Skip to content

[SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting#18354

Closed
bdrillard wants to merge 4 commits intoapache:branch-2.1from
bdrillard:class_splitting_2.1
Closed

[SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting#18354
bdrillard wants to merge 4 commits intoapache:branch-2.1from
bdrillard:class_splitting_2.1

Conversation

@bdrillard
Copy link

@bdrillard bdrillard commented Jun 19, 2017

What changes were proposed in this pull request?

This is a backport patch for Spark 2.1.x of the class splitting feature over excess generated code as was merged in #18075.

How was this patch tested?

The same test provided in #18075 is included in this patch.

@bdrillard bdrillard changed the title Class splitting 2.1 [SPARK-18016][SQL][Catalyst] Backport of Class Splitting, #18075 Jun 19, 2017
@bdrillard bdrillard changed the title [SPARK-18016][SQL][Catalyst] Backport of Class Splitting, #18075 [SPARK-18016][SQL][Catalyst] Backport of Class Splitting Jun 19, 2017
@maor121
Copy link

maor121 commented Jun 20, 2017

This is a very important fix for a very common use case. In our company we had to create a workaround in many different cases. Are there any plans to merge this into 2.0.x as well?
We cannot upgrade to 2.1, and we really need this fix on 2.0.x as well :)

@bdrillard
Copy link
Author

bdrillard commented Jun 20, 2017

@maor121 Yeah, I should be able to backpatch to 2.0.x as well in a separate PR.

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Jun 20, 2017

Hi, @bdrillard .
Did you make a backport for branch-2.2?

@bdrillard
Copy link
Author

bdrillard commented Jun 20, 2017

@dongjoon-hyun I had not yet. I hadn't realized master was already on a 2.3.0-SNAPSHOT. I can also write a backport for branch-2.2.

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Jun 20, 2017 •
edited
Loading

Yep. To backport, we must land this on branch-2.2 first.
In addition, you can use the same title except branch name like the following.

  • [SPARK-18016][SQL][CATALYST][BRANCH-2.2] Code Generation: Constant Pool Limit - Class Splitting
  • [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting

@bdrillard
Copy link
Author

bdrillard commented Jun 20, 2017

@dongjoon-hyun Great, thanks for the heads up. I'll prepare a branch-2.2 backport PR and circle back to this PR with the link as confirmation.

@bdrillard bdrillard changed the title [SPARK-18016][SQL][Catalyst] Backport of Class Splitting [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Jun 20, 2017
@bdrillard
Copy link
Author

bdrillard commented Jun 21, 2017

@dongjoon-hyun I've prepared the PR for the branch-2.2 backport, #18377.

@cloud-fan
Copy link
Contributor

cloud-fan commented Jun 21, 2017

ok to test

@SparkQA
Copy link

SparkQA commented Jun 21, 2017

Test build #78394 has finished for PR 18354 at commit 5685072.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 21, 2017

Test build #78401 has finished for PR 18354 at commit 51a59cb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Jun 22, 2017
@cloud-fan
[SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Po…
6b37c86 
…ol Limit - Class Splitting

## What changes were proposed in this pull request?

This is a backport patch for Spark 2.1.x of the class splitting feature over excess generated code as was merged in #18075.

## How was this patch tested?

The same test provided in #18075 is included in this patch.

Author: ALeksander Eskilson <alek.eskilson@cerner.com>

Closes #18354 from bdrillard/class_splitting_2.1.
@cloud-fan
Copy link
Contributor

cloud-fan commented Jun 22, 2017

thanks, merging to 2.1!

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 24, 2017

@bdrillard, sounds properly backported. Would you close this?

@manojlds
Copy link

manojlds commented Jul 24, 2017

Finding it difficult to track. Is this available in a 2.1 release?

@cloud-fan
Copy link
Contributor

cloud-fan commented Jul 24, 2017

it was backported and then reverted because it's too risky, we can close this PR.

@manojlds
Copy link

manojlds commented Jul 24, 2017

@cloud-fan - thanks. Is there any known workaround for this?

@bdrillard bdrillard mentioned this pull request Jul 24, 2017
Closed
@cloud-fan
Copy link
Contributor

cloud-fan commented Jul 27, 2017

Unfortunately there is no workaround... You can upgrade to spark 2.3 or do the backport and make a new release in your company. cc @rxin any ideas for this case?

@HyukjinKwon HyukjinKwon mentioned this pull request Jul 31, 2017
Closed
@asfgit asfgit closed this in 3a45c7f Aug 5, 2017
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon @zifeif2
[INFRA] Close stale PRs
6639102 
## What changes were proposed in this pull request?

This PR proposes to close stale PRs, mostly the same instances with apache#18017

Closes apache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory …
Closes apache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage.
Closes apache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation
Closes apache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers
Closes apache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key…
Closes apache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples
Closes apache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python
Closes apache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage
Closes apache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins
Closes apache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP]
Closes apache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job
Closes apache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable
Closes apache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator
Closes apache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset
Closes apache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns
Closes apache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work
Closes apache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly
Closes apache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column
Closes apache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService
Closes apache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication
Closes apache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone
Closes apache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000)
Closes apache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos
Closes apache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table
Closes apache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit…
Closes apache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex
Closes apache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable
Closes apache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting
Closes apache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages
Closes apache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery
Closes apache#18432 - resolve com.esotericsoftware.kryo.KryoException
Closes apache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer
Closes apache#18585 - SPARK-21359
Closes apache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala

Added:
Closes apache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I…
Closes apache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0
Closes apache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to …
Closes apache#18667 - Fix the simpleString used in error messages
Closes apache#18782 - Branch 2.1

Added:
Closes apache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads

Added:
Closes apache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread
Closes apache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable
Closes apache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server

Added:
Closes apache#18827 - Merge pull request 1 from apache/master

## How was this patch tested?

N/A

Author: hyukjinkwon <gurwls223@gmail.com>

Closes apache#18780 from HyukjinKwon/close-prs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@bdrillard @maor121 @dongjoon-hyun @cloud-fan @SparkQA @HyukjinKwon @manojlds