Skip to content

[minor] [core] Warn users who try to cache RDDs with dynamic allocation on. #5751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[minor] [core] Warn users who try to cache RDDs with dynamic allocation on. #5751

wants to merge 2 commits into from

Conversation

vanzin
Copy link
Contributor

@vanzin vanzin commented Apr 28, 2015

No description provided.

@vanzin
Copy link
Contributor Author

vanzin commented Apr 28, 2015

Example:

scala> val rdd = sc.parallelize(1 to 100).cache()
15/04/28 12:03:34 WARN SparkContext: Dynamic allocation currently does not support cached RDDs. Cached data for RDD 0 will be deallocated when executors are removed.

sryza
sryza reviewed Apr 28, 2015
View reviewed changes
core/src/main/scala/org/apache/spark/SparkContext.scala
_executorAllocationManager.foreach { _ =>
logWarning(
s"Dynamic allocation currently does not support cached RDDs. Cached data for RDD " +
s"${rdd.id} will be deallocated when executors are removed.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we say "lost" instead of "deallocated"? Deallocate has a connotation to me of purposely returned to a pool, where as this is more a situation of unpurposely returned to nowhere accessible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about lost but thought it would be too strong, since the data can be recomputed. But sure, it sounds more correct than "deallocated".

@sryza
Copy link
Contributor

sryza commented Apr 28, 2015

Minor comment. Otherwise LGTM.

@andrewor14
Copy link
Contributor

LGTM

1 similar comment
@pwendell
Copy link
Contributor

LGTM

@SparkQA
Copy link

SparkQA commented Apr 28, 2015

Test build #31152 has finished for PR 5751 at commit 9efb9da.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

asfgit pushed a commit that referenced this pull request Apr 28, 2015
[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocati…
7e21026 
…on on.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #5751 from vanzin/cached-rdd-warning and squashes the following commits:

554cc07 [Marcelo Vanzin] Change message.
9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
@asfgit asfgit closed this in 28b1af7 Apr 28, 2015
@andrewor14
Copy link
Contributor

I merged this into master 1.3 and all the way back to 1.2 thanks.

asfgit pushed a commit that referenced this pull request Apr 28, 2015
[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocati…
d208047 
…on on.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #5751 from vanzin/cached-rdd-warning and squashes the following commits:

554cc07 [Marcelo Vanzin] Change message.
9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
@SparkQA
Copy link

SparkQA commented Apr 28, 2015

Test build #31158 has finished for PR 5751 at commit 554cc07.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31158/
Test FAILed.

@vanzin vanzin deleted the cached-rdd-warning branch May 4, 2015 21:43
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 14, 2015
[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocati…
7104af3 
…on on.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#5751 from vanzin/cached-rdd-warning and squashes the following commits:

554cc07 [Marcelo Vanzin] Change message.
9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
@andrewor14 andrewor14 mentioned this pull request May 26, 2015
Closed
asfgit pushed a commit that referenced this pull request May 28, 2015
[MINOR] [CORE] Warn about caching if dynamic allocation is enabled (1.3)
33e1539 
This is a resubmit of #5751 for branch-1.3. The previous cherry-pick caused a build break that was later [reverted](2254576). Originally written by vanzin.

Author: Andrew Or <andrew@databricks.com>

Closes #6421 from andrewor14/warn-da-cache-1.3 and squashes the following commits:

25cbb53 [Andrew Or] If DA is enabled, warn about caching
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocati…
361a70a 
…on on.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes apache#5751 from vanzin/cached-rdd-warning and squashes the following commits:

554cc07 [Marcelo Vanzin] Change message.
9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@vanzin @sryza @andrewor14 @pwendell @SparkQA @AmplabJenkins