[SPARK-5979][SPARK-6031][SPARK-6032][SPARK-6047] Refactoring for --packages -> Move to SparkSubmitDriverBootstrapper #4754

brkyvz · 2015-02-25T01:48:48Z

This PR is an umbrella PR for 3 JIRAs. Here're the explanations:

SPARK-5979: All dependencies with the groupId org.apache.spark passed through --packages, were being excluded from the dependency tree on the assumption that they would be in the assembly jar. This is not the case, therefore the exclusion rules had to be defined more explicitly.
SPARK-6031: When using pyspark or running a python program through spark-submit, py4j was not picking up the dynamically loaded jars on the driver. Moving the code for --packages to SparkSubmitDriverBootstrapper solves this. However, this issue still remains for --jars.
SPARK-6032: Ivy prints a whole lot of logs while retrieving dependencies. These were printed to System.out. Moved the logging to System.err.

@tdas Would you care to try this? I think it should solve your problem

SparkQA · 2015-02-25T03:13:46Z

Test build #27927 has finished for PR 4754 at commit 941c65e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class CheckAnalysis

tdas · 2015-02-25T03:14:32Z

@pwendell Please take a look at this. I think you reviewed the original PR of this feature.

pwendell · 2015-02-25T06:15:52Z

LGTM

…with --jars

brkyvz · 2015-02-25T06:25:34Z

@tdas added a hack to include the jars on --driver-extra-classpath. Can you try your test now?

SparkQA · 2015-02-25T07:24:31Z

Test build #27942 has finished for PR 4754 at commit e3ca1b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-02-25T07:51:53Z

Test build #27943 has finished for PR 4754 at commit 5191f3a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-02-25T09:30:13Z

I tested. Still not working. I enabled verbose logging on spark-submit and saw this

[tdas @ Zion spark2] bin/spark-submit --verbose --master local[4] --repositories https://repository.apache.org/content/repositories/orgapachespark-1069/ --packages org.apache.spark:spark-streaming-kafka_2.10:1.3.0 examples/src/main/python/streaming/kafka_wordcount.py localhost:2181 test
Using properties file: null
Parsed arguments:
  master                  local[4]
  deployMode              null
  executorMemory          null
  executorCores           null
  totalExecutorCores      null
  propertiesFile          null
  driverMemory            null
  driverCores             null
  driverExtraClassPath    null
  driverExtraLibraryPath  null
  driverExtraJavaOptions  null
  supervise               false
  queue                   null
  numExecutors            null
  files                   null
  pyFiles                 null
  archives                null
  mainClass               null
  primaryResource         file:/Users/tdas/Projects/Spark/spark2/examples/src/main/python/streaming/kafka_wordcount.py
  name                    kafka_wordcount.py
  childArgs               [localhost:2181 test]
  jars                    null
  packages                org.apache.spark:spark-streaming-kafka_2.10:1.3.0
  repositories            https://repository.apache.org/content/repositories/orgapachespark-1069/
  verbose                 true

Spark properties used, including those specified through
 --conf and those from the properties file null:



Ivy Default Cache set to: /Users/tdas/.ivy2/cache
The jars for the packages stored in: /Users/tdas/.ivy2/jars
https://repository.apache.org/content/repositories/orgapachespark-1069/ added as a remote repository with the name: repo-1
:: loading settings :: url = jar:file:/Users/tdas/Projects/Spark/spark2/assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.spark#spark-streaming-kafka_2.10 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
    confs: [default]
    found org.apache.spark#spark-streaming-kafka_2.10;1.3.0 in repo-1
    found org.apache.kafka#kafka_2.10;0.8.1.1 in list
    found com.yammer.metrics#metrics-core;2.2.0 in list
    found org.slf4j#slf4j-api;1.7.10 in list
    found org.xerial.snappy#snappy-java;1.1.1.6 in list
    found com.101tec#zkclient;0.3 in list
    found log4j#log4j;1.2.17 in list
    found org.spark-project.spark#unused;1.0.0 in list
:: resolution report :: resolve 370ms :: artifacts dl 17ms
    :: modules in use:
    com.101tec#zkclient;0.3 from list in [default]
    com.yammer.metrics#metrics-core;2.2.0 from list in [default]
    log4j#log4j;1.2.17 from list in [default]
    org.apache.kafka#kafka_2.10;0.8.1.1 from list in [default]
    org.apache.spark#spark-streaming-kafka_2.10;1.3.0 from repo-1 in [default]
    org.slf4j#slf4j-api;1.7.10 from list in [default]
    org.spark-project.spark#unused;1.0.0 from list in [default]
    org.xerial.snappy#snappy-java;1.1.1.6 from list in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
    ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
    confs: [default]
    0 artifacts copied, 8 already retrieved (0kB/7ms)
Main class:
org.apache.spark.deploy.PythonRunner
Arguments:
file:/Users/tdas/Projects/Spark/spark2/examples/src/main/python/streaming/kafka_wordcount.py
/Users/tdas/.ivy2/jars/spark-streaming-kafka_2.10.jar,/Users/tdas/.ivy2/jars/kafka_2.10.jar,/Users/tdas/.ivy2/jars/unused.jar,/Users/tdas/.ivy2/jars/metrics-core.jar,/Users/tdas/.ivy2/jars/snappy-java.jar,/Users/tdas/.ivy2/jars/zkclient.jar,/Users/tdas/.ivy2/jars/slf4j-api.jar,/Users/tdas/.ivy2/jars/log4j.jar
localhost:2181
test
System properties:
SPARK_SUBMIT -> true
spark.submit.pyFiles -> /Users/tdas/.ivy2/jars/spark-streaming-kafka_2.10.jar,/Users/tdas/.ivy2/jars/kafka_2.10.jar,/Users/tdas/.ivy2/jars/unused.jar,/Users/tdas/.ivy2/jars/metrics-core.jar,/Users/tdas/.ivy2/jars/snappy-java.jar,/Users/tdas/.ivy2/jars/zkclient.jar,/Users/tdas/.ivy2/jars/slf4j-api.jar,/Users/tdas/.ivy2/jars/log4j.jar
spark.files -> file:/Users/tdas/Projects/Spark/spark2/examples/src/main/python/streaming/kafka_wordcount.py,file:/Users/tdas/.ivy2/jars/spark-streaming-kafka_2.10.jar,file:/Users/tdas/.ivy2/jars/kafka_2.10.jar,file:/Users/tdas/.ivy2/jars/unused.jar,file:/Users/tdas/.ivy2/jars/metrics-core.jar,file:/Users/tdas/.ivy2/jars/snappy-java.jar,file:/Users/tdas/.ivy2/jars/zkclient.jar,file:/Users/tdas/.ivy2/jars/slf4j-api.jar,file:/Users/tdas/.ivy2/jars/log4j.jar
spark.app.name -> kafka_wordcount.py
spark.jars -> file:/Users/tdas/.ivy2/jars/spark-streaming-kafka_2.10.jar,file:/Users/tdas/.ivy2/jars/kafka_2.10.jar,file:/Users/tdas/.ivy2/jars/unused.jar,file:/Users/tdas/.ivy2/jars/metrics-core.jar,file:/Users/tdas/.ivy2/jars/snappy-java.jar,file:/Users/tdas/.ivy2/jars/zkclient.jar,file:/Users/tdas/.ivy2/jars/slf4j-api.jar,file:/Users/tdas/.ivy2/jars/log4j.jar
spark.master -> local[4]
spark.driver.extraClassPath -> /Users/tdas/.ivy2/jars/spark-streaming-kafka_2.10.jar,/Users/tdas/.ivy2/jars/kafka_2.10.jar,/Users/tdas/.ivy2/jars/unused.jar,/Users/tdas/.ivy2/jars/metrics-core.jar,/Users/tdas/.ivy2/jars/snappy-java.jar,/Users/tdas/.ivy2/jars/zkclient.jar,/Users/tdas/.ivy2/jars/slf4j-api.jar,/Users/tdas/.ivy2/jars/log4j.jar
Classpath elements:
/Users/tdas/.ivy2/jars/spark-streaming-kafka_2.10.jar
/Users/tdas/.ivy2/jars/kafka_2.10.jar
/Users/tdas/.ivy2/jars/unused.jar
/Users/tdas/.ivy2/jars/metrics-core.jar
/Users/tdas/.ivy2/jars/snappy-java.jar
/Users/tdas/.ivy2/jars/zkclient.jar
/Users/tdas/.ivy2/jars/slf4j-api.jar
/Users/tdas/.ivy2/jars/log4j.jar

So i can see that the relevant jars are being added to the classpath elements but pyspark is still unable to find org.apache.spark.streaming.kafka.KafkaUtils (from /Users/tdas/.ivy2/jars/spark-streaming-kafka_2.10.jar).

Lets debug this tomorrow morning.

…dency-fix

tdas · 2015-02-25T18:24:40Z

No I verified the class does exist in jar
On Feb 25, 2015 10:07 AM, "Burak Yavuz" notifications@github.com wrote:

nvm, it should be in spark-streaming-kafka_2.10.jar

—
Reply to this email directly or view it on GitHub
#4754 (comment).

…dency-fix

SparkQA · 2015-02-25T23:58:01Z

Test build #27963 has finished for PR 4754 at commit d9e3cf0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-02-26T02:02:27Z

@tdas @pwendell @andrewor14
This is ready for code review. Moved the resolve method to DriverBootstrapper. In case the DriverBootstrapper is not called, the resolution takes place inside SparkSubmit like before. This should work (tested it with TD's example + my own package examples)

SparkQA · 2015-02-26T02:59:52Z

Test build #27973 has finished for PR 4754 at commit 43c3cb2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-02-26T04:00:39Z

Jenkins, test this again.

brkyvz · 2015-02-26T04:02:21Z

It might not be a flaky test. I might have broken some Yarn feature. I'm
going to check once I get home.
On Feb 25, 2015 8:01 PM, "Tathagata Das" notifications@github.com wrote:

Jenkins, test this again.

—
Reply to this email directly or view it on GitHub
#4754 (comment).

tdas · 2015-02-26T04:03:05Z

Ohh... okay.

…dency-fix

SparkQA · 2015-02-26T05:02:41Z

Test build #622 has finished for PR 4754 at commit 43c3cb2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-02-26T05:12:30Z

@tdas The latest commit fixed the issue, feel free to test

SparkQA · 2015-02-26T08:12:26Z

Test build #27989 has finished for PR 4754 at commit c73aabe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-02-26T08:18:10Z

Test build #27990 has finished for PR 4754 at commit 7f958c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-02-26T16:39:00Z

Test build #28007 has finished for PR 4754 at commit b7a9e93.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-02-26T16:52:00Z

This passed locally. What the...
On Feb 26, 2015 8:39 AM, "UCB AMPLab" notifications@github.com wrote:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28007/
Test FAILed.

—
Reply to this email directly or view it on GitHub
#4754 (comment).

andrewor14 · 2015-02-26T17:42:09Z

retest this please

SparkQA · 2015-02-26T18:49:09Z

Test build #28011 has finished for PR 4754 at commit b7a9e93.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit b7a9e93.

SparkQA · 2015-02-26T21:38:36Z

Test build #28015 has finished for PR 4754 at commit 994869e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-02-26T22:25:49Z

Test build #28019 has finished for PR 4754 at commit 44dbf67.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-02-26T22:34:46Z

Flaky test this time... @tdas, can you have this retested please?

srowen · 2015-02-26T22:35:50Z

Jenkins, retest this please

SparkQA · 2015-02-27T00:09:27Z

Test build #28023 has finished for PR 4754 at commit 44dbf67.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-02-27T00:17:54Z

@srowen Thank you!

tdas · 2015-02-27T00:38:28Z

@brkyvz I think you need to address a couple of more JIRAs in this PR. 4 aint enough ;)

pwendell · 2015-02-27T01:44:51Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmitDriverBootstrapper.scala

+      newClasspath += sys.props("path.separator") + 
+        resolvedMavenCoordinates.mkString(sys.props("path.separator"))
+      submitArgs = 
+        Array("--packages-resolved", resolvedMavenCoordinates.mkString(",")) ++ submitArgs


Can we thread this through using an environment variable _PACKAGES_RESOLVED? Having this as an extra flag forces you to make args here mutable, which is sort of strange.

pwendell tdas This is the safer parts of PR #4754: - SPARK-5979: All dependencies with the groupId `org.apache.spark` passed through `--packages`, were being excluded from the dependency tree on the assumption that they would be in the assembly jar. This is not the case, therefore the exclusion rules had to be defined more explicitly. - SPARK-6032: Ivy prints a whole lot of logs while retrieving dependencies. These were printed to `System.out`. Moved the logging to `System.err`. Author: Burak Yavuz <brkyvz@gmail.com> Closes #4802 from brkyvz/simple-streaming-fix and squashes the following commits: e0f38cb [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into simple-streaming-fix bad921c [Burak Yavuz] [SPARK-5979][SPARK-6032] Smaller safer fix (cherry picked from commit 6d8e5fb) Signed-off-by: Patrick Wendell <patrick@databricks.com>

pwendell tdas This is the safer parts of PR #4754: - SPARK-5979: All dependencies with the groupId `org.apache.spark` passed through `--packages`, were being excluded from the dependency tree on the assumption that they would be in the assembly jar. This is not the case, therefore the exclusion rules had to be defined more explicitly. - SPARK-6032: Ivy prints a whole lot of logs while retrieving dependencies. These were printed to `System.out`. Moved the logging to `System.err`. Author: Burak Yavuz <brkyvz@gmail.com> Closes #4802 from brkyvz/simple-streaming-fix and squashes the following commits: e0f38cb [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into simple-streaming-fix bad921c [Burak Yavuz] [SPARK-5979][SPARK-6032] Smaller safer fix

pwendell · 2015-02-28T07:01:01Z

@brkyvz let's close this issue for now and keep it in our back pocket. We can use it if we decide to put this in the 1.3 branch down the line.

brkyvz added 3 commits February 24, 2015 17:43

[SPARK-5979] Made exclusions more refined

838d301

added fake core dependency that should be excluded

9bacee8

removed unused imports

941c65e

Rerouted ivy output to system.err

e3ca1b7

add ivy jars to driver-extra-classpath explicitly for pyspark issues …

5191f3a

…with --jars

Merge branch 'master' of github.com:apache/spark into streaming-depen…

0070af8

…dency-fix

brkyvz added 2 commits February 25, 2015 14:52

removed redundant commands

d9e3cf0

Merge branch 'master' of github.com:apache/spark into streaming-depen…

c9c11ef

…dency-fix

brkyvz added 2 commits February 25, 2015 17:53

added windows support

f25c55b

removed debug line

43c3cb2

tdas mentioned this pull request Feb 26, 2015

[SPARK-6027][SPARK-5546] Fixed --jar and --packages not working for KafkaUtils and improved error message #4779

Closed

brkyvz added 3 commits February 25, 2015 20:23

Merge branch 'master' of github.com:apache/spark into streaming-depen…

a4bc489

…dency-fix

tried to fix yarn test

12e2764

prepend --packages-resolved instead of append

f06a754

brkyvz changed the title ~~[SPARK-5979] Made --package exclusions more refined~~ [SPARK-5979][SPARK-6031][SPARK-6032] Refactoring for --packages -> Move to SparkSubmitDriverBootstrapper Feb 26, 2015

brkyvz added 2 commits February 25, 2015 22:47

address review comments

c73aabe

fixed wrong comment

7f958c1

move if condition

b7a9e93

brkyvz added 2 commits February 26, 2015 12:15

Revert "move if condition"

994869e

This reverts commit b7a9e93.

fix windows checking

44dbf67

brkyvz changed the title ~~[SPARK-5979][SPARK-6031][SPARK-6032] Refactoring for --packages -> Move to SparkSubmitDriverBootstrapper~~ [SPARK-5979][SPARK-6031][SPARK-6032][SPARK-6047] Refactoring for --packages -> Move to SparkSubmitDriverBootstrapper Feb 27, 2015

pwendell reviewed Feb 27, 2015
View reviewed changes

brkyvz mentioned this pull request Feb 27, 2015

[SPARK-5979][SPARK-6032] Smaller safer --packages fix #4802

Closed

brkyvz closed this Feb 28, 2015

brkyvz deleted the streaming-dependency-fix branch February 3, 2019 20:56

[SPARK-5979][SPARK-6031][SPARK-6032][SPARK-6047] Refactoring for --packages -> Move to SparkSubmitDriverBootstrapper #4754

[SPARK-5979][SPARK-6031][SPARK-6032][SPARK-6047] Refactoring for --packages -> Move to SparkSubmitDriverBootstrapper #4754

Uh oh!

Conversation

brkyvz commented Feb 25, 2015

Uh oh!

SparkQA commented Feb 25, 2015

Uh oh!

tdas commented Feb 25, 2015

Uh oh!

pwendell commented Feb 25, 2015

Uh oh!

brkyvz commented Feb 25, 2015

Uh oh!

SparkQA commented Feb 25, 2015

Uh oh!

SparkQA commented Feb 25, 2015

Uh oh!

tdas commented Feb 25, 2015

Uh oh!

tdas commented Feb 25, 2015

Uh oh!

SparkQA commented Feb 25, 2015

Uh oh!

brkyvz commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

tdas commented Feb 26, 2015

Uh oh!

brkyvz commented Feb 26, 2015

Uh oh!

tdas commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

brkyvz commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

brkyvz commented Feb 26, 2015

Uh oh!

andrewor14 commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 26, 2015

Uh oh!

brkyvz commented Feb 26, 2015

Uh oh!

srowen commented Feb 26, 2015

Uh oh!

SparkQA commented Feb 27, 2015

Uh oh!

brkyvz commented Feb 27, 2015

Uh oh!

tdas commented Feb 27, 2015

Uh oh!

pwendell Feb 27, 2015

Choose a reason for hiding this comment

Uh oh!

pwendell commented Feb 28, 2015

Uh oh!

Uh oh!