[SPARK-16751] Upgrade derby to 10.12.1.1 #14379

a-roberts · 2016-07-27T12:53:56Z

What changes were proposed in this pull request?

Version of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1

The CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs

How was this patch tested?

Existing tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder.

I then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present.

I don't know if this would also remove it from the assembly jar in our 1.x branches.

## What changes were proposed in this pull request? Version of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1 The CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs ## How was this patch tested? Existing tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder. I then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present. I don't know if this would also remove it from the assembly jar in our 1.x branches.

SparkQA · 2016-07-27T12:59:38Z

Test build #62922 has finished for PR 14379 at commit 910520e.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

a-roberts · 2016-07-27T13:05:07Z

Noticed the /home/jenkins/workspace/SparkPullRequestBuilder/dev/test-dependencies.sh problem, need to regen (I slipped up here when upgrading Hadoop last time too), on the case...

shellberg · 2016-07-27T13:15:31Z

Thanks Adam for putting together the PR - your more efficient at it than me ;-) - to propose upgrading this component and responding to my developer mailing list posting. I agree that this affects multiple prior Spark releases, and hoping this would make the cut for content towards a 1.6.x release at least too.

a-roberts · 2016-07-27T13:31:01Z

dev/test-dependencies.sh --replace-manifest then git status to see the changed files, updated the PR

a-roberts · 2016-07-27T13:32:30Z

Jenkins retest this please

rxin · 2016-07-27T15:13:06Z

Does this break the runtime for Spark SQL running locally with Hive support, when there is no external catalog setup?

SparkQA · 2016-07-27T15:18:35Z

Test build #62923 has finished for PR 14379 at commit 8016840.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-27T15:42:22Z

Test build #62924 has finished for PR 14379 at commit 8016840.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-07-27T16:45:29Z

Yeah I don't think you can remove derby from packaging. If you don't have a metastore server, Spark (well, the Hive libraries) will create a local derby database and use it.

a-roberts · 2016-07-27T17:03:54Z

Interesting point, and you'd think that requires derby classes, out of curiosity what actually happens and what commands would we use to hit the problem?

I've noticed with bin/spark-shell we get a metastore_db folder and derby.log, without derby on our CP do we get NoClassDefFound exceptions? I can give this a try tomorrow if there's no quick answer here

If this is the case then we can simply remove the test I added and let's ship derby 10.12.1.1 not 10.11.1.1 instead

vanzin · 2016-07-27T17:08:43Z

I don't know what exact error you'd get, but you would almost certainly get some error. Removing the test scope (and updating the PR title) should be all that's needed here.

We actually do want to include it in the Spark distribution

SparkQA · 2016-07-27T21:29:47Z

Test build #62940 has finished for PR 14379 at commit 3233b15.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-27T23:22:45Z

Test build #62943 has finished for PR 14379 at commit f3815cf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-07-27T23:24:54Z

Oh I see, so this is actually packaged, not strictly part of tests. Well we should do it anyway. Not sure if the risk actually surfaces in Spark but better to update and be safe.

a-roberts · 2016-07-28T17:36:42Z

One failure at https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62943/testReport/

which looks to be a timeout or flaky test and not caused by this PR, let's test once more to see if it consistent with this as the "age" on Jenkins doesn't help (usually gives us a history of this test failing)

[info] DirectKafkaStreamSuite:
[info] - basic stream receiving with multiple topics and smallest starting offset (6 seconds, 566 milliseconds)
[info] - pattern based subscription *** FAILED *** (21 seconds, 148 milliseconds)

a-roberts · 2016-07-28T17:36:58Z

Jenkins retest this please

SparkQA · 2016-07-28T19:44:55Z

Test build #62973 has finished for PR 14379 at commit f3815cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Version of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1 The CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs Existing tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder. I then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present. I don't know if this would also remove it from the assembly jar in our 1.x branches. Author: Adam Roberts <aroberts@uk.ibm.com> Closes #14379 from a-roberts/patch-4. (cherry picked from commit 04a2c07) Signed-off-by: Sean Owen <sowen@cloudera.com>

srowen · 2016-07-29T11:46:49Z

Merged to master/2.0/1.6

Version of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1 The CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs Existing tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder. I then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present. I don't know if this would also remove it from the assembly jar in our 1.x branches. Author: Adam Roberts <aroberts@uk.ibm.com> Closes #14379 from a-roberts/patch-4. (cherry picked from commit 04a2c07) Signed-off-by: Sean Owen <sowen@cloudera.com>

srowen · 2016-07-29T13:06:07Z

Shoot, this made the lint check fail in 1.6 since there was a hadoop-1 dependency file to update there too. I'll hot-fix it forward, as that's an easy and trivial thing to correct.

shellberg · 2016-07-29T13:07:50Z

Thanks for merging into 1.6.x and the 2.X lines.

Version of derby upgraded based on important security info at VersionEye. Test scope added so we don't include it in our final package anyway. NB: I think this should be backported to all previous releases as it is a security problem https://www.versioneye.com/java/org.apache.derby:derby/10.11.1.1 The CVE number is 2015-1832. I also suggest we add a SECURITY tag for JIRAs Existing tests with the change making sure that we see no new failures. I checked derby 10.12.x and not derby 10.11.x is downloaded to our ~/.m2 folder. I then used dev/make-distribution.sh and checked the dist/jars folder for Spark 2.0: no derby jar is present. I don't know if this would also remove it from the assembly jar in our 1.x branches. Author: Adam Roberts <aroberts@uk.ibm.com> Closes apache#14379 from a-roberts/patch-4. (cherry picked from commit 04a2c07) Signed-off-by: Sean Owen <sowen@cloudera.com> (cherry picked from commit b6f6075)

… 10.12.1.1 security fix ## What changes were proposed in this pull request? See #14379 ; I failed to note in back-porting to 1.6 that an additional Hadoop 1 deps file would need to be updated. This makes that change. ## How was this patch tested? Jenkins tests. Author: Sean Owen <sowen@cloudera.com> Closes #14403 from srowen/SPARK-16751.2.

… 10.12.1.1 security fix ## What changes were proposed in this pull request? See apache#14379 ; I failed to note in back-porting to 1.6 that an additional Hadoop 1 deps file would need to be updated. This makes that change. ## How was this patch tested? Jenkins tests. Author: Sean Owen <sowen@cloudera.com> Closes apache#14403 from srowen/SPARK-16751.2. (cherry picked from commit 03913af)

a-roberts added 5 commits July 27, 2016 14:29

Remove derby dependency for Hadoop 2.2 profile

3c8ad8f

Remove derby dependency for Hadoop 2.3 profile

8a14014

Remove derby dependency for Hadoop 2.4 profile

dd72a1b

Remove derby dependency for Hadoop 2.6 profile

404209d

Remove derby dependency for Hadoop 2.7 profile

8016840

Revert test scope change for derby

3233b15

We actually do want to include it in the Spark distribution

a-roberts changed the title ~~[SPARK-16751] Upgrade Derby, remove from package~~ [SPARK-16751] Upgrade derby to 10.12.1.1 Jul 27, 2016

a-roberts added 5 commits July 27, 2016 22:43

Add updated derby (10.12.1.1) in 2.2 deps file

0d7dead

Add updated derby (10.12.1.1) in 2.3 deps file

67d5c28

Add updated derby (10.12.1.1) in 2.4 deps file

9765559

Add updated derby (10.12.1.1) in 2.6 deps file

f8573fd

Add updated derby (10.12.1.1) in 2.7 deps file

f3815cf

asfgit closed this in 04a2c07 Jul 29, 2016

srowen mentioned this pull request Jul 29, 2016

[SPARK-16751] [HOTFIX] Also update hadoop-1 deps file to reflect derby 10.12.1.1 security fix #14403

Closed

[SPARK-16751] Upgrade derby to 10.12.1.1 #14379

[SPARK-16751] Upgrade derby to 10.12.1.1 #14379

Uh oh!

Conversation

a-roberts commented Jul 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jul 27, 2016

Uh oh!

a-roberts commented Jul 27, 2016

Uh oh!

shellberg commented Jul 27, 2016

Uh oh!

a-roberts commented Jul 27, 2016

Uh oh!

a-roberts commented Jul 27, 2016

Uh oh!

rxin commented Jul 27, 2016

Uh oh!

SparkQA commented Jul 27, 2016

Uh oh!

SparkQA commented Jul 27, 2016

Uh oh!

vanzin commented Jul 27, 2016

Uh oh!

a-roberts commented Jul 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanzin commented Jul 27, 2016

Uh oh!

SparkQA commented Jul 27, 2016

Uh oh!

SparkQA commented Jul 27, 2016

Uh oh!

srowen commented Jul 27, 2016

Uh oh!

a-roberts commented Jul 28, 2016

Uh oh!

a-roberts commented Jul 28, 2016

Uh oh!

SparkQA commented Jul 28, 2016

Uh oh!

srowen commented Jul 29, 2016

Uh oh!

srowen commented Jul 29, 2016

Uh oh!

shellberg commented Jul 29, 2016

Uh oh!

Uh oh!

a-roberts commented Jul 27, 2016 •

edited

Loading

a-roberts commented Jul 27, 2016 •

edited

Loading