[SPARK-23654][BUILD] remove jets3t as a dependency of spark

[steveloughran]

What changes were proposed in this pull request?

With the update of bouncy-castle JAR in Spark 2.3; jets3t doesn't work any more; hence the s3n
connector to S3 is dead. Only one person has noticed so far. The hadoop s3n connector is never going to be updated to work with a later version of jets3t, instead the code has just been cut from hadoop 3.x entirely..

This patch remove the declarations of jets3t from the POMs which include it (root and spark-core), so it is not being packaged up.

Of the transitive dependencies, one is pinned at the same version, the others removed.

• javax.activation is kept at version 1.1.1. by an explicit import; without this it would be downgraded to version 1.0.
• bcprov-jdk15on-1.58.jar, base64-2.3.8.jar, java-xmlbuilder-1.1.jar are all removed. They are not directly used in Spark.

How was this patch tested?

Seeing what jenkins has to say about the missing/changed dependencies.

[SparkQA]

Test build #89800 has finished for PR 21146 at commit c011acc.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

As promised, dependencies fail

diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/pr-deps/spark-deps-hadoop-2.6
index 32b2e4f..609eeb9 100644
--- a/dev/deps/spark-deps-hadoop-2.6
+++ b/dev/pr-deps/spark-deps-hadoop-2.6
@@ -1,7 +1,7 @@
 JavaEWAH-0.3.2.jar
 RoaringBitmap-0.5.11.jar
 ST4-4.0.4.jar
-activation-1.1.1.jar
+activation-1.1.jar
 aircompressor-0.8.jar
 antlr-2.7.7.jar
 antlr-runtime-3.4.jar
@@ -21,8 +21,6 @@ automaton-1.11-8.jar
 avro-1.7.7.jar
 avro-ipc-1.7.7.jar
 avro-mapred-1.7.7-hadoop2.jar
-base64-2.3.8.jar
-bcprov-jdk15on-1.58.jar
 bonecp-0.8.0.RELEASE.jar
 breeze-macros_2.11-0.13.2.jar
 breeze_2.11-0.13.2.jar
@@ -101,7 +99,6 @@ jackson-module-paranamer-2.7.9.jar
 jackson-module-scala_2.11-2.6.7.1.jar
 jackson-xc-1.9.13.jar
 janino-3.0.8.jar
-java-xmlbuilder-1.1.jar
 javassist-3.18.1-GA.jar
 javax.annotation-api-1.2.jar
 javax.inject-1.jar

[srowen]

Seems OK to remove if it's really not being used, except that it means it's not leaking onto the user classpath anymore, right? that is, this can potentially cause user code to stop working if this goes in 2.4, or am I missing something? Could go in for 3.0. The other dependency changes seem OK.

[steveloughran]

It's not going to stop user code from working as the bouncy castle version on the classpath means that Jets3t doesn't actually work.

The fact that nobody has complained about this must count as a metric of how many people use jets3t :)

More of an issue is the fact that the httpclient in 2.3+ isn't compatible with the AWS SDK in Hadoop 2.8.x. ...

[steveloughran]

The transitive dependencies are a separate issue. Jets3t pulls in 3 JARs which nothing else seems to need, but which transitively go onto the spark CP

downgraded

• javax.activation : could/should be bumped up to 1.1.1 again;

missing

• bcprov-jdk15on-1.58.jar ? really part of bouncy castle
• base64-2.3.8.jar, There are so many base-64 encoders on the average classpath nobody will be short of that one
• java-xmlbuilder-1.1.jar ? there are newer versions; removing it and letting recipients choose their own would be wiser

[srowen]

It seems reasonable to proceed to remove this, and just reflect the rest of the dep changes accordingly.

I wonder if updating bcprov actually makes it work or something?

Or, could one say that updating bouncy castle at some point was an error that caused this to stop working? if it was long ago, well, maybe no use in undoing it. But if it happened very recently maybe we need to revisit.

[steveloughran]

There's usually good reasons for upgrading crypto stuff like bouncy castle; nothing to feel bad about.

How about I take this patch & add the explicit activation 1.1.1 ref to reinstate it, leave the rest out?

[srowen]

I guess leave a note about why the activation artifact is managed there. Safer to keep it I guess. That artifact will almost surely not change.

[steveloughran]

BTW, the activation framework (primariy used for some mime type stuff) is still being developed, now on github at @javaee https://github.com/javaee/activation. At least it is being maintained

[steveloughran]

OK, I've reinstated javax.activation 1.1.1 as an export from spark core (over v 1.1), point to this JIRA in the comments, and updated the -deps lists to remove the others.

Removing jets3t is the right thing to do. It's never going to work again & while the AWS SDK is equally troublesome to keep up to date, the hadoop-aws's move to a shaded JAR removes transitive dependency conflict as a source of friction and pain.

[srowen]

This is fine; if it's a version only ever used once, I also find it OK to inline to avoid one level of indirection.

[steveloughran]

I'll inline. It's not like its one of those dependencies that anyone doing their own build will ever want to change.

[srowen]

Hm, so bouncy castle is also removed? Maybe I miss what bcprov is, but I think it's the "bouncycastle provider". If so, the license in licenses-binary/ and in LICENSE-binary should be removed. And there's an entry separately in the parent POM that brings this in on account of jets3t, it says. That can go?

[steveloughran]

if there's nothing else for bouncy castle then it's cleaner for classpaths if it goes, as for imports.

looking at Hadoop, it it pulls in bcprov-jdk16, but only for testing. and with out any explicit calls on bouncy castle APIs that I can see. Presumably its registered as a crypto

     <dependency>
       <groupId>org.bouncycastle</groupId>
       <artifactId>bcprov-jdk16</artifactId>
       <version>1.46</version>
       <scope>test</scope>
     </dependency>

If it is going, I should update this PR & JIRA name to make clear it is gone

[srowen]

Sounds like it's already been removed, so the other 'remnants' should go.

[srowen]

@steveloughran I'll merge this if you can remove the bouncy castle license info and take one more pass at comments here.

[SparkQA]

Test build #92614 has finished for PR 21146 at commit 965c73c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

@steveloughran since I wanted to expedite this -- it relates to ECCNs which I'm fixing up -- I opened #22081 WDYT?

[steveloughran]

closing as #22081 supercedes it

[SparkQA]

Test build #94702 has finished for PR 21146 at commit 9adc0ec.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[steveloughran]

FYI, I just did a kinesis test run with this PR on a JVM with the unlimited JCE installed (explicitly verified by shasum of the relevant JARs); failure with cert errors.

success on second attempt after a Kinesis throttling exception
- retry success on second attempt after a Kinesis dependency exception
- retry failed after a shutdown exception
- retry failed after an invalid state exception
- retry failed after unexpected exception
- retry failed after exhausting all retries
WithAggregationKinesisBackedBlockRDDSuite:
[2018-08-13 22:01:12.872175] [0x000070000b6f9000] [info] [kinesis_producer.cc:79] Created pipeline for stream "KinesisTestUtils-3116048591482212471"
[2018-08-13 22:01:12.872918] [0x000070000b6f9000] [info] [shard_map.cc:83] Updating shard map for stream "KinesisTestUtils-3116048591482212471"
[2018-08-13 22:01:13.040026] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to monitoring.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:13.054034] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:15.943229] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:18.924473] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:21.919673] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:25.121685] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:27.925785] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:30.917030] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:33.960962] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:36.926987] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:39.912528] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
[2018-08-13 22:01:42.911743] [0x000070000b905000] [error] [http_client.cc:148] Failed to open connection to kinesis.us-west-2.amazonaws.com:443 : certificate verify failed
org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite *** ABORTED ***
  java.lang.IllegalArgumentException: requirement failed: Need data to be sent to multiple shards
  at scala.Predef$.require(Predef.scala:224)
  at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests$$anonfun$beforeAll$1.apply$mcV$sp(KinesisBackedBlockRDDSuite.scala:47)
  at org.apache.spark.streaming.kinesis.KinesisFunSuite$class.runIfTestsEnabled(KinesisFunSuite.scala:41)
  at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:25)
  at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDTests.beforeAll(KinesisBackedBlockRDDSuite.scala:42)
  at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
  at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
  at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
  at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1210)
  at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1257)
  ...

the assert failure is possibly a followon from the previous problem

[steveloughran]

And if you bump up the kinesis client and AWS SDK version to 1.11.271, those failures go away.

Run completed in 15 minutes, 28 seconds.
Total number of tests run: 59
Suites: completed 9, aborted 0
Tests: succeeded 59, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:09 min
[INFO] Finished at: 2018-08-13T22:35:17-07:00
[INFO] ------------------------------------------------------------------------

[SparkQA]

Test build #94770 has finished for PR 21146 at commit 3ac1d63.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #94769 has finished for PR 21146 at commit 853ee1c.

• This patch fails from timeout after a configured wait of `300m`.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4258 has finished for PR 21146 at commit 3ac1d63.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

Closing now we have a test run with the combination of: No jets3t, no bouncy castle, upgraded kinesis. all the kinesis tests now run