Skip to content

Commit c3f5e07

Browse files
pwendellpwendell
committed
SPARK-1121: Include avro for yarn-alpha builds
This lets us explicitly include Avro based on a profile for 0.23.X builds. It makes me sad how convoluted it is to express this logic in Maven. @tgraves and @sryza curious if this works for you. I'm also considering just reverting to how it was before. The only real problem was that Spark advertised a dependency on Avro even though it only really depends transitively on Avro through other deps. Author: Patrick Wendell <pwendell@gmail.com> Closes apache#49 from pwendell/avro-build-fix and squashes the following commits: 8d6ee92 [Patrick Wendell] SPARK-1121: Add avro to yarn-alpha profile
1 parent fd31adb commit c3f5e07

File tree

18 files changed

+234
-12
lines changed

18 files changed

+234
-12
lines changed

‎bagel/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project Bagel</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<dependencies>
3549
<dependency>
3650
<groupId>org.apache.spark</groupId>

‎core/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project Core</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
35+
a Hadoop 0.23.X issue -->
36+
<profiles>
37+
<profile>
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<dependencies>
3549
<dependency>
3650
<groupId>org.apache.hadoop</groupId>

‎docs/building-with-maven.md‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,3 @@ The maven build includes support for building a Debian package containing the as
7676
$ mvn -Pdeb -DskipTests clean package
7777

7878
The debian package can then be found under assembly/target. We added the short commit hash to the file name so that we can distinguish individual packages built for SNAPSHOT versions.
79-
80-
## A note about Hadoop version 0.23.x
81-
82-
For building spark with hadoop 0.23.x and also yarn, you will have to manually add a dependency on avro (org.apache.avro, avro, 1.7.4).

‎examples/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project Examples</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<repositories>
3549
<repository>
3650
<id>apache-repo</id>

‎external/flume/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project External Flume</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<dependencies>
3549
<dependency>
3650
<groupId>org.apache.spark</groupId>

‎external/kafka/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project External Kafka</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<dependencies>
3549
<dependency>
3650
<groupId>org.apache.spark</groupId>

‎external/mqtt/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project External MQTT</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<repositories>
3549
<repository>
3650
<id>mqtt-repo</id>

‎external/twitter/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project External Twitter</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<dependencies>
3549
<dependency>
3650
<groupId>org.apache.spark</groupId>

‎external/zeromq/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project External ZeroMQ</name>
3232
<url>http://spark.apache.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<dependencies>
3549
<dependency>
3650
<groupId>org.apache.spark</groupId>

‎graphx/pom.xml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,20 @@
3131
<name>Spark Project GraphX</name>
3232
<url>http://spark-project.org/</url>
3333

34+
<profiles>
35+
<profile>
36+
<!-- SPARK-1121: SPARK-1121: Adds an explicit dependency on Avro to work around
37+
a Hadoop 0.23.X issue -->
38+
<id>yarn-alpha</id>
39+
<dependencies>
40+
<dependency>
41+
<groupId>org.apache.avro</groupId>
42+
<artifactId>avro</artifactId>
43+
</dependency>
44+
</dependencies>
45+
</profile>
46+
</profiles>
47+
3448
<dependencies>
3549
<dependency>
3650
<groupId>org.apache.spark</groupId>

0 commit comments

Comments
 (0)