Improved build configuration #480

witgo · 2014-04-22T07:35:58Z

1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x
2, Fix SPARK-1491: maven hadoop-provided profile fails to build
3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency
4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces)

AmplabJenkins · 2014-04-22T07:37:55Z

Can one of the admins verify this patch?

srowen · 2014-04-22T08:27:20Z

pom.xml

@@ -793,6 +831,157 @@
  </build>

  <profiles>
+    <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
+    <profile>


I don't think this copy-and-paste approach is reasonable to maintain. Have you tried writing a rule that triggers on a version range of the hadoop-client?

I do not know how to do, an example?

Scratch that. I tried a bunch of variations on this and can't find a way to activate the profile based on a range. I think it's best to activate this profile explicitly with -P rather than maintain all this duplication, still.

srowen · 2014-04-22T14:34:56Z

pom.xml

@@ -892,10 +1081,11 @@
        <dependency>
          <groupId>org.apache.zookeeper</groupId>
          <artifactId>zookeeper</artifactId>
+          <version>3.4.5</version>


+1, this looks like it was unspecified. Does this version need to vary with Hadoop version? And also in the SBT build?

curator-recipes 2.4.0 => zookeeper 3.4.5
hbase 0.94.6 => zookeeper 3.4.5
kafka_2.10 0.8.0 => zookeeper 3.3.4
hadoop is not directly depend on the zookeeper

I see, this declaration is just there to make zookeeper "provided" in order to not conflict with a cluster's copy/version of ZK from Hadoop. Hadoop version kind of matters in that sense. I wonder if this is actually best left unspecified then, since there is no intent to enforce any particular version of ZK, just make sure it is not in the artifact when built with this profile?

SPARK-1064,PR 102.There is no equivalent feature in sbt.

pwendell · 2014-04-23T17:58:36Z

pom.xml

@@ -793,6 +833,17 @@
  </build>

  <profiles>
+    <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
+    <profile>
+      <id>hadoop-0.23</id>


I think in the past we had trouble defining this profile in the parent pom. The issue was that profiles were not correctly inherited by the sub projects, because profiles are not transative. I don't remember how exactly this manifested though.

I have not found this problem in the test.

pwendell · 2014-04-29T03:54:34Z

docs/building-with-maven.md

@@ -39,6 +39,9 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit
    # Cloudera CDH 4.2.0 with MapReduce v1
    $ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package

+    # Apache Hadoop 0.23.x
+    $ mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package
+
 For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "yarn-alpha" or "yarn" profile and set the "hadoop.version", "yarn.version" property:


now that we support building without YARN... this should say "you can enable" instead of "you should enable"

pwendell · 2014-04-29T04:14:34Z

Jenkins, test this please.

AmplabJenkins · 2014-04-29T04:17:57Z

Merged build triggered.

AmplabJenkins · 2014-04-29T04:18:06Z

Merged build started.

pwendell · 2014-04-29T04:46:01Z

So I looked through this a bunch and ran several builds with different permutations. This looks really good! Thanks @witgo I'm sure it took some time to figure all this out. I'm going to merge this pending tests because I'd like to cut an RC tonight and I don't want to change the build around after we ask people to test stuff.

AmplabJenkins · 2014-04-29T05:35:02Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-29T05:35:02Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14552/

pwendell · 2014-04-29T06:17:22Z

I've merged this, thanks.

1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x 2, Fix SPARK-1491: maven hadoop-provided profile fails to build 3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency 4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces) Author: witgo <witgo@qq.com> Closes #480 from witgo/format_pom and squashes the following commits: 03f652f [witgo] review commit b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence 7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence 0da4bc3 [witgo] merge master d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom e345919 [witgo] add avro dependency to yarn-alpha 77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency 1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 934f24d [witgo] review commit cf46edc [witgo] exclude jruby 06e7328 [witgo] Merge branch 'SparkBuild' into format_pom 99464d2 [witgo] fix maven hadoop-provided profile fails to build 0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x 6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml (cherry picked from commit 030f2c2) Conflicts: sql/catalyst/pom.xml sql/core/pom.xml sql/hive/pom.xml

witgo · 2014-04-29T06:47:56Z

Cool!

@mridulm

Handful of 0.9 fixes This patch addresses a few fixes for Spark 0.9.0 based on the last release candidate. @mridulm gets credit for reporting most of the issues here. Many of the fixes here are based on his work in apache#477 and follow up discussion with him.

1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x 2, Fix SPARK-1491: maven hadoop-provided profile fails to build 3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency 4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces) Author: witgo <witgo@qq.com> Closes apache#480 from witgo/format_pom and squashes the following commits: 03f652f [witgo] review commit b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence 7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence 0da4bc3 [witgo] merge master d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom e345919 [witgo] add avro dependency to yarn-alpha 77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency 1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 934f24d [witgo] review commit cf46edc [witgo] exclude jruby 06e7328 [witgo] Merge branch 'SparkBuild' into format_pom 99464d2 [witgo] fix maven hadoop-provided profile fails to build 0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x 6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml

@mridulm

Handful of 0.9 fixes This patch addresses a few fixes for Spark 0.9.0 based on the last release candidate. @mridulm gets credit for reporting most of the issues here. Many of the fixes here are based on his work in apache#477 and follow up discussion with him. (cherry picked from commit 77b986f) Signed-off-by: Patrick Wendell <pwendell@gmail.com>

###### _excavator_ is a bot for automating changes across repositories. Changes produced by the roomba/latest-gradle-wrapper-oss check. {runtimeCheckDesc} To enable or disable this check, please contact the maintainers of Excavator.

This patch fixes the issue mentioned in PR#1519. The default size of devstack swift is 2G, if we enable octavia or we create some new images into glance during test. We will get failure. The key reason for this is the swift device size is too small. So this patch will increase the size through pass a env parameter in devstack local conf. Close: theopenlab/openlab#222

… incorrect result (apache#475) (apache#480)

witgo added 2 commits April 17, 2014 21:45

Maintain consistent SparkBuild.scala, pom.xml

6851bec

Fix compile spark core error with hadoop 0.23.x

0c6c1fc

witgo changed the title ~~Improved build configuration~~ [WIP]Improved build configuration Apr 22, 2014

srowen reviewed Apr 22, 2014
View reviewed changes

fix maven hadoop-provided profile fails to build

99464d2

srowen reviewed Apr 22, 2014
View reviewed changes

witgo added 5 commits April 22, 2014 23:26

Merge branch 'SparkBuild' into format_pom

06e7328

exclude jruby

cf46edc

review commit

934f24d

Merge branch 'master' of https://github.com/apache/spark into format_pom

1a162d7

Fix org.scala-lang: * inconsistent versions dependency

62d0862

witgo changed the title ~~[WIP]Improved build configuration~~ Improved build configuration Apr 23, 2014

witgo mentioned this pull request Apr 23, 2014

SPARK-1119 and other build improvements #502

Closed

pwendell reviewed Apr 23, 2014
View reviewed changes

witgo added 7 commits April 24, 2014 10:26

Merge branch 'master' of https://github.com/apache/spark into format_pom

77fad08

add avro dependency to yarn-alpha

e345919

Merge branch 'master' of https://github.com/apache/spark into format_pom

d1718ed

merge master

0da4bc3

fix SPARK-1629: Spark Core missing commons-lang dependence

6902c91

Merge branch 'master' of https://github.com/apache/spark into format_pom

7382a07

revert fix SPARK-1629: Spark Core missing commons-lang dependence

bee920d

witgo mentioned this pull request Apr 29, 2014

Remove jruby #583

Closed

pwendell reviewed Apr 29, 2014
View reviewed changes

witgo added 2 commits April 29, 2014 13:01

Merge branch 'master' of https://github.com/apache/spark into format_pom

b452680

review commit

03f652f

asfgit closed this in 030f2c2 Apr 29, 2014

witgo deleted the format_pom branch April 29, 2014 06:46

arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020

[SPARK-508] MapR-DB OJAI Connector for Spark isNull condition returns…

a3852ef

… incorrect result (apache#475) (apache#480)

RolatZhang pushed a commit to RolatZhang/spark that referenced this pull request Aug 15, 2022

KE-37170 fix broadcastManager NPE (apache#480)

c8413c8

RolatZhang pushed a commit to RolatZhang/spark that referenced this pull request Aug 15, 2022

KE-37170 fix broadcastManager NPE (apache#480)

58ba8fe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved build configuration #480

Improved build configuration #480

witgo commented Apr 22, 2014

AmplabJenkins commented Apr 22, 2014

srowen Apr 22, 2014

witgo Apr 22, 2014

srowen Apr 22, 2014

srowen Apr 22, 2014

witgo Apr 22, 2014

srowen Apr 22, 2014

witgo Apr 22, 2014

pwendell Apr 23, 2014

witgo Apr 24, 2014

pwendell Apr 29, 2014

pwendell commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

pwendell commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

pwendell commented Apr 29, 2014

witgo commented Apr 29, 2014

Improved build configuration #480

Improved build configuration #480

Conversation

witgo commented Apr 22, 2014

AmplabJenkins commented Apr 22, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwendell commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

pwendell commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

pwendell commented Apr 29, 2014

witgo commented Apr 29, 2014