Skip to content

Commit 898cde5

Browse files
WenboZhaoCurtis Howard
authored andcommitted
Update Spark build instruction (apache#8)
(cherry picked from commit b406258)
1 parent 5872296 commit 898cde5

File tree

2 files changed

+16
-16
lines changed

2 files changed

+16
-16
lines changed

README.md

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -7,32 +7,32 @@ In order to build this package, you need to build and install `cook jobclient` f
77
git clone https://github.com/twosigma/Cook.git
88
cd Cook/jobclient
99
mvn package
10-
mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file -Dfile=target/cook-jobclient-0.1.0.jar -DpomFile=pom.xml
10+
mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file \
11+
-Dfile=target/cook-jobclient-0.1.2-snapshot.jar \
12+
-DpomFile=pom.xml
1113
```
1214

13-
Now, we are ready to build the Spark distribution as follows.
14-
15+
Now, we are ready to build the Spark distribution as follows. Note that if you are using Java 7, we
16+
probably need to increase heap size used by Maven a little bit. However, if you are on Java 8, you
17+
could ignore the following step.
1518
```
16-
# Install package to local m2 repository
17-
build/mvn install -DskipTests=true -Dscala-2.11 -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.4jco
18-
19-
# Build jar for release without hive support
20-
./make-distribution.sh --tgz --skip-java-test --scala-version 2.11 -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.4jco
21-
22-
# Build jar for release with hive support
23-
./make-distribution.sh --tgz --skip-java-test --scala-version 2.11 -Phive -Phive-thriftserver -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.4jco
19+
export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m"
20+
```
21+
Then, we could
22+
```
23+
./dev/make-distribution.sh --tgz --name hadoop-provided-scala2.11 -Dscala-2.11 -Phadoop-2.6,hadoop-provided,hive -DskipTests
2424
```
2525

2626
The tarball will be created with the hadoop version and scala version
2727
embedded in the tarball name. Additionally, we use `git describe
2828
--tags` to create the spark version, rather than just taking what's in
29-
the pom.xml files. This way, we get a tarball name that looks like
29+
the pom.xml files. This way, we get a tarball name that looks like
3030

31-
spark-1.6.1-31-g9dc4df0-bin-hadoop2.6.0-cdh5.4.4jco-scala2.10.tgz
31+
spark-2.0.2-31-g9dc4df0-bin-hadoop-provided-scala2.11.tgz
3232

3333
rather than
3434

35-
spark-1.6.1-bin-2.6.0-cdh5.4.4jco.tgz
35+
spark-2.0.2-bin-hadoop-provided-scala2.11.tgz
3636

3737
and thus we can manage multiple internal releases on the same upstream
3838
version, and also manage our scala version dependencies appropriately.

core/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,9 @@
3434
<url>http://spark.apache.org/</url>
3535
<dependencies>
3636
<dependency>
37-
<groupId>com.twosigma</groupId>
37+
<groupId>twosigma</groupId>
3838
<artifactId>cook-jobclient</artifactId>
39-
<version>0.1.0</version>
39+
<version>0.1.2-snapshot</version>
4040
</dependency>
4141
<dependency>
4242
<groupId>org.apache.avro</groupId>

0 commit comments

Comments
 (0)