-
Notifications
You must be signed in to change notification settings - Fork 47
add projects for maven, gradle and sbt for java and scala #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,3 @@ | ||
SparkBuildExamples* | ||
gradle* | ||
*/.gradle | ||
*/build | ||
*/target | ||
*/gradle | ||
.gradle |
This file was deleted.
This file was deleted.
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# Example projects for using DSE Analytics | ||
|
||
These are template projects that illustrate how to build Spark Application written in Java or Scala with | ||
Maven, SBT or Gradle. Navigate to project that implements simple write-to-/read-from-Cassandra | ||
application with language and build tool of your choice. | ||
|
||
## Dependencies | ||
|
||
Compiling Spark Applications depends on `Apache Spark` and optionally on `Spark Cassandra Connector` | ||
jars. Projects `dse` and `oss` show two different ways of supplying these dependencies. | ||
Both projects are built and executed with similar commands. | ||
|
||
### DSE | ||
|
||
If you are planning to execute your Spark Application on a DSE cluster, you can use `dse` project | ||
template which will automatically download (and use during compilation) all jars available in DSE cluster. | ||
Please mind DSE version specified in build file, it should should match the one in your cluster. | ||
|
||
### OSS | ||
|
||
If you are planning to execute your Spark Application against Open Source Apache Spark and Open Source | ||
Apache Cassandra, use `oss` project template where all dependencies have to be specified manually in | ||
build files. Please mind dependencies versions, these should match the ones in your execution environment. | ||
|
||
### Additional dependencies | ||
|
||
Prepared projects use extra plugins so additional dependencies can be included with your | ||
application's jar. All you need to do is add dependencies in build configuration file. | ||
|
||
## Building & running | ||
|
||
### Sbt | ||
|
||
`sbt clean assembly` | ||
`dse spark-submit --class com.datastax.spark.example.scala.WriteRead target/scala-2.10/writeRead-assembly-0.1.jar` | ||
|
||
### Gradle | ||
|
||
`gradle clean build` | ||
`dse spark-submit --class com.datastax.spark.example.scala.WriteRead build/libs/writeRead-0.1.jar` | ||
|
||
### Maven | ||
|
||
`mvn clean package` | ||
`dse spark-submit --class com.datastax.spark.example.scala.WriteRead target/writeRead-0.1-dep.jar` | ||
|
||
|
||
|
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
build | ||
.idea |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
group 'com.datastax.spark.example' | ||
version '0.1' | ||
apply plugin: 'java' | ||
|
||
repositories { | ||
mavenCentral() | ||
maven { | ||
url "https://datastax.artifactoryonline.com/datastax/public-repos/" | ||
} | ||
} | ||
|
||
configurations { | ||
provided | ||
compile.extendsFrom provided | ||
} | ||
|
||
// Please make sure that following DSE version matches your DSE cluster version. | ||
dependencies { | ||
provided "com.datastax.dse:dse-spark-dependencies:5.0.1" | ||
// compile "your:dependecy:1.0.0" | ||
} | ||
|
||
jar { | ||
dependsOn configurations.runtime | ||
from { | ||
(configurations.runtime - configurations.provided).collect { it.isDirectory() ? it : zipTree(it) } | ||
} { | ||
exclude "META-INF/*.SF" | ||
exclude "META-INF/*.DSA" | ||
exclude "META-INF/*.RSA" | ||
} | ||
} | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
rootProject.name = 'writeRead' |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../../../sbt/dse/src/main/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
build | ||
.idea |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
group 'com.datastax.spark.example' | ||
version '0.1' | ||
apply plugin: 'java' | ||
|
||
repositories { | ||
mavenCentral() | ||
} | ||
|
||
configurations { | ||
provided | ||
compile.extendsFrom provided | ||
} | ||
|
||
// Please make sure that following dependencies have versions corresponding to the ones in your cluster. | ||
dependencies { | ||
provided "org.apache.spark:spark-core_2.10:1.6.1" | ||
provided "org.apache.spark:spark-sql_2.10:1.6.1" | ||
provided "com.datastax.spark:spark-cassandra-connector_2.10:1.6.0" | ||
// compile "your:dependecy:1.0.0" | ||
} | ||
|
||
jar { | ||
dependsOn configurations.runtime | ||
from { | ||
(configurations.runtime - configurations.provided).collect { it.isDirectory() ? it : zipTree(it) } | ||
} { | ||
exclude "META-INF/*.SF" | ||
exclude "META-INF/*.DSA" | ||
exclude "META-INF/*.RSA" | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
rootProject.name = 'writeRead' |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../../../sbt/dse/src/main/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
*.iml | ||
.idea/ | ||
target |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
<modelVersion>4.0.0</modelVersion> | ||
|
||
<groupId>com.datastax.spark.example</groupId> | ||
<artifactId>writeRead</artifactId> | ||
<version>0.1</version> | ||
<packaging>jar</packaging> | ||
|
||
<properties> | ||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> | ||
</properties> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>com.datastax.dse</groupId> | ||
<artifactId>dse-spark-dependencies</artifactId> | ||
<version>5.0.1</version> | ||
<scope>provided</scope> | ||
</dependency> | ||
<!-- Your dependencies, 'provided' are not included in jar --> | ||
</dependencies> | ||
|
||
<repositories> | ||
<repository> | ||
<id>DataStax-Repo</id> | ||
<url>https://datastax.artifactoryonline.com/datastax/public-repos/</url> | ||
</repository> | ||
</repositories> | ||
|
||
<build> | ||
<plugins> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-compiler-plugin</artifactId> | ||
<version>3.5.1</version> | ||
<configuration> | ||
<source>1.8</source> | ||
<target>1.8</target> | ||
</configuration> | ||
</plugin> | ||
<plugin> | ||
<artifactId>maven-assembly-plugin</artifactId> | ||
<version>2.6</version> | ||
<configuration> | ||
<descriptor>src/assembly/dep.xml</descriptor> | ||
</configuration> | ||
<executions> | ||
<execution> | ||
<id>create-archive</id> | ||
<phase>package</phase> | ||
<goals> | ||
<goal>single</goal> | ||
</goals> | ||
</execution> | ||
</executions> | ||
</plugin> | ||
</plugins> | ||
</build> | ||
</project> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../../../../scala/maven/dse/src/assembly/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../../../sbt/dse/src/main/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
*.iml | ||
.idea/ | ||
target |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May want to add a quick comment "The provided configuration behaves the same as the sbt "provided" keyword which will cause jars to be excluded from assembled fat-jar"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This example
jar
command is going to technically build a fat jar incase that wasn't clear :)