Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
SparkBuildExamples*
gradle*
*/.gradle
*/build
*/target
*/gradle
.gradle
20 changes: 0 additions & 20 deletions GradleVsDSE/build.gradle

This file was deleted.

1 change: 0 additions & 1 deletion GradleVsDSE/src

This file was deleted.

14 changes: 0 additions & 14 deletions GradleVsOSS/build.gradle

This file was deleted.

48 changes: 48 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Example projects for using DSE Analytics

These are template projects that illustrate how to build Spark Application written in Java or Scala with
Maven, SBT or Gradle. Navigate to project that implements simple write-to-/read-from-Cassandra
application with language and build tool of your choice.

## Dependencies

Compiling Spark Applications depends on `Apache Spark` and optionally on `Spark Cassandra Connector`
jars. Projects `dse` and `oss` show two different ways of supplying these dependencies.
Both projects are built and executed with similar commands.

### DSE

If you are planning to execute your Spark Application on a DSE cluster, you can use `dse` project
template which will automatically download (and use during compilation) all jars available in DSE cluster.
Please mind DSE version specified in build file, it should should match the one in your cluster.

### OSS

If you are planning to execute your Spark Application against Open Source Apache Spark and Open Source
Apache Cassandra, use `oss` project template where all dependencies have to be specified manually in
build files. Please mind dependencies versions, these should match the ones in your execution environment.

### Additional dependencies

Prepared projects use extra plugins so additional dependencies can be included with your
application's jar. All you need to do is add dependencies in build configuration file.

## Building & running

### Sbt

`sbt clean assembly`
`dse spark-submit --class com.datastax.spark.example.scala.WriteRead target/scala-2.10/writeRead-assembly-0.1.jar`

### Gradle

`gradle clean build`
`dse spark-submit --class com.datastax.spark.example.scala.WriteRead build/libs/writeRead-0.1.jar`

### Maven

`mvn clean package`
`dse spark-submit --class com.datastax.spark.example.scala.WriteRead target/writeRead-0.1-dep.jar`



4 changes: 0 additions & 4 deletions SbtAssemblyVsDSE/.gitignore

This file was deleted.

15 changes: 0 additions & 15 deletions SbtAssemblyVsDSE/README

This file was deleted.

31 changes: 0 additions & 31 deletions SbtAssemblyVsDSE/build.sbt

This file was deleted.

1 change: 0 additions & 1 deletion SbtAssemblyVsDSE/project/assembly.sbt

This file was deleted.

1 change: 0 additions & 1 deletion SbtAssemblyVsDSE/src

This file was deleted.

4 changes: 0 additions & 4 deletions SbtVsDSE/.gitignore

This file was deleted.

12 changes: 0 additions & 12 deletions SbtVsDSE/README

This file was deleted.

28 changes: 0 additions & 28 deletions SbtVsDSE/build.sbt

This file was deleted.

2 changes: 2 additions & 0 deletions java/gradle/dse/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
build
.idea
33 changes: 33 additions & 0 deletions java/gradle/dse/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
group 'com.datastax.spark.example'
version '0.1'
apply plugin: 'java'

repositories {
mavenCentral()
maven {
url "https://datastax.artifactoryonline.com/datastax/public-repos/"
}
}

configurations {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May want to add a quick comment "The provided configuration behaves the same as the sbt "provided" keyword which will cause jars to be excluded from assembled fat-jar"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example jar command is going to technically build a fat jar incase that wasn't clear :)

provided
compile.extendsFrom provided
}

// Please make sure that following DSE version matches your DSE cluster version.
dependencies {
provided "com.datastax.dse:dse-spark-dependencies:5.0.1"
// compile "your:dependecy:1.0.0"
}

jar {
dependsOn configurations.runtime
from {
(configurations.runtime - configurations.provided).collect { it.isDirectory() ? it : zipTree(it) }
} {
exclude "META-INF/*.SF"
exclude "META-INF/*.DSA"
exclude "META-INF/*.RSA"
}
}

1 change: 1 addition & 0 deletions java/gradle/dse/settings.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
rootProject.name = 'writeRead'
1 change: 1 addition & 0 deletions java/gradle/dse/src/main
2 changes: 2 additions & 0 deletions java/gradle/oss/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
build
.idea
31 changes: 31 additions & 0 deletions java/gradle/oss/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
group 'com.datastax.spark.example'
version '0.1'
apply plugin: 'java'

repositories {
mavenCentral()
}

configurations {
provided
compile.extendsFrom provided
}

// Please make sure that following dependencies have versions corresponding to the ones in your cluster.
dependencies {
provided "org.apache.spark:spark-core_2.10:1.6.1"
provided "org.apache.spark:spark-sql_2.10:1.6.1"
provided "com.datastax.spark:spark-cassandra-connector_2.10:1.6.0"
// compile "your:dependecy:1.0.0"
}

jar {
dependsOn configurations.runtime
from {
(configurations.runtime - configurations.provided).collect { it.isDirectory() ? it : zipTree(it) }
} {
exclude "META-INF/*.SF"
exclude "META-INF/*.DSA"
exclude "META-INF/*.RSA"
}
}
1 change: 1 addition & 0 deletions java/gradle/oss/settings.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
rootProject.name = 'writeRead'
1 change: 1 addition & 0 deletions java/gradle/oss/src/main
3 changes: 3 additions & 0 deletions java/maven/dse/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
*.iml
.idea/
target
60 changes: 60 additions & 0 deletions java/maven/dse/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.datastax.spark.example</groupId>
<artifactId>writeRead</artifactId>
<version>0.1</version>
<packaging>jar</packaging>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-spark-dependencies</artifactId>
<version>5.0.1</version>
<scope>provided</scope>
</dependency>
<!-- Your dependencies, 'provided' are not included in jar -->
</dependencies>

<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://datastax.artifactoryonline.com/datastax/public-repos/</url>
</repository>
</repositories>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<configuration>
<descriptor>src/assembly/dep.xml</descriptor>
</configuration>
<executions>
<execution>
<id>create-archive</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
1 change: 1 addition & 0 deletions java/maven/dse/src/assembly
1 change: 1 addition & 0 deletions java/maven/dse/src/main
3 changes: 3 additions & 0 deletions java/maven/oss/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
*.iml
.idea/
target
Loading