add projects for maven, gradle and sbt for java and scala #1
Conversation
|
Are there meant to be Gradle examples as well? This PR seems to remove them all. |
|
@bcantoni yes, there are moved to |
|
These should all be converted to use the now publicly available DataStax Spark dependency that Artem and Predrag built |
|
will do |
|
In this PR I see two build.gradle files which are being deleted, but none which are being added or moved. Under scala and java I just see maven and sbt dirs. Regarding the readme file and discussion of dependencies, would it help to have a few concrete examples with real version numbers listed? This seems to be a tricky part for people, so I think a couple of real examples here would be helpful, like:
|
assembly works for all of them sbt and gradle can figure out DSE classpath (thanks Russ)
|
@bcantoni you are right, sorry, my .gitignore file ate gradle examples |
dse is for users with DSE environment (these projects use DSE connector dependency) oss id for open source users with manual dependencies
|
@RussellSpitzer moved to publicly available DataStax Spark dependency that Artem and Predrag built @bcantoni channged to project structure to dse vs oss. dse uses dependency that Russ mentioned, which is available only for 5.0.1. Do we want to have "custom" examples for older versions of DSE? |
|
The version compatibility table on the connector site is good and we can point to it. However it (intentionally) does not list the DSE version. In any case if I take a step upwards, I think we want to answer the use case of "I'm using DSE version x.y, how do I set up my Spark project to work with it?". A table full of versions would probably be impossible to maintain, so maybe just an explanation of how to find and set the Spark and Spark Connector versions based on the DSE version you're using. The DSE Release Notes pages might be helpful in this regard. |
| } | ||
| } | ||
|
|
||
| configurations { |
There was a problem hiding this comment.
May want to add a quick comment "The provided configuration behaves the same as the sbt "provided" keyword which will cause jars to be excluded from assembled fat-jar"
There was a problem hiding this comment.
This example jar command is going to technically build a fat jar incase that wasn't clear :)
|
I left a few notes |
|
@RussellSpitzer applied your remarks @bcantoni at the moment there is no way to use current
I'll talk to Artem when he is back from vacation and get back to you. Let's treat (1) as a backup plan. |
| sqlContext.setKeyspace("ks"); | ||
| List<Tuple2<Integer, Integer>> sqlReadData = sqlContext.sql("SELECT * FROM kv").javaRDD() | ||
| // Read Data Using the Hive Context | ||
| List<Tuple2<Integer, Integer>> sqlReadData = hiveContext |
There was a problem hiding this comment.
Had to ditch select statement because I couldn't get rid of the exception "table kv.ks couldn't be found".
There was a problem hiding this comment.
Without DSE the metastore won't exist, but you can make an entry by registering a DataFrame or by using the following syntax
|
I'm +1, This is a great starting point for our users. We can always do more tweaks later |
|
Russ, could you please merge this (I do not have required privileges)? |
assembly works for all of them
sbt and gradle can figure out DSE classpath (thanks Russ)