This is a collection of Java 8 and Apache Spark examples and concepts, from basic to advanced. It explain basic concepts introduced by Java 8 and how you can merge them with Apache Spark.
The current tutorial or set of examples provide a way of understand Spark 2.0 in details but also to get familiar with Java 8 and it new features like lambda, Stream and reaction programming.
Java 8 is the latest version of Java which includes two major changes: Lambda expressions and Streams. Java 8 is a revolutionary release of the world’s #1 development platform. It includes a huge upgrade to the Java programming model and a coordinated evolution of the JVM, Java language, and libraries. Java 8 includes features for productivity, ease of use, improved polyglot programming, security and improved performance. Welcome to the latest iteration of the largest, open, standards-based, community-driven platform.
1- Lambda Expressions, a new language feature, has been introduced in this release. They enable you to treat functionality as a method argument, or code as data. Lambda expressions let you express instances of single-method interfaces (referred to as functional interfaces) more compactly.
2- Classes in the new java.util.stream package provide a Stream API to support functional-style operations on streams of elements. The Stream API is integrated into the Collections API, which enables bulk operations on collections, such as sequential or parallel map-reduce transformations including performance improvement for HashMaps with Key Collisions.
Apache Spark™ is a fast and general engine for large-scale data processing. Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.
A good way of using these examples is by first cloning the repo, and then starting your own Spark Java 8.
Java 8 can be download here. After the installation you need to be sure that the version you are using is java 8, you can check that by running:
java -version
In order to setup Spark locally in you machine you should download the spark version from here. Then you should follow the next steps:
> tar zxvf spark-xxx.tgz
> cd spark-xxx
> build/mvn -DskipTests clean package
After the compilation and before running your first example you should add to your profile the SPARK MASTER Variable:
> export SPARK_LOCAL_IP=127.0.0.1
To be sure that you spark is installed properly in your machine you can run the first example from spark:
> ./bin/run-example SparkPi
Some of the datasets we will use in this learning tutorial are:
- Tweets Archive from @ypriverol is used in the word count
- We will be using datasets from the KDD Cup 1999. The results of this competition can be found here.
The reference book for these and other Spark related topics is:
- Learning Spark by Holden Karau, Andy Konwinski, Patrick Wendell, and Matei Zaharia.
The following examples can be examined individually, although there is a more or less linear 'story' when followed in sequence. By using different datasets they try to solve a related set of tasks with it.
Here a list of the most basic examples in Spark-Java8 and definition of the most basic concepts in Spark.
1- SparkWordCount: About How to create a simple JavaRDD in Spark.
2- MaptoDouble: How to generate general statistics about an RDD in Spark
3- SparkAverage: How to compute the average of a set of numbers in Spark.
1- SparkSampling: Basic Spark Sampling using functions sample and takesample.