Skip to content
This repository has been archived by the owner on Nov 11, 2022. It is now read-only.

Add Dataflow, version 2.0.0-beta1, as a distribution of Apache Beam #519

Merged
merged 13 commits into from
Jan 5, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update top-level README.md and CONTRIBUTING.md to reflect what this b…
…ranch contains
  • Loading branch information
davorbonaci committed Jan 5, 2017
commit 2393729cc0747f750c2d54d2c75b5fa79de7984e
14 changes: 9 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
Want to contribute? Great! First, read this page (including the small print at
the end).

Google Cloud Dataflow SDK is a distribution of Apache Beam. If you'd like to
change anything under the `org.apache.beam.*` namespace, please submit that
change directly to the [Apache Beam](https://github.com/apache/beam) project.

This repository contains code to build the Dataflow distribution of Beam, and
some Dataflow-specific code. Only changes to how the distribution is built, or
the Dataflow-specific code under the `com.google.cloud.dataflow.*` namespace,
can be merged here.

### Before you contribute
Before we can use your code, you must sign the
[Google Individual Contributor License Agreement](https://developers.google.com/open-source/cla/individual?csw=1)
Expand All @@ -21,11 +30,6 @@ frustration later on.
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose.

### Organization
During our review and triage of incoming pull requests, we'll advise whether to
include your contribution into the mainline SDK, or to maintain it within the
separate group of [community-contributed modules](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib).

### The small print
Contributions made by corporations are covered by a different agreement than
the one above, the Software Grant and Corporate Contributor License Agreement.
68 changes: 24 additions & 44 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,13 @@

[Google Cloud Dataflow](https://cloud.google.com/dataflow/) provides a simple,
powerful programming model for building both batch and streaming parallel data
processing pipelines. This repository hosts the open-sourced Cloud Dataflow SDK
for Java, which can be used to run pipelines against the Google Cloud Dataflow
Service.
processing pipelines.

Dataflow SDK for Java is a distribution of a portion of the
[Apache Beam](https://beam.apache.org) project. This repository hosts the
code to build this distribution and any Dataflow-specific code/modules. The
underlying source code is hosted in the
[Apache Beam repository](https://github.com/apache/beam).

[General usage](https://cloud.google.com/dataflow/getting-started) of Google
Cloud Dataflow does **not** require use of this repository. Instead:
Expand All @@ -25,13 +29,19 @@ environments like Eclipse or Apache Maven:
[DataflowJavaSDK-examples](https://github.com/GoogleCloudPlatform/DataflowJavaSDK-examples)
repository.

However, if you'd like to contribute to the SDK, write your own PipelineRunner,
or just dig in for the fun of it, please stay with us here!
<!-- 1. If you are using [Eclipse](https://eclipse.org/) integrated development
environment (IDE), the
[Cloud Dataflow Plugin for Eclipse](https://cloud.google.com/dataflow/getting-started-eclipse)
provides tools to create and execute Dataflow pipelines locally and on the
Dataflow Service. -->

## Status [![Build Status](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK.svg?branch=v2)](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK)

## Status [![Build Status](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK)
This branch is a work-in-progress for the Dataflow SDK for Java, version 2.0.0.
It is currently supported on the Cloud Dataflow service in Beta.

Both the SDK and the Dataflow Service are generally available, open to all
developers, and considered stable and fully qualified for production use.
<!--Both the SDK and the Dataflow Service are generally available, open to all
developers, and considered stable and fully qualified for production use.-->

## Overview

Expand All @@ -48,25 +58,17 @@ for execution.
* [`PipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/PipelineRunner.java):
specifies where and how the pipeline should execute.

We provide three PipelineRunners:
We provide two runners:

1. The [`DirectPipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java)
1. The [`DirectRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java)
runs the pipeline on your local machine.
2. The [`DataflowPipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java)
1. The [`DataflowRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java)
submits the pipeline to the Dataflow Service, where it runs using managed
resources in the [Google Cloud Platform](https://cloud.google.com) (GCP).
3. The [`BlockingDataflowPipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/BlockingDataflowPipelineRunner.java)
submits the pipeline to the Dataflow Service via the `DataflowPipelineRunner`
and then prints messages about the job status until the execution is complete.

The SDK is built to be extensible and support additional execution environments
beyond local execution and the Google Cloud Dataflow Service. In partnership
with [Cloudera](https://www.cloudera.com/), you can run Dataflow pipelines on
an [Apache Spark](https://spark.apache.org/) backend using the
[`SparkPipelineRunner`](https://github.com/cloudera/spark-dataflow).
Additionally, you can run Dataflow pipelines on an
[Apache Flink](https://flink.apache.org/) backend using the
[`FlinkPipelineRunner`](https://github.com/dataArtisans/flink-dataflow).
beyond local execution and the Google Cloud Dataflow Service. Apache Beam
contains additional SDKs, runners, IO connectors, etc.

## Getting Started

Expand All @@ -77,35 +79,12 @@ module provides a set of basic Java APIs to program against.
* The [`examples`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/examples)
module provides a few samples to get started. We recommend starting with the
`WordCount` example.
* The [`contrib`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/contrib)
directory hosts community-contributed Dataflow modules.

The following command will build both the `sdk` and `example` modules and
install them in your local Maven repository:

mvn clean install

You can speed up the build and install process by using the following options:

1. To skip execution of the unit tests, run:

mvn install -DskipTests

2. While iterating on a specific module, use the following command to compile
and reinstall it. For example, to reinstall the `examples` module, run:

mvn install -pl examples

Be careful, however, as this command will use the most recently installed SDK
from the local repository (or Maven Central) even if you have changed it
locally.

If you are using [Eclipse](https://eclipse.org/) integrated development
environment (IDE), the
[Cloud Dataflow Plugin for Eclipse](https://cloud.google.com/dataflow/getting-started-eclipse)
provides tools to create and execute Dataflow pipelines locally and on the
Dataflow Service.

After building and installing, you can execute the `WordCount` and other
example pipelines by following the instructions in this
[README](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/examples/README.md).
Expand All @@ -121,5 +100,6 @@ on GitHub to report any bugs, comments or questions regarding SDK development.
## More Information

* [Google Cloud Dataflow](https://cloud.google.com/dataflow/)
* [Apache Beam](https://beam.apache.org/)
* [Dataflow Concepts and Programming Model](https://cloud.google.com/dataflow/model/programming-model)
* [Java API Reference](https://cloud.google.com/dataflow/java-sdk/JavaDoc/index)