Getting started with Google Cloud Dataflow

Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. This guides you through all the steps needed to run an Apache Beam pipeline in the Google Cloud Dataflow runner.

Setting up your Google Cloud project

The following instructions help you prepare your Google Cloud project.

1. Install the Cloud SDK.

   ℹ️ This is not required in Cloud Shell since it already has the Cloud SDK pre-installed.

2. Create a new Google Cloud project and save the project ID in an environment variable.

   Click here to create a new project

   # Save your project ID in an environment variable for ease of use later on.
   export PROJECT=your-google-cloud-project-id

3. Setup the Cloud SDK to your GCP project.

   gcloud init

4. Enable billing.

5. Enable the Dataflow API.

   Click here to enable the API

6. Authenticate to your Google Cloud project.

   gcloud auth application-default login

   ℹ️ For more information on authentication, see the Authentication overview page.

   To learn more about the permissions needed for Dataflow, see the Dataflow security and permissions page.

Setting up a Python development environment

For instructions on how to install Python, virtualenv, and the Cloud SDK, see the Setting up a Python development environment guide.