Dataflow Solution Guides

Welcome to the Dataflow Solution Guides!

The Dataflow Solution Guides offer full end-to-end deployment for the most common streaming solutions to run on Dataflow.

This repository contains the following assets for each guide:

• Full Terraform code to spawn all the necessary Google Cloud infrastructure
• Pipelines code in Python, Java and Go (coming soon) for a sample pipeline for each use case

Solution guides

This the list of solution guides available at this moment:

Guide	Description	Development status
GenAI & Machine Learning Inference	Real-time inference with local GenAI models, using a GPU	Ready ✅
ETL / Integration	Real-time change data capture from a Spanner database to BigQuery	Ready ✅
Log Replication & Analytics	Real-time log replication into Splunk	Beta 🏭
Marketing Intelligence	Real-time marketing intelligence, using an AutoML model deployed in Vertex	Beta 🏭
Clickstream Analytics	Real-time clickstream analytics with Bigtable enrichment / data hydration	Work in progress 🔨
IoT Analytics	Real-time Internet of Things (IoT) analytics with Bigtable enrichment & models deployed in Vertex AI	Work in progress 🔨
Anomaly Detection	Real-time detection of anomalies in a stream of data leveraging GenAI with models deployed in Vertex AI	Beta 🏭
Customer Data Platform	Real-time customer data platform that unifies a customer view from different sources.	Beta 🏭
Gaming Analytics	Real-time analyis of gaming data to enhance live gameplay & offer targeting	Beta 🏭

Repository structure

• terraform: This directory contains the Terraform code for deploying the necessary Google Cloud infrastructure for each use case.
• pipelines: This directory contains the Python, Java, and Go code for the sample pipelines.
• use_cases: This directory contains the documentation of each use case

Getting help

• GitHub Issues: Report any issues or ask questions on the GitHub repository.
  • https://github.com/GoogleCloudPlatform/dataflow-solution-guides/issues
• Stack Overflow: Search for existing solutions or ask questions on Stack Overflow using the google-cloud-dataflow tag:
  • https://stackoverflow.com/questions/tagged/google-cloud-dataflow

Contributing

Your contributions to this repository are welcome.

• Fork and Pull Request: Fork the repository and submit a pull request with your changes.
• Follow the Contribution Guidelines: Please follow the contribution guidelines outlined in the CONTRIBUTING.md file.

Disclaimer

This is not an officially supported Google product. The code in this repository is for demonstrative purposes only.