Skip to content

Latest commit

 

History

History

2022_airflow_summit

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
DATAPROC_EXPANSION_README.md
DATAPROC_EXPANSION_README.md
 
 
README.md
README.md
 
 
__init__.py
__init__.py
 
 
bigquery_permissions.py
bigquery_permissions.py
 
 
bigquery_permissions_test.py
bigquery_permissions_test.py
 
 
constraints.txt
constraints.txt
 
 
data_analytics_dag.py
data_analytics_dag.py
 
 
data_analytics_dag_expansion.py
data_analytics_dag_expansion.py
 
 
data_analytics_dag_expansion_test.py
data_analytics_dag_expansion_test.py
 
 
data_analytics_dag_test.py
data_analytics_dag_test.py
 
 
data_analytics_process.py
data_analytics_process.py
 
 
data_analytics_process_expansion.py
data_analytics_process_expansion.py
 
 
data_analytics_process_expansion_test.py
data_analytics_process_expansion_test.py
 
 
data_analytics_process_test.py
data_analytics_process_test.py
 
 
data_processing_helper.py
data_processing_helper.py
 
 
data_processing_helper_test.py
data_processing_helper_test.py
 
 
ghcn-stations-processed.csv
ghcn-stations-processed.csv
 
 
ghcnd-stations.txt
ghcnd-stations.txt
 
 
holidays.csv
holidays.csv
 
 
noxfile_config.py
noxfile_config.py
 
 
parallel_work.py
parallel_work.py
 
 
parallel_work_test.py
parallel_work_test.py
 
 
requirements-test.txt
requirements-test.txt
 
 
requirements.txt
requirements.txt
 
 
retries.py
retries.py
 
 
retries_test.py
retries_test.py
 
 
test_data.csv
test_data.csv
 
 
test_data_expansion.csv
test_data_expansion.csv
 
 

README.md

Airflow Summit 2022 Sample DAGs

DAGs in this directory are part of the curriculum for the Cloud Composer Workshop at the 2022 Airflow summit. Be sure to register and join us there!

parallel_work

Running a number of tasks in parallel showcases autoscaling in a Cloud Composer environment.

data_analytics_dag

Runs a basic Data Analytics workflow using BigQuery, Cloud Storage, and Dataproc Serverless. More detailed documentation can be found for this DAG in the Composer documentation

data_analytics_dag_expansion

This DAG is nearly identical to data_analytics_dag only it features a more complex Dataproc job. For more info, refer to the README

retries

Showcasing how retries can be used when API calls fail.

bigquery_permissions

Demo showing error message when there are missing permissions to query a BigQuery dataset.