GitHub - da-199/news-headlines: Pipeline that extracts data from sites for a dashboard.

ETL Architecture Overview:

Files in the sources folder are scheduled to extract data on an hourly basis at the start of the hour.
Transformations on tables are performed.
generate_queries.py turns the dataframe into SQL queries.
database_connection.py loads the data to PostgreSQL by running the queries.

Data Model

Google Data Studio Visualization

Name		Name	Last commit message	Last commit date
Latest commit History 170 Commits
sources		sources
table_definitions		table_definitions
transformations		transformations
.gitignore		.gitignore
README.md		README.md
database_connection.py		database_connection.py
generate_queries.py		generate_queries.py
get_secrets.py		get_secrets.py
main.py		main.py
requirements.txt		requirements.txt
s3_output.py		s3_output.py

Provide feedback