Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on May 14, 2022

• :books: Move directory tree to the start of README

  ismaildawoodjee

  authored
  a904ec3

  Copy full SHA for a904ec3

Commits on Jan 31, 2022

• :gear: Add metadata file to source-data

  ismaildawoodjee

  committed
  7cde912

  Copy full SHA for 7cde912

Commits on Jan 30, 2022

• :beetle: Change email address from main.tf (too late though)

  ismaildawoodjee

  committed
  f7f4bad

  Copy full SHA for f7f4bad

• :gear: Change Airflow retry task duration to 30 min

  ismaildawoodjee

  committed
  5b86fdb

  Copy full SHA for 5b86fdb

• :books: Images and README documentation

  ismaildawoodjee

  committed
  8561ab5

  Copy full SHA for 8561ab5

Commits on Jan 23, 2022

• :gear: Formatting and changing file name

  ismaildawoodjee

  committed
  4c65a1d

  Copy full SHA for 4c65a1d

• :books: Airflow deadlocks

  ismaildawoodjee

  committed
  0ada9b0

  Copy full SHA for 0ada9b0

Commits on Jan 22, 2022

• :arrow_up: Scale EMR cores from 1 node to 2

  Show description for 621aa7d
  ismaildawoodjee

  committed
  621aa7d

  Copy full SHA for 621aa7d

Commits on Jan 21, 2022

• :books: Add the main data pipeline architecture image

  ismaildawoodjee

  committed
  3c366d0

  Copy full SHA for 3c366d0

• :beetle: Fix EC2 internet ingress port being 8088 instead of 8080

  ismaildawoodjee

  committed
  7dfeb7d

  Copy full SHA for 7dfeb7d

• :arrow_up: Ensure idempotency for the final task of loading into supersetdb

  ismaildawoodjee

  committed
  1b7d426

  Copy full SHA for 1b7d426

• :beetle: Fix TimeReceived column being non-unique when upserting

  ismaildawoodjee

  committed
  2e31ca2

  Copy full SHA for 2e31ca2

• :gear: Update file name to use the file with full data

  ismaildawoodjee

  committed
  0155547

  Copy full SHA for 0155547

• :gear: Make the condition matching for Monday to be more readable

  ismaildawoodjee

  committed
  e22104d

  Copy full SHA for e22104d

• :beetle: Specify 'depends_on_past' to be false in order to prevent deadlocks

  Show description for 9692ad5
  ismaildawoodjee

  committed
  9692ad5

  Copy full SHA for 9692ad5

• :gear: Update csv file names and .git files

  ismaildawoodjee

  committed
  918a9e3

  Copy full SHA for 918a9e3

Commits on Jan 15, 2022

• :arrow_up: Example files for specifying credentials; upgate .gitignore

  ismaildawoodjee

  committed
  146b47e

  Copy full SHA for 146b47e

• :gear: Unignore a one-day sample of dated data on GitHub

  ismaildawoodjee

  committed
  4f95f07

  Copy full SHA for 4f95f07

• :gear: Unignore the dated sample of data on GitHub

  ismaildawoodjee

  committed
  5a41d1a

  Copy full SHA for 5a41d1a

• :gear: Uncomment out the Terraform code for provisioning EMR

  ismaildawoodjee

  committed
  fd358eb

  Copy full SHA for fd358eb

• :beetle: Fix for today's data being ETL'ed instead of yesterday's data

  ismaildawoodjee

  committed
  f2c14ae

  Copy full SHA for f2c14ae

• :star: Define ostgres foreign data wrapper and dblink functions for Postgres-Postgres data movement

  ismaildawoodjee

  committed
  809663d

  Copy full SHA for 809663d

• :gear: Change column names to use underscores for Redshift data

  ismaildawoodjee

  committed
  cb060f2

  Copy full SHA for cb060f2

• :star: Redshift data transfer to EC2 Postgres instance using the 'dblink' wrapper

  ismaildawoodjee

  committed
  15e5eee

  Copy full SHA for 15e5eee

• :books: Small clarification with SQL code comments

  ismaildawoodjee

  committed
  4e44256

  Copy full SHA for 4e44256

• :gear: Add while clause to wait for Redshift cluster to get paused.

  ismaildawoodjee

  committed
  0ae8be6

  Copy full SHA for 0ae8be6

• :arrow_up: Add Airflow task for transferring Redshift data to EC2 Postgres instance

  ismaildawoodjee

  committed
  f731b7f

  Copy full SHA for f731b7f

Commits on Jan 12, 2022

• :star: Full IaC for provisioning an EC2 instance, bootstrapping, and deploying Superset and Postgres containers

  ismaildawoodjee

  committed
  292b2eb

  Copy full SHA for 292b2eb

• :gear: Staged data format should be CSV; minor formatting

  ismaildawoodjee

  committed
  fa5661b

  Copy full SHA for fa5661b

• :gear: Availability check to skip task when Redshift is already in the desired state

  ismaildawoodjee

  committed
  d6adb39

  Copy full SHA for d6adb39

• :arrow_up: Ensure S3-to-Redshift task's idempotency by upserting data

  ismaildawoodjee

  committed
  44be838

  Copy full SHA for 44be838

• :books: Small clarification to source database update method

  ismaildawoodjee

  committed
  004c467

  Copy full SHA for 004c467

• :gear: Ignore EC2 key-pair security credential files

  ismaildawoodjee

  committed
  f462957

  Copy full SHA for f462957

Commits on Jan 8, 2022

• :gear: Fix task ID saying 'parquet' instead of 'csv'

  ismaildawoodjee

  committed
  b26b354

  Copy full SHA for b26b354

• :arrow_up: Airflow operator to transfer data from S3 to Redshift

  ismaildawoodjee

  committed
  9e219ab

  Copy full SHA for 9e219ab

Pagination

Previous

Next