Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

Terraform module for Amazon MWAA(Apache Airflow)

Terraform module for a PostgreSQL-backed Apache Airflow instance

Setup guidelines for infrastructure setup of open-source technologies

A project that sets up an AWS ECS infrastructure and hosts an Apachi Airflow running inside a docker container.

This project demonstrates how to ensure data security and compliance with industry regulations. This includes the use of GCP IAM for access control, GCP KMS for data encryption, GCP SCC for sensitive data discovery and classification, and GCP Audit Logs for logging and auditing.

This project illustrates real-time data processing and analytics. This project uses Apache Kafka for capturing and streaming real-time data, GCP Cloud Functions for processing data in real-time, GCP PubSub for real-time notifications, and GCP Looker Studio for real-time data visualization.

This project focuses on maintaining data quality and consistency across different data sources. This project features Google Cloud Dataflow for data cataloging, Apache Airflow for ETL, Google Cloud Data Catalog for visual data preparation, and Snowflake for high-quality data storage and analysis.

This project focuses on scalable data processing and query performance optimisation. It uses Snowflake for data warehousing, GCP Cloud Functions for serverless compute, and Apache Kafka for real-time data streaming. It leverages the serverless capabilities of the systems for scalability and performance.