Quickstart PySpark with Anaconda on AWS/EMR using Terraform

A batch Data Pipeline that retrieves data from a user purchase table and a movie review table and is transformed to form a user behaviour metric table.

Deploying Delta Live Tables Pipelines on AWS with Terraform

This repository contains the codebase for the BuildItAll Big Data Processing Platform, a case study project designed to manage large daily data for a hypothetical Belgian client.

E2E Spark data pipelines with engineering fundamentals

Project that incorporates TerraForm to create AWS infrastructure using S3, Lambda, and DynamoDB tables for ocean and river data 🐢

Implemented a data processing pipeline using Amazon EMR to transform monthly vendor sales data from CSV format to a clean, analyzed dataset accessible through Amazon Athena.

Via AWS resource for EMR on EKS (i.e. emr-containers), I provide a more grounded, applicable examples/explanations for using this service

Streaming pipeline using AWS MSK and AWS EMR with Spark, retrieving the data from Twitter Streams API

Data lake project for a US based Insurance Company

This repository contains Terraform code to deploy a serverless batch processing architecture on AWS, designed to replace an on-premises system with a scalable, reliable, and maintainable cloud solution.

Запуск и управление приложениями для Spark и PySpark в сервисе Yandex Data Processing.

Esse repositório é para o aprendizado de recursos da aws, utilizando terraform, python e spark voltado para o universo de dados