A Python framework for data processing on GCP.

Solution Accelerators for Serverless Spark on GCP, the industry's first auto-scaling and serverless Spark as a service

Debussy is an opinionated Data Architecture and Engineering framework, enabling data analysts and engineers to build better platforms and pipelines.

Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

ecommerce GCP Streaming pipeline ― Cloud Storage, Compute Engine, Pub/Sub, Dataflow, Apache Beam, BigQuery and Tableau; GCP Batch pipeline ― Cloud Storage, Dataproc, PySpark, Cloud Spanner and Tableau

An educational project to build an end-to-end pipline for near real-time and batch processing of data further used for visualisation and a machine learning model.

opens a chrome browser to a dataproc cluster

Demonstration of Google Cloud Dataproc for running PySpark jobs

Project for the Data Engineering Zoomcamp by DataTalks.Club

✈ A Spark-based ETL Pipeline for the OpenSky and OpenFlights Datasets

Data Workflows with GCP Dataproc, Apache Airflow and Apache Spark

Collected data about from three sources, one opinion-based social media in twitter, research data in New York Times, and the third is the common crawl data for the same topic or key phrase, and from similar time periods. Processed the three data sets collected individually using classical big data methods like Map Reduce in Google Dataproc Clust…

Repositório para armazenar artefatos de um trabalho da disciplina de Computação Distribuída.

Orchestrating ETL pipeline in Google cloud using Apache Airflow

💥🚗 USA Accidents Data Engineering Project

Digital Innovation One - Desafio GCP Dataproc. O desafio consiste em efetuar um processamento de dados utilizando o produto Dataproc do GCP. Esse processamento irá efetuar a contahem das palavras de um livro e informar quantas vezes cada palavra aparece no mesmo.

Orchestration Dataproc serverless job with Airflow

This project demonstrates how to build a real-time product recommendation system using Pub/Sub Lite and Apache Spark with Dataproc