Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time.
A lightweight CLI tool for versioning data alongside source code and building data pipelines.
Beneath is a serverless real-time data platform ⚡️
ops0 is an AI-powered natural language DevOps CLI native to Claude AI with ansible, terraform, kubernetes, aws, azure and docker operations in a single cli. An open-source alternative to complex DevOps workflows, manual operations, etc. 🤖 ⚡ 👉 Natural Language DevOps Automation & Troubleshooting Tool
A high-performance, extremely flexible, and easily extensible universal workflow engine.
This open-source Terraform provider enables users to seamlessly integrate the Monte Carlo data reliabillity platform into their infrastructure as a code (IaC) workflows.
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
A simple data processing pipeline supporting FIFO, fixed & dynamic worker pools, and broadcast stages.
Lightweight data streaming application that monitors SQL Server CDC-enabled tables for changes and streams events to various output destinations. Ideal for real-time analytics, event-driven architectures, and seamless integration with cloud-native workflow
Ministream is a small, stand-alone, real-time event messaging streaming server
This project implements an ETL (Extract, Transform, Load) pipeline in Go for ingesting cryptocurrency market data from the CoinGecko API.
A set of plugins (mappers, sinks, etc.) for Numaflow pipelines
CLI Application holding a sentiment analysis data (Twitter tweets) pipeline with its own Web API to query results in the database. Written entirely in Go.
Sigzag is an observability utility and backend service for datlin and is used to monitor, sign and log data pipeline transactions.
Playing with Apache Beam Tour: https://tour.beam.apache.org
Kubernetes-native data pipeline platform and orchestration
Add a description, image, and links to the data-pipelines topic page so that developers can more easily learn about it.
To associate your repository with the data-pipelines topic, visit your repo's landing page and select "manage topics."