Visual, durable data pipelines powered by Go and Temporal.
DataFlow is an open-core platform for building, running, and observing data pipelines. Use the visual canvas or AI-assisted builder, promote immutable versions between environments, and let Temporal resume work safely after failures.
The community code is licensed under GNU AGPL-3.0. Enterprise-only code in
apps/workflow-go/ee/ is source-available under Elastic License 2.0.
DataFlow is pre-release software. It is suitable for local evaluation and controlled pilots; production readiness and high availability depend on your deployment and backing services.
Try the hosted app at dataflow.cohestra.dev, or run it locally with Docker Compose:
git clone https://github.com/Cohestra/cohestra-dataflow.git
cd cohestra-dataflow
cp .env.example .env
node scripts/gen-worker-keypair.js
docker compose up -dOpen http://localhost:3002. For local Kubernetes, an existing cluster, or a
GCP/Terraform deployment, use the self-hosting guide.
- Visual pipeline canvas and AI-assisted pipeline drafting.
- Durable execution, retries, schedules, backfills, pause, resume, and cancel.
- Immutable versions with Integration and Production promotion gates.
- Runtime lineage, data quality, alerts, analytics, audit events, and OpenLineage.
- 13 built-in sources and 12 built-in sinks, with saved credentials and OAuth where supported.
- Tenant-aware PostgreSQL row-level security and encrypted credentials and payloads.
| Layer | Components | Responsibility |
|---|---|---|
| Experience | React/Vite web app | Canvas, catalog, runs, lineage, and administration |
| API | Go API | Identity, tenants, pipeline lifecycle, run control, monitoring, and billing |
| Orchestration | Temporal + Go workflow workers | Durable DAG history, queues, retries, timers, and signals |
| Data plane | Go activity workers | Connector I/O, checkpoints, payload handling, and sink writes |
| State | PostgreSQL, Redis, ClickHouse, object storage | Authoritative metadata, events, analytics, and encrypted large payloads |
See the visual architecture guide for the full runtime flow.
Open Connectors in DataFlow, save a connection, test it, and select that
connection from a source or sink node. The authenticated REST API can also be
used from fetch, curl, or any HTTP client library. No product source changes
or image rebuilds are required.
See the connector catalog and API examples.
| Path | Purpose |
|---|---|
apps/web |
React application, routes, and UI components |
apps/workflow-go |
Go API, Temporal workers, connector runtime, and storage |
packages/shared |
Shared pipeline, catalog, lineage, and entitlement contracts |
connectors/manifests |
Deploy-time declarative connector manifests |
db |
PostgreSQL and ClickHouse migrations |
deploy/helm/dataflow |
Kubernetes Helm chart |
infra |
Optional Terraform deployment reference |
docs |
Architecture, operations, security, API contracts, and decisions |
tests |
Cross-runtime contracts and deployment verification |
| Guide | Use it for |
|---|---|
| Documentation home | User-facing documentation index |
| Architecture | Runtime flow, process boundaries, and data ownership |
| Self-hosting | Docker, Kubernetes/Helm, and GCP/Terraform |
| Connectors | Supported sources and sinks, UI setup, and API use |
| Backend contracts | HTTP and Temporal compatibility surface |
| Security | Security baseline and release controls |
| Roadmap | Pending work and acceptance criteria |
Read CONTRIBUTING.md, CODE_OF_CONDUCT.md, and GOVERNANCE.md. Report vulnerabilities privately using SECURITY.md.
- Community code: GNU AGPL-3.0
- Enterprise directory: Elastic License 2.0
- Required notices: NOTICE