Add Airbyte template for data pipeline deployment #609
Conversation
Co-authored-by: Siumauricio <47042324+Siumauricio@users.noreply.github.com>
Co-authored-by: Siumauricio <47042324+Siumauricio@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a new Airbyte template for deploying the open-source data movement platform. The implementation creates a complete multi-service Docker Compose setup with Dokploy integration, including proper domain configuration, secure password generation, and persistent storage for all data.
Key changes:
- Added Airbyte blueprint with 7-service architecture (webapp, server, worker, db, temporal, temporal-db, minio)
- Configured secure credential management using Dokploy's password helpers
- Added metadata entry to meta.json with proper tags and links
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| meta.json | Added Airbyte entry with version 1.7.8, description, links, and categorization tags |
| blueprints/airbyte/template.toml | Configured domain mapping and environment variables with secure password generation |
| blueprints/airbyte/docker-compose.yml | Implemented 7-service architecture with proper dependencies, persistent volumes, and environment configuration |
| restart: unless-stopped | ||
| environment: | ||
| POSTGRES_USER: temporal | ||
| POSTGRES_PASSWORD: temporal |
There was a problem hiding this comment.
The hardcoded password 'temporal' for the temporal-db service should be made configurable using a variable. Add a new variable like temporal_db_password = \"${password:32}\" in template.toml and reference it here as ${TEMPORAL_DB_PASSWORD}.
There was a problem hiding this comment.
Applied the changes in commit 52d2fc6. Added temporal_db_password = "${password:32}" variable in template.toml and updated both temporal-db and temporal services to use ${TEMPORAL_DB_PASSWORD}.
| DB: postgresql | ||
| DB_PORT: 5432 | ||
| POSTGRES_USER: temporal | ||
| POSTGRES_PWD: temporal |
There was a problem hiding this comment.
The hardcoded password 'temporal' for the Temporal service should be made configurable using a variable. This should reference the same ${TEMPORAL_DB_PASSWORD} variable suggested for the temporal-db service to maintain consistency.
| env = [ | ||
| "AIRBYTE_DB_PASSWORD=${db_password}", | ||
| "MINIO_ROOT_PASSWORD=${minio_password}" | ||
| ] |
There was a problem hiding this comment.
The Docker socket mount required by the worker service (/var/run/docker.sock) should be documented here or in a comment. While this mount is intentionally handled in docker-compose.yml rather than template.toml, it would be helpful to note this requirement for users reviewing the configuration.
| ] | |
| ] | |
| # Note: The Airbyte worker service requires a Docker socket mount at /var/run/docker.sock. | |
| # This mount is intentionally configured in docker-compose.yml rather than via config.mounts. |
There was a problem hiding this comment.
Added documentation comment in commit 52d2fc6 explaining that the Docker socket mount is intentionally configured in docker-compose.yml rather than via config.mounts.
…et requirement Co-authored-by: Siumauricio <47042324+Siumauricio@users.noreply.github.com>
built with Refined Cloudflare Pages Action⚡ Cloudflare Pages Deployment
|
|
in the temporal service i'm getting this TEMPORAL_ADDRESS is not set, setting it to 172.22.0.9:7233 in the worker im getting this Unsetting empty environment variable 'VERSION' / | ()/ / __ / / / | ()/ / __ / / / | ()/ / __ / / in the webapp service i'm getting this logs /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh |
Co-authored-by: Siumauricio <47042324+Siumauricio@users.noreply.github.com>
... Fixed both issues in commit db877e7:
|
What is this PR about?
New PR of Airbyte - an open-source data integration platform for building ETL/ELT pipelines with 200+ connectors.
Template Structure:
${password:32}helper for all database and storage passwordsKey Configuration:
Architecture Notes:
/var/run/docker.sock(required for Airbyte to execute connector jobs in isolated containers) - documented in template.tomlpostgres12driver configurationSTORAGE_TYPE: MINIOconfigurationSecurity Features:
${password:32}helperConfiguration Fixes:
postgres12(Temporal only supports: mysql8, postgres12, postgres12_pgx, cassandra)STORAGE_TYPE: MINIOadded to server and worker services for proper MinIO storage backend configurationChecklist
Before submitting this PR, please make sure that:
Issues related (if applicable)
Closes #609
Screenshots or Videos
N/A - Template deployment validation performed via
docker manifest inspectfor all 7 images anddocker compose configsyntax validation. Deployment issues resolved and verified working.Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.