chore(ci): use docker compose profiles for Azurite instead of separate file #44990
Conversation
…e file The batch exports tests use Azurite (Azure Blob Storage emulator) which was configured via a separate docker-compose file at `products/batch_exports/backend/tests/docker-compose.yml`. This required passing `-f` flags in CI and documentation, making it harder to maintain. Docker Compose profiles are the idiomatic way to handle optional services. The Azurite service now lives in docker-compose.base.yml with `profiles: ['batch-exports']`, only starting when explicitly requested via `--profile batch-exports`. Note: `docker compose down` without a profile only stops services without profiles, so CI uses `--profile '*'` on down to ensure cleanup of all services.
There was a problem hiding this comment.
Pull request overview
This PR consolidates Azurite test infrastructure by moving it from a separate docker-compose file into the main docker-compose.base.yml using Docker Compose profiles. This improves developer experience by eliminating the need for multiple -f flags when starting services.
Changes:
- Deleted separate docker-compose file at
products/batch_exports/backend/tests/docker-compose.yml - Added
objectstorage-azureservice todocker-compose.base.ymlwithprofiles: ['batch-exports'] - Updated CI workflow to use
--profile batch-exportsinstead of-fflag pattern - Updated documentation to reflect new profile-based approach
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| products/batch_exports/backend/tests/docker-compose.yml | Deleted separate compose file for Azurite service |
| docker-compose.base.yml | Added objectstorage-azure service with batch-exports profile |
| docker-compose.dev.yml | Extended objectstorage-azure service from base |
| products/batch_exports/backend/tests/temporal/README.md | Updated documentation to use --profile flag instead of -f flag |
| .github/workflows/ci-backend.yml | Updated CI commands to use profile syntax and wildcard profile on down |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| objectstorage-azure: | ||
| image: mcr.microsoft.com/azure-storage/azurite:3.35.0 | ||
| restart: on-failure | ||
| profiles: ['batch-exports'] | ||
| command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --location /data | ||
| ports: | ||
| - '10000:10000' |
There was a problem hiding this comment.
The original docker-compose.yml file included container_name: posthog_azurite which is missing in this migration. This could cause issues if any scripts, tests, or documentation rely on this specific container name for connecting to or managing the Azurite service. Consider adding container_name: posthog_azurite to maintain compatibility.
There was a problem hiding this comment.
Maybe good to do this, although shouldn't really change much.
tomasfarias
left a comment
tomasfarias
left a comment
There was a problem hiding this comment.
I disagree with some of the concerns, but I don't really have a strong opinion about this tbh, and profiles do seem to also tackle the main pain point I was trying to avoid (having everybody run a service they do not need).
So, LGTM! 
❌ Hobby deploy smoke test: FAILEDFailing fast because: Timed out after 45 minutes Connection errors: 173 Run 20988887681 | Consecutive failures: 1 |
2 participants
Problem
When Azure Blob Storage batch exports were added (#43977), a separate docker-compose file was created at
products/batch_exports/backend/tests/docker-compose.ymlto avoid "polluting the top-level directory." This required passing-f products/batch_exports/backend/tests/docker-compose.ymlin CI and documentation.Why this is a devex concern
-fflag pattern is easy to forget and clutters CI commandsSolution
Move the Azurite service to
docker-compose.base.ymlwithprofiles: ['batch-exports']. The service only starts when--profile batch-exportsis passed, keeping the default dev stack unchanged.Gotcha
docker compose down(without profile) only stops services without profiles. CI now uses--profile '*'on down to ensure all services are cleaned up regardless of which profiles were used to start them.