[CAI-644] Add Clickhouse ECS service to the dev environment #1782
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds AWS Lambda Runtime Interface Emulator (RIE) to the development Dockerfile for local testing purposes. Updates README to include instructions for creating and filling in the Google service account JSON file.
Added `platform` specifications and restructured services, including new `langfuse-web`, `langfuse-worker`, `clickhouse`, and `minio` setups. Removed unused `networks` references and added health checks for critical services. These changes modernize and enhance the infrastructure to support new functionality and improve service reliability.
Corrected a misspelling in the README subtitle for better clarity. Ensures professionalism and avoids confusion for users following setup instructions.
Removed obsolete variables from .env.example and updated relevant configurations in docker/compose.yaml to ensure consistency. Adjusted ports and Langfuse initialization parameters to align with the updated environment setup.
Moved sensitive and configurable environment variables to `.env` files for improved readability and security. Updated comments in `docker/compose.yaml` to suggest consolidating environment variables into a dedicated `.env.langfuse` file.
# Conflicts: # apps/chatbot/docker/compose.yaml
…AI-302-add-clickhouse
…icies Introduced a new monitoring module defining resources for ECS tasks, services, ECR repositories, IAM roles, and related security groups. Includes lifecycle policies for ECR to retain 5 recent images and the deployment of Clickhouse with Fargate and EFS integration. Updated main and variables to integrate the module.
# Conflicts: # apps/chatbot/.env.example # apps/chatbot/docker/compose.yaml
Deleted an obsolete file containing initialization logs for Terraform backend. This file was unnecessary and contributed no functional value to the codebase, improving clarity and reducing repository clutter.
…o-v3' into CAI-302-upgrade-local-langfuse-to-v3 # Conflicts: # apps/chatbot/.env.example # apps/chatbot/docker/compose.yaml
…opa/developer-portal into CAI-302-upgrade-local-langfuse-to-v3
Deleted an obsolete file containing initialization logs for Terraform backend. This file was unnecessary and contributed no functional value to the codebase, improving clarity and reducing repository clutter.
Removed the `-p chatbot` flag from Docker Compose commands as it was unnecessary for the current setup. This improves maintainability and reduces potential confusion in script execution. No functional changes were introduced.
Uncommented and restored ECS cluster and EFS security group configuration to enable functionality. Switched ClickHouse image source to Docker Hub and added AWS region and VPC variables for deployment consistency. Deprecated ECR references for ClickHouse and adjusted related logic accordingly.
Introduce a new monitoring module to manage infrastructure components. Configure CloudWatch log groups, Secrets Manager integration, and service discovery for Clickhouse in ECS. Prepared resources for secure credentials and DNS namespace for service health checks.
Commented out unused security groups in `security_group.tf` to simplify resource management. Updated ECS environment variables to use local variables for better consistency and maintainability. Removed outdated instructions from `README.md` for improved clarity.
amanfrinati
requested review from
a team,
MarcoPonchia,
marcobottaro and
tommaso1
as code owners
🦋 Changeset detectedLatest commit: f5840bf The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Moved Langfuse-related environment variables from multiple files into a dedicated `.env.langfuse` for improved organization and maintenance. Updated `.gitignore`, `.env.example`, and Docker compose configurations to reflect the change. This centralization simplifies environment management and enhances clarity across the project.
Reintroduces previously commented-out IAM policy definitions and role policy attachments for ECS task execution and task roles in the Langfuse module. This ensures proper permissions for accessing AWS services such as Secrets Manager, S3, ECR, and Elastic FileSystem.
# Conflicts: # apps/chatbot/src/modules/chatbot.py
Reintroduces previously commented-out IAM policy definitions and role policy attachments for ECS task execution and task roles in the Langfuse module. This ensures proper permissions for accessing AWS services such as Secrets Manager, S3, ECR, and Elastic FileSystem.
Jira Pull Request LinkThis Pull Request refers to the following Jira issue CAI-644 |
|
This PR exceeds the recommended size of 800 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant updates to the chatbot's infrastructure and local development environment, focusing on monitoring, observability, and modernization of dependencies. The most notable changes are the addition of a new monitoring module, the upgrade of Langfuse to v3, and the integration of Clickhouse into the ECS stack. Several improvements were also made to Docker Compose configurations, including new services and streamlined scripts.
List of Changes
monitoringmodule toapps/infrastructure, including CloudWatch log groups forlangfuse_workerandclickhouse, and integrated it into the main stack.Motivation and Context
Mandatory to integrate Langfuse v3
How Has This Been Tested?
Screenshots (if appropriate):
Types of changes
Checklist: