[GCP] Cloud Pipeline shall support Cloud SQL #4009
Description
Background
The goal of this task is to migrate the Cloud Pipeline platform from its current self-hosted PostgreSQL 16 instance to Google Cloud SQL. Google Cloud SQL is a fully managed database service that offers greater reliability, built-in backups, automated updates, and seamless scalability. By adopting Cloud SQL, the platform can reduce the operational overhead associated with managing a self-hosted database, while simultaneously improving the stability and maintainability of the system.
The transition will require the platform to be configured to utilize Google Cloud SQL as the primary backend for storing application state and metadata. This includes updating connection configurations, ensuring proper support for Cloud SQL's authentication mechanisms, such as IAM-based access, and implementing secure data transfer between the application and the managed database. Infrastructure-as-Code deployment scripts and documentation will also need to be updated to include automated provisioning of Cloud SQL instances with appropriate configurations.