[FR]GCP CloudSQL Project Configurability

[stwomack]

Is your feature request related to a problem? Please describe.
GCP has quota limits on GCP projects for managed services such as cloudsql service instances. The default is 40.
GCP recommends provisioning cloudsql instances in a new gcp project once Service instance quota is reached.

Describe the solution you'd like
It would be great feature to add to cloud service broker to provide gcp project as a parameter during provisioning of creating service such as cloudsql instances, so that gcp service quota limit constraint can be overcome when using CF create service in TAS.

Describe alternatives you've considered
Bumping the GCP Quota for max CloudSQL instances in a project. Google recommends against this.

Additional Context
GCP Service broker currently cannot be configured to provision cloudsql instances in multiple GCP projects.

Priority
Low

Priority Context
This looks like it might already be possible with this broker. Just depends on if the config option will be in the tile config.

Platform
GCP

Applicable Services
CloudSQL

[erniebilling]

@stwomack Each service supports the project ID as a provision parameter and it is set as a global default. I is possible to override at provision time currently with cf create-service ... -c '{"project":"<your project id>"}'

It is also possible to configure as part of a plan definition, so an administrator could offer plans to developers that target specific projects.

Are there other configuration options you might like to see?

[stwomack]

@erniebilling What you've explain goes beyond the request, but I'll share it with them to get their thoughts. Thanks!

[dominikmueller]

We are using the original GCP Service Broker (with the builtin provider option / direct API calls) in our company and modified it, so our services get deployed to separate projects.
But first of all: great to see that there are efforts in releasing a terraform based service-broker! We've been waiting for it a long time.

I can share a pitfall/limitation that comes to my mind for this feature:

Using the private IP feature of CloudSQL might collide with the approach to implement multi-project capability.

1. Looking at the current implementation/terraform code, it seems that bind operations are run using the 'terraform postgresql-provider'.
   To my understanding this provider needs direct access to the database instance, meaning that the service broker running the terraform-workspace needs to have access to the private IP of the CloudSQL instance.
   Since the private IP is only accessible through a VPC peering with the Google services VPC network, it is only accessible from within the same project where the instance was created. Thus it would be difficult to access the instance from outside the project running the service broker / the service broker would not be able to connect and manage databases and users.

2. Creating private CloudSQL instances in another project than where the CF foundation is deployed might lead to the same issue as described above. Apps would not be able to connect to the private IP of the instance -> they would need some kind of tunnel running in the target project.

Using the gcp terraform provider to create bindings might solve the first issue (since database- and user-management would get handled through the sqladmin API) but still - afaik - there would not be a solution to access the private IPs from another project because of VPC Peering Restrictions.

We for our part created a tool which creates a "service-project" for each cf Organization. Inside the GCP SB we are mapping the ORG-GUIDs to Project-IDs and because of the above restrictions, we are not using the private IP feature.
All our instance management is handled through the cloudsql API and our apps use the cloudsql proxy implementation to connect to the cloudsql instance (no matter where it resides).

Please let me know when I missed something or when my explanation is faulty! :-)

[omerbensaadon]

@dominikmueller thank you for the thoughtful reply! I'll be sure ENG follows up...

In the meantime, feel free to join us in the #cloudservicebroker-beta channel in the Cloud Foundry Slack! Would love to get you involved in some of the conversations we've had over there.

[erniebilling]

@dominikmueller You bring up two issues on our radar, but not yet fully hashed out.

The first seems to be addressable with the approach taken for mysql in https://github.com/GoogleCloudPlatform/service-broker-plugins and we will investigate.

Could the second be solved with VPC Sharing instead of VPC Peering? Do private IP's work in this scenario?

[omerbensaadon]

@dominikmueller @stwomack I've put a Spike in the backlog to reflect this question... we will certainly get to addressing this problem in a clean (as possible) fashion before GA

[omerbensaadon]

Spike completed!

Should be able to use https://www.terraform.io/docs/providers/google/r/sql_user.html for user credentials on mysql, postgres and mssql.

This is being worked now, and #44 should be fixed as a result of this change! Thank you for escalating this @stwomack @dominikmueller!

I'll make sure to close this Issue with a reference to that pull request.

[erniebilling]

We understand this problem now, but likely won't fix before GA. If it becomes an issue for customers, we will happily revisit.