feat(webhook-datasource): add enabling of Cloud pubsub API (SSPROD-45515)

[SKosier]

No description provided.

[haresh-suresh]

@SKosier I understand the idea here to auto enable the apis needed for CDR. We already have a different way we want to do this in Modular install : https://www.figma.com/design/Ew1XcXFjr5ipwqx30TY2gN/D1456---Modular-Onboarding?node-id=6072-251832&node-type=SECTION&t=DByL7dyOSfVEmuEW-0

Full modular GCP design : https://www.figma.com/design/Ew1XcXFjr5ipwqx30TY2gN/D1456---Modular-Onboarding?node-id=4016-683964&node-type=CANVAS&t=6KzcKts8m2HpAloO-0

Basically as a separate step / TF prior to running the install TF scripts. The reasons for this are :

• We want to give the user agency over what they would like enable before running it
• We also want this general patten before each feature getting installed

Note this is already in development and aimed to be tested and released in a couple of weeks time.
So I would request you to close this one out and we will refer all future installs to use the modular GCP install workflow.

Let me know if there are any questions.
cc: @nkraemer-sysdig

[SKosier]

Hey @haresh-suresh , @nkraemer-sysdig !

while modular GCP is on-the-way, we need to find the way to make current GCP onboarding flow 'working'. Often, our customers successfully onboard, the terraform deployment is successful, and all the right roles are provisioned. However, customers still see no events. I believe I have found the reason behind this issue.

For Pub/Sub service to work without any problems, there has to be a dedicated service agent created under-the-hood. Small context: service agents are created automatically when certain APIs or services are enabled in a project, and they handle the internal operations required to manage and execute (they are also Google-managed and hidden from users in the UI).

I managed to reproduce the issue our customers have and I noticed, the 'no-events issue' happens when pubsub service agent @gcp-sa-pubsub.iam.gserviceaccount.com is not present. One of the reasons for the missing pubsub service agent is if Pub/Sub API (in charge of creating it) has never been enabled.

I understand the reasons behind having a unique behavior towards enabling different APIs, but if we can make onboarding flow robust while we wait for the modular-CDR, I vote for that path forward.

cc @mbertrone @legobrick

[haresh-suresh]

After discussing offline, we agreed that it makes sense to go ahead with change in the uber / legacy TF snippets, with the following caveats:

• We will not follow this pattern in Modular GCP install, in that case there will be a separate step for enablement for all apis needed.
• This change only targets one project, whereas the one in Modular will attempt to auto enable it across all projects in an org
• @SKosier will verify to see, if we need to update the pre-requisites in the legacy / uber wizards to make sure that an user has the right roles to perform the api enablement

Approving this in the meanwhile 👍