GCP gcloud bulk-export as an audit trail #1981

bernokl · 2021-04-29T00:35:07Z

Terraformer to dump files from existing infrastructure to support K8s Infra Migration
Terraformer can be used to "reverse Terraform" existing infrastructure as part of the Migration of K8s Infrastructure.

To achieve this a person with full access to the relevant infrastructure is needed to run it for us

hh · 2021-04-29T00:56:07Z

/assign @spiffxp @dims @ameukam

spiffxp · 2021-04-29T21:23:55Z

Is this not runnable by members of the auditor group? What resources does this support? How is this different than gcloud supporting tf as an output format?

hh · 2021-04-30T03:42:38Z

I wasn't aware that gcloud supported tf as an output format. Do you have any links to docs? My bit of searching didn't seem to pull up anything.

I was able to get terraformer to work, however with DNS and get quite few errors:

terraformer import google --projects=kubernetes-public --resources=dns

....
Error when reading or editing DNSResourceDnsRecordSet \
  "k8s-dev/k8s.dev./SOA": googleapi: Error 403: Forbidden, forbidden
WARN: Fail read resource from provider, wait 300ms before retry
....

I was able to dump a few other items, but may need to re-run it for each region we have resources in:

terraformer import google --projects=kubernetes-public \
--resources=backendServices,bigQuery,firewall,gcs,gke,globalAddresses,globalForwardingRules,healthChecks,iam,instanceTemplates,monitoring,networks,project,pubsub,routes,sslCertificates,targetHttpProxies,targetHttpsProxies,urlMaps

Not sure if the output is ready for public consumption.

So I created a private repo and invited @ameukam @spiffxp and @dims to review before I make public.

https://github.com/ii/k8s-infra-terraformer-dump

(invites can be accepted by visiting https://github.com/ii/k8s-infra-terraformer-dump/invitations)

hh · 2021-04-30T03:58:42Z

List of supported Resources: https://github.com/GoogleCloudPlatform/terraformer/blob/master/docs/gcp.md

ameukam · 2021-04-30T08:37:44Z

@hh You can check the bulk export with gcloud here : https://cloud.google.com/sdk/gcloud/reference/alpha/resource-config/bulk-export.

bernokl · 2021-05-02T22:29:20Z

Thank you @ameukam
It looks like I am still missing some permissions?

gcloud alpha resource-config bulk-export \
--resource-format=terraform \
--organization=840466421052 \
--path=~/foo/dump1/ 

ERROR: (gcloud.alpha.resource-config.bulk-export) Error executing export:: 
[error in 'config-connector' version '1.45.0': 
error exporting asset inventory: error response from exportassets request: 
googleapi: Error 403: Request denied by Cloud IAM., forbidden

hh · 2021-05-02T23:37:26Z

$ config-connector bulk-export --output=./ --organization=840466421052

error in 'config-connector' version '1.46.0':
error exporting asset inventory:
error response from exportassets request:
googleapi: Error 403: Request denied by Cloud IAM., forbidden

$ gcloud auth list
         Credentialed Accounts
ACTIVE             ACCOUNT
*                  hh@ii.coop

hh · 2021-05-02T23:38:30Z

I'm not sure what the iam permission we need here.

bernokl · 2021-05-03T21:03:29Z

Terraformer has been running against k8s for 8 plus hours, still not finished.
I added the additional generated content to https://github.com/ii/k8s-infra-terraformer-dump.
I will add the rest when it is generated

spiffxp · 2021-05-06T02:57:46Z

#2010 should give you cloudasset permissions

bernokl · 2021-05-10T07:09:56Z

Is there something still outstanding to get permissions for gcloud bulk-export ?#1981 (comment)
I created an issue to track this in cncf-infra https://github.com/ii/cncf-infra/issues/37

ameukam · 2021-05-10T18:57:46Z

Thank you @ameukam
It looks like I am still missing some permissions?

gcloud alpha resource-config bulk-export \
--resource-format=terraform \
--organization=840466421052 \
--path=~/foo/dump1/ 

ERROR: (gcloud.alpha.resource-config.bulk-export) Error executing export:: 
[error in 'config-connector' version '1.45.0': 
error exporting asset inventory: error response from exportassets request: 
googleapi: Error 403: Request denied by Cloud IAM., forbidden

I don't recognize this organization ID. We use

 gcloud organizations describe 758905017065
creationTime: '2017-07-06T04:25:01.691Z'
displayName: kubernetes.io
lifecycleState: ACTIVE
name: organizations/758905017065
owner:
  directoryCustomerId: C01vqsvdr

bernokl · 2021-05-13T23:37:53Z

I was using project and not organization id This worked:

gcloud alpha resource-config bulk-export 
--resource-format=terraform 
--organization=758905017065 
--path=~/tmp/dump1/

Based on

gcloud organizations list
DISPLAY_NAME ID DIRECTORY_CUSTOMER_ID
ii.coop 65205373123 C03j7jfp3
kubernetes.io 758905017065 C01vqsvdr

TY for the assist from @ameukam

bernokl · 2021-05-14T21:04:38Z

To run bulk export on individual projects I need permissions to enable the cloudasset api. I can do this on sandbox, but not on individual projects.

For example:

Enabling service [cloudasset.googleapis.com] on project [k8s-infra-prow-build]...
ERROR: (gcloud.alpha.resource-config.bulk-export) PERMISSION_DENIED: Permission denied to enable service [cloudasset.googleapis.com]

From what I can see auditors should have the ability to view cloudassets, but I do not have the permissions to enable them.
Can I request enabling cloudasset on the k8s projects, or alternatively can I have a service account that can enable them for the duration of the audit script run?

riaankleinhans · 2021-05-16T17:10:32Z

/assign @bernokl

riaankleinhans · 2021-05-16T20:41:09Z

/retitle GCP gcloud bulk-export as an audit trail

spiffxp · 2021-05-16T22:24:17Z

I think you can enable and use it against your project. I suspect it doesn't need to be enabled in all projects you want to target

hh · 2021-05-17T04:35:19Z

We have it working against the sandbox.

What we are wanting is to use it as a replacement for https://github.com/kubernetes/k8s.io/blob/main/audit/audit-gcp.sh.

spiffxp · 2021-05-17T23:06:18Z

Right. Do you have a sample PR that does this? You are not blocked.

Gcloud has a billing project flag that can be used to signify which project you want to charge the service to, which could be your project for dev purposes. Otherwise it depends on who is invoking it and what their active project is. Service accounts default to the project they live in

bernokl · 2021-05-19T02:28:08Z

Thank you, here is an example of what an export would look like.
I ran:

gcloud alpha resource-config bulk-export 
--resource-format=terraform 
--project=k8s-infra-ii-sandbox
--billing-project=k8s-infra-ii-sandbox 
--path=/tmp/k8s-infra-ii-sandbox/

and got
https://github.com/ii/k8s-infra-dump/tree/main/resource-config-bulk-export

bernokl · 2021-05-19T03:00:46Z

I did test running the export specifying the billing-project:

gcloud alpha resource-config bulk-export 
--resource-format=terraform 
--project=k8s-staging-apisnoop 
--billing-project=k8s-infra-ii-sandbox 
--path=/tmp/k8s-staging-apisnoop

But still get permission denied:

API [cloudasset.googleapis.com] is required to continue, but is not 
enabled on project [k8s-staging-apisnoop]. Would you like to enable 
and retry (this will take a few minutes)? (y/N)?  y

Enabling service [cloudasset.googleapis.com] on project [k8s-staging-apisnoop]...
ERROR: (gcloud.alpha.resource-config.bulk-export) PERMISSION_DENIED: Permission denied to enable service [cloudasset.googleapis.com]

In the console I see:
You are missing the required permission serviceusage.services.enable on project k8s-staging-apisnoop

spiffxp · 2021-05-19T19:42:50Z

I'll need to read more about this command. I'm not sure we want this service enabled in every target project

spiffxp · 2021-05-20T22:19:46Z

/area infra/auditing

bernokl · 2021-05-20T23:16:56Z

Update.
I explored using the output of a bulk-export to create the tf files. With a one line change of the dumped file I can create the resources.
Here is a org file of my explorations: https://github.com/ii/org/blob/main/research/k8s-infra-wg/sandbox-infra-tf.org

From my brief exploration bulk-export is a very accurate depiction of our project. This is not a comment on further exploration of dangers of enabling serviceusage I understand security needs to be the first concern. Just sharing exciting findings.

spiffxp · 2021-05-21T00:22:29Z

My invite for https://github.com/ii/k8s-infra-dump/tree/main/resource-config-bulk-export expired, if you get a chance to resend I can take more of a look

bernokl · 2021-05-23T20:37:46Z

Resent. Sorry for the delay, let me know if you still have issues

k8s-triage-robot · 2021-08-21T21:25:38Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-ci-robot assigned ameukam, dims and spiffxp Apr 29, 2021

spiffxp mentioned this issue May 6, 2021

infra/gcp/roles: update custom org roles #2010

Merged

spiffxp added the wg/k8s-infra label May 7, 2021

k8s-ci-robot assigned bernokl May 16, 2021

k8s-ci-robot changed the title ~~Help running terraformer against infra we are duplicating as part of migrations~~ GCP gcloud bulk-export as an audit trail May 16, 2021

spiffxp mentioned this issue May 18, 2021

[Umbrella Issue] Auditing improvements #1657

Open

k8s-ci-robot added the area/audit Audit of project resources, audit followup issues, code in audit/ label May 20, 2021

spiffxp mentioned this issue Jun 11, 2021

Validate configuration changes #1734

Open

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 21, 2021

bernokl closed this as completed Aug 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCP gcloud bulk-export as an audit trail #1981

GCP gcloud bulk-export as an audit trail #1981

bernokl commented Apr 29, 2021

hh commented Apr 29, 2021

spiffxp commented Apr 29, 2021

hh commented Apr 30, 2021 •

edited

Loading

hh commented Apr 30, 2021

ameukam commented Apr 30, 2021

bernokl commented May 2, 2021 •

edited

Loading

hh commented May 2, 2021 •

edited

Loading

hh commented May 2, 2021

bernokl commented May 3, 2021

spiffxp commented May 6, 2021

bernokl commented May 10, 2021

ameukam commented May 10, 2021

bernokl commented May 13, 2021

bernokl commented May 14, 2021

riaankleinhans commented May 16, 2021

riaankleinhans commented May 16, 2021

spiffxp commented May 16, 2021

hh commented May 17, 2021

spiffxp commented May 17, 2021

bernokl commented May 19, 2021 •

edited

Loading

bernokl commented May 19, 2021 •

edited

Loading

spiffxp commented May 19, 2021

spiffxp commented May 20, 2021

bernokl commented May 20, 2021

spiffxp commented May 21, 2021

bernokl commented May 23, 2021

k8s-triage-robot commented Aug 21, 2021

GCP gcloud bulk-export as an audit trail #1981

GCP gcloud bulk-export as an audit trail #1981

Comments

bernokl commented Apr 29, 2021

hh commented Apr 29, 2021

spiffxp commented Apr 29, 2021

hh commented Apr 30, 2021 • edited Loading

hh commented Apr 30, 2021

ameukam commented Apr 30, 2021

bernokl commented May 2, 2021 • edited Loading

hh commented May 2, 2021 • edited Loading

hh commented May 2, 2021

bernokl commented May 3, 2021

spiffxp commented May 6, 2021

bernokl commented May 10, 2021

ameukam commented May 10, 2021

bernokl commented May 13, 2021

bernokl commented May 14, 2021

riaankleinhans commented May 16, 2021

riaankleinhans commented May 16, 2021

spiffxp commented May 16, 2021

hh commented May 17, 2021

spiffxp commented May 17, 2021

bernokl commented May 19, 2021 • edited Loading

bernokl commented May 19, 2021 • edited Loading

spiffxp commented May 19, 2021

spiffxp commented May 20, 2021

bernokl commented May 20, 2021

spiffxp commented May 21, 2021

bernokl commented May 23, 2021

k8s-triage-robot commented Aug 21, 2021

hh commented Apr 30, 2021 •

edited

Loading

bernokl commented May 2, 2021 •

edited

Loading

hh commented May 2, 2021 •

edited

Loading

bernokl commented May 19, 2021 •

edited

Loading

bernokl commented May 19, 2021 •

edited

Loading