ci-k8sio-audit periodic job #20742

hh · 2021-02-04T18:18:54Z

Developing using https://github.com/kubernetes/test-infra/blob/master/prow/build_test_update.md#using-pj-on-kindsh

k8s-ci-robot · 2021-02-04T18:18:56Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hh · 2021-02-04T22:12:18Z

I’m trying to use ./config/pj-on-kind.sh to verify a job that needs to use gcloud. It will eventually run in run in the k8s-infra-prow-build-trusted cluster and
use the k8s-infra-gcp-auditor GCP service account via workload identity. I’m trying to figure how how to pass my own GCP , gcloud auth login generated creds into the local prow job so it has access to run gcloud w/ audit permissions.... will cross post this to #sig-testing in slack.

spiffxp

Some suggestions

spiffxp · 2021-02-04T23:25:09Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+    testgrid-alert-email: k8s-infra-alerts@kubernetes.io
+    testgrid-num-failures-to-alert: '1'


I would change these while you're iterating on this job

spiffxp · 2021-02-04T23:26:46Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+        ./audit-gcp.sh
+        git diff
+        git add --all
+        git commit -m "audit: update as of $(date +%Y-%m-%d)"


As a start, you could have this job fail if there are changes to commit, and pass if there are none

I may swing back around to that. Trying to get a full run to ensure all the tooling works.

hh · 2021-02-10T19:23:45Z

/retitle ci-k8sio-audit periodic job

spiffxp

Some suggestions

spiffxp · 2021-02-10T19:35:42Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+      - -c
+      - |
+        echo "Some initial debuging to figure out what it looks like here" >&2
+        pwd


periodic jobs will have the working directory set to the root of the repo specified by the first ref in extra_refs (if specified)

ref: https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md

Thanks, that's useful!

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

spiffxp · 2021-02-10T19:39:02Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+        git config user.name "Kubernetes Prow Robot"
+        git config user.email "k8s.ci.robot@gmail.com"


This should correspond to the GH_USER being used, and I would advise we not use k8s.ci.robot

spiffxp · 2021-02-10T19:40:26Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+        echo -n "Calculate github user from token: " >&2
+        GH_TOKEN=$(cat /etc/github-token/oauth)
+        GH_USER=$(curl -H "Authorization: token $GH_TOKEN" "https://api.github.com/user" 2>/dev/null | sed -n "s/\s\+\"login\": \"\(.*\)\",/\1/p")
+        FORK_GH_BRANCH=autoaudit-${PROW_INSTANCE_NAME:-prow}


not blocking but what's the purpose of the PROW_INSTANCE_NAME substitution?

It's a bit of copy-pasta from here:

test-infra/prow/cmd/autobump/autobump.sh

Lines 29 to 31 in ff516b7

# Set this to something more specific if the repo hosts multiple Prow instances.

# Must be a valid to use as part of a git branch name. (e.g. no spaces)

PROW_INSTANCE_NAME="${PROW_INSTANCE_NAME:-prow}"

Possibly not necessary as I don't expect there to be multiple FORK_GH_BRANCH, but I suspect this repo does host multiple prow instances.

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

spiffxp · 2021-02-10T19:47:57Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+    volumes:
+    - name: github
+      secret:
+        secretName: oauth-token


I need to double check if we have this. If we don't, I suggest we not use k8s-ci-robot, and for the short-term use a user under your control (using fejta-bot as the model, which is not an org member)

Hmm... I'm not sure how to load secrets like this into the production prow instance.
I generated an oauth-token for @cncf-ci and can send that in a secure way as soon as I learn how. :)

Suggested change

secretName: oauth-token

secretName: cncf-ci-github-token

gcloud secrets versions access \ --project=k8s-infra-prow-build-trusted \ --secret=cncf-ci-github-token \ latest | \ kubectl --context=gke_k8s-infra-prow-build-trusted_us-central1_prow-build-trusted \ apply -f -

secret/cncf-ci-github-token created

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

It's not complete yet, but it may be eaiser to merge and iterate. Co-authored-by: Aaron Crickenberger <spiffxp@google.com>

hh · 2021-02-11T20:05:02Z

Created a google secret in the format of a k8s secret in the test-pods namespace:

kubectl create secret  --dry-run=client \
  -n test-pods \
  generic cncf-ci-github-token \
  --from-literal=token=${GH_TOKEN} \
  -o yaml \
| gcloud secrets versions add \
  --project=k8s-infra-prow-build-trusted \
  cncf-ci-github-token \
  --data-file=-

Created version [1] of the secret [cncf-ci-github-token].

spiffxp

Added the secret @hh transferred to me via secret manager, job needs some config updates to match name of/in the secret

spiffxp · 2021-02-11T20:24:39Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+    volumes:
+    - name: github
+      secret:
+        secretName: oauth-token


Suggested change

secretName: oauth-token

secretName: cncf-ci-github-token

gcloud secrets versions access \ --project=k8s-infra-prow-build-trusted \ --secret=cncf-ci-github-token \ latest | \ kubectl --context=gke_k8s-infra-prow-build-trusted_us-central1_prow-build-trusted \ apply -f -

secret/cncf-ci-github-token created

spiffxp · 2021-02-11T20:27:49Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+        echo "Ensure gcloud creds are working" >&2
+        gcloud config list
+        echo -n "Calculate github user from token: " >&2
+        GH_TOKEN=$(cat /etc/github-token/oauth)


Suggested change

GH_TOKEN=$(cat /etc/github-token/oauth)

GH_TOKEN=$(cat /etc/github-token/token)

Fixed in 7663f60

This also adds dynamically pulling the email and name via the token rather than needing to hard code them in the job. Note that it's the default email that is used for the token provided. Also switched to parsing curl output via jq rather than sed. GNU sed and BSD sed act a bit differently and I was getting inconsistent results on OSX vs Linux.

spiffxp

/approve
/lgtm
/wg k8s-infra
/sig testing

k8s-ci-robot · 2021-02-11T20:42:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hh, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/wg-k8s-infra/trusted/OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2021-02-11T20:53:48Z

@hh: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key wg-k8s-infra-trusted.yaml using file config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

In response to this:

Fixes kubernetes/k8s.io#244

Developing using https://github.com/kubernetes/test-infra/blob/master/prow/build_test_update.md#using-pj-on-kindsh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

spiffxp · 2021-02-11T21:04:18Z

config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml

+  extra_refs:
+  - org: kubernetes
+    repo: k8s.io
+    base_ref: master


https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-k8sio-audit/1359969296511406080

d'oh, this should have been main not master, my bad for missing this on review

k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Feb 4, 2021

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 4, 2021

k8s-ci-robot requested review from bartsmykla and dims February 4, 2021 18:19

hh force-pushed the ci-k8sio-audit branch from cd651ca to c0fb0b3 Compare February 4, 2021 18:22

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 4, 2021

hh changed the title ~~ci-k8sio-audit perodic job / fixes #244~~ WIP: ci-k8sio-audit perodic job / fixes #244 Feb 4, 2021

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 4, 2021

spiffxp suggested changes Feb 4, 2021

View reviewed changes

k8s-ci-robot assigned spiffxp Feb 4, 2021

hh force-pushed the ci-k8sio-audit branch 4 times, most recently from 8de871c to d7c5700 Compare February 10, 2021 18:37

k8s-ci-robot changed the title ~~WIP: ci-k8sio-audit perodic job / fixes #244~~ ci-k8sio-audit periodic job Feb 10, 2021

k8s-ci-robot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. labels Feb 10, 2021

spiffxp suggested changes Feb 10, 2021

View reviewed changes

Add ci-k8sio-audit perodic job

bf51391

It's not complete yet, but it may be eaiser to merge and iterate. Co-authored-by: Aaron Crickenberger <spiffxp@google.com>

hh force-pushed the ci-k8sio-audit branch from 3393d07 to bf51391 Compare February 11, 2021 02:48

spiffxp mentioned this pull request Feb 11, 2021

results of running audit script as of 2021-01-13 kubernetes/k8s.io#1534

Merged

spiffxp suggested changes Feb 11, 2021

View reviewed changes

spiffxp approved these changes Feb 11, 2021

View reviewed changes

k8s-ci-robot added wg/k8s-infra sig/testing Categorizes an issue or PR as relevant to SIG Testing. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Feb 11, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 11, 2021

k8s-ci-robot merged commit 492140b into kubernetes:master Feb 11, 2021

k8s-ci-robot added this to the v1.21 milestone Feb 11, 2021

spiffxp reviewed Feb 11, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci-k8sio-audit periodic job #20742

ci-k8sio-audit periodic job #20742

hh commented Feb 4, 2021

k8s-ci-robot commented Feb 4, 2021

hh commented Feb 4, 2021

spiffxp left a comment

spiffxp Feb 4, 2021

spiffxp Feb 4, 2021

hh Feb 11, 2021

hh commented Feb 10, 2021

spiffxp left a comment

spiffxp Feb 10, 2021

hh Feb 11, 2021

spiffxp Feb 10, 2021

spiffxp Feb 10, 2021

hh Feb 11, 2021

spiffxp Feb 10, 2021

hh Feb 11, 2021

spiffxp Feb 11, 2021

hh commented Feb 11, 2021 •

edited

Loading

spiffxp left a comment

spiffxp Feb 11, 2021

spiffxp Feb 11, 2021

hh Feb 11, 2021

spiffxp left a comment

k8s-ci-robot commented Feb 11, 2021

k8s-ci-robot commented Feb 11, 2021

spiffxp Feb 11, 2021

		testgrid-alert-email: k8s-infra-alerts@kubernetes.io
		testgrid-num-failures-to-alert: '1'

		git config user.name "Kubernetes Prow Robot"
		git config user.email "k8s.ci.robot@gmail.com"

	# Set this to something more specific if the repo hosts multiple Prow instances.
	# Must be a valid to use as part of a git branch name. (e.g. no spaces)
	PROW_INSTANCE_NAME="${PROW_INSTANCE_NAME:-prow}"

	GH_TOKEN=$(cat /etc/github-token/oauth)
	GH_TOKEN=$(cat /etc/github-token/token)

ci-k8sio-audit periodic job #20742

ci-k8sio-audit periodic job #20742

Conversation

hh commented Feb 4, 2021

k8s-ci-robot commented Feb 4, 2021

hh commented Feb 4, 2021

spiffxp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hh commented Feb 10, 2021

spiffxp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hh commented Feb 11, 2021 • edited Loading

spiffxp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 11, 2021

k8s-ci-robot commented Feb 11, 2021

Choose a reason for hiding this comment

hh commented Feb 11, 2021 •

edited

Loading