add scripts for backups #377

listx · 2019-09-26T08:05:47Z

This is a rough cut of some tooling for basic backups from prod to a
backup.

listx · 2019-10-01T22:52:44Z

After this gets reviewed + merged, I plan on adding Prow jobs in k8s/test-infra repo to make use of these scripts at regular intervals (the more frequent, the better).

I could also port the "record_consistency.sh" logic into the promoter itself it seems like a conflict of interest, because the whole point of this PR is to add secondary checks to make sure the promoter is behaving properly.

listx · 2019-10-21T21:00:04Z

@thockin This is much simpler than the earlier PR (1 script instead of 3). PTAL

listx · 2019-10-23T00:18:33Z

This is blocked by #376

thockin

Shouldn't this be under infra/gcp ?

We have a turtle-at-the-bottom problem. We shift the focus to gcrane now - how can we build trust in that? I'd suggest we lock to a specific SHA256.

thockin · 2019-10-23T16:12:02Z

infra/backup_tools/README.md

+# Backup tools
+
+The tools in this folder are designed to backup the
+{asia,eu,us}.gcr.io/k8s-artifacts-prod GCRs.


If all 3 are supposed to be identical, do we need backups of all 3? Or is it just easier that way?

I only did it for symmetry (knee-jerk reaction).

Philosophically if I choose 1 region to back up out of the 3, then that 1 region becomes special and I don't like this.

thockin · 2019-10-23T16:13:03Z

infra/backup_tools/backup.sh

+
+# USAGE NOTES
+#
+# Backs up prod registries. This is a thin orchestrator as all the heavy lifting


comment implies copy_with_date.sh and record_consistency.sh are separate files, but no such file exists.

Oops, will fix.

thockin · 2019-10-23T16:15:34Z

infra/backup_tools/backup.sh

+
+# Copy each region to its backup.
+for prod_repo in "${prod_repos[@]}"; do
+    copy_with_date "${prod_repo}" "${prod_repo}-backup" "${sa_key_path}"


NB I changed "-backup" to "-bak" in latest push of the ensure script (because we overflowed allowable project names :)

thockin · 2019-10-23T16:17:22Z

infra/backup_tools/backup.sh

+    docker run \
+        -v "${sa_key_path}":/auth.json \
+        --env GOOGLE_APPLICATION_CREDENTIALS=/auth.json \
+        gcr.io/go-containerregistry/gcrane \


Do we need a particular tag? Assuming :latest is kind of scary...

What protects this backup tool from being compromised?

Do we need a particular tag? Assuming :latest is kind of scary...

Yes, I should have referenced a sha256 digest here. Will change to not use :latest.

What protects this backup tool from being compromised?

Good question. For now I think the sha256 is the only guarantee. If we need to update we would have to run some e2e test that ensures that the new version doesn't change behavior. Short of that, at least manual testing of some sort.

thockin · 2019-10-23T16:49:05Z

Also, if pinning to a SHA, comment why :)

listx · 2019-10-23T18:51:31Z

I removed the unnecessary call to gcloud (activating creds) because they are only used inside the gcrane container.

thockin

Thanks!

/lgtm
/approve

thockin · 2019-10-29T15:30:22Z

infra/gcp/backup_tools/backup.sh

+    # folders more easily traversable from a human perspective.
+    timestamp="$(date -u +"%Y/%m/%d/%H")"
+
+    source_gcr_repo="${1}" # "us.gcr.io/k8s-artifacts-prod"


you should probably check $# and abort if the args are the wrong number, ok for followup

This is a rough cut of some tooling for basic backups from prod to a backup. The `backup.sh` script will run from a periodic Prow job every hour. This script expects the "k8s-artifacts-prod-backup" GCR to exist, as well as a service account that has write access to this backup location.

listx · 2019-10-30T15:54:39Z

/assign @cblecker

rajibmitra · 2019-11-01T13:12:18Z

infra/gcp/backup_tools/backup.sh

+
+    # Perform backup by copying all images recursively over.
+    docker run \
+        -v "${sa_key_path}":/auth.json \


will it be a good idea to use buildkit here @listx

I haven't heard of that. Care to elaborate? :)

Please ignore me. all looks good.

cblecker · 2019-11-02T22:31:54Z

/lgtm
/approve

k8s-ci-robot · 2019-11-02T22:32:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cblecker, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cblecker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot assigned justinsb Sep 26, 2019

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 26, 2019

k8s-ci-robot assigned thockin Sep 26, 2019

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 26, 2019

k8s-ci-robot requested review from cblecker and nikhita September 26, 2019 08:06

k8s-ci-robot added the wg/k8s-infra label Sep 26, 2019

listx force-pushed the dr branch from 7d82fa7 to 54b4cdf Compare October 21, 2019 19:04

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 21, 2019

listx changed the title ~~WIP: add scripts for disaster recovery~~ add scripts for backups Oct 21, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2019

listx mentioned this pull request Oct 22, 2019

add backup job kubernetes/test-infra#14904

Merged

thockin reviewed Oct 23, 2019

View reviewed changes

listx force-pushed the dr branch 2 times, most recently from c4ab926 to fd0bb3e Compare October 23, 2019 18:49

listx force-pushed the dr branch from fd0bb3e to 3660813 Compare October 23, 2019 20:03

listx requested a review from thockin October 28, 2019 21:59

thockin reviewed Oct 29, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 29, 2019

listx force-pushed the dr branch from 3660813 to b49a3c3 Compare October 30, 2019 15:53

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 30, 2019

k8s-ci-robot assigned cblecker Oct 30, 2019

rajibmitra reviewed Nov 1, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 2, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 2, 2019

k8s-ci-robot merged commit 2d26f91 into kubernetes:master Nov 2, 2019

listx deleted the dr branch June 22, 2020 03:51

rajibmitra mentioned this pull request Sep 6, 2021

REQUEST: New membership for rajibmitra kubernetes/org#2954

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add scripts for backups #377

add scripts for backups #377

listx commented Sep 26, 2019 •

edited

Loading

listx commented Oct 1, 2019

listx commented Oct 21, 2019

listx commented Oct 23, 2019

thockin left a comment

thockin Oct 23, 2019

listx Oct 23, 2019

thockin Oct 23, 2019

listx Oct 23, 2019

thockin Oct 23, 2019

listx Oct 23, 2019

thockin Oct 23, 2019

listx Oct 23, 2019

thockin commented Oct 23, 2019

listx commented Oct 23, 2019

thockin left a comment

thockin Oct 29, 2019

listx Oct 30, 2019

listx commented Oct 30, 2019

rajibmitra Nov 1, 2019

listx Nov 1, 2019

rajibmitra Sep 4, 2021

cblecker commented Nov 2, 2019

k8s-ci-robot commented Nov 2, 2019

add scripts for backups #377

add scripts for backups #377

Conversation

listx commented Sep 26, 2019 • edited Loading

listx commented Oct 1, 2019

listx commented Oct 21, 2019

listx commented Oct 23, 2019

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Oct 23, 2019

listx commented Oct 23, 2019

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

listx commented Oct 30, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cblecker commented Nov 2, 2019

k8s-ci-robot commented Nov 2, 2019

listx commented Sep 26, 2019 •

edited

Loading