Support for GCS WAL-E backups #620

alfredw33 · 2019-07-15T14:45:05Z

Added the ability to support the following environment variables:

WAL_GS_BUCKET
GOOGLE_APPLICATION_CREDENTIALS

to be passed down to the spilo images. This should address #198.

alfredw33 · 2019-08-03T13:33:50Z

I've updated the scripts to check for Darwin (MacOS) and removed mac specific things.

I changed some things that require your eyes :

Moved the -r option from the Makefile to right after the cp command.
Added the requirement for greadlink if you are on Mac to use run.sh (If you are okay with that)

Other then that everything seems to be working and tests are passing. Do you think we can merge this PR in now?

FxKu · 2019-08-16T10:29:05Z

Thanks for your contribution and also for providing tests. I'm fine with the support for two more Spilo env variables, but would ask you to move all the changes on the e2e setup to another PR to have a cleaner separation. Is that ok for you?

Jan-M · 2019-08-16T10:42:02Z

docs/reference/operator_parameters.md

+  The default is empty.
+
+* **gcp_credentials**
+  Used to set the GOOGLE_APPLICATION_CREDENTIALS environment variable for the pods.


I don't immedately see where you in fact use the secret mount part?

Having the secrets in pod env var like this exposes them to a lot of readers, would be a proper secret ref be the better choice.

I thought we support all required google fields already.

I thought so too and I could be wrong on what it would take to get this working in GCP without my changes but from reading the documentation it wasn't clear on how to get this to work for GCP.

GOOGLE_APPLICATION_CREDENTIALS env variable will get populated with a file path where a credentials.json file (or whatever you call it) to the service account who has the IAM rules to push to the GCS configured bucket. This service account is then used be SPILO to push the WAL-E information the GCS bucket.

We then use the file mount features to mount the credientails.json to the same path as the path specified by the GOOGLE_APPLICATION_CREDENTIALS

We are using this in production for our applications in GCP. Would it be helpful on giving an example on how we did this?

Yes, it might be worth to add a sub-chapter here in the admin docs. Wondering if #946 could also be used in the future, but for now, let's take the setup which you use in production

FxKu · 2019-11-27T11:02:17Z

@alfredw33 could you rebase and remove the changes in Makefile and run.sh script of e2e tests, so that this PR is only about introducing GCE config parameters? We have switched to go modules in the meantime, which also simplified the e2e setup.

Sorry, for the long pause.

alfredw33 · 2019-12-05T14:54:26Z

I will try to do it this weekend. Sorry for the late reply.

schmitch · 2020-03-16T15:56:31Z

is there any update on this? @alfredw33

alfredw33 · 2020-03-25T23:53:51Z

Hey,

I'll take a look at how to split these out into two different commits this weekend. I will commit to delivering the makefile / run.sh changes to support Mac development. If I have time I'll get the other commit in.

Should I Cancel this PR and make two separate ones? Does that make more sense?

FxKu · 2020-03-26T11:06:34Z

@alfredw33 we are now using go modules to install kind so the makefile changes should not apply anymore. Simply remove these changes and rebase with master.

alfredw33 · 2020-05-06T21:17:34Z

Hey,

I've rebased and made the changes to get my tests to run. One thing I'm struggling with right now is that I'm getting errors on running the e2e tests where it complains about the go-client for k8. There seems to be some issues around this and considering I haven't updated any dependencies I'm not sure why this is not working for me.

Is there a specific go version I should be using. I've upgraded to 1.14.2 but maybe I need to use 1.14?

…ment variables

alfredw33 · 2020-05-12T03:11:24Z

@FxKu I think I got most of the asks in. Let me know if there is anything else you want me to do.

Thanks!

FxKu · 2020-05-13T08:02:33Z

charts/postgres-operator/values.yaml

+  # GCS Bucket to use for shipping WAL segments with WAL-E
+  # wal_gs_bucket: ""
+
+  # GCP Credentials for setting the GOOGLE_APPLICATION_CREDNETIALS environment variable


Suggested change

# GCP Credentials for setting the GOOGLE_APPLICATION_CREDNETIALS environment variable

# GCP credentials for setting the GOOGLE_APPLICATION_CREDENTIALS environment variable

FxKu · 2020-05-13T08:03:08Z

charts/postgres-operator/values.yaml

@@ -200,6 +200,12 @@ configAwsOrGcp:
  # S3 bucket to use for shipping WAL segments with WAL-E
  # wal_s3_bucket: ""

+  # GCS Bucket to use for shipping WAL segments with WAL-E


Suggested change

# GCS Bucket to use for shipping WAL segments with WAL-E

# GCS bucket to use for shipping WAL segments with WAL-E

FxKu · 2020-05-13T08:14:52Z

manifests/postgresql-operator-default-configuration.yaml

@@ -91,6 +91,8 @@ configuration:
    # kube_iam_role: ""
    # log_s3_bucket: ""
    # wal_s3_bucket: ""
+    # wal_gs_bucket: ""
+    # gcp_credentials: ""


Could you add these examples also in the configmap? But please with alphabetical sorting there.

FxKu · 2020-05-13T08:20:12Z

Thanks @alfredw33 ! Few minor things:

Could you cover the two new configuration fields in the validation of the OperatorConfiguration CRD, too? Here's a guide to the files, you have to change.
Please, do not change the go.mod and go.sum files. Reset your changes there. There's still one small change in the e2e run shellscript, which isn't necessary.

alfredw33 · 2020-05-19T01:50:04Z

Thanks @alfredw33 ! Few minor things:

Could you cover the two new configuration fields in the validation of the OperatorConfiguration CRD, too? Here's a guide to the files, you have to change.

Please, do not change the go.mod and go.sum files. Reset your changes there. There's still one small change in the e2e run shellscript, which isn't necessary.

@FxKu - I think I completed what you asked for. Let me know if I've missed anything.

docs/administrator.md

e2e/run.sh

FxKu · 2020-05-22T09:16:11Z

👍

kungfuchicken · 2020-05-29T15:43:44Z

@alfredw33 @FxKu any progress on this? I realize it's only a week since last update, but I'm holding on a deployment pending the merge ;)

alfredw33 · 2020-06-02T04:16:37Z

@alfredw33 @FxKu any progress on this? I realize it's only a week since last update, but I'm holding on a deployment pending the merge ;)

I wasn't aware that I needed to do something else. Did I miss anything that is preventing you to merge. Looks like 1/2 approvals have been given.

Jan-M · 2020-06-03T15:24:25Z

👍

FxKu · 2020-06-03T15:35:03Z

Thanks @alfredw33 for your contribution and patience ;)

alfredw33 · 2020-06-13T13:23:20Z

@FxKu It's my pleasure! 😃

alfredw33 requested review from avaczi, CyberDem0n, erthalion, FxKu, Jan-M, RafiaSabih and sdudoladov as code owners July 15, 2019 14:45

alfredw33 force-pushed the master branch from 53aece3 to 14afcff Compare July 29, 2019 19:17

Jan-M reviewed Aug 16, 2019

View reviewed changes

alfredw33 added 3 commits May 7, 2020 09:55

Support for WAL_GS_BUCKET and GOOGLE_APPLICATION_CREDENTIALS environt…

5ff7a0a

…ment variables

Fixed merge issue but also removed all changes to support macos.

3dab520

Updated test to new format

664554f

alfredw33 force-pushed the master branch from f2d06d8 to 664554f Compare May 9, 2020 17:49

Missed macos specific changes

d0c6d27

alfredw33 force-pushed the master branch from 1a676a8 to d0c6d27 Compare May 9, 2020 19:49

FxKu reviewed May 13, 2020

View reviewed changes

FxKu added this to the 1.6 milestone May 13, 2020

alfredw33 force-pushed the master branch from 93849cc to 25afe63 Compare May 18, 2020 19:46

Added documentation and addressed comments

c782bc2

alfredw33 force-pushed the master branch from 25afe63 to c782bc2 Compare May 18, 2020 23:26

FxKu reviewed May 22, 2020

View reviewed changes

docs/administrator.md Outdated Show resolved Hide resolved

Update docs/administrator.md

601da9e

FxKu reviewed May 22, 2020

View reviewed changes

docs/administrator.md Outdated Show resolved Hide resolved

Update docs/administrator.md

28fce10

FxKu reviewed May 22, 2020

View reviewed changes

e2e/run.sh Outdated Show resolved Hide resolved

Update e2e/run.sh

f574266

FxKu merged commit 2b0def5 into zalando:master Jun 3, 2020

	# GCP Credentials for setting the GOOGLE_APPLICATION_CREDNETIALS environment variable
	# GCP credentials for setting the GOOGLE_APPLICATION_CREDENTIALS environment variable

	# GCS Bucket to use for shipping WAL segments with WAL-E
	# GCS bucket to use for shipping WAL segments with WAL-E

Support for GCS WAL-E backups #620

Support for GCS WAL-E backups #620

Uh oh!

Conversation

alfredw33 commented Jul 15, 2019

Uh oh!

alfredw33 commented Aug 3, 2019

Uh oh!

FxKu commented Aug 16, 2019

Uh oh!

Jan-M Aug 16, 2019

Choose a reason for hiding this comment

Uh oh!

alfredw33 May 11, 2020

Choose a reason for hiding this comment

Uh oh!

FxKu May 13, 2020

Choose a reason for hiding this comment

Uh oh!

FxKu commented Nov 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alfredw33 commented Dec 5, 2019

Uh oh!

schmitch commented Mar 16, 2020

Uh oh!

alfredw33 commented Mar 25, 2020

Uh oh!

FxKu commented Mar 26, 2020

Uh oh!

alfredw33 commented May 6, 2020

Uh oh!

alfredw33 commented May 12, 2020

Uh oh!

FxKu May 13, 2020

Choose a reason for hiding this comment

Uh oh!

FxKu May 13, 2020

Choose a reason for hiding this comment

Uh oh!

FxKu May 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FxKu commented May 13, 2020

Uh oh!

alfredw33 commented May 19, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FxKu commented May 22, 2020

Uh oh!

kungfuchicken commented May 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alfredw33 commented Jun 2, 2020

Uh oh!

Jan-M commented Jun 3, 2020

Uh oh!

FxKu commented Jun 3, 2020

Uh oh!

alfredw33 commented Jun 13, 2020

Uh oh!

Uh oh!

FxKu commented Nov 27, 2019 •

edited

Loading

FxKu May 13, 2020 •

edited

Loading

kungfuchicken commented May 29, 2020 •

edited

Loading