backend: add Google Cloud storage support

[a-hilaly]

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Fixes #61

Proposed Changes

• Add cloud.google.com/go/storage dependency
• Implement Cloud Storage backend
• Add Cloud Storage to plugins config
• Edit flags/environment variables

Description

Checklist

• Read the CONTRIBUTING document. (It's checked since you are alread here.)
• Read the CODE OF CONDUCT document.
• Add tests to cover changes.
• Make sure your code follows the code style of this project
• Make sure CI and all other PR checks are green OR
• Code compiles correctly.
• Created tests which fail without the change (if possible).
• All new and existing tests passed.
• Add your changes to CHANGELOG.
• Improve and update the README (if necessary).
• Make sure documentation is up-to-date. Same file should also be updated in plugin index by sending a PR to that repository.

[kakkoyun]

@a-hilaly This looks great so far. Thanks a lot for the contribution, this is one of the most wanted backends because GCP is very popular nowadays.

I'm sure you've noticed our contribution guidelines, we're also trying to improve those, PTAL #71. It'd be great if you could complete the PR checklist and rebase your branch.

And ping me whenever you need a review.

[a-hilaly]

@kakkoyun Done ! For the CHANGELOG.md file should i add bump the version and add changes to a [1.0.5] section or juste update the [1.0.4] one ?

[kakkoyun]

@a-hilaly Thanks a lot. I'll have a look at it this weekend.
For the Changelog, could you please create an Unreleased titled section at the top and put your changes there. I'll handle the rest before release.

[kakkoyun]

So far so good, I have highlighted a couple of points. If you could fix those, that'd be great.
A major blocker is the lack of tests, We should add tests, if it's possible proper integration tests, you can check examples in plugin_test.go.

[a-hilaly]

@kakkoyun Thanks for the review, i made the changes as requested. I still have the e2e tests to write. Will need to understand how they are working in order to add the correct tests.

[kakkoyun]

Hey @a-hilaly, since you worked on GCP and aware of the AWS integration, I would like to have your opinion on https://github.com/google/go-cloud. This project is an abstraction over AWS, GCP and Azure. Maybe we can use this for object store integrations and provide integrations for other vendors like Azure as well. I'm not suggesting anything, just food for thought.

[a-hilaly]

I think that's a very good idea ! it will be a tree birds one stone deal 😄 i also like the fact that it uses only the standard sdk for each cloud nothing more https://github.com/google/go-cloud/blob/master/go.mod
drone-cache will also benefit whenever go-cloud implement support a new blop storage provider.
One thing i cannot find in go-cloud is encryption and ACL support, maybe i'm looking on the wrong place 🤔

[kakkoyun]

Thanks for the comment. Good point I'll check it.
Let's finish up with this PR since we are nearly at the end of it.
I have created an issue for go-cloud #76.

[kakkoyun]

@a-hilaly Would you be able to push this through? Let me know if you need any help.

[a-hilaly]

@kakkoyun Yes i'm working on this PR now. Need to rebase and add the plugin tests. Not sure how to use APIKey based access with minio (S3 uses SecretKey and AccessKey)

[kakkoyun]

@a-hilaly Cool thanks! I'm not sure if minio could be used to test google cloud storage locally, we need to check. If not there must be a someway to emulate google cloud storage, we should just find it.

[a-hilaly]

i made some research around GCS testing technics and i have found these two interesting projects:

• fake CS server we can run as a docker container (https://github.com/fsouza/fake-gcs-server)
• a mock library https://github.com/googleapis/google-cloud-go-testing/tree/master/storage/stiface

I personally like the fake gcs server it could be very useful to write e2e tests. What do you think @kakkoyun ?

[kakkoyun]

@a-hilaly I'm %100 with you on this one, just go with e2e tests. We only have e2e tests right now anyway.
fake-gcs-server it is 👍

[a-hilaly]

@kakkoyun i tried to add the end to end tests, bucket creation and object creation works as expected but reading object fails with a weird error:

~/source/drone-cache(gcs-support*) » go test -v -timeout 30s github.com/meltwater/drone-cache/cache/backend -run ^TestCloudStorageTruth
=== RUN   TestCloudStorageTruth
--- FAIL: TestCloudStorageTruth (0.71s)
    cloud_storage_test.go:66: googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>UserProjectAccountProblem</Code><Message>User project billing account not in good standing.</Message><Details>The billing account for project 47866975936 is disabled in state absent</Details></Error>
FAIL
FAIL    github.com/meltwater/drone-cache/cache/backend  0.860s
FAIL

i'll try to investigate more this error with different fake-gcs-servers versions

[kakkoyun]

Hey @a-hilaly, I will try to give this one a push this weekend or in the next week.

What's left? Do you need help with something specific?

[a-hilaly]

Hey @kakkoyun, i couldn't have all the end2end test fixed. Specifically this part https://github.com/A-Hilaly/drone-cache/blob/gcs-support/cache/backend/cloud_storage_test.go#L69-L73 . I'm not sure why this happens, i'm suspecting the fake-gcs-server.
EDIT: i remember trying to switch version for the fake-gcs-server because some of them were too buggy, i'll try to build my own image and see if that works

[kakkoyun]

@a-hilaly I have rebased it, and I'm going to merge it.
I'll address testing issues in #45 before we cut v1.1.0 release.