feat: Support Bucket/Object lock operations

[stephenplusplus]

This is a recreation of the original PR: #320. Head there for discussions held during development of this feature.

To Dos

• Tests
  • System
  • Unit

References

• https://docs.google.com/document/d/1wRpiAxTHZoypR1NLVP97Cj0WCGVKx43Tivso0xXhyEA/edit?ts=5b60f543#
• https://www.googleapis.com/discovery/v1/apis/storage/v1/rest

This introduces new behavior:

Buckets

• bucket#lock()
  Lock a previously-defined retention policy. This will prevent changes to the policy.

• bucket#removeRetentionPeriod()
  Remove an already-existing retention policy from this bucket.

• bucket#setRetentionPeriod(durationInSeconds)
  Lock all objects contained in the bucket, based on their creation time.

Files

• file#getExpirationDate()
  Get a Date object representing the earliest time this file will expire.

---

Additionally, bucket#getMetadata(), bucket#setMetadata(), file#getMetadata(), and file#setMetadata() are available if users wish to interact with the raw resource schemas as defined by the API.

[stephenplusplus]

Just a ping @frankyn that this needs an official approval again (nothing has changed since the last PR I accidentally merged). That way, we're good to go when release time comes.

[frankyn]

Thanks @stephenplusplus, I think one additional convenience method similar to getExpirationDate() would be helpful for RetentionPolicy.effectiveTime.

[frankyn]

Sorry @stephenplusplus, it'd be better to just return the entire retentionPolicy object instead of just the getEffectiveTime() as it also contains isLocked another helpful boolean and the retentionPeriod.

Ignore my comments this is fine.

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

[JustinBeckwith]

@frankyn @stephenplusplus is this something we can update and merge?

[googlebot]

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

[stephenplusplus]

@JustinBeckwith ! We're trying to fit this change in with the TypeScript changes that have come recently. I'm getting lots of errors trying to run npm test that I thought you might be able to help with:

$ npm t

> @google-cloud/storage@2.0.3 test /Users/stephen/dev/nodejs-storage
> npm run test-only


> @google-cloud/storage@2.0.3 pretest-only /Users/stephen/dev/nodejs-storage
> npm run compile


> @google-cloud/storage@2.0.3 compile /Users/stephen/dev/nodejs-storage
> tsc -p .

system-test/storage.ts:905:14 - error TS2339: Property 'then' does not exist on type 'void'.

905             .then(() => bucket.getMetadata())
                 ~~~~

system-test/storage.ts:919:14 - error TS2339: Property 'then' does not exist on type 'void'.

919             .then(() => bucket.setRetentionPeriod(RETENTION_DURATION_SECONDS))
                 ~~~~

system-test/storage.ts:934:14 - error TS2339: Property 'then' does not exist on type 'void'.

934             .then(() => bucket.setRetentionPeriod(RETENTION_DURATION_SECONDS))
                 ~~~~

system-test/storage.ts:949:14 - error TS2339: Property 'then' does not exist on type 'void'.

949             .then(() => bucket.setRetentionPeriod(RETENTION_DURATION_SECONDS))
                 ~~~~

system-test/storage.ts:971:14 - error TS2339: Property 'then' does not exist on type 'void'.

971             .then(() => FILE.save('data'));
                 ~~~~

system-test/storage.ts:984:14 - error TS2339: Property 'then' does not exist on type 'void'.

984             .then(response => {
                 ~~~~

system-test/storage.ts:1000:14 - error TS2339: Property 'then' does not exist on type 'void'.

1000             .then(response => {
                  ~~~~

system-test/storage.ts:1015:41 - error TS2339: Property 'then' does not exist on type 'void'.

1015         return FILE.getExpirationDate().then(response => {
                                             ~~~~

system-test/storage.ts:1030:20 - error TS2345: Argument of type 'File' is not assignable to parameter of type 'never'.

1030         FILES.push(file);
                        ~~~~

system-test/storage.ts:1044:16 - error TS2339: Property 'setMetadata' does not exist on type 'never'.

1044           file.setMetadata({temporaryHold: null}, err => {
                    ~~~~~~~~~~~

system-test/storage.ts:1050:18 - error TS2339: Property 'delete' does not exist on type 'never'.

1050             file.delete(next);

Any words of wisdom as I stare down these angry lines?

[googlebot]

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

[stephenplusplus]

I went with just using the callback style for the methods that had issues, and it seems to work now.

System tests work locally for me.

[frankyn]

From System tests

 1) storage
       without authentication
         public data
           should list and download a file:
     Uncaught DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
      at emitWarning (internal/process/promises.js:92:15)
      at emitPendingUnhandledRejections (internal/process/promises.js:109:11)
      at process._tickCallback (internal/process/next_tick.js:189:7)
  2) storage
       without authentication
         private data
           should not download a file:
      Uncaught AssertionError [ERR_ASSERTION]: false == true
      + expected - actual
      -false
      +true
      at DestroyableTransform.file.download.err (system-test/storage.ts:159:11)
      at DestroyableTransform.f (node_modules/once/once.js:25:25)
      at /tmpfs/src/github/nodejs-storage/node_modules/through2/through2.js:19:12
      at _combinedTickCallback (internal/process/next_tick.js:131:7)
      at process._tickCallback (internal/process/next_tick.js:180:9)
  3) storage
       without authentication
         private data
           should not upload a file:
     Uncaught TypeError: Cannot read property 'indexOf' of undefined
      at Pumpify.file.save.err (system-test/storage.ts:167:27)
      at Pumpify.Duplexify._destroy (node_modules/duplexify/index.js:191:15)
      at /tmpfs/src/github/nodejs-storage/node_modules/duplexify/index.js:182:10
      at _combinedTickCallback (internal/process/next_tick.js:131:7)
      at process._tickCallback (internal/process/next_tick.js:180:9)
  4) storage
       write, read, and remove files
         stream write
           should resume an upload after an interruption:
     Uncaught Error: write after end
      at writeAfterEnd (node_modules/readable-stream/lib/_stream_writable.js:288:12)
      at Upload.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:332:20)
      at Duplexify._write (node_modules/duplexify/index.js:208:22)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at Duplexify.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at DestroyableTransform.ondata (node_modules/readable-stream/lib/_stream_readable.js:619:20)
      at addChunk (node_modules/readable-stream/lib/_stream_readable.js:291:12)
      at readableAddChunk (node_modules/readable-stream/lib/_stream_readable.js:278:11)
      at DestroyableTransform.Readable.push (node_modules/readable-stream/lib/_stream_readable.js:245:10)
      at DestroyableTransform.Transform.push (node_modules/readable-stream/lib/_stream_transform.js:148:32)
      at DestroyableTransform.afterTransform (node_modules/readable-stream/lib/_stream_transform.js:91:10)
      at DestroyableTransform.onData [as _transform] (node_modules/hash-stream-validation/index.js:26:5)
      at DestroyableTransform.Transform._read (node_modules/readable-stream/lib/_stream_transform.js:184:10)
      at DestroyableTransform.Transform._write (node_modules/readable-stream/lib/_stream_transform.js:172:83)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at DestroyableTransform.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at DestroyableTransform.ondata (node_modules/readable-stream/lib/_stream_readable.js:619:20)
      at addChunk (node_modules/readable-stream/lib/_stream_readable.js:291:12)
      at readableAddChunk (node_modules/readable-stream/lib/_stream_readable.js:278:11)
      at DestroyableTransform.Readable.push (node_modules/readable-stream/lib/_stream_readable.js:245:10)
      at DestroyableTransform.Transform.push (node_modules/readable-stream/lib/_stream_transform.js:148:32)
      at DestroyableTransform.afterTransform (node_modules/readable-stream/lib/_stream_transform.js:91:10)
      at DestroyableTransform.noop [as _transform] (node_modules/through2/through2.js:26:3)
      at DestroyableTransform.Transform._read (node_modules/readable-stream/lib/_stream_transform.js:184:10)
      at DestroyableTransform.Transform._write (node_modules/readable-stream/lib/_stream_transform.js:172:83)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at DestroyableTransform.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at Pumpify.Duplexify._write (node_modules/duplexify/index.js:208:22)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at Pumpify.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at DestroyableTransform.ondata (node_modules/readable-stream/lib/_stream_readable.js:619:20)
      at addChunk (node_modules/readable-stream/lib/_stream_readable.js:291:12)
      at readableAddChunk (node_modules/readable-stream/lib/_stream_readable.js:278:11)
      at DestroyableTransform.Readable.push (node_modules/readable-stream/lib/_stream_readable.js:245:10)
      at DestroyableTransform.Transform.push (node_modules/readable-stream/lib/_stream_transform.js:148:32)
      at DestroyableTransform._transform (system-test/storage.ts:1766:26)
      at DestroyableTransform.Transform._read (node_modules/readable-stream/lib/_stream_transform.js:184:10)
      at DestroyableTransform.Transform._write (node_modules/readable-stream/lib/_stream_transform.js:172:83)
      at doWrite (node_modules/readable-stream/lib/_stream_writable.js:428:64)
      at writeOrBuffer (node_modules/readable-stream/lib/_stream_writable.js:417:5)
      at DestroyableTransform.Writable.write (node_modules/readable-stream/lib/_stream_writable.js:334:11)
      at ReadStream.ondata (_stream_readable.js:639:20)
      at addChunk (_stream_readable.js:263:12)
      at readableAddChunk (_stream_readable.js:250:11)
      at ReadStream.Readable.push (_stream_readable.js:208:10)
      at fs.read (fs.js:2058:12)
      at FSReqWrap.wrapper [as oncomplete] (fs.js:658:17)

[stephenplusplus]

@JustinBeckwith could the system test errors above just be a Kokoro thing? Can anyone verify they're failing locally for them as well?

[frankyn]

@stephenplusplus when I run this locally I see the following:

npm install 

system-test/storage.ts:1564:47 - error TS2322: Type 'string | Buffer' is not assignable to type 'Uint8Array'.
  Type 'string' is not assignable to type 'Uint8Array'.

1564                   data = Buffer.concat([data, chunk]);

[googlebot]

A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

[JustinBeckwith]

@kinwa91 are we close on the system tests for kokoro passing here on master? If not, can you manually verify this PR is gtg?

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

[googlebot]

CLAs look good, thanks!

[JustinBeckwith]

Well @frankyn I'm pretty sure we fixed the system tests and exhausted the GCS bucket create quota 🤣 If you can fix that on long-door-651 this should be gtg. Anything I can do to help: