storage: support specifying a generation

[stephenplusplus]

Fixes #406
Fixes #498

• Docs
• Unit tests
  • Bucket
  • File
• System tests
• Needs to work

This is a first attempt at allowing a user to lock a File instance to a generation:

var gcloud = require("gcloud")({ projectId: "my-project" })
var storage = gcloud.storage()

var myFile = storage.file("yay.txt", { generation: 234234324 })
// use myFile as normally, however all operations are locked to that generation

Issue 1

Resolved 👍

#409 (comment)

How should normal operations on a File be changed to match a file locked to a generation?

These are pretty straightforward:

• copy() - specify sourceGeneration in the query
• createReadStream() / download() - specify generation in the query
• delete() - specify generation in the query
• getMetadata() - specify generation in the query
• setMetadata() - specify generation in the query

However, when you update the file with createWriteStream(), it uses ifGenerationMatch in the query to mean: only write to this file if the current generation on the remote file is the same as the generation specified for the File instance. So, if you're 3 generations back trying to update the file, it wouldn't work.

• should we use ifGenerationMatch in this way or should we just let the user write to the file regardless?
• should we update the generation lock on the File instance to the new generation?

Issue 2

Resolved 👍

Bucket must be created with {versioning: { enabled: true }}.

This has also been difficult to test with the real API. Here is a script I wrote to attempt to create a file, update it, then access its first state:

var async = require("async")
var gcloud = require("gcloud")({
  keyFilename: "/Users/stephen/dev/keyfile.json",
  projectId: "nth-circlet-705"
})

var storage = gcloud.storage()
var bucket = storage.bucket("stephens-new-bucket")

var fileExists = false
var generations = []

var fileName = Date.now()
var file = bucket.file(fileName)

async.eachSeries("ab".split(""), upload, log)

function upload(appendData, callback) {
  var writeStream = file.createWriteStream()
  writeStream.on("error", callback)
  writeStream.on("complete", function () {
    file.getMetadata(function (err, metadata) {
      if (err) return callback(err)

      console.log(appendData, "=>", metadata.generation)

      // store the generation of the file after writing to it
      generations.push(metadata.generation)

      // continue to the next append
      setTimeout(callback, 60000 * 20) // :-o - 20 minutes
    })
  })

  if (!fileExists) {
    // create the file
    writeStream.end(appendData)
  } else {
    // 1. get the current contents of the file
    // 2. append data to it
    // 3. write it to the remote file
    file.download(function (err, data) {
      if (err) return callback(err)

      writeStream.end(String(data) + appendData)
    })
  }
}

function log(err) {
  if (err) throw err

  async.eachSeries(generations, function (generation, callback) {
    console.log("downloading generation " + generation + "...")

    bucket.file(fileName, { generation: generation }).download(function (err, contents) {
      if (err) return callback(err)
      console.log(String(contents))
      callback()
    })
  }, console.log)
}

But that results in:

> node ./
a => 1424723174921000
b => 1424724376222000
downloading generation 1424723174921000...
{ [Error: Not Found]
  errors: [ { domain: 'global', reason: 'notFound', message: 'Not Found' } ],
  code: 404,
  message: 'Not Found',
  response: undefined }

I can use bucket.file(fileName, { generation: /* b's value */ }) fine, but I can't get a's. I'm inclined to believe this is me not understanding how to use generations, and not a bug in the code.

[ryanseys]

should we use ifGenerationMatch in this way or should we just let the user write to the file regardless?

Yes, we should use ifGenerationMatch. And the API call can fail, and they can handle that in some way.

should we update the generation lock on the File instance to the new generation?

Nope I think if they specify a generation, keep it at that generation. If they don't specify a generation, it always represents the latest version. If they specify a generation that HAPPENS to be the current generation, still keep it at that generation.

[ryanseys]

I think generation is an edge-case so having it fail in most scenarios is okay, only when it makes the most sense to use, should it be used. Also instead of trying to figure out or guess how it will be used by developers, I say delegate the logic to the API.

[stephenplusplus]

Revisiting this one to clean up the PR queue. Issues were resolved. Most importantly, versioning has to be enabled at the bucket level. We can perhaps create a convenience method for this (bucket.enableVersioning()) or just let it be done via metadata:

storage.createBucket('name', {
  versioning: {
    enabled: true
  }
})

storage.bucket('name').setMetadata({
  versioning: {
    enabled: true
  }
})

Either way, writing some tests and will ping when it's ready for a closer look.

[stephenplusplus]

This is ready for a closer review.

Calling bucket.getFiles({ versions: true }) will return version-locked File objects to your callback.

// @javorosas

[ryanseys]

Seems the tests are failing?

[stephenplusplus]

Not anymore!