Figure out how to do browser-based uploading, and document it well.

[jgeewax]

So there's this concept of "browser-based uploads" which basically is the following process:

1. Client sends the server an (untrusted) policy document (basically says "I'm going to upload file which is 10mb and has metadata = {...}).
2. Server gets this request, validates it, and if it likes it, sends back a signature (basically says "SGTM, here's my stamp of approval").
3. Client is able to upload exactly that data directly to Cloud Storage, it's "credentials" for this single request being the policy document (and the signature).

So... a couple of questions:

1. Can we even use gcloud-node in the client-side of a web-app?
2. If so, we need a simple way of creating typical policy documents (as well as weird ones).
3. Then we need a way of generating a signature (server-side) for a policy document.
   Note that we're not wiring together any RPCs or anything. Just providing a method that takes a policy and spits back a signature for the user to serialize and send around however they want.
4. Then we need a way of calling upload (or similar) using the policy document and signature.

The code I'm expecting here would look like...

Server-side:

bucket.signPolicy(policy, function(signature) {
  // The signature can be serialized or whatever else we want now.
});

Client-side:

var policy = bucket.policy({ /* Lots of configuration here... */ });
$.post({ /* Send the policy up to the method above */ }, function(response) {
  // Create an HTML form (?) that has a field called signature = response.data ?
});

/cc @Capstan

[stephenplusplus]

@jgeewax @stellanhaglund will this solution work?

• Docs: https://cloud.google.com/storage/docs/access-control?hl=en#signing-resumable
• Issue: Authorization: Bearer your_auth_token (how to get auth_token) #818
• Solution: Authorization: Bearer your_auth_token (how to get auth_token) #818 (comment)
  (this can be simplified with https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.21.0/storage/file?method=createResumableUpload)

[stephenplusplus]

I made a couple of examples here showing two different ways to handle uploads. https://github.com/stephenplusplus/gcloud-upload-example

[stellanhaglund]

Thanks alot!!

I just added that but now I'm trying to download a file using

app.get('/file/:imageId', function(req, res){

var remoteFile = bucket.file(req.params.imageId);
var localFilename = 'uploads/' + req.params.imageId;

var src = remoteFile.createReadStream()

src
.on('error', function(err) {})
.on('response', function(response) {
// Server connected and responded with the specified status and headers.
})
.on('end', function() {
// The file is fully downloaded.
})
// .pipe(fs.createWriteStream(localFilename));

res.attachment(localFilename);
src.pipe(res);

})

but for larger files I just get about 1.2 mb downloaded..

2015-09-17 1:27 GMT+02:00 Stephen Sawchuk notifications@github.com:

I made a couple of examples here showing two different ways to handle
uploads. https://github.com/stephenplusplus/gcloud-upload-example

—
Reply to this email directly or view it on GitHub
#641 (comment)
.

[stephenplusplus]

Which version of gcloud are you running?

[stellanhaglund]

"gcloud": "^0.21.0",

2015-09-17 15:28 GMT+02:00 Stephen Sawchuk notifications@github.com:

Which version of gcloud are you running?

—
Reply to this email directly or view it on GitHub
#641 (comment)
.

[stephenplusplus]

Hmm, I haven't been able to reproduce. How large are the files you're having trouble with? I've tried with a 10MB file without issues.

I added onto the example with some options of how to download files: https://github.com/stephenplusplus/gcloud-upload-example/blob/1d6cbecb31aa2dcd98d1c8177e6c474b80b5d1f5/server.js#L89

[stellanhaglund]

about 300mb, and the files will probably be up to some gb's

2015-09-17 16:16 GMT+02:00 Stephen Sawchuk notifications@github.com:

Hmm, I haven't been able to reproduce. How large are the files you're
having trouble with? I've tried with a 10MB file without issues.

I added onto the example with some options of how to download files:
https://github.com/stephenplusplus/gcloud-upload-example/blob/1d6cbecb31aa2dcd98d1c8177e6c474b80b5d1f5/server.js#L89

—
Reply to this email directly or view it on GitHub
#641 (comment)
.

[stephenplusplus]

Would you mind opening a new issue about this? Also, please test that it still doesn't work isolated from express, e.g.:

remoteFile.createReadStream().pipe(fs.createWriteStream('./backup-file'))

[stephenplusplus]

I think we've got this one covered with file#createResumableUpload().

Related, I just stumbled upon this project over the weekend: https://github.com/makeomatic/ms-files-gce

If there's more we can do, let me know!