blob.py is unable to handle a partial rewrite and fails with KeyError on lack of resource

[80nashi]

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/storage/google/cloud/storage/blob.py#L871

871:        self._set_properties(api_response['resource'])

This relies on the API to always return 'resource'. However, it is optional as per the doc.
https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite
"""
This property is present in the response only when copying completes.
"""

As such, when the write was partial, done: false is returned and resource not present. Thus failing with the following KeyError.

Traceback (most recent call last): 
File "xxx.py", line 304, in test_011_CopyFileFromBucketToBucketNewName 
new_destination_file_name=new_file_name) 
File "xxx.py, line 276, in copyFileFromBucketToBucket 
token, rewritten, total = destination_blob.rewrite(source_blob, token=token) 
File "/home/jenkins/venv/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 681, in rewrite 
self._set_properties(api_response['resource']) 
KeyError: 'resource' 

The most easy fix would be to push _set_properties down under after done: false check:

E.g.;

       api_response = client._connection.api_request(
            method='POST', path=source.path + '/rewriteTo' + self.path,
            query_params=query_params, data=self._properties, headers=headers,
            _target_object=self)
        # self._set_properties(api_response['resource'])  ## Comment out this line
        rewritten = int(api_response['totalBytesRewritten'])
        size = int(api_response['objectSize'])

        if api_response['done']:
            return None, rewritten, size

        self._set_properties(api_response['resource']) # and move here 
        return api_response['rewriteToken'], rewritten, size

[verbal]

I don't think your solution will work. I came across the same issue, but when it is not done, it falls through. You probably want to move it inside the if api_response['done'].

[80nashi]

Ah, you are totally right. Scrap my suggestion.
I wasn't sure how _set_properties is used in the API and I mistook the "done" check in the wrong way, of course.

[daspecster]

Thanks for reporting @80nashi!

If I understand correctly from the docs, the resource is only available when the copy is complete?

A resource containing the metadata for the copied-to object. This property is present in the response only when copying completes.

@tseaver, @dhermes maybe we should be raising an exception here?

I'm not sure from the docs, but is there polling that should be available for rewriting so you can check if the copy is done?

[80nashi]

As I understand the method and the API, the method is expected to return the rewrite token without raising an exception, in case of partial write (done: false and resource non-existent).
Then the user program can call the method again with the non-null, non empty string rewrite token, repeat it until done: true and resource present.

[80nashi]

So yeah, the following can be a possible fix as verba suggested.
However, I'm not sure what _set_properties is expected to do in case of done: false.

        # self._set_properties(api_response['resource'])  ## Comment out this line
        rewritten = int(api_response['totalBytesRewritten'])
        size = int(api_response['objectSize'])

        if api_response['done']:
            self._set_properties(api_response['resource']) # and move here
            return None, rewritten, size

        return api_response['rewriteToken'], rewritten, size

[daspecster]

@80nashi if you want to try and install from master it appears that @lukesneeringer addressed this.