Optimize uploading to S3

[genben]

Is it necessary to send GET request to S3 every time the file is uploaded?

In methods S3FS.setbinfile() and S3FS.setbytes(), it sends two GET requests. First time, it checks if the the parent directory exists (calls S3FS.isdir()). Second time, it checks if the URL is not a directory (calls S3FS.getinfo()).

These operations are quite expensive. The effective upload speed increases at least x3 times when these methods are not called.

Here is the existing code:

def setbinfile(self, path, file):
    _path = self.validatepath(path)
    _key = self._path_to_key(_path)

    if not self.isdir(dirname(path)):
        raise errors.ResourceNotFound(path)
    try:
        info = self.getinfo(path)
        if info.is_dir:
            raise errors.FileExpected(path)
    except errors.ResourceNotFound:
        pass

    with s3errors(path):
        self.client.upload_fileobj(file, self._bucket_name, _key)

Proposed solution:

def setbinfile(self, path, file):
    _path = self.validatepath(path)
    _key = self._path_to_key(_path)

    with s3errors(path):
        self.client.upload_fileobj(file, self._bucket_name, _key)

And similar changes should be applied to setbytes() method.

[willmcgugan]

It is unfortunate, but in order to be a compliant implementation (and pass the tests), it has to throw the same set of exceptions as all the other implementations. I'm open to suggestion regarding optimisations. I'd also consider api additions that didn't break old interfaces.

[genben]

Maybe, just introduce the additional option to S3FS contstructor, which would disable/enable such validation?
Something like validate_dst_path=False (need better name).
For backward compatibility, the validation can be enabled by default.

[willmcgugan]

That's an idea. How about strict which defaults to True, but would omit expensive validation if set to False.

[genben]

Yes. This name ("strict") is much better.

[genben]

I have made the mentioned changes to the forked repository: miarec@756aff3

Probably, the comment could be improved.

[willmcgugan]

Looks good. Would you like to submit a PR for that? I'll get it in the next release.