Repository navigation
Fetching file in subfolder returns ResourceNotFound, error fetching info #17
Description
Opening file on subfolder couses ResourceNotFound error.
Versions:
- fs (2.0.17)
- fs-s3fs (0.1.5)
To reproduce
Working:
In [1]: import fs_s3fs
In [2]: ffs = fs_s3fs.S3FS('bucket', dir_path='test')
In [3]: with ffs.open('999', 'wb') as f: f.write(b'test')
In [4]: with ffs.open('999', 'rb') as f: f.read()
Not working
In [5]: import fs_s3fs
In [6]: ffs = fs_s3fs.S3FS('bucket')
In [7]: with ffs.open('test/999', 'wb') as f: f.write(b'test')
In [8]: with ffs.open('test/999', 'rb') as f: print(f.read())
Stacktrace
----> 1 with ffs.open('test/999', 'rb') as f: print(f.read())
/usr/lib/python3.5/site-packages/fs/base.py in open(self, path, mode, buffering, encoding, errors, newline, **options)
965 validate_open_mode(mode)
966 bin_mode = mode.replace('t', '')
--> 967 bin_file = self.openbin(path, mode=bin_mode, buffering=buffering)
968 io_stream = iotools.make_stream(
969 path,
/usr/lib/python3.5/site-packages/fs_s3fs/_s3fs.py in openbin(self, path, mode, buffering, **options)
537
538 if self.strict:
--> 539 info = self.getinfo(path)
540 if info.is_dir:
541 raise errors.FileExpected(path)
/usr/lib/python3.5/site-packages/fs_s3fs/_s3fs.py in getinfo(self, path, namespaces)
421 self._get_object(dir_path, _dir_key)
422 except errors.ResourceNotFound:
--> 423 raise errors.ResourceNotFound(path)
424
425 if _path == '/':
ResourceNotFound: resource 'test/999' not found
Activity
willmcgugan commentedon Nov 22, 2017
I think I know why that might be. Because S3 doesn't support true directories, S3FS creates empty files to represent directories. But in your first example there is no
test/directory.If you do the following prior to your first example, does it work?
Hernrup commentedon Nov 23, 2017
Thanks for the fast answer!
Yes, it does work if
makediris introduced.The wierd thing is that
with ffs.open('test/999', 'wb') as f: f.write(b'test')does produce the file in the non-working example. It is only not able to retrieve it again. Its a very unexpected and inconsistent behaviour.My suggestion would be to either:
mkdirwillmcgugan commentedon Nov 23, 2017
Yes, you're right. I'm leaning towards documenting it. The alternative would be to introduce additional checks (i.e. requests), which may be too expensive. Although there is already a
strictflag to relax these requirements.In general though, if you use S3FS to create everything, you shouldn't see any surprising behaviour. The test suite is extensive.
m0nzderr commentedon Dec 4, 2017
+1
willmcgugan commentedon Jan 31, 2018
I've documented this behaviour here.
Unfortunately I don't see a solution that wouldn't be prohibitively expensive.
Hernrup commentedon Jan 31, 2018
It's absolutely sufficient that it's documented. Thank you for taking the time. Unfortunately the documentation does not touch on this issue. It's states that as long as you use s3fs to create the file you should be safe. That is clearly not the case here. The behaviour that needs documenting is that you will have a bad time if you don't explicitly create the subfolder when creating a file with s3fs.
willmcgugan commentedon Feb 1, 2018
Yes, you're right. I conflated this with another issue. The correct behaviour would be to throw an exception if you try to open a file in a directory that doesn't exist.
Hernrup commentedon Feb 1, 2018
My opinion is that the correct behaviour whould be to create the folder if it does not exist when the user is trying to create a file in a subfolder.
That whould keep the internal s3 structure as intended and is from my point of view the expected behaviour.
It is not an issue on read but rather that write with s3fs can corrupt the s3 filestructure.
Throwing an error stating that the folder is missing on write whould be sufficient in mittigating the confusing behaviour. But I insist that the fix is needed an write and not read.
willmcgugan commentedon Feb 1, 2018
The behaviour is dictated by the PyFilesystem interface. S3FS has to remain consistent with the other implementations.
Hernrup commentedon Feb 1, 2018
Of course. Thank you for you're efforts and the quick responses. I'll see if I can get the time to send you a PR. I am fairly certain this should be possible without breaking any assumptions or interfaces but I could ofcourse be wrong.
Documenting this for now whould be very acceptable.
Thanks once again.
willmcgugan commentedon Feb 2, 2018
Fixed in 0.1.7