-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bugfixes: python3, s3 paths with subfolders #178
Conversation
@danilop I'm leaving this PR open for review, but for our actual use we have reverted to commit c5cf4f8 (before PR 167 was merged), with the python 3 fixes from here, because we were running into various issues with the more recent changes on master. One fairly major concern was that the randomly named temp cache folders never get removed, and a new one is created each time yas3fs is restarted. Even if the cache is cleared, these (usually) empty folders apparently hang around forever. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello @rebkwok. Thanks for your commits!
I've already tried to fix this part and ended up with the following construction:
data = self.cache.get(path, 'data')
if data and not os.path.exists(self.cache.get_cache_filename(path)):
logger.debug("Cache leak found for '%s', cleaning up..." % (path))
self.cache.delete(path)
with self.cache.lock and self.cache.new_locks[path]:
del self.cache.new_locks[path]
del self.cache.unused_locks[path]
os.path.exists(path) didn't work for me well in some situations. See PR#168
@rebkwok , it would be great if you could fix the conflicts! |
If the s3 prefix is more than one folder deep, tempfile.mkdtemp interprets the prefix as a path with subfolders and fails because the subfolder it's looking for doesn't exist
Conflicts fixed, but I'm still dubious about actually using this as per previous comment |
Are you talking about the temp folders which aren't got removed? That might be similar issue to this one: #14 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure this commit is required anymore - 399acdb
@rebkwok would you please create a new PR with your changes against the latest master? |
@jazzl0ver I rebased this PR already when I fixed the conflicts, so it is against the latest master |
@rebkwok, the point is that the items 5 and 6 from your PR seem to revert the changes I made before to fix some bugs. May I ask you to remove that items from this PR, so I could merge the ones that seem OK. For the items 5 and 6 - let's discuss them separately to make sure they won't break anything. |
I have updated the PR description. The replace() fix (item 5) is still needed, but i have revised that commit. Item 6 fixes some inconsistent tabs/spaces - not sure when those were introduced, but they stop yas3fs from even starting up for me locally. |
A few fixes for python 3 bugs and a couple of others I came across when upgrading to the latest master:
e
when raising the last exception, however it will never get to this point withe
available, because it comes from a previously raised exception.replace()
, otherwise raises an error on python 3