Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfixes: python3, s3 paths with subfolders #178

Merged
merged 6 commits into from
Oct 9, 2018

Conversation

rebkwok
Copy link
Contributor

@rebkwok rebkwok commented Aug 28, 2018

A few fixes for python 3 bugs and a couple of others I came across when upgrading to the latest master:

  1. d98c9b0: YAS3FSPlugin. load_from_file tries to use e when raising the last exception, however it will never get to this point with e available, because it comes from a previously raised exception.
  2. 95e9402: Wrap some dict keys in list() when iterating over them, otherwise it fails on python 3 because it's modifying the dict while iterating over it
  3. bcbd913: Python 3 doesn't let you raise simple strings as exceptions any more
  4. f4b8e39: The changes from commit 692e36c in the last PR merged introduce a bug if your s3 path has subfolders. (e.g. if my s3 path is s3://mybucket/foo/bar it will try to do tempfile.mkdtemp with the prefix 'mybucket-foo/bar-random' which fails because it's looking for a folder '/tmp/mybucket-foo/' which doesn't exists. This just replaces / with - in the s3_prefix so the temp folder created is 'mybucket-foo-bar-random'. Also updates the README for --cache-path to match the updated parameter help text
  5. 9739764: make sure dirname in replace_empty_dirs is a str before doing replace(), otherwise raises an error on python 3
  6. 40cb448: Fix inconsistent tabs and spaces introduced at some point which are making things break for me (yas3fs will not start up without this fix)

@rebkwok
Copy link
Contributor Author

rebkwok commented Sep 11, 2018

@danilop I'm leaving this PR open for review, but for our actual use we have reverted to commit c5cf4f8 (before PR 167 was merged), with the python 3 fixes from here, because we were running into various issues with the more recent changes on master. One fairly major concern was that the randomly named temp cache folders never get removed, and a new one is created each time yas3fs is restarted. Even if the cache is cleared, these (usually) empty folders apparently hang around forever.

Copy link
Collaborator

@jazzl0ver jazzl0ver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @rebkwok. Thanks for your commits!
I've already tried to fix this part and ended up with the following construction:

            data = self.cache.get(path, 'data')
            if data and not os.path.exists(self.cache.get_cache_filename(path)):
                logger.debug("Cache leak found for '%s', cleaning up..." % (path))
                self.cache.delete(path)
                with self.cache.lock and self.cache.new_locks[path]:
                    del self.cache.new_locks[path]
                del self.cache.unused_locks[path]

os.path.exists(path) didn't work for me well in some situations. See PR#168

@jazzl0ver
Copy link
Collaborator

@rebkwok , it would be great if you could fix the conflicts!

Becky Smith added 4 commits October 2, 2018 16:20
If the s3 prefix is more than one folder deep, tempfile.mkdtemp
interprets the prefix as a path with subfolders and fails because the
subfolder it's looking for doesn't exist
@rebkwok
Copy link
Contributor Author

rebkwok commented Oct 2, 2018

Conflicts fixed, but I'm still dubious about actually using this as per previous comment

@jazzl0ver
Copy link
Collaborator

Are you talking about the temp folders which aren't got removed? That might be similar issue to this one: #14

Copy link
Collaborator

@jazzl0ver jazzl0ver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure this commit is required anymore - 399acdb

@jazzl0ver
Copy link
Collaborator

@rebkwok would you please create a new PR with your changes against the latest master?

@rebkwok
Copy link
Contributor Author

rebkwok commented Oct 4, 2018

@jazzl0ver I rebased this PR already when I fixed the conflicts, so it is against the latest master

@jazzl0ver
Copy link
Collaborator

@rebkwok, the point is that the items 5 and 6 from your PR seem to revert the changes I made before to fix some bugs. May I ask you to remove that items from this PR, so I could merge the ones that seem OK. For the items 5 and 6 - let's discuss them separately to make sure they won't break anything.

@rebkwok
Copy link
Contributor Author

rebkwok commented Oct 9, 2018

I have updated the PR description. The replace() fix (item 5) is still needed, but i have revised that commit. Item 6 fixes some inconsistent tabs/spaces - not sure when those were introduced, but they stop yas3fs from even starting up for me locally.

@jazzl0ver jazzl0ver merged commit 18a894c into danilop:master Oct 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants