Prevent returning cached entry if the entry is degenerate #1873
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When `DatabricksFileSystem.ls` is called with a directory, it will query the API and build a directory listing, then cache it. If a subsequent `ls` is called on a directory that is child to the first directory, the results that are returned are for the child directory itself, as though it were just a file. The correct behavior is to get the contents of this child direcctory. This PR ignores such responses so that `ls` will call the API again with the child directory to get its contents. These results are then cached as usual and available for future repeated `ls` calls with the child directory.
|
I think I've fixed the 3.9 test issues (annotation issues). Not sure if there's anything I can do about the 3.12 ones (rate-limiting issues) |
|
OK, let's give this a try and see how it goes for others |
|
The dbfs tests only run on py3.9 currently, and you can see they are now failing. This might just mean having to re-record the vcr data, but I am not sure. |
|
Sorry for the delay, my annual leave happened =) The VCR cassettes are re-recorded and the tests should pass now (they do locally). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When
DatabricksFileSystem.lsis called with a directory, it will query the API and build a directory listing, then cache it.If a subsequent
lsis called on a directory that is child to the first directory, the results that are returned are for the child directory itself, as though it were just a file. The correct behavior is to get the contents of this child directory.This PR ignores such responses so that
lswill call the API again with the child directory to get its contents. These results are then cached as usual and available for future repeatedlscalls with the child directory.This addresses issue: #1865 by overriding
AbstractFileSystem._ls_from_cachewith a new implementation that only uses the parent's cache entry if it indicates thatpathis not a "directory". If it is a "directory", nothing is returned to allow forDatabricksFileSystem.lsto make a new API call to DBFS to get the contents of that path.Also, if
pathis not found in the cached listing of its parent (if it exists), then it strongly suggests thatpathwas created since the parent was cached. Using this information,DatabricksFileSystem.lswill invalidate the parent's cache entry (by deleting it) before continuing.