Skip to content

Commit

Permalink
[s3] Properly exclude empty folders during 'listdir'
Browse files Browse the repository at this point in the history
Users can create empty folders on S3, by creating an object which name
ends with a forward slash, for example `folder/`. Using the `listdir`
operation on an empty folder returns `[], ["."]`. This is unexpected,
there are no file named `.` in the folder.

The reason for its presence is AWS S3 returning the following content
when listing the content of the directory:
```
{
    'Contents': [
        {... 'Key': 'folder/', ...}
    ],
   ...
}
```
Then, `posixpath.relpath("folder/", "folder/")` returns ".".
  • Loading branch information
francoisfreitag authored and jschneier committed Sep 19, 2021
1 parent fe6dd57 commit 66f4f8e
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 1 deletion.
1 change: 1 addition & 0 deletions AUTHORS
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ By order of apparition, thanks:
* Zoe Liao (S3 docs)
* Jonathan Ehwald
* Dan Hook
* François Freitag (S3)


Extra thanks to Marty for adding this in Django,
Expand Down
4 changes: 3 additions & 1 deletion storages/backends/s3boto3.py
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,9 @@ def listdir(self, name):
for entry in page.get('CommonPrefixes', ()):
directories.append(posixpath.relpath(entry['Prefix'], path))
for entry in page.get('Contents', ()):
files.append(posixpath.relpath(entry['Key'], path))
key = entry['Key']
if key != path:
files.append(posixpath.relpath(key, path))
return directories, files

def size(self, name):
Expand Down
21 changes: 21 additions & 0 deletions tests/test_s3boto3.py
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,27 @@ def test_storage_listdir_subdir(self):
self.assertEqual(dirs, ['path'])
self.assertEqual(files, ['2.txt'])

def test_storage_listdir_empty(self):
# Files:
# dir/
pages = [
{
'Contents': [
{'Key': 'dir/'},
],
},
]

paginator = mock.MagicMock()
paginator.paginate.return_value = pages
self.storage._connections.connection.meta.client.get_paginator.return_value = paginator

dirs, files = self.storage.listdir('dir/')
paginator.paginate.assert_called_with(Bucket=None, Delimiter='/', Prefix='dir/')

self.assertEqual(dirs, [])
self.assertEqual(files, [])

def test_storage_size(self):
obj = self.storage.bucket.Object.return_value
obj.content_length = 4098
Expand Down

0 comments on commit 66f4f8e

Please sign in to comment.