Description
Running img2dataset with parquet as both input and output, using s3 as well (input/output).
At the end of the download it tries to delete the temp dir created for the feather files, but this dir is already removed.
I have removed the "fs.rm(tmp_dir...)" line to debug, no error and temp dir is removed.
When output is not an S3, it works as expected.
It seems that it might be related to the implementation of fsspec and s3fs.
After the last shard to be read is removed (last feather file), the whole path will be removed.
i did not test it this on other object storage, As a directory on S3 is just a "placeholder" (prefix), if this is the last file in the prefix, it will remove the whole prefix.
It is also not creating the temp dir first, as there are no files, it will create the temp dir just when placing the first feather file, and also delete the whole path when removing the last feather file.