Add option to delete temporary files (e.g. extracted files) when loading dataset

[thomwolf]

I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution.

[lhoestq]

Hi !
If we want something more general, we could either

1. delete the extracted files after the arrow data generation automatically, or
2. delete each extracted file during the arrow generation right after it has been closed.

Solution 2 is better to save disk space during the arrow generation. Is it what you had in mind ?

The API could look like

load_dataset(..., delete_extracted_files_after_usage=True)

In terms of implementation, here are some directions we could take for each solution:

1. get the list of the extracted files from the DownloadManager and then delete them after the dataset is processed. This can be implemented in download_and_prepare I guess
2. maybe wrap and mock open in the builder to make it delete the file when the file is closed.

[thomwolf]

Also, if I delete the extracted files they need to be re-extracted again instead of loading from the Arrow cache files

[albertvillanova]

I think we already opened an issue about this topic (suggested by @stas00): duplicated of #2481?

This is in our TODO list... 😅

[albertvillanova]

I think the deletion of each extracted file could be implemented in our CacheManager and ExtractManager (once merged to master: #2295, #2277). 😉

[thomwolf]

Oh yes sorry, I didn't check if this was a duplicate

[albertvillanova]

Nevermind @thomwolf, I just mentioned the other issue so that both appear linked in GitHub and we do not forget to close both once we make the corresponding Pull Request... That was the main reason! 😄

[thomwolf]

Ok yes. I think this is an important feature to be able to use large datasets which are pretty much always compressed files.

In particular now this requires to keep the extracted file on the drive if you want to avoid reprocessing the dataset so in my case, this require using always ~400GB of drive instead of just 200GB (which is already significant).

Two nice features would be to:

• allow to delete the extracted files without loosing the ability to load the dataset from the cached arrow-file
• streamlined decompression when only the currently read file is extracted - this might require to read the list of files from the extracted archives before processing them?

[thomwolf]

Here is a sample dataset with 2 such large compressed JSON files for debugging: https://huggingface.co/datasets/thomwolf/github-python

[thomwolf]

Note that I'm confirming that with the current master branch of dataset, deleting extracted files (without deleting the arrow cache file) lead to re-extracting these files when reloading the dataset instead of directly loading the arrow cache file.

[lhoestq]

Hi ! That's weird, it doesn't do that on my side (tested on master on my laptop by deleting the extracted folder in the download cache directory). You tested with one of the files at https://huggingface.co/datasets/thomwolf/github-python that you have locally ?