Get dataset size from DVC data registry

[RafayAK]

Hi, I would like to request a feature to query dataset sizes from a DVC registry.

I have created a data registry using the documentation and added a dataset to my remote storage through the simple dvc add my_dataset and dvc push commands.
Now, before I import the dataset from a registry I want to make sure I have enough disk space. So, something like the following would be really useful.

dvc du -hs https://github.com/my_data_registry my_dataset

• du: which stands for Disk Usage is a standard Unix command to query file sizes.
  options h and s represent the following:
  - h: is a human-readable format so in bytes, megabytes, gigabytes etc
  - s: is to sum the sizes of all files in a directory.

If the DVC team thinks there are better ways to do this without adding a new feature I'm all ears.

[efiop]

Hi @RafayAK .

Thanks for the request! We were thinking about adding sizes to dvc list output as a part of #3256 , but dvc du also sounds pretty good. It would be easy to implement after the pre-requisites for #3256 are implemented, which is planned to be out in the upcoming weeks.

[copaah]

I am moving this from the Discord chat.

When these ideas are implemented, a nice feature would be for dvc pull to estimate the size of whatever it is pulling and then check if the required disk space is available.

I have encountered long dvc pulls fails, because I hadn't ensured that sufficient disk space was available.

[jorgeorpinel]

I think something like dvc list --size could be enough (avoid a new command) and indeed like @copaah mentioned, pulling (fetching really) and maybe also import/get could include this check by default and throw an error if there's not enough size in the workspace drive.

[pared]

For the record, discussion mentioned by @copaah is here.

[efiop]

For the record. We currently have sizes saved into dvcfiles/lockfiles (though they are not guaranteed to be there) and we have a info/size() calls in our fs (e.g. RepoFileSystem), so a very nice first iteration could leverage that as a quick and simple solution with a new dvc du command.

Good example repo to try this out on would be https://github.com/iterative/dataset-registry , where we have size/nfiles fields in dvc-files there.