load_dataset() function's cache_dir does not seems to work

[andyzhu]

Describe the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine.

It seems the cache_dir parameter cannot change the dataset saving directory from the default

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

Steps to reproduce the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine.

It seems the cache_dir parameter cannot change the dataset saving directory from the default

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

Expected behavior

the dataset should be saved to the cache_dir points to

Environment info

datasets version: 2.14.5
macos X: Ventura 13.4.1 (c)

[mariosasko]

Can you share the error message?

Also, it would help if you could check whether huggingface_hub's download behaves the same:

from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)

In the next major release, we aim to switch to huggingface_hub for file download/caching, but we could align the cache_dir's umask behavior earlier than this if their solution works for your use case.

[srinjoym-cerebras]

@mariosasko, Hey , is there any update on this?

[Dipeshtamboli]

Yes, I am still facing the same issue.

from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)

But this is working for me.

Thanks.

[srinjoym-cerebras]

Could you clarify a bit? What if I wanted only 100 samples from train split, how would i change the above? Thanks

[torotoki]

I have exactly same issue