redpajama_v1/book jsonl file is empty

[YilunKuang]

Hi,

I was trying to download redpajama_v1/book, but the following link corresponding to this dataset is unavailable:

https://data.together.xyz/redpajama-data-1T/v1.0.0/book/book.jsonl

[TianhuaTao]

Hi. I just checked the lastest url list from https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt . It seems they removed the books subset from it. And the original url for book is indeed not available now. It seems to be an issue on the Redpajama side, possibly related to copyright.

[TianhuaTao]

Yeah, a copyright issue for sure.
https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T

Gutenberg and Books3
Defunct: The 'book' config is defunct and no longer accessible due to reported copyright infringement for the Book3 dataset contained in this config.

[YilunKuang]

I see. Thanks for looking into it :) Feel free to close this issue.