How to load the dataset?

[zhangshuibai]

Hi, I noticed the dataset you provided in huggingface stores meta info only, will you provide instruction about how to transform the meta info into text?

[koalazf99]

Thank you for your interest!

the dataset you provided in huggingface stores meta info only

We only did this for our recalled code data. And the preview on huggingface dataset only shows this part.
This is intended since the official stack-v2 data only contain the SWHIDs to download the code files and not the content of the files itself due to license issue. (Please check https://huggingface.co/datasets/bigcode/the-stack-v2).

For downloading code data, you could refer to s3 download tutorials. Maybe this snippet is a very simple one for you.

All other data should be directly available. Please feel free to use.

[pengzhangzhi]

thanks! Still v confused. How to know what example is recalled data and what isn't? And for un-recalled data, how to load them and what's the column name for them?

[pengzhangzhi]

Any chance to share a script of how you load the data for ur training in the paper?

[koalazf99]

Sorry for the confusion. firstly, megamath contains multiple subsets: including web, code, and synthetic data.

1. We give sample code in readme to let you download by different subsets. The missing text only happens in code subset because the issue I mentioned above.

2. All existing data in megamath is ready. I am using the word recalled since the code data is curated via recalling from stack-v2 dataset.

3. To load code data, you need to download the code subset in current hf dataset first, then use the blob id of each file to download the code data from aws s3. We use same downloading snippets provided in the stack-v2.

Hope these information help.