Closed
Description
Describe the bug
Seen when running https://github.com/huggingface/lighteval/blob/v0.9.2/docs/source/using-the-python-api.mdx:
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8348.54 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 734635.07 examples/s]
Generating test split: 100%|██████████| 102/102 [00:00<00:00, 53517.51 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 11630.29 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 7508.60 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 695671.32 examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 116282.34 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 13007.43 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 7937.74 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 823297.30 examples/s]
Generating test split: 100%|██████████| 235/235 [00:00<00:00, 252410.10 examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 34174.84 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8760.03 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 793201.77 examples/s]
Generating test split: 100%|██████████| 114/114 [00:00<00:00, 125301.53 examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 16442.88 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8727.22 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 546932.61 examples/s]
Generating test split: 100%|██████████| 145/145 [00:00<00:00, 103925.85 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 15917.66 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 5739.33 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 560104.46 examples/s]
Generating test split: 100%|██████████| 378/378 [00:00<00:00, 382127.48 examples/s]
Generating validation split: 100%|██████████| 41/41 [00:00<00:00, 53538.75 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8198.41 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 658148.99 examples/s]
Generating test split: 100%|██████████| 126/126 [00:00<00:00, 95756.90 examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 25137.10 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8504.27 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 793643.73 examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 114786.64 examples/s]
Generating validation split: 100%|██████████| 10/10 [00:00<00:00, 16783.93 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 8680.26 examples/s]
Generating auxiliary_train split: 100%|██████████| 99842/99842 [00:00<00:00, 730517.51 examples/s]
Generating test split: 100%|██████████| 310/310 [00:00<00:00, 309358.61 examples/s]
Generating validation split: 100%|██████████| 32/32 [00:00<00:00, 45206.38 examples/s]
To Reproduce
Run the script without HF cache
Expected behavior
Not to generate splits forever.
Version info
lighteval[math]==0.9.2