Fix ci

[lhoestq]

...after last pr errors

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005748 / 0.011353 (-0.005605)	0.004109 / 0.011008 (-0.006899)	0.067017 / 0.038508 (0.028509)	0.031950 / 0.023109 (0.008841)	0.239939 / 0.275898 (-0.035959)	0.266339 / 0.323480 (-0.057141)	0.003176 / 0.007986 (-0.004809)	0.003556 / 0.004328 (-0.000773)	0.050725 / 0.004250 (0.046475)	0.047711 / 0.037052 (0.010658)	0.251048 / 0.258489 (-0.007441)	0.287049 / 0.293841 (-0.006792)	0.029919 / 0.128546 (-0.098627)	0.012562 / 0.075646 (-0.063085)	0.212903 / 0.419271 (-0.206369)	0.036570 / 0.043533 (-0.006963)	0.240975 / 0.255139 (-0.014164)	0.266473 / 0.283200 (-0.016726)	0.019959 / 0.141683 (-0.121724)	1.152224 / 1.452155 (-0.299931)	1.186046 / 1.492716 (-0.306671)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.095836 / 0.018006 (0.077829)	0.303402 / 0.000490 (0.302913)	0.000210 / 0.000200 (0.000010)	0.000042 / 0.000054 (-0.000012)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.020552 / 0.037411 (-0.016859)	0.063619 / 0.014526 (0.049093)	0.076969 / 0.176557 (-0.099588)	0.123368 / 0.737135 (-0.613767)	0.077005 / 0.296338 (-0.219334)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.282005 / 0.215209 (0.066796)	2.794144 / 2.077655 (0.716489)	1.463569 / 1.504120 (-0.040551)	1.334295 / 1.541195 (-0.206899)	1.387198 / 1.468490 (-0.081292)	0.707654 / 4.584777 (-3.877123)	2.341698 / 3.745712 (-1.404014)	2.865131 / 5.269862 (-2.404731)	1.945168 / 4.565676 (-2.620509)	0.077926 / 0.424275 (-0.346349)	0.005470 / 0.007607 (-0.002137)	0.336498 / 0.226044 (0.110454)	3.330262 / 2.268929 (1.061334)	1.865574 / 55.444624 (-53.579050)	1.536932 / 6.876477 (-5.339545)	1.720960 / 2.142072 (-0.421113)	0.794753 / 4.805227 (-4.010475)	0.133491 / 6.500664 (-6.367173)	0.042437 / 0.075469 (-0.033032)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.976788 / 1.841788 (-0.865000)	11.895137 / 8.074308 (3.820829)	9.211969 / 10.191392 (-0.979423)	0.141798 / 0.680424 (-0.538626)	0.014354 / 0.534201 (-0.519847)	0.306044 / 0.579283 (-0.273239)	0.265016 / 0.434364 (-0.169348)	0.340877 / 0.540337 (-0.199460)	0.470449 / 1.386936 (-0.916487)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006134 / 0.011353 (-0.005219)	0.004023 / 0.011008 (-0.006985)	0.050419 / 0.038508 (0.011911)	0.033853 / 0.023109 (0.010744)	0.266799 / 0.275898 (-0.009099)	0.291248 / 0.323480 (-0.032232)	0.004474 / 0.007986 (-0.003511)	0.002847 / 0.004328 (-0.001481)	0.049895 / 0.004250 (0.045645)	0.041160 / 0.037052 (0.004108)	0.278818 / 0.258489 (0.020329)	0.314027 / 0.293841 (0.020186)	0.032303 / 0.128546 (-0.096243)	0.012367 / 0.075646 (-0.063279)	0.061495 / 0.419271 (-0.357776)	0.033512 / 0.043533 (-0.010021)	0.266168 / 0.255139 (0.011029)	0.283129 / 0.283200 (-0.000071)	0.018674 / 0.141683 (-0.123009)	1.124453 / 1.452155 (-0.327701)	1.164527 / 1.492716 (-0.328189)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.098522 / 0.018006 (0.080516)	0.315069 / 0.000490 (0.314579)	0.000202 / 0.000200 (0.000002)	0.000053 / 0.000054 (-0.000001)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022809 / 0.037411 (-0.014602)	0.078409 / 0.014526 (0.063883)	0.088558 / 0.176557 (-0.087998)	0.130004 / 0.737135 (-0.607131)	0.090507 / 0.296338 (-0.205832)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.291323 / 0.215209 (0.076114)	2.836363 / 2.077655 (0.758708)	1.548889 / 1.504120 (0.044769)	1.423857 / 1.541195 (-0.117337)	1.461667 / 1.468490 (-0.006823)	0.714956 / 4.584777 (-3.869821)	0.948170 / 3.745712 (-2.797542)	3.036151 / 5.269862 (-2.233711)	1.923824 / 4.565676 (-2.641853)	0.078002 / 0.424275 (-0.346273)	0.005198 / 0.007607 (-0.002409)	0.337007 / 0.226044 (0.110963)	3.310255 / 2.268929 (1.041327)	1.910371 / 55.444624 (-53.534253)	1.619855 / 6.876477 (-5.256622)	1.682093 / 2.142072 (-0.459979)	0.789903 / 4.805227 (-4.015324)	0.132117 / 6.500664 (-6.368547)	0.041312 / 0.075469 (-0.034157)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.997658 / 1.841788 (-0.844130)	12.447878 / 8.074308 (4.373570)	10.277662 / 10.191392 (0.086270)	0.143580 / 0.680424 (-0.536844)	0.016472 / 0.534201 (-0.517729)	0.307235 / 0.579283 (-0.272048)	0.125469 / 0.434364 (-0.308895)	0.339525 / 0.540337 (-0.200813)	0.427371 / 1.386936 (-0.959566)