Allow polars as valid output type

[psmyth94]

I was trying out polars as an output for a map function and found that it wasn't a valid return type in validate_function_output. Thought that we should accommodate this by creating and adding it to the allowed_processed_input_types variable.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

Cool ! Sorry for the late review. Can you add a test to make sure map() with a polars-output function works ?
You can place the test in test_arrow_dataset.py

[psmyth94]

Hello @lhoestq, I added the test and modified update_data to include polars as an updatable type. Although, it seems pretty redundant to do the type checks both before validate_function_output and then immediately afterward within the call stack. Could consider adding allowable_types in validation_function_output.

[lhoestq]

Cool ! thanks for the additions :)

It looks all good to me as is. Not a big deal to have it at two places, since it's not for the same objective (check whether the type is a data update type, and whether the type is a valid type)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005530 / 0.011353 (-0.005823)	0.004012 / 0.011008 (-0.006996)	0.062474 / 0.038508 (0.023966)	0.031896 / 0.023109 (0.008787)	0.239620 / 0.275898 (-0.036278)	0.264694 / 0.323480 (-0.058785)	0.003199 / 0.007986 (-0.004786)	0.003141 / 0.004328 (-0.001187)	0.048726 / 0.004250 (0.044475)	0.044795 / 0.037052 (0.007743)	0.250661 / 0.258489 (-0.007828)	0.279658 / 0.293841 (-0.014183)	0.029857 / 0.128546 (-0.098689)	0.012293 / 0.075646 (-0.063353)	0.203626 / 0.419271 (-0.215646)	0.036284 / 0.043533 (-0.007249)	0.241678 / 0.255139 (-0.013461)	0.259380 / 0.283200 (-0.023820)	0.020400 / 0.141683 (-0.121283)	1.142334 / 1.452155 (-0.309821)	1.199068 / 1.492716 (-0.293648)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.097348 / 0.018006 (0.079341)	0.303468 / 0.000490 (0.302978)	0.000219 / 0.000200 (0.000019)	0.000044 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018646 / 0.037411 (-0.018766)	0.062374 / 0.014526 (0.047848)	0.074585 / 0.176557 (-0.101972)	0.120380 / 0.737135 (-0.616755)	0.075685 / 0.296338 (-0.220653)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.277488 / 0.215209 (0.062279)	2.741734 / 2.077655 (0.664080)	1.451901 / 1.504120 (-0.052219)	1.341712 / 1.541195 (-0.199482)	1.395209 / 1.468490 (-0.073282)	0.736334 / 4.584777 (-3.848443)	2.358225 / 3.745712 (-1.387487)	2.951838 / 5.269862 (-2.318023)	1.892027 / 4.565676 (-2.673649)	0.077913 / 0.424275 (-0.346362)	0.005188 / 0.007607 (-0.002419)	0.328790 / 0.226044 (0.102745)	3.259387 / 2.268929 (0.990459)	1.826102 / 55.444624 (-53.618522)	1.526635 / 6.876477 (-5.349842)	1.576392 / 2.142072 (-0.565680)	0.786244 / 4.805227 (-4.018983)	0.133909 / 6.500664 (-6.366756)	0.044544 / 0.075469 (-0.030925)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.965314 / 1.841788 (-0.876474)	11.786831 / 8.074308 (3.712523)	9.568519 / 10.191392 (-0.622873)	0.140628 / 0.680424 (-0.539796)	0.014442 / 0.534201 (-0.519759)	0.300876 / 0.579283 (-0.278407)	0.262647 / 0.434364 (-0.171717)	0.339141 / 0.540337 (-0.201196)	0.430254 / 1.386936 (-0.956683)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006020 / 0.011353 (-0.005333)	0.004191 / 0.011008 (-0.006818)	0.050006 / 0.038508 (0.011498)	0.033247 / 0.023109 (0.010138)	0.270677 / 0.275898 (-0.005221)	0.299539 / 0.323480 (-0.023941)	0.004391 / 0.007986 (-0.003595)	0.002825 / 0.004328 (-0.001504)	0.048573 / 0.004250 (0.044322)	0.042461 / 0.037052 (0.005409)	0.283812 / 0.258489 (0.025323)	0.324302 / 0.293841 (0.030461)	0.033264 / 0.128546 (-0.095282)	0.012405 / 0.075646 (-0.063241)	0.060298 / 0.419271 (-0.358973)	0.034833 / 0.043533 (-0.008700)	0.271133 / 0.255139 (0.015994)	0.290712 / 0.283200 (0.007512)	0.019762 / 0.141683 (-0.121920)	1.138644 / 1.452155 (-0.313511)	1.204628 / 1.492716 (-0.288088)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.096171 / 0.018006 (0.078164)	0.308916 / 0.000490 (0.308427)	0.000213 / 0.000200 (0.000013)	0.000046 / 0.000054 (-0.000009)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023077 / 0.037411 (-0.014334)	0.078865 / 0.014526 (0.064339)	0.091031 / 0.176557 (-0.085526)	0.133536 / 0.737135 (-0.603599)	0.093308 / 0.296338 (-0.203030)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.301466 / 0.215209 (0.086257)	2.995190 / 2.077655 (0.917535)	1.616545 / 1.504120 (0.112425)	1.472572 / 1.541195 (-0.068622)	1.477191 / 1.468490 (0.008701)	0.730240 / 4.584777 (-3.854537)	0.966591 / 3.745712 (-2.779121)	2.979970 / 5.269862 (-2.289892)	1.908275 / 4.565676 (-2.657401)	0.081346 / 0.424275 (-0.342929)	0.005150 / 0.007607 (-0.002458)	0.349066 / 0.226044 (0.123022)	3.504363 / 2.268929 (1.235435)	1.973355 / 55.444624 (-53.471270)	1.659337 / 6.876477 (-5.217139)	1.701282 / 2.142072 (-0.440790)	0.813493 / 4.805227 (-3.991735)	0.133537 / 6.500664 (-6.367127)	0.041207 / 0.075469 (-0.034262)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.020368 / 1.841788 (-0.821420)	12.444848 / 8.074308 (4.370540)	10.113832 / 10.191392 (-0.077560)	0.137782 / 0.680424 (-0.542642)	0.015217 / 0.534201 (-0.518984)	0.300419 / 0.579283 (-0.278864)	0.128868 / 0.434364 (-0.305496)	0.342831 / 0.540337 (-0.197506)	0.443036 / 1.386936 (-0.943900)