Optimize Unnest and implement skip_nulls=true if specified

[smiklos]

Which issue does this PR close?

related to #7087.
Closes #6961

Rationale for this change

This implements the new preserve_nulls option in unnest exec.
I've also adjusted the code to avoid copying and rely only on the take kernel as needed.

Todo

• Set preserve_nulls to false -> If we really want align with clickhouse etc.

What changes are included in this PR?

The code will now respect the flag preserve_nulls in UnnestOptions.
This way we can also make the sql api consistent with Clickhouse/duckdb while allowing dataframe users to choose between implementations.

Are these changes tested?

Yes, added some new tests as well.

Are there any user-facing changes?

No

No

[alamb]

This is really nicely done -- thank you @smiklos 🏆

cc @vincev and @izveigor and @jayzhan211 who I think where also interested in this API

[alamb]

❤️

[alamb]

This might be easier to understand

Suggested change

	let take_index =
	<Int32Type as ArrowPrimitiveType>::Native::from_usize(
	take_offset + i,
	)
	.unwrap();
	builder.append_value(take_index);
	let take_index = take_offset + i;
	builder.append_value(take_index as i32);

[alamb]

🎉

[smiklos]

@alamb Thanks for the review!

Do we want to set preserve_nulls = false anywhere in the code? I recall the idea was to keep the dataframe api use preserve_nulls = true but when using sql unnest would drop the nulls

[vincev]

Do we want to set preserve_nulls = false anywhere in the code? I recall the idea was to keep the dataframe api use preserve_nulls = true but when using sql unnest would drop the nulls

Setting preserve_null = false for the dataframe api would be a breaking change for users as data pipelines that use dataframe unnest would now produce different results.

[alamb]

Do we want to set preserve_nulls = false anywhere in the code? I recall the idea was to keep the dataframe api use preserve_nulls = true but when using sql unnest would drop the nulls

I recommend that we merge this PR in without any changes to the default behavior and then we can propose a follow on PR with just the default behavior change (for SQL) by itself for discussion. That will make it easier to follow what exactly the proposed change in behavior is.