`sort_fuzz` testing DX improvements

[blaginin]

Is your feature request related to a problem or challenge?

datafusion/core/tests/fuzz_cases is a very cool tool! It helped to spot #16228. However, once the issue was discovered, it tool several hours to understand a problem and create a reproducible example

Partially that's because debug info to reproduce the problem right now is very big and not very useful:

Describe the solution you'd like

Ideas I have to improve the DX:

• Store dataset in a file instead of printing to stdout. The current version is too big and a bit messy - and it also makes reproducing issues harder.
• Streamline seeds. There are some hardcoded seeds, but it's unclear how to call a function with arguments to get the same error-inducing dataset. It'd help to include a hint on how to generate a test batch for debugging later.
• Add docs on how to extend, use, and maintain the suite (including the modules above).

Describe alternatives you've considered

No response

Additional context

No response

[jonathanc-n]

I agree with this, I've encountered some of these, especially this:

Streamline seeds. There are some hardcoded seeds, but it's unclear how to call a function with arguments to get the same error-inducing dataset. It'd help to include a hint on how to generate a test batch for debugging

I'm thinking the plan would be to:

1. During failure we either set a environment variable to the seed and have it rerun, or print it out (or both)
2. Reproduce the dataset with the saved seed + save it to an arrow file
   Something like (example where seed = 1 failed):
   FUZZ_SEED=1 cargo test fuzz_case_example
3. Have a reader for the arrow file
4. Add documentation to show how to do this in fuzz_cases

Let me know if this is what you had in mind

[jonathanc-n]

@2010YOUY01 I think you'd be interested

[blaginin]

hey @jonathanc-n

set an environment variable to the seed

I think it's a good idea to print the seed used, but not sure we should implicitly set the env variable.

One challenge with reproducing via seeds as currently implemented is that a single seed produces a group of fuzzy tasks, so it's unclear how to reference one specific task. Maybe dumping to a parquet or arrow_ipc file is the only simple way to make reproduction simple...?