Whitespace normalization of nested column coerced as string column in JSONL inputs

[shrshi]

Description

Addresses #15280

Whitespace normalization is expected to remove unquoted whitespace characters in JSON lines inputs. However, in the cases where the JSON line is invalid due to an unquoted whitespace occurring in between numbers or literals, the existing normalization implementation is incorrect since it removes these invalidating whitespaces and makes the line valid.

This PR implements the normalization as a post-processing step on only nested columns forced as string columns.
Idea:

1. Create a single buffer by concatenating the rows of the string column. Create segment offsets and lengths array for concatenated buffer
2. Run a complementary whitespace normalization FST i.e. NOP for non-whitespace and quoted whitespace characters, and output indices of unquoted whitespace characters
3. Update segment lengths based on the number of output indices between segment offsets
4. Remove characters at output indices from concatenated buffer.
5. Return updated buffer, segment lengths and updated segment offsets

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vuule]

No major concerns

[revans2]

I am seeing some test failures with this patch where some of the data being read is not being normalized. I think that the issue is coming from the idea that the data needs to be "mixed" when in reality we want to normalize the white space for any row in a column where the output is a string but the input included LIST or STRUCT parts. For example an input of

{"data": {"A": 0, "B": 1}}

with a requested schema of data STRING results in

{"A": 0, "B": 1}

As if the normalization didn't happen. But if I add in more lines to make it a mixed type, then it passes.

{"data": {"A": 0, "B": 1}}
{"data": "test"}

[revans2]

That fixed all of the issues I was having and all of the tests for white space normalization now pass.

[vuule]

having UI troble, flushing the comments to try to fix it

[shrshi]

/ok to test

[shrshi]

/ok to test

[shrshi]

/ok to test

[vuule]

Great stuff!

[karthikeyann]

Looks good to me.
only one suggestion on memory usage for stencil.

[karthikeyann]

Nice work!
LGTM 👍

[shrshi]

/merge

[karthikeyann]

/ok to test

[karthikeyann]

Posting offline discussion here: (to avoid upper_bound and atomics).
Use a permutation iterator instead outbuf_indices.data(), to directly scatter to stencil.
Use the original FST (output non-whitespace), and use a permutation iterator to create stencil.
then use HistogramRange to create the offsets?

auto out = thrust::make_transform_output_iterator(
    thrust::discard_iterator{},
    [stencil=stencil.begin()](auto const& i) -> bool { stencil[i] = true; return true; });

[karthikeyann]

/ok to test