How to strip a field while preserving quotes?

[vdmitrienko]

Hi @osiegmar

I have more questions in the scope of junit-team/junit5#4339 🙂

Currently, when a field is stripped, whitespaces wrapped in quotes are preserved. Consider the following example, using single quotes as the quote character:

'foo ' -> foo

I wonder if it's possible to implement this using the available customization options. Using the FieldModifiers.STRIP expectedly results in a foo without trailing whitespaces.

[osiegmar]

Hi @vdmitrienko,

That's a wonderful example of "malformed" data where it's hard to find consensus about the correct handling.

As per RFC 4180, there is no such thing as whitespaces around quotes.

record = field *(COMMA field)
field = (escaped / non-escaped)
escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE
non-escaped = *TEXTDATA

Currently, the parser removes (escaped) quotes within the field data before it is passed to the CsvRecordHandler. As the CsvRecordHandler is the one that calls field modifiers, it's too late to implement special treatment there.

Some (not-yet-implemented) mechanism would be required to access raw data before the field modifiers are applied.

Another thing that comes to mind that would have a greater impact: whitespaces BEFORE a quote, as this would cause the parser to (spec-conforming) read the data as an unquoted field including the quotes as regular characters. What is your expectation here?

Best, Oliver

[vdmitrienko]

whitespaces BEFORE a quote

That's a very good question, consider we have such a string: 'foo ' . The process is as follows:

1. Trim leading and trailing spaces: 'foo '.
2. After spaces trimmed, this string is considered "quoted".
3. Remove the quotes but preserve the whitespaces inside the quotes: foo .

After reading your response and reviewing the FastCSV source code, I feel like a separate, non-spec-conforming, parser is needed to support such custom behavior, as the current one operates in a completely different way.

[osiegmar]

Transferred to #140