fix: surface special-format conversion errors instead of swallowing them (#3988)

[jqnatividad]

Summary

Closes part of #3988. Config::resolve_converted previously swallowed polars-native special-format conversion failures (Avro, Parquet, Arrow IPC, JSON, JSONL, compressed CSV/TSV/SSV) and fell back to reading the original — often binary — bytes as delimited text. The command then emitted garbage with a success exit code. Only .zip surfaced the error.

For a data-wrangling tool, silently producing corrupt output that looks successful is the worst failure mode — it can flow downstream into joins, DB loads, etc. unnoticed. This PR makes a detected special format that fails to convert a hard error for all formats, consistent with the existing .zip behavior. The single escape hatch remains QSV_SKIP_FORMAT_CHECK, which opts into reading the raw bytes as-is.

Behavior change

⚠️ User-visible: an unconvertible special-format input now errors (non-zero exit) instead of emitting garbage. It only triggers where conversion was already failing (the user was already getting garbage), so this turns a silent corruption into a clear error. Documented in CHANGELOG.md under [Unreleased].

Masked failures this surfaced (and fixed)

These tests only ever passed because the asserted substring is physically embedded in the raw file bytes — so the substring check passed whether or not conversion actually worked:

• slice_from_avro — the NYC311-5.avro fixture is unreadable by current polars (avro-error: OutOfSpec). Regenerated it from the canonical NYC311-5.csv via the same polars AvroWriter (qsv sqlp ... --format avro), verified it slices to the exact expected row, and strengthened the assertion to a full-row match.
• slice_from_jsonl — strengthened to a full-object comparison, parsed as JSON (order-independent, since polars does not preserve JSONL column order).
• slice_from_jsonl_with_decimal_precision{,_vs_float} — these used an invalid pschema shape (missing the metadata field) and targeted a capability polars doesn't have: its JSONL reader cannot apply a Decimal type (it refuses to coerce JSON numbers or strings into a Decimal column). Reworked both onto the compressed-CSV reader — the path that genuinely applies a Decimal pschema and preserves 25-digit precision vs f64's ~17 — and renamed to slice_from_csvgz_with_decimal_precision{,_vs_float}.

No #[ignore]s are introduced — all affected tests pass for the right reason.

Verification

• test_slice: 140 passed, 0 ignored.
• Special-format-input modules (count/json/profile/snappy/sniff/sqlp/stats/to/jsonl/tojsonl): 1005 passed, 0 failed.
• cargo clippy -F all_features clean; cargo +nightly fmt applied.

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity

Metric	Results
Complexity	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[Copilot]

Pull request overview

This PR makes special-format input conversion failures (e.g., Avro/Parquet/Arrow IPC/JSON/JSONL and compressed delimited inputs) fail fast instead of silently falling back to reading raw bytes as delimited text, preventing “successful” commands from emitting corrupt/garbage output. It also strengthens slice tests that previously passed for the wrong reason, and documents the behavior change.

Changes:

• Make Config::resolve_converted return a hard error on special-format conversion failure unless QSV_SKIP_FORMAT_CHECK is set.
• Strengthen slice integration tests to validate full converted rows / decimal precision behavior rather than substring checks.
• Add an [Unreleased] changelog entry describing the user-visible behavior change and the test fixture adjustments.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.

File	Description
src/config.rs	Surfaces special-format conversion failures as InvalidInput errors unless explicitly opted out via QSV_SKIP_FORMAT_CHECK.
tests/test_slice.rs	Updates several slice tests to assert real conversion results (full-row / decimal precision) instead of substring checks that could pass on raw bytes.
CHANGELOG.md	Documents the behavior change under [Unreleased].

Comments suppressed due to low confidence (2)

tests/test_slice.rs:1325

• This block runs slice twice (assert_success then stdout), but Workdir::stdout does not assert exit success. Capture Output once and reuse its stdout to avoid masking failures.

    wrk.assert_success(&mut cmd);

    // Get the output with Decimal schema
    let decimal_output: String = wrk.stdout(&mut cmd);

tests/test_slice.rs:1344

• Same issue as above: assert_success runs the command once and stdout runs it again without checking the status. Use a single Output to ensure this assertion can’t pass if the command exits non-zero.

    wrk.assert_success(&mut cmd);

    // Get the output with Float64 schema
    let float_output: String = wrk.stdout(&mut cmd);