feat: auto-decompress compressed remote sources — lookup tables & get/dc: (#1417) by jqnatividad · Pull Request #3987 · dathere/qsv

jqnatividad · 2026-06-13T18:20:58Z

Overview

Phase 2 of #1417: compressed remote/downloaded sources are now transparently auto-decompressed. Phase 1 (#3986, merged) handled local files via Config; this PR closes the remaining gap — remote sources were still cached as opaque bytes, so a fetched .csv.gz was stored as raw gzip (corrupted, un-indexable) and lookup::download_lookup_table used .text() which mangled binary payloads.

What changed

Shared decompress_source helper (always-compiled diskcache core, usable by both get ingest and the lookup downloader). Detects compression by the source's outer extension: .zip (first tabular entry via util::select_zip_entry) and .sz are always available; .gz/.zlib need flate2 and .zst needs zstd, each gated with an actionable build-feature error rather than silently caching compressed bytes.

get/diskcache ingest decompresses before caching, so the stored blob is always plain CSV — auto-indexable, with a correct record_count and valid dc: read-back:

ingest_local decompresses before store_blob.
ingest_http/ingest_cloud use a new IngestSink that streams uncompressed sources as before (bounded memory) but buffers-and-decompresses compressed ones on finish (the codec needs the whole payload anyway). Download loops unchanged.
CacheEntry gains optional inner_ext (serde default → back-compat); tabular_temp_name prefers it so a data.tsv.gz materializes its dc: temp as .tsv.

Remote lookup tables (lookup.rs): .text() → .bytes() + decompress_source; write_cache_file(&[u8]). The cache file is named with the source's inner tabular extension (lookup_cache_ext) so the delimiter is correct on both fresh download and cache hit. Read path unchanged.

Codec wiring (Cargo.toml): flate2+zstd → feature_capable (validate/template/describegpt in qsv & qsvmcp), flate2 → get (datapusher_plus), both → luau. Zero Cargo.lock churn — both codecs already compile transitively in these builds.

Commits

feat(diskcache): shared decompress_source + get ingest decompression
feat(lookup): auto-decompress remote lookup tables
build: wire flate2/zstd into lookup-consuming features
test: compressed get ingest, local and remote
docs: README

Reviewer note: src/diskcache.rs shows large line churn, but git diff -w is ~259 lines — the rest is rustfmt re-indenting the ingest_http/ingest_cloud async blocks after a tuple type changed. View with whitespace hidden (?w=1).

Out of scope

Nested special formats (e.g. a .parquet inside a .zip); HTTP/3; derived stats/frequency caches for dc:. A .zip whose inner tabular file is non-CSV-delimited can't be known from the URL for the lookup cache name (defaults to csv) — documented limitation.

Verification

Build + clippy clean for -F all_features, -F lite, and qsvmcp. Full suites pass: all_features 3068+493, lite 2137+92, 0 failures. New tests: 5 local + 1 remote get-decompression, 1 lookup_cache_ext unit test.

🤖 Generated with Claude Code

Add a shared `decompress_source(source, body) -> Decompressed { bytes, inner_ext }` helper in the always-compiled diskcache core (so both `get` ingest and the lookup-table downloader can use it). Detection is by the source's lowercased outer extension: `.zip` (first tabular entry via `util::select_zip_entry`) and `.sz` are always available; `.gz`/`.zlib` need `flate2` and `.zst` needs `zstd`, each gated with an actionable build-feature error rather than silently caching the compressed bytes. `is_compressed_source` is the matching predicate. Wire it into `get`/diskcache ingest so the stored blob is always plain CSV (auto-indexable, correct `record_count`, valid `dc:` read-back): - `ingest_local` decompresses the file before `store_blob`. - `ingest_http`/`ingest_cloud` use a new `IngestSink` that streams uncompressed sources as before (bounded memory) but buffers-and-decompresses compressed ones on `finish` (the codec needs the whole payload, so streaming buys nothing). The download loops are unchanged. - `CacheEntry` gains an optional `inner_ext` (serde default → back-compat with existing on-disk indexes); `tabular_temp_name` prefers it so a `data.tsv.gz` materializes its `dc:` temp as `.tsv` (correct delimiter), not `.csv`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Remote `luau`/`validate`/`describegpt` lookup tables can now be served compressed. `download_lookup_table` reads the response as raw `.bytes()` (not `.text()`, which corrupted a binary/compressed body) and runs the shared `diskcache::decompress_source` on it, keyed on the actually-fetched URL (the CKAN data URL for `ckan://`). `write_cache_file` now takes `&[u8]`. The cache file is named with the source's inner tabular extension (`lookup_cache_ext`, e.g. `foo.tsv.gz` → `{name}.tsv`) so `Config` picks the right delimiter from the cached file's extension when the caller didn't specify one — consistently on both fresh download and cache hit. Cache stays a plain, comment-headed delimited file, so the read path is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Enable remote `.gz`/`.zlib`/`.zst` decompression for every command that loads lookup tables or uses `get`/`dc:`: - `feature_capable` += flate2, zstd (covers validate/template/describegpt in the qsv & qsvmcp builds). - `get` += flate2 (it already had zstd; covers datapusher_plus). - `luau` += flate2, zstd. `.zip`/`.sz` need no codec deps. Zero Cargo.lock churn — flate2 & zstd already compile transitively (via fetch/get/polars) in these builds. qsvlite is unaffected (no feature_capable; diskcache/lookup aren't compiled there). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Local: `get` a boston311-100.csv.{zip,sz,gz,zlib,zst}, then assert `dc:` reads 100 decompressed rows and cache-list records record_count 100. gz/zlib/zst gated on their codec features; zip/sz ungated. - Remote: serve boston311-100.csv.gz over the actix mock (raw bytes, no Content-Encoding) and assert `dc:` read-back is decompressed — exercises the IngestSink buffer-and-decompress path, not just ingest_local. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Note that compressed remote/downloaded sources (lookup tables and get/dc: cached resources) are auto-decompressed before caching, and which codecs need a build feature. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

codacy-production · 2026-06-13T18:21:45Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 56 complexity

Metric Results

Complexity 56

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

Copilot

Pull request overview

This PR completes the “remote sources” half of #1417 by transparently auto-decompressing compressed downloaded/remote inputs before they are cached, so dc: and lookup-table consumers operate on plain tabular bytes (and can select the correct delimiter via the inner extension).

Changes:

Add a shared diskcache::decompress_source helper (plus is_compressed_source) and plumb it through get ingestion (local + HTTP + cloud).
Fix remote lookup-table downloads to fetch raw bytes (.bytes()), decompress based on the fetched URL, and cache with a stable “inner tabular” extension.
Wire feature flags (get, luau, feature_capable) so gzip/zlib/zstd codecs are available in builds that consume remote compressed sources; add integration/unit tests and docs.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`src/diskcache.rs`	Introduces shared decompression helpers, stores `inner_ext` in cache metadata, and updates `get` ingestion to decompress before caching.
`src/lookup.rs`	Switches lookup downloads from `.text()` to `.bytes()`, decompresses, and names cache files using an inferred inner tabular extension.
`tests/test_get.rs`	Adds tests covering local + remote compressed ingestion and `dc:` read-back/record counting.
`Cargo.toml`	Updates feature wiring so codec deps are included where remote decompression is expected to work.
`README.md`	Documents that remote/downloaded compressed sources are now decompressed before caching and notes feature requirements.

…errors (#1417) Copilot review feedback on PR #3987: - diskcache (HIGH): `IngestSink` now STREAMS decompression of gz/zlib/zst remote downloads into `BlobSink` via the crates' `write::*Decoder`s (bounded memory), instead of buffering the whole compressed + decompressed payload. `.zip` (needs random access) and `.sz` (snap has no push-decoder) keep the full-buffer path; codec-absent gz/zlib/zst also fall back to it so `decompress_source` still surfaces the actionable build-feature error. `BlobSink` gains an `io::Write` impl (inherent `write` renamed to `push`) so decoders can write into it. zstd's write Decoder buffers output, so its finalization flushes before `into_inner` (a new remote-.zst test caught a 70/100-row truncation otherwise). - diskcache: `decompress_source` error messages are now source-neutral (dropped the misleading `get:` prefix) and the missing-codec errors name the real features (standard qsv/qsvmcp builds include them). - lookup: wrap the decompress error in `io::Error` (cleaner than a bare String). - tests: assert command success before reading stdout (capture Output once); add a remote-.zst test exercising the streaming zstd decoder. Verified: build + clippy clean for all_features, lite, qsvmcp; full suites pass (all_features 3069+493, lite 2137+92, 0 failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ber support (#1417) roborev 2921 (Medium): the streaming gz path used flate2::write::GzDecoder (single member), regressing multi-member/concatenated gzip that the buffered read::MultiGzDecoder path accepted — members after the first were silently dropped. Switch the streaming decoder to flate2::write::MultiGzDecoder (same write/finish API), preserving both streaming (bounded memory) and multi-member semantics. Test: a 2-member gzip fixture (boston311-100 split header+50 / last 50, gzip'd separately and concatenated) served remotely; a single-member decoder would yield 50 rows, multi-member yields 100. Verified: build + clippy clean (-F all_features); full suite 3070+493, 0 failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jqnatividad and others added 5 commits June 13, 2026 14:20

jqnatividad requested a review from Copilot June 13, 2026 18:54

Copilot started reviewing on behalf of jqnatividad June 13, 2026 18:55 View session

Copilot AI reviewed Jun 13, 2026

View reviewed changes

Comment thread src/lookup.rs Outdated

Comment thread src/diskcache.rs Outdated

Comment thread src/diskcache.rs

Comment thread src/diskcache.rs

Comment thread tests/test_get.rs Outdated

Comment thread tests/test_get.rs

Comment thread tests/test_get.rs Outdated

jqnatividad and others added 2 commits June 13, 2026 15:23

jqnatividad merged commit c1eb777 into master Jun 13, 2026
19 checks passed

jqnatividad deleted the feature/1417-remote-decompression branch June 13, 2026 20:29

This was referenced Jun 13, 2026

Phase 3 follow-ups for auto-decompression (#1417): sftp, dc: derived caches, local-file streaming, dual-path unification #3988

Open

luau: support automatic lookup table decompression - support zip, gz and snappy formats #1417

Closed

BrewTestBot mentioned this pull request Jun 15, 2026

qsv 21.1.0 Homebrew/homebrew-core#288003

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: auto-decompress compressed remote sources — lookup tables & get/dc: (#1417)#3987

feat: auto-decompress compressed remote sources — lookup tables & get/dc: (#1417)#3987
jqnatividad merged 7 commits into
masterfrom
feature/1417-remote-decompression

jqnatividad commented Jun 13, 2026

Uh oh!

codacy-production Bot commented Jun 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jqnatividad commented Jun 13, 2026

Overview

What changed

Commits

Out of scope

Verification

Uh oh!

codacy-production Bot commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codacy-production Bot commented Jun 13, 2026 •

edited

Loading