icechunk (earth-mover/icechunk#1776) introduced dictionary-compressed virtual chunk locations. The manifest FlatBuffer schema has location_dictionary, compression_algorithm, and compressed_location fields that we don't read yet.
Current state
What's needed
- Wait for fzstd#18 to land and release
- Update
manifest-parser.ts to read compressedLocation from ChunkRef
- Update
manifest-parser.ts to read locationDictionary/compressionAlgorithm from Manifest
- Decompress locations at parse time (matching Rust impl), priority: compressed_location → location → chunk_id → inline
- Add tests (will need synthetic fixtures or wait for upstream data)
References
- Upstream schema:
manifest.fbs — ChunkRef.compressed_location, Manifest.location_dictionary, Manifest.compression_algorithm
icechunk (earth-mover/icechunk#1776) introduced dictionary-compressed virtual chunk locations. The manifest FlatBuffer schema has
location_dictionary,compression_algorithm, andcompressed_locationfields that we don't read yet.Current state
fzstddoesn't support dictionary decompression but there's an open PR adding it: Support loading a dictionary for decompression 101arrowz/fzstd#18zstdifywhich has dictionary support but ran into issues I think related to fix(zstdify): correct sequence RLE decoding semantics bhouston/zstdify#3What's needed
manifest-parser.tsto readcompressedLocationfrom ChunkRefmanifest-parser.tsto readlocationDictionary/compressionAlgorithmfrom ManifestReferences
manifest.fbs—ChunkRef.compressed_location,Manifest.location_dictionary,Manifest.compression_algorithm