kzstd

A pure-Kotlin, multiplatform Zstandard (zstd) codec with dictionary support. It produces and reads standard zstd frames that interoperate with libzstd in both directions, and it has zero runtime dependencies — just the Kotlin standard library, on every target.

kzstd was extracted from TAKPacket-SDK, where it replaced three native binding stacks — zstd-jni on the JVM, a per-target libzstd cinterop on Kotlin/Native, and @bokuweb/zstd-wasm on JS/Wasm — with one implementation that compiles everywhere Kotlin does.

Targets

JVM · JS (browser + Node) · Wasm/JS · Wasm/WASI · and nine Kotlin/Native targets: iOS (arm64, simulator-arm64, x64), macOS (arm64), tvOS (arm64, simulator-arm64), Linux (x64, arm64), and Windows (mingw-x64).

Install

// Maven Central
implementation("org.meshtastic:kzstd:0.1.2")

Usage

import org.meshtastic.kzstd.Zstd
import org.meshtastic.kzstd.ZstdDictionary
import org.meshtastic.kzstd.ZstdException

// Without a dictionary
val frame = Zstd.compress(data)
val original = Zstd.decompress(frame, maxSize = 64 * 1024)

// With a dictionary — digest it once, reuse it everywhere
val dict = ZstdDictionary(dictionaryBytes)   // parses tables + indexes content once
val small = Zstd.compress(data, dict)
val back = Zstd.decompress(small, dict, maxSize = 64 * 1024)

• ZstdDictionary(bytes) digests a dictionary once in its constructor (parsing its entropy tables and indexing its content) and is immutable afterward, so a single instance is safe to share across threads and cheap to reuse. bytes may be a trained dictionary (zstd --train / ZDICT) or any raw byte prefix.
• maxSize on decompress is a required decompression-bomb guard: decoding stops and throws if the output would exceed it.
• Failures surface as a single ZstdException.

Deviations and current limits

• No streaming. The API is one-shot only — no InputStream/OutputStream interface; each call handles a whole frame from one byte array, with no cross-call state, so every frame is independently decodable (what packet and mesh transports need).
• level (1–22) governs match-finding search depth only. The encoder uses a single fixed greedy/lazy strategy at every level — it does not implement zstd's other per-level parameters (window log, target length, etc.) — but a higher level does search more candidate matches per position, which can shrink output at the cost of more work. Frames remain fully libzstd-compatible at every level.
• Blocks are compressed independently (no cross-block matching). Zstd.compress cuts input into zstd's 128 KiB Block_Maximum_Size chunks and emits one multi-block frame. Each chunk is matched only against itself and the dictionary, never against an earlier block's output, so a large input compresses less well than a windowed encoder manages — 3 MB of synthetic JSON telemetry lands between libzstd's levels 3 and 19. Entropy tables and the repeat offsets ARE carried across blocks. A windowed matcher is a planned improvement.
• Total input (dictionary content + data) is capped at 128 MiB. Beyond that, the window a frame must declare exceeds libzstd's default decompression limit (ZSTD_WINDOWLOG_LIMIT_DEFAULT), so Zstd.compress throws ZstdException rather than emit a frame most real-world libzstd consumers would refuse to decode.
• Huffman-coded literals are single-stream and directly described. The encoder builds a Huffman table from a block's own literals, but writes only the single-stream layout (so it applies to at most 1023 bytes of literals per block) and only the direct 4-bit weight tree description (so a block containing a literal byte above 128 falls back to raw literals). Both limits are encoder-side only — Zstd.decompress reads the 4-stream layout and FSE-compressed weight descriptions that libzstd emits. The 1023-byte cap is why a full 128 KiB block keeps raw literals: on large inputs the ratio comes from the sequence tables alone.

Interoperability

kzstd reads frames produced by libzstd (including dictionary-compressed frames that use the dictionary's Huffman/FSE entropy tables), and libzstd reads frames produced by kzstd — including frames kzstd itself entropy-codes: dictionary frames using the dictionary's trained tables and repeat-offset codes, and dictionary-free frames using Huffman/FSE tables built from the block's own data (or the RLE forms, when a block, its literals or a symbol stream is constant). Each of those forms is picked only when it is the smallest valid encoding. The test suite cross-checks both directions against zstd-jni (a JVM-test-only oracle, never a runtime dependency).

Building & testing

./gradlew build         # compile every target, run tests, check the API baseline
./gradlew jvmTest       # JVM tests only (includes the libzstd interop oracle)
./gradlew apiDump       # refresh the binary-compatibility API baseline

The test dictionary (src/commonTest's TestVectors) is a genuinely trained zstd dictionary regenerated reproducibly by scripts/train_test_dict.py (requires the zstd CLI) — its content is generic structured JSON, not domain data.

Contributing

Contributions are welcome. See CLAUDE.md for the architecture, build and test commands, and the design invariants, and CHANGELOG.md for release notes. Run ./gradlew build (JDK 21) before opening a PR, and refresh the binary-compatibility baseline with ./gradlew apiDump after any public-API change.

License

GPL-3.0. See LICENSE.