feat(node:zlib): Transform-stream objects + Brotli fns (#1843) - #1866
Merged
Conversation
Implement the zlib stream interface in perry-ext-zlib (the #466 well-known-bound crate for node:zlib): createGzip/createGunzip/ createDeflate/createInflate/createDeflateRaw/createInflateRaw/ createUnzip/createBrotliCompress/createBrotliDecompress return real stream objects with .write/.end/.on('data'|'end'|'error'|'finish'| 'close')/.once/.pipe/.flush/.close. Compression is synchronous: input is buffered across .write() and the codec runs on .end(), with 'data'/'end' deferred onto a queue drained by the main-thread pump so listeners registered after .write() still fire and .pipe() forwards chunks (returns dest for chaining). Wired into perry-stdlib's event loop via a new external-zlib-pump feature (mirrors external-net-pump): dispatch arm + pump + has_active externs, activated by optimized_libs when the flip strips compression. A bundled equivalent lives in perry-stdlib's compression feature for the no-flip path. Also adds Brotli one-shots (brotliCompressSync/ brotliDecompressSync + async) — the brotli crate was already a dep. Fix two latent handle-deref segfaults: is_class_object_ptr and js_value_typeof low-address guards (< 0x1008 / > 0x10000) only covered the tiny net/fastify id space; a mid-range POINTER_TAG registry handle (zlib uses a 0x60000 stream base) sailed past them and segfaulted reading [handle-8]/[handle+12]. Both now reject < 0x100000, the same handle/real-pointer boundary js_native_call_method uses, so typeof aStreamHandle === "object" and every handle subsystem's latent id ceiling rises from 0x1008 to 0x100000. Make the sync one-shots buffer-aware so gunzipSync(Buffer.concat(...)) / gunzipSync(fs.readFileSync(...)) read via BufferHeader instead of misreading a real Buffer as a StringHeader. Stream/brotli round-trips, pipe chains, and Z_* constants are byte-for-byte identical to node --experimental-strip-types.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the
node:zlibTransform-stream interface (cluster 1 of #1843, the highest-leverage gap) plus Brotli functions (cluster 2) and fixes two latent handle-dereference segfaults surfaced along the way.node:zlibis routed to theperry-ext-zlibcrate viawell_known_bindings.toml(#466), so the work lands there, wired into perry-stdlib's event loop through a newexternal-zlib-pumpfeature (mirroringexternal-net-pump). A bundled equivalent lives in perry-stdlib'scompressionfeature for the no-flip path.What now works
zlib.createGzip()/createGunzip()/createDeflate()/createInflate()/createDeflateRaw()/createInflateRaw()/createUnzip()/createBrotliCompress()/createBrotliDecompress()return real stream objects supporting:.write(chunk)/.end([chunk])(Buffer or string chunks).on('data' | 'end' | 'error' | 'finish' | 'close', cb)/.once/.addListener.pipe(dest)(forwardsdata→dest.write,end→dest.end, returnsdestfor chaining).flush()/.close()/.destroy()Plus one-shot
zlib.brotliCompressSync/brotliDecompressSync/brotliCompress/brotliDecompress(thebrotlicrate was already acompressiondep).Compression is synchronous, so input is buffered across
.write()and the codec runs once on.end(); the resultingdata/endevents are deferred onto a queue drained by the main-thread pump, so listeners registered after.write()still fire and.pipe()can forward chunks.End-to-end output is byte-for-byte identical to
node --experimental-strip-typesfor gzip/deflate/brotli stream round-trips,.pipe()chains, andzlib.constants.Z_*.Runtime segfault fixes (general)
Two
is_class_object_ptr/js_value_typeoflow-address guards only rejected the tiny net/fastify id space (< 0x1008/> 0x10000). A native-module registry handle in the mid range (zlib uses a0x60000stream base to stay clear of those id spaces) sailed past them and segfaulted dereferencing[handle - 8]/[handle + 12]. Both now reject anything< 0x100000— the same handle/real-pointer boundaryjs_native_call_methodalready uses (real heap objects always live well above it). This also raises every handle subsystem's latent id ceiling from0x1008to0x100000, and makestypeof aStreamHandle === "object".Buffer-aware one-shot input
gzipSync/gunzipSync/deflateSync/inflateSync/brotli*Syncread their argument through a buffer-aware reader, so a realBuffer/Uint8Array(e.g.gunzipSync(Buffer.concat(chunks)),gunzipSync(fs.readFileSync(...))) is read via itsBufferHeaderrather than misread as aStringHeader.Out of scope / follow-ups
.flush([cb])is a no-op (the buffer-until-end model has nothing to flush mid-stream); incremental-flush tests liketest-zlib-flush-drain-longblock.jsare not addressed here.dataemission (output currently emits on.end()).Closes #1843 (cluster 1 + cluster 2 + cluster 3 Z_* constants).