feat: add Uint8Array support for zero-copy worker_threads transfer#3183
Open
umeshmore45 wants to merge 1 commit intoredis:masterfrom
Open
feat: add Uint8Array support for zero-copy worker_threads transfer#3183umeshmore45 wants to merge 1 commit intoredis:masterfrom
umeshmore45 wants to merge 1 commit intoredis:masterfrom
Conversation
- Add Uint8Array to type system (BlobStringReply, SimpleStringReply, VerbatimStringReply) - Implement Uint8Array encoding/decoding in RESP protocol handlers - Support Uint8Array as command arguments across all commands - Add comprehensive test coverage for Uint8Array operations - Fix type guards in HSET, GEOSEARCH, AGGREGATE, SEARCH commands - Fix Buffer/Uint8Array handling in time-series helpers and cluster-slots - Maintain full backward compatibility with Buffer Enables zero-copy transfer to worker_threads via ArrayBuffer transfer. Closes redis#3106
|
Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset. In case there are security findings, they will be communicated to you as a comment inside the PR. Hope you’ll enjoy using Jit. Questions? Comments? Want to learn more? Get in touch with us. |
Collaborator
|
Thanks @umeshmore45, i will take a look |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds comprehensive
Uint8Arraysupport to enable zero-copy data transfer to worker_threads, significantly improving performance for multi-threaded applications handling large payloads.Motivation
Currently, Redis values are returned as
Buffer, which requires copying when transferring to worker threads. WithUint8Arraysupport, the underlyingArrayBuffercan be transferred with zero-copy semantics.Performance Benefits:
For a 10MB payload, this eliminates 10MB of memory copying when transferring between threads.
Usage Example
Changes Made
Type System (
types.ts):BlobStringReply,SimpleStringReply,VerbatimStringReplyto supportUint8ArrayUnwrapConstructorandReplyWithTypeMappingto handleUint8ArrayUint8ArraytoRedisArgumenttypeDecoder (
decoder.ts):Encoder (
encoder.ts):Command Support:
HSET,GEOSEARCH,SEARCH,AGGREGATEto accept Uint8Array argumentscluster-slots.tsandtime-series/helpers.tsTesting:
Breaking Changes
None! This is fully backward compatible:
Bufferremains the default typeUint8Arrayis opt-in viawithTypeMapping()Files Changed
Closes #3106
Checklist
npm testpass with this change (including linting)?Note: Documentation can be added in a follow-up PR if needed. The code includes inline comments and the PR description provides usage examples.