-
-
Notifications
You must be signed in to change notification settings - Fork 34.7k
sqlite: avoid extra copy for large text binds #61580
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sqlite: avoid extra copy for large text binds #61580
Conversation
|
Review requested:
|
ea75fcc to
b7abb6f
Compare
|
Benchmarks (local, macOS, Release build) Compared with base branch (main) Summary:
change primarily improves large text binds; other workloads appear neutral within measurement noise. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #61580 +/- ##
==========================================
- Coverage 89.78% 89.77% -0.01%
==========================================
Files 673 673
Lines 203775 203852 +77
Branches 39166 39181 +15
==========================================
+ Hits 182956 183017 +61
- Misses 13139 13153 +14
- Partials 7680 7682 +2
🚀 New features to boost your workflow:
|
92bb01c to
8e4016b
Compare
When binding UTF-8 strings to prepared statements, transfer ownership of malloc-backed Utf8Value buffers to SQLite to avoid an extra copy for large strings. Use sqlite3_bind_blob64() when binding BLOB parameters.
8e4016b to
da58cfd
Compare
|
Hi @Renegade334 and @mcollina When you have a moment, could you please take a look at this PR and share your feedback? Thanks so much! |
mcollina
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
|
HI @mcollina some flaky tests failed, is it ready to merge? |
|
All CI passed @geeksilva97 should it be merged? |
yep. let's do this |
|
Landed in 346ad95 |
When binding UTF-8 strings to prepared statements, transfer ownership of malloc-backed Utf8Value buffers to SQLite to avoid an extra copy for large strings. Use sqlite3_bind_blob64() when binding BLOB parameters. PR-URL: #61580 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Edy Silva <edigleyssonsilva@gmail.com> Reviewed-By: René <contact.9a5d6388@renegade334.me.uk> Reviewed-By: Zeyu "Alex" Yang <himself65@outlook.com>
When binding UTF-8 strings to prepared statements, transfer ownership of malloc-backed Utf8Value buffers to SQLite to avoid an extra copy for large strings. Use sqlite3_bind_blob64() when binding BLOB parameters. PR-URL: #61580 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Edy Silva <edigleyssonsilva@gmail.com> Reviewed-By: René <contact.9a5d6388@renegade334.me.uk> Reviewed-By: Zeyu "Alex" Yang <himself65@outlook.com>
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [node](https://nodejs.org) ([source](https://github.com/nodejs/node)) | patch | `25.6.0` → `25.6.1` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>nodejs/node (node)</summary> ### [`v25.6.1`](https://github.com/nodejs/node/releases/tag/v25.6.1): 2026-02-10, Version 25.6.1 (Current), @​aduh95 [Compare Source](nodejs/node@v25.6.0...v25.6.1) ##### Notable Changes - \[[`47df4328d7`](nodejs/node@47df4328d7)] - **build,deps**: replace `cjs-module-lexer` with `merve` (Yagiz Nizipli) [#​61456](nodejs/node#61456) ##### Commits - \[[`47df4328d7`](nodejs/node@47df4328d7)] - **build,deps**: replace cjs-module-lexer with merve (Yagiz Nizipli) [#​61456](nodejs/node#61456) - \[[`a727054503`](nodejs/node@a727054503)] - **deps**: upgrade npm to 11.9.0 (npm team) [#​61685](nodejs/node#61685) - \[[`c78c49ed6b`](nodejs/node@c78c49ed6b)] - **deps**: update amaro to 1.1.7 (Node.js GitHub Bot) [#​61730](nodejs/node#61730) - \[[`4790816d9b`](nodejs/node@4790816d9b)] - **deps**: update minimatch to 10.1.2 (Node.js GitHub Bot) [#​61732](nodejs/node#61732) - \[[`8c71740e8a`](nodejs/node@8c71740e8a)] - **deps**: update undici to 7.21.0 (Node.js GitHub Bot) [#​61683](nodejs/node#61683) - \[[`e559ef6ab1`](nodejs/node@e559ef6ab1)] - **deps**: update googletest to [`56efe39`](nodejs/node@56efe39) (Node.js GitHub Bot) [#​61605](nodejs/node#61605) - \[[`300de2bb5a`](nodejs/node@300de2bb5a)] - **deps**: update amaro to 1.1.6 (Node.js GitHub Bot) [#​61603](nodejs/node#61603) - \[[`e71e9505ef`](nodejs/node@e71e9505ef)] - **dns**: fix Windows SRV ECONNREFUSED by adjusting c-ares fallback detection (notvivek12) [#​61453](nodejs/node#61453) - \[[`439b816bc7`](nodejs/node@439b816bc7)] - **doc**: clarify EventEmitter error handling in threat model (Matteo Collina) [#​61701](nodejs/node#61701) - \[[`c1c6641f23`](nodejs/node@c1c6641f23)] - **doc**: mention default option for test runner env (Steven) [#​61659](nodejs/node#61659) - \[[`41ec451f98`](nodejs/node@41ec451f98)] - **doc**: fix --inspect security warning section (Tim Perry) [#​61675](nodejs/node#61675) - \[[`bb90ef2356`](nodejs/node@bb90ef2356)] - **doc**: document `url.format(urlString)` as deprecated under DEP0169 (René) [#​61644](nodejs/node#61644) - \[[`513df82e6f`](nodejs/node@513df82e6f)] - **doc**: update to Visual Studio 2026 manual install (Mike McCready) [#​61655](nodejs/node#61655) - \[[`9409d30736`](nodejs/node@9409d30736)] - **doc**: deprecation add more codemod (Augustin Mauroy) [#​61642](nodejs/node#61642) - \[[`75a7a67151`](nodejs/node@75a7a67151)] - **doc**: fix grammatical error in README.md (ayj8201) [#​61653](nodejs/node#61653) - \[[`821e59e884`](nodejs/node@821e59e884)] - **doc**: correct tools README Boxstarter link (Mike McCready) [#​61638](nodejs/node#61638) - \[[`4998f539a0`](nodejs/node@4998f539a0)] - **doc**: update `server.dropMaxConnection` link (YuSheng Chen) [#​61584](nodejs/node#61584) - \[[`9383ac4ab7`](nodejs/node@9383ac4ab7)] - **http**: implement slab allocation for HTTP header parsing (Mert Can Altin) [#​61375](nodejs/node#61375) - \[[`e90eb1d561`](nodejs/node@e90eb1d561)] - **meta**: persist sccache daemon until end of build workflows (René) [#​61639](nodejs/node#61639) - \[[`ade36ac367`](nodejs/node@ade36ac367)] - **meta**: bump github/codeql-action from 4.31.9 to 4.32.0 (dependabot\[bot]) [#​61622](nodejs/node#61622) - \[[`26638bd67f`](nodejs/node@26638bd67f)] - **meta**: bump step-security/harden-runner from 2.14.0 to 2.14.1 (dependabot\[bot]) [#​61621](nodejs/node#61621) - \[[`eaa9a96cb6`](nodejs/node@eaa9a96cb6)] - **meta**: bump actions/setup-python from 6.1.0 to 6.2.0 (dependabot\[bot]) [#​61627](nodejs/node#61627) - \[[`fd98187828`](nodejs/node@fd98187828)] - **meta**: bump cachix/cachix-action (dependabot\[bot]) [#​61626](nodejs/node#61626) - \[[`820c1d021c`](nodejs/node@820c1d021c)] - **meta**: bump actions/setup-node from 6.1.0 to 6.2.0 (dependabot\[bot]) [#​61625](nodejs/node#61625) - \[[`72a4136bd5`](nodejs/node@72a4136bd5)] - **meta**: bump actions/cache from 5.0.1 to 5.0.3 (dependabot\[bot]) [#​61624](nodejs/node#61624) - \[[`e3ef6cb3bc`](nodejs/node@e3ef6cb3bc)] - **meta**: bump peter-evans/create-pull-request from 8.0.0 to 8.1.0 (dependabot\[bot]) [#​61623](nodejs/node#61623) - \[[`020a836202`](nodejs/node@020a836202)] - **meta**: bump actions/stale from 10.1.0 to 10.1.1 (dependabot\[bot]) [#​61620](nodejs/node#61620) - \[[`0df72f07c8`](nodejs/node@0df72f07c8)] - **meta**: bump actions/checkout from 6.0.1 to 6.0.2 (dependabot\[bot]) [#​61619](nodejs/node#61619) - \[[`d147c08b83`](nodejs/node@d147c08b83)] - **module**: do not invoke resolve hooks twice for imported cjs (Joyee Cheung) [#​61529](nodejs/node#61529) - \[[`a2843f8556`](nodejs/node@a2843f8556)] - **net**: defer synchronous destroy calls in internalConnect (RajeshKumar11) [#​61658](nodejs/node#61658) - \[[`7fb7030781`](nodejs/node@7fb7030781)] - **repl**: fix flaky test-repl-programmatic-history (Matteo Collina) [#​61614](nodejs/node#61614) - \[[`d4c9b5cf5b`](nodejs/node@d4c9b5cf5b)] - **sqlite**: avoid extra copy for large text binds (Ali Hassan) [#​61580](nodejs/node#61580) - \[[`aa1b3661d9`](nodejs/node@aa1b3661d9)] - **sqlite**: use DictionaryTemplate for run() result (Mert Can Altin) [#​61432](nodejs/node#61432) - \[[`9c8ad7e881`](nodejs/node@9c8ad7e881)] - **src**: elide heap allocation in structured clone implementation (Anna Henningsen) [#​61703](nodejs/node#61703) - \[[`c4ecfef93d`](nodejs/node@c4ecfef93d)] - **src**: use simdutf for one-byte string UTF-8 write in stringBytes (Mert Can Altin) [#​61696](nodejs/node#61696) - \[[`28905b9734`](nodejs/node@28905b9734)] - **src**: consolidate C++ ReadFileSync/WriteFileSync utilities (Joyee Cheung) [#​61662](nodejs/node#61662) - \[[`e90cec2f69`](nodejs/node@e90cec2f69)] - **test**: restraint version replacement pattern in snapshots (Chengzhong Wu) [#​61748](nodejs/node#61748) - \[[`adce20c0a1`](nodejs/node@adce20c0a1)] - **test**: print stack immediately avoiding GC interleaving (Chengzhong Wu) [#​61699](nodejs/node#61699) - \[[`7643bc8999`](nodejs/node@7643bc8999)] - **test**: fix case-insensitive path matching on Windows (Matteo Collina) [#​61682](nodejs/node#61682) - \[[`23d1ecf66f`](nodejs/node@23d1ecf66f)] - **test**: fix flaky test-performance-eventloopdelay (Matteo Collina) [#​61629](nodejs/node#61629) - \[[`99012a88ed`](nodejs/node@99012a88ed)] - **test**: remove duplicate wpt tests (Filip Skokan) [#​61617](nodejs/node#61617) - \[[`a8b32b8ce1`](nodejs/node@a8b32b8ce1)] - **test**: fix race condition in watch mode tests (Matteo Collina) [#​61615](nodejs/node#61615) - \[[`086a5a5a25`](nodejs/node@086a5a5a25)] - **test**: update WPT for url to [`e3c46fd`](nodejs/node@e3c46fdf55) (Node.js GitHub Bot) [#​61602](nodejs/node#61602) - \[[`f0574fd419`](nodejs/node@f0574fd419)] - **test**: use the skipIfNoWatch() utility function (Luigi Pinca) [#​61531](nodejs/node#61531) - \[[`b064ddc221`](nodejs/node@b064ddc221)] - **test**: unify assertSnapshot common patterns (Chengzhong Wu) [#​61590](nodejs/node#61590) - \[[`17122e521b`](nodejs/node@17122e521b)] - **test\_runner**: fix test enqueue when test file has syntax error (Edy Silva) [#​61573](nodejs/node#61573) - \[[`bad3f02dd9`](nodejs/node@bad3f02dd9)] - **tools**: enforce removal of `lts-watch-*` labels on release proposals (Antoine du Hamel) [#​61672](nodejs/node#61672) - \[[`a8f33fd6bd`](nodejs/node@a8f33fd6bd)] - **tools**: use ubuntu-slim runner in meta GitHub Actions (Tierney Cyren) [#​61663](nodejs/node#61663) - \[[`c843e447ca`](nodejs/node@c843e447ca)] - **tools**: test `--shared-merve` in `test-shared` workflow (Antoine du Hamel) [#​61649](nodejs/node#61649) - \[[`2fedc03f96`](nodejs/node@2fedc03f96)] - **tools**: update OpenSSL to 3.5.5 in `test-shared` (Antoine du Hamel) [#​61551](nodejs/node#61551) - \[[`1c1db94670`](nodejs/node@1c1db94670)] - **tools,win**: upgrade install additional tools to Visual Studio 2026 (Mike McCready) [#​61562](nodejs/node#61562) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever MR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi45Ny4wIiwidXBkYXRlZEluVmVyIjoiNDIuOTcuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiUmVub3ZhdGUgQm90IiwiYXV0b21hdGlvbjpib3QtYXV0aG9yZWQiLCJkZXBlbmRlbmN5LXR5cGU6OnBhdGNoIl19-->
Summary
Utf8Valuebuffer to SQLite to avoid an extra copy for larger strings.sqlite3_bind_blob64()for BLOB parameters to avoid truncation for larger payloads.