-
Notifications
You must be signed in to change notification settings - Fork 61
Comparing changes
Open a pull request
base repository: WordPress/sqlite-database-integration
base: trunk
head repository: WordPress/sqlite-database-integration
compare: parser-experiments
- 7 commits
- 16 files changed
- 1 contributor
Commits on Jun 6, 2026
-
Bring in lexer optimization and CI improvements (#424, #425)
Squash the two upstream PRs into a single base commit so further parser-performance experiments can be stacked on top. - #424: Optimize the MySQL lexer for a ~2x speedup (inline whitespace skipping, dispatch reordering, single-byte operator table, inlined keyword lookup, cached length) plus a lexer benchmark CI job. - #425: Disable Xdebug in CI, build the Rust extension in release mode with caching, and consolidate the unit-test workflows into one matrix. Refs: #424 #425
Configuration menu - View commit details
-
Copy full SHA for 05be33f - Browse repository at this point
Copy the full SHA 05be33fView commit details -
Add experimental LALR(1) parser for the MySQL grammar
Explore a bottom-up parser as an alternative to the backtracking LL parser in WP_Parser. Three pieces: - grammar-tools/build-lalr-table.php: an offline LALR(1) table generator. It builds the canonical LR(0) automaton (6.3k states), computes LALR(1) lookaheads by propagation, and emits ACTION/GOTO tables. The raw ANTLR-derived grammar yields ~345k reduce/reduce conflicts, almost all from the overlapping keyword-as-identifier rules. Two language-preserving transforms fix this: refactoring the keyword sets into shared disjoint atom-classes, and merging structurally-identical rules. That cuts reduce/reduce conflicts to ~8.4k. - class-wp-mysql-lalr-parser.php: a runtime shift-reduce parser. The grammar is not LALR(1) (e.g. INSERT ... VALUES vs a VALUES table-value-constructor needs two tokens of lookahead), so conflicted cells keep all candidate actions and the parser forks GLR-style, with failed-configuration memoization to keep the backtracking tractable. It builds a WP_Parser_Node AST with fragment inlining. - run-lalr-benchmark.php: corpus runner mirroring run-parser-benchmark.php. Results on the 69,577-query corpus: matches the LL parser's grammar coverage (8 failures vs 9, all shared grammar/lexer gaps; zero LALR-specific failures) at ~1.75x the throughput (~2,400 vs ~1,350 QPS). The generated table is large and regenerable, so it is gitignored.
Configuration menu - View commit details
-
Copy full SHA for fe627b5 - Browse repository at this point
Copy the full SHA fe627b5View commit details
Commits on Jun 7, 2026
-
Store the LALR table as a comb vector (112x smaller)
Replace the verbose var_export table (~30 MB) with a displacement-packed (comb-vector) layout, the standard compact LR-table representation. Generator changes: - Default-reduction: drop the dominant per-state reduce into a separate default[] array (most states reduce unconditionally, so their rows vanish). - Row sharing: dedupe identical ACTION/GOTO rows; dedupe conflict action lists. - Displacement packing: place all rows into one value[] array with per-row base offsets and a check[] guard, for both ACTION and GOTO. - Store the big integer arrays as base64(gzdeflate(pack 'l*')) so the file is small on disk and decodes once into memory-efficient packed PHP arrays. Runtime changes: - Look up actions via value[base[row]+col] guarded by check[], falling back to the per-state default; new integer action encoding (shift/accept/conflict/ reduce); GOTO consulted the same way. The generated table drops from ~30,895 KB to ~277 KB (112x). Parse results and throughput are unchanged (8 prefix / 51 strict corpus failures; ~2,300 QPS), and the loaded footprint is ~13 MB of flat int arrays instead of 100+ MB of nested arrays.
Configuration menu - View commit details
-
Copy full SHA for 55e48ae - Browse repository at this point
Copy the full SHA 55e48aeView commit details -
Pack the comb table's column maps and conflict lists
Encode the remaining var_export fields compactly: store the ACTION/GOTO column maps as ordered symbol-id lists (column = position, map rebuilt at load) and flatten the conflict action lists to a length-prefixed integer stream, both via the same base64(gzdeflate(pack 'l*')) path as the other arrays. Table drops from ~277 KB to ~205 KB. Rule names are intentionally left as a plain array for now. Parse results and throughput are unchanged.
Configuration menu - View commit details
-
Copy full SHA for 8be4931 - Browse repository at this point
Copy the full SHA 8be4931View commit details -
Apply per-column defaults to the GOTO comb vector
Most states GOTO the same target for a given nonterminal, so store that target once per column as a default and keep only the exceptions in the comb vector. The GOTO comb array drops from ~163k slots to ~29k (5.6x), guarded by g_check with the per-column default as the fallback. Table drops from ~205 KB to ~178 KB. Parse results and throughput unchanged.
Configuration menu - View commit details
-
Copy full SHA for 2d4c260 - Browse repository at this point
Copy the full SHA 2d4c260View commit details -
Add single-production inlining experiment (off by default, net-negative)
Add an opt-in --inline grammar transform that removes single-production nonterminals (pure aliases, no blowup unlike full unit elimination) plus a --stage=units diagnostic. Default builds are unchanged. Measured result on the corpus, so the negative outcome is reproducible: inlining the %f fragments shrinks the automaton (7920 -> 7537 states, table 178 -> 164 KB) but RAISES reduce/reduce conflicts (8365 -> 8859) and roughly halves throughput (~2320 -> ~1160 QPS) because this GLR parser is bound by conflict forking, not reduce-step count. --inline=all (also real rules) is worse still: it increases states. So inlining is not adopted; the intermediate fragments are load-bearing for parse determinism.
Configuration menu - View commit details
-
Copy full SHA for 5ee35f7 - Browse repository at this point
Copy the full SHA 5ee35f7View commit details
Commits on Jun 8, 2026
-
Give the LALR benchmark memory headroom and bound the fail-memo
The benchmark hit PHP's 128M CLI default: the GLR fallback forks heavily on a few pathological queries (peak ~450 MB over the corpus). Have the benchmark raise memory_limit to 1G when it is lower (never lowering it, leaving -1 alone), and cap the failed-configuration memo at 200k entries (down from an absurd 2M) to bound per-query memory without losing the pruning that keeps those queries from blowing up.
Configuration menu - View commit details
-
Copy full SHA for 83f3d9a - Browse repository at this point
Copy the full SHA 83f3d9aView commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff trunk...parser-experiments