Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: WordPress/sqlite-database-integration
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: trunk
Choose a base ref
...
head repository: WordPress/sqlite-database-integration
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: parser-experiments
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 7 commits
  • 16 files changed
  • 1 contributor

Commits on Jun 6, 2026

  1. Bring in lexer optimization and CI improvements (#424, #425)

    Squash the two upstream PRs into a single base commit so further
    parser-performance experiments can be stacked on top.
    
    - #424: Optimize the MySQL lexer for a ~2x speedup (inline whitespace
      skipping, dispatch reordering, single-byte operator table, inlined
      keyword lookup, cached length) plus a lexer benchmark CI job.
    - #425: Disable Xdebug in CI, build the Rust extension in release mode
      with caching, and consolidate the unit-test workflows into one matrix.
    
    Refs:
    #424
    #425
    JanJakes committed Jun 6, 2026
    Configuration menu
    Copy the full SHA
    05be33f View commit details
    Browse the repository at this point in the history
  2. Add experimental LALR(1) parser for the MySQL grammar

    Explore a bottom-up parser as an alternative to the backtracking LL parser
    in WP_Parser. Three pieces:
    
    - grammar-tools/build-lalr-table.php: an offline LALR(1) table generator. It
      builds the canonical LR(0) automaton (6.3k states), computes LALR(1)
      lookaheads by propagation, and emits ACTION/GOTO tables. The raw ANTLR-derived
      grammar yields ~345k reduce/reduce conflicts, almost all from the overlapping
      keyword-as-identifier rules. Two language-preserving transforms fix this:
      refactoring the keyword sets into shared disjoint atom-classes, and merging
      structurally-identical rules. That cuts reduce/reduce conflicts to ~8.4k.
    
    - class-wp-mysql-lalr-parser.php: a runtime shift-reduce parser. The grammar is
      not LALR(1) (e.g. INSERT ... VALUES vs a VALUES table-value-constructor needs
      two tokens of lookahead), so conflicted cells keep all candidate actions and
      the parser forks GLR-style, with failed-configuration memoization to keep the
      backtracking tractable. It builds a WP_Parser_Node AST with fragment inlining.
    
    - run-lalr-benchmark.php: corpus runner mirroring run-parser-benchmark.php.
    
    Results on the 69,577-query corpus: matches the LL parser's grammar coverage
    (8 failures vs 9, all shared grammar/lexer gaps; zero LALR-specific failures)
    at ~1.75x the throughput (~2,400 vs ~1,350 QPS). The generated table is large
    and regenerable, so it is gitignored.
    JanJakes committed Jun 6, 2026
    Configuration menu
    Copy the full SHA
    fe627b5 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2026

  1. Store the LALR table as a comb vector (112x smaller)

    Replace the verbose var_export table (~30 MB) with a displacement-packed
    (comb-vector) layout, the standard compact LR-table representation.
    
    Generator changes:
    - Default-reduction: drop the dominant per-state reduce into a separate
      default[] array (most states reduce unconditionally, so their rows vanish).
    - Row sharing: dedupe identical ACTION/GOTO rows; dedupe conflict action lists.
    - Displacement packing: place all rows into one value[] array with per-row
      base offsets and a check[] guard, for both ACTION and GOTO.
    - Store the big integer arrays as base64(gzdeflate(pack 'l*')) so the file is
      small on disk and decodes once into memory-efficient packed PHP arrays.
    
    Runtime changes:
    - Look up actions via value[base[row]+col] guarded by check[], falling back to
      the per-state default; new integer action encoding (shift/accept/conflict/
      reduce); GOTO consulted the same way.
    
    The generated table drops from ~30,895 KB to ~277 KB (112x). Parse results and
    throughput are unchanged (8 prefix / 51 strict corpus failures; ~2,300 QPS),
    and the loaded footprint is ~13 MB of flat int arrays instead of 100+ MB of
    nested arrays.
    JanJakes committed Jun 7, 2026
    Configuration menu
    Copy the full SHA
    55e48ae View commit details
    Browse the repository at this point in the history
  2. Pack the comb table's column maps and conflict lists

    Encode the remaining var_export fields compactly: store the ACTION/GOTO column
    maps as ordered symbol-id lists (column = position, map rebuilt at load) and
    flatten the conflict action lists to a length-prefixed integer stream, both via
    the same base64(gzdeflate(pack 'l*')) path as the other arrays.
    
    Table drops from ~277 KB to ~205 KB. Rule names are intentionally left as a
    plain array for now. Parse results and throughput are unchanged.
    JanJakes committed Jun 7, 2026
    Configuration menu
    Copy the full SHA
    8be4931 View commit details
    Browse the repository at this point in the history
  3. Apply per-column defaults to the GOTO comb vector

    Most states GOTO the same target for a given nonterminal, so store that target
    once per column as a default and keep only the exceptions in the comb vector.
    The GOTO comb array drops from ~163k slots to ~29k (5.6x), guarded by g_check
    with the per-column default as the fallback.
    
    Table drops from ~205 KB to ~178 KB. Parse results and throughput unchanged.
    JanJakes committed Jun 7, 2026
    Configuration menu
    Copy the full SHA
    2d4c260 View commit details
    Browse the repository at this point in the history
  4. Add single-production inlining experiment (off by default, net-negative)

    Add an opt-in --inline grammar transform that removes single-production
    nonterminals (pure aliases, no blowup unlike full unit elimination) plus a
    --stage=units diagnostic. Default builds are unchanged.
    
    Measured result on the corpus, so the negative outcome is reproducible:
    inlining the %f fragments shrinks the automaton (7920 -> 7537 states, table
    178 -> 164 KB) but RAISES reduce/reduce conflicts (8365 -> 8859) and roughly
    halves throughput (~2320 -> ~1160 QPS) because this GLR parser is bound by
    conflict forking, not reduce-step count. --inline=all (also real rules) is
    worse still: it increases states. So inlining is not adopted; the intermediate
    fragments are load-bearing for parse determinism.
    JanJakes committed Jun 7, 2026
    Configuration menu
    Copy the full SHA
    5ee35f7 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2026

  1. Give the LALR benchmark memory headroom and bound the fail-memo

    The benchmark hit PHP's 128M CLI default: the GLR fallback forks heavily on a
    few pathological queries (peak ~450 MB over the corpus). Have the benchmark
    raise memory_limit to 1G when it is lower (never lowering it, leaving -1 alone),
    and cap the failed-configuration memo at 200k entries (down from an absurd 2M)
    to bound per-query memory without losing the pruning that keeps those queries
    from blowing up.
    JanJakes committed Jun 8, 2026
    Configuration menu
    Copy the full SHA
    83f3d9a View commit details
    Browse the repository at this point in the history
Loading