Repository navigation
parser: +feat Choose the post-processing of the regex conversion with Re2Options - #949
Merged
Merged
Conversation
This was referenced Oct 15, 2026
… Re2Options create_nfa() tied the conversion and its post-processing together through positional booleans: it always trimmed the result, removed epsilon transitions unless use_epsilon was set, and reduced it with use_reduce, so a caller could not get a raw conversion out of the parser, and builder::create_from_regex() and the Python from_regex() exposed only the reduction. The new Re2Options struct (keep_epsilon, epsilon, encoding, trim, reduce) is taken by new overloads of create_nfa() and builder::create_from_regex(). Its defaults reproduce the previous behaviour, and the existing overloads forward to it with a trimmed result, so no current caller changes. The Python from_regex() gains the trim and keep_epsilon keyword arguments, with defaults matching the previous behaviour. Part of #793.
Adda0
force-pushed
the
issue-793-re2-options-pr
branch
from
October 15, 2026 18:24
fae99d5 to
9370fe2
Compare
Adda0
force-pushed
the
issue-793-re2-opcodes-pr
branch
from
October 15, 2026 18:24
995f158 to
e5386fc
Compare
Adda0
added a commit
that referenced
this pull request
Oct 16, 2026
The RE2 converter only logged opcodes it did not handle and treated a match as the end of its instruction list. Two regex families came out with a wrong language: - A non-greedy repetition puts the match before the repeated part in the same list, so the entries after the match were cut off: 'a*?' accepted only the empty word and 'a+?' only 'a'. - RE2 rewrites the alternation of a '[\x00-\xff]*' loop and a match to an AltMatch instruction, e.g., in '(?s)a.*' or 'a[\x00-\xff]*'. The converter skipped it together with the alternatives following it in the list, so these regexes produced the empty language. A match, a fail and an AltMatch entry now continue to the next entry of their list like every other non-last entry; a fail adds no transition of its own. Any other opcode, which cannot occur in a flattened program, throws std::runtime_error instead of being dropped silently. The 'b\w+?' test expected the matching semantics of the lazy repetition (only 'be'). The automaton represents the language of the regex, which is the language of 'b\w+', so the test now expects the longer words too. Part of #793. ### Verification - New test `mata::parser::create_nfa() instructions without a consuming transition of their own` checks `a*?`, `a+?`, `a??b`, `(ab)*?c`, `(?s)a.*`, `a[\x00-\xff]*`, `(?s).*`, `(?s).*?b`, `[^\x00-\xff]` and `a|[^\x00-\xff]` with and without epsilon transitions and reduction. With the continuation of match/fail/AltMatch entries disabled, it fails on `a*?`, `a+?`, `a[\x00-\xff]*`, `(?s).*` and `(?s)a.*`. - On devel, `(?s)a.*` and `a[\x00-\xff]*` give the empty language and log `unhandled 1 in convertProgToNfa`. `a*?` accepts only the empty word. ### Stack This is the first of three PRs for #793: 1. this PR, which fixes the opcode handling; 2. #949, `Re2Options`, stacked on this PR; 3. #950, the direct epsilon-free conversion, stacked on #949. Overlap: open PR #875 also appends tests to `tests/re2parser.cc`. Both add new test cases at the end of the file, so any conflict is trivial.
Adda0
added a commit
that referenced
this pull request
Oct 16, 2026
…ransitions (#950) create_nfa() always translated the RE2 program with an epsilon transition for every nop, capture and empty-width instruction and for every continuation of an instruction list, trimmed the result, and then ran the global remove_epsilon() to get the epsilon-free NFA every default caller asks for. The epsilon-free branch of the converter (create_state_cache_without_epsilon(), get_mapped_states() and the reversed edge copying) was never reached and, as its FIXME said, did not work. The epsilon-free conversion is now direct. A state stands for the instruction list the program continues at: the start and the target of each byte range. Its closure follows the non-consuming instructions and the list continuations to byte ranges and matches; the state gets the byte-range transitions of the closure and is final when the closure contains a match. A worklist from the start creates states only for reachable lists, so RE2's unanchored prefix never becomes a state. On the regexes of the test suite and of the new benchmark, the conversion leaves no useless state behind; only branches that can never match, such as '[^\x00-\xff]' in Latin1, do, so trim stays enabled by default. The explicit epsilon translation stays as the keep_epsilon mode, with the caller's epsilon symbol, rewritten as a plain worklist over the reachable instructions. The broken branch, its state cache and the FIXME are gone. Tests: a seeded differential test generates regexes covering every instruction form (byte ranges with and without case folding, captures, alternation lists, empty branches, greedy, non-greedy and counted repetitions, '(?s:.)' loops compiled to AltMatch, anchors) and checks the direct conversion, untrimmed and reduced, against the epsilon translation followed by remove_epsilon(), and against the language of the generated syntax tree on all words up to length 4. The bundled RE2 has no matcher, so RE2 itself cannot serve as the oracle. bench-regex-conversion times the conversions, remove_epsilon(), trim() and reduce() separately. Part of #793. ### Measurements Release build, g++, pinned to one core (`taskset -c 13`). `perf` harness: median wall time of a `create_nfa(regex, false, 306, reduce)` call. Each value is the median of 5 alternating base/new processes with 3 calls each. Base = `devel` at b6f7fd1; new = this stack on the same base. It has since been rebased onto `devel` at 8b4c09b; the only conflict was #939's casts in the removed `convert_pro_to_nfa`, and the converter itself is unchanged. The number of states and transitions of the trimmed result is identical between base and new for every regex. Without reduction (`reduce = false`, the path used by `replace.cc`): | regex | base [ms] | new [ms] | states / transitions | |---|---:|---:|---| | `(?:[a-z]b\|c[d-f]){1000}` | 9.74 | 2.44 | 3001 / 31000 | | 2000-character literal | 1.62 | 0.53 | 2001 / 2000 | | `a?` × 300 followed by `b` | 0.88 | 0.47 | 302 / 601 | | `(?:a?b?c?){100}d` | 7504 | 2.14 | 302 / 45451 | | `(?:((a)*)*b\|(c(d)*)){200}` | 4.07 | 0.60 | 801 / 2594 | | `(?:(?:ab\|cd)*?e+?(?:f\|g\|)){200}` | 1.72 | 0.47 | 801 / 2197 | | `a*b\|c(de)*` | 0.0147 | 0.0084 | 6 / 8 | | `[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}` | 0.060 | 0.025 | 7 / 238 | With the default reduction: | regex | base [ms] | new [ms] | |---|---:|---:| | `(?:((a)*)*b\|(c(d)*)){200}` | 23.1 | 19.5 | | `(?:(?:ab\|cd)*?e+?(?:f\|g\|)){200}` | 23.2 | 21.3 | | `a*b\|c(de)*` | 0.031 | 0.025 | | `[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}` | 0.262 | 0.220 | `(?:a?b?c?){100}d` hits the forward-chain case of `remove_epsilon()` (#240): 7.0 s of the 7.5 s go to `remove_epsilon()` (see the stage timers below). Note that the base language of the lazy `(?:(?:ab|cd)*?e+?…)` regex differs, because the base still had the bug fixed by #948. Stage timers from the new `bench-regex-conversion` (seconds, one run): | regex | convert_with_epsilon | remove_epsilon | convert_direct | trim | reduce | |---|---:|---:|---:|---:|---:| | `(?:[a-z]b\|c[d-f]){1000}` | 0.0047 | 0.0052 | 0.0025 | 0.0005 | 1.9173 | | 2000-character literal | 0.0005 | 0.0004 | 0.0003 | 0.0001 | 0.0186 | | `a?` × 300 followed by `b` | 0.0005 | 0.0002 | 0.0004 | 0.0000 | 0.0031 | | `(?:a?b?c?){100}d` | 0.0002 | 6.9871 | 0.0019 | 0.0004 | 0.0200 | | `(?:((a)*)*b\|(c(d)*)){200}` | 0.0009 | 0.0019 | 0.0006 | 0.0001 | 0.0220 | | `(?:(?:ab\|cd)*?e+?(?:f\|g\|)){200}` | 0.0006 | 0.0008 | 0.0004 | 0.0000 | 0.0206 | Useless states (#793 item 6): over the 136 regexes used in the tests (`create_nfa`/`create_from_regex` literals and regex lists) plus the 8 benchmark regexes, the untrimmed direct conversion had 7788 states, and `trim()` removed none of them. Language equivalence against the old conversion: every regex of that corpus was converted with devel and with this stack, written as `.mata` files and compared with `are_equivalent()`, both reduced and unreduced. 270 of 272 automata are equivalent. The two that differ are `b\w+?` (plain and reduced), whose devel result at the time was the bug fixed by #948. The randomized differential test adds 400 generated regexes on every run. ### Scope Last of three PRs for #793; the first two, #948 and #949, are merged. Not in this PR: - RE2 acceptance as an oracle (plan step 7): the bundled RE2 is trimmed to its parser and compiler and has no matcher. The test uses the language of the generated syntax tree instead. - Instrumentation counters in the library (plan step 6): the counts above come from the benchmark and from a throwaway harness. - Separate timers for parsing and compiling: the public API runs both inside `create_nfa()`, so they are part of the conversion timers. Found while testing, and left out of scope: the bundled RE2 parser drops the folded case of a letter that it merges into a character class that already holds that letter. `a|(?i:a)` parses to `[a]`, and `[ab]|(?i:a)` parses to `[a-b]`, so `A` is rejected. The generator avoids this case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
create_nfa() tied the conversion and its post-processing together
through positional booleans: it always trimmed the result, removed
epsilon transitions unless use_epsilon was set, and reduced it with
use_reduce, so a caller could not get a raw conversion out of the
parser, and builder::create_from_regex() and the Python from_regex()
exposed only the reduction.
The new Re2Options struct (keep_epsilon, epsilon, encoding, trim,
reduce) is taken by new overloads of create_nfa() and
builder::create_from_regex(). Its defaults reproduce the previous
behaviour, and the existing overloads forward to it with a trimmed
result, so no current caller changes. The Python from_regex() gains
the trim and keep_epsilon keyword arguments, with defaults matching
the previous behaviour.
Part of #793.
The existing
create_nfa(pattern, use_epsilon, epsilon_value, use_reduce, encoding)overloads andcreate_from_regex(regex, bool reduce = true)are unchanged and keep their behaviour.Verification
mata::parser::create_nfa() post-processing options: the defaults give the same automaton as the old overloads, both directly and throughcreate_from_regex();keep_epsilonkeeps epsilon transitions labelled with the requested symbol (500), and none labelled 306;trim = falsekeeps the useless state ofa[^\x00-\xff];encodingreaches the parser (UTF-8\x{100}isc4 80, and Latin1 rejects it).test_post_processing_optionscoverskeep_epsilonandtrim.Stack
Second of three PRs for #793. Based on #948 (opcode handling), so the base of this PR is that branch. GitHub retargets it to
develonce #948 is merged.Overlap:
start_statelines ofconvert_pro_to_nfa, which the third PR of this stack removes;parser.pxd;from_matainparser.pyx, next tofrom_regex;src/nfa/builder.ccin other functions.