Skip to content

parser: +feat Choose the post-processing of the regex conversion with Re2Options - #949

Merged
Adda0 merged 1 commit into
develfrom
issue-793-re2-options-pr
Oct 16, 2026
Merged

Adda0 merged 1 commit into
develfrom
issue-793-re2-options-pr

Conversation

@Adda0

@Adda0 Adda0 commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

create_nfa() tied the conversion and its post-processing together
through positional booleans: it always trimmed the result, removed
epsilon transitions unless use_epsilon was set, and reduced it with
use_reduce, so a caller could not get a raw conversion out of the
parser, and builder::create_from_regex() and the Python from_regex()
exposed only the reduction.

The new Re2Options struct (keep_epsilon, epsilon, encoding, trim,
reduce) is taken by new overloads of create_nfa() and
builder::create_from_regex(). Its defaults reproduce the previous
behaviour, and the existing overloads forward to it with a trimmed
result, so no current caller changes. The Python from_regex() gains
the trim and keep_epsilon keyword arguments, with defaults matching
the previous behaviour.

Part of #793.

struct Re2Options {
	bool keep_epsilon{false};
	Symbol epsilon{306};
	Encoding encoding{Encoding::Latin1};
	bool trim{true};
	bool reduce{true};
};
nfa::Nfa create_nfa(const std::string& pattern, const Re2Options& options);
Nfa builder::create_from_regex(const std::string& regex, const parser::Re2Options& options);
parser.from_regex(regex, encoding='utf-8', reduce=True, trim=True, keep_epsilon=False)

The existing create_nfa(pattern, use_epsilon, epsilon_value, use_reduce, encoding) overloads and create_from_regex(regex, bool reduce = true) are unchanged and keep their behaviour.

Verification

  • C++ test mata::parser::create_nfa() post-processing options: the defaults give the same automaton as the old overloads, both directly and through create_from_regex(); keep_epsilon keeps epsilon transitions labelled with the requested symbol (500), and none labelled 306; trim = false keeps the useless state of a[^\x00-\xff]; encoding reaches the parser (UTF-8 \x{100} is c4 80, and Latin1 rejects it).
  • Python test test_post_processing_options covers keep_epsilon and trim.
  • This PR changes only the API, so there is no performance claim. The conversion pipeline itself is the same.

Stack

Second of three PRs for #793. Based on #948 (opcode handling), so the base of this PR is that branch. GitHub retargets it to devel once #948 is merged.

Overlap:

… Re2Options

create_nfa() tied the conversion and its post-processing together
through positional booleans: it always trimmed the result, removed
epsilon transitions unless use_epsilon was set, and reduced it with
use_reduce, so a caller could not get a raw conversion out of the
parser, and builder::create_from_regex() and the Python from_regex()
exposed only the reduction.

The new Re2Options struct (keep_epsilon, epsilon, encoding, trim,
reduce) is taken by new overloads of create_nfa() and
builder::create_from_regex(). Its defaults reproduce the previous
behaviour, and the existing overloads forward to it with a trimmed
result, so no current caller changes. The Python from_regex() gains
the trim and keep_epsilon keyword arguments, with defaults matching
the previous behaviour.

Part of #793.
@Adda0
Adda0 force-pushed the issue-793-re2-options-pr branch from fae99d5 to 9370fe2 Compare October 15, 2026 18:24
@Adda0
Adda0 force-pushed the issue-793-re2-opcodes-pr branch from 995f158 to e5386fc Compare October 15, 2026 18:24
Base automatically changed from issue-793-re2-opcodes-pr to devel October 16, 2026 05:40
Adda0 added a commit that referenced this pull request Oct 16, 2026
The RE2 converter only logged opcodes it did not handle and treated a
match as the end of its instruction list. Two regex families came out
with a wrong language:

- A non-greedy repetition puts the match before the repeated part in the
  same list, so the entries after the match were cut off: 'a*?' accepted
  only the empty word and 'a+?' only 'a'.
- RE2 rewrites the alternation of a '[\x00-\xff]*' loop and a match to
an
  AltMatch instruction, e.g., in '(?s)a.*' or 'a[\x00-\xff]*'. The
  converter skipped it together with the alternatives following it in
  the list, so these regexes produced the empty language.

A match, a fail and an AltMatch entry now continue to the next entry of
their list like every other non-last entry; a fail adds no transition of
its own. Any other opcode, which cannot occur in a flattened program,
throws std::runtime_error instead of being dropped silently.

The 'b\w+?' test expected the matching semantics of the lazy repetition
(only 'be'). The automaton represents the language of the regex, which
is the language of 'b\w+', so the test now expects the longer words too.

Part of #793.

### Verification

- New test `mata::parser::create_nfa() instructions without a consuming
transition of their own` checks `a*?`, `a+?`, `a??b`, `(ab)*?c`,
`(?s)a.*`, `a[\x00-\xff]*`, `(?s).*`, `(?s).*?b`, `[^\x00-\xff]` and
`a|[^\x00-\xff]` with and without epsilon transitions and reduction.
With the continuation of match/fail/AltMatch entries disabled, it fails
on `a*?`, `a+?`, `a[\x00-\xff]*`, `(?s).*` and `(?s)a.*`.
- On devel, `(?s)a.*` and `a[\x00-\xff]*` give the empty language and
log `unhandled 1 in convertProgToNfa`. `a*?` accepts only the empty
word.

### Stack

This is the first of three PRs for #793:
1. this PR, which fixes the opcode handling;
2. #949, `Re2Options`, stacked on this PR;
3. #950, the direct epsilon-free conversion, stacked on #949.

Overlap: open PR #875 also appends tests to `tests/re2parser.cc`. Both
add new test cases at the end of the file, so any conflict is trivial.
@Adda0
Adda0 merged commit 7efcb1e into devel Oct 16, 2026
50 of 51 checks passed
@Adda0
Adda0 deleted the issue-793-re2-options-pr branch October 16, 2026 06:56
Adda0 added a commit that referenced this pull request Oct 16, 2026
…ransitions (#950)

create_nfa() always translated the RE2 program with an epsilon
transition for every nop, capture and empty-width instruction and for
every continuation of an instruction list, trimmed the result, and then
ran the global remove_epsilon() to get the epsilon-free NFA every
default caller asks for. The epsilon-free branch of the converter
(create_state_cache_without_epsilon(), get_mapped_states() and the
reversed edge copying) was never reached and, as its FIXME said, did not
work.

The epsilon-free conversion is now direct. A state stands for the
instruction list the program continues at: the start and the target of
each byte range. Its closure follows the non-consuming instructions and
the list continuations to byte ranges and matches; the state gets the
byte-range transitions of the closure and is final when the closure
contains a match. A worklist from the start creates states only for
reachable lists, so RE2's unanchored prefix never becomes a state. On
the regexes of the test suite and of the new benchmark, the conversion
leaves no useless state behind; only branches that can never match,
such as '[^\x00-\xff]' in Latin1, do, so trim stays enabled by default.

The explicit epsilon translation stays as the keep_epsilon mode, with
the caller's epsilon symbol, rewritten as a plain worklist over the
reachable instructions. The broken branch, its state cache and the
FIXME are gone.

Tests: a seeded differential test generates regexes covering every
instruction form (byte ranges with and without case folding, captures,
alternation lists, empty branches, greedy, non-greedy and counted
repetitions, '(?s:.)' loops compiled to AltMatch, anchors) and checks
the direct conversion, untrimmed and reduced, against the epsilon
translation followed by remove_epsilon(), and against the language of
the generated syntax tree on all words up to length 4. The bundled RE2
has no matcher, so RE2 itself cannot serve as the oracle.
bench-regex-conversion times the conversions, remove_epsilon(), trim()
and reduce() separately.

Part of #793.

### Measurements

Release build, g++, pinned to one core (`taskset -c 13`). `perf`
harness: median wall time of a `create_nfa(regex, false, 306, reduce)`
call. Each value is the median of 5 alternating base/new processes with
3 calls each. Base = `devel` at b6f7fd1; new = this stack on the same
base. It has since been rebased onto `devel` at 8b4c09b; the only
conflict was #939's casts in the removed `convert_pro_to_nfa`, and the
converter itself is unchanged. The number of states and transitions of
the trimmed result is identical between base and new for every regex.

Without reduction (`reduce = false`, the path used by `replace.cc`):

| regex | base [ms] | new [ms] | states / transitions |
|---|---:|---:|---|
| `(?:[a-z]b\|c[d-f]){1000}` | 9.74 | 2.44 | 3001 / 31000 |
| 2000-character literal | 1.62 | 0.53 | 2001 / 2000 |
| `a?` × 300 followed by `b` | 0.88 | 0.47 | 302 / 601 |
| `(?:a?b?c?){100}d` | 7504 | 2.14 | 302 / 45451 |
| `(?:((a)*)*b\|(c(d)*)){200}` | 4.07 | 0.60 | 801 / 2594 |
| `(?:(?:ab\|cd)*?e+?(?:f\|g\|)){200}` | 1.72 | 0.47 | 801 / 2197 |
| `a*b\|c(de)*` | 0.0147 | 0.0084 | 6 / 8 |
| `[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}` | 0.060 | 0.025 | 7 / 238 |

With the default reduction:

| regex | base [ms] | new [ms] |
|---|---:|---:|
| `(?:((a)*)*b\|(c(d)*)){200}` | 23.1 | 19.5 |
| `(?:(?:ab\|cd)*?e+?(?:f\|g\|)){200}` | 23.2 | 21.3 |
| `a*b\|c(de)*` | 0.031 | 0.025 |
| `[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}` | 0.262 | 0.220 |

`(?:a?b?c?){100}d` hits the forward-chain case of `remove_epsilon()`
(#240): 7.0 s of the 7.5 s go to `remove_epsilon()` (see the stage
timers below). Note that the base language of the lazy
`(?:(?:ab|cd)*?e+?…)` regex differs, because the base still had the bug
fixed by #948.

Stage timers from the new `bench-regex-conversion` (seconds, one run):

| regex | convert_with_epsilon | remove_epsilon | convert_direct | trim
| reduce |
|---|---:|---:|---:|---:|---:|
| `(?:[a-z]b\|c[d-f]){1000}` | 0.0047 | 0.0052 | 0.0025 | 0.0005 |
1.9173 |
| 2000-character literal | 0.0005 | 0.0004 | 0.0003 | 0.0001 | 0.0186 |
| `a?` × 300 followed by `b` | 0.0005 | 0.0002 | 0.0004 | 0.0000 |
0.0031 |
| `(?:a?b?c?){100}d` | 0.0002 | 6.9871 | 0.0019 | 0.0004 | 0.0200 |
| `(?:((a)*)*b\|(c(d)*)){200}` | 0.0009 | 0.0019 | 0.0006 | 0.0001 |
0.0220 |
| `(?:(?:ab\|cd)*?e+?(?:f\|g\|)){200}` | 0.0006 | 0.0008 | 0.0004 |
0.0000 | 0.0206 |

Useless states (#793 item 6): over the 136 regexes used in the tests
(`create_nfa`/`create_from_regex` literals and regex lists) plus the 8
benchmark regexes, the untrimmed direct conversion had 7788 states, and
`trim()` removed none of them.

Language equivalence against the old conversion: every regex of that
corpus was converted with devel and with this stack, written as `.mata`
files and compared with `are_equivalent()`, both reduced and unreduced.
270 of 272 automata are equivalent. The two that differ are `b\w+?`
(plain and reduced), whose devel result at the time was the bug fixed by
#948. The randomized differential test adds 400 generated regexes on
every run.

### Scope

Last of three PRs for #793; the first two, #948 and #949, are merged.

Not in this PR:
- RE2 acceptance as an oracle (plan step 7): the bundled RE2 is trimmed
to its parser and compiler and has no matcher. The test uses the
language of the generated syntax tree instead.
- Instrumentation counters in the library (plan step 6): the counts
above come from the benchmark and from a throwaway harness.
- Separate timers for parsing and compiling: the public API runs both
inside `create_nfa()`, so they are part of the conversion timers.

Found while testing, and left out of scope: the bundled RE2 parser drops
the folded case of a letter that it merges into a character class that
already holds that letter. `a|(?i:a)` parses to `[a]`, and `[ab]|(?i:a)`
parses to `[a-b]`, so `A` is rejected. The generator avoids this case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant