Different PCRE result /w and /wo JIT enabled

[mvorisek]

see php/php-src#11374

repro in C code: php/php-src#11374 (comment)

/cc @iluuu1994 @Wirone

[zherczeg]

Summary:
Pattern: (?<all>(?:(?:a(?&all))|(b))(c?))
Input: aabc

It looks like the code works as intended, but things might be changed. As far as I remember, recursive calls are considered as a "subroutine calls", which uses brackets as private variables. Since (b) is part of the recursion, its match is not visible outside. I remember I used this fact for various optimizations, e.g. when atomic groups are failed to match. Is there any change of recursions since then?

[mvorisek]

I agree with this JITted definition as returning "subroutine calls" matches (when not matched in the "main call") is completely useless. So non-JITted version should be probably fixed.

Also JITted/non-JITted results should be fuzzed and compared to prevent such inconsistencies in the future.

[zherczeg]

It seems perl agrees with me:

$str = "aabc";
if ($str =~ /(?<all>(?:(?:a(?&all))|(b))(c?))/) {
   print "'$1' '$2' '$3'\n";
}

$2 and $3 are empty

[zherczeg]

I completely misunderstood this request, I thought the jit code is wrong.

[mvorisek]

I completely misunderstood this request, I thought the jit code is wrong.

The JITted result seems better to me, but it is a question about the capture groups/recursion PCRE definition. Someone from core PCRE team should be able to tell. But non-JITted/JITted output should be definitely always the same.

[PhilipHazel]

This is an area where PCRE and Perl have differed and not differed over the years. PCRE had recursion (= subroutine calls) before Perl did. There is a section in pcre2compat.3 entitled "Differences in recursion processing between PCRE2 and Perl", but Perl is a moving target, so it may no longer be accurate.

You are right in thinking that JIT and non-JIT should give the same answers. At least, most of the time. The majority of the standard tests are run with and without JIT to ensure that this is the case. Clearly, none of the existing tests show up this anomaly.

Since JIT agrees with Perl, I suppose it is the interpreter (non-JIT match) that should change. I have not looked at the interpreter code, but I suspect that it is not trivial to arrange to unset (or possibly unwind) captures from inside a subroutine call.

"The core PCRE team" is mentioned above. For the record, there are really only three of us: me for the compiler and interpreter, Zoltan for the JIT, and Carlo, who has provided a number of patches in several areas. The bad news is that I have recently had my 79th birthday, and working with tricky code such as the PCRE interpreter has become harder than it used to be. (Thinking about recursion has always been hard, of course. :-) When I get some time, I will look at the interpreter and see what the prospects for changing it are. Meanwhile, if anybody else wants a challenge......

[antman3351]

Hi,
I don't know if this is any help, I noticed in a PHP project a difference between JIT e non when using this regex <(\w+)[\s\w\-]+id="S44_i89ew"> against this string <br><div id="S44_i89ew"> the \w in the [ ]+ is what is causing the problem but []* works.

PCRE 10.35 produced the same results, and from version 10.37 it differs. Unfortunately on 3v4l.org there isn't a PHP version with 10.36.

Results can be seen here: https://3v4l.org/189jh

[PhilipHazel]

There is no difference in 10.42, the current PCRE2 release, at least not using pcre2test.

[carenas]

There is no difference in 10.42, the current PCRE2 release,

the issue reported by @antman3351 is independent and was fixed in 4851890

[mvorisek]

Based on https://onlinephp.io/c/55ec2 it seems it is not fixed and the difference between JIT/non-JIT still remains, furthermore, the issue is in non-JITted version - #261 (comment) - and the link above fixes JIT.

[carenas]

it seems it is not fixed

correct, the issue you reported hasn't been fixed yet. I had updated my comment to better explain the context of it, apologies for the confusion.

% ./pcre2test -jit
PCRE2 version 10.43-DEV 2023-04-14 (8-bit)
  re> /(?<all>(?:(?:a(?&all))|(b))(c?))/
data> aabc
 0: aabc
 1: aabc
 2: <unset>
 3:
data> aabc\=no_jit
 0: aabc
 1: aabc
 2: b
 3: 

trying to understand the practical impact of it though, as it would seem that regardless of what the actual contents are in capture 2, it should be irrelevant (kind of undefined behaviour in C) to the data the user of the regex was looking for (capture 1 and 3 are identical, assuming 3 actually makes sense as well).

wouldn't the expression be as correct and probably more efficient written as?

(?<all>(?:(?>a(?&all))|b)c?)

[mvorisek]

trying to understand the practical impact of it though, as it would seem that regardless of what the actual contents are in capture 2, it should be irrelevant (kind of undefined behaviour in C) to the data the user of the regex was looking for (capture 1 and 3 are identical, assuming 3 actually makes sense as well).

wouldn't the expression be as correct and probably more efficient written as?

I discovered the problem on https://github.com/PHP-CS-Fixer/PHP-CS-Fixer/blob/v3.34.0/src/DocBlock/TypeExpression.php#L47 regex. In https://github.com/PHP-CS-Fixer/PHP-CS-Fixer/blob/v3.34.0/src/DocBlock/TypeExpression.php#L372 the 2nd condition would not be needed if the subroutine matches would not capture (current behaviour /w JIT) - the subroutine matches are simply very misleading in upper routine contexes.