Enable case-sensitive LeadingStrings with frequency-based heuristic#124736
Open
danmoseley wants to merge 11 commits intodotnet:mainfrom
Open
Enable case-sensitive LeadingStrings with frequency-based heuristic#124736danmoseley wants to merge 11 commits intodotnet:mainfrom
danmoseley wants to merge 11 commits intodotnet:mainfrom
Conversation
Member
Author
|
@MihuBot benchmark Regex |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enables case-sensitive multi-string prefix optimization for regex patterns by introducing a frequency-based heuristic to decide between SearchValues<string> (Teddy/Aho-Corasick) and IndexOfAny with character sets. Previously, case-sensitive prefixes were disabled due to regressions in patterns with low-frequency starting characters (e.g., uppercase letters, digits). The new heuristic uses empirical byte frequency data from Rust's aho-corasick crate to determine if starting characters are common enough in typical text to warrant multi-string search, or rare enough that IndexOfAny remains a better filter.
Changes:
- Uncommented and enabled case-sensitive prefix optimization with a frequency guard
- Added
HasHighFrequencyStartingCharsmethod to evaluate whether prefix starting characters are high-frequency (threshold >= 200) - Added
AsciiCharFrequencyRanktable containing the first 128 entries from BurntSushi's BYTE_FREQUENCIES data
stephentoub
reviewed
Feb 23, 2026
.../System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexFindOptimizations.cs
Outdated
Show resolved
Hide resolved
danmoseley
force-pushed
the
regex-redux/leading-strings-frequency
branch
from
fc2a7e2 to
eb39721
Compare
stephentoub
reviewed
Feb 23, 2026
.../System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexFindOptimizations.cs
Outdated
Show resolved
Hide resolved
stephentoub
reviewed
Feb 23, 2026
.../System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexFindOptimizations.cs
Outdated
Show resolved
Hide resolved
|
See benchmark results at https://gist.github.com/MihuBot/d9e8eb967e28c7cdfbcf0682a8b546b3 |
danmoseley
force-pushed
the
regex-redux/leading-strings-frequency
branch
from
eb39721 to
e877181
Compare
danmoseley
force-pushed
the
regex-redux/leading-strings-frequency
branch
2 times, most recently
from
da12aa4 to
3744268
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexFindOptimizations.cs:232
leadingStringsFrequencyuses -1 as the sentinel for "not computed / not applicable", but the subsequent checks use> 0. A valid computed frequency can be0(e.g., prefixes starting with '\x00' or other chars with 0 frequency in the table), which would incorrectly skip the LeadingStrings-vs-set comparison. Consider tracking availability viacaseSensitivePrefixes is not nulland checkingleadingStringsFrequency >= 0(or using a separatebool) so computed-0 still participates in the heuristic.
if (leadingStringsFrequency > 0)
{
bool preferLeadingStrings = true;
src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexFindOptimizations.cs:307
- Same sentinel issue as above: the fallback
if (leadingStringsFrequency > 0)will skip using computed case-sensitive prefixes if the computed value is0, potentially leavingFindModeasNoSearchwhen no other strategy is selected. Use an availability check likecaseSensitivePrefixes is not null && leadingStringsFrequency >= 0(or a dedicated flag) instead of> 0.
// If we have case-sensitive leading prefixes and nothing else was selected, use them.
if (leadingStringsFrequency > 0)
{
LeadingPrefixes = caseSensitivePrefixes!;
src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexPrefixAnalyzer.cs:1512
- The PR description says this change is based on Rust aho-corasick
BYTE_FREQUENCIESranks and a rank threshold (>= 200), but the implementation is using the existingRegexPrefixAnalyzer.Frequencytable of percentage occurrences (generated from runtime/Gutenberg text) and compares summed percentages. Either the description needs updating to reflect the actual heuristic/table used, or the code needs to align with the described rank-based approach.
/// <summary>Percent occurrences in source text (100 * char count / total count).</summary>
internal static ReadOnlySpan<float> Frequency =>
[
0.000f /* '\x00' */, 0.000f /* '\x01' */, 0.000f /* '\x02' */, 0.000f /* '\x03' */, 0.000f /* '\x04' */, 0.000f /* '\x05' */, 0.000f /* '\x06' */, 0.000f /* '\x07' */,
src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexFindOptimizations.cs:255
- This PR introduces new behavior for
RegexOptions.Compiled/NonBacktrackingwhere case-sensitiveLeadingStringsmay now be selected based on a frequency heuristic. There are existingRegexFindOptimizationsTests, but none appear to cover this new decision logic in compiled/NB modes; adding a few targeted test cases (e.g., where compiled should choose LeadingStrings vs LeadingSet depending on starter frequency, and a non-ASCII starter that must not engage the heuristic) would help prevent future regressions.
// Compute case-sensitive leading prefixes, but don't commit yet. We'll compare
// their starting-char frequency against the best FixedDistanceSet below to decide
// which strategy to use.
caseSensitivePrefixes = RegexPrefixAnalyzer.FindPrefixes(root, ignoreCase: false) is { Length: > 1 } csp ? csp : null;
leadingStringsFrequency = caseSensitivePrefixes is not null ? SumStartingCharFrequencies(caseSensitivePrefixes) : -1;
}
// Build up a list of all of the sets that are a fixed distance from the start of the expression.
List<FixedDistanceSet>? fixedDistanceSets = RegexPrefixAnalyzer.FindFixedDistanceSets(root, thorough: !interpreter);
Debug.Assert(fixedDistanceSets is null || fixedDistanceSets.Count != 0);
// See if we can make a string of at least two characters long out of those sets. We should have already caught
// one at the beginning of the pattern, but there may be one hiding at a non-zero fixed distance into the pattern.
if (fixedDistanceSets is not null &&
FindFixedDistanceString(fixedDistanceSets) is (string String, int Distance) bestFixedDistanceString)
{
FindMode = FindNextStartingPositionMode.FixedDistanceString_LeftToRight;
FixedDistanceLiteral = ('\0', bestFixedDistanceString.String, bestFixedDistanceString.Distance);
return;
}
// As a backup, see if we can find a literal after a leading atomic loop. That might be better than whatever sets we find, so
// we want to know whether we have one in our pocket before deciding whether to use a leading set (we'll prefer a leading
// set if it's something for which we can search efficiently).
(RegexNode LoopNode, (char Char, string? String, StringComparison StringComparison, char[]? Chars) Literal)? literalAfterLoop = RegexPrefixAnalyzer.FindLiteralFollowingLeadingLoop(root);
// If we got such sets, we'll likely use them. However, if the best of them is something that doesn't support an efficient
// search and we did successfully find a literal after an atomic loop we could search instead, we prefer the efficient search.
// For example, if we have a negated set, we will still prefer the literal-after-an-atomic-loop because negated sets typically
// contain _many_ characters (e.g. [^a] is everything but 'a') and are thus more likely to very quickly match, which means any
// vectorization employed is less likely to kick in and be worth the startup overhead.
if (fixedDistanceSets is not null)
{
// Sort the sets by "quality", such that whatever set is first is the one deemed most efficient to use.
// In some searches, we may use multiple sets, so we want the subsequent ones to also be the efficiency runners-up.
RegexPrefixAnalyzer.SortFixedDistanceSetsByQuality(fixedDistanceSets);
// If we have case-sensitive leading prefixes, compare the frequency of their starting characters
// against the best fixed-distance set's characters. If the best set isn't more selective than the
// starting chars (i.e. its frequency is at least as high), prefer LeadingStrings (SearchValues)
// which can match full multi-character prefixes simultaneously. Also prefer LeadingStrings when
// the best set is negated or range-based (no Chars), since those are weak filters.
if (leadingStringsFrequency > 0)
{
bool preferLeadingStrings = true;
if (fixedDistanceSets[0].Chars is { } bestSetChars &&
!fixedDistanceSets[0].Negated)
{
ReadOnlySpan<float> frequency = RegexPrefixAnalyzer.Frequency;
Debug.Assert(frequency.Length == 128);
float bestSetFrequency = 0;
foreach (char c in bestSetChars)
{
bestSetFrequency += c < frequency.Length ? frequency[c] : 0;
}
preferLeadingStrings = bestSetFrequency >= leadingStringsFrequency;
}
if (preferLeadingStrings)
{
LeadingPrefixes = caseSensitivePrefixes!;
FindMode = FindNextStartingPositionMode.LeadingStrings_LeftToRight;
#if SYSTEM_TEXT_REGULAREXPRESSIONS
LeadingStrings = SearchValues.Create(LeadingPrefixes, StringComparison.Ordinal);
#endif
return;
}
Member
Author
|
@MihuBot benchmark Regex |
Member
Author
|
I also ran regex tests locally to verify it's still good, so I think this ready for final (?) review. |
Member
Author
|
In my local runs, i get these wins. all others unchanged, no regressions
|
Member
Author
|
How would this affect C# (and F#) on the leaderboard at https://programming-language-benchmarks.vercel.app/problem/regex-redux? ( the original does not have a [GeneratedRegex] entry yet.) Currently the top .NET entry is 6th (AOT+generated). I didn't measure AOT, but assuming the ratio is the same as for jit, we'd get to 3rd on the list. Rust would still be over 2x faster and the main reason is very likely because its regex engine operates on UTF-8 and ours uses UTF-16 so there's just twice the bytes to process, meaning SIMD has to do more chunks,. .. |
|
See benchmark results at https://gist.github.com/MihuBot/75eeb6e2a31171fca2c8dfc5a605f245 |
Member
Author
|
OK fishy mihubot was good before, but second run now doesn't match my local good results. Let me see |
Member
Right, that's another thing to explore as part of the heuristic: not just sum or average of frequencies but e.g. the number of characters in the search set. |
Member
|
@MihuBot benchmark Regex https://github.com/MihaZupan/performance/tree/compiled-regex-only -medium -arm |
Member
Author
|
Good point, if I understand right: always use IndexOf for single char. You can't get faster than that and it indicidentally is less arch sensitive. For >1 char prefixes, still use the freq table, use IndexOfAny if predicted low freq and Teddy for high freq. I'll try this -- Dan |
Single-char IndexOf has much higher throughput than multi-string search (Teddy/Aho-Corasick), so never switch away from it regardless of character frequency. This fixes the Sherlock|Street regression where the heuristic was switching to LeadingStrings for a single 'S' char that barely exceeded the 0.6 threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
That seems to have fixed it. Locally
others unchanged. |
Member
Author
|
@MihuBot benchmark Regex |
Member
Author
|
@MihuBot benchmark Regex https://github.com/MihaZupan/performance/tree/compiled-regex-only -medium -arm |
Member
Author
I'll try making exactly 2 use IndexOfAny and measure that also |
Member
Author
|
A 2 char cutover eliminates the regex-redux win (as key pattern has a 2 char set). I think we should stick with the 1 char cutover in this PR. That gives us wins or parity on all scenarios, and in fact more of a win than the previous commit had. -- Dan |
Member
|
@MihuBot regexdiff |
|
733 out of 18857 patterns have generated source code changes. Examples of GeneratedRegex source diffs"\\b(in)\\b" (658 uses)[GeneratedRegex("\\b(in)\\b", RegexOptions.IgnoreCase | RegexOptions.Singleline)] // Any possible match is at least 2 characters.
if (pos <= inputSpan.Length - 2)
{
- // The pattern matches a character in the set [Nn] at index 1.
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 1; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_409072BF36F03A4496ACC585815833300ABA306360D979616ACDCED385DDC8FB);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i + 1).IndexOfAny('N', 'n');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- if (((span[i] | 0x20) == 'i'))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x03,
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_409072BF36F03A4496ACC585815833300ABA306360D979616ACDCED385DDC8FB = SearchValues.Create(["IN", "iN", "In", "in"], StringComparison.Ordinal);
}
}"([uú]ltim[oa])\\b" (548 uses)[GeneratedRegex("([uú]ltim[oa])\\b", RegexOptions.IgnoreCase | RegexOptions.Singleline)] private bool TryFindNextPossibleStartingPosition(ReadOnlySpan<char> inputSpan)
{
int pos = base.runtextpos;
- char ch;
// Any possible match is at least 6 characters.
if (pos <= inputSpan.Length - 6)
{
- // The pattern matches a character in the set [Mm] at index 4.
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 5; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_5F1D4359E5DF98DCF4B95FDCBFEF2A013E0C46941AEBD0349C6180A4A0372176);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i + 4).IndexOfAny('M', 'm');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- if (((ch = span[i]) < 128 ? ("\0\0\0\0\0 \0 "[ch >> 4] & (1 << (ch & 0xF))) != 0 : RegexRunner.CharInClass((char)ch, "\0\b\0UVuvÚÛúû")) &&
- ((span[i + 1] | 0x20) == 'l'))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x03,
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_5F1D4359E5DF98DCF4B95FDCBFEF2A013E0C46941AEBD0349C6180A4A0372176 = SearchValues.Create(["ULT", "uLT", "ÚLT", "úLT", "UlT", "ulT", "ÚlT", "úlT", "ULt", "uLt", "ÚLt", "úLt", "Ult", "ult", "Últ", "últ"], StringComparison.Ordinal);
}
}"^refs\\/heads\\/tags\\/(.*)|refs\\/heads\\/( ..." (391 uses)[GeneratedRegex("^refs\\/heads\\/tags\\/(.*)|refs\\/heads\\/(.*)|refs\\/tags\\/(.*)|refs\\/(.*)|origin\\/tags\\/(.*)|origin\\/(.*)$")] private bool TryFindNextPossibleStartingPosition(ReadOnlySpan<char> inputSpan)
{
int pos = base.runtextpos;
- char ch;
// Any possible match is at least 5 characters.
if (pos <= inputSpan.Length - 5)
{
- // The pattern matches a character in the set [/i] at index 4.
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 4; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_30C2953D90C322A099E43E6AF5B7857C062C1C20BFC2B2BFB4A15149DFB3AFEF);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i + 4).IndexOfAny('/', 'i');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- if ((((ch = span[i + 3]) == 'g') | (ch == 's')) &&
- (((ch = span[i + 2]) == 'f') | (ch == 'i')))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
/// <summary>Whether <see cref="s_defaultTimeout"/> is non-infinite.</summary>
internal static readonly bool s_hasTimeout = s_defaultTimeout != Regex.InfiniteMatchTimeout;
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_30C2953D90C322A099E43E6AF5B7857C062C1C20BFC2B2BFB4A15149DFB3AFEF = SearchValues.Create(["refs/heads/tags/", "refs/heads/", "refs/tags/", "refs/", "origin/tags/", "origin/"], StringComparison.Ordinal);
}
}"\\b(from).+(to)\\b.+" (316 uses)[GeneratedRegex("\\b(from).+(to)\\b.+", RegexOptions.IgnoreCase | RegexOptions.Singleline)] // Any possible match is at least 8 characters.
if (pos <= inputSpan.Length - 8)
{
- // The pattern matches a character in the set [Mm] at index 3.
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 7; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_DA0DF7757216159252C4FA00AB5982AAA4403D2C43304873401C53E36F92CA04);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i + 3).IndexOfAny('M', 'm');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- if (((span[i] | 0x20) == 'f') &&
- ((span[i + 2] | 0x20) == 'o'))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x03,
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_DA0DF7757216159252C4FA00AB5982AAA4403D2C43304873401C53E36F92CA04 = SearchValues.Create(["FROM", "fROM", "FrOM", "frOM", "FRoM", "fRoM", "FroM", "froM", "FROm", "fROm", "FrOm", "frOm", "FRom", "fRom", "From", "from"], StringComparison.Ordinal);
}
}"\\b(?<year>((1[5-9]|20)\\d{2})|2100)\\b" (309 uses)[GeneratedRegex("\\b(?<year>((1[5-9]|20)\\d{2})|2100)\\b", RegexOptions.IgnoreCase | RegexOptions.Singleline)] private bool TryFindNextPossibleStartingPosition(ReadOnlySpan<char> inputSpan)
{
int pos = base.runtextpos;
- char ch;
- uint charMinusLowUInt32;
// Any possible match is at least 4 characters.
if (pos <= inputSpan.Length - 4)
{
- // The pattern begins with a character in the set [12].
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 3; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_74309B1520D1FC139D75B2BB6987007481E9B777F41E19A028B21AB7FC28BA45);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i).IndexOfAny('1', '2');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- // The primary set being searched for was found. 2 more sets will be checked so as
- // to minimize the number of places TryMatchAtCurrentPosition is run unnecessarily.
- // Make sure they fit in the remainder of the input.
- if ((uint)(i + 2) >= (uint)span.Length)
- {
- goto NoMatchFound;
- }
-
- if (((int)((0xC7C00000U << (short)(charMinusLowUInt32 = (ushort)(span[i + 1] - '0'))) & (charMinusLowUInt32 - 32)) < 0) &&
- ((ch = span[i + 2]) < 128 ? char.IsAsciiDigit(ch) : RegexRunner.CharInClass((char)ch, "\0\u0002\u000101\t")))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x03,
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_74309B1520D1FC139D75B2BB6987007481E9B777F41E19A028B21AB7FC28BA45 = SearchValues.Create(["15", "16", "17", "18", "19", "20", "2100"], StringComparison.Ordinal);
}
}"\\b(et\\s*(le|la(s)?)?)\\b.+" (291 uses)[GeneratedRegex("\\b(et\\s*(le|la(s)?)?)\\b.+", RegexOptions.IgnoreCase | RegexOptions.Singleline)] // Any possible match is at least 3 characters.
if (pos <= inputSpan.Length - 3)
{
- // The pattern matches a character in the set [Tt] at index 1.
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 2; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_40190A5AE82B92C9577FE9A45CD09B22413116F9859390E6536F6EF2E5085EA1);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i + 1).IndexOfAny('T', 't');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- if (((span[i] | 0x20) == 'e'))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x03,
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_40190A5AE82B92C9577FE9A45CD09B22413116F9859390E6536F6EF2E5085EA1 = SearchValues.Create(["ET", "eT", "Et", "et"], StringComparison.Ordinal);
}
}"\\b(https?://|ftp://|www\\.)[\\w\\d\\._/\\-~ ..." (267 uses)[GeneratedRegex("\\b(https?://|ftp://|www\\.)[\\w\\d\\._/\\-~%@()+:?&=#!]*[\\w\\d/]")] private bool TryFindNextPossibleStartingPosition(ReadOnlySpan<char> inputSpan)
{
int pos = base.runtextpos;
- char ch;
// Any possible match is at least 5 characters.
if (pos <= inputSpan.Length - 5)
{
- // The pattern begins with a character in the set [fhw].
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 4; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_7E253910419F77A121C9FCE57096BB0770898849401D67ABDAF3E17D2F5F21FD);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i).IndexOfAny('f', 'h', 'w');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- // The primary set being searched for was found. 2 more sets will be checked so as
- // to minimize the number of places TryMatchAtCurrentPosition is run unnecessarily.
- // Make sure they fit in the remainder of the input.
- if ((uint)(i + 3) >= (uint)span.Length)
- {
- goto NoMatchFound;
- }
-
- if ((((ch = span[i + 3]) == '.') | (ch == ':') | (ch == 'p')) &&
- (((ch = span[i + 1]) == 't') | (ch == 'w')))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x03,
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_7E253910419F77A121C9FCE57096BB0770898849401D67ABDAF3E17D2F5F21FD = SearchValues.Create(["http", "ftp://", "www."], StringComparison.Ordinal);
}
}"\\b(?<unit>decennio?|ann[oi]|mes[ei]|settima ..." (255 uses)[GeneratedRegex("\\b(?<unit>decennio?|ann[oi]|mes[ei]|settiman[ae]|giorn[oi])\\b", RegexOptions.ExplicitCapture | RegexOptions.Singleline)] private bool TryFindNextPossibleStartingPosition(ReadOnlySpan<char> inputSpan)
{
int pos = base.runtextpos;
- char ch;
- uint charMinusLowUInt32;
// Any possible match is at least 4 characters.
if (pos <= inputSpan.Length - 4)
{
- // The pattern begins with a character in the set [adgms].
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 3; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_7F16DACEF575A8087BAF8FAC14BC47F4D2214D513F67C3250A958A2D52D6440A);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i).IndexOfAny(Utilities.s_ascii_92200800);
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- // The primary set being searched for was found. 2 more sets will be checked so as
- // to minimize the number of places TryMatchAtCurrentPosition is run unnecessarily.
- // Make sure they fit in the remainder of the input.
- if ((uint)(i + 2) >= (uint)span.Length)
- {
- goto NoMatchFound;
- }
-
- if ((((ch = span[i + 1]) == 'e') | (ch == 'i') | (ch == 'n')) &&
- ((int)((0x8018C000U << (short)(charMinusLowUInt32 = (ushort)(span[i + 2] - 'c'))) & (charMinusLowUInt32 - 32)) < 0))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
- /// <summary>Supports searching for characters in or not in "adgms".</summary>
- internal static readonly SearchValues<char> s_ascii_92200800 = SearchValues.Create("adgms");
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_7F16DACEF575A8087BAF8FAC14BC47F4D2214D513F67C3250A958A2D52D6440A = SearchValues.Create(["decenni", "anni", "anno", "mese", "mesi", "settiman", "giorni", "giorno"], StringComparison.Ordinal);
}
}"(quest['oa]|corrente)" (244 uses)[GeneratedRegex("(quest['oa]|corrente)", RegexOptions.ExplicitCapture | RegexOptions.Singleline)] private bool TryFindNextPossibleStartingPosition(ReadOnlySpan<char> inputSpan)
{
int pos = base.runtextpos;
- char ch;
// Any possible match is at least 6 characters.
if (pos <= inputSpan.Length - 6)
{
- // The pattern begins with a character in the set [cq].
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 5; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_763B6F2775AD886009FE306897080DA0D914A7D65E7AECC2B25D4344D9F8EF63);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i).IndexOfAny('c', 'q');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- // The primary set being searched for was found. 2 more sets will be checked so as
- // to minimize the number of places TryMatchAtCurrentPosition is run unnecessarily.
- // Make sure they fit in the remainder of the input.
- if ((uint)(i + 3) >= (uint)span.Length)
- {
- goto NoMatchFound;
- }
-
- if ((((ch = span[i + 1]) == 'o') | (ch == 'u')) &&
- char.IsBetween(span[i + 3], 'r', 's'))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
/// <summary>Whether <see cref="s_defaultTimeout"/> is non-infinite.</summary>
internal static readonly bool s_hasTimeout = s_defaultTimeout != Regex.InfiniteMatchTimeout;
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_763B6F2775AD886009FE306897080DA0D914A7D65E7AECC2B25D4344D9F8EF63 = SearchValues.Create(["quest'", "questa", "questo", "corrente"], StringComparison.Ordinal);
}
}"\\b(da(l(l[oae'])?|i|gli)?|tra|fra|entro)(\\ ..." (228 uses)[GeneratedRegex("\\b(da(l(l[oae'])?|i|gli)?|tra|fra|entro)(\\s+(il|l[aeo']|gli|i))?\\b", RegexOptions.ExplicitCapture | RegexOptions.Singleline)] private bool TryFindNextPossibleStartingPosition(ReadOnlySpan<char> inputSpan)
{
int pos = base.runtextpos;
- uint charMinusLowUInt32;
// Any possible match is at least 2 characters.
if (pos <= inputSpan.Length - 2)
{
- // The pattern matches a character in the set [anr] at index 1.
- // Find the next occurrence. If it can't be found, there's no match.
- ReadOnlySpan<char> span = inputSpan.Slice(pos);
- for (int i = 0; i < span.Length - 1; i++)
+ // The pattern has multiple strings that could begin the match. Search for any of them.
+ // If none can be found, there's no match.
+ int i = inputSpan.Slice(pos).IndexOfAny(Utilities.s_indexOfAnyStrings_Ordinal_99936EDA919D5A1A2F2418047924F8D2D31078642E1958EC7E8A32CF569E2511);
+ if (i >= 0)
{
- int indexOfPos = span.Slice(i + 1).IndexOfAny('a', 'n', 'r');
- if (indexOfPos < 0)
- {
- goto NoMatchFound;
- }
- i += indexOfPos;
-
- if (((int)((0xE0008000U << (short)(charMinusLowUInt32 = (ushort)(span[i] - 'd'))) & (charMinusLowUInt32 - 32)) < 0))
- {
- base.runtextpos = pos + i;
- return true;
- }
+ base.runtextpos = pos + i;
+ return true;
}
}
// No match found.
- NoMatchFound:
base.runtextpos = inputSpan.Length;
return false;
}
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x03,
0xFE, 0xFF, 0xFF, 0x87, 0xFE, 0xFF, 0xFF, 0x07
};
+
+ /// <summary>Supports searching for the specified strings.</summary>
+ internal static readonly SearchValues<string> s_indexOfAnyStrings_Ordinal_99936EDA919D5A1A2F2418047924F8D2D31078642E1958EC7E8A32CF569E2511 = SearchValues.Create(["da", "tra", "fra", "entro"], StringComparison.Ordinal);
}
}For more diff examples, see https://gist.github.com/MihuBot/e53dc1d081be99a109ec2d2700619645 JIT assembly changesFor a list of JIT diff regressions, see Regressions.md Sample source code for further analysisconst string JsonPath = "RegexResults-1788.json";
if (!File.Exists(JsonPath))
{
await using var archiveStream = await new HttpClient().GetStreamAsync("https://mihubot.xyz/r/FHqnss4");
using var archive = new ZipArchive(archiveStream, ZipArchiveMode.Read);
archive.Entries.First(e => e.Name == "Results.json").ExtractToFile(JsonPath);
}
using FileStream jsonFileStream = File.OpenRead(JsonPath);
RegexEntry[] entries = JsonSerializer.Deserialize<RegexEntry[]>(jsonFileStream, new JsonSerializerOptions { IncludeFields = true })!;
Console.WriteLine($"Working with {entries.Length} patterns");
record KnownPattern(string Pattern, RegexOptions Options, int Count);
sealed class RegexEntry
{
public required KnownPattern Regex { get; set; }
public required string MainSource { get; set; }
public required string PrSource { get; set; }
public string? FullDiff { get; set; }
public string? ShortDiff { get; set; }
public (string Name, string Values)[]? SearchValuesOfChar { get; set; }
public (string[] Values, StringComparison ComparisonType)[]? SearchValuesOfString { get; set; }
} |
|
See benchmark results at https://gist.github.com/MihuBot/2bd6f75ed057df7b8fdd4b847f57e444 |
Member
|
Some of the diffs are confusing to me, like why this one wasn't previously already handled by the case-insensitive support: |
|
See benchmark results at https://gist.github.com/MihuBot/709e349b42165b2fbd1de51500f56381 |
Member
Author
|
@MihuBot benchmark Regex |
Member
Author
|
(I think we didn't get mihubot on x64 on the last commit, I'll do this to compare with ARM64 |
Member
Author
|
Below is the AI analysis of the code gen issue. which is correct as far as I can tell. Question is whether we should explore adding the proposed "fix" to this PR. It seems like this code gen diff should itself be an improvement, albeit inadvertent. These are not patterns from the benchmarks though, so I'll measure one locally. I spot checked a bunch of diffs and they all have this pattern, btw, so seems like just this one explanation. === For There are two pre-existing gaps that cause this pattern to fall through: Gap 1:
Gap 2:
The How the PR "rescues" it The PR's new code (line 218-220) calls Stephen's point This works but is suboptimal — ideally Gap 1 should be fixed so The fix would be a one-line addition around else if (child.Kind is RegexNodeKind.Capture)
{
// Descend into capture group to find the string inside
// (similar to how FindPrefixesCore handles Capture)
}...though that would require restructuring since the method iterates children flat rather than recursing. |
Member
Author
|
Looks like for this example pattern it's same/an improvement (of course depends on the particular text). I guess searching for
I think I need guidance on whether I should pursue fixing this in this PR. Either way, we'd have a diff: it would just be a code improvement to code not changed in this PR. -- Dan |
Member
Author
|
@MihuBot benchmark Regex https://github.com/MihaZupan/performance/tree/compiled-regex-only -medium -arm |
|
See benchmark results at https://gist.github.com/MihuBot/1bd2835a353c7b0b1f70f8caef9848e8 |
|
See benchmark results at https://gist.github.com/MihuBot/8d26ffeae5729831b9bbdcbf1fc3c316 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I thought it would be interesting to see whether AI could take another look at the commented out search strategy originally introduced by @stephentoub in #98791 to see whether we can enable it and keep the wins without the regressions that caused it to be commented out.
AI tried various experiments, and got to a dead end. I recalled the frequency table approach that Ripgrep uses (credit to @BurntSushi). Turns out that fixes the regressions entirely. This means our engine now has assumptions built in about char frequencies in ASCII (only) text. That's an approach that's been proven in ripgrep, one of the fastest engines, for 10 years, and it turns out to work OK for regex-redux as well because a, c, g, t are relatively high frequency in English anyway. Code unchanged if pattern has anything other than ASCII (see benchmark results below).
This gives us a nice win on regex-redux, a few other wins in existing tests, and no regressions.
Note: a char frequency table already existed in
RegexPrefixAnalyzer.csfor ranking which fixed-distance character sets are most selective. Our new table serves a different purpose: deciding whether to useLeadingStringsvsFixedDistanceSetsat all. The two are complementary.====
When a regex has multiple alternation prefixes (e.g.
a|b|c|...), this change decides whether to useSearchValues<string>(Teddy/Aho-Corasick) or fall through toFixedDistanceSets(IndexOfAny) based on the frequency of the starting characters.High-frequency starters (common letters like lowercase vowels) benefit from multi-string search; low-frequency starters (uppercase, digits, rare consonants) are already excellent
IndexOfAnyfilters. Non-ASCII starters bail out (no frequency data), preserving baseline behavior.Benchmark results (444 benchmarks, BDN A/B with --statisticalTest 3ms)
Leipzig win is because the pattern is
Tom.{10,25}river|river.{10,25}Tomso there is a short prefix that is common in the text; with this change it noticesris common andTfairly common in English text, so it switches toSearchValueswhich looks forTomandriversimultaneously, causing far fewer false starts.regex-redux win is because it was previously looking for short, very common prefixes naively, and now (specifically because the pattern chars are common) it changed to use
SearchValues(ie Aho-Corasick/Teddy) to search for the longer strings simultaneously.No regressions detected. All MannWhitney tests report Same for non-improved benchmarks.
Key design decisions
BYTE_FREQUENCIESfrom @BurntSushi's aho-corasick crateLeadingStrings; below 200 falls through toFixedDistanceSetsCompanion benchmarks: dotnet/performance#5126
New benchmark results (not yet in dotnet/performance, won't be picked up by PR bot)
These benchmarks are from the companion PR dotnet/performance#5126.
Binary didn't regress even though it's ASCII with non English frequencies because the pattern has chars that are not particularly common in English, so it uses the old codepath. It's hard to hypothesize about what one might search for in a binary file; searching for a bunch of leading lower case ASCII chars might regress somewhat. We've never particularly tested on binary before, and I don't recall seeing any bugs mentioning binary, so I don't think this is particularly interesting.
NonASCII didn't regress since as previously mentioned, the non ASCII leading chars in the pattern (presumably likely for any searching of non ASCII text) causes it to choose the existing codepath.
All MannWhitney tests report Same -- no regressions on binary or non-ASCII input.