Repository navigation
fix(security): replace backtracking regex in ParametersUtils with a linear scanner (Fixes #1638) - #1654
Conversation
…inear scanner
The pattern used to find ${...} expressions relied on nested lookaheads and
backreferences. It is applied to every string task parameter, including task
output returned by workers, and its matching time grows polynomially with
the nesting depth: ${${...}} nested 1000 deep (~3 KB) takes over 30 seconds.
Find the expressions with a single pass that tracks the brace depth instead.
The semantics are unchanged, including $${ escaping, nested expressions and
an unclosed expression hiding the rest of the string; a test compares the
scanner with the previous pattern on every string of a small alphabet up to
length 6.
Fixes conductor-oss#1638
There was a problem hiding this comment.
Approving. Thanks for the thorough writeup and testing.
This should also result in a small performance boost for most scenarios -- adversarial was polynomial (now O(n)), and benign lookups have now improved from worse-than-linear to strictly O(n) as well.
One thing worth noting, now there's an extreme-depth stack limit that will throw StackOverflowError -- which is a strict upgrade over just burning a core to the ground :) but just thought it was worth noting here as an acknowledgement of behavior change that COULD be noticed.
|
Thanks for the review! Good call on the StackOverflowError: the nested-expression substitution recurses once per nesting level, so at those depths it now fails fast instead of hanging in the regex. Happy to add an explicit depth limit in a follow-up if you think it's worth it. |
Pull Request type
NOTE: Please remember to run
./gradlew spotlessApplyto fix any format violations.Changes in this PR
ParametersUtils.PATTERNfound${...}expressions with nested lookaheads and backreferences, and it is applied to every string task parameter, including output returned by workers. Its matching time grows polynomially with the nesting depth: on my machine${${...}}nested 500 deep takes ~2.4 s and nested 1000 deep (a ~3 KB string) ~37 s, per parameter.This PR removes the pattern and finds the expressions with a single pass over the string (
ParametersUtils.findExpressions): an expression starts at a${that is not preceded by$and ends at its matching}, tracked with a brace-depth counter.replaceVariablesiterates over the resulting ranges instead of regex matches; the substitution logic itself is unchanged.Semantics are preserved, including
$${escaping, nested expressions being part of the outer range, and the existing behaviour that an unclosed${hides everything after it. To check this I ran the old pattern and the scanner over every string up to length 9 over the alphabet$ { } a, plus runs including newlines and brace-only strings up to length 12 (~2.6M inputs in total) with identical results.Tests added to
ParametersUtilsTest:testFindExpressions: expected ranges for representative inputs.testFindExpressionsMatchesLegacyPattern: differential test against the previous pattern (kept in the test only, as the reference for its semantics) on every string up to length 6 over$ { } a \n.testReplaceIsNotVulnerableToReDoS: adversarial inputs throughreplace()(depth-1000 nesting, 100k unclosed braces, 50k expressions, 100k closing braces) under a 10 s timeout. It does not complete with the previous pattern../gradlew :conductor-core:testpasses locally (972 tests),spotlessApplyapplied.Issue #1638
Alternatives considered
${...}with nesting is not a regular language, so a regex cannot match balanced braces without lookaheads/backreferences of the kind that caused the problem; only a depth-limited approximation would be possible, which would change behaviour for deeper nesting.