fix(#5886): bound SQL MATCHES and openCypher =~ against catastrophic regex backtracking - #5983
Conversation
…regex backtracking
java.util.regex backtracking never polls an interrupt flag or checks a deadline, so a
pathological pattern (e.g. (.*a){20}$) keeps its worker thread busy for as long as the
backtracking takes, and arcadedb.command.timeout cannot stop it. Both regex entry points now
run through TimeBoundRegex, which wraps the matched input so charAt() throws once a deadline
elapses, the one call site java.util.regex reaches on every backtracking step. The deadline is
a new dedicated setting, arcadedb.command.regexTimeout (default 1000ms), active independently
of and even when arcadedb.command.timeout is left at its default of disabled.
SQL LIKE/ILIKE were audited too: QueryHelper.convertForRegExp escapes every metacharacter
except % and ?, so the generated pattern can never contain the grouping/alternation/nested
quantifiers catastrophic backtracking requires, and is left unchanged.
|
Tick the box to add this pull request to the merge queue (same as
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 13 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Review: bound SQL
|
…equence() Code review on #5983 pointed out that each subSequence()-derived DeadlineBoundCharSequence started its own charAt() call counter, so the every-256-calls check cadence restarted per sub-sequence. java.util.regex's own backtracking indexes with charAt()/length() and does not currently go through subSequence() for matches(), so this was not reachable today, but sharing one counter by reference across a sequence and everything derived from it via subSequence() removes the assumption instead of just documenting it.
|
Thanks for the review. Responses to each point: 1. Default 2. Per-row config lookup. Leaving as-is: it's a 3. Allocation/indirection overhead. Ran an informal check: 200k matches/round, 20 rounds, warmed up, literal-ish pattern ( 4. Fixed in 16af3ef — 5. Test timing margin. Agreed low-risk as noted; keeping the 5s margin on a 200ms deadline for CI-runner headroom. |
|
Decision on point 1: keeping the 1000ms default. Regex matches are normally sub-millisecond even on large text, legitimate patterns needing more time are rare, and it's a per-database setting anyone hitting the edge case can raise. A tight default closes the security gap immediately for anyone running with |
|
Review: fix(#5886) bound SQL Nice, focused fix. The core idea, wrap the matched Findings
Things that look solid
Overall this is a well-reasoned, well-tested fix for a real DoS vector. The main thing I'd want addressed or explicitly acknowledged before merge is whether the multi-value MATCHES per-item timeout budget (finding 1) is intentional or should be tightened to an aggregate deadline. |
…t from the right config Two issues from code review on #5983: 1. A multi-value MATCHES (a WHERE against a list/array-typed property) called TimeBoundRegex.matches() once per item, each starting its own fresh timeoutMillis budget, so N catastrophic items could still tie up the thread for N * regexTimeout instead of one bounded evaluation. TimeBoundRegex now exposes newDeadline()/matchesUntil() so MatchesCondition computes one deadline before the loop and shares it across every item. 2. Writing that regression test surfaced a real bug: both MatchesCondition and RegexExpression read the timeout via context.getConfiguration(), which for a plain SELECT/=~ evaluation is a fresh, disconnected ContextConfiguration - nothing wires it to the database's settings, so it silently fell back to the compiled-in 1000ms default and ignored any per-database arcadedb.command.regexTimeout override. The existing arcadedb.command.timeout handling in SelectExecutionPlanner uses context.getDatabase().getConfiguration() for exactly this reason; both regex entry points now do the same.
|
Pushed fc07cb1 addressing both points from the second review: Finding 1 (multi-value per-item timeout budget): fixed as suggested. Writing the regression test for that surfaced a more serious, separate bug: both Net effect: the fix was still correct and on by default (the compiled-in 1000ms default was always being applied), but a database operator lowering or raising All tests green: |
ReviewSolid fix for a real DoS vector, and the iteration history in the commits (sharing the deadline across Correctness
Worth considering
Test coverageGood coverage of the actual bug (issue reproducer at both the Minor
No security or compilation concerns found beyond the performance/default-value considerations above. |
…ole evaluation Code review on #5983 asked for this to be explicit rather than just implied by "bounded": a catastrophic item in a list-typed property fails the query with TimeoutException instead of silently excluding just that item, since swallowing it would let the pattern keep consuming its full budget on every future scan undetected.
|
Thanks — responses to the remaining points, none of which needed a code change beyond one doc line: Per-match overhead on the hot path. Already measured this for the earlier review (200k matches/round, 20 rounds, warmed up, literal-ish pattern): plain Default 1000ms being a behavior change on upgrade. Decision already made and recorded above: keeping 1000ms. Regex matches are normally sub-millisecond even on large text; the setting is per-database and easy to raise for the rare legitimate case that needs more. Whole-query abort vs. per-row on multi-value timeout. Agreed this deserved to be explicit rather than implied — added a paragraph to the release notes (dfc7817) confirming it's intentional: a catastrophic item fails the query with Test gap: no test proving a normal match is unaffected by a pattern that exercises At this point I believe all findings from both review passes are either fixed or addressed, and nothing outstanding is blocking. Let me know if anything above should be revisited. |
ReviewSolid fix for a real ReDoS gap, and the Security - the stated audit missed two other user-reachable regex sitesThe PR says "LIKE/ILIKE were audited too... left unchanged," which is correct for those, but the same catastrophic-backtracking class the PR fixes for
Possible bug - new NPE surface via
|
…d, plus polish Code review on #5983 found two more user-reachable catastrophic-backtracking sites the original audit missed, both now routed through TimeBoundRegex like MATCHES/=~: - text.regexReplace(string, regex, replacement) SQL function: its only defense was a 500-char pattern-length cap plus catch(StackOverflowError), neither of which bounds a time-based (not stack-depth-based) catastrophic match. TimeBoundRegex gains a replaceAll() counterpart to matches() for this. - Full-text search's Lucene RegexpQuery support (the /pattern/ query syntax) in FullTextQueryExecutor#collectRegexpMatches: a pathological pattern is matched against every token in what is, absent a literal prefix, a full index scan - arguably a worse instance of the same bug. LSMTreeFullTextIndex gains a getDatabase() accessor (mirroring the internal underlyingIndex.getMutableIndex().getDatabase() idiom already used elsewhere in that class) so the executor can read arcadedb.command.regexTimeout. Also from that review: TimeBoundRegex now catches StackOverflowError alongside a deadline trip - a sufficiently pathological pattern can blow the (recursive) backtracking stack before the next 256-call checkpoint, and that failure mode gets the same TimeoutException as an explicit deadline. The nanoTime() comparison switched to the overflow-safe form System.nanoTime()'s own Javadoc recommends. TextRegexReplace falls back to a plain ContextConfiguration when context/context.getDatabase() is unset, since (unlike a WHERE-clause condition, always evaluated with a database bound to the context) this SQL function is a lower-level entry point tests in this codebase already call directly with a null context. Widened the multi-value MATCHES test's timing margin (5 items/800ms -> 10 items/1000ms) per the review's CI-flakiness concern, without losing its ability to catch a regression to the old N * regexTimeout behavior.
|
Pushed 691bc73 addressing everything from this review: Security - two missed entry points (the big one): fixed.
Agreed this was arguably worse than the two originally-fixed sites (RegexpQuery runs the pathological pattern against every token in a full scan) and appreciate the catch — the original audit only looked at the SQL/Cypher condition operators, not SQL functions or search-query engines. Possible NPE via
Test coverage gap (multi-value margin too tight): fixed. Widened from 5 items/800ms to 10 items/1000ms — same discriminating power (expected ~200-300ms if shared, ~2000ms if not) with a wider absolute margin on both sides for CI-runner slack. Performance / default value: no new action, already addressed in earlier rounds (measured negligible overhead, decision made to keep the 1000ms default). New tests: |
ReviewNice fix overall - wrapping the Bug: sibling wildcard-query path in the same file is still unbounded
// engine/src/main/java/com/arcadedb/index/fulltext/FullTextQueryExecutor.java:625 and :635
return regex.matcher(token).matches();
Two things make this concretely reachable, not just theoretical:
Suggest wrapping both Minor: null-database handling is inconsistent between entry points
Nits
Test coverageGood regression coverage for the entry points that were fixed ( I reviewed statically (couldn't execute |
…d the wildcard-query sibling Code review on #5983 found FullTextQueryExecutor#collectWildcardMatches (the WildcardQuery sibling of the RegexpQuery path fixed last commit) still unbounded, and pointed out that wildcardToRegex()'s '*' -> '.*' translation reproduces catastrophic backtracking on its own, without needing grouping/alternation/nested quantifiers - a sequence of several .* segments is enough (the classic ReDoS pattern class this is: catastrophic backtracking has more than one shape, and the original audit only ruled out one of them). That directly contradicts this PR's own earlier conclusion that SQL LIKE/ILIKE were safe: QueryHelper.convertForRegExp produces the exact same '.*'-sequence shape for '%'. Verified empirically (not just theoretically) that '%a%a%a%a%a%a%a%a%a%a%a%a%a%a%a%a%a%a%a%a%c' against a string of plain 'a's hangs exactly like the issue's own (.*a){20}$ reproducer. Fixed both: - QueryHelper.like()/likeUntil() now route through TimeBoundRegex, mirroring matches()/ matchesUntil(). LikeOperator, ILikeOperator, and the native query engine's SelectOperator (a third, independent LIKE/ILIKE evaluation path) all updated, including the same shared-deadline treatment LikeOperator/ILikeOperator already needed for their multi-value (list-typed left operand) case. - FullTextQueryExecutor#collectWildcardMatches now shares one deadline across its full index scan, same as collectRegexpMatches. LikeOperator/ILikeOperator needed the same null-database fallback TextRegexReplace already had: LikeOperatorTest/ILikeOperatorTest call op.execute(null, ...) directly as an established convention, which the previous, unguarded context.getDatabase() call in this same PR would have NPE'd on. New tests: QueryHelperLikeRegexTimeoutTest (utility-level like()/likeUntil() plus SQL-level LIKE and ILIKE regression tests), FullTextQueryExecutorTest#catastrophicWildcardQueryIsAborted ByRegexTimeout. Reran LikeOperatorTest/ILikeOperatorTest (the null-database convention), QueryTest, SelectStatementExecutionTest, SelectOperatorsExtendedTest, SelectExecutionTest - all green.
|
Pushed 87f33fc. This round found something worth flagging explicitly: the wildcard-query finding didn't just mean fixing a sibling method - it invalidated this PR's own earlier "LIKE/ILIKE are structurally safe" conclusion. Bug: sibling wildcard-query path unbounded — fixed, and it corrects an earlier mistake of mine. Fixed all of it: Caught my own regression before it shipped: New tests: At this point every entry point named across four review rounds - |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 17 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Review: bound SQL
|
…review points Code review on #5983: 1. TimeBoundRegex.newDeadline() computed System.nanoTime() + (timeoutMillis * 1_000_000L) with no overflow guard. arcadedb.command.regexTimeout is admin-configurable, and an oversized value overflowing that arithmetic could wrap into a deadline already in the past - making every match abort immediately instead of applying the requested (very long) timeout, the opposite of the intended effect for a large value. Now uses Math.addExact/multiplyExact and falls back to the disabled (Long.MAX_VALUE) sentinel on overflow, which is the safe direction for a value that was clearly meant to mean "a lot of time" rather than "none at all." 2. MatchesCondition/RegexExpression call context.getDatabase().getConfiguration() unguarded, unlike LikeOperator/ILikeOperator/TextRegexReplace's null-safe fallback. Investigated in an earlier round and confirmed safe (both are only ever constructed by the parser and evaluated with a database already bound to the context - no direct-unit-test entry point the way there is for a standalone operator/function); added a comment explaining why instead of an unneeded guard, per the review's own suggestion. 3. TimeBoundRegex's class Javadoc now states plainly that both matches() and replaceAll() were verified empirically (not just by reading JDK source) to route every char access through charAt() for the literal-heavy patterns this PR's own tests exercise, since some regex fast paths read a pattern's literal prefix a different way and would silently escape the bound if they did. New test: newDeadlineDoesNotOverflowOnAnOversizedTimeout.
|
Pushed 9adab87 addressing all four points: 1. Inconsistent null-database fallback: investigated (already done in an earlier round) and documented rather than guarded. Confirmed 2. Deadline overflow: fixed, and it's a real one. 3. GC pressure: no new action — already measured at ~30ns/match (noise-level) in an earlier round, consistent with your own assessment that it's "probably negligible relative to 4. Boyer-Moore/fast-path concern: documented. You're right that this PR's own tests already provide decent evidence (the literal-heavy All tests green: |
Review: fix(#5886) bound SQL MATCHES / openCypher =~ against catastrophic regex backtrackingOverall this is a well-executed, carefully-scoped security fix. The core insight ( Possible gap: PromQL label matchers look like the same bug, unfixed
That's exactly the shape of gap this PR's own "3rd review pass" found in SQL LIKE: a sequence of unparenthesized quantified segments (e.g. Minor points
Test coverageGood - the new tests assert both correctness (normal matches still work) and the abort-near-deadline behavior with generous timing margins to avoid CI flakiness, and the multi-value budget-sharing regression test (10 items within ~1s instead of ~2s) is a meaningful regression guard, not just a smoke test. Nothing here blocks merging as a fix for the originally reported issue - the PromQL point is the only thing I'd want a maintainer opinion on before calling the ReDoS surface fully closed. |
… disabled-timeout stack overflow Code review on #5983, 5th round: PromQLEvaluator#matchesPostFilters (the =~/!~ label matcher for TimeSeries PromQL queries) called Pattern.matcher(strVal).matches() directly, guarded only by a static REDOS_CHECK that flags parenthesized nested-quantifier/alternation shapes ((a+)+, (a|aa)+). A sequence of unparenthesized quantified segments (a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*ac, PromQL's equivalent of the LIKE/wildcard gap fixed last commit) has no '(' for that check to catch and reaches the unbounded match uncaught - verified empirically before touching anything, same as the LIKE case. Routed both RE/NRE match calls through TimeBoundRegex.matchesUntil, sharing one deadline across the whole row scan a label matcher runs against (matchesPostFilters runs once per row), not a fresh one per row. Also from that round: TimeBoundRegex.run() converted a StackOverflowError to a TimeoutException unconditionally, even when the bound was explicitly disabled (regexTimeout <= 0), producing a message blaming arcadedb.command.regexTimeout for a failure unrelated to it (a raw JDK regex recursion-depth issue). Now only wraps it when the bound is actually active; when disabled it propagates as itself. New test: PromQLEvaluatorIntegrationTest#unparenthesizedCatastrophicPatternIsAbortedByRegexTimeout, verified against the exact reproducer shape (confirmed catastrophic and paren-free before writing the test, same discipline as the LIKE fix).
|
Pushed c9d816b addressing everything from this round: PromQL Misleading message on disabled timeout + stack overflow: fixed. Possible silent fallback via Pattern compilation per call in New test: At this point every regex-based query surface I can find in the codebase — |
|
Reviewed the diff (docs + 1. Inconsistent null-safety for 2. Shared deadline over a full scan makes the 1000ms default fairly aggressive for bulk paths (moderate/design) 3. Test coverage gaps (minor)
4. Minor duplication (nit) Things that look right:
Nice find on the follow-on entry points (full-text |
… review round 1. MatchesCondition/RegexExpression now fall back to a plain ContextConfiguration when context.getDatabase() is null, matching LikeOperator/ILikeOperator/TextRegexReplace's existing pattern instead of being the only unguarded call sites in this PR. Investigated twice now (rounds 5 and 6) and confirmed not reachable in practice, but the review's own point stands: the guard costs nothing, so relying solely on that audit holding forever isn't worth the asymmetry with every other modified call site. 2. QueryHelper.like() now delegates to likeUntil() instead of duplicating the null/empty guard and pattern-compile logic. 3. Documented the sharper edge of the upgrade-behavior tradeoff explicitly: full-text RegexpQuery/WildcardQuery and PromQL =~/!~ share one regexTimeout budget across an entire scan, so a large *non-catastrophic* scan that legitimately takes >1s purely because there's a lot of data now fails instead of completing slowly - recommend raising arcadedb.command.regexTimeout for such workloads. New tests closing the two coverage gaps the review found: - SelectExecutionTest#catastrophicLikeIsAbortedByRegexTimeout: the native query engine's SelectOperator.like/ilike (a third, independent LIKE/ILIKE path) had no dedicated regression test proving the abort, unlike every other modified entry point. - QueryHelperLikeRegexTimeoutTest#multiValueLikeSharesOneTimeoutBudgetAcrossItems: mirrors MatchesConditionTest's multi-value test for LikeOperator/ILikeOperator's BY-ITEM branch, which had shared-deadline logic but no test proving it (only the single-value LIKE/ILIKE paths were covered before).
|
Pushed bf3bfdb addressing all four points: 1. Inconsistent null-safety: fixed. Agreed on the second ask — 2. Shared-deadline-over-full-scan + 1000ms default is aggressive for bulk paths: documented explicitly. Added a standalone paragraph to the release notes calling this out as its own tradeoff (distinct from the earlier "any single match >1s now fails" note) — a legitimate, non-catastrophic full-text or PromQL scan over a lot of data can now fail on 3. Test coverage gaps: both closed.
4. All tests green: |
|
Reviewed the diff (didn't build locally - One finding I think is worth resolving before merge, plus a couple of smaller points. Potential issue:
|
Review: ReDoS protection for MATCHES/
|
…as documentation 10th review round came back with no correctness bugs or security gaps - three minor, non-blocking points, all addressed: - SelectOperator's LIKE/ILIKE bypassed the new GlobalConfiguration.getValueAsLong(Database) helper, calling record.getDatabase().getConfiguration().getValueAsLong(...) directly instead. Functionally identical (record.getDatabase() is never null there), but the helper's whole purpose is centralizing this lookup so call sites can't drift apart - switched to it for consistency with every other entry point. - LIKE/ILIKE's residual per-row (not per-scan) budget and TimeBoundRegex's StackOverflowError handling were both re-confirmed as already-documented, conscious trade-offs from earlier rounds - no new action. - The release notes had accumulated ten rounds of "Nth review pass found..." narration, useful during development but not what a published release note should read like. Rewritten as a single, direct description of the final state: what's covered, how deadline-sharing works, the one documented exception (LIKE/ILIKE per-row budget), and the upgrade-behavior tradeoff - without narrating the review process that got there. Reran SelectExecutionTest (25) and SelectOperatorsExtendedTest (12) after the SelectOperator change - all green.
|
Pushed 4cab15f. This round came back clean — "No correctness bugs or security gaps found" — with three minor, non-blocking points, all addressed: 1. 2. LIKE/ILIKE residual per-row budget & broad 3. Release notes reading as an internal changelog: rewritten. Fair nit — ten review rounds had left the notes narrating "a 6th review pass found...", "a 7th review pass caught...", which is exactly the kind of process detail that belongs in this PR thread, not in what ships as user-facing documentation. Rewrote the whole section as a single, direct description of the final state: what's covered, how the deadline-sharing design works (including the one documented exception for All tests green: Ten review rounds, two of which caught genuine, independently-verified bugs in the fix itself (not just missed entry points), and this one found none. I think this is a solid stopping point unless something new surfaces. |
Review: ReDoS hardening for regex-based query/validation surfaces (#5886)This is a well-scoped, thoroughly-tested fix for a real DoS vector (CWE-1333, catastrophic regex backtracking). The design is sound: Code quality
Potential issues / follow-ups
PerformanceThe zero-cost-when-disabled design ( SecuritySolid improvement. Confirms Test coverageExcellent - Overall: nice, careful piece of security engineering. The open items above are refinements/follow-ups, not blockers. |
… choice 11th review round (came back with no blockers) asked for a doc note explaining why SQLMethodNormalize lets TimeoutException propagate raw while TextRegexReplace rewraps it to IllegalArgumentException. Added: TextRegexReplace rewraps to preserve a contract that pre-dates this issue (it already converted PatternSyntaxException the same way); .normalize() had no such prior contract, so letting TimeoutException through as-is is actually the more consistent choice - it's the shape MatchesCondition/RegexExpression/LIKE/full-text/PromQL all surface.
|
Pushed 989392f. Thanks for the thorough final pass — this is a great summary to close on. 1. Inconsistent exception contract ( 2. 3. Upgrade-behavior callout prominence: looked at the document structure — there's no separate top-level "highlights" list this section reports into; each 4. All tests green ( This round confirmed no bugs and no blockers — just refinements. Across 11 rounds: 2 real, independently-verified bugs in the fix itself (a PR: #5983 |
ReviewThis is a thorough, well-tested fix for a real ReDoS gap ( 1.
|
…the 12th review round The 12th review round found the exact "shared deadline across a scan/query, not per item" principle this issue establishes elsewhere had been missed at three more sites: 1. DocumentValidator.validateField() gave each REGEXP-constrained property on a document its own fresh regexTimeout budget - a document with N such properties, each crafted to backtrack catastrophically, could cost up to N * regexTimeout instead of one bounded validation. This is the entry point the PR itself calls out as most exposed (no query privileges needed), so the gap mattered more here than anywhere else. validate() now computes one deadline and threads it through validateField() via a new overload; the existing 2-arg validateField() (no evidence of external callers, but public API) keeps computing its own per-call deadline for standalone single-field use. 2. PromQLEvaluator.evaluateRange() calls evaluate() once per step (up to MAX_RANGE_STEPS = 1,000,000), and each step's evaluateVectorSelector()/evaluateMatrixSelector() computed its own fresh regexDeadline - correctly shared across rows within one step, but not across steps. At the 1000ms default that's a theoretical ~11.5 days worst case for a crafted =~/!~ range query. Fixed by caching the deadline as a PromQLEvaluator instance field, computed once lazily - safe because a fresh evaluator is created per top-level query (see SQLFunctionPromQL), never reused across executions the way RegexExpression's AST nodes are cached by CypherStatementCache (that distinction is exactly what made an instance field wrong there and right here). 3. TextRegexReplace and SQLMethodNormalize's pattern argument both receive a CommandContext but recomputed a fresh deadline on every call instead of using it to share one across a query's rows, unlike MatchesCondition/RegexExpression which use the same context for exactly this. TimeBoundRegex gained replaceAllUntil() (the deadline-based counterpart to replaceAll(), mirroring matches()/matchesUntil()) so both functions could adopt the same context.getCachedValue()/setCachedValue() pattern. Also, per the same round's minor note: MatchesCondition's deadline cache key changed from "5886_MATCHES_CONTEXT_DEADLINE" to "__MATCHES_DEADLINE__" - still provably outside the "MATCHES_" pattern-cache namespace, but without embedding an issue number a future reader would have to look up to understand. New tests: DocumentValidationTest#regExpValidationSharesOneTimeoutBudgetAcrossAllPropertiesOnOneDocument, PromQLEvaluatorIntegrationTest#rangeQuerySharesOneTimeoutBudgetAcrossAllSteps, TextRegexReplaceTest#multiRowReplaceSharesOneTimeoutBudgetAcrossRows, SQLMethodAdditionalCoverageTest#normalizeMultiRowSharesOneTimeoutBudgetAcrossRows, TimeBoundRegexTest#replaceAllUntilSharesOneDeadlineAcrossASeries. Reran the full affected suite (135 tests) plus a broader sweep (CRUDTest, SQLFunctionsTest, QueryTest) - all green.
|
Pushed 8e3a0f5. All three points were real, valid instances of the exact principle this PR establishes elsewhere — fixed all of them. 1. 2. PromQL 3. Minor cache-key naming: fixed too, since I was touching that file anyway — New tests for all four: This was a genuinely valuable round — three real architectural gaps in the core principle this PR is built around, all now closed consistently across every entry point. |
Review: ReDoS hardening for MATCHES/=~/LIKE/full-text/PromQL/REGEXPThis is a well-scoped, thoroughly tested fix for a real DoS vulnerability. I checked out the merge ref locally, read every changed file (not just the diff), and cross-verified the trickier claims against the current code rather than taking the PR description at face value. Summary below. What's solid
Points worth a second look
TestingCompiled and cross-checked call sites against the current tree (checked out the PR merge ref) rather than running the full Maven build in this sandbox (build tooling wasn't available here). The test suite itself looks comprehensive: a dedicated Overall: solid fix, worth merging. The main thing I'd want a second pair of eyes on before release is point 1 (default-on behavior change for existing large-scan workloads) and confirming perf numbers for point 2. |
…dContext MatchesCondition, RegexExpression, TextRegexReplace, and SQLMethodNormalize each hand-rolled the same getCachedValue/newDeadline/setCachedValue pattern with their own cache key. That duplication is exactly what produced the cache-key-collision bug fixed earlier in this PR, so a shared CommandContext.getOrComputeRegexDeadline() default method now owns the get-or-compute logic; call sites just pick a collision-free key.
|
Addressed the duplication point from the last review round: Compiled clean and re-ran the full affected suite ( On the other reconfirmed-but-not-actioned points from the last round: no code change intended.
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 18 |
🟢 Coverage 86.90% diff coverage · -6.56% coverage variation
Metric Results Coverage variation ✅ -6.56% coverage variation Diff coverage ✅ 86.90% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (46c507f) 153675 119612 77.83% Head commit (9124855) 186103 (+32428) 132639 (+13027) 71.27% (-6.56%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#5983) 84 73 86.90% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Review: bound SQL
|
DocumentValidator.validate() computed a regexTimeout deadline unconditionally on every document validation, even for the common case of a document type with no REGEXP-constrained properties. Now computed once, on first actual need, mirroring the lazy PromQLEvaluator.regexDeadline() pattern already used elsewhere in this fix.
|
Addressed the two easy, actionable points from the last round; the other two are genuinely non-blocking nits with no further action needed. 1. 4. Stale PR description: fixed. Rewrote it to describe the actual final scope (all covered entry points, deadline sharing, the two bugs caught during review) instead of the original narrow MATCHES/ 2. 3. Upgrade-note double-check: no code change - this is a request to verify behavior in practice before release, not a code or doc gap. The upgrade note itself is already accurate and prominent (bolded paragraph in Compiled clean, |
Review: ReDoS mitigation for regex entry points (#5886)This is a well-executed, security-critical change. I read through Correctness
One nuance worth a look (not a security hole)I traced For SQL's parallel bucket scan, though, Everything else
Test coverageExcellent, every covered entry point has both a "matches/replaces normally" test and a "catastrophic pattern aborted near the deadline" test, and every entry point with shared-budget semantics has a dedicated multi-row/multi-item timing test that would fail (>=2000ms instead of <1000ms) if the sharing regressed. Nice work on this one, the audit trail through the review passes (documented inline as "Nth review pass") and the regression tests that specifically target bugs caught during that process are a good pattern. |
…orker Verified against FetchFromTypeExecutionStep.syncPullParallel and BasicCommandContext.copy(): each parallel scan worker gets its own CommandContext.copy() (cachedValues is not carried over), so getOrComputeRegexDeadline() computes independently per worker rather than sharing one deadline across the whole query the way the sequential path does. A type scanned across N buckets in parallel is bounded by N * regexTimeout, not one shared budget. Not fixing the sharing itself: cachedValues is a plain, non-thread-safe HashMap, and aliasing it across worker threads to share the deadline would turn getOrComputeRegexDeadline's check-then-set into a genuine race. Bucket count is schema/DDL-controlled, not attacker-controlled the way row/item counts are, so this is a much narrower gap than the one this issue closes - documenting it accurately is the proportionate fix.
|
Verified the parallel-bucket-scan nuance from the last round rigorously before deciding how to respond - traced it end to end:
So the claim is correct: each parallel worker's Not fixing the sharing itself. I looked at actually sharing one deadline across workers and concluded it would trade a narrow, non-attacker-controlled gap for a real bug: Fixed instead: corrected the "shares one deadline across an entire query execution" claim - both the Compiled clean; this is a doc/Javadoc-only change with no test impact. Pushed as 96d3376. |
Review: bound regex against catastrophic backtrackingThis is a large, iteratively self-reviewed PR (18 commits, each addressing a previous review round), and it shows: the commit history documents several real bugs (the Finding:
|
…ator instances SQLFunctionPromQL constructs a fresh PromQLEvaluator on every promql() SQL function call, so it never benefited from the CommandContext-cached deadline the other per-row regex-bounding call sites (text.regexReplace(), .normalize()) already share. Added PromQLEvaluator.setRegexDeadline() to override the lazily-computed default, and wired SQLFunctionPromQL to resolve it from the context before evaluating. Also adds SQL-function-level coverage: promQLSqlFunctionAbortsOnCatastrophicPattern proves a catastrophic =~ label matcher is bounded when reached through SELECT promql(...), not just through PromQLEvaluator.evaluateInstant() directly as the existing tests already covered.
|
Verified the The code characteristic is real: The specific claimed consequence didn't hold up empirically. I reproduced the exact scenario described ( Fixed anyway, for a narrower, real reason. Rather than leave this as a documented non-issue, I added Test added honestly. I didn't reuse the "many identical rows" pattern from other tests in this PR, since I'd just shown it doesn't discriminate a regression here. Instead: Full |
Review: ReDoS hardening (#5886)This is a thorough, well-scoped fix for a real problem: A few things worth a look before merge: 1. Every regex/LIKE/MATCHES evaluation now allocates on the hot path, by default
2.
|
|
This round re-confirms three trade-offs already settled in earlier rounds rather than raising new ones - no code changes needed, restating why for the record: 1. Hot-path allocation from 2. 3. The review's own conclusion: "solid and mergeable... not blockers." Agreed - no further action this round. |
|
`unit-tests` failed on the last CI run (2 failures: `Issue4960LSMCloseDuringCompactionTest#closeCancelsScheduledCompaction` and `DictionaryMultiPageTest#aDictionaryPageThatIsClaimedButMissingFailsLoudly`). Investigated rigorously before treating as a real regression:
Conclusion: pre-existing CI flakes unrelated to this PR, not a regression. Re-running the failed job once the run finishes (can't re-run failed jobs while the workflow run is still in progress). |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5983 +/- ##
==========================================
+ Coverage 69.08% 69.10% +0.01%
==========================================
Files 1797 1798 +1
Lines 153675 153760 +85
Branches 32514 32525 +11
==========================================
+ Hits 106168 106249 +81
- Misses 34102 34108 +6
+ Partials 13405 13403 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 18 |
🟢 Coverage 86.90% diff coverage · -6.56% coverage variation
Metric Results Coverage variation ✅ -6.56% coverage variation Diff coverage ✅ 86.90% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (46c507f) 153675 119612 77.83% Head commit (9124855) 186103 (+32428) 132648 (+13036) 71.28% (-6.56%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#5983) 84 73 86.90% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|
`ha-integration-tests` failed a second consecutive time on retry, same two tests as before: `Issue5410AbandonedTicketReleaseIT#abandonedEntryReleasesItsPhase2TicketOnceApplied` and `Issue5569SlotMergeDeleteRaftIT#mergedDeletesReplicateIntact`. Per the instruction to treat a repeat failure more seriously than a first one, compared both runs in detail rather than just retrying again blind:
Same test identities fail both times, but the specific node index and record/seq numbers differ between runs. That pattern is what a genuine timing/replication race looks like (non-deterministic which node/record loses the race), not what a deterministic bug from this PR's diff would produce (which would fail the same way every time, or not fail at all if truly unrelated). Both tests are in `arcadedb-ha-raft` - Raft leader-election, ticket-release, and slot-merge-delete replication internals - with zero file overlap with this PR's changes (regex/`CommandContext`/PromQL/`DocumentValidator`). This also matches project history: `ha-integration-tests` is a documented chronic flake independent of PR content (tracked separately, running red a large fraction of the time regardless of what's being merged). Retrying once more. If it fails a third time with the same non-deterministic pattern, I'll treat that as sufficient confirmation of the known pre-existing chronic flake rather than continuing to retry indefinitely, and will flag that explicitly rather than silently blocking on it. |
|
`ha-integration-tests` has now failed 3 consecutive times. Summary of the investigation across all 3 attempts before stopping the retry loop, per plan: Attempts 1 & 2: same two tests failed both times - `Issue5410AbandonedTicketReleaseIT#abandonedEntryReleasesItsPhase2TicketOnceApplied` and `Issue5569SlotMergeDeleteRaftIT#mergedDeletesReplicateIntact` - with different specific node indices and record/seq numbers each run (Node 0/record=4/seq=25, then Node 2/record=7/seq=40), the signature of a non-deterministic timing/replication race, not a deterministic bug. Attempt 3: the same two tests failed again (Node 2/record=8/seq=17 this time - again different data), plus a third, different test failed: `Issue4743LargeRecordReplicationIT#recordLargerThanTheOldCeilingReplicatesToEveryNode`, with `Schema Cannot create type 'Blob' because already exists` - a test-isolation/schema-state-pollution error, a different failure class entirely from the other two. All three failing tests are in `arcadedb-ha-raft` (Raft leader-election, ticket-release, slot-merge-delete, and large-record replication internals) - zero file overlap with this PR's changes (regex/`CommandContext`/PromQL/`DocumentValidator`). The variety of failure types and non-deterministic data across 3 independent runs points to general instability of this test suite/CI runner rather than a regression introduced by this PR. This also matches this suite's documented status as a chronic flake independent of PR content. Not retrying a 4th time. Every other check on this PR is green, including `unit-tests` (which failed once and passed clean on retry) and the code review itself (round 18: "solid and mergeable... not blockers"). Bringing the merge decision to @lvca given `ha-integration-tests` is a required check that hasn't gone green across 3 attempts despite being unrelated to this PR's diff. |
Summary
Fixes #5886.
java.util.regexnever polls an interrupt flag or checks a deadline while backtracking - the only thing it calls on every backtracking step isCharSequence.charAt()on the input. A pathological pattern like(.*a){20}$against a 41-character string triggers catastrophic backtracking and is still running 30+ seconds later, andarcadedb.command.timeoutcannot stop it. Anyone who can reach a regex-based entry point with an attacker-controlled pattern - or who forwards user-supplied patterns from an application layer - can permanently tie up a worker thread with a handful of such operations.What started as a fix for SQL
MATCHESand openCypher=~(the two entry points named in the issue) grew, through the review process on this PR, into an audit of every user-reachable regex call site in the engine.Fix
TimeBoundRegex(new,com.arcadedb.utility) wraps the matched input in aCharSequencewhosecharAt()throws once a deadline elapses - the one interception pointjava.util.regexoffers, formatches(),replaceAll(), andsplit()alike. The deadline comes from a new dedicated setting,arcadedb.command.regexTimeout(default 1000ms, per-database), independent ofarcadedb.command.timeoutand active even when that setting is left at its own default of disabled (0). The deadline is checked every 256charAt()calls (a bitmask check) rather than every call, keeping it off the hot path for the overwhelming majority of patterns that never come close to that many backtracking steps.Covered entry points:
MATCHESand openCypher=~LIKE/ILIKE, including the native query engine'sSelectOperatorpath (a sequence of unparenthesized.*/*segments reproduces the same catastrophic shape without any grouping/alternation/nested quantifiers - the original "LIKE is structurally safe" audit conclusion was wrong)text.regexReplace()and the.normalize()SQL method's optional pattern argumentRegexpQuery(/pattern/) andWildcardQuery(*/?) support=~/!~label matchers, including every step of a range queryREGEXPproperty validation (CREATE PROPERTY ... REGEXP <pattern>) - the most exposed of the group, since it runs on every insert/update of a validated property through any write path (REST, any wire protocol) with no query privileges needed.split(delimiter)'s SQL method got a different fix: unlike its three siblings (split()function,text.split(), Cyphersplit()), which all treat the delimiter as a literal viaPattern.quote(...), it passed the delimiter straight toString.split(regex)unescaped. Matched to its siblings, removing the risk entirely rather than just bounding it.Deadline sharing. A regex evaluated once per row/token/item/property over a scan must not get a fresh timeout budget per item, or a table/index/document shaped so every item triggers catastrophic backtracking costs
itemCount * regexTimeoutinstead of one bounded operation.MATCHES,=~,text.regexReplace(), and.normalize()share one deadline across an entire query execution via a newCommandContext.getOrComputeRegexDeadline()helper; full-text and schemaREGEXPvalidation share one deadline across the whole scan/document; PromQL shares one deadline across an entire query's lifetime, including every step of a range query.LIKE/ILIKEare the one exception:BinaryCompareOperatorhas noCommandContextto cache a deadline on, and widening that interface was judged out of proportion here, so each row gets its own budget.Upgrade note. Because the deadline is shared per query/scan rather than per match, and defaults to an enabled 1000ms, a legitimate non-catastrophic operation that previously ran slowly to completion can now fail with
TimeoutExceptioninstead. Raisearcadedb.command.regexTimeoutfor any database with large-scan regex-heavy workloads.Full detail in
docs/release-26.9.1.md.Testing
Dedicated
TimeBoundRegexTestfor the utility, plus regression tests at every covered entry point (MatchesConditionTest,OpenCypherWhereClauseTest,QueryHelperLikeRegexTimeoutTest,SelectExecutionTest,LikeOperatorTest/ILikeOperatorTest,TextRegexReplaceTest,SQLMethodNormalizeTest,SQLMethodSplitTest,FullTextQueryExecutorTest,PromQLEvaluatorIntegrationTest,DocumentValidationTest), each covering "matches/replaces normally", "catastrophic pattern is aborted near the deadline", and (where sharing applies) "one budget is shared across N rows/items/steps/properties".Two real bugs were caught and fixed by the review process, both with regression tests that fail without the fix:
MatchesConditionTest#patternTextDeadlineDoesNotCollideWithTheDeadlineCacheKey).CypherStatementCache, causing spurious timeouts on later benign executions of the same cached query (OpenCypherWhereClauseTest#regexDeadlineDoesNotLeakAcrossCachedQueryExecutions).Test plan
mvn -pl engine compile/test-compileclean