Summary
Split out from #833: a string-syntax proximity query ("a b"~n) and the equivalent XML <near slop="n"> query can match a genuinely different set of documents for the same nominal slop value, when word order varies in the indexed text. This is separate from (and deeper than) the exist:match-highlighting bug fixed in #833 / #6756 — it's about which documents match at all, not just how matches get highlighted.
Root cause
The two syntaxes compile to different Lucene query types with different, both intentional and documented, slop semantics:
- String syntax (
"a b"~n) compiles to Lucene's PhraseQuery. Its slop is an edit distance that tolerates term reordering — see PhraseQuery#getSlop()'s javadoc: "the fox is quick" matches "quick fox" at an edit distance of 3, i.e. a full reversal is tolerated, just at higher cost.
- XML syntax (
<near slop="n">, default ordered="yes") compiles to SpanNearQuery. Its slop is a plain positional gap; ordered near never matches out-of-order terms, at any slop.
This was confirmed with a controlled reproduction (see ftt:slop-string-vs-xml-reordering-disagreement in extensions/indexes/lucene/src/test/xquery/lucene/ft-match.xql, added in #6756): for text with foul@position 0 and fair@position 6 (reversed relative to the query "fair foul"), '"fair foul"~7' matches (edit distance formula: |(0-6) - 1| = 7), but <near slop="7"><term>fair</term><term>foul</term></near> (ordered) never matches at any slop. This is very likely what's happening with "fair is foul, and foul is fair" in the Shakespeare corpus used in the original #833 report (queries 5/6, "fair foul"~2).
What #6756 already did about this
Rather than silently changing either syntax's default matching behavior (which would change real search results for existing indexed content and saved queries — a compatibility risk), #6756 added an opt-in phrase-as-near query option (ft:query/ft:query-field/ft:search) that rewrites a parsed PhraseQuery into an equivalent ordered SpanNearQuery, so a query can explicitly request <near>-equivalent matching. It also documents the semantic difference on the three functions' signatures.
What's left here
phrase-as-near is opt-in and only rewrites PhraseQuery (plus BooleanQuery/BoostQuery wrapping it) into an ordered SpanNearQuery. There's no equivalent to request the reverse (make <near> tolerate PhraseQuery-style reordering), nor an unordered variant of the option.- Whether a one-time reconciliation at query-construction time (rather than an opt-in escape hatch) is worth the compatibility risk is an open design question for a maintainer to weigh in on.
LuceneUtil.asProximityTerms's highlighting also has a known related limitation (documented in its javadoc): a genuine PhraseQuery reordering match gets no highlight span at all, since the highlighter only scans forward. Out of scope for a fix there too (would need buffering, since matching backwards isn't possible on a forward-only token stream).
References
Summary
Split out from #833: a string-syntax proximity query (
"a b"~n) and the equivalent XML<near slop="n">query can match a genuinely different set of documents for the same nominal slop value, when word order varies in the indexed text. This is separate from (and deeper than) theexist:match-highlighting bug fixed in #833 / #6756 — it's about which documents match at all, not just how matches get highlighted.Root cause
The two syntaxes compile to different Lucene query types with different, both intentional and documented, slop semantics:
"a b"~n) compiles to Lucene'sPhraseQuery. Its slop is an edit distance that tolerates term reordering — seePhraseQuery#getSlop()'s javadoc:"the fox is quick"matches"quick fox"at an edit distance of 3, i.e. a full reversal is tolerated, just at higher cost.<near slop="n">, defaultordered="yes") compiles toSpanNearQuery. Its slop is a plain positional gap; ordered near never matches out-of-order terms, at any slop.This was confirmed with a controlled reproduction (see
ftt:slop-string-vs-xml-reordering-disagreementinextensions/indexes/lucene/src/test/xquery/lucene/ft-match.xql, added in #6756): for text withfoul@position 0 andfair@position 6 (reversed relative to the query"fair foul"),'"fair foul"~7'matches (edit distance formula:|(0-6) - 1| = 7), but<near slop="7"><term>fair</term><term>foul</term></near>(ordered) never matches at any slop. This is very likely what's happening with "fair is foul, and foul is fair" in the Shakespeare corpus used in the original #833 report (queries 5/6, "fair foul"~2).What #6756 already did about this
Rather than silently changing either syntax's default matching behavior (which would change real search results for existing indexed content and saved queries — a compatibility risk), #6756 added an opt-in
phrase-as-nearquery option (ft:query/ft:query-field/ft:search) that rewrites a parsedPhraseQueryinto an equivalent orderedSpanNearQuery, so a query can explicitly request<near>-equivalent matching. It also documents the semantic difference on the three functions' signatures.What's left here
phrase-as-nearis opt-in and only rewritesPhraseQuery(plusBooleanQuery/BoostQuerywrapping it) into an orderedSpanNearQuery. There's no equivalent to request the reverse (make<near>tolerate PhraseQuery-style reordering), nor an unordered variant of the option.LuceneUtil.asProximityTerms's highlighting also has a known related limitation (documented in its javadoc): a genuinePhraseQueryreordering match gets no highlight span at all, since the highlighter only scans forward. Out of scope for a fix there too (would need buffering, since matching backwards isn't possible on a forward-only token stream).References
phrase-as-nearoption)PhraseQuery#getSlop()javadoc (Lucene 10.4): the edit-distance/reordering-tolerance definition