[BUG] ft:query-vector k ignores a multi-document predicate and throws past 341 documents

[duncdrum]

Describe the bug

k is applied only by a direct call, and only while the node set spans at most 341 documents. collection(...)//article[ft:query-vector(., $vec, $k)] returns every article once each article is in its own document. vector-search.xqm still passes because its articles share one file: query-vector-arity-three expects 2 of 3.

LuceneIndexWorker.buildDocsFilterQuery builds that limit. It adds one IntField.newExactQuery("docId", id) per document as a SHOULD clause. Lucene flattens those to about three clauses each, and BooleanQuery caps at 1024, so 341 documents return 5 and 342 throw maxClauseCount is set to 1024. The only caller is buildVectorFilterQuery. QueryVector.parseK does pass k into KnnFloatVectorQuery. Hit order is separate and already documented: sort with ft:score.

Expected behavior

Both the path form and ft:query-vector($nodes, $vec, $k) return $k nodes, including when each node is its own document and when the set is larger than 341 documents. The one-document cases in vector-search.xqm stay as they are.

To Reproduce

Precomputed 4-d vectors, no ONNX. On a 342-document collection the path test returns 342 and the direct test throws. Both should return 5. Same pattern measured at 30 documents: path 30, direct 5.

xquery version "3.1";

module namespace t="http://exist-db.org/xquery/test";

declare namespace test="http://exist-db.org/xquery/xqsuite";
declare namespace xmldb="http://exist-db.org/xquery/xmldb";

import module namespace ft="http://exist-db.org/xquery/lucene";

declare variable $t:N := 342;
declare variable $t:K := 5;
declare variable $t:VEC := [1.0, 0.0, 0.0, 0.0];

declare variable $t:xconf :=
    <collection xmlns="http://exist-db.org/collection-config/1.0">
        <index xmlns:xs="http://www.w3.org/2001/XMLSchema">
            <lucene>
                <text qname="article">
                    <vector-field name="embedding" expression="embedding"
                        dimension="4" similarity="cosine" encoding="text"/>
                </text>
            </lucene>
        </index>
    </collection>;

declare variable $t:COLL := "/db/test-" || translate(string(current-dateTime()), "-:TZ.+", "");
declare variable $t:CONF_COLL := "/db/system/config/db/" || substring-after($t:COLL, "/db/");

declare
    %test:setUp
function t:setup() {
    let $_ := xmldb:create-collection("/db/system", "config")
    let $_ := xmldb:create-collection("/db/system/config", "db")
    let $_ := xmldb:create-collection("/db", substring-after($t:COLL, "/db/"))
    let $_ := xmldb:create-collection("/db/system/config/db", substring-after($t:COLL, "/db/"))
    let $_ := xmldb:store($t:CONF_COLL, "collection.xconf", $t:xconf)
    let $_ :=
        for $i in 1 to $t:N
        return xmldb:store(
            $t:COLL,
            "a" || $i || ".xml",
            <article><embedding>1.0 0.0 0.0 0.0</embedding></article>
        )
    return xmldb:reindex($t:COLL)
};

declare
    %test:tearDown
function t:tearDown() {
    if (xmldb:collection-available($t:COLL)) then xmldb:remove($t:COLL) else (),
    if (xmldb:collection-available($t:CONF_COLL)) then xmldb:remove($t:CONF_COLL) else ()
};

(: Path form. Today this returns 342. :)
declare
    %test:assertEquals(5)
function t:path-form-honors-k() {
    count(collection($t:COLL)//article[ft:query-vector(., $t:VEC, $t:K)])
};

(: Direct call. Today this throws maxClauseCount at 342 documents. :)
declare
    %test:assertEquals(5)
function t:direct-call-honors-k() {
    count(ft:query-vector(collection($t:COLL)//article, $t:VEC, $t:K))
};

Suggested home: extensions/indexes/lucene/src/test/xquery/lucene/query-vector-k.xqm, beside vector-search.xqm.

Screenshots

n/a

Context (please always complete the following information)

Build: eXist-7.0.0-SNAPSHOT (9e51fdf62f20913bb925a3b06aa9957c224f9d94)
Java: 21.0.12 (Eclipse Adoptium)
OS: Linux 7.0.12-linuxkit (aarch64)

Additional context

Docker image duncdrum/existdb:experimental. No conf.xml change. Replace the per-document boolean in buildDocsFilterQuery with one set query over docId. Then make the path predicate one search over the step, as ft:query does. Doing only the rewrite makes a collection of this size throw, and a caller that catches the error drops the vector leg.