Lucene104ScalarQuantizedVectorsWriter includes deleted docs when recalculating the centroid

[Pulkitg64]

Description

Lucene104ScalarQuantizedVectorsWriter computes a per-field centroid before quantizing the full-precision vectors during a merge. The centroid is the mean of the full-precision vector values gathered from the segments being merged, and it is later used as the reference point for scalar quantization of every vector in the merged segment.

When any source segment has deletions, the recalculation path does not skip the deleted documents. (Code Ref)

static int calculateCentroid(MergeState mergeState, FieldInfo fieldInfo, float[] centroid)
      throws IOException {
    assert fieldInfo.getVectorEncoding().isFloatingPoint();
    // clear out the centroid
    Arrays.fill(centroid, 0);
    int count = 0;
    for (int i = 0; i < mergeState.knnVectorsReaders.length; i++) {
      KnnVectorsReader knnVectorsReader = mergeState.knnVectorsReaders[i];
      if (knnVectorsReader == null) continue;
      count += accumulateCentroid(knnVectorsReader, fieldInfo, centroid);
    }
    if (count == 0) {
      return count;
    }
    for (int i = 0; i < centroid.length; i++) {
      centroid[i] /= count;
    }
    if (fieldInfo.getVectorSimilarityFunction() == COSINE) {
      VectorUtil.l2normalize(centroid);
    }
    return count;
  }

  private static int accumulateCentroid(
      KnnVectorsReader reader, FieldInfo fieldInfo, float[] centroid) throws IOException {
    FloatVectorValues vectorValues = floatingPointVectorValues(reader, fieldInfo);
    if (vectorValues == null) {
      return 0;
    }
    int count = 0;
    KnnVectorValues.DocIndexIterator iterator = vectorValues.iterator();
    for (int doc = iterator.nextDoc(); doc != NO_MORE_DOCS; doc = iterator.nextDoc()) {
      count++;
      float[] vector = vectorValues.vectorValue(iterator.index());
      for (int j = 0; j < vector.length; j++) {
        centroid[j] += vector[j];
      }
    }
    return count;
  }

Version and environment details

No response