Skip to content

compact() returns another key's vector from get() (no removals required) #9

Description

@titusz

Summary

After calling compact(), get(key) returns another key's vector. No removals are required —
compact() on a freshly built index with zero deletions is enough to break retrieval for every key
sampled.

The graph itself survives (search self-recall stays ~99%), so this is specifically the key→vector
retrieval path: slot reordering performed by compact() is not reflected in the lookup that get()
resolves through.

compact() is not exposed on the Python Index wrapper, but it is bound on the compiled module
(python/lib.cpp:1480, i.def("compact", &compact_index<index_at>, ...)), so it is reachable as
index._compiled.compact(threads=...). That makes it an attractive-looking trap for anyone who goes
looking for a way to repair a degraded graph.

Reproduction

Pure usearch, stock MetricKind.Hamming, zero removals:

"""compact() leaves get() returning another key's vector."""

import numpy as np
from usearch.index import Index, MetricKind, ScalarKind

N = 2_000
rng = np.random.default_rng(7)
vecs = rng.integers(0, 256, size=(N, 32), dtype=np.uint8)
keys = np.arange(1, N + 1, dtype=np.uint64)
owner = {vecs[i].tobytes(): i + 1 for i in range(N)}

index = Index(ndim=256, metric=MetricKind.Hamming, dtype=ScalarKind.B1)
index.add(keys, vecs)


def report(label):
    print(label)
    for k in (1, 2, 3, 500, 1000, 2000):
        got = np.asarray(index.get(int(k)), dtype=np.uint8).ravel()
        print(f"  get({k:>4}) returns the vector of key {owner.get(got.tobytes(), '<unknown>')}")


report("before compact():")
index._compiled.compact(threads=0)  # not exposed on the Index wrapper
report("\nafter compact():")

hits = sum(
    int(np.atleast_1d(index.search(vecs[k - 1], 1).keys).ravel()[0]) == k for k in range(1, 201)
)
print(f"\nsearch self-recall after compact(): {hits}/200 (graph itself is fine)")
print(f"index size: {len(index)}")

Output:

before compact():
  get(   1) returns the vector of key 1
  get(   2) returns the vector of key 2
  get(   3) returns the vector of key 3
  get( 500) returns the vector of key 500
  get(1000) returns the vector of key 1000
  get(2000) returns the vector of key 2000

after compact():
  get(   1) returns the vector of key 52
  get(   2) returns the vector of key 302
  get(   3) returns the vector of key 1802
  get( 500) returns the vector of key 1992
  get(1000) returns the vector of key 111
  get(2000) returns the vector of key 1541

search self-recall after compact(): 197/200 (graph itself is fine)
index size: 2000

Every sampled key returns a wrong vector. len(index) and contains() remain correct, so the
breakage is silent — a caller that uses get() to read vectors back gets plausible-looking but
wrong data with no error.

Suspected mechanism

index_dense_gt::compact() reorders nodes and rewrites neighbor links to match the new slot
assignments (and updates entry_slot_). The key→slot mapping used by get() appears not to be
rebuilt with the same track_slot_change remapping, so keys resolve to stale slot numbers that now
hold different vectors.

Suggested resolution

Either fix the remapping so compact() keeps get() consistent and then expose it on the Python
Index wrapper, or — if it is known-incomplete — remove/guard the _compiled.compact binding so it
cannot be reached by accident.

A working compact() matters beyond this bug: it is the natural repair for the graph-reachability
defect in the companion issue below.

Environment

  • usearch-iscc 2.24.5 (reports usearch.__version__ == 2.24.0)
  • Python 3.12.0, Windows 10 x86_64

Companion issue

Graph reachability lost after remove()+add() churn — see #8

Downstream tracking issue: iscc/iscc-usearch#30

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions