Skip to content

Supermemory provider ignores options.limit and returns 30 results while every other provider returns 10 — cross-provider comparisons are not like-for-like #64

Open

Description

@rajarshidattapy

Description

runSearchPhase asks every provider for the same number of results:

// src/orchestrator/phases/search.ts:58-62
const results = await provider.search(question.question, {
  containerTag,
  limit: 10,
  threshold: 0.3,
})

Every provider honours that limit except Supermemory, which hardcodes 30:

Provider Code Effective results
supermemory src/providers/supermemory/index.ts:126 — limit: 30 30
mem0 src/providers/mem0/index.ts:180 — top_k: options.limit || 30 10
zep src/providers/zep/index.ts:243 — options.limit || 20 10
filesystem src/providers/filesystem/index.ts:178 — options.limit || 10 10
rag src/providers/rag/index.ts:218 — options.limit || 10 10

Note the Supermemory call also accepts options.threshold on the very next line, so the hardcoded limit reads as an oversight rather than a deliberate provider capability.

Impact

The whole point of the tool is the README's "🆚 Multi-provider comparison: run the same benchmark across providers side-by-side". That comparison is currently unfair in three measurable ways:

  1. Answer quality. runAnswerPhase feeds the full stored result set to the answering model as context (answer.ts:118-123). Supermemory supplies 30 candidate memories; competitors supply 10. More retrieved context is a direct accuracy advantage that has nothing to do with retrieval quality.
  2. MemScore token component. contextTokens is derived from the prompt built out of those results (answer.ts:125-130). Supermemory's token count is inflated by ~3× the context, so its MemScore token figure is not comparable to the others' either.
  3. Retrieval metrics. calculateRetrievalMetrics defaults to k = 10 and slices (retrieval-eval.ts:93-95), so Supermemory is scored on the best 10 of 30 candidates while others are scored on the only 10 they were allowed to return. Precision@10, MRR and NDCG are computed over different candidate pools.

This matters especially because the benchmark is published by the vendor of one of the providers under comparison — the numbers should be defensible on their face.

Suggested fix

-      limit: 30,
+      limit: options.limit || 30,

If a different limit per provider is ever intentional, it belongs in the Provider interface as a declared capability and must be surfaced in the report, not buried in a call site.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions