You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Supermemory provider ignores options.limit and returns 30 results while every other provider returns 10 — cross-provider comparisons are not like-for-like #64
Note the Supermemory call also accepts options.threshold on the very next line, so the hardcoded limit reads as an oversight rather than a deliberate provider capability.
Impact
The whole point of the tool is the README's "🆚 Multi-provider comparison: run the same benchmark across providers side-by-side". That comparison is currently unfair in three measurable ways:
Answer quality.runAnswerPhase feeds the full stored result set to the answering model as context (answer.ts:118-123). Supermemory supplies 30 candidate memories; competitors supply 10. More retrieved context is a direct accuracy advantage that has nothing to do with retrieval quality.
MemScore token component.contextTokens is derived from the prompt built out of those results (answer.ts:125-130). Supermemory's token count is inflated by ~3× the context, so its MemScore token figure is not comparable to the others' either.
Retrieval metrics.calculateRetrievalMetrics defaults to k = 10 and slices (retrieval-eval.ts:93-95), so Supermemory is scored on the best 10 of 30 candidates while others are scored on the only 10 they were allowed to return. Precision@10, MRR and NDCG are computed over different candidate pools.
This matters especially because the benchmark is published by the vendor of one of the providers under comparison — the numbers should be defensible on their face.
Suggested fix
- limit: 30,+ limit: options.limit || 30,
If a different limit per provider is ever intentional, it belongs in the Provider interface as a declared capability and must be surfaced in the report, not buried in a call site.
Activity