-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Part of #10
Add the OpenClaw builtin memory search as a comparison provider in the benchmark suite.
Requirements
- Index the same test corpus with OpenClaw's builtin memory-core provider
- Run the same queries through the builtin
memory_searchtool - Score with the same metrics (Recall@K, MRR, Precision@K)
- Side-by-side comparison output: BM vs builtin per category
Implementation options
- Preferred: Start an OpenClaw instance with builtin memory pointing at the benchmark corpus, query via the memory tools
- Alternative: Replicate the builtin's chunking + embedding + hybrid search logic in a standalone script (more work, but no OpenClaw dependency for CI)
Key comparisons
- Semantic queries: both use vector search, should be similar
- Relational queries: BM has knowledge graph, builtin has only text
- Exact fact queries: builtin has BM25 hybrid, BM has FTS — compare
- Task queries: BM composited search scans tasks, builtin does not
- Context efficiency: BM returns structured observations, builtin returns raw chunks
Depends on
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request