Skip to content

Benchmark: OpenClaw builtin memory-core comparison #13

@bm-clawd

Description

@bm-clawd

Part of #10

Add the OpenClaw builtin memory search as a comparison provider in the benchmark suite.

Requirements

  • Index the same test corpus with OpenClaw's builtin memory-core provider
  • Run the same queries through the builtin memory_search tool
  • Score with the same metrics (Recall@K, MRR, Precision@K)
  • Side-by-side comparison output: BM vs builtin per category

Implementation options

  1. Preferred: Start an OpenClaw instance with builtin memory pointing at the benchmark corpus, query via the memory tools
  2. Alternative: Replicate the builtin's chunking + embedding + hybrid search logic in a standalone script (more work, but no OpenClaw dependency for CI)

Key comparisons

  • Semantic queries: both use vector search, should be similar
  • Relational queries: BM has knowledge graph, builtin has only text
  • Exact fact queries: builtin has BM25 hybrid, BM has FTS — compare
  • Task queries: BM composited search scans tasks, builtin does not
  • Context efficiency: BM returns structured observations, builtin returns raw chunks

Depends on

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions