Skip to content

Choosing the mappings and searching chunks reuse the question's embedding instead of embedding it again in the same turn - #978

Merged
WaylandYang merged 1 commit into
deeplethe:devfrom
Maya-Kid:fix/embed-once-across-search-and-mappings
Sep 27, 2026
Merged

WaylandYang merged 1 commit into
deeplethe:devfrom
Maya-Kid:fix/embed-once-across-search-and-mappings

Conversation

@Maya-Kid

Copy link
Copy Markdown
Contributor

The follow-up suggested in #971's review: search_chunks and mapping_index::relevant share the turn's embedding cache.

Problem

  • Embed the question once per chat turn, not on every graph lookup #971 embeds the question once per turn for the graph lookups, on the turn's ToolSink.
  • Two other places embed the same sentence on their own: choosing the confirmed mappings for the prompt (mapping_index::relevant, before the first model request) and the vector channel of search_chunks (retrieval::hybrid). A question over a base with a data source was embedded up to three times per turn: once for the mappings, once for the search when the model passes the question as its query, once for the first lookup by name.

Change

  • llm_util::EmbedCache names the cache the sink already held (HashMap<String, Vec<f32>>).
  • retrieval::hybrid and mapping_index::relevant take Option<&mut EmbedCache>. Their vector channels read the vector from the cache before reading the model settings, and put a newly embedded vector in it. A cache hit in relevant skips the embedding request and the embed permit; it still reads the settings for the model name, since the mapping search compares only rows embedded by the same model.
  • The chat turn creates the cache before choosing the mappings and moves it into the sink once the sink exists, so the question's vector reaches search_chunks and the graph lookups.
  • search_chunks passes the sink's cache. The HTTP search route, the mappings page and the one-shot RAG fallback pass None and embed as before: they have no turn.
  • Only successful embeddings are cached; failures degrade to BM25 or the lexical channel as they did.

Verification

Run in containers on a Linux host: Rust 1.98.1 and PostgreSQL 16 with pgvector.

Tests, in chat_embed_once_tests.rs

  • a_chat_turn_embeds_its_question_once now scripts search_chunks with the question as its query before find_entities and entity_facts: three steps, one embedding request.
  • mappings_search_and_the_graph_tools_share_one_embedding: relevant on a base with no confirmed mappings embeds the question once; hybrid with the same cache embeds nothing and still runs its vector channel; find_entities on a sink holding the cache picks the entity closest to the question without embedding; a different search query is embedded once and cached; the same calls with None embed again.

On dev (0fbcd38) with the new tests

The control is dev plus the chat-level test (the function-level one needs the new parameters and does not compile there). a_chat_turn_embeds_its_question_once fails there: the embedding service receives two requests, the search's and the lookup's, where one is expected. a_turn_embeds_its_question_and_each_word_once passes there as here (logs 520–522).

This branch

The embedding tests (10), api:: with retrieval and mapping (213), cargo fmt --all --check, cargo clippy --locked --workspace --all-targets -- -D warnings, UTOPIA_TEST_REQUIRE_PDFTOTEXT=1 cargo test --locked --workspace with UTOPIA_TEST_REQUIRE_DB=1 (1,228 passed, 0 failed, 5 pre-existing ignored) and cargo build --locked --workspace all pass (logs 523–530). No web file changes.

Also on a Mac with the same toolchain and a scratch PostgreSQL 16: the embedding tests (10), api::, retrieval and mapping (213), fmt and clippy clean.

Known limits

  • The search reuses the vector only when the model's query is the question verbatim; a rephrased query is embedded once and cached for the rest of the turn.
  • The one-shot RAG fallback embeds the question itself; it runs once per turn and only when the endpoint refused tool calling.

…ding instead of embedding it again in the same turn

Signed-off-by: dada-yan <BinjunYann@gmail.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. One cache for the turn, created before the mappings are chosen and handed to the sink, and the callers without a turn pass None. A cache hit still reads the settings for the model name in the mapping search, which is the right caution. Landing it. Thanks again.

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer review: sound, landing.

@WaylandYang
WaylandYang merged commit b5298b4 into deeplethe:dev Sep 27, 2026
@Maya-Kid
Maya-Kid deleted the fix/embed-once-across-search-and-mappings branch September 27, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants