(EAI-988): Refactor `GenerateResponse` for tool call support #687

mongodben · 2025-04-28T16:13:04Z

Jira: https://jira.mongodb.org/browse/EAI-988

Changes

Simplify GenerateResponse so only takes the request-specific args
- Can pass other stuff in a constructor function
Refactor legacy RAG system to fit into shape of new construct.
- To be removed relatively soon
Include GenerateResponse in args to conversations server.
preserve includeChunksForMaxTokensPossible() utility func from now removed generate user message file

Notes

nlarew

LGTM - some minor comments + a typo that you might wanna fix

nlarew · 2025-04-28T18:50:47Z

packages/chatbot-server-mongodb-public/src/config.ts

@@ -20,6 +20,7 @@ import {
  makeDefaultFindVerifiedAnswer,
  defaultCreateConversationCustomData,
  defaultAddMessageToConversationCustomData,
+  makeLegacyGeneratateResponse,


Suggested change

makeLegacyGeneratateResponse,

makeLegacyGenerateResponse,

typo throughout

nlarew · 2025-04-28T18:57:32Z

packages/mongodb-chatbot-server/src/processors/makeFilterNPreviousMessages.test.ts

@@ -21,25 +15,7 @@ const userMessage = {
 } satisfies Message;

 describe("makeFilterNPreviousMessages", () => {
-  it("should throw an error when there are no messages", async () => {


Just confirming - we don't want to throw anymore if the array is empty?

correct. this is b/c we wont store the system message anymore.

this is for a few reasons:

this is b.c some tool info is in the system msg. so if tool changes, then could be mismatch btwn behavvior and persisted system message

logically, the system msg isn't really part of the converastion, so probably should have never been saved 🤷

nlarew · 2025-04-28T19:01:53Z

packages/mongodb-chatbot-server/src/routes/conversations/addMessageToConversation.ts

+export interface GenerateResponseParams {
+  shouldStream: boolean;
+  latestMessageText: string;
+  clientContext?: ClientContext;
+  customData?: ConversationCustomData;
+  dataStreamer?: DataStreamer;
+  reqId: string;
+  conversation: Conversation;
+  request?: ExpressRequest;
+}
+
+export interface GenerateResponseReturnValue {
+  messages: SomeMessage[];
+}
+
+export type GenerateResponse = (
+  params: GenerateResponseParams
+) => Promise<GenerateResponseReturnValue>;


Should this all be in a separate generateResponse.ts file? Seems like a distinct (if not separate) concern from the route that calls it.

yeah reasonable, will do

verified answer generate response Co-authored-by: Ben Perlmutter <mongodben@mongodb.com>

* (EAI-988): Refactor `GenerateResponse` for tool call support (#687) * refactor GenerateRespose * Clean up imports * consolidate generate user prompt to the legacy file * update test config imports * Fix broken tests * (EAI-989): Refactor verified answers to wrap `GenerateResponse` (#688) verified answer generate response Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> * handle streaming * separate generateresponse * typo fix --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> * (EAI-990): Refactor search as a tool (#705) * refactor GenerateRespose * Clean up imports * consolidate generate user prompt to the legacy file * update test config imports * Fix broken tests * get started * nominally working generate res w/ search * small refactors * aint pretty but fully functional * hacky if more functional * more hack * tools * functional if not pretty * Add processing * working tool calling * making progress * keepin on * Clean config * working e2e * update model version * Remove no longer used stuff * decouple search results for references and whats shown to model * fix scripts build errs * fix broken tests * update default ref links * fix broken tests * Apply suggestions from code review Co-authored-by: Nick Larew <nick.larew@mongodb.com> * revert default reference links * adding missing test --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> Co-authored-by: Nick Larew <nick.larew@mongodb.com> * (EAI-992): Remove `ChatLlm` (#751) * refactor GenerateRespose * Clean up imports * consolidate generate user prompt to the legacy file * update test config imports * Fix broken tests * get started * nominally working generate res w/ search * small refactors * aint pretty but fully functional * hacky if more functional * more hack * tools * functional if not pretty * Add processing * working tool calling * making progress * keepin on * Clean config * working e2e * update model version * Remove no longer used stuff * decouple search results for references and whats shown to model * fix scripts build errs * remove ChatLlm * lite fixes * Remove stub --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> * (EAI-993): deprecate framework (#752) * refactor GenerateRespose * Clean up imports * consolidate generate user prompt to the legacy file * update test config imports * Fix broken tests * get started * nominally working generate res w/ search * small refactors * aint pretty but fully functional * hacky if more functional * more hack * tools * functional if not pretty * Add processing * working tool calling * making progress * keepin on * Clean config * working e2e * update model version * Remove no longer used stuff * decouple search results for references and whats shown to model * fix scripts build errs * fix broken tests * deprecation * build out docs following last week convo * clean up spec + contact * fix merge funk * docs updates --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> * (EAI-1071): Fix broken Atlas OpenAPI ingest (#765) * update to fix broken test * Update packages/ingest-mongodb-public/src/sources/snooty/snootyAstToOpenApiSpec.ts --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> * (EAI-995): add guardrail (#755) * refactor GenerateRespose * Clean up imports * consolidate generate user prompt to the legacy file * update test config imports * Fix broken tests * get started * nominally working generate res w/ search * small refactors * aint pretty but fully functional * hacky if more functional * more hack * tools * functional if not pretty * Add processing * working tool calling * making progress * keepin on * Clean config * working e2e * update model version * Remove no longer used stuff * decouple search results for references and whats shown to model * fix scripts build errs * fix broken tests * update default ref links * fix broken tests * input guardrail refactor * guardrail works well * simpler validity metric * add guardrail to server * add next step todo * llm refusal msg * remove TODO comment * merge fix * fix unnec changes * NL feedback --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> * fix type in text * (EAI-991 & EAI-1050): Evaluate and clean up retrieval as a tool (#757) * refactor GenerateRespose * Clean up imports * consolidate generate user prompt to the legacy file * update test config imports * Fix broken tests * get started * nominally working generate res w/ search * small refactors * aint pretty but fully functional * hacky if more functional * more hack * tools * functional if not pretty * Add processing * working tool calling * making progress * keepin on * Clean config * working e2e * update model version * Remove no longer used stuff * decouple search results for references and whats shown to model * fix scripts build errs * fix broken tests * update default ref links * fix broken tests * input guardrail refactor * guardrail works well * simpler validity metric * add guardrail to server * add next step todo * llm refusal msg * remove TODO comment * evals on new architecture * Get urls in a way that supports verified answers * dont eval on retrieved elems if no context * Cleaner handling * update trace handling * update trace handling * undo git funk * handle undefined case * Fix tracing test --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> * remove console logs + redunancies --------- Co-authored-by: Ben Perlmutter <mongodben@mongodb.com> Co-authored-by: Nick Larew <nick.larew@mongodb.com>

Ben Perlmutter added 2 commits April 28, 2025 12:08

refactor GenerateRespose

2e286c3

Clean up imports

0be9fe2

mongodben changed the base branch from main to retrieval_tool_call April 28, 2025 16:18

Ben Perlmutter added 3 commits April 28, 2025 12:26

consolidate generate user prompt to the legacy file

2556bc2

update test config imports

feee66a

Fix broken tests

f6fe862

nlarew approved these changes Apr 28, 2025

View reviewed changes

mongodben and others added 4 commits April 28, 2025 15:42

(EAI-989): Refactor verified answers to wrap GenerateResponse (#688)

a68391f

verified answer generate response Co-authored-by: Ben Perlmutter <mongodben@mongodb.com>

handle streaming

da1676e

separate generateresponse

b0b3e10

typo fix

2a5af1f

mongodben merged commit c87dc74 into retrieval_tool_call May 27, 2025
1 check failed

mongodben deleted the EAI-988 branch May 27, 2025 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(EAI-988): Refactor `GenerateResponse` for tool call support #687

(EAI-988): Refactor `GenerateResponse` for tool call support #687

Uh oh!

mongodben commented Apr 28, 2025 •

edited

Loading

Uh oh!

nlarew left a comment

Uh oh!

nlarew Apr 28, 2025

Uh oh!

nlarew Apr 28, 2025

Uh oh!

mongodben Apr 28, 2025

Uh oh!

nlarew Apr 28, 2025

Uh oh!

mongodben Apr 28, 2025

Uh oh!

Uh oh!

Uh oh!

(EAI-988): Refactor GenerateResponse for tool call support #687

(EAI-988): Refactor GenerateResponse for tool call support #687

Uh oh!

Conversation

mongodben commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Notes

Uh oh!

nlarew left a comment

Choose a reason for hiding this comment

Uh oh!

nlarew Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

nlarew Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

mongodben Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

nlarew Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

mongodben Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

(EAI-988): Refactor `GenerateResponse` for tool call support #687

(EAI-988): Refactor `GenerateResponse` for tool call support #687

mongodben commented Apr 28, 2025 •

edited

Loading