Correct way to implement hindsight into multi agents system

[djanito]

I’ve spent a lot of time reviewing the Hindsight documentation as well as the recipes, but they don’t exactly match my current use case.

To put it simply, the user will chat through an interface; their query will be sent to an orchestrator agent (built with Google ADK), which will dynamically route the request to sub-agents (still adk) discovered from a LiteLLM A2A agents registry. The request will then be handled by the appropriate sub-agent, and the user will receive a response.

However, I have some questions regarding the correct use of Hindsight:

• Should I create one bank per user, even if it will be a big project or I should create one bank for all users but filter based on a user_id ?
• Is it a good practice if each sub agent have the hindsight MCP server to allow them to retrieve information (read-only) ?
• Should the orchestrator agent be the only one to be able to update memory ?

I created this sequence diagram to illustrator my understanding of integrating hindsight into my project:

sequenceDiagram
    autonumber
    participant U as User
    participant O as Orchestrator (ADK)
    participant H_U as Hindsight (User Bank)
    participant H_S as Hindsight (Shared Bank)
    participant R as LiteLLM Registry
    participant A as FinOps Agent

    Note over U, O: 1. User asks question
    U->>O: "Why is my bill high?" (Session includes user_id)
    
    Note over O, H_U: 2. Dual-Bank Recall (Parallel)
    par Get User Context
        O->>H_U: recall(query="bill high", bank="user-123")
        H_U-->>O: "User has project 'Alpha'"
    and Get Docs
        O->>H_S: recall(query="bill high", bank="global")
        H_S-->>O: "Billing FAQ docs"
    end
    
    Note over O, R: 3. Routing
    O->>O: Reasoning: "Intent is FinOps"
    O->>R: Find Agent("FinOps")
    R-->>O: Agent Endpoint
    
    Note over O, A: 4. Delegation
    O->>A: Send Task + MEMORY CONTEXT

    
    Note over O, U: 6. Response & Retention
    O-->>U: Display Answer
    O->>H_U: retain(content="User asked about bill...", bank="user-123")

Loading

Is my implementation correct or they are things that I can optimize ?

[nicoloboschi]

Thanks for trying out Hindsight

Should I create one bank per user, even if it will be a big project or I should create one bank for all users but filter based on a user_id ?

in general yes, but it depends on how you want to use Hindsight. Hindsight can be used in 2 different ways:

1. Personalized Memory for users - the agent uses one bank per user, storing via retain() all the conversations + recall() to augment and personalize the response
2. Learning Platform - the agent is aware of some knowledge base (your product or the topic of the conversations happening in your product) and it ingests all users conversations - via reflect() it can generate conjecture and offload most of the reasoning from your agent to the platform.

You can also implement both at the same time via tags - but in your use case I'd start with the case 1. So one bank per user.

Is it a good practice if each sub agent have the hindsight MCP server to allow them to retrieve information (read-only) ?

This depends on the purpose of your sub-agents system. Did you split them into sub-agents because they have different goals/features or for horizontal scaling them?
Depending on their purpose, I'd assume the MCP (via tools) is a good help for your system, exposing recall tools for the working user (so the working bank_id)

Should the orchestrator agent be the only one to be able to update memory ?

I see this as good practice if you only store user conversations.

Feel free to reach out to me on Slack to chat more about this! https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg