Add system prompt to clarify the scope of session-sql tool

[ravikiranvm]

Fixes OPS-3138

Additional Notes

Although it's not happening most of the times, LLM once included session-sql tool in the tool selection output even for a simple query like do you have access to my tables?

I've checked the description of this tool:

The description is kinda generic and may not be clear for LLM to use it correctly in our app. So I've modified the system prompt to clarify the scope of this tool

---

Alternative fix: Do we really need this session-sql tool? Can we exclude it from the tools selection?

[linear]

OPS-3138 AI invokes AWS mcp when asked about Tables

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch ops-3138

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR adds clarification to the system prompt about the specific scope of the session-sql tool to prevent LLM from incorrectly selecting it for general database queries. The tool is meant exclusively for AWS cost/billing MCP tool-generated tables and temporary analysis, not for user database queries.

• Added explicit guidance in the system prompt defining when the session-sql tool should be used

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[greptile-apps]

Greptile Overview

Greptile Summary

Added clarifying note to the LLM system prompt to prevent incorrect tool selection for the session-sql tool. The note specifies that session-sql is only for AWS cost/billing MCP-created tables, not for general user database queries.

• Addresses issue where LLM occasionally selected session-sql for general database queries
• Improves tool routing accuracy by explicitly defining the scope of session-sql
• Minor grammatical suggestion: "user's" → "the user's"

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The change is a simple, targeted addition to a system prompt that clarifies tool usage scope. It's a single line addition that improves LLM decision-making without affecting logic or introducing side effects. The change addresses a real issue observed in production and follows the existing pattern of prompt engineering in the codebase.
• No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
packages/server/api/src/app/ai/mcp/llm-query-router.ts	5/5	Added clarifying note to system prompt about session-sql tool scope to prevent incorrect LLM tool selection for general database queries

Sequence Diagram

sequenceDiagram
    participant User
    participant LLM as LLM Query Router
    participant SystemPrompt as getSystemPrompt()
    participant LLMModel as Language Model

    User->>LLM: routeQuery(messages, tools, ...)
    LLM->>SystemPrompt: Generate system prompt
    Note over SystemPrompt: Adds clarifying note:<br/>"session-sql tool is ONLY<br/>for AWS cost/billing tables"
    SystemPrompt-->>LLM: Return enhanced prompt
    LLM->>LLMModel: generateObject(system prompt + messages)
    LLMModel-->>LLM: Returns tool_names & classification
    Note over LLMModel: Now correctly excludes<br/>session-sql for general<br/>database queries
    LLM-->>User: Selected tools & classification

Loading

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

style: Grammatical issue: "user's" should be "the user's" for clarity

Suggested change

	"NOTE: The 'session-sql' tool is ONLY for querying tables created by AWS cost/billing MCP tools and performing temporary analysis. It is NOT for user's database or table related queries. " +
	"NOTE: The 'session-sql' tool is ONLY for querying tables created by AWS cost/billing MCP tools and performing temporary analysis. It is NOT for the user's database or table related queries. " +

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/server/api/src/app/ai/mcp/llm-query-router.ts
Line: 210:210

Comment:
**style:** Grammatical issue: "user's" should be "the user's" for clarity

```suggestion
    "NOTE: The 'session-sql' tool is ONLY for querying tables created by AWS cost/billing MCP tools and performing temporary analysis. It is NOT for the user's database or table related queries. " +
```

How can I resolve this? If you propose a fix, please make it concise.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rita-gorokhod]

I don't like this @ravikiranvm .
There might be other tools in the future that cause this problem.
I think a broader solution is needed, to say that certain tools do not mix with each other.
Please consult with Eran.

[ravikiranvm]

Okay, I will talk to Eran.