Enhance Geospatial Tool Usage #351
Conversation
This commit introduces a two-stage prompting strategy to improve the reliability of geospatial tool usage. The `taskManager` agent now categorizes user queries into 'geospatial', 'web_search', or 'general'. This classification is then passed to the `researcher` agent, which uses it to tailor its system prompt. For geospatial queries, a more directive prompt is used to ensure the `geospatialQueryTool` is prioritized, addressing the issue of underutilization.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the You can disable this status message by setting the WalkthroughThis PR introduces a category classification system to the agent workflow. Query categorization ('geospatial', 'web_search', 'general') is added to the task-manager's decision logic, passed through the action schema, and used by the researcher agent to conditionally apply category-specific prompts. Additionally, image detection in task-manager short-circuits processing, and import paths are normalized. Changes
Sequence DiagramsequenceDiagram
participant User
participant TaskManager
participant Researcher
User->>TaskManager: Submit query (with/without image)
alt Image detected
TaskManager->>TaskManager: Early return
TaskManager-->>User: next: 'proceed'
else No image
TaskManager->>TaskManager: Classify into category<br/>(geospatial|web_search|general)
TaskManager->>Researcher: Call with category
alt Category is 'geospatial'
Researcher->>Researcher: Append geospatial prompt<br/>to system message
end
Researcher->>Researcher: Process with<br/>category-informed system prompt
Researcher-->>TaskManager: Results
TaskManager-->>User: Decision (proceed|inquire)
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
Comment |
PR Compliance Guide 🔍Below is a summary of compliance checks for this PR:
Compliance status legend🟢 - Fully Compliant🟡 - Partial Compliant 🔴 - Not Compliant ⚪ - Requires Further Human Verification 🏷️ - Compliance label |
||||||||||||||||||
PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||||
|
@coderabbitai review this |
![charliecreates[bot]](https://avatars.githubusercontent.com/in/1140780?s=60&v=4){kind=small}
There was a problem hiding this comment.
Primary concern: the overwrite of action.object can drop the default category, undermining the new two-stage flow. The geospatial directive is too strict and may reduce answer quality for mixed queries; prefer prioritization with fallback. Using any for action weakens guarantees around critical control-flow data. Consider making category required in the schema to align with the classification instructions.
Additional notes (1)
- Maintainability |
app/actions.tsx:291-291
Overwritingaction.objecthere can drop the defaultcategory: 'general'you set earlier iftaskManagerreturns an object withoutcategory(e.g., the image-bypass path). That undermines the two-stage flow becauseresearchermay receiveundefinedand miss the geospatial directive. Merge the new fields instead of replacing to preserve defaults.
Summary of changes
Summary of Changes
- Added a
categoryfield with default'general'to the initialaction.objectinapp/actions.tsx. - Passed the
categoryfromtaskManagertoresearcherin thesubmitloop. - Updated
researcheragent to accept an optionalcategoryparameter and appended a new, directive geospatial system prompt whencategory === 'geospatial'. - Rewrote the
taskManagersystem prompt to introduce a two-stage strategy: classify the query (geospatial,web_search,general) and then decide (proceed/inquire). - Extended
nextActionSchemato include an optionalcategoryenum. - Fixed an import path in
mapbox_mcp/hooks.tsto use the alias@/lib/utilsinstead ofQCX/lib/utils.
Related Files
- app/actions.tsx
- lib/agents/researcher.tsx
- lib/agents/task-manager.tsx
- lib/schema/next-action.tsx
- mapbox_mcp/hooks.ts
|
|
||
| async function processEvents() { | ||
| let action: any = { object: { next: 'proceed' } } | ||
| let action: any = { object: { next: 'proceed', category: 'general' } } |
There was a problem hiding this comment.
action is typed as any, which weakens guarantees right where critical control-flow fields (next, category) are introduced. This makes it easier for unintended shapes to slip through and complicates future refactors.
Suggestion
Introduce a minimal local type for the shape you rely on and avoid any:
type NextActionObject = { next: 'inquire' | 'proceed'; category?: 'geospatial' | 'web_search' | 'general' };
let action: { object: NextActionObject } = { object: { next: 'proceed', category: 'general' } };(Alternatively, derive the type from your Zod schema and use that.)
Reply with "@CharlieHelps yes please" if you'd like me to add a commit with this change.
| const geospatial_prompt = `The user's query has been identified as geospatial. | ||
| You MUST use the 'geospatialQueryTool' to answer this question. | ||
| Do not use any other tools. If the query cannot be answered with the geospatial tool, respond that you are unable to answer.`; | ||
|
|
||
| const systemToUse = dynamicSystemPrompt && dynamicSystemPrompt.trim() !== '' ? dynamicSystemPrompt : default_system_prompt; | ||
|
|
||
| let systemToUse = dynamicSystemPrompt && dynamicSystemPrompt.trim() !== '' ? dynamicSystemPrompt : default_system_prompt; | ||
| if (category === 'geospatial') { | ||
| systemToUse = `${systemToUse}\n\n${geospatial_prompt}`; | ||
| } |
There was a problem hiding this comment.
The geospatial directive is overly restrictive: "Do not use any other tools" and "respond that you are unable to answer" if the geospatial tool can’t handle it. This can degrade UX for mixed queries that are primarily geospatial but still require supplemental facts (e.g., POI details) from web search. Prefer a "prioritize geospatial" policy with graceful fallback.
Suggestion
Relax the directive to allow fallback when needed while still prioritizing the geospatial tool:
const geospatial_prompt = `The user's query has been identified as geospatial.
You MUST prioritize using the 'geospatialQueryTool' to answer this question first.
If the geospatial tool is insufficient for a subtask (e.g., factual details or current info), you MAY use other available tools to complete the request.
When you use 'geospatialQueryTool', do not describe how the map will change; provide the textual answer and trust the map will update appropriately.`;Reply with "@CharlieHelps yes please" if you'd like me to add a commit with this change.
| next: z.enum(['inquire', 'proceed']), // "generate_ui" | ||
| category: z.enum(['geospatial', 'web_search', 'general']).optional() |
There was a problem hiding this comment.
You instruct the model to classify every query, but the schema makes category optional. That weakens the guarantee that the researcher will receive the classification and can lead to inconsistent behavior. Align the schema with the prompt by requiring category.
Suggestion
Make category required:
export const nextActionSchema = z.object({
next: z.enum(['inquire', 'proceed']),
category: z.enum(['geospatial', 'web_search', 'general'])
});Note: This will require ensuring all taskManager return paths include a category (e.g., default to 'general' in bypass paths).
Reply with "@CharlieHelps yes please" if you'd like me to add a commit with this change.
|
@ngoiyaeric: I'll review the changes in this PR for you. The implementation looks like an interesting enhancement to improve geospatial query handling through a two-stage prompting strategy. Let me analyze the code changes in detail. ✅ Actions performedReview triggered.
|
This commit introduces a new image search functionality and refactors the agent logic to improve tool selection.
- A new `imageSearchTool` is added, using the Serper API to fetch image results from the web.
- A corresponding `ImageSearchSection` component is created to display the image results in a grid format in the chat UI.
- The `taskManager` agent is updated to classify user queries into categories ('geospatial', 'web_search', 'general').
- The `researcher` agent now uses this classification to tailor its system prompt, encouraging the use of the new image search tool for visual queries and improving the reliability of geospatial tool usage.
2 participants
User description
This change enhances the AI's ability to identify and respond to geospatial queries by implementing a two-stage prompting strategy. The first stage categorizes the query, and the second stage uses that category to tailor the system prompt for the research agent, ensuring the correct tools are used for the job. This should result in more frequent and accurate use of geospatial tools and a better user experience.
PR created automatically by Jules for task 15709345609338375841
PR Type
Enhancement
Description
Implements two-stage prompting strategy for geospatial query classification
Adds query categorization into geospatial, web_search, or general types
Passes category to researcher agent to tailor system prompts
Enhances geospatial tool prioritization with directive prompts
Fixes import path from QCX to standard alias
Diagram Walkthrough
File Walkthrough
hooks.ts
Fix import path aliasmapbox_mcp/hooks.ts
QCX/lib/utilsto@/lib/utilsactions.tsx
Pass query category to researcher agentapp/actions.tsx
category: 'general'action.object.categoryto researcher functionresearcher.tsx
Tailor system prompt based on query categorylib/agents/researcher.tsx
categoryparameter to function signature with type'geospatial'| 'web_search' | 'general'geospatial_promptthat enforces geospatialQueryTool usageis geospatial
systemToUsefrom const to let for conditional modificationtask-manager.tsx
Implement query categorization in task managerlib/agents/task-manager.tsx
web_search, or general
next-action.tsx
Add category field to action schemalib/schema/next-action.tsx
categoryfield to schema as optional enum with three valuesSummary by CodeRabbit