Tool visualization: factory-based tool render with inline inspector #4

Activity

1. JoshuaRowePhantom

   added

   enhancementNew feature or requestNew feature or request

   on Jun 24, 2026
2. JoshuaRowePhantom commented on Jun 25, 2026

   

   JoshuaRowePhantom

   on Jun 25, 2026

   OwnerAuthor

   More actions

   Additional scope: tool-driven GUI status updates

   Some tools want to push a brief status string to the agent status line rather than (or in addition to) producing a chat content block. The canonical example is a report_intent-style tool: it takes a short description of what the agent is currently doing and updates a status field visible at a glance in the UI, without adding noise to the chat transcript.

   New IToolVisualizerFactory side-channel: StatusUpdate

   Extend the visualization result interpretation table with a new return type:

   Returned object	Behaviour
   StatusUpdate	Update the named status field on AgentChatStatusLineViewModel; produce no chat content block (or optionally a collapsed one).

   /// <summary>
   /// Signals that a tool call should update a named status-line field rather than (or in
   /// addition to) emitting a chat content block.
   /// </summary>
   public sealed record StatusUpdate(
       /// <summary>
       /// The well-known field name to update. See <see cref="AgentStatusField"/>.
       /// </summary>
       AgentStatusField Field,
   
       /// <summary>
       /// The new value, or null/empty to clear the field.
       /// </summary>
       string? Value,
   
       /// <summary>
       /// When non-null, a collapsed chat content block is still emitted with this summary label.
       /// When null, no content block is emitted.
       /// </summary>
       string? ChatSummary = null);
   
   public enum AgentStatusField
   {
       /// <summary>Short description of what the agent is currently doing (report_intent).</summary>
       Intent,
   }

   AgentChatStatusLineViewModel changes

   Add a new IntentDisplay property (nullable string) populated from StatusUpdate(Intent, ...) results:

   • Shown in the status line between the brain icon and the model/provider fields.
   • Cleared automatically when IsThinking transitions to false (i.e. the run completes).
   • Shown in a distinct muted style (e.g. italic, dimmed) to distinguish it from model/provider metadata.

   ToolVisualizationContext changes

   No changes needed — StatusUpdate is a pure output type returned by Visualize.

   Routing StatusUpdate through the rendering pipeline

   ChatMessageHtmlModel.Render (or the transformer) calls IToolVisualizerFactory.Visualize for each content block. When the result is a StatusUpdate:

   1. If ChatSummary is non-null, emit a collapsed <details> block (same as the existing generic tool-call fallback) with ChatSummary as the label.
   2. Post the field update to AgentChatStatusLineViewModel via a new IAgentStatusSink interface injected into ChatOutputHtmlModel — the same sink pattern already used by IChatOutputHtmlSink.

   public interface IAgentStatusSink
   {
       void UpdateStatus(AgentStatusField field, string? value);
   }

   AgentChatOutputControl (or AgentViewModel) implements IAgentStatusSink and forwards to AgentChatStatusLineViewModel on the UI thread.

   report_intent tool visualizer

   Add a ReportIntentToolVisualizer inside CopilotToolVisualizerFactory (or a dedicated BuiltinToolVisualizerFactory) that matches:

   • FunctionCallContent where Name == "report_intent" and arguments contain "intent" string
   • Returns StatusUpdate(AgentStatusField.Intent, intentText, ChatSummary: null) — updates the status line silently with no chat block
   • FunctionResultContent for the matching call — returns null (no rendering needed)

   This means report_intent calls become completely invisible in the chat transcript but their current value is always live in the status line while the agent is running.
3. JoshuaRowePhantom

   added

   next-up

   on Jun 27, 2026
4. JoshuaRowePhantom commented on Jun 27, 2026

   

   JoshuaRowePhantom

   on Jun 27, 2026

   OwnerAuthor

   More actions

   Design

   Approach: Introduce a factory-based visualization layer for tool calls/results, with a status-line side-channel for silent status updates.

   Architecture

   1. Core contracts in ViewModels/Visualization/:

      • IToolVisualizerFactory with Visualize(ToolVisualizationContext) returning object?
      • ToolVisualizationContext(AIContent Content)
      • Summary(string Label, string? HtmlBody = null) - renders as <details open>
      • StatusUpdate(AgentStatusField Field, string? Value, string? ChatSummary = null)
      • CompositeToolVisualizerFactory, ToolVisualizationInterpreter
      • WorkspaceVisualizerFactory, CopilotToolVisualizerFactory

   2. IAgentStatusSink injected into ChatOutputHtmlModel; AgentChatOutputControl implements it

   3. AgentChatStatusLineViewModel gains IntentDisplay (nullable string), cleared when IsThinking transitions to false

   4. ChatOutputHtmlRenderer.RenderContent delegates tool content to ToolVisualizationInterpreter

   5. Inspector affordance: every content block HTML includes a small inspect button; clicking posts JS message; host opens AIContentInspectorWindow
5. JoshuaRowePhantom commented on Jun 27, 2026

   

   JoshuaRowePhantom

   on Jun 27, 2026

   OwnerAuthor

   More actions

   Tests

   Phantom.Workspaces.Agent.Gui.Tests

   1. CompositeToolVisualizerFactory_ReturnsFirstNonNull

      • Three factories; first returns null, second returns Summary, third returns null → result is Summary

   2. CompositeToolVisualizerFactory_ReturnsNull_WhenAllReturnNull

      • All factories return null → composite returns null

   3. CompositeToolVisualizerFactory_RespectsOrder

      • First factory returns a value → used (second factory not consulted)

   4. ToolVisualizationInterpreter_Summary_RendersDetailsElement

      • Summary("label", "<b>body</b>") → HTML contains <details open, <summary>label</summary>, body

   5. ToolVisualizationInterpreter_NullResult_UsesGenericCollapsibleFallback

      • Factory returns null for FunctionCallContent → same output as old generic collapsible

   6. ToolVisualizationInterpreter_StatusUpdate_PostsToSink

      • Factory returns StatusUpdate(Intent, "doing something", null) → sink receives UpdateStatus(Intent, "doing something"), no HTML emitted

   7. ToolVisualizationInterpreter_StatusUpdate_WithChatSummary_EmitsCollapsedDetailsAndPostsToSink

      • Factory returns StatusUpdate(Intent, "val", "summary label") → HTML is collapsed <details>, sink also receives update

   8. CopilotToolVisualizerFactory_ReportIntent_FunctionCall_ReturnsStatusUpdate

      • FunctionCallContent("call-1", "report_intent", {"intent": "analyzing code"}) → StatusUpdate(Intent, "analyzing code", null)

   9. CopilotToolVisualizerFactory_ReportIntent_FunctionResult_ReturnsNull

      • FunctionResultContent("call-1", "ok") from a report_intent call → null

   10. AgentChatStatusLineViewModel_IntentDisplay_SetByUpdateStatus

       • StatusUpdate sets IntentDisplay to given value

   11. AgentChatStatusLineViewModel_IntentDisplay_ClearedWhenThinkingStops

       • Set IntentDisplay, then complete running item → IntentDisplay becomes null

   12. ChatOutputHtmlRenderer_InspectorAffordance_PresentOnFunctionCallBlock

       • RenderContent for FunctionCallContent includes chat-inspect button

   13. ChatOutputHtmlModel_InspectMessage_HandledByBridge

       • Simulate inspect JS message → AIContentInspectorWindow open request raised
6. JoshuaRowePhantom

   closed this as completedon Jun 27, 2026
7. JoshuaRowePhantom

   added a commit that references this issue on Jun 27, 2026

   Fix #4: Factory-based tool visualization with inline inspector

   ...

   This issue will close once commit 5dae2ed is merged into the 'main' branch.

   5dae2ed
8. JoshuaRowePhantom commented on Jun 30, 2026

   

   JoshuaRowePhantom

   on Jun 30, 2026

   OwnerAuthor

   More actions

   Verification failed

   Checked: Commit 5dae2ed (Fix #4), commit ed49564 (Fix #293), files on features branch tip:

   • ViewModels/Visualization/: IToolVisualizerFactory, CompositeToolVisualizerFactory, ToolVisualizationContext, Summary, StatusUpdate, ToolVisualizationInterpreter, WorkspaceVisualizerFactory, CopilotToolVisualizerFactory
   • ViewModels/IAgentStatusSink.cs, ViewModels/AgentChatStatusLineViewModel.cs (IntentDisplay)
   • ViewModels/DocumentModels/ChatOutputHtmlRenderer.cs (factory delegation)
   • Controls/AIContentInspectorWindow.axaml.cs
   • Controls/AgentChatOutputControl.axaml.cs
   • Agent.Gui.Tests/ToolVisualizationTests.cs

   Found: The factory-based visualization system is fully implemented and tested:

   • IToolVisualizerFactory, CompositeToolVisualizerFactory, ToolVisualizationContext, Summary, StatusUpdate, AgentStatusField
   • ToolVisualizationInterpreter routes factory results to HTML or the status sink
   • WorkspaceVisualizerFactory and CopilotToolVisualizerFactory (including report_intent -> StatusUpdate(Intent))
   • AgentChatStatusLineViewModel.IntentDisplay set by UpdateStatus, cleared when IsThinking -> false
   • ChatOutputHtmlRenderer.RenderContent delegates FunctionCallContent/FunctionResultContent through the factory
   • AIContentInspectorWindow exists (window class and AXAML)
   • InspectorRequested event declared and fired in AgentChatOutputControl for JS inspect messages
   • All factory dispatch, interpreter, and status-line tests present and green

   Missing: Acceptance criterion 3 — Inspector affordance renders on every content block; clicking it opens the full inspector with metadata + raw payload — is not met:

   1. The chat-inspect HTML button was removed by issue Bug: old inline '...' button still visible after #155 - should have been removed #293 (ed49564: "remove old inline inspector affordance button"). The JS inspect message is therefore never sent from the WebView.
   2. AgentChatOutputControl.InspectorRequested has no subscriber anywhere in the codebase — AgentViewModel does not wire it up, so even if the JS message were sent, the AIContentInspectorWindow would never be opened.
   3. Test ChatOutputHtmlModel_InspectMessage_HandledByBridge (specified in the issue test plan) is absent.

   The AIContentInspectorWindow and the InspectorRequested event declaration are orphaned code with no working trigger.
9. JoshuaRowePhantom

   added

   failed-verificationBug failed automated verificationBug failed automated verification

   on Jun 30, 2026
10. JoshuaRowePhantom

    removed

    next-up

    on Jul 17, 2026