Skip to content

Improve hook ergonomics for structured tool failures and call-scoped state #2094

Description

@gold-silver-copper

Motivation

Work on #2092, based on Erik Tews's tool-failure use case, validated the current architectural boundary:

  • tools report structured facts;
  • hooks decide whether the agent continues or terminates;
  • StepEvent::ToolResult exposes structured outcomes and typed extensions;
  • the scratchpad supports run-scoped cooperation between hooks.

The example works with the current API, but implementing and testing it exposed several opportunities to make this pattern safer and easier for users.

Potential improvements

1. Make typed metadata easier to attach to tool errors

Today, attaching ToolResultExtensions to a failed tool execution requires overriding Tool::call_structured, manually calling classify_error, and returning Ok(ToolReturn::failed(...)) with extensions.

Consider a richer error-classification API that can provide both the standard ToolFailure and typed result extensions. Possible shapes include:

  • an additional error_extensions(&Self::Error) -> ToolResultExtensions method; or
  • a richer classification/report type containing the failure and extensions.

The goal is to let a tool report machine-readable error context without taking responsibility for agent lifecycle decisions.

2. Provide per-tool-call scratchpad state

The scratchpad is keyed by type and shared across the run. When tool calls execute concurrently, users must build and correctly maintain their own map keyed by internal_call_id.

A call-scoped facility conceptually similar to:

ctx.scratchpad().for_call(internal_call_id)

could make the safe pattern easier and prevent implementations that accidentally use the latest unrelated result. Any design should preserve the documented concurrency semantics and avoid relying on tool completion order.

3. Add a hook testing harness to test-utils

HookContext is intentionally constructed by the runner and cannot be created by downstream tests. As a result, testing custom hooks and scratchpad cooperation requires driving an entire agent with MockCompletionModel.

A focused test harness could support:

  • setting run metadata and the current turn;
  • dispatching selected StepEvents;
  • registering a HookStack in order;
  • inspecting the resulting Flow and scratchpad state;
  • exercising concurrent/correlated tool-call scenarios.

Full mock-runner tests would remain valuable, but simple hook policies would become easier to test directly.

4. Clarify when to use ToolResult data directly versus the scratchpad

A hook handling StepEvent::ToolResult can inspect outcome and extensions directly. The scratchpad is primarily useful when:

  • multiple hooks cooperate;
  • state must survive across different event types;
  • failures or budgets accumulate across a run.

Documentation and examples should make this distinction explicit so users do not copy data into the scratchpad when the same hook can act on the event directly.

5. Highlight per-event hook ordering versus cross-call concurrency

For a single ToolResult, hooks run in registration order. With concurrent tools, events from different calls may interleave while sharing the same scratchpad.

This distinction is already documented, but it is easy to implement an unsafe policy such as reading ledger.last(). A prominent example should demonstrate correlation by internal_call_id and explain that registration order does not imply global ordering across calls.

6. Consider graceful completion separately from cancellation

Flow::Terminate currently surfaces as PromptError::PromptCancelled, which is appropriate for aborting after a fatal tool failure. Some applications may instead want to stop the loop successfully with an application-provided final result or explanation.

A distinct concept such as Flow::Finish may be worth discussing. This is a broader product/API decision and should not be coupled to the other ergonomic improvements without considering:

  • streaming and non-streaming parity;
  • transcript persistence;
  • final response types;
  • hook-stack short-circuiting;
  • telemetry semantics.

Suggested prioritization

The most immediately actionable items appear to be:

  1. structured extensions on classified errors;
  2. safer call-scoped scratchpad ergonomics;
  3. a downstream hook testing harness.

The documentation changes can accompany those improvements. Graceful completion should be evaluated independently as a larger API decision.

Reference

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions