Skip to content

Expose image generation to every Akeru bot #450

Description

@linear-code

Problem

A bot should be able to make an image whatever model it chats with. Without one Akeru-native tool, a Claude bot could not use a ChatGPT subscription for images, and each provider would expose its own tool names to the bot.

Most of this shipped in #322 (feat: land the Akeru roadmap stack), with later changes in #337, #343, and #353. The remaining gaps are content exclusion, user-confirmed consent for cross-provider edits, and real-subscription proof.

Goal

Every Akeru bot can generate an image through one typed Akeru-native tool, even when its chat model comes from another provider. Each call routes through the bot's own image provider, or the global Image generation default from AKR-70's provider contract. The bot never sees provider-specific tool names.

Required behavior:

  • Generate an image from a prompt
  • Edit supplied images only when the selected provider supports editing
  • Show a clear running state while the image is made
  • Support cancellation and a bounded timeout
  • Use the configured fallback only for availability or provider failures, never for invalid requests
  • Save finished images as local Akeru artifacts and render them in bot and group chats
  • Offer open, save, copy, and reveal actions
  • Keep the image after a reload
  • Keep the provider's original response out of chat and persistence
  • Count provider usage against the calling bot when the provider reports it
  • Never send generated images, prompts, or input images to analytics, observational memory, entity memory, logs, or feedback
  • Never send an input image to a fallback provider silently. Ask before crossing to another provider when an edit includes user images.

Reuse the existing artifact, attachment, rich chat, tool approval, usage, and provider health paths. Do not build a second file store.

Acceptance criteria

Done on main (code audit, 2026-10-06):

  • One typed tool, GenerateImage in packages/contracts/src/akeruTools/catalog.ts, with the request schema ImageGenerationRequest in packages/contracts/src/imageGeneration.ts. Mastra drops the duplicate MCP generate_image tool.
  • Routing order is explicit provider, then bot override, then global default, then fallback order (imageRoutePlan and routeImageRequest in apps/server/src/image-generation/router.ts). An explicitly named provider never falls back.
  • Generate and capability-gated edit for ChatGPT and Grok subscriptions (chatGptImageAdapter, grokImageAdapter in adapters.ts). ChatGPT takes up to four edit images, Grok takes one.
  • The chat shows the tool call as running
  • Stopping the turn cancels the request. Each provider attempt times out after 150 seconds (IMAGE_REQUEST_TIMEOUT).
  • Fallback only for availability, revoked credentials, provider failure, and timeout (IMAGE_FALLBACK_FAILURE_KINDS). Invalid and unsupported requests stop.
  • Images are saved as artifacts and posted as attachment messages that survive reload and restart, in direct and group chats (BotMessageAttachments on web)
  • Open, save, copy, and reveal in ExpandedImageDialog. Reveal appears only when the environment supports a file manager.
  • Provider responses are normalized, and the bot gets a short summary, never the image data
  • Usage is recorded against the responding bot, including in groups. Providers that report no tokens record zero.
  • Tests in ImageGenerationRuntime*.test.ts, router.test.ts, and AgentController.images.catalog.test.ts cover a Claude bot using ChatGPT, a Codex bot using Grok, default and explicit routing, safe fallback, the needs-consent result, cancellation, restart, persistence, group posting and usage, and runtime content exclusion

Still open:

  • Prompts and images stay out of every listed sink. The runtime's own events and usage omit them, but the generic tool events in agentController/Events.ts publish tool arguments, including the prompt, and AkeruPassiveObservationalMemoryProcessor (apps/server/src/provider/mastra/AkeruMemory.ts) stores new messages without filtering image tool calls. Add exclusion and tests covering analytics, observational memory, entity memory, logs, and feedback.
  • Crossing providers during an edit needs the user's confirmation, not only the model's. Today routeImageRequest returns needs-consent, and the bot may retry with allowProvider, but nothing records that the user actually agreed.
  • An end-to-end test renders a finished image in a group chat
  • One real generation succeeds through each supported subscription path (ChatGPT and Grok) before this is called shipped

How to verify

vp test run apps/server/src/image-generation

Then add the exclusion and consent tests above. Finally, in an isolated dev environment with real ChatGPT and Grok subscriptions, have a Claude bot make an image with ChatGPT and a Codex bot make one with Grok, edit an attached image, stop one request mid-run, reload, and check the images in a direct chat and a group chat at desktop and mobile widths.

Out of scope

  • The first-use consent card before any image request reaches a vendor (AKR-19)
  • Image settings in the chat composer
  • API-key image providers

Context

  • Docs: docs/user/image-generation.md, docs/internals/image-generation.md.
  • Settings: ImageGenerationSettingsPanel and the bot editor's image provider picker on web, and the read-only ImageGenerationSummary on mobile.
  • Blocked by AKR-70 (provider contract) and AKR-69 (Image generation settings), both done. Related: AKR-79 (the complete Akeru tool catalog through Mastra), AKR-84, AKR-90.

Created with Claude Opus 5.5 in Claude Code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions