Skip to content

image-input.md

Latest commit

 

History

History
112 lines (82 loc) · 7.55 KB

image-input.md

File metadata and controls

112 lines (82 loc) · 7.55 KB
Raw

Image input and game perception

OpenGameAgent accepts images as durable, bounded input to an agent run. The framework does not bundle a vision model: use a cloud or local API model whose catalog entry declares image input.

Image understanding and generated media are separate paths. Screenshots, crops, minimaps, and tool-produced observations enter the agent as image input. Assets generated by an image, audio, or video model use OpenGameAgent.Media.

Recommended perception stack

Do not serialize every block, voxel, navmesh point, or screen pixel into a prompt. For a large world, give each NPC a layered observation:

  1. Structured local state: visible entities, relations, affordances, inventory, hazards, goals, game time, and authoritative IDs in bounded JSON.
  2. Sparse spatial view: a local BEV, occupancy grid, room graph, chunk summary, or topological map at the coarsest resolution that preserves the current decision.
  3. Selective images: a screenshot or crop when appearance, occlusion, terrain shape, an unknown object, or a player-created structure cannot be represented reliably by state alone.
  4. On-demand queries: tools such as inspect_entity, inspect_region, find_path, or measure_clearance for exact facts after the model selects a target.
  5. Deterministic execution: the model chooses an intent, target, or blueprint; ordinary game code performs exact pathfinding, placement, physics, collision, resource accounting, and animation.

This keeps the prompt semantic and bounded. A block-building NPC can decide what to build from a screenshot, nearby materials, and a coarse local map without receiving millions of coordinates. Once it selects a design, a game-owned blueprint or building tool handles the exact blocks.

Each actor should receive only its own visible scene. Apply fog-of-war, permissions, and secret filtering before creating JSON or image input. Capture images at decision boundaries or when the scene changed materially, not on every render frame. A game can route image-heavy decisions to a vision model and keep routine dialogue or deterministic ticks on a cheaper text model.

Durable attachment lifecycle

Inline image bytes are admitted before the run:

  • PNG, JPEG, WebP, and GIF are supported;
  • the complete batch is validated before any object is published;
  • real image decoding verifies the declared media type, dimensions, byte limit, and pixel limit;
  • the local store writes immutable SHA-256-addressed objects atomically;
  • canonical transcripts and save files contain only bounded attachment references;
  • bytes are resolved only immediately before a model request;
  • the active provider/model is preflighted before attachment bytes are loaded;
  • missing, corrupt, or mismatched objects fail closed;
  • a server read is authorized against the referenced session and actor before the store is touched.

Defaults are 5 MiB per image, 20 images per message, 100 MiB in aggregate, and 40 million decoded pixels per image. Configure lower limits for shipped games where appropriate.

Only user input and tool results become model-visible image history. A tool can return a screenshot as an inline BinaryContent; GameAgentRuntime persists it before the next model turn and before the session checkpoint. Local tool-progress subscribers may receive bounded ephemeral binary previews, as used by generated-media progress. Progress content is neither canonical history nor part of the stock public JSON/SSE projection; persist a final result when the image must survive or cross the server boundary.

In-process example

Install OpenGameAgent.Attachments.Local alongside the runtime, then mount one store below the game's save or application-data directory:

using OpenGameAgent;
using OpenGameAgent.Attachments;
using OpenGameAgent.Attachments.Local;
using OpenGameAgent.Kernel;

var attachments = new FileGameImageAttachmentStore(
    Path.Combine(saveRoot, "agent-attachments"));

var runtime = new GameAgentRuntime(new GameAgentRuntimeOptions(provider, model)
{
    ImageAttachments = attachments,
    ContextProvider = worldContext,
    ToolProvider = gameTools,
    SessionStore = sessionStore,
});

var screenshot = await File.ReadAllBytesAsync(framePath);
var input = new GameInput(
    sessionId: "save-42",
    actorId: "npc-builder",
    type: "scene_changed",
    payloadJson: """{"visibleEntities":["player","workbench"],"region":"base-east"}""",
    moment: new GameMoment("main", tick: 18420),
    inputId: "scene-18420-builder",
    content: new AgentContent[]
    {
        new BinaryContent(
            AgentMediaKind.Image,
            Convert.ToBase64String(screenshot),
            GameImageMediaTypes.Png,
            "builder-view.png"),
    });

var result = await runtime.RunAsync(input);

After admission, the runtime replaces the inline bytes with a GameImageAttachment. Replaying the session resolves the immutable object again; the transcript never stores base64 data.

Request-time projection

Admission limits protect storage; model-request limits protect latency and provider context. Configure the optional projector when a provider should receive resized or selectively omitted images while the authoritative transcript continues to reference the originals:

var runtime = new GameAgentRuntime(new GameAgentRuntimeOptions(provider, model)
{
    ImageAttachments = attachments,
    ImageRequestProjector = new SkiaGameImageRequestProjector(maximumCacheEntries: 128),
    ImageProjectionBudgetSelector = (_, _) => new ValueTask<GameImageProjectionBudget>(
        new GameImageProjectionBudget(
            maximumImages: 4,
            maximumTotalPixels: 8_000_000,
            maximumEncodedBytes: 8_000_000,
            maximumEdgePixels: 2048)),
});

SkiaGameImageRequestProjector preserves aspect ratio, derives WebP request objects, admits the newest images first under the configured count/pixel/byte budget, and replaces omitted images with stable text. Derived data is immutable and content addressed; the original object and transcript reference are never overwritten. Repeated projections reuse a bounded cache.

Subscribe to GameAgentExtensionEvents.ImagesProjected for source-to-request attachment IDs, disposition, transform ID, dimensions, and encoded size. GameModelContextProvenanceExtension persists the same relationship next to the exact model-request identity for replay and evaluation.

Server placement

The stock JSON/SSE server accepts inline image content in GameInput.content and uses the configured attachment directory. The client can call ServerGameAgentClient.ReadImageAttachmentAsync when an authorized UI needs to display an attachment referenced by that session. Multi-user hosts must install an identity-derived owner authorizer; an attachment ID alone grants no access.

Back up and restore session state and its attachment directory together. Immutable content-addressed objects may be shared by multiple references. Retention and orphan collection remain a save/storage policy: never delete an object merely because one transcript branch no longer displays it unless the host has enumerated all authoritative references.

Multi-NPC performance

Image admission does not change actor scheduling. Runs for the same (sessionId, actorId) remain serialized; different actors run concurrently up to GameRuntimeLimits.MaxConcurrentActors. Request projection runs off the engine's render-critical path. Deduplicate identical frames and apply actor importance/distance budgets before enqueuing runs. Shared-world writes still require game-owned revisions or transactions: per-actor ordering is not a global world lock.