vision-agent

A small vision agent that solves perception tasks by calling a sequence of vision tools. You give it a task in plain English and an image, and it figures out which tools to run, runs them in order, threads the intermediate results from one tool into the next, and hands back a structured answer along with the full trajectory of what it did.

The whole thing runs offline on CPU. There are no model downloads, no API keys, and no network calls anywhere in the path. That is deliberate: the tools operate on synthetic images made of solid rectangles and a tiny built in bitmap font, which is enough to exercise a genuine detect, crop, classify, read pipeline without dragging in heavy weights.

What is real and what is a stub

The architecture is real. The agent loop, the tool registry, the argument threading between steps, and every tool's algorithm are all working code, not placeholders.

The one piece that stands in for something heavier is the planner. In a production agent a language model would read the task and the tool signatures and decide the next call. Here a deterministic rule based planner plays that exact role so the tests stay reproducible and fast. The agent loop does not know the planner is rule based, so swapping in a real model later is a single class replacement.

The tools

Each tool returns a uniform ToolResult envelope and declares its inputs and what it produces, so the agent discovers everything through the registry and never hard codes a tool list.

• detector finds solid blobs using connected component labeling and returns a real bounding box and a fill based confidence for each one.
• cropper slices a sub image out of a bounding box, clamping the box to the image bounds.
• classifier names the dominant shape in a crop (square, tall rectangle, wide rectangle, or empty) from the aspect ratio of its filled region.
• ocr reads a short digit string by segmenting the crop into glyph columns and matching each against a built in 5 by 3 font using Hamming distance. It is a genuine template decoder, no language model and no external OCR engine.

How the loop works

1. The planner classifies the task intent (count, classify, or read) and emits a list of tool calls. Argument values can be symbolic references like $input_image, $detection[0].box, $detection_union.box, or $last_crop.
2. The agent runs each step, resolving those references against the state built up so far, and records every call as an Observation.
3. After the plan finishes the agent folds the trajectory into a structured answer keyed by intent.

A counting task runs the detector alone. A classify task runs detect then crop then classify on the first object. A read task detects every glyph, crops the union of all detections so the OCR tool sees the whole string, then reads it.

Layout

src/vision_agent/
  types.py       core dataclasses: BBox, Detection, ToolResult, Observation, AgentResult
  tools.py       the four tools plus the bitmap font and renderer
  registry.py    the tool registry
  planner.py     the deterministic planner that stands in for a language model
  agent.py       the agent loop and answer assembly
  synthetic.py   generators for synthetic scenes and digit images
tests/           pytest behavior tests for tools, planner, and agent

Running it

Install the dependencies and run the suite.

pip install -r requirements.txt
pytest tests/ -q

A quick taste in code:

import numpy as np
from vision_agent import VisionAgent
from vision_agent.synthetic import scene_with_n_objects, shape_image, digit_image

agent = VisionAgent()

print(agent.run("how many objects are there", scene_with_n_objects(3)).answer)
print(agent.run("classify the shape", shape_image("wide")).answer)
print(agent.run("read the number", digit_image("42")).answer)

Tests

The suite has 45 tests covering the tools in isolation, the planner intent mapping and plan shapes, and the agent end to end on synthetic inputs. They are behavior checks: the detector counts the right number of separated objects, the classifier tells the three shapes apart and stays deterministic, the OCR reads back several rendered digit strings, and the agent selects the correct tool sequence for each task and returns the expected structured result. The full run passes on CPU in well under a second.