Medical Imaging Annotation Tool

A self-hosted annotation tool for building medical imaging datasets: bounding boxes, polygon segmentation, dataset/project management, multi-format export, and real AI-assisted labeling (classical CV, runs fully offline/free).

---

---

Features

• Bounding boxes — click-drag to draw, per-class colors
• Segmentation (polygons) — click to place vertices, double-click to close
• Dataset management — multiple projects, image upload, per-image annotation status, live stats (images/annotated/annotation count/classes)
• Annotation export — COCO JSON, YOLO (txt + classes.txt), Pascal VOC (XML)
• AI-assisted labeling — select a rough bounding box and click AI Assist to refine it into a segmentation polygon using OpenCV GrabCut (color/texture-based) or a Canny edge/contour method (better for high-contrast grayscale X-rays). This is genuine, real computer vision that runs locally — no external API, no cost, no internet dependency.

Architecture

medical-imaging-annotator/
├── backend/
│   ├── main.py              # FastAPI app — routes + serves the frontend
│   ├── annotation_store.py  # filesystem-based project/image/annotation storage
│   ├── exporters.py         # COCO / YOLO / Pascal VOC export
│   ├── ai_assist.py         # GrabCut & edge-based segmentation refinement
│   └── requirements.txt
├── frontend/
│   └── index.html           # single-file canvas UI (vanilla JS, no build step)
└── data/                    # created at runtime — your datasets live here
    └── projects/<name>/{images,annotations,classes.json}

No database, no build tooling — projects and annotations are plain files on disk under data/, which makes it trivial to back up, version, or sync a dataset with git/rsync.

Setup

cd backend
pip install -r requirements.txt
uvicorn main:app --reload --port 8000

Open http://localhost:8000 in your browser — the backend serves the frontend directly, so there's nothing else to run.

Using it

1. Create a project (left sidebar).
2. Add label classes (name + color).
3. Upload one or more images.
4. Select Box tool and drag to draw; or Polygon tool and click points, double-click to close.
5. Select a box and click ✨ AI Assist to auto-refine it into a segmentation polygon — review and adjust the points, then Save.
6. Click Save annotations after each image.
7. Pick an export format and click Export dataset to download a zip.

Keyboard shortcuts: B box tool, P polygon tool, V select tool, Delete/Backspace remove selected annotation, Esc cancel in-progress shape.

Annotation schema (internal)

Each image's annotations are stored as:

{
  "image_id": "abcd1234.png",
  "width": 800,
  "height": 600,
  "annotations": [
    { "id": "a1", "type": "bbox", "class_id": 0, "bbox": [x, y, w, h] },
    { "id": "a2", "type": "polygon", "class_id": 1, "points": [[x1,y1], [x2,y2], ...] }
  ]
}

Coordinates are absolute pixel values, origin top-left. Note: YOLO and Pascal VOC exports use axis-aligned bounding boxes only (their standard formats), so polygons are exported as their enclosing box in those two formats — full polygon segmentation is preserved in the COCO export.

Extending it

• Real deep-learning AI-assist: swap/extend ai_assist.py to call a hosted Segment Anything (SAM) endpoint (e.g. Replicate, a self-hosted SAM server) when an API key/URL is configured, falling back to GrabCut otherwise — same pattern used in the Radiology Report Assistant project.
• Auth / multi-user: add a login layer in front of FastAPI if this needs to be shared across a team rather than run locally per-annotator.
• DICOM support: add pydicom to read .dcm files and convert to PNG on upload for annotation, storing the original alongside for reference.
• Model-in-the-loop: add an endpoint that runs a trained detector over unannotated images and pre-populates boxes for the annotator to correct (classic active-learning loop).

Disclaimer

This is a general-purpose annotation/dataset-building tool for engineering and research teams. It does not diagnose, does not interpret images clinically, and stores/handles files locally only — add your own access-control and de-identification steps before using it with real patient data.