Skip to content

Add Gemini 3.6 and 3.7 structured VLM output support - #2504

Open
Xayar145 wants to merge 1 commit into
roboflow:developfrom
Xayar145:feat/gemini-3-6-3-7-vlm
Open

Add Gemini 3.6 and 3.7 structured VLM output support#2504
Xayar145 wants to merge 1 commit into
roboflow:developfrom
Xayar145:feat/gemini-3-6-3-7-vlm

Conversation

@Xayar145

Copy link
Copy Markdown
Before submitting
  • Self-reviewed the code
  • Updated documentation in Google style
  • Added/updated tests
  • All tests pass locally

Description

Adds Google Gemini 3.6 and 3.7 adapters to Detections.from_vlm for the current structured object-detection and segmentation response format.

Type of Change

  • ✨ New feature (non-breaking change which adds functionality)

Motivation and Context

The existing Gemini adapters stop at 3.5 and expect the legacy top-level array and encoded-mask format. Gemini 3.6 and 3.7 structured responses wrap items in a top-level boxes object and represent segmentation masks as normalized polygons. Without dedicated VLM members and parsing, Detections.from_vlm rejects these model variants.

Format reference: https://ai.google.dev/gemini-api/docs/image-understanding#object-detection

Changes Made

  • Register GOOGLE_GEMINI_3_6 and GOOGLE_GEMINI_3_7 with result and argument validation.
  • Parse structured boxes, optional confidence values, class filters, and normalized polygon masks into Detections.
  • Preserve valid empty-mask shapes when class filtering removes every item, and document the new adapters in the capability tables and changelog.

Testing

  • I have tested this code locally

  • I have added unit tests that prove the feature works

  • All new and existing tests pass

  • Focused VLM suite: 114 passed

  • Full suite: 3617 passed, 20 skipped

  • Pre-commit: all hooks passed, including Ruff, mypy, codespell, and Markdown formatting

Additional Notes

The 3.7 adapter reuses the 3.6 structured parser because both versions expose the same box and polygon schema.

@Xayar145
Xayar145 requested a review from SkalskiP as a code owner August 23, 2026 13:41
@CLAassistant

CLAassistant commented Aug 23, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants