Repository navigation
feat(xla): support Molmo2 indexed attention pooling #871
Copy link
Copy link
Closed
Labels
area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)Generation, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadataModel architectures, weights, loading, metadatapriority:lowLow priorityLow prioritystatus:doneCompletedCompletedtype:enhancementNew features, capabilities, or significant additionsNew features, capabilities, or significant additions
Description
Activity
- addedstatus:blockedBlocked by dependencies or other issuesBlocked by dependencies or other issuestype:enhancementNew features, capabilities, or significant additionsNew features, capabilities, or significant additionspriority:lowLow priorityLow priorityarea:modelsModel architectures, weights, loading, metadataModel architectures, weights, loading, metadataarea:inferenceGeneration, sampling, decoding (incl. speculative, DRY)Generation, sampling, decoding (incl. speculative, DRY)
on Jul 22, 2026 - addedstatus:in-progressCurrently being worked onCurrently being worked onstatus:reviewUnder reviewUnder reviewand removedstatus:blockedBlocked by dependencies or other issuesBlocked by dependencies or other issuesstatus:in-progressCurrently being worked onCurrently being worked on
on Jul 24, 2026 27 remaining items
- added 10 commits that reference this issue
on Aug 20, 2026 - added a commit that references this issue
on Aug 21, 2026 - addedstatus:doneCompletedCompletedand removedstatus:reviewUnder reviewUnder review
on Sep 6, 2026
Metadata
Metadata
Assignees
Labels
area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)Generation, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadataModel architectures, weights, loading, metadatapriority:lowLow priorityLow prioritystatus:doneCompletedCompletedtype:enhancementNew features, capabilities, or significant additionsNew features, capabilities, or significant additions
Parent and dependency
src/vision/molmo2_vl.rs,src/vision/encoders/molmo2.rs,src/vision/processors/molmo2.rs, andsrc/models/molmo2.rsProblem
Molmo2 has its own visual/data contract: pre-flattened patches use a Linear patch embedding, position embeddings depend on the patch grid, configured intermediate ViT layers are concatenated,
image_token_poolingcontains patch indices with negative invalid sentinels, attention pooling produces one vector per output token, and a SwiGLU projector is additively merged at scannedimage_patch_idpositions.This differs from Molmo v1's processor-supplied
image_input_idxmapping and must not share the wrong merge/index policy.Goal
Support a pinned Molmo2 checkpoint end to end on XLA with its exact patch-grid, layer-selection, indexed attention-pooling, additive merge, and continuous-batch behavior.
Required implementation
Molmo2Processoroutputs: flattened crops/patches,image_token_pooling,image_grids, andimage_num_crops. Preserve integer dtypes, shapes, and negative pooling sentinels in the prepared payload.image_patch_idcount before native invocation.image_patch_id, preserve position order, and add one projected feature per position. Do not replace text embeddings and do not use Molmo v1image_input_idxsemantics.Non-goals
Validation
Acceptance criteria
image_patch_idmerge match both reference implementations.