[EXPERIMENT][WIP][OpenVINO] Add support for FG-CLIP2

[mlukasze]

⚠️ AUTOMATICALLY GENERATED BY OMEGA AGENT — REQUIRES HUMAN REVIEW ⚠️
This PR was created by an AI agent as part of automated model enablement.
A human maintainer must review and approve it before it can be considered for merge.
Do NOT merge without human review and sign-off.

What does this PR do?

Adds OpenVINO export and inference support for FG-CLIP2 (qihoo360/fg-clip2-so400m, model_type=fgclip2), a bilingual (EN/中文) dual-tower zero-shot image classification model with a SigLIP2-NaFlex-style contrastive vision/text embedding architecture (custom trust_remote_code modeling, loaded via Fgclip2Model).

Closes #84.

Architecture summary

• Dual-tower CLIP/SigLIP-style model: separate vision and text encoders producing image_embeds/text_embeds/logits_per_image/logits_per_text, reusing the existing OVModelForZeroShotImageClassification runtime class (no new OV runtime class needed).
• The vision tower is SigLIP2-NaFlex-style: pixel_values is pre-patchified to (batch, num_patches, num_channels * patch_size * patch_size), and a spatial_shapes (batch, 2) side-tensor drives a per-sample F.interpolate call that resizes learned positional embeddings to the image's grid.
• Export is pinned to FG-CLIP2's own fixed canonical resolution (square sqrt(num_patches) x sqrt(num_patches) patch grid) so the per-sample interpolation loop traces to one static shape -- the same trade-off the existing siglip (fixed-resolution) OV config already makes. Arbitrary-resolution NaFlex dynamism is out of scope for this PR.
• A known upstream bug in modeling_fgclip2.py (position_ids/mask1/mask2 in Fgclip2TextEmbeddings are precomputed at __init__ time as non-persistent buffers/plain attributes, so from_pretrained's fast-init path never (re)materializes them, causing an IndexError in plain PyTorch, independent of OpenVINO) is worked around in Fgclip2ModelPatcher for the export path.

Accuracy validation (perception-mode fallback, not WWB)

FG-CLIP2's task (zero-shot-image-classification) is canonical_type=perception, so the standard WWB text-generation pipeline does not apply. Accuracy was validated by accuracy-supervisor via manual image/text embedding cosine-similarity against the eager PyTorch reference, on 4 real image/caption pairs, CPU + iGPU (GPU.0):

Precision	Device	Mean image-embed cos-sim	Mean text-embed cos-sim	Top-1 caption agreement	Status
FP16	CPU	99.99998%	99.99999%	100%	✅ PASS
FP16	GPU.0	99.981%	99.999%	100%	✅ PASS
INT8 (weight-only)	CPU	99.784%	99.906%	100%	✅ PASS
INT8 (weight-only)	GPU.0	99.813%	99.912%	100%	✅ PASS
INT4 (weight-only, default group_size=128+adjust)	CPU	91.835%	99.248%	75% (3/4)	⚠️ expected quantization loss (see below)
INT4 (weight-only, default group_size=128+adjust)	GPU.0	91.852%	99.268%	75% (3/4)	⚠️ expected quantization loss (see below)

Verdict: ok (this is the first accuracy run for this model_type, so it establishes the baseline rather than regressing against one). Full breakdown: agent-results/accuracy-supervisor/verdict.json / agent-results/accuracy-supervisor/summary.md.

Known limitations / follow-ups

• Export is at a fixed canonical resolution (SigLIP2-NaFlex-style dynamic per-sample interpolation is traced to one static shape); true arbitrary-resolution NaFlex dynamism is out of scope for this change.
• walk_type is pinned to the model's own default ("short") for the exported text tower variant -- a compile-time Python string kwarg, not a tensor branch.
• Full/activation-based INT8 quantization is not supported: real, variously-sized calibration images do not match the fixed canonical patch grid baked in at export time, causing a Broadcast shape-mismatch during statistics collection. Weight-only INT8/INT4 are the supported quantization paths (no default _DEFAULT_INT8_FQ_CONFIGS entry is registered; documented inline in configuration.py).
• INT4 group_size sweep finding (not implemented in this PR, out of scope): the current default INT4 recipe (group_size=128, group_size_fallback="adjust", since intermediate_size=4304 is not evenly divisible by 128) shows meaningful fidelity loss (91.8% image-embed cosine similarity, 75% top-1 agreement). A CPU-only group_size sweep by accuracy-supervisor found group_size=16 (an exact divisor of intermediate_size=4304, avoiding the fallback path entirely) recovers image-embed cosine similarity to 96.8% and top-1 caption agreement to 100%, with no regression on any other measured axis (text-embed cosine also improves slightly, 99.59% vs 99.25%). group_size=32 also fully recovers top-1 agreement (94.9% image cosine). Recommend a human maintainer or a follow-up iteration consider registering group_size=16 (or 32) as the default INT4 weight-compression group size for fgclip2 in _DEFAULT_4BIT_WQ_CONFIGS, instead of the generic 128+adjust fallback. Full sweep data: agent-results/accuracy-supervisor/verdict.json (known_issue.experiment).

Pending human action -- tiny test fixture upload

The tiny test fixture optimum-intel-internal-testing/tiny-random-Fgclip2Model referenced by the new test entries has not been uploaded to the Hub yet (this agent has no upload access). A standalone, no-argument creation script is provided at agent-results/optimum-intel/create_tiny_fgclip2.py (real Fgclip2Model class, random weights, scaled-down but shape/divisibility-invariant-preserving config). A human maintainer needs to run it and upload the result to that Hub id before the newly added tests (test_modeling.py, test_export.py, test_exporters_cli.py) can pass in CI. All new test code paths were validated locally against a freshly generated tiny model (temporarily pointing MODEL_NAMES["fgclip2"] at the local directory, reverted before commit) -- 2 passed (see test_run.log). Running against the real (not-yet-uploaded) Hub id fails deterministically with RepositoryNotFoundError (401), which is the expected/documented blocked state, not a code defect.

Quality gate results

• check_pr_quality.py: black PASS, syntax PASS, test_coverage PASS (hard gates). mutation shows a pre-existing warning unrelated to this change (37 locations in model_configs.py/model_patcher.py all belong to VideoChatFlashQwenConfig/QwenModelPatcher, not touched by this diff). base_class PASS (no recent breaking upstream changes to the base classes used here).
• scan_open_prs.py: no duplicate/open PRs found for fgclip2/clip.

Mandatory Reuse Audit

New/changed symbol	Existing candidate(s) inspected	Shared behavior	Decision	Remaining model-specific delta
Fgclip2OpenVINOConfig	SiglipOpenVINOConfig	Dual-tower CLIP-style export config: same logits_per_image/logits_per_text/text_embeds/image_embeds output contract, same fixed-resolution vision-input trade-off	Configured/subclassed (class Fgclip2OpenVINOConfig(SiglipOpenVINOConfig))	Overrides DUMMY_INPUT_GENERATOR_CLASSES (adds spatial_shapes) and inputs (declares pixel_values/spatial_shapes shapes matching the pre-patchified NaFlex-style vision tower); no attention_mask input, matching siglip's own convention
DummyFgclip2VisionInputGenerator	DummyVisionInputGenerator (base), DummyDeepseekOCR2VisionTilesInputGenerator/DummyGemma4UnifiedVisionInputGenerator (closest sibling patterns with non-standard vision dummy inputs)	Generates dummy vision tower inputs for export tracing	Subclassed (class DummyFgclip2VisionInputGenerator(DummyVisionInputGenerator))	Generates pre-patchified pixel_values (batch, num_patches, channels*patch*patch) instead of (batch, channels, height, width), plus a synthetic constant spatial_shapes tensor; pins batch_size=1 because the vision-tower embeddings module unrolls a Python loop over spatial_shapes.shape[0] at trace time, baking image batch size into the graph -- none of the inspected siblings need this
Fgclip2ModelPatcher	ModelPatcher (base), SAMModelPatcher/QwenModelPatcher (closest siblings that also mutate model.config/precompute tensors in __init__)	Pre-export model-object patching before tracing	Genuinely model-specific (subclasses generic ModelPatcher, no shared logic reused beyond the base contract)	Recomputes position_ids/mask1/mask2 for Fgclip2TextEmbeddings to work around an upstream from_pretrained fast-init bug specific to this checkpoint's custom modeling code -- not a general pattern applicable elsewhere
OVModelForZeroShotImageClassification.forward (extended, not replaced)	itself (pre-existing clip/siglip code path)	Runs OV inference for CLIP-style zero-shot image classification models	Reused unchanged for clip/siglip; conditionally extended (if "spatial_shapes" in self.input_names) rather than forked into a new class	Adds an optional spatial_shapes input, defaulted to the fixed canonical grid size derived from pixel_values.shape[1] when the caller does not supply one -- a no-op for every existing model that does not declare a spatial_shapes input
_DEFAULT_4BIT_WQ_CONFIGS["qihoo360/fg-clip2-so400m"] entry	other dual-tower/encoder-only entries in the same dict	Data-free INT4 weight-only compression recipe	Configured (new dict entry, no new code)	group_size_fallback="adjust" needed because intermediate_size=4304 is not divisible by the default group_size=128 (see Known limitations above for the follow-up sweep)

No symbol in this diff is a renamed copy or a bare super()-only override; every override adds a concrete, documented, model-specific behavioral delta.

Regression check

No shared runtime files (convert.py, modeling_decoder.py, modeling_diffusion.py, modeling_seq2seq.py, modeling_visual_language.py) were modified. model_patcher.py, model_configs.py, and input_generators.py were extended with new, additive classes only (no existing class/function body was modified) -- a regression run against an existing model of the same family was not required, and check_pr_quality.py's base_class check confirms no upstream base-class drift affects this change.

Installation instructions

pip install git+https://github.com/mlukasze/optimum-intel.git@enable/qihoo360-fg-clip2-so400m
pip install --pre -U openvino openvino-tokenizers nncf --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

Exporting cmd-line

optimum-cli export openvino -m qihoo360/fg-clip2-so400m --task zero-shot-image-classification ov_fgclip2

Inference script

import requests
from PIL import Image
from transformers import AutoProcessor
from optimum.intel import OVModelForZeroShotImageClassification

model_id = "qihoo360/fg-clip2-so400m"
model = OVModelForZeroShotImageClassification.from_pretrained("ov_fgclip2")
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
candidate_labels = ["a photo of a cat", "a photo of a dog"]

inputs = processor(text=candidate_labels, images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=-1)
print(list(zip(candidate_labels, probs[0].tolist())))

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.