About Visual Prompt Encoder.

[ysysys666]

Thank you for your excellent work! I have a question to ask.

How can deformable attention be used to extract features from specific regions? How are the reference points initialized? Are they based on the centers of the bounding boxes, or are they determined differently? Additionally, how many sampling points are typically used?

Looking forward to your response!

[wangSir0202]

Waiting for an answer. +1

[Mountchicken]

Hi @ysysys666 @wangSir0202
Sorry for the late reply

How can deformable attention be used to extract features from specific regions?

Technically, Instead of attending to all spatial locations, Deformable Attention attends to a small set of learned, sparse sampling points around a reference point. each query in Deformable attention attends to a small set of learnable sampling points around a reference point (or box), rather than the entire feature map. For example, in visual prompts like point or box prompts, each query in cross-attention uses the point or box as the reference point. Then, it computes some offsets and gathers information about the objects in those regions.

Here is a code snap that might help

cross_attn = MSDeformAttn(d_model=256, n_levels=4, n_heads=8, n_points=4)
# n_points is the sampling points
visual_prompt_feature = cross_attn(query, reference_points, memory)

Are they based on the centers of the bounding boxes

Deformable attention supports two types of reference points:

Point format (xy) – typically representing specific coordinates (like the center of a region), and

Box format (xywh) – representing a region by its center coordinates (x, y), width (w), and height (h).

:param reference_points            (N, Length_{query}, n_levels, 2), range in [0, 1], top-left (0,0), bottom-right (1, 1), including padding area
                                        or (N, Length_{query}, n_levels, 4), add additional (w, h) to form reference boxes

In T-Rex2, both types of prompts are supported

how many sampling points are typically used

We use 4