pooler of dino-v2 is newly initialized when loading the pre-trained model

[garychan22]

System Info

I have encountered the following warning when loading facebook/dinov2-giant:

Some weights of Dinov2Model were not initialized from the model checkpoint at models--facebook--dinov2-giant and are newly initialized:['pooler.dense.bias', 'pooler.dense.weight']

does this mean that we cannot utilize the output.pooler_output as the global feature?

transformers 4.32.0.dev0

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import os
from transformers import AutoImageProcessor, AutoModel
from PIL import Image

image_path = 'COCO_val2014_000000000285.jpg'
image = Image.open(image_path)

processor = AutoImageProcessor.from_pretrained(r"models--facebook--dinov2-giant")
model = AutoModel.from_pretrained(r"models--facebook--dinov2-giant")

inputs = processor(images=[image] * 2, return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state # bs x 257 x 1536, all features [global, patch ....]
global_feature = outputs.pooler_output # bs x 1536, global image feature after pooling layer

print(last_hidden_states.size())

Expected behavior

expect the outputs.pooler_output is meaningful

[sgugger]

cc @younesbelkada

[ValerioL29]

Hi, dear HF staffs.

I found a similar issue when I am playing with the DINO v1 model.

dino-vits16

The following warning message came out after the loading process -

Some weights of ViTModel were not initialized from the model checkpoint at /data/models/dino/dino-vits16 and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

The DINOv2 models work fine since it is fixed in #25392 .

[amyeroberts]

@ValerioL29 Would you like to open a PR to fix this for dino v1?

[ValerioL29]

@ValerioL29 Would you like to open a PR to fix this for dino v1?

Yes, for sure. I'm glad to help! 🤗

[ValerioL29]

Hi, dear HF staffs.

I found a similar issue when I am playing with the DINO v1 model.

dino-vits16

The following warning message came out after the loading process -

Some weights of ViTModel were not initialized from the model checkpoint at /data/models/dino/dino-vits16 and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

The DINOv2 models work fine since it is fixed in #25392 .

@NielsRogge Hi Niels, I have done some research on the issue. I found that the transformers' implementation of ViT is kinda different from the original implementation of DINO. I printed out the modules -

• 🤗transformers ViTModel

from transformers import ViTModel
model = ViTModel.from_pretrained('facebook/dino-vits16')
print(model)

'''
ViTModel(
  (embeddings): ViTEmbeddings(
    (patch_embeddings): ViTPatchEmbeddings(
      (projection): Conv2d(3, 384, kernel_size=(16, 16), stride=(16, 16))
    )
    (dropout): Dropout(p=0.0, inplace=False)
  )
  (encoder): ViTEncoder(
    (layer): ModuleList(
      (0-11): 12 x ViTLayer(
        (attention): ViTSdpaAttention(
          (attention): ViTSdpaSelfAttention(
            (query): Linear(in_features=384, out_features=384, bias=True)
            (key): Linear(in_features=384, out_features=384, bias=True)
            (value): Linear(in_features=384, out_features=384, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
          )
          (output): ViTSelfOutput(
            (dense): Linear(in_features=384, out_features=384, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
          )
        )
        (intermediate): ViTIntermediate(
          (dense): Linear(in_features=384, out_features=1536, bias=True)
          (intermediate_act_fn): GELUActivation()
        )
        (output): ViTOutput(
          (dense): Linear(in_features=1536, out_features=384, bias=True)
          (dropout): Dropout(p=0.0, inplace=False)
        )
        (layernorm_before): LayerNorm((384,), eps=1e-12, elementwise_affine=True)
        (layernorm_after): LayerNorm((384,), eps=1e-12, elementwise_affine=True)
      )
    )
  )
  (layernorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)
  (pooler): ViTPooler(
    (dense): Linear(in_features=384, out_features=384, bias=True)
    (activation): Tanh()
  )
)
'''

• Following the DINO github repo guides and load it from torch hub

import torch
model = torch.hub.load('facebookresearch/dino:main', 'dino_vits16')
print(model)

'''
VisionTransformer(
  (patch_embed): PatchEmbed(
    (proj): Conv2d(3, 384, kernel_size=(16, 16), stride=(16, 16))
  )
  (pos_drop): Dropout(p=0.0, inplace=False)
  (blocks): ModuleList(
    (0-11): 12 x Block(
      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
      (attn): Attention(
        (qkv): Linear(in_features=384, out_features=1152, bias=True)
        (attn_drop): Dropout(p=0.0, inplace=False)
        (proj): Linear(in_features=384, out_features=384, bias=True)
        (proj_drop): Dropout(p=0.0, inplace=False)
      )
      (drop_path): Identity()
      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
      (mlp): Mlp(
        (fc1): Linear(in_features=384, out_features=1536, bias=True)
        (act): GELU(approximate='none')
        (fc2): Linear(in_features=1536, out_features=384, bias=True)
        (drop): Dropout(p=0.0, inplace=False)
      )
    )
  )
  (norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
  (head): Identity()
)
'''

The difference is that the original implementation of the DINO backbone does not contain a pooler (or head?) after the final layer norm module but with an nn.Identity head. I guess that's why the ViTPooler's weights are newly initialized and then trigger the warning message.

Luckily the ViTModel has a boolean parameter add_pooling_layer to control whether to initialize the ViTPooler in the model or not. After setting it to be False when loading the pretrained DINO model with the from_pretrained method, everything works just fine.

from transformers import ViTModel
model = ViTModel.from_pretrained(
    'facebook/dino-vits16',
    add_pooling_layer=False
)
# No more warning messages about pooler weights are newly initialized.

So, I think there's no need to open a new PR, right? Perhaps we can add some notes on this within the model card's code example so the user will know it is by design. 🤩

[amyeroberts]

@ValerioL29 A note on the model card seems like a good solution :)

[ValerioL29]

@ValerioL29 A note on the model card seems like a good solution :)

Yeah, I think so! I can open a PR on the Hub 😃

[NielsRogge]

Thanks for looking into this and opening a PR! As I'm not part of the facebook org on the hub I'm not able to merge these PRs. Will reach out to someone at Meta who can merge them.