merge origin/main

README.md

@@ -37,6 +37,8 @@ This source code is licensed under the Apache 2.0 license found in the LICENSE f
 OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks 
 (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) 
 to a simple sequence-to-sequence learning framework. For more information, please refer to our paper: [Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](http://arxiv.org/abs/2202.03052).
+
+We welcome contributions to our project. Feel free to contact us or send us issues/PRs!
 <br></br>
 
 
@@ -51,7 +53,7 @@ Also we provide Colab notebooks for you to better perceive the procedures. Click
 <br></br>
 
 # News
-* 2022.3.08: Realease the pretrained checkpoint of OFA-Base in [checkpoints.md](checkpoints.md). To use OFA-Base, you just need to load `ofa_base.pt` and change `arch=ofa_large` to `arch=ofa_base` in the training scripts.
+* 2022.3.08: Released the pretrained checkpoint of OFA-Base in [checkpoints.md](checkpoints.md). To use OFA-Base, you just need to load `ofa_base.pt` and change `--arch=ofa_large` to `--arch=ofa_base` in the training scripts.
 * 2022.3.07: Released the finetuning & inference code/checkpoints for **Image Classification**, which achieves **85.0 accuracy on ImageNet-1K, slightly better than reported in OFA paper**.
 * 2022.3.04: Released the finetuning & inference code/checkpoints for **Text-to-Image Generation**.
 * 2022.3.03: Released the finetuning & inference code/checkpoints for **SNLI-VE** and **GLUE**.
@@ -64,9 +66,11 @@ Also we provide Colab notebooks for you to better perceive the procedures. Click
 
 # TODO
 * [x] To release finetuning and inference codes for multimodal downstream tasks soon, including image captioning, VQA, text-to-image generation, SNLI-VE, referring expression, comprehension, etc. 
-* [ ] To release finetuning and inference codes for unimodal downstream tasks soon. 
+* [x] To release finetuning and inference codes for unimodal downstream tasks soon. 
 * [ ] To release codes for pretraining soon.
 * [ ] To integrate more downstream tasks concerning more modalities to our OFA framework.
+* [ ] To release smaller models, including OFA-medium, OFA-tiny, as well as OFA-edge. 
+* [ ] To release OFA for Chinese.
 <br></br>
 
 # Approach
@@ -115,7 +119,7 @@ To release soon:)
 <br></br>
 
 # Finetuning & Inference
-Below we provide methods for finetuning and inference on different downstream tasks. We provide both the pretrained checkpoint of OFA-Large and OFA-Base in [checkpoints.md](checkpoints.md), the following scripts is running for OFA-Large. If you want to use OFA-Base, just change the `restore_file` to the path where ofa_base.pt is located, and change `arch=ofa_large` to `arch=ofa_base`. **Note that the optimal hyperparameters for the Base model may be different from the Large model and requires proper hyperparameter tuning.
+Below we provide methods for finetuning and inference on different downstream tasks. We provide both pretrained OFA-Large and OFA-Base in [checkpoints.md](checkpoints.md). The scripts in this section are prepared for OFA-Large. If you want to use OFA-Base, just modify `--restore-file` to the path where `ofa_base.pt` is located and change `--arch=ofa_large` to `--arch=ofa_base`. **Note that the optimal hyperparameters for the Base model may be different from the Large model and requires proper hyperparameter tuning.**
 ## Image Captioning
 We provide procedures to reproduce our results of image captioning on our paper below.
 <details>

criterions/clip_scst_loss.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import math
 from dataclasses import dataclass, field

criterions/label_smoothed_cross_entropy.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import math
 from dataclasses import dataclass, field

criterions/scst_loss.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import math
 import string

data/cv_data/image_classify_dataset.py

@@ -1,7 +1,8 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 from io import BytesIO
 
 import logging
@@ -192,4 +193,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/data_utils.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 try:
     from collections.abc import Iterable
@@ -598,4 +598,4 @@ def raise_if_valid_subsets_unintentionally_ignored(train_cfg) -> None:
     if ignored_paths:
         advice = "Set --combine-val to combine them or --ignore-unused-valid-subsets to ignore them."
         msg = f"Valid paths {ignored_paths} will be ignored. {advice}"
-        raise ValueError(msg)
+        raise ValueError(msg)

data/file_dataset.py

@@ -1,3 +1,8 @@
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 import os
 import torch
 import pickle
@@ -99,4 +104,4 @@ def __getitem__(self, index):
         column_l = self._reader.readline().rstrip("\n").split(self.separator)
         self.data_cnt += 1
         column_l = [dtype(column_l[col_id]) for col_id, dtype in zip(self.selected_col_ids, self.dtypes)]
-        return column_l
+        return column_l

data/mm_data/caption_dataset.py

@@ -1,7 +1,8 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 from io import BytesIO
 
 import logging
@@ -151,4 +152,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/mm_data/image_gen_dataset.py

@@ -1,7 +1,8 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 from io import BytesIO
 
 import logging

data/mm_data/refcoco_dataset.py

@@ -1,7 +1,8 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 from io import BytesIO
 
 import logging
@@ -165,4 +166,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/mm_data/snli_ve_dataset.py

@@ -1,7 +1,8 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 from io import BytesIO
 
 import logging
@@ -199,4 +200,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/mm_data/vqa_gen_dataset.py

@@ -1,7 +1,8 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 from io import BytesIO
 
 import logging

data/nlu_data/cola_dataset.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import warnings
@@ -135,4 +135,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/nlu_data/mnli_dataset.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import warnings
@@ -140,4 +140,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/nlu_data/mrpc_dataset.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import warnings
@@ -138,4 +138,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/nlu_data/qnli_dataset.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import warnings
@@ -138,4 +138,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/nlu_data/qqp_dataset.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import warnings
@@ -138,4 +138,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/nlu_data/rte_dataset.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import warnings
@@ -138,4 +138,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/nlu_data/sst2_dataset.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import warnings
@@ -135,4 +135,4 @@ def collater(self, samples, pad_to_length=None):
         Returns:
             dict: a mini-batch with the following keys:
         """
-        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)
+        return collate(samples, pad_idx=self.pad, eos_idx=self.eos)

data/ofa_dataset.py

@@ -1,3 +1,8 @@
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 import logging
 import re
 import torch.utils.data

evaluate.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3 -u
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import logging
 import os

models/ofa/ofa.py

@@ -1,7 +1,8 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
+
 """
 OFA
 """

models/ofa/unify_multihead_attention.py

@@ -1,7 +1,7 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
+# Copyright 2022 The OFA-Sys Team. 
+# All rights reserved.
+# This source code is licensed under the Apache 2.0 license 
+# found in the LICENSE file in the root directory.
 
 import math
 from typing import Dict, Optional, Tuple