vision-language-pretraining

Here are 32 public repositories matching this topic...

salesforce / LAVIS

LAVIS - A One-stop Library for Language-Vision Intelligence

deep-learning salesforce image-captioning deep-learning-library vision-framework vision-and-language multimodal-deep-learning multimodal-datasets vision-language-transformer vision-language-pretraining visual-question-anwsering

Updated Oct 11, 2024
Jupyter Notebook

DAMO-NLP-SG / Video-LLaMA

Star

[EMNLP 2023 Demo] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

llama large-language-models video-language-pretraining vision-language-pretraining cross-modal-pretraining blip2 minigpt4 multi-modal-chatgpt

Updated Jun 4, 2024
Python

deepseek-ai / DeepSeek-VL

Star

DeepSeek-VL: Towards Real-World Vision-Language Understanding

foundation-models vision-language-pretraining vision-language-model

Updated Apr 24, 2024
Python

[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

chatbot llama clip mulit-modal vision-language vicuna gpt-4 vision-language-pretraining llava video-chatboat video-conversation

Updated Aug 27, 2024
Python

deepseek-ai / Janus

Star

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

multimodal any-to-any foundation-models llm vision-language-pretraining

Updated Oct 31, 2024
Python

Sense-GVT / DeCLIP

Star

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

multi-model clip big-model zero-shot self-supervised image-text vision-language-pretraining

Updated Sep 19, 2022
Python

TXH-mercury / VALOR

Star

Codes and Models for VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

vision-language-pretraining audio-language-pretraining audiovisual-language-pretraining multimodal-representation-learning

Updated May 28, 2024
Python

mbzuai-oryx / VideoGPT-plus

Star

Official Repository of paper VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding

chatbot clip image-encoder video-encoder multimodal dual-encoder vision-language vicuna gpt4 vision-language-pretraining llava video-conversation video-chatbot llama3 gpt4o phi-3-mini

Updated Aug 11, 2024
Python

sail-sg / ptp

Star

[CVPR2023] The code for 《Position-guided Text Prompt for Vision-Language Pre-training》

cross-modality vlp vision-language-pretraining

Updated Jun 7, 2023
Python

Surrey-UP-Lab / RegionSpot

Star

Recognize Any Regions

open-world object-detection zero-shot instance-segmentation auto-labeling vision-language-pretraining open-vocabulary vision-language-model multimodal-representation-learning vision-foundation-model vision-language-foundation-model

Updated Sep 27, 2024
Python

jusiro / FLAIR

Star

[MedIA'24] FLAIR: A Foundation LAnguage-Image model of the Retina for fundus image understanding.

medical-imaging fundus-image-analysis foundation-models vision-language-pretraining

Updated May 15, 2024
Python

ArrowLuo / SegCLIP

Star

PyTorch implementation of ICML 2023 paper "SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation"

transfer-learning semantic-segmentation contrastive-learning zero-shot-semantic-segmentation vision-language-pretraining open-vocabulary open-vocabulary-semantic-segmentation

Updated Jun 28, 2023
Python

vgthengane / Continual-CLIP

Star

Official repository for "CLIP model is an Efficient Continual Learner".

baseline clip continual-learning vision-language-pretraining foundational-models

Updated Dec 13, 2022
Python

marslanm / Multimodality-Representation-Learning

Star

This repository provides a comprehensive collection of research papers focused on multimodal representation learning, all of which have been cited and discussed in the survey just accepted https://dl.acm.org/doi/abs/10.1145/3617833 .

cross-modal multimodal-deep-learning multimodal-datasets transformer-models multimodal-pre-trained-model vision-language-pretraining multimodal-applications multimodal-pretext