Codebase for Aria - an Open Multimodal Native MoE

[ICML'24] Data and code for our paper "Training-Free Long-Context Scaling of Large Language Models"

A new multi-shot video understanding benchmark Shot2Story with comprehensive video summaries and detailed shot-level captions.

Accelerating the development of large multimodal models (LMMs) with lmms-eval

A lightweight flexible Video-MLLM developed by TencentQQ Multimedia Research Team.

Code implementation of synthetic continued pretraining

LaVIT: Empower the Large Language Model to Understand and Generate Visual Content

EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

[CVPR 2023] Official Pytorch code for PROB: Probabilistic Objectness for Open World Object Detection

Qwen2-VL is the multimodal large language model series developed by Qwen team, Alibaba Cloud.

official impelmentation of Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input

GPT4V-level open-source multi-modal model based on Llama3-8B

Towards Large Multimodal Models as Visual Foundation Agents

LVBench: An Extreme Long Video Understanding Benchmark

Pytorch implementation of Twelve Labs' Video Foundation Model evaluation framework & open embeddings

[ICML 2024 Oral] Official code repository for MLLM-as-a-Judge.

[Neurips 24' D&B] Official Dataloader and Evaluation Scripts for LongVideoBench.

VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use th…

ACL'24 (Oral) Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

Utilities intended for use with Llama models.

【NeurIPS 2024】Dense Connector for MLLMs

[ECCV 2024] Elysium: Exploring Object-level Perception in Videos via MLLM

Fast and memory-efficient exact attention