OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

⭐ If our project helps you, please give us a star on GitHub to support us!

📅 Updates

• [Aug 2026] 🥰 Online demo is available at HF Space.
• [Aug 2026] 🚀 Released the datasets and evaluation scripts.
• [Aug 2026] 🎉 Released the Paper.
• [Jul 2026] 🔥 Released Post-Training weights for OneEmo.
• [Jul 2026] 💻 Inference code are now live!

---

💡About OneEmo

English | Chinese

OneEmo targets video emotion intelligence, unifying emotion perception, emotion understanding, and emotion interaction within a single multimodal reasoning model. It covers sentiment analysis (MSA), basic emotion recognition (B-MER), open-vocabulary emotion recognition(OV-MER), intention recognition (MIR), humor & sarcasm understanding (MHU & MSU), empathic response generation (ERG) and emotional support conversation (ESC). Compared with models of similar scale, OneEmo achieves SOTA results on eight emotion tasks. The training pipeline consists of two core components:

1. EmoWorld-130K: Distills domain knowledge from expert models across multiple emotion tasks into a unified training dataset with explicit reasoning trajectories.
2. Emo-Chord: Performs an off-policy cold start, then introduces expert trajectory data from the same task family as an auxiliary target during GRPO, and performs credit assignment through a unified multi-task reward system, ultimately unlocking the reasoning potential of compact models.

Resources

Resource	URL	Purpose
OneEmo-Base	Hugging Face Model	Curriculum learning stage 1 weight for cold start
OneEmo	Hugging Face Model	Final Emo-Chord model weights for evaluation
EmoWorld-130K	Hugging Face Dataset	Unified emotion reasoning dataset for training
Qwen3.5-4B	Hugging Face Model	Initialization model for SFT and RL
all-mpnet-base-v2	Hugging Face Model	S-BERT model
Qwen2.5-7B-Instruct	Hugging Face Model	RL thought reward judge model weight
DeepSeek-V4-flash	-	Automated judge for interaction tasks
MiMo-v2.5-Pro	-	Automated judge for interaction tasks
GPT-4.1-mini	-	Automated judge for interaction tasks

Directory Structure

.
├── assets/logo.png                 # Multimedia assets
├── config.py                       # Configuration for raw data, labels, videos, and model paths
├── dataset.py                      # Raw video data loading and sample matching
├── datas/
│   ├── sftnew/                     # SFT data, including assistant reasoning and answers
│   ├── rl/                         # RL data, including solution, perception, and other reward fields
│   └── eval/                       # Inference/evaluation data except MerUnibench
├── reward/
│   ├── reward_plugin.py            # format, process, answer rewards
│   └── emotion_wheel/              # OVMER label mapping resources
├── scripts/                        # Scripts for SFT, GRPO, Emo-Chord, rollout, and LoRA weight merging
├── eval/                           # Evaluation scripts
│   ├── eval_perception_understanding.py  # Perception/understanding benchmark evaluation
│   ├── eval_interaction.py         # ERG/ESC interaction quality evaluation (LLM judge)
│   ├── prompt.py                   # ERG/ESC judge prompt templates
│   ├── requirement.py              # Eval dependency list
│   ├── emotion_wheel/              # OVMER label mapping resources (eval)
│   ├── my_affectgpt/               # Evaluation toolkit (dataset/metric builders)
│   └── toolkit/                    # Utilities (readers, qwen, vllm helpers)
├── environment.yml                 # Conda environment
└── requirments.txt                 # Pip dependency list

Environment Setup

Key Version Dependencies

We used the following environment for training and evaluation; we recommend using the same configuration. For training, a Linux GPU environment compatible with the CUDA driver is recommended, preferably with the locked versions provided in this repository.

Component	Version
Python	3.12.0
PyTorch	2.10.0+cu128
torchvision	0.25.0+cu128
CUDA Runtime	12.8 (PyTorch cu128)
Transformers	5.2.0
ms-swift	4.1.3
vLLM	0.19.0
DeepSpeed	0.19.0
TRL	0.29.1
flash-attn	2.8.3

Install with Conda

conda env create -f environment.yml
conda activate oneemo

If you do not want to use the full Conda lock file directly, you can create a base environment with the following commands and then install the dependencies:

conda create -n oneemo python=3.12 -y
conda activate oneemo
pip install -r requirments.txt

Model and Data Download

OneEmo-Base (Curriculum stage1) Weights (if you want to skip SFT stage)

mkdir -p ckpts
hf download Jiaha0Hu4ng/OneEmo-Base \
  --local-dir ckpts/OneEmo-Base

EmoWorld-130K

mkdir -p datas
hf download Jiaha0Hu4ng/EmoWorld-130K \
  --repo-type dataset \
  --local-dir datas

The training commands in scripts/ directly read the in-repository prepared datas/sftnew/*.json and datas/rl/*.json. Therefore, downloading EmoWorld-130K alone will not automatically change the existing training data; to use the downloaded data, you need to organize it into the multimodal message format that ms-swift can read, and update the --dataset or --chord_sft_dataset argument in the corresponding scripts to the new JSON path.

Qwen3.5-4B

The training scripts use a placeholder path /path/to/your/OneEmo/ckpts/Qwen3.5-4B for the Qwen3.5-4B. After downloading Qwen3.5-4B, please save it to a stable local directory, for example:

Then update the --model or MODEL_PATH placeholder in the following locations:

• MODEL_PATH in config.py, used as the default model path by inference_swift.py and inference_vllm.py;
• scripts/train_sft.sh;
• scripts/train_sft_phase1.sh;
• the base model path in scripts/trans2Vllm_sft.sh;
• any other script paths that still contain the /path/to/your/resource/ placeholder.

You can use the following command to check for unreplaced placeholder paths:

rg -n "/path/to/your/OneEmo/ckpts/Qwen3.5-4B" config.py scripts datas

Raw Video and Data Path Configuration

The repository centralizes raw data paths in config.py. After downloading or organizing the raw data, you need to check at least the following four categories of configuration:

DATA_DIR = {
    "MIntRec": "/path/to/your/resource/MIntRec",
    "MIntRec2": "/path/to/your/resource/MIntRec2",
    # other datasets...
}

PATH_TO_RAW_VIDEO = {
    "MIntRec": os.path.join(DATA_DIR["MIntRec"], "video"),
    "MIntRec2": os.path.join(DATA_DIR["MIntRec2"], "video"),
}

PATH_TO_LABEL = {
    "MIntRec": os.path.join(DATA_DIR["MIntRec"], "label.npz"),
}

PATH_TO_TRANSCRIPTIONS = {
    "MIntRec": os.path.join(DATA_DIR["MIntRec"], "transcription-engchi-polish.csv"),
}

The actual fields and directory names must match your raw data directory. dataset.py uses these mappings to locate videos, labels, and transcriptions; TESTSET_JSON is used to read samples directly from evaluation JSON files. You should also check the following in config.py:

• DATA_DIR: root directory for each dataset;
• PATH_TO_RAW_VIDEO: video directory;
• PATH_TO_LABEL: label files;
• PATH_TO_TRANSCRIPTIONS: subtitle or transcription files;
• TESTSET_JSON: path to the evaluation JSON files.

Note: The videos field in datas/sftnew/*.json and datas/rl/*.json stores absolute paths to the original videos. Modifying config.py alone will not update these already generated JSON files; therefore, after downloading the data JSON files, you need to manually change the video paths inside them to the paths where you store the video data, or batch-update the videos paths in the JSON files. RL data should also preserve fields such as task_type, solution, perception, and reasoning_ref, otherwise the corresponding reward functions will not work properly.

Curriculum Learning SFT

Stage 1: Multi-task Cold Start

Phase 1 uses scripts/train_sft_phase1.sh and covers task families such as MSA, MER, MIR, MHD, MSD, and ERG. The default configuration uses LoRA, bfloat16, two GPUs, and DeepSpeed ZeRO-2:

conda activate oneemo
bash scripts/train_sft_phase1.sh

The default output directory is:

ckpts/training_ckpts/sft_oneemo_p1/

The CUDA_VISIBLE_DEVICES=0,1, NPROC_PER_NODE=2, base model path, and data sampling size in the script can be adjusted according to your GPU memory and machine configuration.

Export the Stage 1 Model

Phase 1 produces a LoRA checkpoint. Before running the export, modify the three paths in scripts/trans2Vllm_sft.sh:

--model <local path to Qwen3.5-4B>
--adapters <path to the Phase 1 checkpoint>
--output_dir <path to the exported full model>

Then run:

bash scripts/trans2Vllm_sft.sh

The exported directory must match the --model in the Phase 2 script. Currently train_sft_phase2.sh defaults to:

ckpts/inference_ckpts/sft/oneemo_p1_e5_20260618

Stage 2: Curriculum Learning Extension

Phase 2 continues training on the model exported from Phase 1, and adds more ERG task data and the ESC task:

bash scripts/train_sft_phase2.sh

The default output directory is:

ckpts/training_ckpts/sft_oneemo_p2/

If you do not use the default directory, you need to update --model in scripts/train_sft_phase2.sh first. Phase 2 also uses LoRA; before subsequent inference or RL, you typically need to export the selected checkpoint as a merged full model.

Emo-Chord Training

ERG/ESC rewards require a local Sentence-BERT model. When the repository already contains ckpts/all-mpnet-base-v2, set:

export VIDEMO_SENTENCE_BERT_MODEL=/path/to/your/OneEmo/ckpts/all-mpnet-base-v2

If you use the process reward in train_sft_then_chord.sh, you need to prepare a judge service and set the actual service address:

export VIDEMO_JUDGE_BASE_URL=http://127.0.0.1:15555/v1
export VIDEMO_JUDGE_MODEL=Qwen2.5-7B-Instruct
bash scripts/train_sft_then_chord.sh

The default judge address in the code is an internal address of the current development environment and should not be treated as a universal configuration. train_grpo_chord.sh uses format answer by default and does not enable process; the actual reward combination is determined by --reward_funcs and --reward_weights in the script.

Inference

inference_swift.py uses the TransformersEngine, while inference_vllm.py uses the vLLM/OpenAI-compatible interface. Both read raw video, label, and transcription configuration via config.py and dataset.py.

# Start the vLLM model first, then run inference
python inference_vllm.py --datasets merunibench --model_path {path} --verbose

# Directly use swift infer for inference
CUDA_VISIBLE_DEVICES=0 python inference_swift.py \
  --datasets merunibench \
  --model_path {path} \
  --verbose \
  --think true \
  --attn_impl sdpa

--datasets accepts dataset groups defined in config.py, such as merunibench, mir, msd, mhd, erg, and esc, or you can pass a single dataset name directly.

Evaluation

please see the Instruction

Notes

• The licenses of the original videos and each dataset are determined by their original publishers. Please follow the corresponding dataset licenses and terms of use before using the data.

Citation

If you use this codebase, or otherwise found our work valuable, please consider giving a star ⭐ and cite the papers:

@article{huang2026oneemo,
      title={OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction}, 
      author={Jiahao Huang and Zheng Lian and Jingyi Zhang and Zhide Chen and Xiaojiang Peng and Shaonan Wang},
      year={2026},
      journal={arXiv preprint arXiv:2608.06013},
}

@InProceedings{Huang_2026_CVPR,
    author    = {Huang, Jiahao and Lin, Fengyan and Yang, Xuechao and Feng, Chen and Zhu, Kexin and Yang, Xu and Chen, Zhide},
    title     = {Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {22986-22997}
}

Acknowledgements

The training and evaluation pipeline of OneEmo is built on top of several excellent open-source projects. We would like to specially thank:

• ms-swift: for providing an efficient and easy-to-use training framework.
• AffectGPT: whose dataset and evaluation toolkit provided important references and foundations for building the emotion understanding task benchmarks.