This repository contains action-centric captioning pipelines for two major video segmentation datasets:
- A2D Dataset (Actor-Action Dataset)
- ViCaS Dataset
The goal is to generate action-centric captions that enable Referring Image Segmentation (RIS) models to better understand actions, not just object categories.
Despite recent advances in RIS, most models are trained on datasets like RefCOCO, where captions are predominantly noun-centric, such as:
"The man on the left", "The dog in the back"
These descriptions lack action-level semantics. However, RIS is a task that often requires distinguishing between multiple objects of the same category — a situation where action-based disambiguation is critical.
To unlock action-level comprehension, we need captions like:
"The man is kicking a ball",
"The child is climbing the stairs"
This repository addresses that gap by proposing pipelines that generate such verb-rich, action-centered captions using A2D and ViCaS datasets.
- Improves the ability of RIS models to distinguish same-category objects based on actions
- Enhances verb understanding in RIS models
This project builds on:
✅ Designed and implemented full pipelines for both A2D and ViCaS
✅ Applied movability-based filtering, action-focused instruction tuning
✅ Significantly enriched RIS training data to enhance verb-level understanding
If you have any questions, feel free to reach out!