[ECCV 2026] VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting

Daeun Lee, Shoubin Yu, Yue Zhang, Mohit Bansal

UNC Chapel Hill

---

📺 Method Overview

📄 Abstract

Video reasoning requires models to locate and track question-relevant evidence across frames. While reinforcement learning (RL) with verifiable rewards improves accuracy, it still struggles to achieve reliable spatio-temporal grounding during the reasoning process. Moreover, improving grounding typically relies on scaled training data or inference-time perception tools, which increases annotation cost or computational cost. To address this challenge, we propose VisionCoach, an input-adaptive RL framework that improves spatio-temporal grounding through visual prompting as training-time guidance. During RL training, visual prompts are selectively applied to challenging inputs to amplify question-relevant evidence and suppress distractors. The model then internalizes these improvements through self-distillation, enabling grounded reasoning directly on raw videos without visual prompting at inference. VisionCoach consists of two components: (1) Visual Prompt Selector, which predicts appropriate prompt types conditioned on the video and question, and (2) Spatio-Temporal Reasoner, optimized with RL under visual prompt guidance and object-aware grounding rewards that enforce object identity consistency and multi-region bounding-box IoU. Extensive experiments demonstrate that VisionCoach achieves state-of-the-art performance across diverse video reasoning, video understanding, and temporal grounding benchmarks (V-STAR, VideoMME, World-Sense, VideoMMMU, PerceptionTest, and Charades-STA), while maintaining a single efficient inference pathway without external tools.

---

🔧 Environment Setup

Run the following script to set up the environment:

bash ./setup.sh

---

📁 Dataset

We use the same dataset as Open-o3-Video. The data structure is organized as follows:

data/videos/
├── stgr/ # STGR (Spatio-Temporal Grounded Reasoning)
│ ├── plm/ # PLM subset
│ │ ├── kfs/ # Keyframes
│ │ └── videos/ # Video files
│ └── temporal_grounding/ # Temporal grounding subset
│ ├── kfs/
│ └── videos/
├── tvg_r1/ # TVG-R1
│ └── videomind_data/
├── videor1/ # Video-R1
├── videoespresso/ # VideoEspresso
├── treevgr/ # TreeVGR
├── gqa/ # GQA └── timerft/ # TimeRFT 

---

🚀 Training

Cold start initialization:

bash ./src/scripts/run_sft_video.sh

Reinforcement learning with GRPO:

bash ./src/scripts/run_grpo_video.sh

---

📊 Evaluation

We provide benchmark evaluation including V-STAR, VideoMME, VideoMMMU, World-Sense, and PerceptionTest.

cdeval
bash ./scripts/eval_all.sh

---

✅ TODOs

• VP-Selector training code
• Visual prompting generation code

---

📝 Citation

If you use our work or our implementation in this repo, or find them helpful, please consider citing:

@misc{lee2026visioncoachreinforcinggroundedvideo,
title={VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting}, author={Daeun Lee and Shoubin Yu and Yue Zhang and Mohit Bansal},
year={2026},
eprint={2603.14659},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.14659}, }

---

🙏 Acknowledgements

We sincerely thank the following projects for their contributions to this work:

• Open-o3-Video
• Video-R1
• R1-V
• ObjectMLLM