Efficient Long-Form Video Understanding: Scaling Perception, Selecting Evidence, and Adapting Computation

Long-form video understanding requires capturing temporal events, identifying relevant visual evidence, and reasoning across extended sequences. However, processing long videos is computationally demanding: adapting visual representations consumes substantial training memory, vision-language models have limited visual context, and extended reasoning incurs unnecessary costs for simpler questions. This dissertation investigates how computation can be allocated more effectively across perception, evidence selection, and reasoning.

The first part addresses efficient representation learning for temporal action detection. While frozen visual features limit task adaptation, full end-to-end fine-tuning requires substantial memory. AdaTAD introduces lightweight temporal-informative adapters that enable task-specific learning with frozen backbones, allowing larger models and longer video inputs. ETAD further reduces gradient computation, while OpenTAD provides a unified framework for studying and evaluating temporal action detection methods.

The second part focuses on selecting relevant visual evidence under limited context budgets. Uniform frame sampling often misses important moments, particularly when relevant information is sparsely distributed. BOLT introduces training-free, query-conditioned frame selection that balances relevance and temporal diversity, enabling vision-language models to better utilize their limited visual context without additional training.

The final part investigates adaptive reasoning for video understanding. Extended chain-of-thought reasoning benefits complex questions but can waste computation or cause overthinking on simpler tasks. VideoAuto-R1 introduces a confidence-adaptive framework that first generates a direct answer and then determines whether further reasoning is necessary. This allows the model to reduce unnecessary reasoning while maintaining competitive accuracy.

Together, these contributions establish a unified principle for efficient long-form video understanding: allocating computation according to the downstream task during representation learning, query relevance during evidence selection, and prediction confidence during reasoning. This enables more scalable and computationally efficient video understanding systems.