TimeScope is an open-source benchmark designed to measure how well vision-language models understand long videos. By adding short “needle” clips into videos ranging from 1 minute to 8 hours, it evaluates three skills:

Recent advances in multimodal AI have produced models claiming to understand hour-long videos. This trend mirrors progress in long-context language models, which excel at reasoning over lengthy text. Following this, vision-language systems now advertise context windows that can handle thousands of frames. But these claims require a closer look: do these models truly demonstrate understanding of the sequence of events? Are they limited to surface-level retrieval \ recognition? It's crucial to ask if their capabilities are being overstated.

Text benchmarks such as HELM and RULER have exposed the fragility of long-context claims, showing that models often struggle when tasks demand more than simple retrieval, like reasoning or aggregation at long context lengths. In the video domain, however, we're still playing catch-up. The most common test, Video Needle in a Haystack (VideoNIAH), injects static images as "needles" into videos, effectively measuring visual search rather than true temporal dynamics. As a result, even top-tier models advertising massive frame capacities are rarely trained beyond ~256 frames and see sharp drops on benchmarks like Video-MME when pushed further.

This measurement gap leaves us wondering: What does it really mean for a model to "understand" long videos? To address this, we're excited to introduce TimeScope , a new open-source benchmark hosted on Hugging Face. TimeScope probes the limits of long-video capabilities by inserting several short (~5-10 second) video clips —our "needles"—into base videos ranging from 1 minute to 8 hours.

Source link