StreamClarify

WACV 2027

StreamClarify

Proactive Clarification for Grounded
Streaming Video Question Answering

Zhuchen ShaoBo ChenKlara Nahrstedt

University of Illinois Urbana-Champaign

Check when to ask.
Retrieve past video with user guidance.

Method animationVideo questions

(a) Streaming input

Observed drill-scene frame at 0 seconds
0 s
Observed drill-scene frame at 15 seconds
15 s
Observed drill-scene frame at the 20-second query
20 s · query

Question Qi “What color is this?”

Future frames are unavailable.

(b) When to ask

Would the answer rely on video evidence rather than general knowledge?

Yes / No judgmententropy H

H ≤ τAnswer directly active TP-KV only

H > τRequest guidance

(c) Memory for re-answering

After requested guidance is available

Active contextbounded TP-KV
Searchable archiveseparate CPU storage

User guidance suppliedretrieve

Temporary answering context

active context retrieved frames

Conceptual overview of trigger selection and memory.

Setting

A question arrives while the video continues

A frozen VLM receives a question at time ti. It can use only the video observed so far.

Bounded active context. Keep recent detail. Compress older context within a fixed token budget.

A separate archive. Store searchable representations on the CPU. This archive sits outside the active token budget.

Core method

Triggering and memory

About this step

01 / Observe

Keep recent detail, compress older context.

StreamClarify-QA

Questions over streaming video

1,837 timestamped questions across 355 videos. Each question uses the video observed so far. Play short excerpts from StreamBench and OVO-Bench below.

Standard

995

Well-specified questions with a supported answer.

Finding the red teapot
01:04–01:12
StreamBench · Question at 01:20

QuestionWhere is a red teapot?

Reference answerThere is a red teapot on the cabinet opposite the refrigerator.

Ambiguous

302

Questions with an unclear intended referent.

Which object is “this”?
00:12–00:20
StreamBench · Question at 00:20

QuestionWhat color is this?

Reference answerThe electric drill you're holding is likely black and red.

“This” could refer to the drill or the sweatshirt.

Source question

What color is the electric drill I'm holding?

Unanswerable

540

No supported substantive answer option.

A missing color option
00:00–00:05
StreamBench · Question at 00:16

QuestionWhat color is this man's hat?

  1. The hat is brown.
  2. None of the above
  3. The hat is blue.
  4. The hat is red.

Reference answerNone of the above

The hat is black. Black is missing from the offered choices.

Original video excerpts from the observed prefix. Answers are dataset reference annotations, not model predictions.

How the dataset is constructed
01Source video + timed questions

260 StreamBench videos
95 OVO-Bench videos

02Timestamp-aligned captions

Dense video descriptions guide
query transformation and checking.

03Three question types

Retain, rewrite, or construct
questions.

Final benchmark composition
SourceVideosStandardAmbiguousUnanswerableTotal QA
StreamBench2609132543891,556
OVO-Bench958248151281

Source selection and captions. OVO videos contain native HLD questions. Queries less than 5 seconds apart are filtered. MiniCPM-V 4.5 captions use 2 FPS, 448-pixel inputs, 6-second windows and a 3-second stride. Source questions are included; answers are withheld.

Ambiguity review. Replace specific noun phrases with pronouns. Human review retains 302 of 404 candidates with plausible alternative referents.

Unanswerable checks. For 389 StreamBench questions, check distractors against caption prefixes, remove the correct option and add “None of the above”. Screen all raw video prefixes; review 7 flagged cases. Keep the 151 native OVO-Bench HLD labels unchanged.

Example annotations and frame provenance.

Experimental setup

What each experiment measures

Three frozen 8B backbones: LLaVA-OV-1.5, InternVL-3.5, and Qwen3-VL. No task-specific fine-tuning.

10K active tokens · 20 s recent window · 0.25 compression ratio · top-16 retrieval

Archived paper results. Three queries in one video included later frames in their saved context. The release fixes query order; aggregate scores have not been rerun.

StudyQuestions and comparisonWhat is measured
01Trigger selection

1,837 questions · three backbones. Random and entropy selection use comparable rates. Selected queries receive oracle guidance.

Initial-error precision and recall. Answer accuracy and latency.

02Matched 6K context

Qwen3-VL · each source separately · clarification off. Methods share a 6K active-context limit and retain their own memory designs.

Accuracy under a shared active-context limit. CPU storage is separate.

03Standard configuration

Each source and backbone separately · clarification off. Each method uses its standard memory configuration.

Accuracy, time to first token, and processing FPS. Charts below show Qwen3-VL.

04Generated-question PoC

302 ambiguous questions · Qwen3-VL. An evaluation-only referent-aware gate checks follow-ups. Accepted questions receive a fixed reply; rejected questions use direct retrieval.

Simulated accuracy, corrections, and harms. Entropy selection uses 162 questions.

Metrics and evaluation protocol
Answer scoring
Qwen3-14B judges answers using offline captions. These captions are judge inputs only.
Micro / Macro
Micro averages questions. Macro averages the three question types.
Trigger precision
How many selected answers were initially wrong?
Trigger recall
How many initial errors were selected?
Uncertainty score
Ranks initial errors. It does not identify their cause or predict clarification benefit.
Confidence intervals
Paired 95% intervals resample whole videos.
Latency
No-clarification and clarification timings are separate. Gate time is reported separately; simulated replies exclude real-user delay.
6K context
Counts active visual context, not total storage. Methods retain different memory mechanisms; CPU storage is separate.

Results

Trigger selection and bounded context

Selecting initially incorrect answers

Trigger precision · 1,837 questions · comparable selection rates

Experiment details and full numbers

Additional study · Qwen3-VL

Generated questions: a proof of concept

Five recorded cases: corrections, retrieval success, and a rejected follow-up. Original video excerpts; simulated replies.

Observed video

Step through the recorded clarification branch.

Loading example

    Recorded model outputs and simulated replies.

    Retrieved frames and simulation protocol

    The gate knows the annotated referent. It accepts a follow-up or falls back to direct retrieval. Accepted questions receive a fixed referent-only reply. Retrieved frames are candidates, not verified answer evidence.

    Citation

    Download BibTeX
    @inproceedings{shao2027streamclarify,
  title     = {StreamClarify: Proactive Clarification for Grounded Streaming Video Question Answering},
  author    = {Shao, Zhuchen and Chen, Bo and Nahrstedt, Klara},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  year      = {2027}
}