Dataset Interactive audio-visual response generation

InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos

Audio-video models are trained to render captions. InteracVid instead pairs a query with the real audio-visual response it triggered, at scale, from live-chat livestreams.

Chi Zhang Haoyang Shi Yueyi Liu Zhaokun Yan Yishu Yin Yuhang Wu Miao Liu^✉

College of AI, Tsinghua University

^✉ Corresponding author · imzc.2004@gmail.com · miaoliu@mail.tsinghua.edu.cn

Paper arXiv Dataset Model Code Browse the clips

Overview of InteracVid: preceding audio-visual context, a live-chat query, and the real audio-visual response that follows. — Figure 1. Each sample keeps the full interaction structure: preceding audio-visual context, an external live-chat query, and the real response that follows.

454K

context-query-response triplets

59K

unique source livestreams, four content domains

979h

real response footage: 76 h real-query, 903 h reconstructed

100

held-out genuine live-chat queries, split by channel

99.7%

clips carry speech; 42% carry extra non-speech sound

Abstract

Learning to respond, not to describe

Recent audio-video models synthesize high-fidelity synchronized content, but the supervision they learn from is descriptive: they render captions rather than produce responses caused by an external interaction.

InteracVid is the first open-source large-scale dataset for that missing supervision. A metadata-aware pipeline extracts over 454K context-query-response triplets from more than 59K livestreams. Fine-tuning on it improves both interaction planning and audio-video generation on a held-out benchmark of 100 genuine live-chat queries.

The response is always real. Where a livestream preserves time-stamped live-chat, the query is observed; where it does not, a vision-language model reconstructs it. The audio-visual target is real footage in both branches, so the reconstructed subset is synthetic only in its input condition.

Dataset samples

Query, spoken response, caption

Twelve per group. Play them with sound on — the point of this data is the audio.

Verbatim live-chat comments with the streamer's spoken reply and the audio-visual caption, from a fixed random sample of 100 clips.

Generation

Ten held-out queries, three responses each

Left to right: the streamer's real reaction, the off-the-shelf co-generator, and the same backbone fine-tuned on InteracVid. Each generated response is eight seconds, conditioned on a single context frame.

Counterfactual samples

Answering the query, not just the room

The query is replaced while the visual context is held fixed. A model that ignored the query would keep producing the original reaction; relevance scored against the substituted query instead holds at 3.98 against 4.03.

Results

Key tables

Responses are generated in two stages, so the planner and the generator can be fine-tuned separately and each contribution attributed. MOVA is the generator throughout; semantic scores are 1–5 ratings from a fixed VLM judge on 100 held-out genuine queries.

p(Y | C, Q) ≈ p(Y | T, C) AV co-generator · p(T | C, Q) Interaction planner

**Table 1. Semantic evaluation of the two-stage pipeline on the genuine-query held-out set.** Planner and generator are fine-tuned separately, isolating each stage. Values after ± are standard errors of the mean; OVRL is the mean of the three dimensions.
Model	Plan. ft.	Gen. ft.	Relev. ↑	Approp. ↑	Expr. ↑	OVRL ↑
(a) Benefits of fine-tuning the AV co-generator
Oracle (GT)	—	✗	4.89 ± 0.20	4.61 ± 0.18	3.15 ± 0.10	4.22
Oracle (GT)	—	✓	4.97 ± 0.15	4.54 ± 0.19	3.41 ± 0.09	4.31
Δ (+ gen. ft.)			+0.08	−0.07	+0.26	+0.09
Qwen3.5-9B	✗	✗	1.85 ± 0.19	1.65 ± 0.17	1.73 ± 0.09	1.74
Qwen3.5-9B	✗	✓	3.60 ± 0.16	2.99 ± 0.19	2.88 ± 0.09	3.16
Δ (+ gen. ft.)			+1.75	+1.34	+1.15	+1.41
InternVL3.5-8B	✗	✗	1.79 ± 0.18	1.43 ± 0.18	1.56 ± 0.09	1.59
InternVL3.5-8B	✗	✓	3.60 ± 0.17	3.24 ± 0.18	2.79 ± 0.09	3.21
Δ (+ gen. ft.)			+1.81	+1.81	+1.23	+1.62
Qwen3-VL-8B	✗	✗	1.93 ± 0.11	1.57 ± 0.10	1.68 ± 0.08	1.73
Qwen3-VL-8B	✗	✓	2.96 ± 0.16	2.50 ± 0.15	2.25 ± 0.10	2.57
Δ (+ gen. ft.)			+1.03	+0.93	+0.57	+0.84
(b) Benefits of fine-tuning the interaction planner
Qwen3.5-9B	✗	✓	3.60 ± 0.16	2.99 ± 0.19	2.88 ± 0.09	3.16
Qwen3.5-9B	✓	✓	3.91 ± 0.16	3.63 ± 0.19	3.25 ± 0.09	3.60
Δ (+ plan. ft.)			+0.31	+0.64	+0.37	+0.44
InternVL3.5-8B	✗	✓	3.60 ± 0.17	3.24 ± 0.18	2.79 ± 0.09	3.21
InternVL3.5-8B	✓	✓	3.89 ± 0.10	3.36 ± 0.16	2.89 ± 0.09	3.38
Δ (+ plan. ft.)			+0.29	+0.12	+0.10	+0.17
Qwen3-VL-8B	✗	✓	2.96 ± 0.16	2.50 ± 0.15	2.25 ± 0.10	2.57
Qwen3-VL-8B	✓	✓	3.39 ± 0.13	3.14 ± 0.13	2.71 ± 0.10	3.08
Δ (+ plan. ft.)			+0.43	+0.64	+0.46	+0.51

Fine-tuning the generator on InteracVid is the dominant effect, worth +1.41 to +1.62 OVRL across planners. Tuning the planner adds a consistent but smaller +0.17 to +0.51. Oracle captions still beat the best learned planner by 0.71 OVRL, so planning a plausible response, not rendering one, is the bottleneck.

**Table 2. Human evaluation of generated interactive responses.** Ten raters each scored ten sampled videos across eight configurations, blind to which produced which. All values are human means. **Bold** marks the best configuration per backbone.
Planner + generator	Relev. ↑	Approp. ↑	Expr. ↑	Video ↑	Audio ↑	OVRL ↑
Oracle (GT) + MOVA (pt.)	4.33	3.89	3.72	3.17	3.61	3.98
Oracle (GT) + MOVA (ft.)	4.41	3.98	3.96	4.00	3.98	4.12
Δ (+ gen. ft.)	+0.08	+0.09	+0.24	+0.83	+0.37	+0.14
Qwen3.5 (pt.) + MOVA (pt.)	2.62	1.92	2.22	1.93	2.48	2.25
Qwen3.5 (pt.) + MOVA (ft.)	3.55	3.14	3.17	3.53	3.79	3.29
Qwen3.5 (ft.) + MOVA (ft.)	4.00	3.59	3.63	3.81	3.96	3.74
Δ (+ gen. ft.)	+0.93	+1.22	+0.95	+1.60	+1.31	+1.04
Δ (+ plan. ft.)	+0.45	+0.45	+0.46	+0.28	+0.17	+0.45
InternVL3.5 (pt.) + MOVA (pt.)	2.04	1.63	1.48	1.37	1.81	1.72
InternVL3.5 (pt.) + MOVA (ft.)	3.86	3.08	3.26	3.86	3.84	3.40
InternVL3.5 (ft.) + MOVA (ft.)	3.96	3.48	3.63	3.63	3.78	3.69
Δ (+ gen. ft.)	+1.82	+1.45	+1.78	+2.49	+2.03	+1.68
Δ (+ plan. ft.)	+0.10	+0.40	+0.37	−0.23	−0.06	+0.29

Ten raters, blind to configuration, reproduce that structure: generator tuning is again dominant (+1.68 and +1.04 OVRL), planner tuning again smaller (+0.29 and +0.45). Configuration-level agreement with the automatic judge is Pearson r = 0.947–0.979 per dimension, and every contrast agrees in sign.

**Table 4. Quantitative evaluation of the audio-visual co-generator.** Block (b) is the main comparison, same backbone before and after fine-tuning; **bold** marks the better of the two. Block (a) groups generators used without fine-tuning; its last two rows are one model conditioned on the first frame alone or additionally on preceding audio and video, where underline marks the better of the pair.
Method	Video quality					Audio quality				A–V sync.
Method	ID ↑	SC ↑	BC ↑	MS ↓	TF ↓	CE ↑	CU ↑	PQ ↑	CER ↓	LC ↑	LD ↓	AVA ↑
(a) Zero-shot generators: no task-specific fine-tuning
OVI	0.905	0.971	0.969	0.008	0.011	5.112	5.576	6.002	0.251	1.984	8.483	0.240
UniAVGen	0.914	0.971	0.973	0.013	0.016	5.242	5.811	6.518	0.694	1.403	9.868	0.316
UniVerse-1	0.911	0.986	0.976	0.004	0.005	3.413	4.352	4.879	0.596	0.354	10.69	0.304
LTX-2.3 (first frame)	0.888	0.969	0.961	0.011	0.015	5.177	6.126	6.841	0.485	2.066	8.308	0.218
LTX-2.3 (+ preceding A/V)	0.882	0.972	0.960	0.011	0.015	5.665	6.463	7.126	0.452	2.056	7.466	0.181
(b) Main comparison: same backbone, oracle caption
MOVA (pt.)	0.868	0.950	0.949	0.023	0.029	5.217	5.823	6.273	0.193	1.456	10.06	0.248
Ours (ft.)	0.881	0.962	0.960	0.014	0.019	5.392	6.154	6.625	0.179	1.519	9.651	0.288
Δ %	+1.5%	+1.3%	+1.2%	+39.2%	+34.7%	+3.4%	+5.7%	+5.6%	+7.1%	+4.3%	+4.1%	+16.3%

ID: identity score. SC / BC / MS / TF: subject consistency, background consistency, motion smoothness, temporal flickering. CE / CU / PQ: content enjoyment, content usefulness, production quality. CER: character error rate over generated speech. LC / LD: lip-sync confidence and distance. AVA: AV-Align. Arrows follow the paper's convention.

Fine-tuning also improves every low-level metric over the pretrained initialization, so the audio and video targets carry the fidelity and synchronization needed to train on. All rows are conditioned on oracle captions, which isolates generator adaptation from planning.

Citation

BibTeX

@article{interacvid2026,
  title   = {InteracVid: Building a Real Interactive Audio-Visual Response
             Dataset from Live-Chat Videos},
  author  = {Zhang, Chi and Shi, Haoyang and Liu, Yueyi and Yan, Zhaokun
             and Yin, Yishu and Wu, Yuhang and Liu, Miao},
  year    = {2026},
  eprint  = {2608.01157},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url     = {https://arxiv.org/abs/2608.01157}
}