Paper Tracker Daily Reading

多模态大模型

3 papers

No. 1

Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Training

Hanfeng Lu, Tianyu Feng, Suyi Li, Yuheng Zhao, Wei Gao, Shaopan Xiong, Ju Huang, Siran Yang, Jiamang Wang, Lin Qu, Wei Wang

Sourcearxiv

Published2026-08-14

Updated2026-08-14

Classcs.LG · cs.DC

TLDR提出Rollplex运行时，通过将VLM的prefix计算与rollout解码并行，并采用阶段感知内存管理和并行感知权重共享，在保持同步RL语义的同时显著提升训练吞吐。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Vision-language models (VLMs) enable embodied agents to reason and act from visual observations and language instructions. Reinforcement learning (RL) post-training enhances these capabilities using task feedback, but current on-policy RL runtimes execute rollout, reference scoring, and actor training in strict serial phases. While effective for text-only RL, this phase-granular execution is wasteful for VLMs, where processing dense video inputs and prompt prefixes occupies a large fraction of each phase. Because prefix processing is independent of the generated response, it can be run alongside rollout decoding, which leaves GPU compute capacity underutilized, without breaking synchronous on-policy semantics. We present Rollplex, a runtime that decomposes the reference and training phase and moves the prefix computation into the rollout decode window. Realizing this schedule requires more than concurrent kernel launches: naive colocation of Qwen2.5-VL-32\,B requires roughly 165\,GiB per GPU, while rollout and training prefer different tensor-parallel (TP) degrees and weight layouts. Rollplex addresses these constraints with two mechanisms. Phase-aware memory management controls HBM residency according to producer--consumer lifetimes. Parallelism-aware weight sharing uses the same physical storage for layout-compatible tensors across distinct TP degrees and reconstructs only incompatible tensors, avoiding a complete second actor copy. On 32 H800 GPUs, Rollplex achieves $1.23\times$--$1.30\times$ speedup over serial colocation and $1.57\times$--$2.24\times$ over disaggregation under the same GPU budget, while preserving the synchronous RL update.

Motivation

现有RL后训练运行时按阶段串行执行rollout、参考评分和actor训练，对VLM而言，处理密集视频和前缀占用大量计算，但前缀计算独立于生成响应，可并行化，当前串行方式浪费GPU资源。

Method

Rollplex将参考和训练阶段分解，把前缀计算移入rollout解码窗口；采用阶段感知内存管理控制HBM驻留，以及并行感知权重共享，在不同TP度间复用布局兼容的张量，仅重建不兼容部分，避免完整第二份actor副本。

Result

在32块H800 GPU上，相比串行共置获得1.23x-1.30x加速，相比同预算下的分离部署获得1.57x-2.24x加速，同时保持同步RL更新语义。

Conclusion

Rollplex通过重新调度计算和优化资源利用，有效提升VLM RL后训练效率，证明了在保持同步性的前提下利用前缀并行性的可行性。

No. 2

Improving Generalization Robustness of Multimodal RLVR

Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng, Chenrui Zhou, Lama Moukheiber, Yixing Ma, Bin Xu, Jiajun Song, Zhenglin Wan, Wangbo Zhao, Jiasheng Tang, Bohan Zhuang, Fan Wang, Yang You

Sourcearxiv

Published2026-08-09

Updated2026-08-14

TLDR本文提出PIRL，通过动态三值奖励和基于嵌入空间对抗的一致性正则化，增强RLVR对提示扰动的鲁棒性，在压力测试中性能下降不超过1%，优于GRPO。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only $\le 1\%$, where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.

Motivation

RLVR虽能提升多模态大模型准确性，但对问题改写或模板变化敏感，导致在医疗VQA等高风险场景中不可靠。原因在于二元奖励混淆格式与内容，且训练分布覆盖不足。

Method

PIRL包含两个组件：动态三值奖励（区分格式错误与内容错误）和基于嵌入空间对抗的一致性正则化（强制模型对语义等价的扰动提示产生不变策略）。

Result

在压力测试中，PIRL平均准确率下降≤1%，而GRPO下降约3%；在动态评估中，PIRL也取得最小性能下降。

Conclusion

通过分离格式与语义奖励并施加策略不变性，PIRL显著提升RLVR的鲁棒性，为高风险场景下的可靠部署提供了有效方案。

No. 3

CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations

Shanmukha Vellamcheti, Uday Kiran Kothapalli, Disharee Bhowmick, Sathyanarayanan N. Aakur

Sourcearxiv

Published2026-03-22

Updated2026-08-14

TLDR本文提出CVT-Bench基准，系统评估多模态大语言模型在空间任务中的状态完整性，发现模型在孤立任务上表现良好，但在跨视角和竞争场景下准确率、一致性和可实现性显著下降，且现有提示或场景表示无法完全解决该问题。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Multimodal large language models (MLLMs) perform strongly on isolated spatial tasks, but whether their predictions remain persistent and mutually coherent across viewpoints and competing scenes is unclear. We formalize this behavioral property as spatial-state integrity and introduce CVT-Bench, a factorial diagnostic suite spanning two domains (CVT-Synthetic, CVT-Real), four context regimes (CVT-Isolated, irrelevant filler, attribute filler, CVT-Competing), three representations (Image, Text/BBox, Scene Graph), and ten viewpoint conditions (nine azimuths from $0^\circ$ to $360^\circ$ at $45^\circ$ increments, plus Top) with target views withheld. Across 200 scenes and 11,833 relational queries, we measure counterfactual accuracy, cycle consistency, the normalized survival coefficient, spatial realizability, and context-specific interference. Five state-of-the-art MLLMs exhibit rapid persistence loss and frequently produce jointly unrealizable states despite high local accuracy, with failures amplified by competing contexts and natural-scene complexity. Matched irrelevant and attribute filler controls show that prompt length, position, or loss of scene access alone are insufficient explanations. On average, Text/BBox improves accuracy, persistence, and realizability, while Scene Graph provides complementary gains; neither eliminates the instability. Thus, isolated spatial accuracy substantially overestimates robustness, establishing spatial-state integrity as a distinct evaluation target. The complete benchmark and codebase will be publicly released.

Motivation

现有评估仅关注孤立空间任务，忽略了模型预测在跨视角和竞争场景下的持久性与相互一致性，导致对模型鲁棒性的高估。

Method

构建CVT-Bench，包含合成与真实场景、四种上下文（孤立、无关填充、属性填充、竞争）、三种表示（图像、文本/边界框、场景图）和十种视角条件，通过五个指标（反事实准确率、循环一致性、归一化生存系数、空间可实现性、上下文干扰）评估五个SOTA MLLM。

Result

模型在孤立任务上准确率高，但跨视角时持久性快速下降，常产生联合不可实现的状态；竞争上下文和自然场景复杂度加剧失败；无关填充和属性填充对照表明提示长度、位置或场景访问缺失不足以解释失败；文本/边界框和场景图表示有改善，但无法消除不稳定性。

Conclusion

孤立空间准确率显著高估模型鲁棒性，空间状态完整性应作为独立评估维度，现有MLLM在此方面存在明显缺陷。

视频多模态理解

3 papers

No. 1

EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory

Le Zhang, Ke Sun

Sourcearxiv

Published2026-08-12

Updated2026-08-12

Classcs.CV · cs.AI · cs.CL · cs.HC

TLDR提出EgoCITE框架，通过上下文增强索引和时间感知检索，解决长时程自我中心视频问答中的索引不可靠和忽略时间意图问题，显著提升准确率并降低成本。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Long-horizon egocentric memory transforms continuous first-person video and audio into a searchable record of past experiences. We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent. To address both bottlenecks, we introduce EgoCITE (Egocentric Context-augmented Indexing and Time-aware Evidence retrieval), a long-horizon agentic memory framework for egocentric QA. EgoCITE comprises three components. EgoScheme uses local multimodal context to turn fragmentary video captions and speech transcripts into self-contained atomic memory indices. EgoIndex organizes complementary action, activity, utterance, and conversation representations into searchable multi-view memory indices at multiple granularities. EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved evidence. We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench in terms of answer accuracy and target-event retrieval alignment. EgoCITE improves accuracy over agentic memory baselines by at least 4.4--14.2\% while achieving 36$\times$ lower cost than long-context LLM agents.

Motivation

现有长时程自我中心记忆系统存在两个瓶颈：基于上下文贫乏字幕的索引不可靠，以及检索忽略问题的时间意图。

Method

EgoCITE包含三个组件：EgoScheme利用局部多模态上下文将片段化字幕和语音转为自包含原子索引；EgoIndex组织多视图、多粒度记忆索引；EgoRetrv结合语义搜索与问题条件的时间相关性评分和证据筛选。

Result

在EgoLifeQA、EgoMem和EgoR1-Bench上，EgoCITE相比智能体记忆基线准确率提升4.4%-14.2%，且成本比长上下文LLM智能体低36倍。

Conclusion

EgoCITE通过上下文增强索引和时间感知检索，有效提升长时程自我中心问答的准确性和效率，验证了其优越性。

No. 2

StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos

Julian Spravil, Sebastian Houben, Sven Behnke

Sourcearxiv

Published2026-08-12

Updated2026-08-12

TLDR本文提出StrAD基准，将长视频自动音频描述（AD）重新定义为流式密集视频字幕生成，无需真实时间戳，支持微调和零样本模型，并在多个基准上取得领先结果。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Visual content is the dominant medium of communication, yet without audio descriptions (ADs), it remains inaccessible to blind and low-vision people. ADs narrate context-relevant visual events during natural audio pauses. Manually creating ADs is expensive, limiting coverage to a small fraction of available content. Most existing automatic AD generation methods frame the task as video clip captioning, requiring ground-truth timestamps and additional context cues such as character databases. Current benchmarks reinforce this framing, consisting of short video segments paired with automatic or task-mismatched annotations. We introduce StrAD, a benchmark for long-form AD generation on full-length videos spanning diverse genres such as movies, documentaries, short films, performances, and video games. We reformulate AD generation as streaming dense video captioning. Our approach processes full-length videos with a sliding window, inserting ADs into existing transcripts without ground-truth timestamps, and supports both fine-tuned models and zero-shot prompting of vision-language models. On the segment-level task with given timestamps, our fine-tuned StrAD-FT sets the state of the art on CMD-AD with 36.3 CIDEr (+10.0 over Shot-by-shot), establishes a reference point on StrAD (51.0 CIDEr), and remains competitive on MAD-Eval at 24.9 CIDEr. On the full-video streaming task, StrAD-FT reaches a SODA score of 2.4 against 1.1 for our zero-shot baseline StrAD-Zero, though both exhibit limitations in temporal localization and narrative coherence. While prior work has tackled full-video AD generation in an offline, multi-stage fashion, ours is the first streaming approach, generating ADs on the fly without ground-truth timestamps. StrAD makes progress on full-video AD generation measurable, a prerequisite for scaling accessibility.

Motivation

手动创建音频描述成本高，覆盖有限；现有自动方法依赖时间戳和额外上下文，基准局限于短片段，无法处理完整长视频。

Method

采用滑动窗口处理完整视频，将AD生成视为流式密集视频字幕任务，在现有转录中插入描述，无需真实时间戳；支持微调模型（StrAD-FT）和零样本视觉语言模型（StrAD-Zero）。

Result

在CMD-AD上达到36.3 CIDEr（+10.0），在StrAD上51.0 CIDEr，在MAD-Eval上24.9 CIDEr；全视频流式任务中StrAD-FT的SODA为2.4，优于StrAD-Zero的1.1。

Conclusion

StrAD是首个流式全视频AD生成方法，使长视频AD生成可测量，但时间定位和叙事连贯性仍有局限，为未来工作奠定基础。

No. 3

Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models

Oscar Chew, Serhii Honcharenko, Qian-Hui Chen, Patricia Lu, Dishant Zaveri, Khoa D. Doan, Kuan-Hao Huang

Sourcearxiv

Published2026-05-26

Updated2026-08-11

Classcs.CV · cs.CL

TLDR本文提出DistractionBench基准，通过插入无关广告片段测试视频大模型，发现所有模型普遍存在将不同片段中的主体与事件错误关联的幻觉，称为“事件袋”行为，表明模型缺乏时间接地能力。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

A key capability for video understanding is reliably linking subjects to events across time, yet whether Video Large Language Models (VideoLLMs) actually achieve this remains unclear. In this work, we introduce DistractionBench to evaluate whether VideoLLMs can robustly link subjects and events in the presence of unrelated video segments. Through controlled interventions, such as inserting short advertisement clips into longer videos, we show that VideoLLMs frequently hallucinate interactions between entities from different segments, incorrectly attributing actions from injected advertisements to subjects in the main video. We characterize this systematic hallucination as bag-of-events (BoE) behavior, where models process videos as collections of events rather than temporally structured sequences. Evaluating 11 popular VideoLLMs, we find that all models exhibit substantial BoE behavior. Our findings suggest that VideoLLMs lack reliable mechanisms for temporal grounding and motivate the development of models with more robust subject-event association.

Motivation

视频理解的关键能力是跨时间可靠地将主体与事件关联，但现有VideoLLMs是否真正具备此能力尚不明确。

Method

设计DistractionBench基准，通过在长视频中插入短广告片段等受控干预，评估模型在存在无关片段时的主体-事件关联能力，并量化幻觉行为。

Result

评估11种流行VideoLLMs，发现所有模型均表现出显著的事件袋行为，即错误地将广告中的动作归因于主视频中的主体。

Conclusion

VideoLLMs缺乏可靠的时间接地机制，需要开发更稳健的主体-事件关联模型。

多模态检索与 RAG

3 papers

No. 1

HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation

Yin Li, Ziyang Hu, Zhiyu Guo, Xiangyu Liu, Wenbin Li, Boo-Ho Yang, Rav Lawana, Ziyue Li, Wei Zeng, Fugee Tsung

Sourcearxiv

Published2026-08-14

Updated2026-08-14

Classcs.IR · cs.AI

TLDR提出层级感知的多模态RAG框架（HAM-RAG），利用文档层级结构作为检索和生成的锚定信号，提升证据选择、放置和局部图文对齐；并构建HAM-Bench基准，在多个数据集上显著优于非层级基线。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement. We propose HAM-RAG, a Hierarchy-Aware Multimodal RAG framework for structure-faithful interleaved generation. HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation, contextualizing textual and visual evidence and preserving source position and local text-image relations in the prompt. We further introduce HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents. Across multiple backbones, HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline. On Wukong, HAM-RAG improves Img-CBS by 24.2% over the strongest non-hierarchical baseline, demonstrating substantially better local text-image alignment. The main experiments and ablation study together demonstrate that document hierarchy is a key grounding signal for faithful image selection, placement, and local text-image alignment. These findings highlight the value of hierarchy-aware grounding for reliable multimodal assistants that generate answers faithful to the source organization, procedural structure, and local text-image evidence of structured documents, such as technical manuals, maintenance guides, and industrial SOPs. The code is available at https://github.com/MCCodeAI/HAM-RAG.git.

Motivation

现有多模态RAG方法将结构化文档扁平化为孤立的文本和图像单元，破坏了源文档的组织结构和局部图文逻辑，导致证据选择与放置不忠实。

Method

HAM-RAG利用文档层级作为检索和生成的锚定信号，在提示中保留源位置和局部图文关系，从而对文本和视觉证据进行上下文化处理。

Result

在多个骨干网络上，HAM-RAG在多模态平均指标上比最强非层级基线提升17.3%；在Wukong数据集上，Img-CBS提升24.2%，显著改善局部图文对齐。

Conclusion

文档层级是忠实图像选择、放置和局部图文对齐的关键锚定信号，对技术手册、维护指南和工业SOP等结构化文档的可靠多模态助手具有重要价值。

No. 2

S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modeling

Xueqi Wang, Zhigang Wang, Runqing Zhang, Zhenqi Jia, Junfeng Zhao

Sourcearxiv

Published2026-08-14

Updated2026-08-14

TLDR提出S2Dialog框架，用于多模态对话级别的语义与风格检索，通过文本和声学检索器及对比学习，在DailyTalk数据集上取得优异性能。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Multimodal dialogue retrieval aims to retrieve dialogues from multimodal dialogue banks that are similar to a target dialogue in terms of both textual semantics and acoustic conversational styles. Such dialogue-level retrieval is crucial for many dialogue-related tasks, including Emotion Recognition in Conversation, Spoken Dialogue Systems, and Conversational Speech Synthesis, where external dialogue examples can provide valuable semantic and stylistic references. However, existing retrieval methods are still largely limited to utterance-level or unimodal matching, and often fail to capture the global semantic coherence and stylistic consistency of an entire dialogue. To address this gap, we propose S2Dialog, a unified framework for dialogue-level semantic-style retrieval from multimodal dialogue banks. Specifically, S2Dialog consists of a Dialogue-level Textual Retriever and a Dialogue-level Acoustic Retriever, which encode the textual and acoustic modalities of a dialogue into dialogue-level representations, respectively. To further enhance multimodal retrieval, we introduce Dialogue-level Textual-Acoustic Contrastive Learning, which aligns semantically and stylistically similar dialogues while distinguishing unrelated ones. Extensive experiments on the multimodal dialogue dataset DailyTalk demonstrate that S2Dialog achieves outstanding retrieval performance.

Motivation

现有检索方法局限于话语级或单模态匹配，无法捕捉整个对话的全局语义一致性和风格一致性，而对话级检索对情感识别、对话系统等任务至关重要。

Method

S2Dialog包含对话级文本检索器和对话级声学检索器，分别编码文本和声学模态为对话级表示，并引入对话级文本-声学对比学习，对齐语义和风格相似的对话，区分不相关对话。

Result

在多模态对话数据集DailyTalk上的大量实验表明，S2Dialog取得了突出的检索性能。

Conclusion

S2Dialog有效解决了对话级多模态检索问题，验证了对话级语义-风格联合建模的优越性。

No. 3

Generative Universal Multimodal Retrieval with Dual-role Identifiers

Kaipeng Li, Haitao Yu, Xuanchen Zhou

Sourcearxiv

Published2026-08-13

Updated2026-08-13

Classcs.IR · cs.AI

TLDR提出DrIG，一种基于双角色标识符的生成式多模态检索框架，通过顺序解码和集合重解释缓解前缀错误，在M-BEIR和文生图数据集上超越现有生成式基线，并接近稠密检索性能。

PDF Abstract

阅读摘要与笔记点击展开

Abstract

Generative information retrieval (GIR) has emerged as a compelling alternative to the conventional index-retrieve-then-rank retrieval pipeline by training a generator to produce the identifiers of relevant items directly. Despite its promise, a number of open challenges still remain. First, constrained left-to-right decoding is vulnerable to prefix-level errors and local optima. Second, most prior GIR research remains largely unimodal, leaving instruction-aware retrieval across text, image, and mixed image-text items underexplored. Third, although discrete identifier-based GIR offers higher efficiency, its retrieval accuracy still lags behind that of the cutting-edge dense-vector-based retrieval methods. Motivated by these challenges, we propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers, which supports diverse retrieval tasks across multiple modalities and domains. Each candidate is assigned a single residual-quantized identifier that serves two complementary roles. In its sequential role, the identifier is decoded autoregressively, where the first token explicitly models modality and the remaining tokens capture progressively finer semantics. In its set-based role, the same tokens are reinterpreted as an unordered set to provide a prefix-independent relevance prior, which guides constrained beam search and alleviates local-optimum errors. Extensive experiments on the M-BEIR benchmark and the text-to-image evaluation datasets show that:(1)DrIG consistently outperforms state-of-the-art generative multimodal baselines across diverse tasks, while hybrid reranking achieves a favorable efficiency-effectiveness trade-off against strong dense retrievers. (2)Ablation and scaling analyses reveal how the base LMM, beam size, reranking depth, and fusion strategy affect retrieval performance, providing practical guidance for system design.

Motivation

现有生成式检索存在前缀级错误和局部最优问题，且大多局限于单模态，检索精度落后于稠密向量方法。

Method

为每个候选分配一个残差量化标识符，兼具顺序角色（首token建模模态，后续token细化语义）和集合角色（作为无序集合提供前缀无关的相关性先验，指导约束束搜索）。

Result

在M-BEIR和文生图数据集上，DrIG一致优于最先进的生成式多模态基线；混合重排在效率与效果间取得良好平衡，并优于强稠密检索器。

Conclusion

DrIG验证了双角色标识符的有效性，消融和扩展分析为系统设计提供了实用指导。