VLM WikiVision-Language Model 资料整理与学习笔记看到了什么、发生了什么、为什么发生、何时响应、如何持续低延迟运行参考资料见文末Resources目录Course计算机视觉基础Transformer 与多模态基础ConceptVLM 基本架构视频理解与视频推理PaperVLM 基座视频推理主动交互When to Speak长期记忆实时推理Streaming with ThinkingBenchmarkDatasetProjectBook学习路线Resources一、Course计算机视觉基础Stanford CS231n: Deep Learning for Computer VisionCNN、视觉表征、检测与分割基础Hugging Face Community Computer Vision Course从视觉基础到 Vision TransformerUvA Deep Learning TutorialsPyTorch、Attention、ViT 等可运行教程Dive into Deep LearningComputer Vision卷积、检测、分割与数据增强Transformer 与多模态基础The Illustrated Transformer先建立 Attention、Encoder、Decoder 的直觉The Annotated Transformer用 PyTorch 阅读 Transformer 实现Hugging Face LLM CourseTokenizer、Transformer、微调与推理Stanford CS336: Language Modeling from Scratch语言模型训练与系统基础LLaVA-NeXT Documentation图像、视频、多模态模型与评测生态二、ConceptVLM 基本架构Image / Video ↓ Vision EncoderViT / CLIP ↓ Projector / Adapter / Q-Former ↓ Visual Tokens Text Tokens ↓ Large Language Model ↓ Caption / QA / Reasoning / ActionVision Encoder把像素转换成 patch tokenProjector / Adapter将视觉特征映射到 LLM 的表示空间Visual TokenLLM 能消费的视觉序列Position Encoding / RoPE表达时间与空间位置SFT使用图文、视频问答与 caption 数据训练模型Prefill / Decode / KV Cache决定多模态模型的推理速度与显存视频理解与视频推理视频比单图多了时间轴。模型不仅要识别物体还要处理动作、状态变化与事件边界事件先后顺序、持续时间和因果关系跨帧目标跟踪与空间关系长视频记忆与证据检索音频、字幕和画面的同步视频推理可以按输出和思考形式分为CoT-based Video Reasoning以语言 Chain-of-Thought 为主CoF-based Video Reasoning以帧、视频生成或视觉状态变化作为推理过程Interleaved Video Reasoning视频、图像与文本交替进入推理链Streaming Video Reasoning视频持续到来模型只能使用过去和当前信息Streaming Video Understanding离线视频模型可以提前看到完整视频Streaming 模型在时刻t tt只能使用I t { F 1 , F 2 , … , F t } \mathcal{I}_t \{F_1,F_2,\ldots,F_t\}It​{F1​,F2​,…,Ft​}不能预览F t 1 F_{t1}Ft1​也不能依赖未来证据。核心问题分为两类Proactive / When to Act什么时候回答、提醒、输出 caption 或保持沉默Reactive / How to Sustain如何控制不断增长的视觉 token、记忆、KV Cache 和计算量常见系统形态Video Stream → Frame/Chunk Buffer → Trigger/Gate → VLM ↓ ↓ ↓ Short-term Memory Speak? Response ↓ Long-term Memory / RetrievalStreaming input视频帧持续到达Streaming output模型逐 token 输出文字Dataset streaming训练时边下载边读取数据三、PaperVLM 基座CLIP: Learning Transferable Visual Models From Natural Language Supervision视觉与文本对比学习基础Flamingo: a Visual Language Model for Few-Shot Learning跨注意力连接视觉编码器与冻结 LLMBLIP-2: Bootstrapping Language-Image Pre-trainingQ-Former 对齐视觉与语言LLaVA: Visual Instruction TuningCode视觉指令微调经典工作Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any ResolutionCode动态分辨率与多模态 RoPEQwen2.5-VL Technical ReportCode长视频、定位与视觉 AgentLLaVA-OneVision: Easy Visual Task TransferCode统一单图、多图和视频任务OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence利用运动与残差信号进行 codec-aligned 视觉编码Video-LLM 基座Video-LLaVA: Learning United Visual Representation by Alignment Before ProjectionCodeVideoChat2: Advancing Spatial-Temporal Modeling and Post-Training in Video-LLMsCodeLLaVA-NeXT: A Strong Zero-shot Video Understanding ModelCodeLongVA: Long Context Transfer from Language to VisionCodeLongVU: Spatiotemporal Adaptive Compression for Long Video-Language UnderstandingCode视频推理Language-centric / CoTVideo-R1: Reinforcing Video Reasoning in MLLMsCodeVideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-TuningCodeThinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video ReasoningCodeOpen-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal EvidenceCodeVideo-Thinker: Sparking “Thinking with Videos” via Reinforcement LearningCodeVision-centric / CoFChain-of-Frames: Advancing Video Understanding via Frame-Aware ReasoningCodeAre Video Models Ready as Zero-Shot Reasoners? MME-CoFDatasetThinking with Video: Video Generation as a Promising Multimodal Reasoning ParadigmCodeInterleaved ReasoningThinking With VideosCode通过工具调用在推理中主动重看视频ViTCoT: Video-Text Interleaved Chain-of-ThoughtCodeLongVT: Thinking with Long Videos via Native Tool CallingCode主动交互When to Speak生成式 Token 触发让模型生成EOS、silence、response或动作 token统一学习“是否回答”和“回答内容”。VideoLLM-online: Online Video Large Language Model for Streaming VideoCodeStreaming EOS 代表保持沉默What to Say and When to Say ItCodenext与feedback动作 tokenLiveCC: Learning Video LLM with Streaming Speech Transcription at ScaleCode实时赛事解说与 Streaming EOSProAssist: Proactive Assistant Dialogue GenerationCode帧级 EOS 与负样本下采样Streaming Video Instruction TuningCodeSilence、Standby、Response 三状态 token辅助 Head / Detector 触发使用轻量分类器、路由器或 Activation Model 判断是否调用大模型。MMDuet: VideoLLM Knows When to SpeakCodeInformative Head 与 Relevance HeadDispiderCode解耦感知、决策和响应StreamMindCodeEvent-Gated Cognition面向高帧率视频ViSpeakCodeInformative Head 判断视觉反馈时机StreamBridgeCode小 Activation Model 驱动离线 Video-LLMProact-VLCodeFLAG token 与 gated response headFeature / Event 触发TimeChat-Online: 80% Visual Tokens are Naturally RedundantCode差分 token drop 和场景变化触发LiveStarCode通过 perplexity 验证是否需要输出QueryStreamCodequery-aware differential pruning 与相关性触发ColorTriggerCode灰度常开、彩色按需激活CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inferencemotion vector 引导 patch pruningI 帧刷新 KV Cache强化学习优化响应时机MMDuet2Code多轮 RL 学习 Reply / No ReplyThinking in Streaming VideoCodeWatch–Think–Speak 与 streaming RLVR长期记忆滑动窗口与淘汰A Simple Baseline for Streaming Video UnderstandingCode固定最近帧窗口是重要 baselineStreamingVLM: Real-Time Understanding for Infinite Video StreamsCodeAttention Sink、滑动窗口和连续 RoPEProact-VLCode双 Cache 滑窗与 Reverse-RoPE eviction层次化与事件记忆VideoChat-Online / OVBenchCodePyramid Memory BankStreamChatCode短期记忆、长期记忆树与对话检索StreamForestCodePersistent Event Memory ForestVideoScaffoldCode弹性事件切分与层次化聚合EventMemAgentCode事件中心双层记忆与工具调用FluxMemCode短、中、长期自适应层次记忆Token 与 KV Cache 压缩VideoScanCode每帧压缩为 Semantic Carrier TokenInfiniPot-VCode时间冗余和 Value-Norm 驱动 KV 压缩StreamMemProjectquery-agnostic KV pruning 与 mergingStreamingTOMCode视觉 token 压缩与 4-bit KV MemoryStreamKVCode分段 KV 检索与压缩HERMESCode将 KV Cache 组织成感觉、工作与长期记忆检索增强记忆ReKVCodeKV Cache 下放 CPU/磁盘并按问题检索Flash-VStreamCode摘要记忆与细节增强记忆CogStreamCode事件压缩与历史对话检索WeaveTimeCode不确定性触发的粗到细历史检索实时推理编码—解码并行Speak While WatchingCode并行感知与生成Think-as-You-SeeCodeParallel Dual KV CacheThink While WatchingCode异步 Watch–Think Pipeline稀疏调用与视觉计算VideoLLM-MoDMixture-of-Depths 跳过冗余视觉层计算LION-FSCodeFast Path 判断时机Slow Path 生成回答StreamMindCode轻量 Gate 持续运行重模型按事件调用STC: Hierarchical Token CompressionCodeViT 特征复用与 LLM 前 token pruningAutoGazeCode在 ViT 前自回归选择少量多尺度 patch推理系统关键指标TTFTTime to First TokenTPOTTime per Output TokenRTF推理耗时 / 视频时长实时系统通常要求R T F 1 RTF 1RTF1P50 / P95 latency平均体验与尾延迟Model call ratio真正调用重模型的帧或 chunk 比例Visual tokens/s视觉侧处理吞吐Peak GPU memory峰值显存Streaming with ThinkingStreamingCoTCodeStreaming VideoQA 与多模态 CoT 数据Video Streaming ThinkingCode边看边思考VST-SFT VST-RLThink While WatchingCode持续 segment memory 与并行推理Thinking in Streaming VideoCodeWatch–Think–Speak 与 reasoning-compressed memoryThink-as-You-SeeCode因果 streaming attention 与双 KV Cache四、BenchmarkStreaming QA、记忆与推理StreamingBenchCode Data实时、全局与上下文理解OVBenchCode Data过去记忆、当前感知与未来预测OVO-BenchCode DataBackward Tracing、实时感知和主动响应StreamBenchCode长短期记忆与多轮交互SVBenchCode时序多轮视频对话RTV-BenchCode持续感知、理解和推理OST-BenchCode在线时空场景理解ODV-BenchDataset自动驾驶在线理解RIVERCode实时交互、记忆、感知与预判主动响应与响应时机OmniMMICode Data流式理解、告警、轮次切换与动作规划ViSpeakCode Data视觉唤醒、打断与反馈PROASSISTCode Data第一视角任务指导与响应时机ProactiveVideoQACode DataWeb、Ego、剧集与异常场景ESTP-Bench / Eyes Wide OpenCodeJust-in-Time 响应与主动请求高清帧Proact-VL Live Gaming Benchmark实时游戏解说、协同解说和用户指导视频推理综合评测Video-MMECode短、中、长视频综合理解MVBenchCode多任务时空理解VideoVistaCode视频理解与推理VideoReasonBenchCode视觉中心复杂推理Video-HolmesCode复杂长视频推理VCRBenchCode长视频因果推理五、DatasetStreaming Caption 与 NarrationLive-CC-5M大规模流式 ASR 视频预训练数据Live-WhisperX-526K密集实时解说与指令微调OmniStar-RNG实时 narration、dense caption 和视频文本对齐MMDuetIT多答案 grounded QA、主动响应与时间定位Streaming QA 与交互OVBenchOVO-BenchSVBenchStreamBenchODV-BenchTemporalBench数据设计时需要显式保存video/chunk、时间戳、历史对话、事件区间、响应内容以及silence/respond标签避免训练阶段意外读取未来信息。六、ProjectNVIDIA Live VLM WebUIDocs摄像头实时 VLM WebUIVideoLLM-onlineStreaming EOS 基础实现LiveCC实时视频 commentaryStreamBridge将离线 Video-LLM 改造成主动 Streaming AssistantStreamingVLM无限流、固定显存与 KV Cache 管理SimpleStream固定滑动窗口 baseline适合首先复现StreamMind高帧率事件门控方案Proact-VL实时 AI companion 与游戏场景OneVision-Encoder ModelsCodec-aligned Vision Encoder七、SurveyTowards Online Interactors: A Comprehensive Survey on Streaming Video Understanding围绕主动触发、长期记忆、稀疏计算和评测整理 Streaming Video UnderstandingA Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming视频生成、理解与流式处理综述The Landscape of Video Reasoning: Tasks, Paradigms and BenchmarksCoT、CoF、Interleaved 与 Streaming Video Reasoning八、BlogVision TransformerThe Illustrated Vision TransformerAn Image is Worth 16x16 WordsHugging Face Vision Transformer ExplainedVideo 与 CodecFFmpeg DocumentationA Beginner’s Guide for FFMPEGVideo Coding Concepts帧率、I/P/B 帧、压缩与码率推理与 KV CachevLLM: Easy, Fast, and Cheap LLM ServingPagedAttention PaperHugging Face KV Cache StrategiesFlashAttention: Fast and Memory-Efficient Exact Attention九、BookDive into Deep Learning深度学习、Transformer 与计算机视觉入门Deep Learning for Vision Systems视觉任务工程实践Computer Vision: Algorithms and Applications经典计算机视觉理论Natural Language Processing with TransformersTransformers 与 Hugging Face 实践Designing Machine Learning Systems数据、训练、部署和监控十、路线Stage 1VLM 基础学习 PyTorch Tensor、Module、Dataset 和 GPU 推理理解 ViT、Patch Embedding、Attention 和 RoPE阅读 CLIP、BLIP-2、LLaVA、Qwen2-VL跑通单图 caption 与 VQAStage 2Video-LLM使用 FFmpeg / Decord 解码视频并均匀抽帧理解[T,C,H,W] → visual tokens → LLM数据流跑通 Video-MME 或 MVBench 的小规模评测比较帧数、分辨率、视觉 token 数与显存、延迟的关系Stage 3Streaming Baseline将视频切成严格因果的 chunk禁止使用未来帧实现固定长度滑动窗口在每个 chunk 输出 caption、answer 或 silence记录 TTFT、P95、RTF、显存和 model call ratioStage 4主动触发与长期记忆对比固定周期、帧差、codec residual 和轻量分类 Head加入silence/response或独立 response head实现短期滑窗 长期文本/事件/KV 记忆在 OVO-Bench、SVBench、RTV-Bench 上做消融Stage 5高帧率与 VLA使用 lightweight gate 持续感知重模型按事件调用将输出从 caption 扩展为离散 action token测量 action latency、decision FPS、return 和 model call ratio研究 codec motion vector / residual 引导的区域级 patch selectionSimpleStreambaseline → VideoLLM-onlineEOS / silence → StreamBridge独立触发器 → StreamMind高帧率稀疏调用 → StreamingVLM / HERMESKV Cache 与长期记忆 → CodecSight / OneVision-EncoderCodec-aware 推理Resourcessotayang/Awesome-Streaming-Video-Understandingydyhello/Awesome-VLM-Streaming-VideoLJungang/Awesome-Video-Reasoning-LandscapeVideo-Reason/Awesome-Video-Reasoningyunlong10/Awesome-LLMs-for-Video-Understandingyunlong10/Awesome-Video-LMM-Post-TrainingzhengxuJosh/Awesome-Multimodal-Spatial-Reasoning资料更新较快论文发表状态、代码和模型权重以原项目页面为准。欢迎提交 PR 补充课程、论文复现、实验记录和中文笔记。