LLMs之Benchmark之MMSU《MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark》翻译与解读导读MMSU 提出了一套以语言学理论为支撑、以真实音频为基础的47 任务、多维度感知推理评测框架暴露了当前 SpeechLLMs 在声学感知与跨层次推理上的显著短板并给出了通过更真实数据、细粒度监督与任务驱动设计来改进模型的明确方向该基准对推动更“理解式”的语音智能具有重要参考价值。 背景痛点● 覆盖缺口现有评测过度侧重语义层面忽视口语中常见而关键的现象如口吃/自纠、讽刺、非言语声、重音/停顿/延长、误读/双关、混合语码等。这些现象携带重要的语用与声学线索直接影响机器理解“说什么”和“怎么说”的能力。● 音频真实性不足许多基准大量依赖 TTS 合成音频无法反映真实人声的声学变异导致评测对现实场景泛化能力不足。痛点说明合成语音缺乏自然的韵律、非语言噪声和口语变体低估了模型在真实语音下的感知难度。● 评估缺乏语言学基础现有任务设计很少基于系统的语言学理论导致评测在语音—语义—语用多层次的联结上存在盲点。痛点说明没有把音系、韵律、修辞、句法、语义与副语言paralinguistics系统性地纳入任务设计。 具体的解决方案● 基准构建MMSU — 大规模多任务口语理解与推理基准。方案概要构建5,000 条专家标注的多项选择题覆盖 47 个任务24 个感知任务 23 个推理任务并按语言学分层语音学/韵律/修辞/句法/语义/副语言组织。● 数据来源与真实性保证方案举措以开放真实数据为主针对缺失的语音现象聘请专业配音演员录制定向样本并用少量真实说话者录音 有选择地用多声道 TTS 做补充以保证声学多样性与真实性。●专家与人机混合标注方案举措设计专家参与的问卷/题目方案并采用“专家-in-the-loop”的增强利用 LLM 生成候选干扰项再由人工复核最后多轮人工质检与专家复审确保题目质量与标注一致性。 核心思路与步骤流程化说明● 总体设计思路三层次语言学架构→ 感知 vs 推理两大维度 → 细分为语言学语义/音系与副语言说话人特质/说话风格子域。步骤小结先理论驱动任务设计再数据收集/合成最后严格人工复核。●四阶段数据构建流程●● 阶段一—语言学框架与任务设计与语言学专家协作确定覆盖现象与任务模版基于文献与理论。●● 阶段二—题目收集与选项增强从权威教材/在线资源采题使用 LLM作者使用 GPT-4o生成候选干扰项再由专家筛选与调整。●● 阶段三—音频收集与定制录制优先开放数据集素材针对音系类缺失项组织专业配音录制补充少量真实说话者录音并适度用 TTS 增强。●● 阶段四—人工复核与质量把控10 名训练过的标注者多轮标注过滤最终由领域专家审定并标注任务类型/子域标签。 优势● 广度与深度优势覆盖 47 个任务远超多数同类基准既有感知层面也有推理层面能评估从声学线索到复杂语用推理的端到端能力。优势说明可以暴露模型在不同语言学层面的薄弱点便于针对性改进。● 真实性与多样性优势以真实人声 专业录制为主包含非语音声音、口音、情感、韵律多样性能更好测试模型在自然语音中的稳定性。● 语言学理论支撑任务设计系统依托语音学、韵律学、修辞学、句法与语义学等理论提升了基准的解释性与诊断能力。优势说明有助于将评测输出转化为可理解的模型改进方向。 论文的一些结论与观点侧重经验与建议● 当前模型表现与人类差距明显实验结果显示在人类均值 ~89.7% 的参照下最佳模型Gemini-1.5-Pro仅约 60.68%说明现有 SpeechLLMs 在综合口语理解与推理上仍有大量提升空间。建议需要在声学感知与跨层次推理能力上同时增强训练与架构。●感知错误是主要失误来源论文的误差分析表明“感知错误Perceptual Errors”是各模型的主因其次是推理错误与知识盲区。建议应在模型训练中增加对细粒度声学特征如重音、延长、非言语声的显式监督或对比学习目标。●不同模型优势互补实验显示不同模型在不同任务上表现差异明显如某些模型擅长性别预测、某些在双关/押韵/韵律判读上更好。建议未来可探索模型集成、模块化专门声学感知模块 通用推理模块或多任务联合训练策略。●噪声鲁棒性观察在加入不同强度高斯噪声后整体性能下降有限且部分模型如 Gemini-1.5-Pro 与 Qwen2.5-Omni 在噪声下保持相对稳定显示模型确实在利用音频信号而非纯文本偏差做判断。建议继续在更复杂噪声与环境下测试并在训练中引入更丰富噪声增强以提高泛化。●数据与领域不足影响专业任务对口音、特殊发音或领域知识相关任务模型表现受限表明需要更多针对性训练数据或领域自适应策略。建议收集覆盖性更强的口音与语料或采用少量标注实现领域微调。目录《MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark》翻译与解读Abstract1、IntroductionFigure 1:Overview of the MMSU dataset: MMSU incorporates fine-grained acoustic features, quality assurance through linguistic experts-guided data creation, and tasks across 47 distinct perception and reasoning skills for comprehensive spoken language understanding.图 1MMSU 数据集概述MMSU 融合了精细的声学特征通过语言专家指导的数据创建来确保质量并涵盖 47 种不同的感知和推理技能任务以实现全面的口语理解。6 Conclusion《MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark》翻译与解读地址论文地址https://arxiv.org/abs/2506.04779时间2026年01月15日作者香港中文大学AbstractSpeech inherently contains rich acoustic information that extends far beyond the textual language. In real-world spoken communication, effective interpretation often requires integrating semantic meaning (e.g., content), paralinguistic features (e.g., emotions, speed, pitch) and phonological characteristics (e.g., prosody, intonation, rhythm), which are embedded in speech. While recent multimodal Speech Large Language Models (SpeechLLMs) have demonstrated remarkable capabilities in processing audio, their ability to perform fine-grained perception and complex reasoning in natural speech remains largely unexplored. To address this gap, we introduce MMSU, a comprehensive benchmark designed specifically for understanding and reasoning in speech. MMSU comprises 5,000 meticulously curated audio-question-answer triplets across 47 distinct tasks. Notably, linguistic theory forms the foundation of speech language understanding (SLU), yet existing benchmarks have paid insufficient attention to this fundamental aspect and fail to capture the broader linguistic picture. To ground our benchmark in linguistic principles, we systematically incorporate a wide range of linguistic phenomena, including phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics. Through a rigorous evaluation of 22 advanced SpeechLLMs, we identify substantial room for improvement in existing models. MMSU establishes a new standard for comprehensive assessment of SLLU, providing valuable insights for developing more sophisticated human-AI speech interaction systems. MMSU benchmark is available at https://huggingface.co/datasets/ddwang2000/MMSU.语音本身蕴含着丰富的声学信息这些信息远远超出了文本语言的范畴。在现实世界的口语交流中有效的解读往往需要整合语义例如内容、副语言特征例如情感、语速、音高以及音韵特征例如韵律、语调、节奏这些都包含在语音之中。尽管近期的多模态语音大语言模型SpeechLLMs在处理音频方面展现出了卓越的能力但它们在自然语音中的细粒度感知和复杂推理方面的能力仍有待深入探索。为了解决这一空白我们推出了 MMSU这是一个专门用于语音理解和推理的全面基准。MMSU 包含了 5000 个精心策划的音频-问题-答案三元组涵盖了 47 个不同的任务。值得注意的是语言理论构成了语音语言理解SLU的基础然而现有的基准对这一基本方面关注不足未能全面捕捉语言的全貌。为了使我们的基准测试建立在语言学原理之上我们系统地纳入了广泛的语言现象包括语音学、韵律学、修辞学、句法学、语义学和副语言学。通过对 22 个先进的语音语言模型进行严格评估我们发现现有模型存在很大的改进空间。MMSU 建立了对语音语言理解单元进行综合评估的新标准为开发更复杂的语音人机交互系统提供了宝贵的见解。MMSU 基准测试可在 https://huggingface.co/datasets/ddwang2000/MMSU 获取。1、IntroductionRecent advancements in Speech Large Language Models (SpeechLLMs) (Ji et al., 2024; Arora et al., 2025; Chu et al., 2024; Zhang et al., 2023; Ghosh et al., 2025) have attracted significant attention in the field of multimodal large models (Yin et al., 2024; Caffagni et al., 2024; Fu et al., 2024; Chen et al., 2025). SpeechLLMs are designed to process and understand audio inputs, enabling them to handle a wide range of audio-related tasks. Yet, how well these models can perceive nuanced speech signals in real-world communication still remains largely unexplored. Unlike text, spoken language is distinguished by unique acoustic features that allow speakers to convey intentions beyond surface-level literal information through elements such as prosody, intonation, and emotion. In other words, to facilitate effective human-computer interactions, we need to fully understand not only what the speaker said, but also how they said it and what they truly meant.However, achieving holistic spoken language understanding (SLU) is challenging, as existing benchmarks fail to capture the full spectrum of SLU, particularly in authentic scenarios. We identify three key limitations of current evaluation systems: (i) Lack coverage of critical spoken phenomena in daily life. Existing benchmarks for SpeechLLMs predominantly focus on semantic-level tasks (Chen et al., 2024; Gao et al., 2024; Si et al., 2024; Yang et al., 2024; Wang et al., 2024a), while many common phenomena in daily speech have been largely overlooked. Examples include spontaneous disfluencies, sarcasm, self-corrections, non-verbal sounds, prosody variations (e.g., stress, pause, intonation, prolonged sound), mispronunciations, pun interpretation, and code-switching. (ii) Limited expressive and diverse authentic audio resources. Most current benchmarks heavily rely on TTS-synthesized audio (Gao et al., 2024; Chen et al., 2024; Ao et al., 2025; Cheng et al., 2025), which fails to capture the nuanced acoustic variability inherent in human speech, limiting their ability to evaluate models under realistic communicative conditions. (iii) Absence of linguistic principles in evaluation design. Linguistics provides the theoretical foundation for understanding how humans produce, perceive, and interpret spoken language (Chomsky and Halle, 1991; Partee et al., 1990; Lyons, 1968). A true SLU system should not merely rely on extracting surface-level semantics, but involves decoding and deep reasoning over multiple linguistic layers from phonological cues, prosodic patterns, and rhetorical structures. However, existing benchmarks neglect linguistic principles in their evaluation, leading to potentially biased assessments and critical blind spots. This gap hampers progress in developing SpeechLLMs capable of capturing speech’s full complexity.近期语音大语言模型SpeechLLMsJi 等人2024 年Arora 等人2025 年Chu 等人2024 年Zhang 等人2023 年Ghosh 等人2025 年的进展在多模态大模型领域Yin 等人2024 年Caffagni 等人2024 年Fu 等人2024 年Chen 等人2025 年引起了广泛关注。语音大语言模型旨在处理和理解音频输入从而能够应对各种与音频相关的任务。然而这些模型在真实交流中对细微语音信号的感知能力如何仍是一个未被充分探索的领域。与文本不同口语通过诸如语调、声调和情感等元素能够传达出表面文字信息之外的意图。换句话说为了实现有效的人机交互我们不仅要理解“说话者说了什么”还要理解“他们是怎么说的”以及“他们真正的意思是什么”。然而实现全面的口语理解SLU颇具挑战性因为现有的基准测试未能涵盖 SLU 的全部范围尤其是在真实场景中。我们发现当前评估系统存在三个关键局限性一对日常生活中关键的口语现象缺乏覆盖。现有的针对语音语言模型的基准测试主要集中在语义层面的任务上Chen 等人2024 年Gao 等人2024 年Si 等人2024 年Yang 等人2024 年Wang 等人2024a而日常口语中许多常见的现象却被很大程度上忽略了。这些现象包括自发的不流畅、讽刺、自我纠正、非言语声音、语调变化如重音、停顿、语调、延长音、发音错误、双关语解读以及语言转换。二缺乏丰富多样的真实音频资源。目前大多数基准测试严重依赖于 TTS 合成的音频Gao 等人2024 年Chen 等人2024 年Ao 等人2025 年Cheng 等人2025 年这无法捕捉到人类言语中固有的细微声学变化从而限制了其在真实交流条件下评估模型的能力。三评估设计中缺乏语言学原则。语言学为理解人类如何生成、感知和解读口语提供了理论基础乔姆斯基和哈勒1991 年帕蒂等人1990 年莱昂斯1968 年。一个真正的口语理解SLU系统不应仅仅依赖于提取表面语义还应涉及对从语音线索、韵律模式到修辞结构等多层语言信息的解码和深度推理。然而现有的基准测试在评估时忽略了语言学原理这可能导致评估结果存在偏差和关键盲点。这种差距阻碍了能够捕捉口语全部复杂性的语音大语言模型SpeechLLM的发展。To address these gaps, we propose MMSU (Massive Multi-task Spoken Language Understanding and Reasoning Benchmark), a comprehensive evaluation framework designed to assess SLU across diverse dimensions. As illustrated in Fig. 1, MMSU is distinguished by three primary features: (1) Fine-grained acoustic features. MMSU captures the most comprehensive range of acoustic information, including diverse non-verbal sounds (e.g., crying, snoring, coughing), English accents (e.g., Indian, British), different emotional states, a variety of prosodic features (e.g., stress, prolonged sounds, pauses), and intonation variations, among others. (2) High-quality data assurance. In contrast to many existing benchmarks that heavily rely on synthetic speech, MMSU is primarily based on real-world data sourced from open-source datasets and professional studio recordings, ensuring acoustic authenticity. Moreover, each task and question undergoes meticulous review by experts to guarantee accuracy and representativeness in evaluation. (3) Pioneering the integration of linguistic principles and comprehensive task coverage. To our knowledge, MMSU is the first benchmark that systematically incorporates linguistic theory into task design. It introduces 47 novel tasks, each targeting different challenges in spoken communication. The benchmark spans multiple linguistic subfields, including phonetics (Ladd, 2008), prosody (Pierre, 1980), rhetoric (Ladd, 2008), syntactics (Carnie, 2007), semantics (Lyons, 1995) and paralinguistics (Trager, 1961). These tasks — such as pun interpretation, disfluency detection, code-switching QA, intonation-based reasoning, and homophone-based reasoning — are unique to MMSU.To validate MMSU’s effectiveness as a benchmark, we conduct an in-depth evaluation and analysis across 22 SpeechLLMs revealing critical insights, such as widespread challenges in phonological perception, difficulty in handling complex reasoning, as well as specific subtask deficiencies. These findings provide valuable guidance for future advancements in SpeechLLMs and help identify areas for targeted improvement.为解决这些不足我们提出了 MMSU大规模多任务语音语言理解和推理基准这是一个全面的评估框架旨在从多个维度评估语音语言理解SLU。如图 1 所示MMSU 具有三个主要特点1精细的声学特征。MMSU 捕获了最全面的声学信息包括各种非言语声音如哭声、鼾声、咳嗽声、英语口音如印度口音、英式口音、不同的情绪状态、多种韵律特征如重音、延长音、停顿以及语调变化等。2高质量的数据保障。与许多依赖合成语音的现有基准不同MMSU 主要基于来自开源数据集和专业录音室的真实世界数据确保声学的真实性。此外每个任务和问题都经过专家的仔细审查以确保评估的准确性和代表性。3开创性地融合语言学原理和全面的任务覆盖。据我们所知MMSU 是首个将语言学理论系统地融入任务设计的基准。它引入了 47 项新颖的任务每项任务都针对口语交流中的不同挑战。该基准涵盖了多个语言学分支领域包括语音学Ladd2008、韵律学Pierre1980、修辞学Ladd2008、句法学Carnie2007、语义学Lyons1995和副语言学Trager1961。这些任务——例如双关语解读、不流畅检测、代码切换问答、基于语调的推理以及基于同音词的推理——都是 MMSU 所独有的。为了验证 MMSU 作为基准的有效性我们在 22 个 SpeechLLM 上进行了深入的评估和分析揭示了关键的见解比如在语音感知方面的普遍挑战、处理复杂推理的困难以及特定子任务的缺陷。这些发现为 SpeechLLM 的未来改进提供了宝贵的指导并有助于确定有针对性的改进领域。Figure 1:Overview of the MMSU dataset: MMSU incorporates fine-grained acoustic features, quality assurance through linguistic experts-guided data creation, and tasks across 47 distinct perception and reasoning skills for comprehensive spoken language understanding.图 1MMSU 数据集概述MMSU 融合了精细的声学特征通过语言专家指导的数据创建来确保质量并涵盖 47 种不同的感知和推理技能任务以实现全面的口语理解。6 ConclusionIn this paper, we introduce MMSU, a comprehensive multi-task benchmark designed to address the complexities of spoken language understanding and reasoning. MMSU encompasses 47 distinct tasks with 5,000 meticulously curated audio samples, covering a broad spectrum of acoustic features. Notably, MMSU is the first benchmark to systematically integrate established linguistic theories across a wide range of subfields, including phonetics, prosody, rhetoric, syntax, semantics, and paralinguistics. MMSU aims to provide a systematic approach to evaluate the capabilities of SpeechLLMs in understanding and reasoning across multiple facets of spoken language in practical contexts. Our evaluation of 22 widely-used open-source and proprietary models reveals that, even for the best-performing model, accuracy achieves only 60.68%. This underscores the considerable challenges that persist in achieving robust and generalized spoken language understanding, which is essential for truly effective human-computer interactions. To facilitate ongoing research and model comparison, we plan to launch and maintain a leaderboard that will serve as a consistent platform for the community to access and compare model performance.在本文中我们介绍了 MMSU这是一个全面的多任务基准旨在解决口语理解与推理的复杂性。MMSU 包含 47 个不同的任务拥有 5000 个精心挑选的音频样本涵盖了广泛的声学特征。值得注意的是MMSU 是首个系统性地整合了包括语音学、韵律学、修辞学、句法学、语义学和副语言学等多个子领域中成熟语言理论的基准。MMSU 旨在为评估 SpeechLLMs 在实际情境中对口语理解与推理的多方面能力提供系统性方法。我们对 22 个广泛使用的开源和专有模型的评估表明即使对于表现最佳的模型准确率也仅达到 60.68%。这凸显了在实现稳健且通用的口语理解方面仍存在巨大挑战而这对于真正有效的人机交互至关重要。为了便于持续的研究和模型比较我们计划推出并维护一个排行榜该排行榜将作为一个统一的平台供社区成员访问并比较模型的性能。