尧图网页设计 
Google Cloud 音频生成实战指南Lyria 3 音乐创作与 Chirp 3 / Gemini-TTS 语音合成【免费下载链接】generative-aiSample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform项目地址: https://gitcode.com/GitHub_Trending/ge/generative-ai本文以开源仓库generative-ai中的 audio/README.md 为主线系统梳理 Google Cloud 音频生成能力的完整技术栈从 Lyria 3 音乐生成文本/图像提示、自定义歌词、多语言流式生成到 Chirp 3 Instant Custom Voice 即时音色克隆、Chirp 3 HD 高清音色的实时与流式合成再到 Gemini-TTS 的提示词驱动语音合成与多说话人对话。读完本文你将掌握两套 APICloud Text-to-Speech API 与 Agent Platform API的选择标准与调用方式并能直接复用仓库中的代码完成从文本到音乐、从录音到自定义音色的端到端实现。仓库中的audio目录按能力分为music音乐生成与speech语音合成与处理两大模块所有教程均以可执行的 Jupyter Notebook 形式提供音乐生成lyria3_music_generation.ipynb、lyria2_music_generation.ipynb语音合成get_started_with_chirp3_instant_custom_voice.ipynb、get_started_with_chirp_3_hd_voices.ipynb、get_started_with_gemini_tts_voices.ipynb语音识别与更多用例参见 audio/speech/README.md一、环境准备启用 API 与初始化客户端四个教程共享同一套环境准备流程主要包括以下步骤。1.1 安装 SDK使用 Agent Platform API 的教程Lyria 3、Gemini-TTS 的 Agent Platform 部分需要安装google-genai%pip install --upgrade --quiet google-genai使用 Cloud Text-to-Speech API 的教程Chirp 3 HD、Gemini-TTS 的 Cloud TTS 部分、Chirp 3 Instant Custom Voice需要安装google-cloud-texttospeech其中 Gemini-TTS 相关字段要求google-cloud-texttospeech 版本不低于 2.31.0%pip install --upgrade --quiet google-cloud-texttospeechChirp 3 教程还依赖ffmpeg用于音频处理notebook 中给出了跨平台安装脚本Linux 通过apt install ffmpegmacOS 通过 Homebrew 的brew install ffmpeg。Instant Custom Voice 教程额外依赖gradio%pip install --upgrade --quiet gradio用于构建演示应用。1.2 认证与项目配置在 Colab 环境中先执行用户认证再设置项目信息。各 notebook 均支持通过环境变量GOOGLE_CLOUD_PROJECT回退读取项目 IDimport sys if google.colab in sys.modules: from google.colab import auth auth.authenticate_user()import os # fmt: off PROJECT_ID [your-project-id] # param {type: string, placeholder: [your-project-id], isTemplate: true} # fmt: on if not PROJECT_ID or PROJECT_ID [your-project-id]: PROJECT_ID str(os.environ.get(GOOGLE_CLOUD_PROJECT)) LOCATION os.environ.get(GOOGLE_CLOUD_REGION, global)随后通过 gcloud 设置项目与 ADCApplication Default Credentials配额项目! gcloud config set project {PROJECT_ID} ! gcloud auth application-default set-quota-project {PROJECT_ID} ! gcloud auth application-default login -q两个 API 的客户端初始化方式不同这一点贯穿全文务必区分Agent Platform 客户端Lyria 3、Gemini-TTSfrom google import genai client genai.Client(enterpriseTrue, projectPROJECT_ID, locationLOCATION)Cloud Text-to-Speech 客户端Chirp 3 HD、Gemini-TTSfrom google.api_core.client_options import ClientOptions from google.cloud import texttospeech_v1beta1 as texttospeech API_ENDPOINT ( f{TTS_LOCATION}-texttospeech.googleapis.com if TTS_LOCATION ! global else texttospeech.googleapis.com ) client texttospeech.TextToSpeechClient( client_optionsClientOptions(api_endpointAPI_ENDPOINT) )端点构造逻辑是TTS_LOCATION为global时直接使用texttospeech.googleapis.com否则使用{区域}-texttospeech.googleapis.com形式如us-central1-texttospeech.googleapis.com。具体可用区域需查阅各功能对应的官方文档Chirp 3 HD 见 endpoints 文档Instant Custom Voice 见 regional availability 文档。二、Lyria 3 音乐生成从文本到完整歌曲lyria3_music_generation.ipynb 演示了如何通过 Google Gen AI SDK for Python 与 Agent Platform 上的 Lyria 3 交互生成高质量的音乐片段、完整歌曲与音乐流。教程核心能力包括从文本提示生成带人声的完整歌曲、从图像提示生成音乐片段、自定义歌词、以及通过 Interactions API 实现多语言音乐流式生成。2.1 模型选择clip 与 pro 两个档位notebook 中加载了预览版模型标识二者对应不同的生成时长与用途music_clip_model lyria-3-clip-preview music_model lyria-3-pro-previewlyria-3-clip-preview生成约 30 秒的音乐片段cliplyria-3-pro-preview生成最长 3 分钟的完整歌曲track。从模型命名可以看出两者均为 preview预览阶段模型使用时以仓库中记录的模型名称为准。2.2 定义音频展示辅助函数由于响应中同时包含文本歌词、歌曲结构说明与内联音频字节notebook 定义了display_output辅助函数统一解析response.parts文本部分通过正则去除[[...]]节标记并规整时间戳格式后以 Markdown 渲染inline_data部分直接以 IPython 的Audio组件播放def display_output(response): for part in response.parts: if part.text: text part.text # Remove section tags, add new lines text re.sub(r\[\[.*?\]\], , text) text re.sub( r\s*\[(\d\.\d)(?::?(\d\.\d))?:?\]:?, r\n\n[\1\2] , text ) text text.replace([:], \n\n) clean_text text.strip() display(Markdown(clean_text)) if part.inline_data: display(Audio(datapart.inline_data.data, autoplayFalse))关键点要同时拿到文本与音频输出必须在请求中把response_modalities设置为[AUDIO, TEXT]。2.3 从文本提示生成完整歌曲编写提示词时notebook 建议从三个维度展开描述风格/流派Style/Genre如 classical、electronic、rock、jazz、hip hop、pop也可以使用更泛化的描述如 cinematic、ambient、lo-fi人声Vocals如需人声描述音色与音域如音域、音调等特征乐器Instruments指定钢琴、合成器、原声吉他、鼓、弦乐、长笛等具体乐器。示例提示词与调用prompt Sophisticated, rhythmic, and aspirational track with crisp 808 percussion, digital plucks, and muted electric guitar rhythmic strums. Include breathy, airy Alto female vocal textures with melodic, minimalist oohs and aahs with heavy reverb and rhythmic delay. response client.models.generate_content( modelmusic_model, contentsprompt, configtypes.GenerateContentConfig(response_modalities[AUDIO, TEXT]), ) display_output(response)默认情况下Lyria 3 生成的所有音乐都会嵌入 SynthIDAI 生成内容水印与 C2PA 内容凭证用于标识内容的 AI 生成属性。2.4 从图像提示生成音乐片段Lyria 3 支持以图像作为参考输入生成音乐。notebook 首先下载示例图片!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/flowers.png然后以types.Part.from_bytes构造图像内容配合文本提示调用lyria-3-clip-preview模型。单次请求最多可携带 10 张图像with open(input_image, rb) as f: image f.read() response client.models.generate_content( modelmusic_clip_model, contents[ types.Part.from_bytes( dataimage, mime_typeimage/png, ), Generate an instrumental clip based on this input image that starts slowly and builds in intensity., ], configtypes.GenerateContentConfig( response_modalities[TEXT, AUDIO], ), ) display_output(response)2.5 自定义歌词在图像之外还可以在提示中直接给定歌词文本并附加速度tempo、流派、风格、人声配置、乐器等补充指令。notebook 的示例把风格描述 完整歌词组织成一个结构化文本块传给模型genre_lyrics Genre: Upbeat, acoustic Folk-Pop with a warm and cuddly vibe. Bright acoustic guitars, a soft shaker rhythm, and a friendly, melodic vocal. Lyrics: Tail wags and a heavy head, Time to curl up in your favorite bed. Soft as a cloud, a dream come true, The perfect spot for a dog like you. response client.models.generate_content( modelmusic_clip_model, contents[ types.Part.from_bytes(dataimage, mime_typeimage/png), genre_lyrics, ], configtypes.GenerateContentConfig(response_modalities[TEXT, AUDIO]), )2.6 Interactions API多语言生成与流式输出Interactions API 是 Agent Platform 提供的统一模型/Agent 交互接口。Lyria 3 支持英语、德语、西班牙语、法语、印地语、日语、韩语、葡萄牙语八种语言既可以直接用这些语言写提示也可以在使用其他语言时显式要求目标语言。非流式调用示例以西班牙语提示为例interaction client.interactions.create( modelmusic_model, inputGenera un tema pop, ) display_interaction_output(interaction)Interactions API 的响应结构与generate_content不同输出位于interaction.outputs列表中每项是带type字段的字典text或audio音频数据为 base64 编码字符串需解码后播放def display_interaction_output(interaction): outputs getattr(interaction, outputs, []) or [] for output in outputs: is_dict isinstance(output, dict) output_type output.get(type) if is_dict else getattr(output, type, None) if output_type text: # ... 文本清洗与 Markdown 渲染 ... elif output_type audio: data output.get(data) if is_dict else getattr(output, data, None) audio_data base64.b64decode(data) display(Audio(dataaudio_data, autoplayFalse))流式生成在interactions.create中设置streamTrue歌词与描述会随模型输出即时返回无需等待整个请求完成。遍历流式事件时通过event.event_type content.delta判断增量内容event.delta中的text字段是文本增量data配合mime_type中的audio判断音频块stream client.interactions.create( modelmusic_model, inputGenerate a song about spending a day in Seoul in Korean., streamTrue, ) for event in stream: if event.event_type content.delta: delta_dict event.delta if isinstance(event.delta, dict) else getattr(event, delta, {}) if text in delta_dict: text delta_dict[text] text text.replace([:], \n[:]) text re.sub(r(\[\d\.\d:\]), r\n\1, text) display(Markdown(text)) elif data in delta_dict and audio in delta_dict.get(mime_type, ): display(Audio(database64.b64decode(delta_dict[data]), autoplayFalse))三、Chirp 3 Instant Custom Voice10 秒录音创建专属音色get_started_with_chirp3_instant_custom_voice.ipynb 介绍 Google Cloud Text-to-Speech API 的Chirp 3 Instant Custom Voice功能用你自己的高质量录音训练出个性化语音模型随后即可用该音色合成音频支持流式与长文本输出且音色创建与合成支持25 种以上语言。3.1 访问限制Allowlist⚠️由于安全考量该音色克隆能力仅对白名单allow-listed用户开放。如需使用必须联系 Google Cloud 团队成员加入白名单。这一点在 notebook 顶部以醒目的警告框标出是使用该功能的前置条件。3.2 获取凭证与构造端点与 Agent Platform 客户端不同Instant Custom Voice 教程直接使用 REST API 调用因此需要通过google.auth获取 OAuth 访问令牌import google.auth import google.auth.transport.requests credentials, _ google.auth.default() authentication google.auth.transport.requests.Request() credentials.refresh(authentication)API_ENDPOINT ( f{TTS_LOCATION}-texttospeech.googleapis.com if TTS_LOCATION ! global else texttospeech.googleapis.com ) ACCESS_TOKEN credentials.token3.3 创建音色克隆密钥Voice Cloning Key核心 API 是POST /v1beta1/voices:generateVoiceCloningKey。请求体包含三部分关键输入reference_audio参考音频预先录制的目标音色样本.wav格式最佳理想情况是几秒钟的清晰语音voice_talent_consent声纹所有人授权音频说话人明确朗读同意脚本的录音consent_script同意脚本文本即授权语句本身。音频统一采用LINEAR16编码、24000Hz 采样率并以 base64 形式放在content字段。notebook 中的实现如下def create_instant_custom_voice_key( reference_audio_bytes: bytes, consent_audio_bytes: bytes ) - str: Creates a temporary custom voice key url fhttps://{API_ENDPOINT}/v1beta1/voices:generateVoiceCloningKey request_body { reference_audio: { audio_config: {audio_encoding: LINEAR16, sample_rate_hertz: 24000}, content: reference_audio_bytes, }, voice_talent_consent: { audio_config: {audio_encoding: LINEAR16, sample_rate_hertz: 24000}, content: consent_audio_bytes, }, consent_script: I am the owner of this voice and I consent to Google using this voice to create a synthetic voice model., language_code: en-US, } headers { Authorization: fBearer {ACCESS_TOKEN}, x-goog-user-project: PROJECT_ID, Content-Type: application/json; charsetutf-8, } response requests.post(url, headersheaders, jsonrequest_body) response.raise_for_status() response_json response.json() return response_json.get(voiceCloningKey)请求成功且项目在白名单内时API 返回voiceCloningKey它是自定义音色的临时标识。本地 WAV 文件通过wav_to_base64辅助函数读取并编码为 base64 字符串即 JSON payload 中音频数据的传输格式。3.4 使用自定义音色合成语音合成接口为POST /v1beta1/text:synthesize。关键差异在于voice字段携带voice_clone.voice_cloning_keydef synthesize_text_with_cloned_voice(voice_key: str, text: str) - None: url fhttps://{API_ENDPOINT}/v1beta1/text:synthesize request_body { input: {text: text}, voice: { language_code: en-US, voice_clone: { voice_cloning_key: voice_key, }, }, audioConfig: {audioEncoding: LINEAR16, sample_rate_hertz: 24000}, } headers { Authorization: fBearer {ACCESS_TOKEN}, x-goog-user-project: PROJECT_ID, Content-Type: application/json; charsetutf-8, } response requests.post(url, headersheaders, jsonrequest_body) response.raise_for_status() response_json response.json() audio_content response_json.get(audioContent) if audio_content: display(Audio(base64.b64decode(audio_content), rate24000))notebook 提供的示例文本展示了其典型用法新闻播报风格且文本内容本身由克隆音色生成text_to_synthesize Breaking news! Chirp 3, Google Clouds audio model, now has Instant Custom Voice. ... synthesize_text_with_cloned_voice(voice_key, text_to_synthesize)3.5 构建 Gradio 演示应用为了交互式地上传音频 → 创建音色 → 合成语音notebook 用 Gradio 构建了一个完整 Web 应用。核心设计用gr.State()保存完整 voice key用于后续合成左侧列Reference Voice参考音频与Consent Audio授权音频两个gr.Audio(typefilepath)组件 Create Voice 按钮 Voice Key 文本框右侧列Text to Synthesize文本框 Generate Speech 按钮 生成音频输出 状态输出底部 Clear All 按钮负责重置整个界面。事件处理链路上create_voice_with_masking包装了create_voice内部依次调用wav_to_base64→create_instant_custom_voice_key并在返回时把 key 掩码显示前 5 位 ...以避免界面泄露完整密钥generate_speech则复用 3.4 节的合成逻辑把 base64 音频解码为np.int16数组并返回(24000, audio_array)元组供 Gradio 播放。全部完成后通过app.launch(shareTrue)启动、app.close()关闭。四、Chirp 3 HD 高清音色实时与流式合成get_started_with_chirp_3_hd_voices.ipynb 介绍基于大语言模型LLM驱动的Chirp 3 HD Voices相比传统 TTS它在真实感和情感表达上显著提升输出高保真、带类人语调和停顿的自然语音。当前提供8 个音色4 男 4 女、支持 31 种语言适用于语音助手、有声书、客户服务等场景。4.1 音色命名与选择音色列表为Aoede、Puck、Charon、Kore、Fenrir、Leda、Orus、Zephyr。完整音色名遵循{language_code}-Chirp3-HD-{voice}命名规范例如en-US-Chirp3-HD-Aoede。notebook 中通过参数下拉框选择音色与语言voice Aoede # 可选: Aoede, Puck, Charon, Kore, Fenrir, Leda, Orus, Zephyr language_code en-US # 可选: de-DE, en-AU, en-GB, en-IN, en-US, fr-FR, hi-IN, pt-BR, ar-XA, es-ES, fr-CA, id-ID, it-IT, ja-JP, tr-TR, vi-VN, bn-IN, gu-IN, kn-IN, ml-IN, mr-IN, ta-IN, te-IN, nl-NL, ko-KR, cmn-CN, pl-PL, ru-RU, th-TH voice_name f{language_code}-Chirp3-HD-{voice} voice texttospeech.VoiceSelectionParams( namevoice_name, language_codelanguage_code, )4.2 实时在线合成实时处理调用synthesize_speech方法输出为 MP3 格式的音频字节response client.synthesize_speech( inputtexttospeech.SynthesisInput(textprompt), voicevoice, # Select the type of audio file you want returned audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.MP3 ), ) display(Audio(response.audio_content))4.3 流式合成Chirp 3 HD 支持通过streaming_synthesize方法进行流式 TTS与处理单次请求的synthesize_speech不同它处理连续文本流并生成对应的音频流。notebook 的实现展示了双向流协议的正确用法——必须先发送一条包含streaming_config的配置请求再逐条发送文本输入请求服务端在客户端半关闭half-close即请求生成器耗尽后开始回推音频def synthesize_streaming( text_iterator: Iterator[str], voice: texttospeech.VoiceSelectionParams, ) - Iterator[bytes]: config_request texttospeech.StreamingSynthesizeRequest( streaming_configtexttospeech.StreamingSynthesizeConfig( voicevoice, ) ) def request_generator() - Iterator[texttospeech.StreamingSynthesizeRequest]: yield config_request for text in text_iterator: yield texttospeech.StreamingSynthesizeRequest( inputtexttospeech.StreamingSynthesisInput(texttext) ) streaming_responses: Iterator[texttospeech.StreamingSynthesizeResponse] ( client.streaming_synthesize(request_generator()) ) for response in streaming_responses: yield response.audio_content为了模拟文本逐步生成的场景notebook 还提供了text_generator辅助函数用正则正向先行断言按句号/感叹号/问号把长文本切分成句子迭代器再逐句送入流式接口。process_streaming_audio函数负责将各分块音频np.int16采样拼接为最终完整音频并支持display_individual_chunks参数逐块预览最终以 24000 Hz 采样率播放。五、Gemini-TTS用自然语言提示词精确控制语音get_started_with_gemini_tts_voices.ipynb 介绍Gemini-TTS——在自然度之外更进一步用文本提示词对生成音频进行细粒度控制。你可以从短片段到长篇叙事自由合成语音通过自然语言提示精确指定风格、口音、语速、语调乃至情感表达还可以创建两个说话人之间、具有相同情感表达与可控性的对话。当前提供30 个音色、80 语言区域locale选项。5.1 双 API 选择决策Gemini-TTS 同时通过两条 API 开放便于不同场景的客户端集成选择 Cloud Text-to-Speech API 的情形已在用 Chirp 3 HD 或其他音色希望以最小增量继续使用同一 API需要指定特定的输出编码类型——Cloud TTS API 可以指定音频编码而 Agent Platform API 输出的是无 WAV 头的 PCM 16bit 24kHz 裸数据如需其他格式须在客户端自行转换需要双向流式——Cloud TTS API 支持多请求多响应交互而 Agent Platform API 仅支持单请求多响应。选择 Agent Platform API 的情形已从 AI Studio 使用 Gemini-TTS希望无缝迁移到 Agent Platform 以获得 Google Cloud 的扩展性与合规能力已在 Agent Platform API 上使用其他模型——统一 API 结构使其只需指定模型名与音色选项即可开始使用 Gemini-TTS。5.2 Cloud TTS API 单说话人合成模型选项包括gemini-3.1-flash-tts-preview、gemini-2.5-flash-tts、gemini-2.5-pro-tts音色选项即第 4.1 节的 8 个 HD 音色加上 Gemini 专属音色Achernar、Achird、Algenib、Algieba、Alnilam、Autonoe、Callirrhoe、Despina、Enceladus、Erinome、Gacrux、Iapetus、Laomedeia、Pulcherrima、Rasalgethi、Sadachbia、Sadaltager、Schedar、Sulafat、Umbriel、Vindemiatrix、Zubenelgenubi等共 30 个。语言代码覆盖en-us、en-gb、zh系cmn-cn、cmn-tw、hi-in、ja-jp、ko-kr等 80 区域。与 Chirp 3 HD 的关键差异在于VoiceSelectionParams需显式指定model_nameMODEL gemini-3.1-flash-tts-preview VOICE Aoede LANGUAGE_CODE en-us voice texttospeech.VoiceSelectionParams( nameVOICE, language_codeLANGUAGE_CODE, model_nameMODEL )合成调用在SynthesisInput中同时携带text与prompt两个字段——prompt 正是 Gemini-TTS提示词驱动能力的核心入口# title Capture emotion with prompts PROMPT You are having a conversation with a friend. Say the following in a happy and casual way TEXT hahaha, i did NOT expect that. can you believe it! response client.synthesize_speech( inputtexttospeech.SynthesisInput(textTEXT, promptPROMPT), voicevoice, audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.MP3 ), ) display(Audio(response.audio_content))5.3 用提示词与表现标签控制语速与情绪控制语速如PROMPT Say the following very fast but still be intelligible配合同一段法务免责声明文本即可得到快速但仍清晰的朗读表现标签Expressive Tags在文本中嵌入[chuckling]、[coughs]、[um]等口语/情绪标签配合提示词控制整体语气。notebook 明确指出这些标签并非严格语法鼓励自由实验不同表达与格式PROMPT Say the following with a sarcastic tone TEXT So.. [chuckling] tell me about this [coughs] AI thing.5.4 多说话人对话合成Gemini-TTS 支持在同一段音频中合成两个说话人的对话。核心配置对象是MultiSpeakerVoiceConfig为每个说话人分配speaker_alias自定义别名用于在文本中引用与speaker_id实际使用的预置音色。输入方式有两种方式一显式轮次语法——使用MultiSpeakerMarkup.Turn结构化描述每句的说话人与文本SPEAKER_ALIAS_1 Zizu SPEAKER_1 Fenrir SPEAKER_ALIAS_2 Gary SPEAKER_2 Orus LANGUAGE_CODE en-gb PROMPT Read the following dialogue between two friends multi_speaker_voice_config texttospeech.MultiSpeakerVoiceConfig( speaker_voice_configs[ texttospeech.MultispeakerPrebuiltVoice( speaker_aliasSPEAKER_ALIAS_1, speaker_idSPEAKER_1 ), texttospeech.MultispeakerPrebuiltVoice( speaker_aliasSPEAKER_ALIAS_2, speaker_idSPEAKER_2 ), ] ) multi_speaker_markup texttospeech.MultiSpeakerMarkup( turns[ texttospeech.MultiSpeakerMarkup.Turn( speakerSPEAKER_ALIAS_1, textHave you tried the new multi-speaker feature on Gemini?, ), texttospeech.MultiSpeakerMarkup.Turn( speakerSPEAKER_ALIAS_2, textYes! I am super excited about it ), ] ) response client.synthesize_speech( inputtexttospeech.SynthesisInput( multi_speaker_markupmulti_speaker_markup, promptPROMPT ), voicetexttospeech.VoiceSelectionParams( language_codeLANGUAGE_CODE, model_nameMODEL, multi_speaker_voice_configmulti_speaker_voice_config, ), audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.LINEAR16 ), )方式二内联对话文本——直接在text字段中用别名: 内容加换行的格式书写整段对话MultiSpeakerMarkup可省略multi_speaker_voice_config仍必须提供response client.synthesize_speech( inputtexttospeech.SynthesisInput( textZizu: Have you tried the new multi-speaker feature on Gemini?\nGary: Yes! I am super excited about it, promptPROMPT, ), voicetexttospeech.VoiceSelectionParams( language_codeLANGUAGE_CODE, model_nameMODEL, multi_speaker_voice_configmulti_speaker_voice_config, ), audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.LINEAR16 ), )5.5 放松安全过滤relax_safety_filtersGemini-TTS 默认对有害内容设有过滤。采用**月度发票计费monthly invoiced billing**的账号可通过AdvancedVoiceOptions中的relax_safety_filters字段放松过滤该字段对非此类计费账号不生效response client.synthesize_speech( inputtexttospeech.SynthesisInput(textTEXT, promptPROMPT), voicevoice, audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.MP3 ), advanced_voice_optionstexttospeech.AdvancedVoiceOptions(relax_safety_filtersTrue), )5.6 流式合成Cloud TTS APICloud TTS API 的流式模式与 Chirp 3 HD 完全一致先发streaming_config请求再发文本输入客户端请求生成器耗尽即触发 half-close随后开始接收音频流。notebook 给出了真实场景的关键建议——流式响应应在到达时立即播放/转发例如在 Web 服务器中通过 WebSocket 用emit(audio, response.audio_content)把每个音频块即时推给前端而非等全部收完。config_request texttospeech.StreamingSynthesizeRequest( streaming_configtexttospeech.StreamingSynthesizeConfig( voicetexttospeech.VoiceSelectionParams( nameVOICE, language_codeLANGUAGE_CODE, model_nameMODEL ) ) ) def request_generator(): yield config_request yield texttospeech.StreamingSynthesizeRequest( inputtexttospeech.StreamingSynthesisInput(textTEXT, promptPROMPT) ) streaming_responses client.streaming_synthesize(request_generator()) for response in streaming_responses: audio_data np.frombuffer(response.audio_content, dtypenp.int16) final_audio_data np.concatenate((final_audio_data, audio_data))该教程还演示了如何测量time to first audio首包延迟与time to completion完成总耗时并据此计算音频时长len(final_audio_data) / 24_000默认采样率 24kHz这为实时应用的延迟评估提供了直接参照。5.7 Agent Platform API 合成与 PCM 后处理Agent Platform API 使用统一的client.models.generate_content接口模型名直接写在参数中语音配置通过speech_config传入prebuilt_voice_config指定预置音色from google import genai from google.genai import types client genai.Client(enterpriseTrue, projectPROJECT_ID, locationLOCATION) response client.models.generate_content( modelgemini-2.5-flash-tts, contentsTEXT, configtypes.GenerateContentConfig( speech_configtypes.SpeechConfig( language_codeen-in, voice_configtypes.VoiceConfig( prebuilt_voice_configtypes.PrebuiltVoiceConfig( voice_nameKore, ) ), ), ), ) data response.candidates[0].content.parts[0].inline_data.data由于 Agent Platform 返回的是无 WAV 头的 PCM 16bit 24kHz 数据notebook 提供了wave_file辅助函数为其补上 WAV 头默认单声道、24000 Hz、16 位采样宽度以便保存与播放def wave_file(filename, pcm, channels1, rate24000, sample_width2) - None: with wave.open(filename, wb) as wf: wf.setnchannels(channels) wf.setsampwidth(sample_width) wf.setframerate(rate) wf.writeframes(pcm) file_name output_speech.wav wave_file(file_name, data) # Saves the file to current directory Audio(output_speech.wav)Agent Platform 的流式合成使用client.models.generate_content_stream逐 chunk 从chunk.candidates[0].content.parts[0].inline_data.data累积音频字节notebook 将其封装为可复用的synthesize(text, model, voice, locale)函数同样测量首包延迟与完成耗时。六、延伸阅读与仓库配套资源audio目录还包含与本主题相关的其他能力可作为进阶方向音乐生成旧版 lyria2_music_generation.ipynb语音识别Speech-to-Textget_started_with_chirp_3_transcription.ipynbChirp 3 转写、gemini_3_5_transcribe.ipynb 与 gemini_3_5_transcribe_live.pyGemini 3.5 实时转写新模型体验gemini_3_1_flash_tts.ipynbGemini 3.1 Flash TTS 生成高级用例multi-speaker-podcast.ipynb用 Gemini 与 TTS 生成多说话人播客、storytelling.ipynb多角色故事旁白配套剧本 macbeth_the_sitcom.json。七、总结Google Cloud 音频生成能力在audio目录中形成了从音乐到语音的完整矩阵Lyria 3通过 Agent Platform 统一客户端以lyria-3-clip-preview/lyria-3-pro-preview两个模型档位覆盖 30 秒片段与 3 分钟完整歌曲支持文本、图像、自定义歌词三种提示形式并以 Interactions API 提供多语言流式输出所有生成内容默认嵌入 SynthID 与 C2PA 凭证Chirp 3 Instant Custom Voice以参考音频 声纹授权双录音换取voiceCloningKey经 REST 接口实现 25 语言的即时音色克隆需白名单并可快速包装为 Gradio 交互应用Chirp 3 HD Voices提供 8 个 LLM 驱动的高清音色支持 31 种语言覆盖实时synthesize_speech与流式streaming_synthesize两种合成路径Gemini-TTS则以提示词为核心把风格、语速、情绪、表现标签和多说话人对话全部纳入可控范围并同时开放 Cloud TTS API支持双向流与自定义编码与 Agent Platform API统一接口、PCM 输出两条集成路线。实际开发中按现有调用栈、输出编码需求、流式形态、是否已接入 Agent Platform四点即可快速完成 API 选型所有代码均可直接在 audio 目录对应 notebook 中运行验证。【免费下载链接】generative-aiSample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform项目地址: https://gitcode.com/GitHub_Trending/ge/generative-ai创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Google Cloud 音频生成实战指南：Lyria 3 音乐创作与 Chirp 3 / Gemini-TTS 语音合成
Google Cloud 音频生成实战指南Lyria 3 音乐创作与 Chirp 3 / Gemini-TTS 语音合成【免费下载链接】generative-aiSample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform项目地址: https://gitcode.com/GitHub_Trending/ge/generative-ai本文以开源仓库generative-ai中的 audio/README.md 为主线系统梳理 Google Cloud 音频生成能力的完整技术栈从 Lyria 3 音乐生成文本/图像提示、自定义歌词、多语言流式生成到 Chirp 3 Instant Custom Voice 即时音色克隆、Chirp 3 HD 高清音色的实时与流式合成再到 Gemini-TTS 的提示词驱动语音合成与多说话人对话。读完本文你将掌握两套 APICloud Text-to-Speech API 与 Agent Platform API的选择标准与调用方式并能直接复用仓库中的代码完成从文本到音乐、从录音到自定义音色的端到端实现。仓库中的audio目录按能力分为music音乐生成与speech语音合成与处理两大模块所有教程均以可执行的 Jupyter Notebook 形式提供音乐生成lyria3_music_generation.ipynb、lyria2_music_generation.ipynb语音合成get_started_with_chirp3_instant_custom_voice.ipynb、get_started_with_chirp_3_hd_voices.ipynb、get_started_with_gemini_tts_voices.ipynb语音识别与更多用例参见 audio/speech/README.md一、环境准备启用 API 与初始化客户端四个教程共享同一套环境准备流程主要包括以下步骤。1.1 安装 SDK使用 Agent Platform API 的教程Lyria 3、Gemini-TTS 的 Agent Platform 部分需要安装google-genai%pip install --upgrade --quiet google-genai使用 Cloud Text-to-Speech API 的教程Chirp 3 HD、Gemini-TTS 的 Cloud TTS 部分、Chirp 3 Instant Custom Voice需要安装google-cloud-texttospeech其中 Gemini-TTS 相关字段要求google-cloud-texttospeech 版本不低于 2.31.0%pip install --upgrade --quiet google-cloud-texttospeechChirp 3 教程还依赖ffmpeg用于音频处理notebook 中给出了跨平台安装脚本Linux 通过apt install ffmpegmacOS 通过 Homebrew 的brew install ffmpeg。Instant Custom Voice 教程额外依赖gradio%pip install --upgrade --quiet gradio用于构建演示应用。1.2 认证与项目配置在 Colab 环境中先执行用户认证再设置项目信息。各 notebook 均支持通过环境变量GOOGLE_CLOUD_PROJECT回退读取项目 IDimport sys if google.colab in sys.modules: from google.colab import auth auth.authenticate_user()import os # fmt: off PROJECT_ID [your-project-id] # param {type: string, placeholder: [your-project-id], isTemplate: true} # fmt: on if not PROJECT_ID or PROJECT_ID [your-project-id]: PROJECT_ID str(os.environ.get(GOOGLE_CLOUD_PROJECT)) LOCATION os.environ.get(GOOGLE_CLOUD_REGION, global)随后通过 gcloud 设置项目与 ADCApplication Default Credentials配额项目! gcloud config set project {PROJECT_ID} ! gcloud auth application-default set-quota-project {PROJECT_ID} ! gcloud auth application-default login -q两个 API 的客户端初始化方式不同这一点贯穿全文务必区分Agent Platform 客户端Lyria 3、Gemini-TTSfrom google import genai client genai.Client(enterpriseTrue, projectPROJECT_ID, locationLOCATION)Cloud Text-to-Speech 客户端Chirp 3 HD、Gemini-TTSfrom google.api_core.client_options import ClientOptions from google.cloud import texttospeech_v1beta1 as texttospeech API_ENDPOINT ( f{TTS_LOCATION}-texttospeech.googleapis.com if TTS_LOCATION ! global else texttospeech.googleapis.com ) client texttospeech.TextToSpeechClient( client_optionsClientOptions(api_endpointAPI_ENDPOINT) )端点构造逻辑是TTS_LOCATION为global时直接使用texttospeech.googleapis.com否则使用{区域}-texttospeech.googleapis.com形式如us-central1-texttospeech.googleapis.com。具体可用区域需查阅各功能对应的官方文档Chirp 3 HD 见 endpoints 文档Instant Custom Voice 见 regional availability 文档。二、Lyria 3 音乐生成从文本到完整歌曲lyria3_music_generation.ipynb 演示了如何通过 Google Gen AI SDK for Python 与 Agent Platform 上的 Lyria 3 交互生成高质量的音乐片段、完整歌曲与音乐流。教程核心能力包括从文本提示生成带人声的完整歌曲、从图像提示生成音乐片段、自定义歌词、以及通过 Interactions API 实现多语言音乐流式生成。2.1 模型选择clip 与 pro 两个档位notebook 中加载了预览版模型标识二者对应不同的生成时长与用途music_clip_model lyria-3-clip-preview music_model lyria-3-pro-previewlyria-3-clip-preview生成约 30 秒的音乐片段cliplyria-3-pro-preview生成最长 3 分钟的完整歌曲track。从模型命名可以看出两者均为 preview预览阶段模型使用时以仓库中记录的模型名称为准。2.2 定义音频展示辅助函数由于响应中同时包含文本歌词、歌曲结构说明与内联音频字节notebook 定义了display_output辅助函数统一解析response.parts文本部分通过正则去除[[...]]节标记并规整时间戳格式后以 Markdown 渲染inline_data部分直接以 IPython 的Audio组件播放def display_output(response): for part in response.parts: if part.text: text part.text # Remove section tags, add new lines text re.sub(r\[\[.*?\]\], , text) text re.sub( r\s*\[(\d\.\d)(?::?(\d\.\d))?:?\]:?, r\n\n[\1\2] , text ) text text.replace([:], \n\n) clean_text text.strip() display(Markdown(clean_text)) if part.inline_data: display(Audio(datapart.inline_data.data, autoplayFalse))关键点要同时拿到文本与音频输出必须在请求中把response_modalities设置为[AUDIO, TEXT]。2.3 从文本提示生成完整歌曲编写提示词时notebook 建议从三个维度展开描述风格/流派Style/Genre如 classical、electronic、rock、jazz、hip hop、pop也可以使用更泛化的描述如 cinematic、ambient、lo-fi人声Vocals如需人声描述音色与音域如音域、音调等特征乐器Instruments指定钢琴、合成器、原声吉他、鼓、弦乐、长笛等具体乐器。示例提示词与调用prompt Sophisticated, rhythmic, and aspirational track with crisp 808 percussion, digital plucks, and muted electric guitar rhythmic strums. Include breathy, airy Alto female vocal textures with melodic, minimalist oohs and aahs with heavy reverb and rhythmic delay. response client.models.generate_content( modelmusic_model, contentsprompt, configtypes.GenerateContentConfig(response_modalities[AUDIO, TEXT]), ) display_output(response)默认情况下Lyria 3 生成的所有音乐都会嵌入 SynthIDAI 生成内容水印与 C2PA 内容凭证用于标识内容的 AI 生成属性。2.4 从图像提示生成音乐片段Lyria 3 支持以图像作为参考输入生成音乐。notebook 首先下载示例图片!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/flowers.png然后以types.Part.from_bytes构造图像内容配合文本提示调用lyria-3-clip-preview模型。单次请求最多可携带 10 张图像with open(input_image, rb) as f: image f.read() response client.models.generate_content( modelmusic_clip_model, contents[ types.Part.from_bytes( dataimage, mime_typeimage/png, ), Generate an instrumental clip based on this input image that starts slowly and builds in intensity., ], configtypes.GenerateContentConfig( response_modalities[TEXT, AUDIO], ), ) display_output(response)2.5 自定义歌词在图像之外还可以在提示中直接给定歌词文本并附加速度tempo、流派、风格、人声配置、乐器等补充指令。notebook 的示例把风格描述 完整歌词组织成一个结构化文本块传给模型genre_lyrics Genre: Upbeat, acoustic Folk-Pop with a warm and cuddly vibe. Bright acoustic guitars, a soft shaker rhythm, and a friendly, melodic vocal. Lyrics: Tail wags and a heavy head, Time to curl up in your favorite bed. Soft as a cloud, a dream come true, The perfect spot for a dog like you. response client.models.generate_content( modelmusic_clip_model, contents[ types.Part.from_bytes(dataimage, mime_typeimage/png), genre_lyrics, ], configtypes.GenerateContentConfig(response_modalities[TEXT, AUDIO]), )2.6 Interactions API多语言生成与流式输出Interactions API 是 Agent Platform 提供的统一模型/Agent 交互接口。Lyria 3 支持英语、德语、西班牙语、法语、印地语、日语、韩语、葡萄牙语八种语言既可以直接用这些语言写提示也可以在使用其他语言时显式要求目标语言。非流式调用示例以西班牙语提示为例interaction client.interactions.create( modelmusic_model, inputGenera un tema pop, ) display_interaction_output(interaction)Interactions API 的响应结构与generate_content不同输出位于interaction.outputs列表中每项是带type字段的字典text或audio音频数据为 base64 编码字符串需解码后播放def display_interaction_output(interaction): outputs getattr(interaction, outputs, []) or [] for output in outputs: is_dict isinstance(output, dict) output_type output.get(type) if is_dict else getattr(output, type, None) if output_type text: # ... 文本清洗与 Markdown 渲染 ... elif output_type audio: data output.get(data) if is_dict else getattr(output, data, None) audio_data base64.b64decode(data) display(Audio(dataaudio_data, autoplayFalse))流式生成在interactions.create中设置streamTrue歌词与描述会随模型输出即时返回无需等待整个请求完成。遍历流式事件时通过event.event_type content.delta判断增量内容event.delta中的text字段是文本增量data配合mime_type中的audio判断音频块stream client.interactions.create( modelmusic_model, inputGenerate a song about spending a day in Seoul in Korean., streamTrue, ) for event in stream: if event.event_type content.delta: delta_dict event.delta if isinstance(event.delta, dict) else getattr(event, delta, {}) if text in delta_dict: text delta_dict[text] text text.replace([:], \n[:]) text re.sub(r(\[\d\.\d:\]), r\n\1, text) display(Markdown(text)) elif data in delta_dict and audio in delta_dict.get(mime_type, ): display(Audio(database64.b64decode(delta_dict[data]), autoplayFalse))三、Chirp 3 Instant Custom Voice10 秒录音创建专属音色get_started_with_chirp3_instant_custom_voice.ipynb 介绍 Google Cloud Text-to-Speech API 的Chirp 3 Instant Custom Voice功能用你自己的高质量录音训练出个性化语音模型随后即可用该音色合成音频支持流式与长文本输出且音色创建与合成支持25 种以上语言。3.1 访问限制Allowlist⚠️由于安全考量该音色克隆能力仅对白名单allow-listed用户开放。如需使用必须联系 Google Cloud 团队成员加入白名单。这一点在 notebook 顶部以醒目的警告框标出是使用该功能的前置条件。3.2 获取凭证与构造端点与 Agent Platform 客户端不同Instant Custom Voice 教程直接使用 REST API 调用因此需要通过google.auth获取 OAuth 访问令牌import google.auth import google.auth.transport.requests credentials, _ google.auth.default() authentication google.auth.transport.requests.Request() credentials.refresh(authentication)API_ENDPOINT ( f{TTS_LOCATION}-texttospeech.googleapis.com if TTS_LOCATION ! global else texttospeech.googleapis.com ) ACCESS_TOKEN credentials.token3.3 创建音色克隆密钥Voice Cloning Key核心 API 是POST /v1beta1/voices:generateVoiceCloningKey。请求体包含三部分关键输入reference_audio参考音频预先录制的目标音色样本.wav格式最佳理想情况是几秒钟的清晰语音voice_talent_consent声纹所有人授权音频说话人明确朗读同意脚本的录音consent_script同意脚本文本即授权语句本身。音频统一采用LINEAR16编码、24000Hz 采样率并以 base64 形式放在content字段。notebook 中的实现如下def create_instant_custom_voice_key( reference_audio_bytes: bytes, consent_audio_bytes: bytes ) - str: Creates a temporary custom voice key url fhttps://{API_ENDPOINT}/v1beta1/voices:generateVoiceCloningKey request_body { reference_audio: { audio_config: {audio_encoding: LINEAR16, sample_rate_hertz: 24000}, content: reference_audio_bytes, }, voice_talent_consent: { audio_config: {audio_encoding: LINEAR16, sample_rate_hertz: 24000}, content: consent_audio_bytes, }, consent_script: I am the owner of this voice and I consent to Google using this voice to create a synthetic voice model., language_code: en-US, } headers { Authorization: fBearer {ACCESS_TOKEN}, x-goog-user-project: PROJECT_ID, Content-Type: application/json; charsetutf-8, } response requests.post(url, headersheaders, jsonrequest_body) response.raise_for_status() response_json response.json() return response_json.get(voiceCloningKey)请求成功且项目在白名单内时API 返回voiceCloningKey它是自定义音色的临时标识。本地 WAV 文件通过wav_to_base64辅助函数读取并编码为 base64 字符串即 JSON payload 中音频数据的传输格式。3.4 使用自定义音色合成语音合成接口为POST /v1beta1/text:synthesize。关键差异在于voice字段携带voice_clone.voice_cloning_keydef synthesize_text_with_cloned_voice(voice_key: str, text: str) - None: url fhttps://{API_ENDPOINT}/v1beta1/text:synthesize request_body { input: {text: text}, voice: { language_code: en-US, voice_clone: { voice_cloning_key: voice_key, }, }, audioConfig: {audioEncoding: LINEAR16, sample_rate_hertz: 24000}, } headers { Authorization: fBearer {ACCESS_TOKEN}, x-goog-user-project: PROJECT_ID, Content-Type: application/json; charsetutf-8, } response requests.post(url, headersheaders, jsonrequest_body) response.raise_for_status() response_json response.json() audio_content response_json.get(audioContent) if audio_content: display(Audio(base64.b64decode(audio_content), rate24000))notebook 提供的示例文本展示了其典型用法新闻播报风格且文本内容本身由克隆音色生成text_to_synthesize Breaking news! Chirp 3, Google Clouds audio model, now has Instant Custom Voice. ... synthesize_text_with_cloned_voice(voice_key, text_to_synthesize)3.5 构建 Gradio 演示应用为了交互式地上传音频 → 创建音色 → 合成语音notebook 用 Gradio 构建了一个完整 Web 应用。核心设计用gr.State()保存完整 voice key用于后续合成左侧列Reference Voice参考音频与Consent Audio授权音频两个gr.Audio(typefilepath)组件 Create Voice 按钮 Voice Key 文本框右侧列Text to Synthesize文本框 Generate Speech 按钮 生成音频输出 状态输出底部 Clear All 按钮负责重置整个界面。事件处理链路上create_voice_with_masking包装了create_voice内部依次调用wav_to_base64→create_instant_custom_voice_key并在返回时把 key 掩码显示前 5 位 ...以避免界面泄露完整密钥generate_speech则复用 3.4 节的合成逻辑把 base64 音频解码为np.int16数组并返回(24000, audio_array)元组供 Gradio 播放。全部完成后通过app.launch(shareTrue)启动、app.close()关闭。四、Chirp 3 HD 高清音色实时与流式合成get_started_with_chirp_3_hd_voices.ipynb 介绍基于大语言模型LLM驱动的Chirp 3 HD Voices相比传统 TTS它在真实感和情感表达上显著提升输出高保真、带类人语调和停顿的自然语音。当前提供8 个音色4 男 4 女、支持 31 种语言适用于语音助手、有声书、客户服务等场景。4.1 音色命名与选择音色列表为Aoede、Puck、Charon、Kore、Fenrir、Leda、Orus、Zephyr。完整音色名遵循{language_code}-Chirp3-HD-{voice}命名规范例如en-US-Chirp3-HD-Aoede。notebook 中通过参数下拉框选择音色与语言voice Aoede # 可选: Aoede, Puck, Charon, Kore, Fenrir, Leda, Orus, Zephyr language_code en-US # 可选: de-DE, en-AU, en-GB, en-IN, en-US, fr-FR, hi-IN, pt-BR, ar-XA, es-ES, fr-CA, id-ID, it-IT, ja-JP, tr-TR, vi-VN, bn-IN, gu-IN, kn-IN, ml-IN, mr-IN, ta-IN, te-IN, nl-NL, ko-KR, cmn-CN, pl-PL, ru-RU, th-TH voice_name f{language_code}-Chirp3-HD-{voice} voice texttospeech.VoiceSelectionParams( namevoice_name, language_codelanguage_code, )4.2 实时在线合成实时处理调用synthesize_speech方法输出为 MP3 格式的音频字节response client.synthesize_speech( inputtexttospeech.SynthesisInput(textprompt), voicevoice, # Select the type of audio file you want returned audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.MP3 ), ) display(Audio(response.audio_content))4.3 流式合成Chirp 3 HD 支持通过streaming_synthesize方法进行流式 TTS与处理单次请求的synthesize_speech不同它处理连续文本流并生成对应的音频流。notebook 的实现展示了双向流协议的正确用法——必须先发送一条包含streaming_config的配置请求再逐条发送文本输入请求服务端在客户端半关闭half-close即请求生成器耗尽后开始回推音频def synthesize_streaming( text_iterator: Iterator[str], voice: texttospeech.VoiceSelectionParams, ) - Iterator[bytes]: config_request texttospeech.StreamingSynthesizeRequest( streaming_configtexttospeech.StreamingSynthesizeConfig( voicevoice, ) ) def request_generator() - Iterator[texttospeech.StreamingSynthesizeRequest]: yield config_request for text in text_iterator: yield texttospeech.StreamingSynthesizeRequest( inputtexttospeech.StreamingSynthesisInput(texttext) ) streaming_responses: Iterator[texttospeech.StreamingSynthesizeResponse] ( client.streaming_synthesize(request_generator()) ) for response in streaming_responses: yield response.audio_content为了模拟文本逐步生成的场景notebook 还提供了text_generator辅助函数用正则正向先行断言按句号/感叹号/问号把长文本切分成句子迭代器再逐句送入流式接口。process_streaming_audio函数负责将各分块音频np.int16采样拼接为最终完整音频并支持display_individual_chunks参数逐块预览最终以 24000 Hz 采样率播放。五、Gemini-TTS用自然语言提示词精确控制语音get_started_with_gemini_tts_voices.ipynb 介绍Gemini-TTS——在自然度之外更进一步用文本提示词对生成音频进行细粒度控制。你可以从短片段到长篇叙事自由合成语音通过自然语言提示精确指定风格、口音、语速、语调乃至情感表达还可以创建两个说话人之间、具有相同情感表达与可控性的对话。当前提供30 个音色、80 语言区域locale选项。5.1 双 API 选择决策Gemini-TTS 同时通过两条 API 开放便于不同场景的客户端集成选择 Cloud Text-to-Speech API 的情形已在用 Chirp 3 HD 或其他音色希望以最小增量继续使用同一 API需要指定特定的输出编码类型——Cloud TTS API 可以指定音频编码而 Agent Platform API 输出的是无 WAV 头的 PCM 16bit 24kHz 裸数据如需其他格式须在客户端自行转换需要双向流式——Cloud TTS API 支持多请求多响应交互而 Agent Platform API 仅支持单请求多响应。选择 Agent Platform API 的情形已从 AI Studio 使用 Gemini-TTS希望无缝迁移到 Agent Platform 以获得 Google Cloud 的扩展性与合规能力已在 Agent Platform API 上使用其他模型——统一 API 结构使其只需指定模型名与音色选项即可开始使用 Gemini-TTS。5.2 Cloud TTS API 单说话人合成模型选项包括gemini-3.1-flash-tts-preview、gemini-2.5-flash-tts、gemini-2.5-pro-tts音色选项即第 4.1 节的 8 个 HD 音色加上 Gemini 专属音色Achernar、Achird、Algenib、Algieba、Alnilam、Autonoe、Callirrhoe、Despina、Enceladus、Erinome、Gacrux、Iapetus、Laomedeia、Pulcherrima、Rasalgethi、Sadachbia、Sadaltager、Schedar、Sulafat、Umbriel、Vindemiatrix、Zubenelgenubi等共 30 个。语言代码覆盖en-us、en-gb、zh系cmn-cn、cmn-tw、hi-in、ja-jp、ko-kr等 80 区域。与 Chirp 3 HD 的关键差异在于VoiceSelectionParams需显式指定model_nameMODEL gemini-3.1-flash-tts-preview VOICE Aoede LANGUAGE_CODE en-us voice texttospeech.VoiceSelectionParams( nameVOICE, language_codeLANGUAGE_CODE, model_nameMODEL )合成调用在SynthesisInput中同时携带text与prompt两个字段——prompt 正是 Gemini-TTS提示词驱动能力的核心入口# title Capture emotion with prompts PROMPT You are having a conversation with a friend. Say the following in a happy and casual way TEXT hahaha, i did NOT expect that. can you believe it! response client.synthesize_speech( inputtexttospeech.SynthesisInput(textTEXT, promptPROMPT), voicevoice, audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.MP3 ), ) display(Audio(response.audio_content))5.3 用提示词与表现标签控制语速与情绪控制语速如PROMPT Say the following very fast but still be intelligible配合同一段法务免责声明文本即可得到快速但仍清晰的朗读表现标签Expressive Tags在文本中嵌入[chuckling]、[coughs]、[um]等口语/情绪标签配合提示词控制整体语气。notebook 明确指出这些标签并非严格语法鼓励自由实验不同表达与格式PROMPT Say the following with a sarcastic tone TEXT So.. [chuckling] tell me about this [coughs] AI thing.5.4 多说话人对话合成Gemini-TTS 支持在同一段音频中合成两个说话人的对话。核心配置对象是MultiSpeakerVoiceConfig为每个说话人分配speaker_alias自定义别名用于在文本中引用与speaker_id实际使用的预置音色。输入方式有两种方式一显式轮次语法——使用MultiSpeakerMarkup.Turn结构化描述每句的说话人与文本SPEAKER_ALIAS_1 Zizu SPEAKER_1 Fenrir SPEAKER_ALIAS_2 Gary SPEAKER_2 Orus LANGUAGE_CODE en-gb PROMPT Read the following dialogue between two friends multi_speaker_voice_config texttospeech.MultiSpeakerVoiceConfig( speaker_voice_configs[ texttospeech.MultispeakerPrebuiltVoice( speaker_aliasSPEAKER_ALIAS_1, speaker_idSPEAKER_1 ), texttospeech.MultispeakerPrebuiltVoice( speaker_aliasSPEAKER_ALIAS_2, speaker_idSPEAKER_2 ), ] ) multi_speaker_markup texttospeech.MultiSpeakerMarkup( turns[ texttospeech.MultiSpeakerMarkup.Turn( speakerSPEAKER_ALIAS_1, textHave you tried the new multi-speaker feature on Gemini?, ), texttospeech.MultiSpeakerMarkup.Turn( speakerSPEAKER_ALIAS_2, textYes! I am super excited about it ), ] ) response client.synthesize_speech( inputtexttospeech.SynthesisInput( multi_speaker_markupmulti_speaker_markup, promptPROMPT ), voicetexttospeech.VoiceSelectionParams( language_codeLANGUAGE_CODE, model_nameMODEL, multi_speaker_voice_configmulti_speaker_voice_config, ), audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.LINEAR16 ), )方式二内联对话文本——直接在text字段中用别名: 内容加换行的格式书写整段对话MultiSpeakerMarkup可省略multi_speaker_voice_config仍必须提供response client.synthesize_speech( inputtexttospeech.SynthesisInput( textZizu: Have you tried the new multi-speaker feature on Gemini?\nGary: Yes! I am super excited about it, promptPROMPT, ), voicetexttospeech.VoiceSelectionParams( language_codeLANGUAGE_CODE, model_nameMODEL, multi_speaker_voice_configmulti_speaker_voice_config, ), audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.LINEAR16 ), )5.5 放松安全过滤relax_safety_filtersGemini-TTS 默认对有害内容设有过滤。采用**月度发票计费monthly invoiced billing**的账号可通过AdvancedVoiceOptions中的relax_safety_filters字段放松过滤该字段对非此类计费账号不生效response client.synthesize_speech( inputtexttospeech.SynthesisInput(textTEXT, promptPROMPT), voicevoice, audio_configtexttospeech.AudioConfig( audio_encodingtexttospeech.AudioEncoding.MP3 ), advanced_voice_optionstexttospeech.AdvancedVoiceOptions(relax_safety_filtersTrue), )5.6 流式合成Cloud TTS APICloud TTS API 的流式模式与 Chirp 3 HD 完全一致先发streaming_config请求再发文本输入客户端请求生成器耗尽即触发 half-close随后开始接收音频流。notebook 给出了真实场景的关键建议——流式响应应在到达时立即播放/转发例如在 Web 服务器中通过 WebSocket 用emit(audio, response.audio_content)把每个音频块即时推给前端而非等全部收完。config_request texttospeech.StreamingSynthesizeRequest( streaming_configtexttospeech.StreamingSynthesizeConfig( voicetexttospeech.VoiceSelectionParams( nameVOICE, language_codeLANGUAGE_CODE, model_nameMODEL ) ) ) def request_generator(): yield config_request yield texttospeech.StreamingSynthesizeRequest( inputtexttospeech.StreamingSynthesisInput(textTEXT, promptPROMPT) ) streaming_responses client.streaming_synthesize(request_generator()) for response in streaming_responses: audio_data np.frombuffer(response.audio_content, dtypenp.int16) final_audio_data np.concatenate((final_audio_data, audio_data))该教程还演示了如何测量time to first audio首包延迟与time to completion完成总耗时并据此计算音频时长len(final_audio_data) / 24_000默认采样率 24kHz这为实时应用的延迟评估提供了直接参照。5.7 Agent Platform API 合成与 PCM 后处理Agent Platform API 使用统一的client.models.generate_content接口模型名直接写在参数中语音配置通过speech_config传入prebuilt_voice_config指定预置音色from google import genai from google.genai import types client genai.Client(enterpriseTrue, projectPROJECT_ID, locationLOCATION) response client.models.generate_content( modelgemini-2.5-flash-tts, contentsTEXT, configtypes.GenerateContentConfig( speech_configtypes.SpeechConfig( language_codeen-in, voice_configtypes.VoiceConfig( prebuilt_voice_configtypes.PrebuiltVoiceConfig( voice_nameKore, ) ), ), ), ) data response.candidates[0].content.parts[0].inline_data.data由于 Agent Platform 返回的是无 WAV 头的 PCM 16bit 24kHz 数据notebook 提供了wave_file辅助函数为其补上 WAV 头默认单声道、24000 Hz、16 位采样宽度以便保存与播放def wave_file(filename, pcm, channels1, rate24000, sample_width2) - None: with wave.open(filename, wb) as wf: wf.setnchannels(channels) wf.setsampwidth(sample_width) wf.setframerate(rate) wf.writeframes(pcm) file_name output_speech.wav wave_file(file_name, data) # Saves the file to current directory Audio(output_speech.wav)Agent Platform 的流式合成使用client.models.generate_content_stream逐 chunk 从chunk.candidates[0].content.parts[0].inline_data.data累积音频字节notebook 将其封装为可复用的synthesize(text, model, voice, locale)函数同样测量首包延迟与完成耗时。六、延伸阅读与仓库配套资源audio目录还包含与本主题相关的其他能力可作为进阶方向音乐生成旧版 lyria2_music_generation.ipynb语音识别Speech-to-Textget_started_with_chirp_3_transcription.ipynbChirp 3 转写、gemini_3_5_transcribe.ipynb 与 gemini_3_5_transcribe_live.pyGemini 3.5 实时转写新模型体验gemini_3_1_flash_tts.ipynbGemini 3.1 Flash TTS 生成高级用例multi-speaker-podcast.ipynb用 Gemini 与 TTS 生成多说话人播客、storytelling.ipynb多角色故事旁白配套剧本 macbeth_the_sitcom.json。七、总结Google Cloud 音频生成能力在audio目录中形成了从音乐到语音的完整矩阵Lyria 3通过 Agent Platform 统一客户端以lyria-3-clip-preview/lyria-3-pro-preview两个模型档位覆盖 30 秒片段与 3 分钟完整歌曲支持文本、图像、自定义歌词三种提示形式并以 Interactions API 提供多语言流式输出所有生成内容默认嵌入 SynthID 与 C2PA 凭证Chirp 3 Instant Custom Voice以参考音频 声纹授权双录音换取voiceCloningKey经 REST 接口实现 25 语言的即时音色克隆需白名单并可快速包装为 Gradio 交互应用Chirp 3 HD Voices提供 8 个 LLM 驱动的高清音色支持 31 种语言覆盖实时synthesize_speech与流式streaming_synthesize两种合成路径Gemini-TTS则以提示词为核心把风格、语速、情绪、表现标签和多说话人对话全部纳入可控范围并同时开放 Cloud TTS API支持双向流与自定义编码与 Agent Platform API统一接口、PCM 输出两条集成路线。实际开发中按现有调用栈、输出编码需求、流式形态、是否已接入 Agent Platform四点即可快速完成 API 选型所有代码均可直接在 audio 目录对应 notebook 中运行验证。【免费下载链接】generative-aiSample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform项目地址: https://gitcode.com/GitHub_Trending/ge/generative-ai创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考