尧图网页设计 
【Bug已解决】429 RESOURCE_EXHAUSTED for Claude Sonnet 3.5 on Vertex AI 解决方案一、现象长什么样你在 Vertex AI 上用 Claude Sonnet 3.5遇到429 RESOURCE_EXHAUSTED与 917 的Quota exceeded (concurrent)不同这里是资源耗尽更偏区域容量维度单个请求、低并发也会偶发高峰期如整点更频繁同一份代码某些区域能过、某些区域直接RESOURCE_EXHAUSTED重试有时成功、有时持续失败一段时间报错不带具体配额名只是笼统的资源耗尽。一句话Vertex AI 在该区域为该模型分配的瞬时容量不足你的请求落在了区域没多余算力的窗口于是被RESOURCE_EXHAUSTED拒绝——这更多是区域容量/时段的供给问题而非你账户的配额设置。二、背景Vertex AI 的RESOURCE_EXHAUSTEDHTTP 429和配额超限quota是两类不同信号配额超限你账户维度的限额QPM、并发用满了需要提额资源耗尽该区域底层加速器GPU/TPU当下没有空闲容量服务于这个模型是平台供给侧的瞬时紧张。Claude Sonnet 3.5 在 Vertex 上很受欢迎某些区域在高峰时段会出现容量吃紧。这时即便你配额没满请求也会被RESOURCE_EXHAUSTED挡下。它通常短暂、可重试且与区域强相关。三、根因根因是目标区域当下没有足够容量服务该模型请求打到 region-A 的 Claude Sonnet 3.5 - region-A 该模型底层容量已满高峰/供给紧张 - Vertex 返回 429 RESOURCE_EXHAUSTED - 换个 region-B容量充裕- 成功这与 917 的per_base_model concurrent quota区别917 是你并发太多这里是区域没容量。处理策略也不同917 降并发即可这里要靠重试 区域故障转移failover。四、最小可运行复现下面用 Python 模拟区域容量耗尽 failover 成功import asyncio import random from dataclasses import dataclass from typing import List dataclass class _VertexRegion: name: str capacity_free: float # 0~1剩余容量比例 async def call(self) - str: if random.random() self.capacity_free: raise RuntimeError(429 RESOURCE_EXHAUSTED) return ok async def try_regions(regions: List[_VertexRegion]) - str: last None for r in regions: try: return await r.call() except RuntimeError as e: last e continue # failover 到下一个区域 raise last async def main(): regions [ _VertexRegion(us-central1, capacity_free0.05), # 紧张 _VertexRegion(us-east5, capacity_free0.8), # 充裕 ] try: print(await try_regions(regions)) except RuntimeError as e: print(ERR:, e) if __name__ __main__: asyncio.run(main())运行后优先紧张的 region 失败、failover 到充裕 region 成功与真实RESOURCE_EXHAUSTED 区域转移同构。五、解决方案第一层最小直接修复最小修复是加重试退避 区域故障转移import asyncio import random from anthropic import AnthropicVertex async def call_with_failover(prompt, regions, max_retries5): delay 1.0 for region in regions: for _ in range(max_retries): try: client AnthropicVertex(project_idp, regionregion) return await client.messages.create( modelclaude-3-5-sonnet-v220241022, max_tokens256, messages[{role: user, content: prompt}], ) except Exception as e: if RESOURCE_EXHAUSTED in str(e) or 429 in str(e): await asyncio.sleep(delay random.random()) delay * 2 continue raise # 该 region 重试耗尽换下一个 region raise RuntimeError(所有区域均 RESOURCE_EXHAUSTED) # 用法把容量充裕的区域放前面 await call_with_failover(hi, [us-east5, us-central1])六、解决方案第二层结构化改进把区域 failover 退避抽成策略集中管理可用区域与重试from dataclasses import dataclass, field import asyncio import random from typing import Awaitable, Callable, List, TypeVar T TypeVar(T) dataclass(frozenTrue) class ClaudeVertexExhaustedPolicy: Vertex RESOURCE_EXHAUSTED 策略区域故障转移 指数退避。 规则 - 维护有序区域列表容量充裕优先 - 429/RESOURCE_EXHAUSTED 时退避重试 - 单区域重试耗尽则 failover 下一区域 regions: List[str] field(default_factorylambda: [us-east5, us-central1]) max_retries: int 5 base_delay: float 1.0 async def run(self, worker: Callable[[str], Awaitable[T]]) - T: delay self.base_delay last_err None for region in self.regions: for _ in range(self.max_retries): try: return await worker(region) except Exception as e: if RESOURCE_EXHAUSTED in str(e) or 429 in str(e): last_err e await asyncio.sleep(delay random.random()) delay * 2 continue raise delay self.base_delay # 换区域重置退避 raise last_err or RuntimeError(RESOURCE_EXHAUSTED) def demo() - None: policy ClaudeVertexExhaustedPolicy(regions[us-east5, us-central1]) print(failover 顺序:, policy.regions) if __name__ __main__: demo()若长期持续RESOURCE_EXHAUSTED应到 GCP 申请该模型的预留/提升区域配额或固定到容量充裕的区域。七、解决方案第三层断言 / CI 守护import asyncio import pytest from your_module import ClaudeVertexExhaustedPolicy def test_failover_order(): policy ClaudeVertexExhaustedPolicy(regions[us-east5, us-central1]) assert policy.regions[0] us-east5 def test_retry_then_failover(): policy ClaudeVertexExhaustedPolicy(max_retries2, base_delay0.0) calls {n: 0} async def worker(region): calls[n] 1 if region us-east5: raise RuntimeError(429 RESOURCE_EXHAUSTED) return ok async def main(): return await policy.run(worker) assert asyncio.run(main()) ok # us-east5 重试2次 us-central1 成功 assert calls[n] 3 def test_all_exhausted_raises(): policy ClaudeVertexExhaustedPolicy(max_retries1, base_delay0.0) async def worker(region): raise RuntimeError(429 RESOURCE_EXHAUSTED) async def main(): try: await policy.run(worker) except RuntimeError: return raised assert asyncio.run(main()) raised def test_non_429_not_retried(): policy ClaudeVertexExhaustedPolicy(max_retries5) calls {n: 0} async def worker(region): calls[n] 1 raise ValueError(real) async def main(): try: await policy.run(worker) except ValueError: pass asyncio.run(main()) assert calls[n] 1CI 里加一条用受控的首区域必耗尽模拟断言能 failover 到次区域成功。八、排查清单报错是RESOURCE_EXHAUSTED还是Quota exceeded前者偏区域容量后者偏账户配额。是否做了区域故障转移把容量充裕的区域放前面。是否加了指数退避重试高峰期靠重试扛过。换区域是否就成功成功即可锁定是区域供给问题。是否长期持续长期应申请预留/提升区域配额。是否混淆了并发配额917与区域容量本篇九、小结Vertex AI 上 Claude Sonnet 3.5 的429 RESOURCE_EXHAUSTED与配额超限不同——它反映的是目标区域当下容量紧张供给侧瞬时不足与并发数无关。最小修复是加重试退避 区域故障转移容量充裕区域优先结构化做法是抽成ClaudeVertexExhaustedPolicy集中管理区域顺序与退避最后用 pytest 守护单区域耗尽能 failover 到下一区域、非 429 不重试让调用在高峰时段也能稳定成功。 【Bug已解决】429 RESOURCE_EXHAUSTED for Claude Sonnet 3.5 on Vertex AI 解决方案
【Bug已解决】429 RESOURCE_EXHAUSTED for Claude Sonnet 3.5 on Vertex AI 解决方案一、现象长什么样你在 Vertex AI 上用 Claude Sonnet 3.5遇到429 RESOURCE_EXHAUSTED与 917 的Quota exceeded (concurrent)不同这里是资源耗尽更偏区域容量维度单个请求、低并发也会偶发高峰期如整点更频繁同一份代码某些区域能过、某些区域直接RESOURCE_EXHAUSTED重试有时成功、有时持续失败一段时间报错不带具体配额名只是笼统的资源耗尽。一句话Vertex AI 在该区域为该模型分配的瞬时容量不足你的请求落在了区域没多余算力的窗口于是被RESOURCE_EXHAUSTED拒绝——这更多是区域容量/时段的供给问题而非你账户的配额设置。二、背景Vertex AI 的RESOURCE_EXHAUSTEDHTTP 429和配额超限quota是两类不同信号配额超限你账户维度的限额QPM、并发用满了需要提额资源耗尽该区域底层加速器GPU/TPU当下没有空闲容量服务于这个模型是平台供给侧的瞬时紧张。Claude Sonnet 3.5 在 Vertex 上很受欢迎某些区域在高峰时段会出现容量吃紧。这时即便你配额没满请求也会被RESOURCE_EXHAUSTED挡下。它通常短暂、可重试且与区域强相关。三、根因根因是目标区域当下没有足够容量服务该模型请求打到 region-A 的 Claude Sonnet 3.5 - region-A 该模型底层容量已满高峰/供给紧张 - Vertex 返回 429 RESOURCE_EXHAUSTED - 换个 region-B容量充裕- 成功这与 917 的per_base_model concurrent quota区别917 是你并发太多这里是区域没容量。处理策略也不同917 降并发即可这里要靠重试 区域故障转移failover。四、最小可运行复现下面用 Python 模拟区域容量耗尽 failover 成功import asyncio import random from dataclasses import dataclass from typing import List dataclass class _VertexRegion: name: str capacity_free: float # 0~1剩余容量比例 async def call(self) - str: if random.random() self.capacity_free: raise RuntimeError(429 RESOURCE_EXHAUSTED) return ok async def try_regions(regions: List[_VertexRegion]) - str: last None for r in regions: try: return await r.call() except RuntimeError as e: last e continue # failover 到下一个区域 raise last async def main(): regions [ _VertexRegion(us-central1, capacity_free0.05), # 紧张 _VertexRegion(us-east5, capacity_free0.8), # 充裕 ] try: print(await try_regions(regions)) except RuntimeError as e: print(ERR:, e) if __name__ __main__: asyncio.run(main())运行后优先紧张的 region 失败、failover 到充裕 region 成功与真实RESOURCE_EXHAUSTED 区域转移同构。五、解决方案第一层最小直接修复最小修复是加重试退避 区域故障转移import asyncio import random from anthropic import AnthropicVertex async def call_with_failover(prompt, regions, max_retries5): delay 1.0 for region in regions: for _ in range(max_retries): try: client AnthropicVertex(project_idp, regionregion) return await client.messages.create( modelclaude-3-5-sonnet-v220241022, max_tokens256, messages[{role: user, content: prompt}], ) except Exception as e: if RESOURCE_EXHAUSTED in str(e) or 429 in str(e): await asyncio.sleep(delay random.random()) delay * 2 continue raise # 该 region 重试耗尽换下一个 region raise RuntimeError(所有区域均 RESOURCE_EXHAUSTED) # 用法把容量充裕的区域放前面 await call_with_failover(hi, [us-east5, us-central1])六、解决方案第二层结构化改进把区域 failover 退避抽成策略集中管理可用区域与重试from dataclasses import dataclass, field import asyncio import random from typing import Awaitable, Callable, List, TypeVar T TypeVar(T) dataclass(frozenTrue) class ClaudeVertexExhaustedPolicy: Vertex RESOURCE_EXHAUSTED 策略区域故障转移 指数退避。 规则 - 维护有序区域列表容量充裕优先 - 429/RESOURCE_EXHAUSTED 时退避重试 - 单区域重试耗尽则 failover 下一区域 regions: List[str] field(default_factorylambda: [us-east5, us-central1]) max_retries: int 5 base_delay: float 1.0 async def run(self, worker: Callable[[str], Awaitable[T]]) - T: delay self.base_delay last_err None for region in self.regions: for _ in range(self.max_retries): try: return await worker(region) except Exception as e: if RESOURCE_EXHAUSTED in str(e) or 429 in str(e): last_err e await asyncio.sleep(delay random.random()) delay * 2 continue raise delay self.base_delay # 换区域重置退避 raise last_err or RuntimeError(RESOURCE_EXHAUSTED) def demo() - None: policy ClaudeVertexExhaustedPolicy(regions[us-east5, us-central1]) print(failover 顺序:, policy.regions) if __name__ __main__: demo()若长期持续RESOURCE_EXHAUSTED应到 GCP 申请该模型的预留/提升区域配额或固定到容量充裕的区域。七、解决方案第三层断言 / CI 守护import asyncio import pytest from your_module import ClaudeVertexExhaustedPolicy def test_failover_order(): policy ClaudeVertexExhaustedPolicy(regions[us-east5, us-central1]) assert policy.regions[0] us-east5 def test_retry_then_failover(): policy ClaudeVertexExhaustedPolicy(max_retries2, base_delay0.0) calls {n: 0} async def worker(region): calls[n] 1 if region us-east5: raise RuntimeError(429 RESOURCE_EXHAUSTED) return ok async def main(): return await policy.run(worker) assert asyncio.run(main()) ok # us-east5 重试2次 us-central1 成功 assert calls[n] 3 def test_all_exhausted_raises(): policy ClaudeVertexExhaustedPolicy(max_retries1, base_delay0.0) async def worker(region): raise RuntimeError(429 RESOURCE_EXHAUSTED) async def main(): try: await policy.run(worker) except RuntimeError: return raised assert asyncio.run(main()) raised def test_non_429_not_retried(): policy ClaudeVertexExhaustedPolicy(max_retries5) calls {n: 0} async def worker(region): calls[n] 1 raise ValueError(real) async def main(): try: await policy.run(worker) except ValueError: pass asyncio.run(main()) assert calls[n] 1CI 里加一条用受控的首区域必耗尽模拟断言能 failover 到次区域成功。八、排查清单报错是RESOURCE_EXHAUSTED还是Quota exceeded前者偏区域容量后者偏账户配额。是否做了区域故障转移把容量充裕的区域放前面。是否加了指数退避重试高峰期靠重试扛过。换区域是否就成功成功即可锁定是区域供给问题。是否长期持续长期应申请预留/提升区域配额。是否混淆了并发配额917与区域容量本篇九、小结Vertex AI 上 Claude Sonnet 3.5 的429 RESOURCE_EXHAUSTED与配额超限不同——它反映的是目标区域当下容量紧张供给侧瞬时不足与并发数无关。最小修复是加重试退避 区域故障转移容量充裕区域优先结构化做法是抽成ClaudeVertexExhaustedPolicy集中管理区域顺序与退避最后用 pytest 守护单区域耗尽能 failover 到下一区域、非 429 不重试让调用在高峰时段也能稳定成功。