mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-11-12 12:17:23 +00:00
fix(codearts): 上游限流改为可声明重试(内核通用能力)+ 会话/连接两处真因
用户连续报「codeart 还是不可用 / empty stream」。把上游原始响应打出来逐条对齐,
三个症状同源,且都不是"模型坏了":
| 场景 | 上游实际返回 |
|---|---|
| 流式、短时间连续请求 | HTTP **200** + 只带 error 的帧 ⇒ 内核判 `empty stream` |
| 非流式、同一条件 | HTTP 400 `TM.00001041: 并发会话数已达上限(3个)` |
| 突发请求 | HTTP 429 `TPM limit … 500000 tokens per minute` |
实测:最后一个请求结束后约 **44~60 秒**释放;同一账号连续请求 3~4 次即占满。
参考实现(示例插件 llm-adapter.ts)的做法是**每 10s 轮询后重试 chat**(最多 30
分钟),所以用户那边"看起来没事"。
## 我们的改法(内核只做通用机制,厂商知识留在插件)
1. **内核新增声明式重试策略** `source.retry`(纯数据:状态码列表 + 响应体子串 +
间隔 + 次数)。内核不认识任何厂商;命中即在同一网关请求内每 10s 重试一次。
流式只在**握手阶段**重试(首帧尚未下发,不会重复输出),非流式整体重试。
2. **插件为 codearts 声明**:400/429 + 命中 `TM.00001041` / `TPM limit` /
`并发会话数已达上限` → 10s × 最多 18 次(≈3 分钟)。
3. **适配器补两个会话细节**:`Connection: close`(keep-alive 空闲连接会被上游
计入并发会话)与稳定 `Session-Id`(同一账号恒定,避免每次请求新建会话)。
4. 错误文案补可行动提示(并发会话上限 / TPM 限流,约 1 分钟释放)。
## 验证(生产)
- 连续 6 次**流式**:6/6 成功(撞上限时自动等 10s 再试,客户端只感觉慢):
`+0s / +12s / +34s / +86s / +100s / +144s`
- 连续 5 次**非流式**:5/5 成功(`+0 / +11 / +20 / +28 / +38s`)
- 修复前:第 4 个请求起必然 TM.00001041 / empty stream。
> 注:`TM.00001005 无正常可用套餐` 属于账号权益问题(不重试、如实报出),
> 与限流要分开处理。
(cherry picked from commit 6454b094bc)
This commit is contained in:
@ -95,7 +95,14 @@ type Source struct {
|
||||
// ProxyURL routes this source's HTTP(S) traffic through an explicit proxy
|
||||
// (e.g. http://127.0.0.1:7890). Empty = direct connection. Needed for
|
||||
// upstreams behind geo/IP blocks that only respond through a proxy.
|
||||
ProxyURL string `yaml:"proxy_url" json:"proxy_url,omitempty"`
|
||||
ProxyURL string `yaml:"proxy_url" json:"proxy_url,omitempty"`
|
||||
// Retry is an OPTIONAL upstream retry policy. It exists for upstreams that
|
||||
// answer a business-level rejection with a status code the kernel would
|
||||
// otherwise surface to the client — e.g. a per-account concurrency cap that
|
||||
// clears by itself after tens of seconds. The kernel stays vendor-agnostic:
|
||||
// the matcher is pure DATA (status list + response-body substrings), so no
|
||||
// provider name ever appears in core.
|
||||
Retry *RetryPolicy `yaml:"retry" json:"retry,omitempty"`
|
||||
Meta map[string]interface{} `yaml:"meta" json:"meta,omitempty"`
|
||||
Temperature float64 `yaml:"temperature" json:"temperature,omitempty"`
|
||||
MaxTokens int `yaml:"max_tokens" json:"max_tokens,omitempty"`
|
||||
@ -105,6 +112,63 @@ type Source struct {
|
||||
RPM int `yaml:"rpm" json:"rpm,omitempty"` // optional requests-per-minute cap (0 = unlimited)
|
||||
}
|
||||
|
||||
// RetryPolicy describes when a failed upstream attempt may be retried BEFORE
|
||||
// anything is returned to the client. All conditions are data:
|
||||
//
|
||||
// OnStatus - retry only when the HTTP status is one of these
|
||||
// OnBodyContains - ... AND the response body contains one of these substrings
|
||||
// (empty = status alone decides)
|
||||
// Delay - wait between attempts ("10s" etc.); parsed on demand
|
||||
// MaxAttempts - TOTAL attempts including the first (1 or 0 = no retry)
|
||||
//
|
||||
// Scope: the retry happens inside one gateway request, so a stream is only
|
||||
// retried before its first chunk reaches the client (the kernel already holds
|
||||
// the first chunk back), and a non-stream call is retried as a whole.
|
||||
type RetryPolicy struct {
|
||||
OnStatus []int `yaml:"on_status,omitempty" json:"on_status,omitempty"`
|
||||
OnBodyContains []string `yaml:"on_body_contains,omitempty" json:"on_body_contains,omitempty"`
|
||||
Delay string `yaml:"delay,omitempty" json:"delay,omitempty"`
|
||||
MaxAttempts int `yaml:"max_attempts,omitempty" json:"max_attempts,omitempty"`
|
||||
}
|
||||
|
||||
// ShouldRetry reports whether one failed attempt matches this policy.
|
||||
func (r *RetryPolicy) ShouldRetry(status int, body string) bool {
|
||||
if r == nil || r.MaxAttempts <= 1 {
|
||||
return false
|
||||
}
|
||||
statusOK := len(r.OnStatus) == 0
|
||||
for _, s := range r.OnStatus {
|
||||
if s == status {
|
||||
statusOK = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !statusOK {
|
||||
return false
|
||||
}
|
||||
if len(r.OnBodyContains) == 0 {
|
||||
return true
|
||||
}
|
||||
for _, sub := range r.OnBodyContains {
|
||||
if sub != "" && strings.Contains(body, sub) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// DelayDuration is the configured pause between attempts (default 10s).
|
||||
func (r *RetryPolicy) DelayDuration() time.Duration {
|
||||
if r == nil || r.Delay == "" {
|
||||
return 10 * time.Second
|
||||
}
|
||||
d, err := time.ParseDuration(r.Delay)
|
||||
if err != nil || d < 0 {
|
||||
return 10 * time.Second
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// Load reads and validates a config file. When the file does not exist yet a
|
||||
// default config is generated at that path first (first-run bootstrap), so a
|
||||
// fresh binary just works: `llmsproxy -config /path/to/config.yaml`.
|
||||
|
||||
Reference in New Issue
Block a user