Compare commits

..

24 Commits

Author SHA1 Message Date
d81bfb1460 chore(version): 1.6.0
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-26 14:21:51 +08:00
a8cff57e24 feat(deploy): 部署前校验 master.key 可解封配置 + llmsproxy -show-secrets
密钥校验(deploy.sh)
- 新增 verify_master_key,在 build/替换任何文件之前执行。失败则二进制与
  配置分毫未动、服务不受影响(已负向验证:缺钥匙、错钥匙两种情况都挡住)
- 走真实的 -show-secrets 解密路径,而不是只检查钥匙文件格式——格式合法
  但内容不匹配(重新生成、恢复了错的备份、换机器)同样会被拒
- 钥匙来源与 config 包一致:LLMS_PROXY_MASTER_KEY 优先,否则
  dirname(runtime_file)/master.key
- 配置里没有密文时跳过并提示(首次加密场景)

-show-secrets
- llmsproxy -show-secrets -config <path>:把凭据打到 stdout 后退出
- 不启动任何东西、不写任何文件(已验证 mtime 不变)
- 加密往返无损:封存前后输出逐字节一致

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-26 14:21:44 +08:00
926b9f6565 feat: WebUI 改为依赖 /api/v1,UI 与 agent 共用一套 API 契约
- sources / sort / keys 三个页面的数据源从 /api/sources 切到 /api/v1/sources
  (写操作仍走 /api/sources:v1 是只读门面,不做变更)
- 编辑弹窗改用 /api/v1/sources/{name}?reveal=credentials(admin-only)取明文 key。
  这是必须的:表单要整体回传源,若不回填 key,改个端口就会把 key 清空。
- 遮蔽视图仍是默认,只有显式 reveal 才返回明文

端到端验证(真浏览器 + 临时实例,非仅 API 测试):
- sources/sort/keys 三页实际发出 GET /api/v1/sources,0 console error
- editSource('demo') → reveal=credentials,#s-key 与 #s-url 正确回填
- 写入往返:改 base_url /v1→/v2 后重开,key 仍在(未被清空)
- 落盘 api_key 明文残留 0、密文 1

测试:+1(reveal 必须 admin,否则任意 user key 可读全部凭据)
变异验证:reveal 去掉 admin 校验 → 403 断言变红

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-26 14:08:44 +08:00
ad28a924a5 feat: 密钥静态加密 + /api/v1 agent 管理 API
密钥加密(写侧封存 / 读侧解封)
- config.yaml 的 sources[].api_key、sources[].headers、keys[].key 落盘即
  AES-256-GCM 密文(enc:v1: 前缀),master.key 复用 runtime store 那把
- 内存里永远是明文:鉴权比对、API 返回新建 key、WebUI 编辑回填都不受影响
- 启动时一次性封存现存明文(幂等,已封存则不写盘);-check 不写文件
- UpsertSourceInYAML 增加 box 参数,新加的源不再以明文落盘
- 解密失败改为硬错误:原先 MustDecrypt 返回密文会被下次 Save 二次封存
  (实测:源 key 18→20、静默损坏),现在启动即失败且配置分毫不动

/api/v1:面向 agent 的管理 API(WebUI 零影响)
- GET /api/v1            机器可读索引,列出每个端点的方法/权限/用途
- GET /api/v1/overview   一次调用看全貌:源 + AUTO 链 + 密钥数 + 健康度
- GET /api/v1/health     仅健康快照
- GET /api/v1/models     按源分组的可路由模型清单
- GET /api/v1/sources[/{name}]  凭据遮蔽后的源
- GET /api/v1/auto       调度链与实时槽位状态
- GET /api/v1/keys       admin only,密钥元数据,绝不回显密钥本身
- 沿用同一套网关 key 鉴权;读端点任意角色,写仍需 admin

测试:15 个新用例(含负向:泄密、越权、写操作必须被拒)
变异验证:maskKey 不遮蔽→红、去掉 admin 校验→红

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-26 14:01:47 +08:00
432129243e chore(version): 1.5.9 2026-09-11 18:19:47 +08:00
c744ee151e feat(opencode): 透传 completion_tokens_details.reasoning_tokens 与上游 cost
回答「opencodego 的用量与费用透传呢」时逐字段核对上游产出,发现 usage 漏了
一项、费用整项丢失。

## 上游实际发什么(实测 opencode.ai/zen/go/v1)

  {
    "choices": [...],
    "usage": { "prompt_tokens": 37, "completion_tokens": 40, "total_tokens": 77,
               "prompt_cache_hit_tokens": 0, "prompt_cache_miss_tokens": 37,
               "prompt_tokens_details": {"cached_tokens": 0},
               "completion_tokens_details": {"reasoning_tokens": 40} },
    "cost": "0"
  }

cost 在**顶层**且是**字符串**。流式时还会单独发一帧:
{"choices":[],"cost":"0"}

## 此前丢了两样

1. completion_tokens_details.reasoning_tokens —— 输出里有多少是思考 token。
   没有它,客户端无法判断 completion_tokens 里多少是可见回答、多少是思考,
   而两者都按输出计费。
2. cost —— 唯一的费用信号,网关整个丢弃。Go 订阅是包月制恒为 "0",
   但 Zen 按量付费模型(以及未来的其它源)有信息量。

顺带修掉一处流式/非流式不一致:命中缓存时上游同时给
prompt_tokens_details.cached_tokens 和独立的 hit/miss,流式路径写成了 elseif,
只留 details,与非流式产出不同(只认独立字段的老客户端会看不到缓存)。

## 实现

- types.TokenUsage += CompletionTokensDetails;UnifiedResponse / UnifiedChunk += Cost
- opencodego/opencodezen 适配器映射两个字段;空 choices 帧改成 usage 与 cost
  都可带(早退只带 usage 会把同帧的 cost 丢干净 —— 新测试先抓到的就是这个)
- Gateway ChatCompletion / ChatChunk += cost,随终帧发(对齐上游的
  {"choices":[],"cost":"0"} 形态)
- Go 兜底 standardSSEChunk 同步支持(openai 系适配器不再漏 reasoning_tokens;
  纯 cost 帧不再被整体丢弃),新增 rawCostString 兼容字符串/数字两种形态

费用只做**搬运**:不解析、不换算、不汇总 —— 它是上游事实,且只有部分上游提供。

## 验证

经网关实测 gozen:deepseek-v4.1-flash,流式与非流式产出逐字段一致:
  prompt_tokens_details.cached_tokens=6784
  prompt_cache_hit_tokens=6784 / miss=148
  completion_tokens_details.reasoning_tokens=16
  cost="0"

测试:TestOpenCodeCostAndReasoningPassthrough(含「无数据不得凭空造字段」反例)、
TestOpenCodeStreamCacheFieldsMatchNonStream、TestTokenUsageMarshalsCompletionTokensDetails。
2026-09-11 18:19:40 +08:00
55f5d7a0f4 chore(version): 1.5.8 2026-09-11 16:54:36 +08:00
c9c09b2ba2 fix(opencode): 采纳客户端真实会话 id + 超窗消息不再被限流措辞封杀
两处都源于同一次排查:pi 到底有没有带会话标识、超窗为什么触发不了压缩。

## 1) 客户端会话 id:pi 一直在发,只是被配置关掉了

之前结论是「通用客户端不发会话 id」——只对了一半。pi 有会话 id,且能发:
pi-ai 的 createClient 在 compat.sendSessionAffinityHeaders 为真时,会把
平台会话 id(uuidv7,整个会话恒定)放到 x-session-affinity /
x-client-request-id / session_id 上。该开关默认 false,而 llmsproxy 的
provider 配置里没开,所以此前一直收不到。

现在网关按优先级采纳:x-session-affinity → x-session-id → session_id →
body 的 prompt_cache_key,并把值经 types.ChatRequest.ClientSession 传到
适配器 meta.client_session。适配器的会号种子优先级变为:
客户端会话 id > 首条 user 消息指纹 > 按源固定。

刻意不采纳 x-client-request-id:名字含 request,部分客户端每请求都换,
拿它当会话会让上游前缀缓存永不命中(pi 总会同时发 x-session-affinity,够用)。

实测:抓 127.0.0.1:8081 的真实 pi 请求,配置打开后收到
x-session-affinity = session_id = x-client-request-id = <子会话 uuid>。
上游缓存确为会话级隔离(同前缀、不同会号:A 冷→命中,B 首次仍为 0),
两个不同 header 值互不命中,反证网关确实采纳了客户端会话 id。

## 2) 超窗消息必须「干净」,否则被同链的限流措辞反向封杀

pi 的 isContextOverflow 先查 NON_OVERFLOW_PATTERNS(/rate limit/、
/too many requests/、Bedrock 前缀),命中就直接判为「非超窗」——**即使
消息里已经有 context_length_exceeded**,pi 也不会压缩重试。

而 AUTO 链的失败消息天生是多 tier 原因的拼接,超窗 tier(gozen 400
maximum context length)常与配额/限流 tier(429 token plan exhausted、
cooling、no free slot)同时出现。此前把 tier 明细原样拼在归一化标记后面,
等于让一条限流 tier 的措辞反过来封杀超窗识别。

现在超窗走独立的干净消息:
  context_length_exceeded: context window is full; reduce the length of
  the messages (gozen/deepseek-v4.1-flash)
只留超窗措辞 + 超窗源名,不带任何其它 tier 的文本。

测试:TestOverflowMessageSurvivesRateLimitedSiblingTier 用 pi 的完整判定
顺序(先 NON_OVERFLOW 后 OVERFLOW)断言同链限流 tier 不再封杀超窗识别;
TestClientSessionFromRequestHeaders / TestClientRequestIDIsNotUsedAsSession /
TestOpenCodePrefersClientSessionID 覆盖会话采纳与优先级。
2026-09-11 16:54:31 +08:00
091c9c961f chore(version): 1.5.7 2026-09-11 16:04:21 +08:00
39b48e556e feat(opencode): per-conversation session via first-user-message fingerprint
Follow-on to the session-stability fix. "Per source" already made the
prefix cache hit, but it puts every conversation into one upstream session.

Using the client's own session id is not possible: capturing real agent
traffic (tcpdump on 127.0.0.1:8081) shows generic OpenAI clients send NO
session identifier at all — no user / session_id / conversation_id /
metadata in the body, and no session header (only X-Stainless-* plus
User-Agent: pi). The x-opencode-session the Go endpoint asks for is an
OpenCode native-client concept that a generic client cannot forward.

Since history is replayed every turn, the FIRST user message is invariant
for the life of a conversation, so it is used as the conversation
fingerprint. The session becomes stable within a conversation and distinct
across conversations; requests with no user message fall back to per-source
stability.

Measured through the gateway (same 5.7k-token prompt): 2nd call
cached_tokens=5504, and an unrelated conversation gets its own session.

Test: TestOpenCodeSessionIsStableForCache covers same-conversation
stability, cross-conversation separation, per-request request ids and the
sessionless fallback.
2026-09-11 16:04:17 +08:00
7df413453b chore(version): 1.5.6 2026-09-11 15:42:26 +08:00
791d198f47 fix(opencode): make x-opencode-session stable so the upstream prefix cache can hit
The opencode adapters derived x-opencode-session from meta.timestamp, i.e. a
brand new session on every request. The upstream prefix cache is
session-scoped, so no request could ever hit it, and the cache fields the
endpoint does report (prompt_tokens_details.cached_tokens,
prompt_cache_hit_tokens/prompt_cache_miss_tokens) always came back 0/absent.

Measured against the live endpoint, same 6032-token prompt:

  fixed session id   -> 2nd call: hit 5888, miss 144
  rotating session id -> every call: hit 0, miss 6032

Fix: derive the session from the source name (stable), matching how
x-opencode-project is already derived. x-opencode-request stays unique per
request — it is only a request identifier, not part of the cache key.
Applied to both opencodego and opencodezen.

Through the gateway the same prompt now reports, on the 2nd call:
  details={'cached_tokens': 5888} hit=5888 miss=144      (non-streaming)
  prompt_tokens_details={'cached_tokens': 5888}          (streaming)

Test: TestOpenCodeSessionIsStableForCache asserts the session is stable
across requests for one source while the request id differs.
2026-09-11 15:42:22 +08:00
1f7d17feac chore(version): 1.5.5 2026-09-11 15:28:17 +08:00
d1a72cd23a fix(opencodego): inject empty reasoning_content on tool-calling turns
v1.5.4 stopped stripping reasoning_content, which fixes clients that send
it — but most agent clients (pi included) never store or replay their
reasoning, keeping only the tool call. OpenCode Go validates the field on
any assistant turn that carries tool_calls and rejects the whole request:

  400 invalid_request_error: The `reasoning_content` in the thinking mode
  must be passed back to the API.

Verified against the live endpoint that an EMPTY string satisfies the
check, so the adapter now fills in "" when a tool-calling assistant turn
has no reasoning_content. Nothing is fabricated: the reasoning shown to
the client is still exactly what the upstream returned for that turn.

Measured: with a tool_call + tool_result history and no reasoning_content,
all 25 configured Go models returned 400 before and all 25 answer
correctly now.

Test: TestOpenCodeGoVsZenReasoning also pins that a plain assistant turn
(no tool calls) must NOT gain the field.
2026-09-11 15:28:14 +08:00
19c4fc3a13 chore(version): 1.5.4 2026-09-11 15:14:14 +08:00
71e9040a45 feat(adapters): split opencode into opencodezen and opencodego
Zen (https://opencode.ai/zen/v1) and Go (https://opencode.ai/zen/go/v1)
are different services with different requirements, and one shared adapter
could not satisfy both.

The decisive difference is reasoning_content:

  * OpenCode Go runs thinking models and REQUIRES the assistant turn's
    reasoning_content to be echoed back. The shared adapter stripped it
    (msg.reasoning_content = nil), so every replay of a thinking turn
    failed with:
      400 invalid_request_error: The `reasoning_content` in the thinking
      mode must be passed back to the API.
    Reproduced directly: the same request with reasoning_content -> 200,
    without -> 400. That is why the Go tier never worked in an agent loop.

  * The Zen free pool must not receive it, so it keeps stripping.

Both adapters keep the earlier fixes they share (never drop an assistant
turn carrying tool_calls; send stream_options only when streaming; role
whitelist; multimodal strip) and the opencode client fingerprint headers —
the Go endpoint additionally REQUIRES x-opencode-session, which the
adapter already sends.

config: localzen -> opencodezen, gozen -> opencodego.
Verified: all 25 gozen models answer correctly through the gateway with a
thinking + tool_call + tool_result history (was 0/25 before), streaming
included; the Zen free models still pass.

Test: TestOpenCodeGoVsZenReasoning pins the Go-keeps / Zen-strips split.
2026-09-11 15:00:45 +08:00
0528941e24 fix(opencode): only send stream_options with stream:true
OpenCode Go (and other strict OpenAI-compatible upstreams) reject a
non-streaming request that carries stream_options with
"stream_options should be set along with stream". The adapter attached it
unconditionally, so every non-stream call through the opencode adapter
failed on those upstreams.

Verified against OpenCode Go: 25/25 configured models now pass a real
completion through the gateway (they previously 400'd).

Test: TestOpenCodeStreamOptionsOnlyWhenStreaming (absent when
non-streaming, include_usage present when streaming).
2026-09-11 13:13:05 +08:00
12af22fda6 chore(version): 1.5.3 2026-09-10 21:57:58 +08:00
9114468753 fix: empty-array content becomes invalid {} on every pass-through adapter; gemini/ollama drop tool calls
Three related forwarding defects found by auditing every adapter with a
tool-calling replay (assistant turn with content:[] + tool_calls).

1) content:[] -> content:{} (all 12 openai-adapter sources, plus
   deepseek/trae/sensenova/agentrouter/github/groq/kimicode/mistral)

   Lua adapters json.decode the request and re-encode it, and an empty Lua
   table is indistinguishable from an empty JSON array — the encoder emits
   {} for both. Agent clients serialise a tool-calling assistant turn with
   no text as content:[], so every pass-through adapter rewrote it to
   content:{} — not valid OpenAI (content is string|array|null). Verified
   against a live upstream: content:[] produced "400 invalid arguments"
   while content:"" was accepted.

   Fixed once at the decode boundary (types.ChatMessage.UnmarshalJSON):
   empty-array content normalises to "" and an empty tool_calls array is
   dropped, so every adapter — including future ones — sees a valid shape.

2) gemini dropped tool_calls and never emitted functionCall /
   functionResponse; the tool role also stayed as an invalid role inside
   contents and system was not moved to systemInstruction.

3) ollama copied only role/content, dropping tool_calls and the call
   attribution entirely (it needs tool_name, not tool_call_id).

Test: TestAdaptersPreserveToolCalls asserts, for every adapter, that the
call id (or function name where the wire format has no id), the function
name, the tool result and the trailing user turn all survive, plus a
negative control for plain text.
2026-09-10 21:57:54 +08:00
611e975456 fix(gateway): 上下文超窗错误归一化,客户端才能压缩重试
问题:上游返回上下文超窗时,客户端(pi)既不压缩也不重试,只看到一条
普通上游错误。链路两处叠加:

1. 措辞不在客户端识别列表里。pi 靠 @earendil-works/pi-ai 的
   OVERFLOW_PATTERNS(25 条正则)判断超窗,而 justworker 返回的是
   「请精简对话历史…(Context window is full…)」——与那 25 条一条都不匹配
   (最接近的 /context window exceeds limit/i 也不命中,因为措辞是 "is full"
   而非 "exceeds limit")。
2. AUTO 链路的每条 tier 错误按 80 字节 OneLine 截断,而 "Context window
   is full" 这类短诊断词常出现在尾部,正好被按字节切掉(直连路径是 160
   才保住)。

修法:
- 新增 overflowMarkers 识别超窗措辞(含中文写法),命中时把客户端可见
  错误归一化为 context_length_exceeded 前缀——它命中 pi 的
  /context[_ ]length[_ ]exceeded/i,超窗因此可被发现并触发压缩重试。
- AUTO 链路每条 tier 错误宽度 80→160,短诊断词不再被截断。

归一化只加前缀,原始诊断信息保留,便于定位是哪一层超窗。

测试:internal/gateway/overflow_err_test.go(5 例,含「标记必须命中 pi
正则」、非超窗不得误标、80 vs 160 宽度的回归对比)。
2026-09-10 20:38:34 +08:00
6733534282 chore(version): 1.5.2 2026-09-10 19:45:31 +08:00
499f0cac2f fix(opencode): never drop an assistant turn that carries tool_calls
An agent client (pi) serialises an assistant turn whose content is only
[thinking, toolCall] as content:[] with tool_calls. The multimodal-strip
pass treated an empty content array as 'nothing left, drop the whole
message' and discarded the tool_calls with it.

The next message is that call's tool result, so it arrived orphaned: the
model saw a result for a call it had never made and re-issued the same
call on every turn — an endless repeated-tool-call loop. Reproduced
against a capture sink: content:[] lost tool_calls, while content:"" and
content:null kept them.

Only messages with neither usable content NOR a tool call now get
dropped. Content that collapses to empty but still has tool_calls or a
tool_call_id is emitted as "" instead.

Test: TestOpenCodeKeepsToolCallWithEmptyContent (plus a negative control
that an image-only message without tool calls is still dropped).
2026-09-10 19:32:47 +08:00
7fb8f96b82 fix(gateway): AUTO scope grants all models — restrict to routing mode only
A key with scope=[AUTO] could previously:
1. request ANY concrete model id directly (hasScopeModel/checkModelScope
   treated AUTO as a wildcard)
2. see the full 56-model list on /v1/models (intersectModels considered
   AUTO as grant-everything)

AUTO now only authorizes the AUTO routing mode. Direct requests to a
specific model require an explicit scope entry.

Also carries agentrouter.lua WAF fingerprint headers (Origin/Referer/
X-Requested-With) already staged on this branch.

Tests: TestHasScopeModelWithSourcePrefix updated; full suite green.
2026-09-10 12:51:40 +08:00
6a34cd6f86 chore(version): 1.4.2 -> 1.5.0 2026-09-06 10:17:43 +08:00
19 changed files with 76 additions and 2382 deletions

View File

@ -24,7 +24,6 @@
### 强大的多租户调度能力
- **多密钥多租户**:支持无限密钥,每个密钥独立角色、模型范围、Token 配额、重置周期
- **按模型配额**:每把 key 的每个模型单独配 token + 请求数配额与重置周期(小时/周/月/自定义 N 小时);一个模型用满只影响该模型,同 key 其它模型照常;耗尽返 429 + `Retry-After` 并点名模型,admin key 永不受限
- **AUTO 智能调度**:基于优先级档位的分级调度,同优先级源自动轮询负载均衡,故障自动毫秒级故障转移
- **自愈冷却**:冷却上限 5 分钟,过半后放行 1 个探测请求,上游/额度恢复即刻回归轮询,无需等满冷却窗口
- **Token 配额管理**:精确到模型级别的 Token 配额控制,支持小时/周/月/自定义小时周期自动重置
@ -179,81 +178,6 @@ sources:
- 客户端用任意一个已授权的密钥明文作为 Bearer(`Authorization: Bearer <key>`)。
- 删除密钥即从运行时存储移除,立即失效。
#### 密钥用量配额
**配额按模型单独设置**:每个密钥的 `models[]` 里,每一条模型范围各自带一份
token 配额与请求数配额。一个模型用满只影响该模型,同一密钥的其它模型照常工作。
```yaml
keys:
- key: sk-gw-<hex>
role: user
name: agent-alice
models:
- model: deepseek-v4-flash
token_quota: 1000000 # 本周期内该 key 用这个模型的 token 预算
req_quota: 20000 # 本周期内的请求次数
period: nhour # "" | hour | week | month | nhour
hours: 6 # 仅 nhour
- model: AUTO
token_quota: 5000000 # AUTO 也是一条独立配额
period: day # ← 这种写法会被拒绝(词表只有 hour/week/month/nhour)
- model: kimi-k3 # 未列配额 = 无限
```
- **没有「整钥总配额」**:这是刻意的设计。整钥总额会让「把 A 模型的额度挪给
B 模型」变成一次全局重分配;按模型独立计费则每个模型各自可控,运维可以
看出哪个模型吃掉了预算。
- `period` 词表:空 = 永不过期(累计总量),`hour` / `week` / `month` = 固定窗口,
`nhour` + `hours` = 自定义小时数。**拼错的周期在写入时就被拒**,不会静默变成
永不过期。
- 配额严格按密钥隔离,且**同一密钥内按模型隔离**:A 密钥用满 `m1` 不会消耗
B 密钥的额度,同一密钥的 `m2` 也不受影响。
- 配额统计含聊天、流式、生图,跨重启从审计日志回放(保留 40 天,覆盖最长的
month 窗口)。
- 配额耗尽返回 **429 + `Retry-After`**(`rate_limit_exceeded`),消息里点名是哪个
模型用满了,客户端可等窗口重置后自动恢复;模型越权才是 403。
**admin 密钥永不受配额限制**(其 scope 上的配额也不强制),避免把管理员
锁在门外。
- 窗口用量按整点小时分桶统计,实际释放比配置窗口最多晚 1 小时(配额宁可晚释放
也不超发)。
- `PUT /api/keys/{key}` 提交 `models` 即同时提交它们的配额(配额就是 scope 的一部分,
不存在会与模型列表脱节的第二份预算)。显式 `0` = 解除该模型的限制。
##### 配额拒绝 vs 容量拒绝:两种「拒绝」含义不同
配额在**入口**检查,不占用任何上游槽位,因此是廉价拒绝;容量不足则要走
`busyWait` 有界等待后才返回 503。实测(容量 8、0.6s/请求、100 并发):
| 情况 | 状态码 | 延迟 | 是否打到上游 |
|---|---|---|---|
| 配额用尽(`req_quota` 50 → 100 并发) | `429 rate_limit_exceeded` + `Retry-After` | **19–21ms** | 否 |
| 容量打满(同一拓扑,未限配额) | `503 upstream_error` | ~2.6s(`busyWait`) | 超出部分否 |
客户端因此可以区分「等窗口重置」与「等上游腾出容量」:前者只需等待,后者
通常该降并发或换源。
##### 配额的开销
配额桶按 (密钥, 模型, 整点小时) 分桶保留 40 天,实测(AMD 7840HS):
- 窗口查询按窗口长度而非保留总量扫描:1h 窗口 49ns、24h 窗口 55ns、
30d 窗口 3.9μs(此前全扫保留总量,960 桶时 5.9μs)。
- 记录一条请求:283ns、3 分配(与引入配额前相同,分配来自 ring buffer)。
- 内存:生产形态(7 密钥 × 8 模型 × 2 源 × 满 40 天 retention)约 **3.7MB**。
按源 pin 的 `source::model` 桶**惰性创建**——只有当某条配额真的 pin 了
某个源时才维护,否则每条记录多写一份桶,在 20 密钥 × 8 模型 × 3 源下会
多占 18MB。
代价:配置 pinned 配额之前的历史用量无法事后按源拆分,所以 **pinned 配额的
首个窗口可能少算**。
```bash
# 配额耗尽时客户端看到(点名了具体模型)
HTTP/1.1 429 Too Many Requests
Retry-After: 2100
{"error":{"type":"rate_limit_exceeded","message":"token quota exceeded for \"deepseek-v4-flash\" (5000000/5000000)"}}
```
### 模型路由
`/v1/chat/completions` 的 `model` 解析顺序:
@ -420,10 +344,7 @@ Environment=MALLOC_ARENA_MAX=2
支持按时间范围导出 CSV;点击模型可生成 pin 到该模型的连接配置。
- **对话页**:流式/非流式调试。
- **密钥页**:创建/编辑网关 key,为每个 key 配模型范围(模型 + 源 + token 配额 + 周期),
管理员管理全部 key,用户只看到自己的 key。每个模型砖块显示自己的配额徽标
(如 `1.0K 77×·1h`,未设配额显示 `∞`),点开可编辑该模型的 token 配额、
请求数配额与重置周期 —— 配额按模型独立生效,一个用满不影响同一 key 的其它模型。
「我的密钥」页对用户展示本 key 的模型范围与各自配额。
管理员管理全部 key,用户只看到自己的 key。
- **优先级页**:拖拽积木配置 AUTO 链档位。
- **源页**:在线增删改上游源(API key 等敏感字段加密落盘)。
- **适配器页**:上传 / 删除 Lua 适配器脚本。

View File

@ -38,11 +38,6 @@ Extracted and independently evolved from the multi-source LLM adapter layer of
`reasoning_content`, `tool_calls`, `usage`).
- **Image generation**: `POST /v1/images/generations`, routed to models with
`kind: image`.
- **Per-model quota**: each key gives every model its own token and request caps
plus a reset period (hour/week/month/custom N hours). One model running out
affects only that model — the key's other models keep working. Exhaustion
answers 429 + `Retry-After` naming the model, so a client resumes when the
window rolls over; admin keys are never capped.
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
Anthropic/Gemini/Ollama are translated automatically.
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
@ -176,103 +171,11 @@ under the `keys` field of the runtime file (encrypted at rest):
delete the seed key.
- The WebUI **Keys page** creates/deletes keys. Each key has a role (`admin`
manages everything, `user` sees only its own key) and an optional **model
scope** (model + source + token quota + reset period). Each model brick shows
its own budget badge (e.g. `1.0K 77×·1h`, `∞` when uncapped); clicking it
edits that model's token quota, request quota and reset period. Quotas apply
per model, so one model running out never blocks the key's others. The
"My key" view shows a user its model scopes and their budgets.
scope** (model + source + token quota + reset period).
- Clients authenticate with any authorized key's plaintext as
`Authorization: Bearer <key>`.
- Deleting a key removes it from the store immediately.
#### Per-key usage quota
**Quotas are per model.** Each key's `models[]` list gives every model its own
token budget and request budget. One model running out affects only that model
— the key's other models keep working.
```yaml
keys:
- key: sk-gw-<hex>
role: user
name: agent-alice
models:
- model: deepseek-v4-flash
token_quota: 1000000 # this key's token budget for this model
req_quota: 20000 # requests within the window
period: nhour # "" | hour | week | month | nhour
hours: 6 # n-hour only
- model: AUTO
token_quota: 5000000 # AUTO is a quota entry like any other
period: hour
- model: kimi-k3 # no quota listed = unlimited
```
- **There is deliberately no key-wide total.** A key-wide cap would make
"move A's budget to B" a global reallocation; per-model budgets keep each
model independently controllable, so it stays visible which model is
actually consuming the spend.
- `period`: empty = never resets (lifetime total); `hour` / `week` / `month` =
fixed windows; `nhour` + `hours` = a custom hour count. **A misspelled
period is rejected at write time** rather than silently becoming a
never-resetting quota.
- Quotas are isolated per key *and* per model within a key: one key exhausting
`m1` never draws on another key's budget, and never blocks the same key's
`m2`.
- Usage counts chat, streaming and image requests, and survives a restart by
replaying the audit log (40 days retained, covering the longest `month`
window).
- An exhausted quota returns **429 + `Retry-After`**
(`rate_limit_exceeded`) and the message names the model that ran out, so a
client resumes when the window rolls over; a model the key may not use stays
403. **Admin keys are never capped** (quotas on their scopes are not
enforced either), so a cap can never lock the operator out.
- Buckets are whole unix hours, so a window frees up at most an hour late
(deliberately freeing late rather than overspending).
- `PUT /api/keys/{key}` submits quotas by submitting `models` — the caps are
part of the scope, so there is no second budget that can drift out of sync
with the model list. An explicit `0` lifts that model's cap.
##### Quota rejection vs capacity rejection
A quota is checked at the entry point and occupies no upstream slot, so it is a
cheap rejection. Running out of capacity instead waits out the bounded
`busyWait` before answering 503. Measured (capacity 8, 0.6 s/request, 100
concurrent clients):
| Case | Status | Latency | Reaches upstream |
|---|---|---|---|
| Quota spent (`req_quota` 50, 100 concurrent) | `429 rate_limit_exceeded` + `Retry-After` | **19–21 ms** | no |
| Capacity exhausted (same topology, no quota) | `503 upstream_error` | ~2.6 s (`busyWait`) | overflow: no |
So a client can tell "wait for the window to roll over" apart from "wait for
upstream capacity" — the first only needs patience, the second usually means
back off concurrency or switch sources.
##### What the quota costs
Buckets are kept per (key, model, whole unix hour) for 40 days. Measured on an
AMD 7840HS:
- A window query scans the window, not the whole retention: 49 ns for 1 h,
55 ns for 24 h, 3.9 µs for 30 d.
- Recording one request: 283 ns, 3 allocations (unchanged from before this
feature; the allocations come from the record ring buffer).
- Memory: the production shape (7 keys x 8 models x 2 sources at full 40-day
retention) costs about **3.7 MB**. The `source::model` bucket used by a
source-pinned quota is created **lazily** — it is maintained only once some
quota actually pins that source, because writing it on every record would
cost an extra 18 MB at 20 keys x 8 models x 3 sources.
The trade-off: usage recorded before a pinned quota was configured cannot be
split by source afterwards, so **a pinned quota may under-count its first
window**.
```
HTTP/1.1 429 Too Many Requests
Retry-After: 2100
{"error":{"type":"rate_limit_exceeded","message":"token quota exceeded for \"deepseek-v4-flash\" (5000000/5000000)"}}
```
### Model routing
`/v1/chat/completions` `model` resolution order:

View File

@ -1,12 +1,12 @@
{
"name": "modelrouter-gui",
"version": "1.7.2",
"version": "1.6.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "modelrouter-gui",
"version": "1.7.2",
"version": "1.6.0",
"license": "Proprietary",
"devDependencies": {
"electron": "^33.0.0",
@ -582,7 +582,7 @@
}
},
"node_modules/@peculiar/webcrypto": {
"version": "1.7.2",
"version": "1.7.1",
"resolved": "https://registry.npmjs.org/@peculiar/webcrypto/-/webcrypto-1.7.1.tgz",
"integrity": "sha512-ODOov0sGMJMf3jPonOkgGqPknTsu+DdQ7kD++gz8aI+aFMOMHFbWAA2taqXXVTdP+OTOQR/znGvSpmkeI0WTYQ==",
"dev": true,

View File

@ -1,6 +1,6 @@
{
"name": "modelrouter-gui",
"version": "1.7.2",
"version": "1.6.0",
"description": "ModelRouter Desktop — embedded ModelRouter core with tray",
"author": "ModelRouter",
"main": "main.js",

View File

@ -377,76 +377,15 @@ type GWKey struct {
Seed bool `yaml:"seed,omitempty" json:"seed,omitempty"` // true if migrated from config gateway_keys
}
// KeyQuota is retained only to carry a scope entry's caps through the admin
// API. Quotas are per model, never per key: there is deliberately no key-wide
// total, so exhausting one model's budget never blocks the others.
type KeyQuota struct {
TokenQuota int64 `json:"token_quota"`
ReqQuota int64 `json:"req_quota"`
Period string `json:"period"`
Hours int64 `json:"hours"`
}
// NormalizeRole defaults an empty role to "user", so a key can never end up in
// a state where no role means "neither admin nor user".
func NormalizeRole(role string) string {
if role == "admin" {
return "admin"
}
return "user"
}
// Validate rejects a quota configuration that could not work as written. A
// period is only meaningful when at least one cap is set, and a cap of zero
// means "unlimited" rather than "deny everything", so those are the only two
// things worth rejecting.
func (q KeyQuota) Validate() error {
if q.TokenQuota < 0 {
return fmt.Errorf("token_quota must be >= 0 (0 = unlimited)")
}
if q.ReqQuota < 0 {
return fmt.Errorf("req_quota must be >= 0 (0 = unlimited)")
}
if q.Hours < 0 {
return fmt.Errorf("hours must be >= 0")
}
if q.TokenQuota > 0 || q.ReqQuota > 0 {
if err := ValidatePeriod(q.Period, q.Hours); err != nil {
return err
}
}
return nil
}
// ValidatePeriod accepts the quota period vocabulary: "" (never resets),
// "hour", "week", "month", or "nhour" with hours >= 1. An unknown period is
// rejected rather than silently treated as "never resets", which would turn a
// typo into an all-time quota — the opposite of what the operator typed.
func ValidatePeriod(period string, hours int64) error {
switch period {
case "", "hour", "week", "month":
return nil
case "nhour":
if hours < 1 {
return fmt.Errorf("period %q needs hours >= 1", period)
}
return nil
}
return fmt.Errorf("period must be one of \"\", hour, week, month, nhour (got %q)", period)
}
// ModelScope is one allowed model for a key, or one AUTO scheduling slot.
// Its TokenQuota and ReqQuota cap THAT entry only, independently of every
// other entry on the same key: a model that runs out of budget stops being
// served while the key's other models keep working. TokenQuota 0 / ReqQuota 0
// = unlimited. Period "" = never resets; "hour"/"week"/"month" are fixed
// windows; "nhour" uses Hours.
// ModelScope is one allowed model for a key, or one AUTO scheduling slot,
// with an optional token quota and reset period. TokenQuota 0 = unlimited;
// Period "" = never resets; "hour"/"week"/"month" are fixed windows; "nhour"
// uses Hours as the window length in hours.
type ModelScope struct {
Model string `yaml:"model" json:"model"`
Source string `yaml:"source,omitempty" json:"source,omitempty"` // optional: pin to one upstream source; "" = any source
Tier int `yaml:"tier,omitempty" json:"tier,omitempty"`
TokenQuota int64 `yaml:"token_quota" json:"token_quota"`
ReqQuota int64 `yaml:"req_quota,omitempty" json:"req_quota,omitempty"`
Period string `yaml:"period,omitempty" json:"period,omitempty"`
Hours int64 `yaml:"hours,omitempty" json:"hours,omitempty"`
}
@ -465,7 +404,6 @@ func (m *ModelScope) UnmarshalJSON(b []byte) error {
Source string `json:"source"`
Tier int `json:"tier"`
TokenQuota int64 `json:"token_quota"`
ReqQuota int64 `json:"req_quota"`
Period string `json:"period"`
Hours int64 `json:"hours"`
}
@ -476,7 +414,6 @@ func (m *ModelScope) UnmarshalJSON(b []byte) error {
m.Source = o.Source
m.Tier = o.Tier
m.TokenQuota = o.TokenQuota
m.ReqQuota = o.ReqQuota
m.Period = o.Period
m.Hours = o.Hours
return nil

View File

@ -249,15 +249,10 @@ func (c *Core) FindKey(key string) (config.GWKey, bool) {
}
// CreateKey builds a new random gateway key and persists it to config.yaml.
// Quotas live on the model scope entries, so a new key's budget is whatever
// its scopes carry.
func (c *Core) CreateKey(name, role string, models []config.ModelScope, note string) (config.GWKey, error) {
c.mu.Lock()
defer c.mu.Unlock()
models = cleanScopes(models)
if err := ValidateScopeQuotas(models); err != nil {
return config.GWKey{}, err
}
key := make([]byte, 16)
if _, err := rand.Read(key); err != nil {
return config.GWKey{}, err
@ -270,7 +265,9 @@ func (c *Core) CreateKey(name, role string, models []config.ModelScope, note str
Note: note,
CreatedAt: time.Now().Unix(),
}
rec.Role = config.NormalizeRole(rec.Role)
if rec.Role == "" {
rec.Role = "user"
}
c.cfg.Keys = append(c.cfg.Keys, rec)
if err := c.saveConfig(); err != nil {
return config.GWKey{}, err
@ -278,16 +275,10 @@ func (c *Core) CreateKey(name, role string, models []config.ModelScope, note str
return rec, nil
}
// UpdateKey mutates a key's name/role/model scope and persists it. The scope
// entries carry their own quotas, so replacing the scope replaces the budgets.
// UpdateKey mutates a key's name/role/model scope and persists it.
func (c *Core) UpdateKey(key, name, role string, models []config.ModelScope, note string) (config.GWKey, error) {
c.mu.Lock()
defer c.mu.Unlock()
if models != nil {
if err := ValidateScopeQuotas(models); err != nil {
return config.GWKey{}, err
}
}
for i, k := range c.cfg.Keys {
if k.Key == key {
if name != "" {
@ -785,20 +776,3 @@ func (c *Core) Close() {
c.vm.Stop()
}
}
// ValidateScopeQuotas checks every scope entry's caps before they are stored.
// A typo in a period must be rejected at write time rather than silently
// becoming a never-resetting budget — the opposite of what was typed.
func ValidateScopeQuotas(entries []config.ModelScope) error {
for _, e := range entries {
if err := (config.KeyQuota{
TokenQuota: e.TokenQuota,
ReqQuota: e.ReqQuota,
Period: e.Period,
Hours: e.Hours,
}).Validate(); err != nil {
return fmt.Errorf("model %q: %w", e.Model, err)
}
}
return nil
}

View File

@ -116,8 +116,6 @@ func (g *Gateway) apiV1Routes(w http.ResponseWriter, r *http.Request) {
"note": k.Note,
"created_at": k.CreatedAt,
"seed": k.Seed,
// Quotas live on the scope entries (k.Models), echoed above;
// there is deliberately no key-wide total.
// The secret itself is never echoed. An operator that needs it
// already has it from creation time or from config.yaml.
"key_prefix": maskKey(k.Key),

View File

@ -8,7 +8,6 @@ import (
"log"
"net/http"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
@ -197,95 +196,35 @@ func intersectModels(models []string, allow []config.ModelScope) []string {
// A scope entry with model "AUTO" only allows requests where the effective
// model is AUTO (the routing mode). It does NOT grant access to specific model
// ids — that requires an explicit scope entry for the model.
// checkModelScope keeps its string-returning signature for callers that only
// need to know whether the request may proceed.
func (g *Gateway) checkModelScope(ctx context.Context, model string) string {
if q := g.checkQuota(ctx, model); q != nil {
return q.msg
}
return ""
}
// quotaRejection is a quota verdict: the message plus whether the client
// should retry. A spent quota is a rate limit (429 + Retry-After), not a
// permission failure (403): a client that sees 403 gives up on the key, while
// one that sees 429 with a retry hint waits and resumes when the window rolls
// over.
type quotaRejection struct {
msg string
retry int64 // seconds until the window resets; 0 = unknown
}
func (q *quotaRejection) Error() string { return q.msg }
// checkQuota validates the effective model against the key's model scope and
// that entry's quota. Returns nil when the request may proceed.
//
// Quotas are per scope entry, never key-wide: a model whose budget is spent
// is refused on its own while the key's other models keep working. The verdict
// carries the remaining seconds of the reset window so a spent budget answers
// 429 + Retry-After (come back when it rolls over) instead of 403 (which reads
// as "this key may never use this model" and makes clients give up).
func (g *Gateway) checkQuota(ctx context.Context, model string) *quotaRejection {
allow := g.allowedModels(ctx)
if allow == nil {
return nil
return ""
}
for _, sc := range allow {
if sc.Model != model {
continue
}
win := AutoPeriodSeconds(sc.Period, sc.Hours)
k := keyID(reqKey(ctx))
if sc.TokenQuota > 0 {
if used := g.scopeTokens(ctx, sc); used >= sc.TokenQuota {
return &quotaRejection{
msg: fmt.Sprintf("token quota exceeded for %q (%d/%d)", model, used, sc.TokenQuota),
retry: AutoSecondsToReset(sc.Period, sc.Hours),
}
used := g.scopeTokens(ctx, sc)
if used >= sc.TokenQuota {
return fmt.Sprintf("token quota exceeded for %q (%d/%d)", model, used, sc.TokenQuota)
}
}
if sc.ReqQuota > 0 {
if used := g.stats.KeyWindowReqs(k, win); used >= sc.ReqQuota {
return &quotaRejection{
msg: fmt.Sprintf("request quota exceeded for %q (%d/%d)", model, used, sc.ReqQuota),
retry: AutoSecondsToReset(sc.Period, sc.Hours),
}
}
}
return nil
return ""
}
return &quotaRejection{msg: fmt.Sprintf("model %q is not allowed for this key", model)}
return fmt.Sprintf("model %q is not allowed for this key", model)
}
// quotaWindowSuffix describes a quota's reset window for an error message, so
// a rejected caller can tell a permanent block from one that clears in an hour.
func quotaWindowSuffix(period string, hours int64) string {
switch {
case period == "hour":
return ", resets hourly"
case period == "week":
return ", resets weekly"
case period == "month":
return ", resets monthly"
case period == "nhour" && hours > 1:
return fmt.Sprintf(", resets every %dh", hours)
}
return ""
} // scopeTokens returns the tokens this key consumed on the scope entry's model
// within its reset window, isolated per key. For an AUTO entry the cap covers
// everything the key routed through AUTO; for a model entry it covers that
// model only.
//
// It reads the per-key hourly buckets rather than the key-blind model
// buckets, so one key's usage can never exhaust another's quota.
// scopeTokens returns the tokens a scope entry has consumed within its reset
// window (total for AUTO / per model otherwise).
func (g *Gateway) scopeTokens(ctx context.Context, sc config.ModelScope) int64 {
k := keyID(reqKey(ctx))
win := AutoPeriodSeconds(sc.Period, sc.Hours)
if sc.Model != "" && strings.EqualFold(sc.Model, "AUTO") {
return g.stats.KeyWindowTokens(k, win)
return g.stats.KeyTokens(k)
}
return g.stats.KeyWindowModelTokens(k, sc.Model, sc.Source, win)
win := AutoPeriodSeconds(sc.Period, sc.Hours)
return g.stats.WindowTokens(sc.Model, sc.Source, win)
}
// hasScopeModel reports whether a model (possibly with a "source-model" /
@ -309,26 +248,6 @@ func (g *Gateway) hasScopeModel(list []config.ModelScope, s string) bool {
return false
}
// writeScopeReject answers a model-scope or quota rejection. A spent quota is
// 429 (rate_limit_exceeded) with Retry-After, so a client waits and resumes
// after the reset; a model the key may not use stays 403 (model_not_allowed),
// because retrying cannot help.
func (g *Gateway) writeReject(w http.ResponseWriter, q *quotaRejection) {
if q.retry > 0 {
w.Header().Set("Retry-After", strconv.FormatInt(q.retry, 10))
writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", q.msg)
return
}
// no window to wait for: the cap is either permanent or key-wide with no
// period. "rate_limit_exceeded" still says "come back after the operator
// raises the cap", which 403 would not.
if strings.Contains(q.msg, "quota exceeded") {
writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", q.msg)
return
}
writeError(w, http.StatusForbidden, "model_not_allowed", q.msg)
}
func (g *Gateway) resolveByModel(model string) ([]*provider.Provider, string) {
if isAuto(model) {
return g.core.Registry().Resolve("AUTO"), ""
@ -391,8 +310,8 @@ func (g *Gateway) handleChat(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusServiceUnavailable, "no_provider", "no auto slot configured")
return
}
if q := g.checkQuota(r.Context(), "AUTO"); q != nil {
g.writeReject(w, q)
if msg := g.checkModelScope(r.Context(), "AUTO"); msg != "" {
writeError(w, http.StatusForbidden, "model_not_allowed", msg)
return
}
ctx := r.Context()
@ -445,8 +364,8 @@ func (g *Gateway) handleChat(w http.ResponseWriter, r *http.Request) {
if effective == "" {
effective = firstModel(cands[0])
}
if q := g.checkQuota(r.Context(), effective); q != nil {
g.writeReject(w, q)
if msg := g.checkModelScope(r.Context(), effective); msg != "" {
writeError(w, http.StatusForbidden, "model_not_allowed", msg)
return
}
ctx := r.Context()
@ -957,19 +876,6 @@ func (g *Gateway) pumpStream(w http.ResponseWriter, rec *Req, chunks <-chan type
// Final usage chunk (OpenAI standard: empty choices + usage before [DONE]).
// Prefer the upstream's exact usage if the stream carried it; fall back to
// the gateway's estimate otherwise.
// Write the upstream's exact numbers back onto the audit record. Without
// this the streamed path kept only the per-chunk byte estimate, so the same
// request recorded ~1.4x its real completion tokens (measured: upstream 100,
// audit 145) while the non-streaming path recorded 100. Two paths, two
// different numbers for one request is a reporting bug, not a rounding one.
if lastUsage != nil {
if lastUsage.Prompt > 0 {
rec.Prompt = int64(lastUsage.Prompt)
}
if lastUsage.Completion > 0 {
rec.Compl = int64(lastUsage.Completion)
}
}
var tut *types.TokenUsage
if lastUsage != nil {
tut = lastUsage
@ -1115,8 +1021,8 @@ func (g *Gateway) handleImage(w http.ResponseWriter, r *http.Request) {
}
if isAuto(model) {
if chain := g.core.AutoImageChain(); chain != nil && len(chain.Tiers) > 0 {
if q := g.checkQuota(r.Context(), "AUTO"); q != nil {
g.writeReject(w, q)
if msg := g.checkModelScope(r.Context(), "AUTO"); msg != "" {
writeError(w, http.StatusForbidden, "model_not_allowed", msg)
return
}
done := g.stats.Begin()
@ -1138,10 +1044,7 @@ func (g *Gateway) handleImage(w http.ResponseWriter, r *http.Request) {
}
rec.OK = true
rec.Status = http.StatusOK
// Image generation has no token concept. Recording len(ImageData)
// (the image COUNT) in completion_tokens mislabels image count as
// tokens and feeds it into the token totals; leave it 0.
rec.ImageCount = len(resp.ImageData)
rec.Compl = int64(len(resp.ImageData))
g.writeRec(rec)
writeJSON(w, http.StatusOK, types.ImageGenResponse{
Created: time.Now().Unix(),
@ -1165,8 +1068,8 @@ func (g *Gateway) handleImage(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusServiceUnavailable, "no_provider", "no image source configured")
return
}
if q := g.checkQuota(r.Context(), effectiveImageModel(model, cands)); q != nil {
g.writeReject(w, q)
if msg := g.checkModelScope(r.Context(), effectiveImageModel(model, cands)); msg != "" {
writeError(w, http.StatusForbidden, "model_not_allowed", msg)
return
}
done := g.stats.Begin()
@ -1190,8 +1093,7 @@ func (g *Gateway) handleImage(w http.ResponseWriter, r *http.Request) {
}
rec.OK = true
rec.Status = http.StatusOK
// Image generation has no token concept — see the AUTO path above.
rec.ImageCount = len(resp.ImageData)
rec.Compl = int64(len(resp.ImageData))
g.writeRec(rec)
writeJSON(w, http.StatusOK, types.ImageGenResponse{
Created: time.Now().Unix(),

View File

@ -1,253 +0,0 @@
package gateway
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"llmsproxy/internal/config"
"llmsproxy/internal/core"
)
// adminGateway builds a gateway whose admin key can call /api/keys.
func adminGateway(t *testing.T, keys ...config.GWKey) *Gateway {
t.Helper()
td := t.TempDir()
cfgPath := td + "/config.yaml"
if err := os.WriteFile(cfgPath, []byte("listen: :0"), 0o644); err != nil {
t.Fatal(err)
}
cfg := &config.Config{
Path: cfgPath,
AdapterDir: td + "/adapters",
RuntimeFile: td + "/runtime.json",
Keys: keys,
}
if err := cfg.ApplyDefaults(); err != nil {
t.Fatal(err)
}
c, err := core.NewFromConfig(cfg)
if err != nil {
t.Fatalf("core: %v", err)
}
t.Cleanup(c.Close)
g, err := New(c, []string{"sk-admin"})
if err != nil {
t.Fatalf("gateway: %v", err)
}
return g
}
func adminReq(t *testing.T, g *Gateway, method, path, body string) *httptest.ResponseRecorder {
t.Helper()
req, _ := http.NewRequest(method, path, strings.NewReader(body))
req.Header.Set("Authorization", "Bearer sk-admin")
req.Header.Set("Content-Type", "application/json")
rr := httptest.NewRecorder()
g.Handler().ServeHTTP(rr, req)
return rr
}
// keyRecord pulls one key's stored record out of the admin list.
func keyRecord(t *testing.T, g *Gateway, secret string) config.GWKey {
t.Helper()
rr := adminReq(t, g, "GET", "/api/keys", "")
if rr.Code != 200 {
t.Fatalf("GET /api/keys: %d %s", rr.Code, rr.Body.String())
}
var out struct {
Keys []config.GWKey `json:"keys"`
}
if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v (%s)", err, rr.Body.String())
}
for _, k := range out.Keys {
if k.Key == secret {
return k
}
}
t.Fatalf("key %q not found in %s", secret, rr.Body.String())
return config.GWKey{}
}
func scopeOf(t *testing.T, k config.GWKey, model string) config.ModelScope {
t.Helper()
for _, m := range k.Models {
if m.Model == model {
return m
}
}
t.Fatalf("scope %q not found in %+v", model, k.Models)
return config.ModelScope{}
}
// Quotas live on the scope entries, not on the key: creating a key with a
// budget means creating scopes that carry it, and they must survive a
// read-back (persisted, not just echoed).
func TestKeyAPICreatesPerModelQuota(t *testing.T) {
g := adminGateway(t, config.GWKey{Key: "sk-admin", Role: "admin"})
rr := adminReq(t, g, "POST", "/api/keys",
`{"name":"agent-x","role":"user","models":[{"model":"m1","token_quota":50000,"req_quota":200,"period":"nhour","hours":6}]}`)
if rr.Code != 200 {
t.Fatalf("create: %d %s", rr.Code, rr.Body.String())
}
var created struct {
Key config.GWKey `json:"key"`
}
if err := json.Unmarshal(rr.Body.Bytes(), &created); err != nil {
t.Fatalf("decode: %v", err)
}
sc := scopeOf(t, created.Key, "m1")
if sc.TokenQuota != 50000 || sc.ReqQuota != 200 || sc.Period != "nhour" || sc.Hours != 6 {
t.Fatalf("created scope did not carry the caps: %+v", sc)
}
back := scopeOf(t, keyRecord(t, g, created.Key.Key), "m1")
if back.TokenQuota != 50000 || back.ReqQuota != 200 || back.Period != "nhour" || back.Hours != 6 {
t.Errorf("read-back lost the caps: %+v", back)
}
}
// Two models on one key carry independent budgets.
func TestKeyAPIKeepsPerModelQuotaIndependent(t *testing.T) {
g := adminGateway(t, config.GWKey{Key: "sk-admin", Role: "admin"})
rr := adminReq(t, g, "POST", "/api/keys",
`{"name":"agent-y","role":"user","models":[{"model":"m1","token_quota":1000,"period":"hour"},{"model":"m2","token_quota":9999,"req_quota":7,"period":"week"}]}`)
if rr.Code != 200 {
t.Fatalf("create: %d %s", rr.Code, rr.Body.String())
}
var created struct {
Key config.GWKey `json:"key"`
}
_ = json.Unmarshal(rr.Body.Bytes(), &created)
back := keyRecord(t, g, created.Key.Key)
if a := scopeOf(t, back, "m1"); a.TokenQuota != 1000 || a.ReqQuota != 0 || a.Period != "hour" {
t.Errorf("m1 caps wrong: %+v", a)
}
if b := scopeOf(t, back, "m2"); b.TokenQuota != 9999 || b.ReqQuota != 7 || b.Period != "week" {
t.Errorf("m2 caps wrong: %+v", b)
}
}
// Sending 0 explicitly lifts that model's cap.
func TestKeyAPIUpdateZeroLiftsCap(t *testing.T) {
g := adminGateway(t, config.GWKey{Key: "sk-admin", Role: "admin"})
rr := adminReq(t, g, "POST", "/api/keys",
`{"name":"z","role":"user","models":[{"model":"m1","token_quota":1000,"period":"hour"}]}`)
if rr.Code != 200 {
t.Fatalf("create: %d %s", rr.Code, rr.Body.String())
}
var created struct {
Key config.GWKey `json:"key"`
}
_ = json.Unmarshal(rr.Body.Bytes(), &created)
secret := created.Key.Key
rr = adminReq(t, g, "PUT", "/api/keys/"+secret,
`{"models":[{"model":"m1","token_quota":0,"req_quota":0,"period":""}]}`)
if rr.Code != 200 {
t.Fatalf("lift: %d %s", rr.Code, rr.Body.String())
}
if back := scopeOf(t, keyRecord(t, g, secret), "m1"); back.TokenQuota != 0 || back.ReqQuota != 0 || back.Period != "" {
t.Errorf("caps not lifted: %+v", back)
}
}
// A misspelled period must be refused, not quietly turned into an all-time
// quota — which is the exact opposite of what the operator typed.
func TestKeyAPIRejectsBadPeriod(t *testing.T) {
g := adminGateway(t, config.GWKey{Key: "sk-admin", Role: "admin"})
rr := adminReq(t, g, "POST", "/api/keys",
`{"name":"bad","role":"user","models":[{"model":"m1","token_quota":1000,"period":"houre"}]}`)
if rr.Code != http.StatusBadRequest {
t.Fatalf("want 400 for a bad period, got %d %s", rr.Code, rr.Body.String())
}
if !strings.Contains(rr.Body.String(), "period") {
t.Errorf("error should name the period field: %s", rr.Body.String())
}
}
func TestKeyAPIRejectsNegativeQuota(t *testing.T) {
g := adminGateway(t, config.GWKey{Key: "sk-admin", Role: "admin"})
rr := adminReq(t, g, "POST", "/api/keys",
`{"name":"bad","role":"user","models":[{"model":"m1","token_quota":-5}]}`)
if rr.Code != http.StatusBadRequest {
t.Fatalf("want 400 for a negative quota, got %d %s", rr.Code, rr.Body.String())
}
}
// The rejection must say WHICH model is over budget, so an operator looking at
// a key with a dozen scopes can tell which one to raise.
func TestKeyAPIRejectionNamesTheModel(t *testing.T) {
g := adminGateway(t, config.GWKey{Key: "sk-admin", Role: "admin"})
rr := adminReq(t, g, "POST", "/api/keys",
`{"name":"bad","role":"user","models":[{"model":"m1","req_quota":-1}]}`)
if rr.Code != http.StatusBadRequest {
t.Fatalf("want 400, got %d", rr.Code)
}
if !strings.Contains(rr.Body.String(), "m1") {
t.Errorf("error should name the offending model: %s", rr.Body.String())
}
}
// A non-admin key must not be able to mint keys.
func TestKeyAPIQuotaIsAdminOnly(t *testing.T) {
g := adminGateway(t,
config.GWKey{Key: "sk-admin", Role: "admin"},
config.GWKey{Key: "sk-u", Role: "user", Models: []config.ModelScope{{Model: "m1", TokenQuota: 10, Period: "hour"}}},
)
req, _ := http.NewRequest("POST", "/api/keys",
strings.NewReader(`{"name":"x","role":"admin","models":[{"model":"m1"}]}`))
req.Header.Set("Authorization", "Bearer sk-u")
req.Header.Set("Content-Type", "application/json")
rr := httptest.NewRecorder()
g.Handler().ServeHTTP(rr, req)
if rr.Code != http.StatusForbidden {
t.Fatalf("non-admin create: want 403, got %d %s", rr.Code, rr.Body.String())
}
// /api/v1/keys exposes the per-model caps but never a secret
rr = adminReq(t, g, "GET", "/api/v1/keys", "")
if rr.Code != 200 {
t.Fatalf("GET /api/v1/keys: %d", rr.Code)
}
if strings.Contains(rr.Body.String(), "sk-u") {
t.Error("/api/v1/keys leaked a key secret")
}
if !strings.Contains(rr.Body.String(), `"token_quota":10`) {
t.Errorf("/api/v1/keys should expose the per-model caps: %s", rr.Body.String())
}
}
// A user must be able to see their own budget: /api/keys/me is the only key
// view a non-admin gets, so a cap missing from it is invisible to the very
// client it constrains.
func TestKeyMeExposesOwnQuota(t *testing.T) {
g := adminGateway(t,
config.GWKey{Key: "sk-admin", Role: "admin"},
config.GWKey{Key: "sk-u", Role: "user", Name: "agent",
Models: []config.ModelScope{{Model: "m1", TokenQuota: 123456, ReqQuota: 42, Period: "week"}}},
)
req, _ := http.NewRequest("GET", "/api/keys/me", nil)
req.Header.Set("Authorization", "Bearer sk-u")
rr := httptest.NewRecorder()
g.Handler().ServeHTTP(rr, req)
if rr.Code != 200 {
t.Fatalf("GET /api/keys/me: %d %s", rr.Code, rr.Body.String())
}
// the endpoint wraps the record: {"key": {...}}
var wrap struct {
Key config.GWKey `json:"key"`
}
if err := json.Unmarshal(rr.Body.Bytes(), &wrap); err != nil {
t.Fatalf("decode: %v (%s)", err, rr.Body.String())
}
sc := scopeOf(t, wrap.Key, "m1")
if sc.TokenQuota != 123456 || sc.ReqQuota != 42 || sc.Period != "week" {
t.Errorf("own quota not visible to the key's owner: %+v", sc)
}
}
var _ = fmt.Sprintf

View File

@ -1,60 +0,0 @@
package gateway
import (
"sync"
"testing"
"time"
)
// The per-key quota buckets are shared maps mutated on every recorded request
// and read on every quota check. Single-threaded unit tests cannot see a race
// there at all, so this drives 64 goroutines through record + read together
// under `go test -race`. It also opts one model into the lazily-created pinned
// bucket mid-flight, which is the path that mutates two bucket maps at once.
func TestConcurrentQuotaAccounting(t *testing.T) {
s := NewStats(0)
keys := []string{"k0", "k1", "k2", "k3", "k4", "k5", "k6"}
models := []string{"m0", "m1", "m2", "AUTO"}
var wg sync.WaitGroup
stop := time.Now().Add(2 * time.Second)
for w := 0; w < 64; w++ {
wg.Add(1)
go func(w int) {
defer wg.Done()
k := keys[w%len(keys)]
m := models[w%len(models)]
now := time.Now()
for time.Now().Before(stop) {
s.Record(Req{Time: now.UnixMilli(), Key: k, Model: m, Source: "src",
Prompt: 100, Compl: 50, OK: true, Status: 200, Type: "chat"})
_ = s.KeyWindowTokens(k, 3600)
_ = s.KeyWindowReqs(k, 3600)
_ = s.KeyWindowModelTokens(k, m, "", 3600)
_ = s.KeyWindowModelTokens(k, m, "src", 3600) // opts into the pinned bucket
}
}(w)
}
wg.Wait()
// Every recorded request must be accounted for exactly once.
s.mu.Lock()
var total int64
for _, hm := range s.keyHour {
for _, v := range hm {
total += v
}
}
recs := int64(0)
for _, r := range s.recs {
recs += r.Prompt + r.Compl
}
s.mu.Unlock()
if total == 0 {
t.Fatal("no tokens accounted")
}
t.Logf("accounted %d tokens across %d keys", total, len(keys))
if recs == 0 {
t.Error("no records retained; the concurrent writers lost work")
}
}

View File

@ -1,125 +0,0 @@
package gateway
import (
"math/rand"
"testing"
"time"
)
// Perf: the per-key quota bookkeeping runs on every recorded request and on
// every quota check, so its cost lands directly on the request path. These
// benchmarks exist to catch a regression that would make the feature
// expensive; the numbers that matter are relative to each other and to the
// pre-change baseline (Record was 275 ns/op with 3 allocs before this work).
func BenchmarkRecordQuotaBucket(b *testing.B) {
s := NewStats(0)
r := Req{Time: time.Now().UnixMilli(), Key: "keyABC", Model: "deepseek-v4-flash",
Source: "deepseek", Prompt: 800, Compl: 200, OK: true, Status: 200, Type: "chat"}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
s.Record(r)
}
}
// A key that has been busy for the whole 40-day retention.
func BenchmarkRecordQuotaBucketFullRetention(b *testing.B) {
s := NewStats(0)
nowH := time.Now().Unix() / 3600
for h := int64(0); h < quotaRetentionHours; h++ {
s.Record(Req{Time: (nowH - h) * 3600 * 1000, Key: "keyABC", Model: "m1",
Source: "deepseek", Prompt: 100, Compl: 100, OK: true, Status: 200})
}
r := Req{Time: time.Now().UnixMilli(), Key: "keyABC", Model: "m1", Source: "deepseek",
Prompt: 100, Compl: 100, OK: true, Status: 200, Type: "chat"}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
s.Record(r)
}
}
// The whole per-request quota check: total tokens + request count.
func BenchmarkQuotaCheckHourly(b *testing.B) {
s := NewStats(0)
for h := 0; h < 24; h++ {
s.Record(Req{Time: time.Now().Add(-time.Duration(h) * time.Hour).UnixMilli(),
Key: "keyABC", Model: "m1", Source: "deepseek", Prompt: 100, Compl: 100, OK: true, Status: 200})
}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
s.KeyWindowTokens("keyABC", 3600)
s.KeyWindowReqs("keyABC", 3600)
}
}
// A 30-day window has to visit 720 hour buckets. This is the worst case the
// quota feature puts on the request path.
func BenchmarkQuotaCheckMonthly(b *testing.B) {
s := NewStats(0)
nowH := time.Now().Unix() / 3600
for h := int64(0); h < 720; h++ {
s.Record(Req{Time: (nowH - h) * 3600 * 1000, Key: "keyABC", Model: "m1",
Prompt: 100, Compl: 100, OK: true, Status: 200})
}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
s.KeyWindowTokens("keyABC", 30*24*3600)
}
}
// TestSumBucketsMatchesFullScan is the correctness guard for the optimization
// that made the quota check proportional to the window instead of to the whole
// retention: the windowed sum must agree with a full scan for every bucket
// placement, or a quota would silently start letting traffic through. It
// caught a real off-by-one (floor instead of ceil on the first hour).
func TestSumBucketsMatchesFullScan(t *testing.T) {
const hour = 3600
base := int64(1_700_000_000) / hour * hour
rng := rand.New(rand.NewSource(7))
for trial := 0; trial < 400; trial++ {
hm := map[int64]int64{}
for i, n := 0, rng.Intn(40)+1; i < n; i++ {
h := base/hour - int64(rng.Intn(1200))
hm[h] += int64(rng.Intn(1000)) + 1
}
now := base + int64(rng.Intn(hour))
for _, sec := range []int64{3600, 2 * 3600, 6 * 3600, 24 * 3600, 7 * 24 * 3600, 30 * 24 * 3600} {
var want int64
cut := now - sec
for h, v := range hm {
if h*hour >= cut {
want += v
}
}
if got := sumBuckets(hm, now, sec); got != want {
t.Fatalf("trial %d sec=%d: sumBuckets=%d, full scan=%d", trial, sec, got, want)
}
}
}
}
// The pinned bucket is maintained only for (key, model) pairs whose quota pins
// a source. Doing it unconditionally doubled the bucket count — measured at
// 26.7 MB for 20 keys x 8 models x 3 sources at full retention, against a
// documented ~32 MB total memory budget — to serve a lookup nobody performs.
func TestPinnedBucketNotMaintainedUnlessWanted(t *testing.T) {
s := NewStats(100)
now := time.Now().UnixMilli()
for i := 0; i < 5; i++ {
s.Record(Req{Time: now, Key: "keyA", Model: "m1", Source: "srcX", Prompt: 100, Compl: 100, OK: true, Status: 200})
}
s.mu.Lock()
_, pinned := s.keyModelHour["keyA"]["srcX::m1"]
_, bare := s.keyModelHour["keyA"]["m1"]
s.mu.Unlock()
if pinned {
t.Error("a pinned bucket exists before any quota read the pin — memory regression")
}
if !bare {
t.Error("the bare-model bucket must always exist; an unpinned quota depends on it")
}
}

View File

@ -1,225 +0,0 @@
package gateway
import (
"testing"
"time"
"llmsproxy/internal/config"
)
// ---- data layer: per-key isolation ----
func TestKeyWindowTokensIsolatesKeys(t *testing.T) {
s := NewStats(100)
now := time.Now().UnixMilli()
s.Record(Req{Time: now, Key: "keyA", Model: "m1", Prompt: 500, Compl: 500, OK: true, Status: 200})
s.Record(Req{Time: now, Key: "keyB", Model: "m1", Prompt: 500, Compl: 500, OK: true, Status: 200})
if got := s.KeyWindowModelTokens("keyA", "m1", "", 3600); got != 1000 {
t.Errorf("keyA model tokens = %d, want 1000 (its own usage only)", got)
}
if got := s.KeyWindowModelTokens("keyB", "m1", "", 3600); got != 1000 {
t.Errorf("keyB model tokens = %d, want 1000", got)
}
// the key-blind bucket stays global on purpose: it backs the AUTO slot
// quota, which limits the whole gateway, not one key.
if got := s.WindowTokens("m1", "", 3600); got != 2000 {
t.Errorf("WindowTokens = %d, want 2000 (global per-model total must be unchanged)", got)
}
}
// A source pin scopes the cap to that one upstream. The pinned bucket is
// maintained lazily (it is only read by quotas that actually pin a source), so
// usage recorded before anything queried the pin is not attributed to it.
func TestKeyWindowTokensSeparatesSourcePin(t *testing.T) {
s := NewStats(100)
now := time.Now().UnixMilli()
s.Record(Req{Time: now, Key: "keyA", Model: "m1", Source: "srcX", Prompt: 100, Compl: 100, OK: true, Status: 200}) // 200 tok
s.Record(Req{Time: now, Key: "keyA", Model: "m1", Source: "srcY", Prompt: 200, Compl: 200, OK: true, Status: 200}) // 400 tok
// The unpinned bucket always counts — it is what a scope entry without a
// source pin reads. 200 + 400.
if got := s.KeyWindowModelTokens("keyA", "m1", "", 3600); got != 600 {
t.Errorf("unpinned model tokens = %d, want 600 (both sources)", got)
}
// Reading a pin opts this (key, model) into pinned accounting.
if got := s.KeyWindowModelTokens("keyA", "m1", "srcX", 3600); got != 0 {
t.Errorf("srcX pin before opt-in = %d, want 0 (the lazy bucket has not accrued yet)", got)
}
// From now on both pins accrue.
more := now + 1
s.Record(Req{Time: more, Key: "keyA", Model: "m1", Source: "srcX", Prompt: 10, Compl: 10, OK: true, Status: 200}) // 20 tok
s.Record(Req{Time: more, Key: "keyA", Model: "m1", Source: "srcY", Prompt: 30, Compl: 30, OK: true, Status: 200}) // 60 tok
if got := s.KeyWindowModelTokens("keyA", "m1", "srcX", 3600); got != 20 {
t.Errorf("srcX-pinned tokens after opt-in = %d, want 20", got)
}
if got := s.KeyWindowModelTokens("keyA", "m1", "srcY", 3600); got != 60 {
t.Errorf("srcY-pinned tokens = %d, want 60 (a pin read for one source must not blind the other)", got)
}
}
// This is the shape of every real chat record: Source is always populated.
// The unpinned bucket must still see the tokens, or a per-model quota without
// a source pin reads an empty bucket and never trips.
func TestKeyWindowModelTokensUnpinnedSeesSourcedTraffic(t *testing.T) {
s := NewStats(100)
s.Record(Req{Time: time.Now().UnixMilli(), Key: "keyA", Model: "m1", Source: "deepseek",
Prompt: 1000, Compl: 1000, OK: true, Status: 200})
if got := s.KeyWindowModelTokens("keyA", "m1", "", 3600); got != 2000 {
t.Errorf("unpinned tokens = %d, want 2000 — an unpinned per-model quota would never trip otherwise", got)
}
}
func TestKeyWindowRespectsResetWindow(t *testing.T) {
s := NewStats(100)
old := time.Now().Add(-30 * 24 * time.Hour).UnixMilli()
s.Record(Req{Time: old, Key: "keyA", Model: "m1", Prompt: 100, Compl: 100, OK: true, Status: 200})
s.Record(Req{Time: time.Now().UnixMilli(), Key: "keyA", Model: "m1", Prompt: 5, Compl: 5, OK: true, Status: 200})
if got := s.KeyWindowTokens("keyA", 3600); got != 10 {
t.Errorf("1h window = %d, want 10 (30-day-old usage must not count)", got)
}
if got := s.KeyWindowTokens("keyA", 0); got != 210 {
t.Errorf("all-time total = %d, want 210", got)
}
}
func TestKeyWindowReqsCountsEveryRequest(t *testing.T) {
s := NewStats(100)
now := time.Now().UnixMilli()
// a failed request and an image request both count: a client that loops on
// failures must still burn its request quota
s.Record(Req{Time: now, Key: "keyA", Model: "m1", Type: "chat", OK: true, Status: 200})
s.Record(Req{Time: now, Key: "keyA", Model: "m1", Type: "chat", OK: false, Status: 500})
s.Record(Req{Time: now, Key: "keyA", Model: "img", Type: "image", OK: true, Status: 200})
s.Record(Req{Time: now, Key: "keyB", Model: "m1", Type: "chat", OK: true, Status: 200})
if got := s.KeyWindowReqs("keyA", 3600); got != 3 {
t.Errorf("keyA reqs = %d, want 3 (chat ok + chat fail + image)", got)
}
if got := s.KeyWindowReqs("keyB", 3600); got != 1 {
t.Errorf("keyB reqs = %d, want 1", got)
}
}
func TestKeyWindowTokensZeroTokensNotCounted(t *testing.T) {
s := NewStats(100)
now := time.Now().UnixMilli()
// a request that reported no usage must not create a bucket entry
s.Record(Req{Time: now, Key: "keyA", Model: "", Type: "chat", OK: true, Status: 200})
if got := s.KeyWindowTokens("keyA", 3600); got != 0 {
t.Errorf("tokens = %d, want 0", got)
}
if got := s.KeyWindowReqs("keyA", 3600); got != 1 {
t.Errorf("reqs = %d, want 1 (request still happened)", got)
}
}
// ---- reset window arithmetic ----
func TestAutoSecondsToReset(t *testing.T) {
if got := AutoSecondsToReset("", 0); got != 0 {
t.Errorf("no period = %d, want 0 (never resets -> no retry hint)", got)
}
now := time.Now().Unix()
for _, tc := range []struct {
period string
hours int64
want int64
}{
{"hour", 0, 3600},
{"week", 0, 7 * 24 * 3600},
{"month", 0, 30 * 24 * 3600},
{"nhour", 6, 6 * 3600},
} {
got := AutoSecondsToReset(tc.period, tc.hours)
if got <= 0 || got > tc.want {
t.Errorf("AutoSecondsToReset(%q,%d) = %d, want in (0,%d]", tc.period, tc.hours, got, tc.want)
}
// must never exceed the window itself
if got < tc.want-now%3600 {
t.Logf("note: %q hint %ds < remaining-window %ds (rounds to hour boundary)", tc.period, got, tc.want-now%3600)
}
if got > tc.want {
t.Errorf("hint %d exceeds window %d", got, tc.want)
}
}
_ = now
}
func TestAutoSecondsToResetNeverExceedsWindow(t *testing.T) {
// "nhour" with a tiny window must not hand out a longer wait than the
// window itself (which would stall a client past its own reset)
for hours := int64(1); hours <= 48; hours++ {
got := AutoSecondsToReset("nhour", hours)
if got <= 0 || got > hours*3600 {
t.Errorf("nhour/%d = %d, want in (0,%d]", hours, got, hours*3600)
}
}
}
// ---- config validation ----
func TestKeyQuotaValidate(t *testing.T) {
cases := []struct {
name string
q config.KeyQuota
wantErr bool
}{
{"unlimited", config.KeyQuota{}, false},
{"tokens hourly", config.KeyQuota{TokenQuota: 1000, Period: "hour"}, false},
{"tokens n-hour", config.KeyQuota{TokenQuota: 1000, Period: "nhour", Hours: 6}, false},
{"reqs weekly", config.KeyQuota{ReqQuota: 100, Period: "week"}, false},
{"no period never resets", config.KeyQuota{TokenQuota: 1000}, false},
{"negative tokens", config.KeyQuota{TokenQuota: -1}, true},
{"negative reqs", config.KeyQuota{ReqQuota: -1}, true},
{"negative hours", config.KeyQuota{TokenQuota: 5, Hours: -1}, true},
{"typo period becomes all-time if accepted", config.KeyQuota{TokenQuota: 5, Period: "houre"}, true},
{"nhour without hours", config.KeyQuota{TokenQuota: 5, Period: "nhour"}, true},
{"nhour with 0 hours", config.KeyQuota{TokenQuota: 5, Period: "nhour", Hours: 0}, true},
{"junk period with only req quota", config.KeyQuota{ReqQuota: 5, Period: "daily"}, true},
{"junk period with no caps is irrelevant", config.KeyQuota{Period: "daily"}, false},
}
for _, tc := range cases {
err := tc.q.Validate()
if tc.wantErr && err == nil {
t.Errorf("%s: expected error, got nil", tc.name)
}
if !tc.wantErr && err != nil {
t.Errorf("%s: unexpected error: %v", tc.name, err)
}
}
}
func TestNormalizeRole(t *testing.T) {
if got := config.NormalizeRole(""); got != "user" {
t.Errorf("empty role = %q, want user", got)
}
if got := config.NormalizeRole("admin"); got != "admin" {
t.Errorf("admin role = %q, want admin", got)
}
if got := config.NormalizeRole("root"); got != "user" {
t.Errorf("unknown role = %q, want user (never escalate)", got)
}
}
// Retention must hold for the lazy pinned buckets too, or a long-lived key
// would grow without bound.
func TestPinnedBucketRespectsRetention(t *testing.T) {
s := NewStats(100)
_ = s.KeyWindowModelTokens("keyA", "m1", "srcX", 3600) // opt in
nowH := time.Now().Unix() / 3600
for h := int64(0); h < quotaRetentionHours+50; h++ {
s.Record(Req{Time: (nowH - h) * 3600 * 1000, Key: "keyA", Model: "m1", Source: "srcX",
Prompt: 10, Compl: 10, OK: true, Status: 200})
}
s.mu.Lock()
n := len(s.keyModelHour["keyA"]["srcX::m1"])
s.mu.Unlock()
if n > quotaRetentionHours {
t.Errorf("pinned bucket holds %d hours, want <= %d", n, quotaRetentionHours)
}
}

View File

@ -1,444 +0,0 @@
package gateway
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"llmsproxy/internal/config"
"llmsproxy/internal/core"
)
// quotaGateway builds a gateway with two user keys that differ only in their
// configured caps, against one mock upstream that reports 4 tokens per call.
func quotaGateway(t *testing.T, keyA, keyB config.GWKey) (*Gateway, *upstreamCtrl) {
t.Helper()
ctrl := &upstreamCtrl{}
up := upstream(t, ctrl)
t.Cleanup(up.Close)
td := t.TempDir()
cfgPath := td + "/config.yaml"
if err := os.WriteFile(cfgPath, []byte("listen: :0"), 0o644); err != nil {
t.Fatal(err)
}
cfg := &config.Config{
Path: cfgPath,
AdapterDir: td + "/adapters",
RuntimeFile: td + "/runtime.json",
Keys: []config.GWKey{keyA, keyB},
Sources: []config.Source{{
Name: "up", BaseURL: up.URL, Adapter: "openai",
Models: []config.Model{{ID: "m1", Priority: 100}},
}},
}
if err := cfg.ApplyDefaults(); err != nil {
t.Fatal(err)
}
c, err := core.NewFromConfig(cfg)
if err != nil {
t.Fatalf("core: %v", err)
}
t.Cleanup(c.Close)
g, err := New(c, []string{keyA.Key, keyB.Key})
if err != nil {
t.Fatalf("gateway: %v", err)
}
return g, ctrl
}
// chatAs issues a chat completion with a specific gateway key and returns the
// recorder plus the decoded error code.
func chatAs(t *testing.T, g *Gateway, key, model string) (*httptest.ResponseRecorder, string) {
t.Helper()
req, _ := http.NewRequest("POST", "/v1/chat/completions",
strings.NewReader(fmt.Sprintf(`{"model":%q,"messages":[{"role":"user","content":"hi"}]}`, model)))
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
rr := httptest.NewRecorder()
g.Handler().ServeHTTP(rr, req)
code := ""
if rr.Code != 200 {
var e struct {
Error struct {
Type string `json:"type"`
} `json:"error"`
}
_ = json.Unmarshal(rr.Body.Bytes(), &e)
code = e.Error.Type
}
return rr, code
}
// A key whose token budget is spent must be refused, and the refusal must be a
// 429 the client can retry after the window rolls over — not a 403 that reads
// as "this key may never use this model".
func TestKeyTokenQuotaBlocksWithRetryAfter(t *testing.T) {
g, _ := quotaGateway(t,
config.GWKey{Key: "sk-a", Role: "user",
Models: []config.ModelScope{{Model: "m1", TokenQuota: 8, Period: "hour"}}},
config.GWKey{Key: "sk-b", Role: "user", Name: "b"},
)
// 4 tokens per call, budget 8 -> the third call crosses it
for i := 1; i <= 2; i++ {
rr, _ := chatAs(t, g, "sk-a", "m1")
if rr.Code != 200 {
t.Fatalf("call %d: want 200, got %d (%s)", i, rr.Code, rr.Body.String())
}
}
rr, code := chatAs(t, g, "sk-a", "m1")
if rr.Code != http.StatusTooManyRequests {
t.Fatalf("third call: want 429, got %d (%s)", rr.Code, rr.Body.String())
}
if code != "rate_limit_exceeded" {
t.Errorf("error code = %q, want rate_limit_exceeded", code)
}
if ra := rr.Header().Get("Retry-After"); ra == "" {
t.Error("429 must carry Retry-After so a client knows when to come back")
} else if n := mustAtoi(t, ra); n <= 0 || n > 3600 {
t.Errorf("Retry-After = %q, want a positive value within the 1h window", ra)
}
if !strings.Contains(rr.Body.String(), "quota") {
t.Errorf("body should name the quota: %s", rr.Body.String())
}
}
// The quota is per key: exhausting one key's budget must not affect another
// key that is allowed the same model.
func TestKeyTokenQuotaIsIsolatedPerKey(t *testing.T) {
g, _ := quotaGateway(t,
config.GWKey{Key: "sk-a", Role: "user",
Models: []config.ModelScope{{Model: "m1", TokenQuota: 4, Period: "hour"}}},
config.GWKey{Key: "sk-b", Role: "user",
Models: []config.ModelScope{{Model: "m1", TokenQuota: 1000, Period: "hour"}}},
)
rr, _ := chatAs(t, g, "sk-a", "m1")
if rr.Code != 200 {
t.Fatalf("keyA first call: want 200, got %d", rr.Code)
}
if rr, code := chatAs(t, g, "sk-a", "m1"); rr.Code != http.StatusTooManyRequests {
t.Fatalf("keyA second call: want 429, got %d (%s)", rr.Code, code)
}
// keyB has its own budget and must still be served
if rr, _ := chatAs(t, g, "sk-b", "m1"); rr.Code != 200 {
t.Fatalf("keyB must be unaffected by keyA's exhausted quota, got %d", rr.Code)
}
}
// A per-model quota inside the scope must also be per key: one key burning a
// shared model budget must not lock the other key out.
func TestScopeModelTokenQuotaIsolatedPerKey(t *testing.T) {
g, _ := quotaGateway(t,
config.GWKey{Key: "sk-a", Role: "user", Models: []config.ModelScope{
{Model: "m1", TokenQuota: 4, Period: "hour"},
}},
config.GWKey{Key: "sk-b", Role: "user", Models: []config.ModelScope{
{Model: "m1", TokenQuota: 4, Period: "hour"},
}},
)
if rr, _ := chatAs(t, g, "sk-a", "m1"); rr.Code != 200 {
t.Fatalf("keyA first call: want 200, got %d", rr.Code)
}
if rr, _ := chatAs(t, g, "sk-a", "m1"); rr.Code != http.StatusTooManyRequests {
t.Fatalf("keyA second call: want 429 (its own model quota), got %d", rr.Code)
}
// keyB shares the model but not the counter — it must be served twice too
if rr, _ := chatAs(t, g, "sk-b", "m1"); rr.Code != 200 {
t.Fatalf("keyB first call: want 200, got %d", rr.Code)
}
if rr, _ := chatAs(t, g, "sk-b", "m1"); rr.Code != http.StatusTooManyRequests {
t.Fatalf("keyB second call: want 429, got %d", rr.Code)
}
}
// An admin key is never capped: a budget set on an admin key must not lock the
// operator out of the gateway they administer.
func TestAdminKeyIsNeverQuotaCapped(t *testing.T) {
g, _ := quotaGateway(t,
config.GWKey{Key: "sk-admin", Role: "admin",
Models: []config.ModelScope{{Model: "m1", TokenQuota: 1, ReqQuota: 1, Period: "hour"}}},
config.GWKey{Key: "sk-b", Role: "user"},
)
for i := 1; i <= 3; i++ {
if rr, _ := chatAs(t, g, "sk-admin", "m1"); rr.Code != 200 {
t.Fatalf("admin call %d: want 200 (admin keys are uncapped), got %d", i, rr.Code)
}
}
}
// A request quota caps sustained volume; RPM-style bursts are not its job but
// the count must still stop the key.
func TestKeyRequestQuotaBlocks(t *testing.T) {
g, ctrl := quotaGateway(t,
config.GWKey{Key: "sk-a", Role: "user",
Models: []config.ModelScope{{Model: "m1", ReqQuota: 2, Period: "hour"}}},
config.GWKey{Key: "sk-b", Role: "user"},
)
for i := 1; i <= 2; i++ {
if rr, _ := chatAs(t, g, "sk-a", "m1"); rr.Code != 200 {
t.Fatalf("call %d: want 200, got %d", i, rr.Code)
}
}
rr, code := chatAs(t, g, "sk-a", "m1")
if rr.Code != http.StatusTooManyRequests || code != "rate_limit_exceeded" {
t.Fatalf("third call: want 429/rate_limit_exceeded, got %d/%s", rr.Code, code)
}
if ctrl.hits != 2 {
t.Errorf("upstream saw %d calls, want 2 (a refused request must not reach upstream)", ctrl.hits)
}
}
// A model outside the scope is still 403, not 429: retrying cannot help.
func TestModelOutsideScopeStaysForbidden(t *testing.T) {
g, _ := quotaGateway(t,
config.GWKey{Key: "sk-a", Role: "user",
Models: []config.ModelScope{{Model: "other-model", TokenQuota: 1000, Period: "hour"}}},
config.GWKey{Key: "sk-b", Role: "user"},
)
rr, code := chatAs(t, g, "sk-a", "m1")
if rr.Code != http.StatusForbidden {
t.Fatalf("want 403, got %d (%s)", rr.Code, rr.Body.String())
}
if code != "model_not_allowed" {
t.Errorf("error code = %q, want model_not_allowed", code)
}
if rr.Header().Get("Retry-After") != "" {
t.Error("a 403 must not advertise a retry time")
}
}
// An unlimited key (no caps at all) is never rejected — the default config
// must keep working exactly as before.
func TestKeyWithoutCapsIsUnlimited(t *testing.T) {
g, _ := quotaGateway(t,
config.GWKey{Key: "sk-a", Role: "user", Models: []config.ModelScope{{Model: "m1"}}},
config.GWKey{Key: "sk-b", Role: "user"},
)
for i := 1; i <= 5; i++ {
if rr, _ := chatAs(t, g, "sk-a", "m1"); rr.Code != 200 {
t.Fatalf("call %d: want 200 (no caps = unlimited), got %d", i, rr.Code)
}
}
}
func mustAtoi(t *testing.T, s string) int64 {
t.Helper()
var n int64
if _, err := fmt.Sscanf(s, "%d", &n); err != nil {
t.Fatalf("Retry-After %q is not an integer: %v", s, err)
}
return n
}
// quotaCtx builds a request context carrying a gateway key, as the auth
// middleware would.
func quotaCtx(t *testing.T, g *Gateway, key string) context.Context {
t.Helper()
return withAuth(context.Background(), key, "user")
}
func nowMSOffset(sec int64) int64 {
return time.Now().Add(time.Duration(sec) * time.Second).UnixMilli()
}
// The key-wide quota and the AUTO slot quota are different limits with
// different scopes: the slot quota is gateway-wide (a shared upstream budget),
// the key quota belongs to one caller. When both are exhausted the caller must
// see the KEY quota, because that is the one it can act on — the slot quota
// would otherwise be reported as "no capacity", which reads like an outage.
func TestKeyQuotaWinsOverSlotQuota(t *testing.T) {
up := upstream(t, &upstreamCtrl{})
defer up.Close()
g := newQuotaGW(t, up.URL,
config.GWKey{Key: "sk-a", Role: "user",
Models: []config.ModelScope{{Model: "AUTO", TokenQuota: 4, Period: "hour"}}})
ctx := quotaCtx(t, g, "sk-a")
// exhaust the key first
g.stats.Record(Req{Time: time.Now().UnixMilli(), Key: keyID("sk-a"), Model: "m1",
Source: "up", Prompt: 100, Compl: 100, OK: true, Status: 200})
rr, code := chatAs(t, g, "sk-a", "AUTO")
if rr.Code != http.StatusTooManyRequests {
t.Fatalf("want 429 from the key quota, got %d (%s)", rr.Code, rr.Body.String())
}
if code != "rate_limit_exceeded" {
t.Errorf("code = %q, want rate_limit_exceeded", code)
}
_ = ctx
}
// A key with NO caps must never be blocked by the slot quota check reaching it
// through the scope path: scopeTokens on an uncapped key returns 0 usage and
// the limit check must treat that as "no cap", not "exhausted".
func TestUncappedKeyNeverBlockedByEmptyScope(t *testing.T) {
up := upstream(t, &upstreamCtrl{})
defer up.Close()
g := newQuotaGW(t, up.URL,
config.GWKey{Key: "sk-a", Role: "user", Models: []config.ModelScope{{Model: "AUTO"}}})
for i := 0; i < 5; i++ {
if rr, _ := chatAs(t, g, "sk-a", "AUTO"); rr.Code != 200 {
t.Fatalf("call %d: want 200 (AUTO scope with no quota), got %d (%s)", i, rr.Code, rr.Body.String())
}
}
}
// newQuotaGW builds a gateway with one mock upstream serving model m1 and the
// given keys.
func newQuotaGW(t *testing.T, upURL string, keys ...config.GWKey) *Gateway {
t.Helper()
td := t.TempDir()
cfgPath := filepath.Join(td, "config.yaml")
if err := os.WriteFile(cfgPath, []byte("listen: :0"), 0o644); err != nil {
t.Fatal(err)
}
cfg := &config.Config{
Path: cfgPath,
AdapterDir: filepath.Join(td, "adapters"),
RuntimeFile: filepath.Join(td, "runtime.json"),
Keys: keys,
Sources: []config.Source{{
Name: "up", BaseURL: upURL, Adapter: "openai",
Models: []config.Model{
{ID: "m1", Priority: 100},
{ID: "m2", Priority: 90},
},
}},
}
if err := cfg.ApplyDefaults(); err != nil {
t.Fatal(err)
}
c, err := core.NewFromConfig(cfg)
if err != nil {
t.Fatalf("core: %v", err)
}
t.Cleanup(c.Close)
secrets := make([]string, 0, len(keys))
for _, k := range keys {
secrets = append(secrets, k.Key)
}
g, err := New(c, secrets)
if err != nil {
t.Fatalf("gateway: %v", err)
}
return g
}
// The core of per-model quotas: a model that runs out of budget must stop
// being served on its own, while every other model on the SAME key keeps
// working. A key-wide total would fail this — it would block m2 because m1 was
// capped, which is exactly the coupling this design removes.
func TestOneModelsQuotaDoesNotBlockAnother(t *testing.T) {
up := upstream(t, &upstreamCtrl{})
defer up.Close()
td := t.TempDir()
cfgPath := td + "/config.yaml"
if err := os.WriteFile(cfgPath, []byte("listen: :0"), 0o644); err != nil {
t.Fatal(err)
}
cfg := &config.Config{
Path: cfgPath, AdapterDir: filepath.Join(td, "adapters"),
RuntimeFile: filepath.Join(td, "runtime.json"),
Keys: []config.GWKey{{
Key: "sk-a", Role: "user", Name: "two-models",
Models: []config.ModelScope{
{Model: "m1", TokenQuota: 4, Period: "hour"},
{Model: "m2", TokenQuota: 1000, Period: "hour"},
},
}},
Sources: []config.Source{{
Name: "up", BaseURL: up.URL, Adapter: "openai",
Models: []config.Model{
{ID: "m1", Priority: 100},
{ID: "m2", Priority: 90},
},
}},
}
if err := cfg.ApplyDefaults(); err != nil {
t.Fatal(err)
}
c, err := core.NewFromConfig(cfg)
if err != nil {
t.Fatalf("core: %v", err)
}
t.Cleanup(c.Close)
g, err := New(c, []string{"sk-a"})
if err != nil {
t.Fatalf("gateway: %v", err)
}
// Spend on m2 FIRST. Without this the two designs are
// indistinguishable: a key-wide counter and m1's own counter would both
// read 0 before m1 is used, so the test would pass either way (it did —
// see the commit that rewrote it).
for i := 1; i <= 3; i++ {
if rr, _ := chatAs(t, g, "sk-a", "m2"); rr.Code != 200 {
t.Fatalf("m2 priming call %d: want 200, got %d", i, rr.Code)
}
}
// m1's budget is 4 and each call costs 4, so the key-wide total (m2+m1) is
// already 12 when m1 starts: a key-wide cap would refuse m1 immediately.
if rr, _ := chatAs(t, g, "sk-a", "m1"); rr.Code != 200 {
t.Fatalf("m1 must be served on its own budget (m2's spend is not its problem), got %d (%s)",
rr.Code, rr.Body.String())
}
if rr, code := chatAs(t, g, "sk-a", "m1"); rr.Code != http.StatusTooManyRequests {
t.Fatalf("m1 second call: want 429, got %d (%s)", rr.Code, code)
}
// m2 must keep working
for i := 1; i <= 3; i++ {
if rr, _ := chatAs(t, g, "sk-a", "m2"); rr.Code != 200 {
t.Fatalf("m2 call %d must be served while m1 is capped, got %d (%s)", i, rr.Code, rr.Body.String())
}
}
// and the message must name m1, not the key
rr, _ := chatAs(t, g, "sk-a", "m1")
if !strings.Contains(rr.Body.String(), "m1") {
t.Errorf("rejection should name the capped model: %s", rr.Body.String())
}
}
// A model with no quota on it is never blocked by a sibling's cap, and an
// uncapped key is never blocked at all.
func TestUncappedModelNeverBlocked(t *testing.T) {
up := upstream(t, &upstreamCtrl{})
defer up.Close()
g := newQuotaGW(t, up.URL,
config.GWKey{Key: "sk-a", Role: "user", Models: []config.ModelScope{
{Model: "m1", TokenQuota: 1, Period: "hour"},
{Model: "m2"},
}})
if rr, _ := chatAs(t, g, "sk-a", "m1"); rr.Code != 200 {
t.Fatalf("m1 first: want 200, got %d", rr.Code)
}
if rr, _ := chatAs(t, g, "sk-a", "m1"); rr.Code != http.StatusTooManyRequests {
t.Fatalf("m1 second: want 429, got %d", rr.Code)
}
for i := 1; i <= 4; i++ {
if rr, _ := chatAs(t, g, "sk-a", "m2"); rr.Code != 200 {
t.Fatalf("m2 (uncapped) call %d: want 200, got %d", i, rr.Code)
}
}
}
// An admin key is never capped even when its scopes carry budgets: a cap that
// locked the operator out would be unrecoverable through the UI.
func TestAdminKeyScopesAreNotEnforced(t *testing.T) {
up := upstream(t, &upstreamCtrl{})
defer up.Close()
g := newQuotaGW(t, up.URL,
config.GWKey{Key: "sk-admin", Role: "admin", Models: []config.ModelScope{
{Model: "m1", TokenQuota: 1, ReqQuota: 1, Period: "hour"},
}})
for i := 1; i <= 4; i++ {
if rr, _ := chatAs(t, g, "sk-admin", "m1"); rr.Code != 200 {
t.Fatalf("admin call %d: want 200 (admin scopes are not enforced), got %d", i, rr.Code)
}
}
}

View File

@ -48,7 +48,13 @@ func (g *Gateway) handleKeysAPI(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid_request", "invalid json: "+err.Error())
return
}
body.Role = config.NormalizeRole(body.Role)
if body.Role == "" {
body.Role = "user"
}
if body.Role != "admin" && body.Role != "user" {
writeError(w, http.StatusBadRequest, "invalid_request", "role must be admin or user")
return
}
rec, err := g.core.CreateKey(body.Name, body.Role, body.Models, body.Note)
if err != nil {
writeError(w, http.StatusBadRequest, "key_error", err.Error())
@ -70,9 +76,6 @@ func (g *Gateway) handleKeysAPI(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid_request", "invalid json: "+err.Error())
return
}
// Each scope entry carries its own token/request caps, so replacing the
// scope replaces the budgets with it — there is no separate key-wide
// quota that could drift out of sync with the models.
rec, err := g.core.UpdateKey(path, body.Name, body.Role, body.Models, body.Note)
if err != nil {
writeError(w, http.StatusBadRequest, "key_error", err.Error())

View File

@ -48,10 +48,6 @@ type Req struct {
Status int `json:"status"`
// Err short error message
Err string `json:"error,omitempty"`
// ImageCount is how many images an image-generation request produced.
// Kept separate from Compl/Prompt: image generation has no token concept,
// so counting images as "completion tokens" would corrupt the token totals.
ImageCount int `json:"image_count,omitempty"`
}
// Stat aggregates counters for one dimension row.
@ -88,21 +84,6 @@ type Stats struct {
auditPath string
replayPartial bool // aggregates built from a bounded audit tail
modelHour map[string]map[int64]int64 // model -> unix-hour bucket -> tokens
// keyModelHour buckets the same tokens as modelHour but keyed by
// (gateway key id, model) so per-key model quotas are isolated from each
// other. modelHour stays key-blind on purpose: it backs the AUTO slot
// quota, which is a gateway-wide limit on a slot, not a per-key one.
keyModelHour map[string]map[string]map[int64]int64 // key -> model -> unix-hour -> tokens
// keyHour buckets a key's total tokens per unix hour, backing the
// whole-key quota (all models of one key share one budget).
keyHour map[string]map[int64]int64
// keyReqHour buckets a key's request count per unix hour, backing the
// key-wide request quota. Counted for every request type (chat, stream,
// image) including failed ones, so a failing client cannot loop for free.
keyReqHour map[string]map[int64]int64
// pinnedWanted records which (key, model) pairs have a quota that pins a
// source, so the pinned bucket is maintained only where it is read.
pinnedWanted map[string]bool
}
const hourSec = 3600
@ -135,18 +116,14 @@ func NewStats(maxRecords int) *Stats {
maxRecords = defaultRingSize
}
return &Stats{
byKey: map[string]*Stat{},
byModel: map[string]*Stat{},
bySrc: map[string]*Stat{},
byKeyModel: map[string]map[string]*Stat{},
byKeySrc: map[string]map[string]*Stat{},
byStatus: map[int]*Stat{},
modelHour: map[string]map[int64]int64{},
keyModelHour: map[string]map[string]map[int64]int64{},
keyHour: map[string]map[int64]int64{},
keyReqHour: map[string]map[int64]int64{},
pinnedWanted: map[string]bool{},
maxRecs: maxRecords,
byKey: map[string]*Stat{},
byModel: map[string]*Stat{},
bySrc: map[string]*Stat{},
byKeyModel: map[string]map[string]*Stat{},
byKeySrc: map[string]map[string]*Stat{},
byStatus: map[int]*Stat{},
modelHour: map[string]map[int64]int64{},
maxRecs: maxRecords,
}
}
@ -341,9 +318,6 @@ func (s *Stats) Record(r Req) {
// aggregateLocked folds r into every aggregate row and the quota window
// bucket. Caller must hold s.mu.
func (s *Stats) aggregateLocked(r Req) {
if r.Key != "" {
s.addKeyReqLocked(r.Key, (r.Time/1000)/hourSec, 1)
}
inc(s.byKey, r.Key, r)
if r.Model != "" {
inc(s.byModel, r.Model, r)
@ -388,12 +362,6 @@ func (s *Stats) aggregateLocked(r Req) {
s.modelHour[key] = hm
}
hm[h] += tok
// per-key buckets: same token split, but isolated per key so one
// key's quota cannot be drained by another key's traffic
if r.Key != "" {
s.addKeyTokenLocked(r.Key, r.Model, r.Source, h, tok)
s.addKeyHourLocked(r.Key, h, tok)
}
// retention: 24*40 = 960 hourly buckets ≈ 40 days of history (covers
// the longest "month" quota window)
if len(hm) > 24*40 {
@ -406,185 +374,6 @@ func (s *Stats) aggregateLocked(r Req) {
}
}
// quotaRetentionHours is how much hourly history the per-key quota buckets
// keep. It matches the modelHour retention (40 days) so the longest "month"
// window is fully covered after a restart, and it is deliberately applied per
// key so an idle key's buckets are reclaimed instead of pinning memory.
const quotaRetentionHours = 24 * 40
// addKeyTokenLocked adds tok to one key's (model[, source]) hourly bucket.
//
// The bare-model bucket is always written: it is what a scope entry without a
// source pin reads, and writing it is what makes that quota work at all (every
// recorded request carries a resolved source, so a source::model-only bucket
// would leave an unpinned quota reading an empty bucket forever).
//
// The pinned "source::model" bucket is written only for (key, model) pairs
// some scope entry actually pins a source for. Maintaining it unconditionally
// doubles the bucket count — measured at 26.7 MB for 20 keys x 8 models x 3
// sources at full retention, against a documented ~32 MB total budget — to
// serve a lookup that almost nobody performs.
func (s *Stats) addKeyTokenLocked(key, model, source string, h, tok int64) {
if model == "" {
return
}
byModel := s.keyModelHour[key]
if byModel == nil {
byModel = map[string]map[int64]int64{}
s.keyModelHour[key] = byModel
}
addBucket(byModel, model, h, tok)
if source == "" {
return
}
pinned := source + "::" + model
if pinned == model || !s.pinnedWanted[key+"\x00"+model] {
return
}
addBucket(byModel, pinned, h, tok)
}
// addBucket adds tok to one named hourly bucket and enforces retention.
func addBucket(byModel map[string]map[int64]int64, name string, h, tok int64) {
hm := byModel[name]
if hm == nil {
hm = map[int64]int64{}
byModel[name] = hm
}
hm[h] += tok
if len(hm) > quotaRetentionHours {
for k := range hm {
if k < h-quotaRetentionHours {
delete(hm, k)
}
}
}
}
// wantPinnedBuckets starts maintaining the pinned (source::model) bucket for a
// key's model, so a quota that pins a source counts traffic from the moment it
// is configured. Usage recorded before that cannot be attributed to a source
// after the fact, so a pinned quota may under-count its first window.
func (s *Stats) wantPinnedBuckets(key, model string) {
if key == "" || model == "" {
return
}
s.pinnedWanted[key+"\x00"+model] = true
}
// addKeyHourLocked adds tok to one key's all-model hourly total.
func (s *Stats) addKeyHourLocked(key string, h, tok int64) {
hm := s.keyHour[key]
if hm == nil {
hm = map[int64]int64{}
s.keyHour[key] = hm
}
hm[h] += tok
if len(hm) > quotaRetentionHours {
for k := range hm {
if k < h-quotaRetentionHours {
delete(hm, k)
}
}
}
}
// KeyWindowReqs returns how many requests one key issued within sec seconds;
// sec <= 0 means all retained history.
func (s *Stats) KeyWindowReqs(key string, sec int64) int64 {
s.mu.Lock()
defer s.mu.Unlock()
return sumBuckets(s.keyReqHour[key], time.Now().Unix(), sec)
}
// addKeyReqLocked adds n requests to a key's hourly count bucket.
func (s *Stats) addKeyReqLocked(key string, h, n int64) {
hm := s.keyReqHour[key]
if hm == nil {
hm = map[int64]int64{}
s.keyReqHour[key] = hm
}
hm[h] += n
if len(hm) > quotaRetentionHours {
for k := range hm {
if k < h-quotaRetentionHours {
delete(hm, k)
}
}
}
}
// sumBuckets totals the hourly buckets inside the window ending at `now`
// (unix seconds). sec <= 0 means "all retained history" (no reset).
//
// It only visits the hour buckets a window can actually cover. The naive full
// scan measured 5.9us/op at the full 40-day retention, and a per-key quota
// check runs on every request (2-3 of these), so the scan cost lands directly
// on the request path; visiting at most sec/3600 + 1 buckets makes it
// proportional to the window instead of to the retained history. The map keys
// are whole unix hours, so the covered range is exact rather than a sample.
func sumBuckets(hm map[int64]int64, now, sec int64) int64 {
if len(hm) == 0 {
return 0
}
if sec <= 0 {
var total int64
for _, v := range hm {
total += v
}
return total
}
nowHour := now / hourSec
// A bucket at hour H covers [H*3600, (H+1)*3600). The pre-existing
// WindowTokens counts a bucket when h*3600 >= now-sec, so the first hour
// that qualifies is ceil((now-sec)/3600) — NOT floor, which would pull in
// one bucket too many (caught by TestSumBucketsMatchesFullScan: a 30-day
// window reported 8878 vs 8649).
firstHour := (now - sec + hourSec - 1) / hourSec
if firstHour < 0 {
firstHour = 0
}
var total int64
for h := firstHour; h <= nowHour; h++ {
total += hm[h]
}
return total
}
// KeyWindowModelTokens returns the tokens one key consumed for one model
// (optionally pinned to a source) within sec seconds; sec <= 0 means all
// retained history. Unlike WindowTokens this is isolated per key, which is
// what a per-key model quota needs.
//
// A source pin matches the pinned bucket exactly. Without a pin the call
// totals the key's bare-model bucket, which counts traffic on every source —
// a quota on "this model" should not stop applying just because the request
// happened to be served by a different upstream.
func (s *Stats) KeyWindowModelTokens(key, model, source string, sec int64) int64 {
s.mu.Lock()
defer s.mu.Unlock()
byModel := s.keyModelHour[key]
if len(byModel) == 0 {
return 0
}
mk := model
if source != "" {
mk = source + "::" + model
// from now on this pair keeps a pinned bucket, so a quota configured
// against this source counts the traffic it is meant to cap
s.wantPinnedBuckets(key, model)
}
return sumBuckets(byModel[mk], time.Now().Unix(), sec)
}
// KeyWindowTokens returns one key's total tokens across all models within sec
// seconds; sec <= 0 means all retained history. It backs the whole-key quota.
func (s *Stats) KeyWindowTokens(key string, sec int64) int64 {
s.mu.Lock()
defer s.mu.Unlock()
return sumBuckets(s.keyHour[key], time.Now().Unix(), sec)
}
// rotateAuditLocked renames the audit file to <path>.<unix>.old once it
// exceeds auditRotateBytes and prunes old files beyond auditKeepOld, keeping
// the newest ones. Caller must hold s.mu.
@ -675,27 +464,6 @@ func AutoPeriodSeconds(period string, hours int64) int64 {
return 0
}
// AutoSecondsToReset returns how many seconds remain until the quota window
// rolls over, for the Retry-After header on a 429. Buckets are whole unix
// hours, so the value is rounded up to the next hour boundary and never
// exceeds one window. 0 means "unknown" (never resets, or no period set), in
// which case the caller should not advertise a retry time.
func AutoSecondsToReset(period string, hours int64) int64 {
sec := AutoPeriodSeconds(period, hours)
if sec <= 0 {
return 0
}
now := time.Now().Unix()
// window = the last sec seconds, bucketed by whole hours; the oldest
// bucket still inside the window expires at its own hour boundary
elapsed := now % hourSec
left := sec - elapsed
if left <= 0 || left > sec {
left = sec
}
return left
}
// WindowTokens returns the tokens consumed for one model (optionally pinned
// to a single source) within the window; sec <= 0 means all time. Buckets are
// whole unix hours, so a sliding window overcounts by up to one hour — an

View File

@ -1,231 +0,0 @@
package gateway
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"llmsproxy/internal/config"
)
// Token accounting must record TOKENS, not byte estimates and not image
// counts. These tests pin the two units bugs found on 2026-09-27:
//
// 1. streamed chat: rec.Compl was only ever the per-chunk byte estimate
// (len(content)/3), even when the upstream sent exact usage on its final
// frame. Measured on production traffic: the same model reported a
// streamed completion median 4-27x its non-streamed median. Upstream 100
// recorded as 145.
// 2. image generation: rec.Compl was set to len(resp.ImageData) — the image
// COUNT — and folded into the token totals.
//
// A fake upstream that reports FIXED numbers makes both mismatches
// unambiguous: any difference between what it reports and what the audit
// record holds is the gateway's doing.
// fixedUsageUpstream reports prompt=prompt completion=compl on both the
// streaming and the non-streaming path. The streaming path emits usage on its
// final frame before [DONE], exactly like OpenAI-compatible upstreams.
func fixedUsageUpstream(t *testing.T, prompt, compl int, images int) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var body map[string]interface{}
_ = json.NewDecoder(r.Body).Decode(&body)
stream, _ := body["stream"].(bool)
usage := fmt.Sprintf(`{"prompt_tokens":%d,"completion_tokens":%d,"total_tokens":%d}`,
prompt, compl, prompt+compl)
// Image generation wire: /v1/images/generations style body.
if strings.Contains(r.URL.Path, "image") {
w.Header().Set("Content-Type", "application/json")
var data []string
for i := 0; i < images; i++ {
data = append(data, `{"b64_json":"AAAA"}`)
}
fmt.Fprintf(w, `{"created":1,"data":[%s]}`, strings.Join(data, ","))
return
}
if !stream {
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"model":"m1","choices":[{"message":{"content":"pong pong pong"},
"finish_reason":"stop"}],"usage":%s}`, usage)
return
}
w.Header().Set("Content-Type", "text/event-stream")
w.WriteHeader(200)
fl, _ := w.(http.Flusher)
send := func(s string) {
fmt.Fprintf(w, "data: %s\n\n", s)
if fl != nil {
fl.Flush()
}
}
// Deliberately verbose payloads: a byte-estimating path records a
// number far from `compl`, so the test would catch it.
send(`{"model":"m1","choices":[{"delta":{"reasoning_content":"` + strings.Repeat("think ", 40) + `"}}]}`)
send(`{"model":"m1","choices":[{"delta":{"content":"` + strings.Repeat("answer ", 40) + `"}}]}`)
send(`{"model":"m1","choices":[{"delta":{},"finish_reason":"stop"}]}`)
send(`{"model":"m1","choices":[],"usage":` + usage + `}`)
send(`[DONE]`)
}))
}
// lastChatRecord returns the most recent audit record whose Type is one of the
// given types.
func lastChatRecord(t *testing.T, g *Gateway, types ...string) Req {
t.Helper()
recs := g.stats.AuditRecords(0, 0, "")
want := map[string]bool{}
for _, ty := range types {
want[ty] = true
}
for i := len(recs) - 1; i >= 0; i-- {
if want[recs[i].Type] {
return recs[i]
}
}
t.Fatalf("no audit record of type %v found (have %d records)", types, len(recs))
return Req{}
}
// The streamed path must record the upstream's exact usage, not a byte
// estimate. Before the fix this recorded 145 for an upstream-reported 100.
func TestStreamedChatRecordsUpstreamUsage(t *testing.T) {
const wantPrompt, wantCompl = 44, 100
up := fixedUsageUpstream(t, wantPrompt, wantCompl, 0)
defer up.Close()
g := newTestGateway(t, config.Source{
Name: "up", BaseURL: up.URL, Adapter: "openai",
Models: []config.Model{{ID: "m1", Priority: 100}},
})
rr := doReq(t, g, "POST", "/v1/chat/completions",
`{"model":"m1","messages":[{"role":"user","content":"hi"}],"stream":true}`)
if rr.Code != 200 {
t.Fatalf("stream status=%d body=%s", rr.Code, rr.Body.String())
}
rec := lastChatRecord(t, g, "stream")
if rec.Prompt != wantPrompt || rec.Compl != wantCompl {
t.Errorf("streamed record prompt=%d completion=%d, want %d/%d — the audit "+
"must carry the upstream's tokens, not a byte estimate",
rec.Prompt, rec.Compl, wantPrompt, wantCompl)
}
}
// Streaming and non-streaming must agree on the same request's tokens. This is
// the cross-path consistency check: the two paths previously recorded 145 and
// 100 for one identical upstream usage value.
func TestStreamedAndNonStreamedAgreeOnTokens(t *testing.T) {
const wantPrompt, wantCompl = 44, 100
up := fixedUsageUpstream(t, wantPrompt, wantCompl, 0)
defer up.Close()
g := newTestGateway(t, config.Source{
Name: "up", BaseURL: up.URL, Adapter: "openai",
Models: []config.Model{{ID: "m1", Priority: 100}},
})
if rr := doReq(t, g, "POST", "/v1/chat/completions",
`{"model":"m1","messages":[{"role":"user","content":"hi"}]}`); rr.Code != 200 {
t.Fatalf("non-stream status=%d body=%s", rr.Code, rr.Body.String())
}
if rr := doReq(t, g, "POST", "/v1/chat/completions",
`{"model":"m1","messages":[{"role":"user","content":"hi"}],"stream":true}`); rr.Code != 200 {
t.Fatalf("stream status=%d body=%s", rr.Code, rr.Body.String())
}
chat, stream := lastChatRecord(t, g, "chat"), lastChatRecord(t, g, "stream")
if chat.Prompt != stream.Prompt || chat.Compl != stream.Compl {
t.Errorf("chat=%d/%d stream=%d/%d — identical upstream usage must record "+
"identically on both paths", chat.Prompt, chat.Compl, stream.Prompt, stream.Compl)
}
}
// A fallback to the byte estimate is still correct WHEN the upstream reports no
// usage at all. Without this the fix could silently zero out every streamed
// record from upstreams that omit usage.
func TestStreamedChatFallsBackToEstimateWithoutUsage(t *testing.T) {
up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/event-stream")
w.WriteHeader(200)
fl, _ := w.(http.Flusher)
send := func(s string) {
fmt.Fprintf(w, "data: %s\n\n", s)
if fl != nil {
fl.Flush()
}
}
send(`{"model":"m1","choices":[{"delta":{"content":"` + strings.Repeat("abc ", 60) + `"}}]}`)
send(`{"model":"m1","choices":[{"delta":{},"finish_reason":"stop"}]}`)
send(`[DONE]`)
}))
defer up.Close()
g := newTestGateway(t, config.Source{
Name: "up", BaseURL: up.URL, Adapter: "openai",
Models: []config.Model{{ID: "m1", Priority: 100}},
})
if rr := doReq(t, g, "POST", "/v1/chat/completions",
`{"model":"m1","messages":[{"role":"user","content":"hi"}],"stream":true}`); rr.Code != 200 {
t.Fatalf("status=%d body=%s", rr.Code, rr.Body.String())
}
rec := lastChatRecord(t, g, "stream")
if rec.Compl <= 0 {
t.Errorf("completion=%d, want a positive byte-estimate fallback when the "+
"upstream reports no usage", rec.Compl)
}
}
// Image generation has no token concept: recording the image COUNT as
// completion tokens corrupted the token totals. The count is reported
// separately and must not leak into Prompt/Compl.
func TestImageGenerationDoesNotRecordTokens(t *testing.T) {
up := fixedUsageUpstream(t, 44, 100, 3)
defer up.Close()
src := config.Source{
Name: "up", BaseURL: up.URL, Adapter: "openai",
Models: []config.Model{{ID: "img-m", Priority: 100, Kind: "image"}},
}
g := newTestGateway(t, src)
rr := doReq(t, g, "POST", "/v1/images/generations",
`{"model":"img-m","prompt":"a cat","n":3}`)
if rr.Code != 200 {
t.Fatalf("image status=%d body=%s", rr.Code, rr.Body.String())
}
rec := lastChatRecord(t, g, "image")
if rec.Compl != 0 || rec.Prompt != 0 {
t.Errorf("image record prompt=%d completion=%d, want 0/0 — image count is "+
"not a token and must not enter the token totals", rec.Prompt, rec.Compl)
}
if rec.ImageCount != 3 {
t.Errorf("image_count=%d, want 3 (the count belongs in its own field)", rec.ImageCount)
}
// And the aggregate must see zero tokens for it.
if got := g.stats.WindowTokens("img-m", "", 3600); got != 0 {
t.Errorf("window tokens for an image model = %d, want 0", got)
}
}
// The audit JSON must carry image_count so the WebUI can show images rather
// than a bogus token figure.
func TestImageCountSurvivesAuditJSON(t *testing.T) {
b, err := json.Marshal(Req{Type: "image", ImageCount: 2, OK: true})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(b), `"image_count":2`) {
t.Errorf("audit JSON %s lacks image_count", b)
}
// Omitempty keeps chat records unchanged.
b2, _ := json.Marshal(Req{Type: "chat", Prompt: 1, Compl: 2, OK: true})
if strings.Contains(string(b2), "image_count") {
t.Errorf("chat record %s should omit image_count", b2)
}
}

View File

@ -818,10 +818,6 @@
kAnySrc: "任意源",
kQuotaB: "Token 配额",
kQuotaHintB: "0 / 留空 = 无限",
kReqQuota: "请求数配额",
kReqQuotaHint: "限制周期内的请求次数。0 / 留空 = 无限。",
kQuotaPerModelHint:
"配额按模型单独设置:创建后点「+ 添加模型」,逐个模型配 token 配额与周期。一个模型用满只影响该模型,同一密钥的其它模型照常。",
kPeriodB: "重置周期",
kPerNothing: "不限",
kPerHour: "每 小时",
@ -984,7 +980,6 @@
thErr: "失败",
thPrompt: "输入 Tokens",
thCompl: "输出 Tokens",
thImgs: "张",
thAvgLat: "平均延迟",
thMaxLat: "最长延迟",
thTime: "时间",
@ -1048,10 +1043,6 @@
kAnySrc: "any source",
kQuotaB: "Token quota",
kQuotaHintB: "0 / empty = unlimited",
kReqQuota: "Request quota",
kReqQuotaHint: "Caps requests per window. 0 / empty = unlimited.",
kQuotaPerModelHint:
"Quotas are per model: after creating the key, use \"Add model\" to give each model its own token budget and reset period. One model running out affects only that model; the key's other models keep working.",
kPeriodB: "Reset period",
kPerNothing: "Never",
kPerHour: "Every hour",
@ -1215,7 +1206,6 @@
thErr: "Err",
thPrompt: "Prompt Tokens",
thCompl: "Completion Tokens",
thImgs: "imgs",
thAvgLat: "Avg latency",
thMaxLat: "Max latency",
thTime: "Time",
@ -1702,20 +1692,7 @@
</div>`;
document.body.appendChild(wrap);
}
// closeTopModal removes the topmost dialog. Several dialogs share the id
// "modal-wrap" and more than one can be open at once (the seed-key notice
// sits on top of the keys page), so `$("#modal-wrap")` resolves to
// whichever comes FIRST in the document — not necessarily the one the
// user is looking at. Handlers that receive the clicked button resolve
// their own dialog with .closest(); handlers with no button reference
// use this, which matches what the user sees.
function closeTopModal() {
const all = [...document.querySelectorAll("#modal-wrap")];
const w = all[all.length - 1];
if (w) w.remove();
return w;
}
function downloadStatsCsv(from, to, btn) {
function downloadStatsCsv(from, to) {
const q = new URLSearchParams({
export: "csv",
from: String(Math.floor(from)),
@ -1723,7 +1700,8 @@
});
if (statsKeyF) q.set("key", statsKeyF);
location.href = "/api/stats?" + q.toString();
(btn ? btn.closest("#modal-wrap") : null) || closeTopModal();
const w = $("#modal-wrap");
if (w) w.remove();
}
// downloadStatsCsvFromForm reads the custom date range from the export
// modal's date inputs and forwards to downloadStatsCsv. (Previously this
@ -1741,11 +1719,12 @@
if (!isFinite(from) || from <= 0) return toast(t("expRange"));
downloadStatsCsv(from, to);
}
function downloadKeysCsv(btn) {
function downloadKeysCsv() {
location.href =
"/api/stats?export=keys-csv" +
(statsKeyF ? "&key=" + encodeURIComponent(statsKeyF) : "");
(btn ? btn.closest("#modal-wrap") : null) || closeTopModal();
const w = $("#modal-wrap");
if (w) w.remove();
}
async function paintStats() {
try {
@ -2195,7 +2174,7 @@
<td class="t-tag">${fmtTime(r.time)}</td>
<td class="num">${r.ok ? `<span class="tag tag-green">${r.status || 200}</span>` : `<span class="tag tag-red" title="${esc(r.error || "")}">${r.status || 500}</span>`}</td>
<td>${esc(keyNames[r.key] ? keyNames[r.key] + " · " + r.key : r.key)}</td><td class="t-tag">${esc(r.type)}</td><td>${esc(r.model)}</td><td>${esc(r.source || "")}</td>
<td class="num">${fmtTok(r.prompt_tokens)}</td><td class="num">${r.type === "image" ? fmtN(r.image_count || 0) + " " + t("thImgs") : fmtTok(r.completion_tokens)}</td>
<td class="num">${fmtTok(r.prompt_tokens)}</td><td class="num">${fmtTok(r.completion_tokens)}</td>
<td class="num">${cacheCell(r)}</td>
<td class="num">${fmtMs(r.latency_ms)}</td></tr>`;
}
@ -2939,10 +2918,9 @@
body: JSON.stringify(payload),
});
toast(t("toastSaved"));
const w = btn && btn.closest("#modal-wrap");
const w = $("#modal-wrap");
if (w) w.remove();
else if (window._modal) window._modal.remove();
else closeTopModal();
renderSources();
} catch (e) {
toast(tFmt("toastSaveFail", e.message));
@ -3099,11 +3077,11 @@
? `<div class="tbl-wrap"><table><tr><th>${t("tName")}</th><th>${t("tURL")}</th><th>${t("tAdapter")}</th><th>${t("tModels")}</th><th></th></tr>` +
tpls
.map(
(tpl) =>
`<tr><td><b>${esc(tpl.name)}</b></td><td>${esc(tpl.base_url)}</td><td>${esc(tpl.adapter)}</td>
<td><div class="src-models">${(tpl.models || []).map((m) => `<span class="tag tag-blue">${esc(m.id)}</span>`).join("")}</div></td>
<td><button class="ghost small" onclick="editTemplate(${JSON.stringify(tpl.name).replace(/["<>]/g, (c) => ({ '"': "&quot;", "<": "&lt;", ">": "&gt;" }[c]))})">${t("srcEdit")}</button>
<button class="danger small" onclick="delTemplate('${escAttr(tpl.name)}')">${t("srcDel")}</button></td></tr>`,
(t) =>
`<tr><td><b>${esc(t.name)}</b></td><td>${esc(t.base_url)}</td><td>${esc(t.adapter)}</td>
<td><div class="src-models">${(t.models || []).map((m) => `<span class="tag tag-blue">${esc(m.id)}</span>`).join("")}</div></td>
<td><button class="ghost small" onclick="editTemplate(${JSON.stringify(t.name).replace(/["<>]/g, (c) => ({ '"': "&quot;", "<": "&lt;", ">": "&gt;" }[c]))})">${t("srcEdit")}</button>
<button class="danger small" onclick="delTemplate('${escAttr(t.name)}')">${t("srcDel")}</button></td></tr>`,
)
.join("") +
"</table></div>"
@ -3203,9 +3181,8 @@
body: JSON.stringify({ name: nm, ...body }),
});
toast(t("toastSaved"));
const w = btn && btn.closest("#modal-wrap");
const w = $("#modal-wrap");
if (w) w.remove();
else closeTopModal();
openTemplateModal();
} catch (e) {
toast(e.message);
@ -3911,7 +3888,7 @@
</select>
<div id="a-hours-box" style="display:none"><label>${t("kPerNHint")}</label>
<input id="a-hours" type="number" min="1" step="1" value="24"></div>
<p><button onclick="scrAddFromForm(this)">${t("kSaveScope")}</button>
<p><button onclick="scrAddFromForm()">${t("kSaveScope")}</button>
<button class="ghost" onclick="this.closest('#modal-wrap').remove()">${t("mCancel")}</button></p>
</div>`;
document.body.appendChild(wrap);
@ -3921,7 +3898,7 @@
});
$("#a-model").focus();
}
function scrAddFromForm(btn) {
function scrAddFromForm() {
const raw = $("#a-model").value.trim();
if (!raw) {
toast(t("kName"));
@ -3944,14 +3921,14 @@
],
});
const li = sortState.lanes.length - 1;
if (btn) btn.closest("#modal-wrap")?.remove();
else closeTopModal();
const w = $("#modal-wrap");
if (w) w.remove();
paintSort([li]);
persistAuto()
.then(() => toast(t("sortHintSave")))
.catch((e) => toast(e.message));
}
function sortScopeSave(li, ji, btn) {
function sortScopeSave(li, ji) {
const it = sortState.lanes[li].models[ji];
if (!it) return;
let q = parseInt($("#q-quota").value);
@ -3960,7 +3937,8 @@
let h = parseInt($("#q-hours").value);
if (isNaN(h) || h < 1) h = 1;
it.meta = { quota: q, period: p, hours: p || q ? h : 0 };
(btn ? btn.closest("#modal-wrap") : null) || closeTopModal();
const w = $("#modal-wrap");
if (w) w.remove();
paintSort([li]);
persistAuto()
.then(() => toast(t("sortHintSave")))
@ -4319,14 +4297,13 @@
const comb = scopeComb(m);
const src = normSrc(m.source);
const attrs = `data-key="${escAttr(key)}" data-model="${escAttr(comb)}"
data-quota="${m.token_quota || 0}" data-reqquota="${m.req_quota || 0}"
data-period="${escAttr(m.period || "")}" data-hours="${m.hours || 0}"`;
data-quota="${m.token_quota || 0}" data-period="${escAttr(m.period || "")}" data-hours="${m.hours || 0}"`;
return `<span class="mb" draggable="true" ${attrs} title="${escAttr(t("kBrickH"))}"
onclick="scopeEdit('${escAttr(key)}','${escAttr(comb)}')"
oncontextmenu="scopeCtx(event,'${escAttr(key)}','${escAttr(comb)}')">
<span class="mb-ico"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" width="10" height="10"><circle cx="12" cy="12" r="10"/></svg></span>
<span class="mb-name">${esc(m.model)}${src ? `<em class="mb-src">${esc(src)}</em>` : ""}</span>
<span class="mb-quota">${esc(scopeQuotaBadge(m))}</span>
<span class="mb-quota">${esc(quantBadge(m.token_quota, m.period, m.hours))}</span>
<span class="copy-b" title="${escAttr(t("kCopyB"))}" onclick="event.stopPropagation();scopeDup('${escAttr(key)}','${escAttr(comb)}')"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" width="12" height="12"><rect x="9" y="9" width="13" height="13" rx="2" ry="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg></span>
<span class="mb-x" title="${escAttr(t("kDelB"))}" onclick="event.stopPropagation();scopeRm('${escAttr(key)}','${escAttr(comb)}')"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" width="12" height="12"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg></span>
</span>`;
@ -4358,16 +4335,6 @@
if (p === "nhour") return "·" + Math.max(1, h) + "h";
return "";
}
// scopeQuotaBadge shows a scope entry's budgets: token quota and, when
// set, the request count. They are per model — a spent budget blocks
// only that model, not the whole key.
function scopeQuotaBadge(m) {
const parts = [];
if (+m.token_quota > 0) parts.push(fmtQuota(m.token_quota));
if (+m.req_quota > 0) parts.push(fmtQuota(m.req_quota) + "\u00d7");
if (!parts.length) return "\u221e";
return parts.join(" ") + periodText(m.period, m.hours);
}
function quantBadge(quota, period, hours) {
quota = +quota || 0;
period = period || "";
@ -4382,21 +4349,16 @@
model: comb[0],
source: src || undefined,
token_quota: parseInt(b.dataset.quota) || 0,
req_quota: parseInt(b.dataset.reqquota) || 0,
period: b.dataset.period || "",
hours: parseInt(b.dataset.hours) || 0,
};
});
}
async function putScope(key, scopes) {
// Each scope entry carries its own quotas, so the whole budget travels
// with the models it applies to. There is no separate key-wide total
// that could drift out of sync with the model list.
const body = { models: scopes };
await api("/api/keys/" + encodeURIComponent(key), {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
body: JSON.stringify({ models: scopes }),
});
}
async function scopePush(key) {
@ -4499,9 +4461,6 @@
<label>${t("kQuotaB")} <span class="muted">${t("kQuotaHintB")}</span></label>
<input id="sc-quota" type="number" min="0" step="1"
placeholder="${escAttr(t("kQuotaHintB"))}" value="${sc.token_quota ? sc.token_quota : ""}">
<label>${t("kReqQuota")} <span class="muted">${t("kReqQuotaHint")}</span></label>
<input id="sc-reqs" type="number" min="0" step="1"
placeholder="${escAttr(t("kReqQuotaHint"))}" value="${sc.req_quota || ""}">
<label>${t("kPeriodB")}</label>
<select id="sc-period">
<option value="" ${!sc.period ? "selected" : ""}>${t("kPerNothing")}</option>
@ -4535,8 +4494,6 @@
const parts = splitCombKey(comb);
let q = parseInt($("#sc-quota").value);
if (isNaN(q) || q < 0) q = 0;
let rq = parseInt($("#sc-reqs").value);
if (isNaN(rq) || rq < 0) rq = 0;
let hours = parseInt($("#sc-hours").value);
if (isNaN(hours) || hours < 1) hours = 1;
const period = $("#sc-period").value;
@ -4545,7 +4502,6 @@
model: parts.model,
source: parts.source || undefined,
token_quota: q,
req_quota: rq,
period,
hours,
};
@ -4555,9 +4511,8 @@
if (btn) btn.disabled = true;
try {
await putScope(key, scopes);
const w = btn && btn.closest("#modal-wrap");
const w = $("#modal-wrap");
if (w) w.remove();
else closeTopModal();
toast(t("kSaved"));
await loadKeys();
} catch (e) {
@ -4706,7 +4661,6 @@
<option value="user">${t("kRoleUser")}</option>
<option value="admin">${t("kRoleAdmin")}</option>
</select>
<p class="muted" style="font-size:12px">${t("kQuotaPerModelHint")}</p>
<label>${t("kNote")}</label>
<input id="kc-note">
<p><button onclick="createKey(this)">${t("kCreateBtn")}</button>
@ -4738,12 +4692,8 @@
if (btn) btn.disabled = false;
return;
}
// Close THIS dialog (resolved from the clicked button), not whichever
// #modal-wrap comes first: the seed-key notice may already be open and
// would otherwise be the one that gets removed.
const w = btn ? btn.closest("#modal-wrap") : null;
const w = $("#modal-wrap");
if (w) w.remove();
else closeTopModal();
$("#k-newbox").innerHTML = `
<div class="key-canvas" style="background:var(--card2)">
<div class="kc-head">

View File

@ -347,60 +347,3 @@ func cssBlock(src, selector string) (string, bool) {
}
return src[i : i+j+1], true
}
// The WebUI ships two locale objects (`zh` and `en`) that every rendered string
// goes through. A key added to only one of them does not error: t() falls back
// to `t("key") || "literal"` in some call sites and to the bare key string in
// others, so the user sees either a hardcoded local string or a raw key name.
// Nothing in the Go test suite noticed — this was found by hand while adding
// thImgs. Pin the key sets so a one-sided edit fails here instead of shipping.
func TestUILocaleKeyParity(t *testing.T) {
src := uiSource(t)
iZh := strings.Index(src, "zh: {")
iEn := strings.Index(src, "en: {")
if iZh < 0 || iEn < 0 || iEn < iZh {
t.Fatalf("locale blocks not found (zh=%d en=%d)", iZh, iEn)
}
zh := src[iZh:iEn]
en := src[iEn:]
// End the EN block at its closing brace, so keys from later objects
// (config templates etc.) do not pollute the comparison.
if end := strings.Index(en, "\n },"); end > 0 {
en = en[:end]
}
keyRe := regexp.MustCompile(`(?m)^\s*([A-Za-z_][A-Za-z0-9_]*)\s*:`)
collect := func(seg string) map[string]bool {
m := map[string]bool{}
for _, g := range keyRe.FindAllStringSubmatch(seg, -1) {
m[g[1]] = true
}
return m
}
zk, ek := collect(zh), collect(en)
// The locale markers themselves are the two block headers.
delete(zk, "zh")
delete(ek, "en")
var missingEN, missingZH []string
for k := range zk {
if !ek[k] {
missingEN = append(missingEN, k)
}
}
for k := range ek {
if !zk[k] {
missingZH = append(missingZH, k)
}
}
if len(missingEN) > 0 {
t.Errorf("translation keys present in zh but missing in en: %v", missingEN)
}
if len(missingZH) > 0 {
t.Errorf("translation keys present in en but missing in zh: %v", missingZH)
}
if len(zk) < 100 {
t.Errorf("only %d zh keys parsed — the block boundary regexp has drifted "+
"and this test is no longer checking anything", len(zk))
}
}

View File

@ -1,267 +0,0 @@
package gateway
import (
"regexp"
"strings"
"testing"
)
// The WebUI dialogs all share the id "modal-wrap", and more than one can be
// open at the same time (the seed-key notice sits on top of the keys page).
// A save handler that closes "the" modal via $("#modal-wrap") therefore removes
// whichever one comes FIRST in the document — which is the wrong dialog: the
// form the user just submitted stays on screen while an unrelated dialog
// vanishes.
//
// This is the same class of bug the api() contract test pins: reviewing inline
// JS by eye does not catch it, and the symptom ("the dialog did not close")
// points away from the cause.
// modalCloseRe finds every `$(...)`-style lookup of the shared modal id.
var modalCloseRe = regexp.MustCompile(`\$\("#modal-wrap"\)`)
// closestModalRe finds the safe form: resolve the dialog from the clicked
// button instead of from the document.
var closestModalRe = regexp.MustCompile(`\.closest\("#modal-wrap"\)`)
func TestUIDialogClosesItselfNotTheFirstModal(t *testing.T) {
// Strip comments first: prose that *names* the unsafe pattern (as the fix's
// own comment does) would otherwise be flagged as a violation.
code := stripJSComments(uiSource(t))
for _, m := range modalCloseRe.FindAllStringIndex(code, -1) {
after := code[m[1]:]
stmtEnd := strings.Index(after, ";")
if stmtEnd < 0 || stmtEnd > 200 {
continue
}
stmt := after[:stmtEnd]
if strings.Contains(stmt, ".remove()") {
line := 1 + strings.Count(code[:m[0]], "\n")
t.Errorf("line %d closes the first #modal-wrap in the document, not its own dialog:\n\t%s",
line, strings.TrimSpace(stmt))
}
}
}
// stripJSComments removes // line comments and /* block */ comments from JS
// embedded in the UI document. It is deliberately simple: the document is our
// own source, and a false negative here only means the check is silent.
func stripJSComments(src string) string {
var out strings.Builder
lines := strings.Split(src, "\n")
inBlock := false
for _, ln := range lines {
trimmed := strings.TrimSpace(ln)
if inBlock {
if strings.Contains(ln, "*/") {
inBlock = false
}
continue
}
if strings.HasPrefix(trimmed, "/*") {
if !strings.Contains(ln, "*/") {
inBlock = true
}
continue
}
if i := strings.Index(ln, "//"); i >= 0 {
before := ln[:i]
if strings.Count(before, `"`)%2 == 0 && strings.Count(before, "'")%2 == 0 {
ln = before
}
}
out.WriteString(ln)
out.WriteString("\n")
}
return out.String()
}
// Every handler that closes a dialog must do it one of the two safe ways.
func TestUIDialogClosuresGoThroughSafePaths(t *testing.T) {
src := stripJSComments(uiSource(t))
for _, fn := range []string{
"downloadStatsCsv", "downloadKeysCsv", "saveSource", "saveTemplate",
"scrAddFromForm", "sortScopeSave", "scopeSave", "createKey",
} {
body, ok := jsFunctionBody(src, fn)
if !ok {
t.Errorf("%s not found", fn)
continue
}
if !strings.Contains(body, `closest("#modal-wrap")`) && !strings.Contains(body, "closeTopModal()") {
t.Errorf("%s closes a dialog with neither .closest nor closeTopModal", fn)
}
}
if !strings.Contains(src, "function closeTopModal(") {
t.Error("closeTopModal helper is missing")
}
}
// The helper must pick the LAST dialog (the topmost one the user sees), not the
// first — that inversion is the whole bug.
func TestUICloseTopModalTakesTheLast(t *testing.T) {
body, ok := jsFunctionBody(uiSource(t), "closeTopModal")
if !ok {
t.Fatal("closeTopModal not found")
}
if !strings.Contains(body, "all.length - 1") {
t.Errorf("closeTopModal does not take the last dialog:\n\t%s", oneLine(body))
}
}
// Handlers that resolve their dialog from a button must actually receive one:
// a signature without the parameter means the .closest() silently yields null
// and the save leaves its form stranded.
func TestUIDialogHandlersReceiveTheirButton(t *testing.T) {
src := stripJSComments(uiSource(t))
for _, fn := range []string{
"downloadStatsCsv", "saveSource", "scrAddFromForm", "sortScopeSave",
"scopeSave", "createKey",
} {
body, ok := jsFunctionBody(src, fn)
if !ok {
t.Errorf("%s not found", fn)
continue
}
if !strings.Contains(body, "closest(\"#modal-wrap\")") {
continue // uses closeTopModal only
}
sig := body[:strings.Index(body, ")")+1]
if !strings.Contains(sig, "btn") {
t.Errorf("%s uses .closest(\"#modal-wrap\") but its signature %s has no button parameter —\n"+
"the lookup would always be null and the form would never close", fn, oneLine(sig))
}
}
}
// Quotas belong to the scope entries, so putScope only has to ship the scope
// list — the caps travel inside it. What must NOT come back is a key-wide
// total: it would be a second budget able to drift out of sync with the models
// it is supposed to cover.
func TestUIPutScopeShipsOnlyScopeQuotas(t *testing.T) {
body, ok := jsFunctionBody(uiSource(t), "putScope")
if !ok {
t.Fatal("putScope not found")
}
code := stripJSComments(body)
if !strings.Contains(code, "models:") || !strings.Contains(code, "scopes") {
t.Error("putScope must ship the scope list the caps live in")
}
for _, gone := range []string{"kquota", "kreqquota", "kperiod", "khours"} {
if strings.Contains(code, gone) {
t.Errorf("putScope still references the removed key-wide quota field %q", gone)
}
}
}
// A model brick carries its own budgets, and the scope editor reads and writes
// both of them: dropping req_quota on the round trip would silently lift a
// request cap every time someone edited a token cap.
func TestUIScopeEditorRoundTripsBothQuotas(t *testing.T) {
src := stripJSComments(uiSource(t))
for _, fn := range []string{"scopeHtml", "readScopes", "scopeEdit", "scopeSave", "scopeQuotaBadge"} {
if _, ok := jsFunctionBody(src, fn); !ok {
t.Errorf("%s not found in the UI source", fn)
}
}
for _, fn := range []string{"scopeHtml", "readScopes", "scopeSave", "scopeQuotaBadge"} {
body, ok := jsFunctionBody(src, fn)
if !ok {
continue
}
if !strings.Contains(body, "req_quota") && !strings.Contains(body, "reqquota") {
t.Errorf("%s does not carry req_quota — a request cap would be lost on edit", fn)
}
}
if form, ok := jsFunctionBody(src, "scopeEdit"); ok && !strings.Contains(form, "sc-reqs") {
t.Error("the scope editor has no request-quota input")
}
}
// The whole key-wide quota surface must be gone from the UI: a badge or a
// button reading a field the server no longer has would render "undefined" or
// silently do nothing.
func TestUIHasNoKeyWideQuotaSurface(t *testing.T) {
src := uiSource(t)
for _, gone := range []string{
"keyCapBadges", "keyQuotaEdit", "keyQuotaSave",
"kq-tokens", "kq-reqs", "kq-period", "kq-hours",
"kc-tokens", "kc-reqs", "kc-period", "kc-hours",
"data-kquota", "data-kreqquota", "data-kperiod", "data-khours",
} {
if strings.Contains(src, gone) {
t.Errorf("UI still references the removed key-wide quota surface %q", gone)
}
}
}
// Existence of the strings is not enough: the badge has to READ the scope's
// fields, and the editor has to read back what the brick writes. A field can
// be present in the source and still never reach the screen — e.g. left in a
// dead branch, or read from a data attribute the writer never sets.
func TestUIKeyQuotaDataflowIsLive(t *testing.T) {
src := stripJSComments(uiSource(t))
badge, ok := jsFunctionBody(src, "scopeQuotaBadge")
if !ok {
t.Fatal("scopeQuotaBadge not found")
}
for _, field := range []string{"m.token_quota", "m.req_quota", "m.period"} {
if !strings.Contains(badge, field) {
t.Errorf("scopeQuotaBadge does not read %q — the cap would never show on the model brick", field)
}
}
brick, ok := jsFunctionBody(src, "scopeHtml")
if !ok {
t.Fatal("scopeHtml not found")
}
edit, ok := jsFunctionBody(src, "scopeEdit")
if !ok {
t.Fatal("scopeEdit not found")
}
brickCode, editCode := stripJSComments(brick), stripJSComments(edit)
reader := stripJSComments(mustBody(t, src, "readScopes"))
for _, ds := range []string{"quota", "reqquota", "period", "hours"} {
if !strings.Contains(brickCode, "data-"+ds+"=") {
t.Errorf("scopeHtml does not write data-%s, so the editor has nothing to prefill", ds)
}
// the editor pre-fills through readScopes(), which is what walks the
// bricks' data attributes — check the reader, not the form
if !strings.Contains(reader, "dataset."+ds) {
t.Errorf("readScopes does not read dataset.%s — editing a brick would save zeros over it", ds)
}
}
// and the form must actually consume what readScopes produced
for _, field := range []string{"sc.token_quota", "sc.req_quota", "sc.period", "sc.hours"} {
if !strings.Contains(editCode, field) {
t.Errorf("scopeEdit does not prefill from %q", field)
}
}
}
func mustBody(t *testing.T, src, fn string) string {
t.Helper()
b, ok := jsFunctionBody(src, fn)
if !ok {
t.Fatalf("%s not found", fn)
}
return b
}
// The quota period vocabulary must match the server's, or the UI can offer a
// value the API rejects.
func TestUIQuotaPeriodsMatchServer(t *testing.T) {
src := uiSource(t)
for _, p := range []string{`value=""`, `value="hour"`, `value="week"`, `value="month"`, `value="nhour"`} {
if !strings.Contains(src, p) {
t.Errorf("UI period select is missing %s", p)
}
}
for _, p := range []string{`"hour"`, `"week"`, `"month"`, `"nhour"`} {
if !strings.Contains(src, `if (p === `+p+`)`) && !strings.Contains(src, `=== `+p+`)`) {
t.Errorf("periodText() does not describe %s, so a badge would omit the window", p)
}
}
}