mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-11-07 17:57:23 +00:00
Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d5bbb0eca5 | |||
| 9cb6254166 | |||
| 4514270e21 | |||
| ef95bc45d0 | |||
| f7e926ccb7 | |||
| 688dce88d1 | |||
| de7553eaa2 | |||
| 79bc55cfbe | |||
| 4b9fb7b056 | |||
| 7941b4110a | |||
| 5ccf068db3 | |||
| b4c5f0f0fc | |||
| 442fa5e3f6 | |||
| 0caa37ea6a | |||
| 33cb2bdb06 | |||
| 0ae7d2c8be | |||
| 02c01582f1 | |||
| 6454b094bc | |||
| 771b95a133 | |||
| 441de434f5 | |||
| 9feef8f4cb |
497
HANDOFF_2026-10-19.md
Normal file
497
HANDOFF_2026-10-19.md
Normal file
@ -0,0 +1,497 @@
|
||||
# ModelRouter / llmsproxy 交接文档
|
||||
|
||||
更新时间:2026-10-19
|
||||
工作目录:`/home/program/llmsproxy`
|
||||
当前分支:`release/v2.0.x`
|
||||
|
||||
## 1. 当前 Git 状态
|
||||
|
||||
最近提交:
|
||||
|
||||
- `c9f3453 fix(codearts): 识别 InferHub 未注册模型`
|
||||
- `12bfd53 fix: 收口渠道源迁移与测试页合并`
|
||||
- `25970a2 refactor(scheduler)!: 移除 priority 废弃字段 + 一通道一源 + discover 复用凭据`
|
||||
|
||||
`c9f3453` 已推送到 `origin/release/v2.0.x`。
|
||||
|
||||
注意:当前工作树在最后一轮添加 provider 回归测试时出现了一个**未完成且可能无法编译的测试编辑**。最后一次工具报错:
|
||||
|
||||
```text
|
||||
internal/provider/provider_test.go:99: missing ',' in argument list
|
||||
```
|
||||
|
||||
下一位 agent 接手后的第一步必须是:
|
||||
|
||||
```bash
|
||||
cd /home/program/llmsproxy
|
||||
git status --short
|
||||
gofmt -w internal/provider/provider_test.go 2>&1 || true
|
||||
go test -tags luajit ./internal/provider/ 2>&1
|
||||
```
|
||||
|
||||
如果测试文件处于半编辑状态,先检查 `TestProviderChatStreamNormalizesStopAfterToolCall`,修正 Go 字符串转义或删除该未完成测试后再继续。不要直接部署当前工作树。
|
||||
|
||||
## 2. 生产服务状态
|
||||
|
||||
生产服务:`llmsproxy.service`
|
||||
|
||||
- 当前服务 active
|
||||
- 监听:`0.0.0.0:8081`
|
||||
- 二进制已通过 `./deploy.sh` 原子替换并重启
|
||||
- 最近一次部署健康检查通过
|
||||
- 当前生产插件 SHA256 前缀曾核对为:`1bf26cbb3abcb107`
|
||||
- 当前生产二进制在最近部署后已更新
|
||||
|
||||
不要把密钥写入交接记录。生产 admin key 保存在本机运维环境中,不应复制到聊天、文档或提交。
|
||||
|
||||
## 3. Channel 源当前状态
|
||||
|
||||
生产源已经从旧的“一账号一源”收口成“一渠道一源”:
|
||||
|
||||
```text
|
||||
channels-trae
|
||||
channels-minimax
|
||||
channels-lobsterai
|
||||
channels-buddy
|
||||
channels-codearts
|
||||
channels-zcode
|
||||
```
|
||||
|
||||
旧形态源已经清理:
|
||||
|
||||
```text
|
||||
channels-<provider>-<account_id>
|
||||
```
|
||||
|
||||
此前旧源和新源并存,造成同一渠道同一模型出现多份,只能靠账号 hex 区分。
|
||||
|
||||
根因已修复:内核 `list_sources` 返回裸数组,但旧插件代码按 `{sources = [...]}` 信封读取,导致迁移清理循环永远为空。当前 `channels.lua` 已兼容:
|
||||
|
||||
- 裸数组
|
||||
- `{sources = [...]}`
|
||||
- `{json = {sources = [...]}}`
|
||||
|
||||
相关代码:
|
||||
|
||||
```text
|
||||
internal/lua/plugins/channels.lua: syncProviderSource
|
||||
internal/core/channels_legacy_source_cleanup_test.go
|
||||
```
|
||||
|
||||
当前生产源列表已实测只有 6 个新形态源。
|
||||
|
||||
## 4. CodeArts 当前状态
|
||||
|
||||
### 已确认可用的部分
|
||||
|
||||
CodeArts 使用:
|
||||
|
||||
```text
|
||||
adapter = codearts
|
||||
base_url = https://snap-access.cn-north-4.myhuaweicloud.com
|
||||
```
|
||||
|
||||
CodeArts 的 chat 请求需要完整 AK/SK/security token 凭据,由插件适配器签名,不是裸 Bearer token。
|
||||
|
||||
真实流式工具调用测试已经成功:
|
||||
|
||||
请求形状:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "channels-codearts:openpangu-2.0-flash",
|
||||
"messages": [{"role": "user", "content": "Use the weather tool for Beijing."}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get weather",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto",
|
||||
"stream": true,
|
||||
"max_tokens": 64
|
||||
}
|
||||
```
|
||||
|
||||
结果:
|
||||
|
||||
- HTTP 200
|
||||
- `Content-Type: text/event-stream`
|
||||
- 返回结构化 `tool_calls`
|
||||
- 函数名 `get_weather`
|
||||
- 参数最终为 `{"city":"Beijing"}`
|
||||
- `finish_reason = tool_calls`
|
||||
|
||||
因此 CodeArts 的流式工具调用链路当前是通的。
|
||||
|
||||
### CodeArts 模型刷新
|
||||
|
||||
生产调用:
|
||||
|
||||
```http
|
||||
POST /api/plugins/http/channels/model/refresh
|
||||
{"platform":"codearts"}
|
||||
```
|
||||
|
||||
实测返回:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"results": [
|
||||
{"platform": "codearts", "ok": true, "count": 11}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
CodeArts 模型目录来自两个签名端点并合并:
|
||||
|
||||
1. `opengw.developer.huaweicloud.com/api/v1/gateway/config`
|
||||
- `result.models`
|
||||
2. `snap-access.cn-north-4.myhuaweicloud.com/v1/model/builtin`
|
||||
- `builtinModels`
|
||||
|
||||
相关解析:
|
||||
|
||||
```text
|
||||
internal/lua/plugins/channels.lua: parseRemoteModels
|
||||
internal/lua/plugins/channels.lua: refreshRemoteModels
|
||||
internal/core/channels_codearts_catalog_test.go
|
||||
```
|
||||
|
||||
### 已确认失效并禁用的 CodeArts 模型
|
||||
|
||||
下面 4 个模型曾收到明确的上游 404:
|
||||
|
||||
```text
|
||||
CodeArts-Pangu
|
||||
deepseek-v4-flash-0731
|
||||
deepseek-v4-pro-0813
|
||||
glm-5.3-flash
|
||||
```
|
||||
|
||||
生产上已通过 `channels/model/set_disabled` 禁用。它们仍可能在 `model/list` 中出现,但 `disabled=true`,不会作为可用模型路由。
|
||||
|
||||
新增的死模型判据:
|
||||
|
||||
```lua
|
||||
model is not registered
|
||||
```
|
||||
|
||||
对应提交:`c9f3453`。
|
||||
|
||||
注意:当前插件的 dead-model 机制只在收到真实请求失败后记录模型;刷新目录不会自动根据“远端目录没有该模型”删除模型,这是刻意设计,避免误删远端目录不完整的平台模型。
|
||||
|
||||
CodeArts 当前仍有一些模型返回 429/TPM 限流,例如:
|
||||
|
||||
```text
|
||||
deepseek-v4-flash
|
||||
deepseek-v4.1-flash
|
||||
```
|
||||
|
||||
429 不是模型失效,不能禁用。
|
||||
|
||||
## 5. 其他 Channel 实测结果
|
||||
|
||||
### Trae
|
||||
|
||||
真实流式 + tools 请求:
|
||||
|
||||
```text
|
||||
channels-trae:DeepSeek-V4-Flash
|
||||
```
|
||||
|
||||
结果:
|
||||
|
||||
- HTTP 200
|
||||
- SSE 正常返回
|
||||
- 结构化 tool call 返回
|
||||
- 但上游最后的结束原因可能是 `stop`,即使前面已经出现 tool-call fragments
|
||||
|
||||
这会让 Agent 把工具调用误判成普通文本完成。
|
||||
|
||||
当前准备中的修复在统一 provider 流式层:
|
||||
|
||||
```text
|
||||
internal/provider/provider.go
|
||||
```
|
||||
|
||||
逻辑是:只要本次流出现真实 tool-call 片段,后续 `FinishReason == "stop"` 就归一成 `tool_calls`。
|
||||
|
||||
另外,Trae 的非流式接口偶尔返回 SSE。当前 `trae.lua` 对非流式 SSE 只取最后一条 data,不能完整合并多条工具调用/推理片段。这是后续仍需处理的协议问题,但不能用非流式手工请求结果推断 Pi 的真实请求模式。
|
||||
|
||||
### Buddy
|
||||
|
||||
真实流式 + tools 请求:
|
||||
|
||||
```text
|
||||
channels-buddy:glm-5.3-flash
|
||||
```
|
||||
|
||||
结果:
|
||||
|
||||
- HTTP 200
|
||||
- SSE 正常返回
|
||||
- tool-call 参数分片可见
|
||||
- 最终有 `finish_reason = tool_calls`
|
||||
- 但某些上游 chunk 的 `tool_calls` 可能是空对象 `{}`
|
||||
|
||||
空对象不是合法的 OpenAI tool-call 数组,部分 Agent 客户端会拒绝。
|
||||
|
||||
当前准备中的修复在:
|
||||
|
||||
```text
|
||||
internal/lua/adapters/buddy.lua
|
||||
```
|
||||
|
||||
只在 `type(delta.tool_calls) == "table" and #delta.tool_calls > 0` 时输出 `tool_calls`,避免 Lua 空表被编码为 `{}`。
|
||||
|
||||
Buddy 的非流式错误:
|
||||
|
||||
```text
|
||||
11101: Non-stream chat request is currently not supported
|
||||
```
|
||||
|
||||
之前出现的该错误来自手工 `curl` 测试中显式设置 `stream:false`,不能归因到 Pi。真实 `stream:true + tools` 已经证明 Buddy 流式工具调用可以工作。
|
||||
|
||||
### MiniMax
|
||||
|
||||
生产日志反复出现:
|
||||
|
||||
```text
|
||||
adapter "anthropic": status=401 body={"error":"invalid access token"}
|
||||
provider channels-minimax: api error 401
|
||||
```
|
||||
|
||||
这是凭据失效问题,不是 tools 形状问题。
|
||||
|
||||
当前源:
|
||||
|
||||
```text
|
||||
channels-minimax
|
||||
adapter = anthropic
|
||||
endpoint = /mavis/api/v1/llm/v1/messages
|
||||
```
|
||||
|
||||
需要用户重新登录/提供有效凭据。不要擅自刷新一次性 refresh token,之前已经发生过 refresh token 被消耗后无法再次使用的事故。
|
||||
|
||||
## 6. 当前未提交的代码改动
|
||||
|
||||
最后一轮已经改动但尚未完成提交:
|
||||
|
||||
```text
|
||||
internal/provider/provider.go
|
||||
internal/lua/adapters/trae.lua
|
||||
internal/lua/adapters/buddy.lua
|
||||
internal/provider/provider_test.go # 可能有半编辑、需先修复
|
||||
```
|
||||
|
||||
已完成的意图:
|
||||
|
||||
1. provider 流式层记录 `sawToolCalls`
|
||||
2. 看到真实 tool-call 后,把后续 `stop` 归一为 `tool_calls`
|
||||
3. Trae 不输出空工具调用表
|
||||
4. Buddy 不输出空工具调用表
|
||||
5. provider 回归测试模拟“tool fragment + stop”场景
|
||||
|
||||
但 provider 测试最后一次编辑工具报错:
|
||||
|
||||
```text
|
||||
internal/provider/provider_test.go:99: missing ',' in argument list
|
||||
```
|
||||
|
||||
因此必须先修测试文件,再提交。
|
||||
|
||||
## 7. 推荐接手顺序
|
||||
|
||||
### 第一步:修复工作树
|
||||
|
||||
```bash
|
||||
cd /home/program/llmsproxy
|
||||
git status --short
|
||||
sed -n '1,150p' internal/provider/provider_test.go
|
||||
gofmt -w internal/provider/provider_test.go 2>&1 || true
|
||||
go test -tags luajit ./internal/provider/ 2>&1
|
||||
```
|
||||
|
||||
若测试函数字符串转义难以修复,先删除半成品测试,再重新用 raw string 写测试。
|
||||
|
||||
### 第二步:跑定向验证
|
||||
|
||||
```bash
|
||||
luac -p internal/lua/adapters/trae.lua
|
||||
luac -p internal/lua/adapters/buddy.lua
|
||||
go test -tags luajit ./internal/provider/
|
||||
go test -tags luajit ./internal/lua/ -run 'Adapter|Buddy|Trae|Stream'
|
||||
```
|
||||
|
||||
### 第三步:构建并部署临时验证版
|
||||
|
||||
修复确认后:
|
||||
|
||||
```bash
|
||||
go build ./...
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
部署前确认:
|
||||
|
||||
```bash
|
||||
sha256sum internal/lua/plugins/channels.lua
|
||||
sha256sum /etc/llmsproxy/plugins/channels.lua
|
||||
systemctl is-active llmsproxy
|
||||
```
|
||||
|
||||
### 第四步:真实流式回归
|
||||
|
||||
不要用 `stream:false` 测试流式问题。使用:
|
||||
|
||||
```bash
|
||||
KEY='<从本机安全环境读取,不写入文档>'
|
||||
TOOL='{"type":"function","function":{"name":"get_weather","description":"Get weather","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}'
|
||||
|
||||
curl --no-buffer --max-time 90 \
|
||||
-H "Authorization: Bearer $KEY" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"model\":\"channels-codearts:openpangu-2.0-flash\",\"messages\":[{\"role\":\"user\",\"content\":\"Use the weather tool for Beijing.\"}],\"tools\":[$TOOL],\"tool_choice\":\"auto\",\"stream\":true,\"max_tokens\":64}" \
|
||||
http://127.0.0.1:8081/v1/chat/completions
|
||||
```
|
||||
|
||||
分别将 model 替换为:
|
||||
|
||||
```text
|
||||
channels-codearts:openpangu-2.0-flash
|
||||
channels-trae:DeepSeek-V4-Flash
|
||||
channels-buddy:glm-5.3-flash
|
||||
```
|
||||
|
||||
验收:
|
||||
|
||||
- HTTP 200
|
||||
- `Content-Type: text/event-stream`
|
||||
- 不能出现 `tool_calls: {}`
|
||||
- 有 tool-call fragments 时最终必须是 `finish_reason: tool_calls`
|
||||
- CodeArts 当前应继续返回结构化工具调用
|
||||
|
||||
### 第五步:CodeArts 刷新验收
|
||||
|
||||
```bash
|
||||
curl -sS -X POST \
|
||||
-H "Authorization: Bearer $KEY" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"platform":"codearts"}' \
|
||||
http://127.0.0.1:8081/api/plugins/http/channels/model/refresh
|
||||
```
|
||||
|
||||
预期:
|
||||
|
||||
```json
|
||||
{"ok":true,"results":[{"platform":"codearts","ok":true,"count":11}]}
|
||||
```
|
||||
|
||||
再检查:
|
||||
|
||||
```bash
|
||||
curl -sS -X POST \
|
||||
-H "Authorization: Bearer $KEY" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"platform":"codearts"}' \
|
||||
http://127.0.0.1:8081/api/plugins/http/channels/model/list
|
||||
```
|
||||
|
||||
预期:远端目录 11 个 + 兜底补位,已确认 404 的 4 个仍保持 `disabled=true`。
|
||||
|
||||
## 8. 发布/安装包状态
|
||||
|
||||
目前没有完成“全部安装包”发布闭环。
|
||||
|
||||
已经发生的事情:
|
||||
|
||||
- 核心包构建成功,但脚本实际使用了 `git describe`,产物版本类似:
|
||||
`2.0.40~4-gc9f3453`
|
||||
- Linux GUI 包构建成功,版本为 `1.9.7`
|
||||
- Windows 主机 Wine 构建失败,生成了一个 264KB 的坏 `ModelRouter Setup 1.9.7.exe`,绝对不能发布
|
||||
- Docker Windows 构建也失败在拉取 Docker Hub 基础镜像 `golang:1.25` 网络超时
|
||||
- apt 无法找到 `wine32:i386`
|
||||
- 旧的 1.9.5/1.9.6 Windows 包是历史产物,不能作为当前版本
|
||||
|
||||
不要把旧 Windows exe 当作新包发布。
|
||||
|
||||
仓库已有 Docker 方案:
|
||||
|
||||
```text
|
||||
cmd/gui/scripts/dist-win-docker.sh
|
||||
cmd/gui/docker/win-builder/Dockerfile
|
||||
```
|
||||
|
||||
但当前 Docker Hub 网络不可用。若要继续做 Windows 包:
|
||||
|
||||
1. 在可访问 Docker Hub 的构建机/CI 执行 Docker 方案;或
|
||||
2. 提供包含 Wine32/Wine64 的完整构建环境;或
|
||||
3. 暂时只发布 Linux/core 包,并明确说明 Windows 包未完成。
|
||||
|
||||
版本脚本注意:
|
||||
|
||||
- 核心包脚本使用位置参数:
|
||||
|
||||
```bash
|
||||
./packaging/core-dist.sh 2.0.41
|
||||
```
|
||||
|
||||
不是 `VERSION=2.0.41 ./packaging/core-dist.sh`。
|
||||
- GUI 版本来自 `cmd/gui/package.json`,环境变量 `VERSION` 不会自动修改 Electron 产品版本。
|
||||
- 清理旧产物后再打包,并在全部包完成后统一计算 `SHA256SUMS`。
|
||||
|
||||
## 9. 参考插件状态
|
||||
|
||||
用户要求按:
|
||||
|
||||
```text
|
||||
https://github.com/Ebony-Vinyl/dsh-our-free-model
|
||||
```
|
||||
|
||||
逐厂商对照参考插件。
|
||||
|
||||
当前本地路径:
|
||||
|
||||
```text
|
||||
/tmp/pi-github-repos/Ebony-Vinyl/dsh-our-free-model
|
||||
```
|
||||
|
||||
但该目录当前不是有效可读源码副本:
|
||||
|
||||
- `vendor/channel-pack/src` 下面只有空目录;
|
||||
- `.git` 没有有效 `HEAD`/refs;
|
||||
- 无法从本地 Git 对象恢复源码;
|
||||
- 本地也没有 `buddy.ts`、`trae.ts`、`codearts.ts` 等文件。
|
||||
|
||||
因此不能声称当前适配器是逐行照参考插件移植的。若要继续按用户要求完成,必须先重新 clone/恢复该 GitHub 仓库源码,确认以下文件实际存在后再对照:
|
||||
|
||||
```text
|
||||
vendor/channel-pack/src/trae.ts
|
||||
vendor/channel-pack/src/buddy.ts
|
||||
vendor/channel-pack/src/codearts.ts
|
||||
vendor/channel-pack/src/*adapter*
|
||||
vendor/channel-pack/src/*model*
|
||||
```
|
||||
|
||||
当前仓库代码只能作为临时协议修复基础,不能代替参考插件源码。
|
||||
|
||||
## 10. 关键结论
|
||||
|
||||
- Pi 是否发送流式请求,历史日志没有保存入站 `stream/tools` 字段,不能靠旧日志判断。
|
||||
- 本次真实 `stream:true + tools` 测试证明 ModelRouter 流式路径能工作:
|
||||
- CodeArts:通过;
|
||||
- Trae:通过但结束原因需归一;
|
||||
- Buddy:通过但需过滤空 `{}`;
|
||||
- MiniMax:凭据 401,独立问题。
|
||||
- 不要再用 `stream:false` 的手工请求推断 Pi 的问题。
|
||||
- CodeArts 刷新接口当前真实返回 `ok=true,count=11`,刷新链路不是当前阻塞点。
|
||||
- 当前最紧急的代码状态是修复 `provider_test.go` 的半编辑,然后验证未提交的统一流式修复。
|
||||
- 未经测试、未重新构建、未重新部署,不要继续改生产账号或刷新一次性凭据。
|
||||
4
Makefile
4
Makefile
@ -2,7 +2,7 @@
|
||||
|
||||
# ModelRouter - single-binary gateway + optional Electron desktop GUI (embedded core)
|
||||
export BINARY
|
||||
BINARY=llmsproxy
|
||||
BINARY=modelrouter
|
||||
export BUILD_DIR
|
||||
BUILD_DIR=build
|
||||
export GUI_DIR
|
||||
@ -20,7 +20,7 @@ all: build
|
||||
# ---- core binary (server users / embedded core) ----
|
||||
build:
|
||||
@mkdir -p $(BUILD_DIR)
|
||||
CGO_ENABLED=1 go build -tags luajit -trimpath -o $(BUILD_DIR)/$(BINARY) ./cmd/llmsproxy
|
||||
CGO_ENABLED=1 go build -tags luajit -trimpath -o $(BUILD_DIR)/$(BINARY) ./cmd/modelrouter
|
||||
@echo "Built: $(BUILD_DIR)/$(BINARY) ($(VERSION))"
|
||||
|
||||
# ---- core distribution packages (deb + rpm + tar.gz) -----
|
||||
|
||||
674
README.md
674
README.md
@ -4,103 +4,46 @@
|
||||
>
|
||||
> **AI-assisted**: 本项目使用 AI 辅助编程(代码与文档由 AI 协作完成,经人工复核)。
|
||||
|
||||
面向多 llms 订阅者,部署在内网,实现一次配置多个服务共同使用的效果。支持指定模型或 auto 模式,按照配置的优先级选择可用模型提供服务。
|
||||
面向多 LLM 订阅者的内网网关:一次配置,多个服务共用一组上游额度。支持指定模型或
|
||||
`AUTO` 模式,按配置的优先级选择可用模型提供服务。
|
||||
|
||||
> **轻量且增源无需重编译**:网关本体是单 Go 二进制(约 10MB,零运行时依赖)。新增/切换上游
|
||||
> 只需在 `config.yaml`(或 WebUI)加一个 `sources` 条目或挂一个 `.lua` 适配器——**不改 Go、
|
||||
> 不重编译**。源与适配器的改动经 WebUI 提交时即时生效(热更新);直接编辑
|
||||
> `config.yaml` 或 `adapter_dir` 下的 `.lua` 文件则需要重启进程生效。
|
||||
网关本体是单个 Go 二进制(约 10 MB,零运行时依赖)。新增或切换上游只需在
|
||||
`config.yaml`(或 WebUI)加一个 `sources` 条目、或挂一个 `.lua` 适配器——**不改 Go、
|
||||
不重编译**。源与适配器的改动经 WebUI 提交时即时生效(热更新);直接编辑
|
||||
`config.yaml` 或 `adapter_dir` 下的 `.lua` 文件需要重启进程生效。
|
||||
|
||||
## 核心优势
|
||||
## 核心能力
|
||||
|
||||
### 极致轻量
|
||||
|
||||
- **单二进制**:编译后 8~12MB(`-s -w` strip 后约 8MB),零运行时依赖(仅依赖系统 libc),部署即用
|
||||
- **低内存占用**:与源数量相关,非与运行时长相关——单源 ~10MB、16 源生产实例 ~32–35MB([实测分解与调优](#内存占用实测与调优))
|
||||
- **日志按需加载**:统计数字来自**全量**审计日志(启动时流式扫一遍即释放,29MB/22 万行约 260ms),但原始记录不常驻内存——默认只加载首屏,下滚自动分页,CSV 导出流式写出(O(1) 内存),离页即释放
|
||||
- **零运行时依赖**:纯 Go + LuaJIT 静态链接,无需安装 Python/Node/Java 等运行时
|
||||
- **启动极快**:冷启动 < 200ms,热重载配置 < 10ms
|
||||
|
||||
### 强大的多租户调度能力
|
||||
|
||||
- **多密钥多租户**:支持无限密钥,每个密钥独立角色、模型范围、Token 配额、重置周期
|
||||
- **按模型配额**:每把 key 的每个模型单独配 token + 请求数配额与重置周期(小时/周/月/自定义 N 小时);一个模型用满只影响该模型,同 key 其它模型照常;耗尽返 429 + `Retry-After` 并点名模型,admin key 永不受限
|
||||
- **AUTO 智能调度**:基于优先级档位的分级调度,同优先级源自动轮询负载均衡,故障自动毫秒级故障转移
|
||||
- **自愈冷却**:冷却上限 5 分钟,过半后放行 1 个探测请求,上游/额度恢复即刻回归轮询,无需等满冷却窗口
|
||||
- **Token 配额管理**:精确到模型级别的 Token 配额控制,支持小时/周/月/自定义小时周期自动重置
|
||||
- **同优先级源负载均衡**:同一优先级档位的多个源,请求自动 Round-Robin 均匀分发,故障毫秒级故障转移
|
||||
- **Source-Model 前缀路由**:支持 `source-model`/`source:model`/`source/model` 精确指定上游源
|
||||
|
||||
### 生产级可靠性
|
||||
|
||||
- **热加载配置**:WebUI 修改源/密钥/适配器/AUTO链即时生效,无需重启
|
||||
- **密钥加密落盘**:AES-256-GCM 加密存储 `api_key`、网关密钥、请求头,master.key 0600 权限保护
|
||||
- **实时源探测**:`GET {base}/models` 定期探测,毫秒级感知上游状态,不污染调度退避状态
|
||||
- **全链路审计**:HTTP 访问日志、登录/配置变更审计、请求记录 CSV 导出、密钥用量统计 CSV 导出
|
||||
|
||||
### 灵活的协议适配
|
||||
|
||||
- **LuaJIT VM**:每个适配器独立 VM + **弹性 worker 池**(按需扩缩容,空闲时自动回收),安全并发,Lua 脚本热加载无需重启
|
||||
- **Lua 适配器协议**:`transform_request` / `transform_response` / `transform_stream_chunk` 双向转换
|
||||
- **动态请求头钩子**:`build_headers(meta)` 支持 HMAC 签名、动态 Header 注入
|
||||
- **多模态透传**:`image_url` 等多模态内容在多源间无损透传,Anthropic/Gemini/Ollama 自动转换
|
||||
|
||||
## 实现
|
||||
|
||||
统一 OpenAI 兼容网关:把多个上游 LLM 源(DeepSeek、Qijiar、OpenAI、Anthropic、Gemini、
|
||||
Groq、Mistral、Ollama、KimiCode…)通过 **Lua 适配器** 做协议转换,对内网暴露
|
||||
一个标准 `OpenAI Chat Completions` 接口(`/v1/chat/completions` + `/v1/models`),
|
||||
支持单次结算、SSE 流式、**AUTO 模型路由**与**多模态**透传。
|
||||
|
||||
从 [HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent) 的多源 LLM 适配层
|
||||
(`internal/agent/api/provider.go` + `internal/lua/adapters/*`)抽离而来并独立演进。
|
||||
|
||||
## 特性
|
||||
|
||||
- **多源**:一个进程内配置任意多个上游源,按请求的 `model` 自动路由。
|
||||
- **AUTO 模式**:`default_model: AUTO` 时按优先级页保存的 AUTO 链档位逐档调度并发请求。
|
||||
- **统一输出**:所有源对外都是 OpenAI 格式(含 `reasoning_content`、`tool_calls`、`usage`)。
|
||||
- **生图**:`POST /v1/images/generations`,`kind: image` 的模型独立路由。
|
||||
- **多模态**:`content` 数组(`image_url` 等)在多源间无损透传,Anthropic/Gemini/Ollama 自动转换。
|
||||
- **LuaJIT VM**:基于 golua 绑定的 LuaJIT,每个适配器独立 VM + worker 池,安全并发。
|
||||
池是 **弹性** 的:`Σ max_concurrent` 只是上限而非预分配,状态按需创建、空闲时自动回收,
|
||||
因此空载网关几乎不持有 Lua 状态(启动 0 个)。
|
||||
- 扩容步长由适配器最大并发决定:`clamp(ceil(max/8), 1, 8)`,且仅在**真实并发争用**
|
||||
(所有现有状态均已占用)时批量预热,顺序流量始终只用 1 个状态。
|
||||
- 缩容步长由当前连接数决定:`clamp(ceil(冗余/(1+占用)), 1, 冗余)`——无连接时一轮收到
|
||||
常驻下限,忙时每轮只释放 1 个,保护热路径。WebUI「适配器」页展示实时池指标。
|
||||
- **thinking 预算**:请求里的推理开关/预算归一为中立的 `thinking:{enabled,budget_tokens}`,
|
||||
适配器映射到各自协议(Anthropic `thinking.budget_tokens`、Gemini `thinkingConfig`、
|
||||
DeepSeek `thinking.type="disabled"`)。
|
||||
- **Lua 适配协议**:每个源挂一个 `.lua` 适配器,完成 `transform_request` /
|
||||
`transform_response` / `transform_stream_chunk` 双向转换,协议差异全在 Lua 层。
|
||||
- **签名 / 请求头钩子**:适配器可定义 `build_headers(meta)`,在 Go 发 HTTP 前
|
||||
动态注入/签名请求头——用于云端 API 校验调用方 app(如 KimiCode 只放行特定
|
||||
agent)。提供 `hmac_sha256_hex` / `sha256_hex` / `base64_encode` 等签名辅助。
|
||||
- **WebUI**:内置管理界面,可在线查看/新增/编辑上游源与模型、配置 AUTO 优先级链、
|
||||
管理密钥模型范围,写入运行时文件持久化。
|
||||
- **密钥加密存储**:运行时文件中的上游 `api_key`、自定义请求头值、网关 key 均以
|
||||
AES-256-GCM 加密落盘(`master.key` 独立 0600,或 `LLMS_PROXY_MASTER_KEY`)。
|
||||
- **实时源探测**:状态页探测各源可达性(`GET {base}/models`,失败回退最小请求),
|
||||
不污染正常调度的退避状态,错误信息在 UI 可悬停查看。
|
||||
- **鉴权**:网关自身用 `gateway_keys` 校验客户端 Bearer key;与上游各自的 key 相互独立。
|
||||
- **流式**:SSE `chat.completion.chunk`,含角色首包与 `[DONE]` 收尾。
|
||||
- **全链路审计**:HTTP 访问日志、登录/配置变更审计、请求记录 CSV 导出、密钥用量统计 CSV 导出
|
||||
- **AES-256-GCM 加密存储**:运行时文件敏感字段加密落盘,master.key 0600 权限,支持环境变量注入主密钥
|
||||
- **统一接口**:对内暴露 OpenAI 兼容的 `/v1/chat/completions`、`/v1/models`、
|
||||
`/v1/images/generations`;所有源对外都是同一套格式(含 `reasoning_content`、
|
||||
`tool_calls`、`usage`)。
|
||||
- **多源 + AUTO 链**:一个进程内任意多个上游源;`default_model: AUTO` 时按优先级页
|
||||
保存的档位逐档调度,源故障自动降级并冷却,恢复后半冷却探测自愈。
|
||||
- **协议适配在 Lua 层**:每个源挂一个 `.lua` 适配器,完成请求/响应/流式分片的双向
|
||||
转换,并可用 `build_headers(meta)` 注入签名头。协议差异不进 Go 内核。
|
||||
- **插件独立**:业务扩展(渠道接入、计费、额度巡检)是外置 Lua 插件,由插件目录加载,
|
||||
不编进二进制、不随发行包分发;插件只通过 `modelrouter` SDK 与内核交互。
|
||||
- **多租户密钥**:网关 key 与上游 key 相互独立,按 key 限定模型范围、AUTO 链与配额。
|
||||
- **加密存储**:上游 key、自定义请求头、网关 key 均以 AES-256-GCM 落盘,
|
||||
`master.key` 单独 0600 权限(或由 `MODELROUTER_MASTER_KEY` 注入)。
|
||||
- **可观测**:状态页实时探测源可达性;HTTP 访问日志、配置变更审计、请求与用量 CSV 导出;
|
||||
写操作不落在请求热路径上。
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
|
||||
./llmsproxy -config config.yaml # 首次运行自动生成默认配置并打印随机 admin key
|
||||
# 需要 LuaJIT 开发库;-tags luajit 是必须的
|
||||
GOMODCACHE=... GOPROXY=off go build -tags luajit -o modelrouter ./cmd/modelrouter
|
||||
|
||||
# 首次运行会生成默认配置,并在日志里打印随机 admin key
|
||||
./modelrouter -config config.yaml
|
||||
```
|
||||
|
||||
> 依赖 [golua](https://github.com/aarzilli/golua)(LuaJIT 绑定)。**必须**带 `-tags luajit`
|
||||
> 构建,否则默认走内置超集 gopher-lua 路径(行为略有差异)。
|
||||
> 依赖 [golua](https://github.com/aarzilli/golua)(LuaJIT 绑定)。不带 `-tags luajit`
|
||||
> 会走 gopher-lua 路径,行为略有差异。
|
||||
|
||||
> **配置不入库**:仓库不携带任何 config 文件(配置文件含密钥)。二进制首次运行时
|
||||
> 会在 `-config` 指定的位置生成默认配置:随机 admin key(打印在启动日志中)、
|
||||
> 仅绑定 `127.0.0.1:8080`。首次登录后请在 WebUI「密钥」页更换管理员密钥。
|
||||
> **配置不入库**:仓库不携带任何 config 文件(其中含密钥)。默认配置只绑定
|
||||
> `127.0.0.1:8080`,首次登录后请在 WebUI「密钥」页更换管理员密钥。
|
||||
|
||||
```bash
|
||||
# 无 key -> 401
|
||||
@ -117,526 +60,73 @@ curl -N -H "Authorization: Bearer $KEY" \
|
||||
http://127.0.0.1:8080/v1/chat/completions
|
||||
```
|
||||
|
||||
任何 OpenAI SDK 把 `base_url` 指到网关地址、`api_key` 用任意一个已创建并授权的网关 key 即可。
|
||||
任何 OpenAI SDK 把 `base_url` 指向网关、`api_key` 换成任一已授权的网关 key 即可。
|
||||
|
||||
## 配置
|
||||
## 文档
|
||||
|
||||
首次运行在 `-config` 路径自动生成默认配置(默认 `./config.yaml`),核心字段:
|
||||
| 主题 | 文档 |
|
||||
|---|---|
|
||||
| 实现与完整特性清单 | [docs/features.md](docs/features.md) |
|
||||
| 配置:`config.yaml`、自动探测模型、多密钥 | [docs/configuration.md](docs/configuration.md) |
|
||||
| 路由:模型路由、AUTO 链、冷却与半冷却、源与适配器、源模板 | [docs/routing.md](docs/routing.md) |
|
||||
| 适配器:Lua 协议、内置适配器清单 | [docs/adapters.md](docs/adapters.md) · [docs/lua-adapters.md](docs/lua-adapters.md)([EN](docs/lua-adapters-en.md)) |
|
||||
| 插件:SDK 契约、钩子、UI 扩展、HTTP 端点 | [docs/plugins.md](docs/plugins.md) |
|
||||
| 桌面端:Headless / Desktop 两种产物、构建与运行 | [docs/gui.md](docs/gui.md) |
|
||||
| 性能:实测内存占用与调优、thinking 预算 | [docs/performance.md](docs/performance.md) |
|
||||
| WebUI 使用 | [docs/webui.md](docs/webui.md) |
|
||||
| 开发:目录职责、测试、分支与发布纪律 | [docs/development.md](docs/development.md) · [docs/git-workflow.md](docs/git-workflow.md)([EN](docs/git-workflow-en.md)) |
|
||||
| 部署与迁移 | [docs/deployment.md](docs/deployment.md) |
|
||||
|
||||
```yaml
|
||||
listen: 127.0.0.1:8080 # 网关监听地址(建议绑内网/回环)
|
||||
gateway_keys: [sk-gw-<随机>] # 初始 admin 密钥(每次安装随机生成,仅首启写入运行时存储用)
|
||||
default_model: AUTO # model 无法路由时自动按优先级链选源
|
||||
adapter_dir: adapters # Lua 适配目录;目录不存在时首启创建并 seed 内置,存在则只读
|
||||
runtime_file: runtime.json # WebUI 编辑的源/密钥/AUTO 链持久化到此文件
|
||||
## 从 llmsproxy 改名
|
||||
|
||||
sources:
|
||||
- name: deepseek
|
||||
base_url: https://api.deepseek.com
|
||||
api_key: sk-... # 也可用 api_key_env: SOME_ENV 引用环境变量(不落盘明文)
|
||||
adapter: deepseek
|
||||
max_concurrent: 8
|
||||
models:
|
||||
- id: deepseek-v4-flash
|
||||
priority: 100 # YAML 源首次启动会 seed 进 AUTO 链的初值
|
||||
kind: chat
|
||||
- id: deepseek-v4-pro
|
||||
priority: 60
|
||||
kind: chat
|
||||
# 静态请求头(比适配器默认优先)
|
||||
headers: { X-Tenant: prod }
|
||||
# 传给 Lua build_headers 的透传元数据
|
||||
meta: { app_id: x, app_secret: y }
|
||||
temperature: 0.7
|
||||
max_tokens: 4096
|
||||
timeout: 120s # 请求超时,默认 120s
|
||||
```
|
||||
产品名早已是 ModelRouter,二进制、配置目录、服务名与环境变量现已统一为
|
||||
`modelrouter`:
|
||||
|
||||
| 旧 | 新 |
|
||||
|---|---|
|
||||
| `llmsproxy`(二进制) | `modelrouter` |
|
||||
| `/etc/llmsproxy/` | `/etc/modelrouter/` |
|
||||
| `llmsproxy.service` | `modelrouter.service` |
|
||||
| `LLMSPROXY_*` | `MODELROUTER_*` |
|
||||
|
||||
### 自动探测模型列表
|
||||
|
||||
不想手抄模型 id 时,只填 URL 和 key,让网关去扫:
|
||||
已部署的机器用迁移脚本升级(**默认 dry-run**,复制不移动,可回滚):
|
||||
|
||||
```bash
|
||||
# 只探测,不写任何东西
|
||||
curl -X POST http://127.0.0.1:8080/api/sources/discover \
|
||||
-H "Authorization: Bearer sk-gw-…" -H "Content-Type: application/json" \
|
||||
-d '{"base_url":"https://api.vendor.com/v1","api_key":"sk-…"}'
|
||||
# → {"ok":true,"base_url":"…","endpoint":"","models":["gpt-x",…],
|
||||
# "image_models":["gpt-image-2"],"models_path":"/models","latency_ms":1387,…}
|
||||
|
||||
# 探测后直接落库
|
||||
curl -X POST http://127.0.0.1:8080/api/sources/discover/apply \
|
||||
-H "Authorization: Bearer sk-gw-…" -H "Content-Type: application/json" \
|
||||
-d '{"base_url":"https://api.vendor.com/v1","api_key":"sk-…","name":"vendor"}'
|
||||
sudo ./packaging/migrate-to-modelrouter.sh # 预演
|
||||
sudo ./packaging/migrate-to-modelrouter.sh --apply # 执行
|
||||
sudo ./packaging/migrate-to-modelrouter.sh --rollback --apply
|
||||
```
|
||||
|
||||
WebUI 源编辑框里也有「自动探测模型」按钮:填好 URL 和 key 点一下,列表自动
|
||||
填好,仍需点保存才生效。
|
||||
|
||||
几点值得知道:
|
||||
|
||||
- **URL 形态不用猜。** 厂商对「`/v1` 放 base 还是放 path」没有共识,探测会
|
||||
依次尝试若干候选并返回答对的那个,同时给出归一化后的 `base_url` +
|
||||
`endpoint`(`endpoint` 平时留空,因为 openai adapter 自带 `/chat/completions`)。
|
||||
- **`image_endpoint` 是另一套规则。** 网关的图像默认路径是**绝对**的
|
||||
`/v1/images/generations`,而 chat 默认是相对的。所以 `base_url` 带 `/v1` 时
|
||||
必须显式给 `image_endpoint: /images/generations`,否则图像请求 404。
|
||||
探测会替你填好。
|
||||
- **图像模型只看上游自己声明的 modality**,绝不从模型名猜。`gpt-image-2`
|
||||
这类 id 也出现在同时提供对话服务的目录上,猜错会让 AUTO 链选中一个只支持
|
||||
图像生成的模型,其后每一层都被它拖垮。
|
||||
- **不会逐模型探活。** 34 个模型的目录跑一轮要花钱且耗时数分钟,`/models`
|
||||
是足够便宜的信号;模型是否真能应答留给正常请求去发现。
|
||||
- **对已存在的源,必须显式 `replace_models: true`** 才会覆盖模型列表。重新
|
||||
探测顺手抹掉为 AUTO 链手工调过的 priority,是最难排查的一类数据丢失。
|
||||
- **`api_key` 留空表示沿用已存值**,不会把凭据清空。
|
||||
|
||||
|
||||
`config.yaml`、`runtime.json`、`master.key`、`adapters/` 均不提交版本库(见
|
||||
`.gitignore`),请随备份带走——`master.key` 丢失后运行时密文无法解密。
|
||||
|
||||
运行文件生成在 `-config` 所在目录:`adapter_dir` / `runtime_file` 自动指向
|
||||
配置文件旁边的 `adapters/` 与 `runtime.json`,从任意工作目录启动都可用。
|
||||
|
||||
运行时文件(`runtime_file`)中的敏感字段自动加密:
|
||||
|
||||
- 加密算法 AES-256-GCM,格式 `enc:v1:<base64>`。
|
||||
- 主密钥来源:环境变量 `LLMS_PROXY_MASTER_KEY`(64 位 hex);否则读取
|
||||
`runtime_file` 同目录的 `master.key`;都不存在时首次启动自动生成 `master.key`(0600)。
|
||||
- 升级时旧明文文件自动兼容:首次运行正常读取,任何 UI 保存操作触发全文件加密迁移。
|
||||
- 注意:**master.key 丢失后密文无法解密**,请随配置一起备份;切勿提交到版本库。
|
||||
|
||||
### 网关密钥(多密钥)
|
||||
|
||||
网关鉴权采用「多密钥 + 角色 + 模型范围」架构,密钥持久化在 `runtime_file` 的
|
||||
`keys` 字段(加密存储):
|
||||
|
||||
- `gateway_keys` 配置只是**初始 admin 密钥种子**:首次启动迁移为运行时 admin
|
||||
key,之后不再参与鉴权管理。
|
||||
- **重要:首次启动后请在 WebUI「密钥」页更换管理员密钥**——初始密钥明文写在
|
||||
`config.yaml` 里(每次安装随机生成),继续使用存在被盗风险;用新密钥登录后删除初始密钥。
|
||||
- WebUI **密钥页**可创建/删除密钥;每个密钥可指定 `admin`(管理全部)或
|
||||
`user`(仅看自己的 key)角色,并配置**模型范围**(模型 + 源 + token 配额 +
|
||||
重置周期)。
|
||||
- 客户端用任意一个已授权的密钥明文作为 Bearer(`Authorization: Bearer <key>`)。
|
||||
- 删除密钥即从运行时存储移除,立即失效。
|
||||
|
||||
#### 密钥用量配额
|
||||
|
||||
**配额按模型单独设置**:每个密钥的 `models[]` 里,每一条模型范围各自带一份
|
||||
token 配额与请求数配额。一个模型用满只影响该模型,同一密钥的其它模型照常工作。
|
||||
|
||||
```yaml
|
||||
keys:
|
||||
- key: sk-gw-<hex>
|
||||
role: user
|
||||
name: agent-alice
|
||||
models:
|
||||
- model: deepseek-v4-flash
|
||||
token_quota: 1000000 # 本周期内该 key 用这个模型的 token 预算
|
||||
req_quota: 20000 # 本周期内的请求次数
|
||||
period: nhour # "" | hour | week | month | nhour
|
||||
hours: 6 # 仅 nhour
|
||||
- model: AUTO
|
||||
token_quota: 5000000 # AUTO 也是一条独立配额
|
||||
period: day # ← 这种写法会被拒绝(词表只有 hour/week/month/nhour)
|
||||
- model: kimi-k3 # 未列配额 = 无限
|
||||
```
|
||||
|
||||
- **没有「整钥总配额」**:这是刻意的设计。整钥总额会让「把 A 模型的额度挪给
|
||||
B 模型」变成一次全局重分配;按模型独立计费则每个模型各自可控,运维可以
|
||||
看出哪个模型吃掉了预算。
|
||||
- `period` 词表:空 = 永不过期(累计总量),`hour` / `week` / `month` = 固定窗口,
|
||||
`nhour` + `hours` = 自定义小时数。**拼错的周期在写入时就被拒**,不会静默变成
|
||||
永不过期。
|
||||
- 配额严格按密钥隔离,且**同一密钥内按模型隔离**:A 密钥用满 `m1` 不会消耗
|
||||
B 密钥的额度,同一密钥的 `m2` 也不受影响。
|
||||
- 配额统计含聊天、流式、生图,跨重启从审计日志回放(保留 40 天,覆盖最长的
|
||||
month 窗口)。
|
||||
- 配额耗尽返回 **429 + `Retry-After`**(`rate_limit_exceeded`),消息里点名是哪个
|
||||
模型用满了,客户端可等窗口重置后自动恢复;模型越权才是 403。
|
||||
**admin 密钥永不受配额限制**(其 scope 上的配额也不强制),避免把管理员
|
||||
锁在门外。
|
||||
- 窗口用量按整点小时分桶统计,实际释放比配置窗口最多晚 1 小时(配额宁可晚释放
|
||||
也不超发)。
|
||||
- `PUT /api/keys/{key}` 提交 `models` 即同时提交它们的配额(配额就是 scope 的一部分,
|
||||
不存在会与模型列表脱节的第二份预算)。显式 `0` = 解除该模型的限制。
|
||||
|
||||
##### 配额拒绝 vs 容量拒绝:两种「拒绝」含义不同
|
||||
|
||||
配额在**入口**检查,不占用任何上游槽位,因此是廉价拒绝;容量不足则要走
|
||||
`busyWait` 有界等待后才返回 503。实测(容量 8、0.6s/请求、100 并发):
|
||||
|
||||
| 情况 | 状态码 | 延迟 | 是否打到上游 |
|
||||
|---|---|---|---|
|
||||
| 配额用尽(`req_quota` 50 → 100 并发) | `429 rate_limit_exceeded` + `Retry-After` | **19–21ms** | 否 |
|
||||
| 容量打满(同一拓扑,未限配额) | `503 upstream_error` | ~2.6s(`busyWait`) | 超出部分否 |
|
||||
|
||||
客户端因此可以区分「等窗口重置」与「等上游腾出容量」:前者只需等待,后者
|
||||
通常该降并发或换源。
|
||||
|
||||
##### 配额的开销
|
||||
|
||||
配额桶按 (密钥, 模型, 整点小时) 分桶保留 40 天,实测(AMD 7840HS):
|
||||
|
||||
- 窗口查询按窗口长度而非保留总量扫描:1h 窗口 49ns、24h 窗口 55ns、
|
||||
30d 窗口 3.9μs(此前全扫保留总量,960 桶时 5.9μs)。
|
||||
- 记录一条请求:283ns、3 分配(与引入配额前相同,分配来自 ring buffer)。
|
||||
- 内存:生产形态(7 密钥 × 8 模型 × 2 源 × 满 40 天 retention)约 **3.7MB**。
|
||||
按源 pin 的 `source::model` 桶**惰性创建**——只有当某条配额真的 pin 了
|
||||
某个源时才维护,否则每条记录多写一份桶,在 20 密钥 × 8 模型 × 3 源下会
|
||||
多占 18MB。
|
||||
代价:配置 pinned 配额之前的历史用量无法事后按源拆分,所以 **pinned 配额的
|
||||
首个窗口可能少算**。
|
||||
|
||||
```bash
|
||||
# 配额耗尽时客户端看到(点名了具体模型)
|
||||
HTTP/1.1 429 Too Many Requests
|
||||
Retry-After: 2100
|
||||
{"error":{"type":"rate_limit_exceeded","message":"token quota exceeded for \"deepseek-v4-flash\" (5000000/5000000)"}}
|
||||
```
|
||||
|
||||
### 模型路由
|
||||
|
||||
`/v1/chat/completions` 的 `model` 解析顺序:
|
||||
|
||||
1. `source/model` 或 `source:model` 前缀 → 指定源;
|
||||
2. 精确匹配某个源的 `model`;
|
||||
3. 配置 `default_model: AUTO` 时 → 走优先级页保存的 AUTO 链(见下);
|
||||
4. 否则回落(返回错误)。
|
||||
|
||||
任何 OpenAI 客户端,只要 `model` 设为某个源的 `name/任意名`,即可锁定走该源;
|
||||
设为 `AUTO`(或网关配了 `default_model: AUTO`)即自动按优先级链选源。
|
||||
|
||||
### AUTO 链(优先级页)
|
||||
|
||||
AUTO 调度**只**由优先级页保存的规则决定,源配置里的 `models[].priority` 数字不再
|
||||
参与调度、也不再显示。**聊天与生图是两套独立的链**,在优先级页面通过「聊天 / 生图」
|
||||
按钮切换编辑:
|
||||
|
||||
- **聊天链**(`runtime_file` 的 `auto` 字段):服务 `POST /v1/chat/completions`
|
||||
的 `model: AUTO` 请求;只接受 `kind: chat` 模型,配置在链里的生图槽位会被忽略。
|
||||
- **生图链**(`runtime_file` 的 `auto_image` 字段):服务 `POST /v1/images/generations`
|
||||
的 `model: AUTO` 请求;只接受 `kind: image` 模型。未配置生图链时,AUTO 生图回落
|
||||
到「所有含生图模型的源、按注册顺序」的旧行为,兼容旧配置。
|
||||
|
||||
每条规则 = 一个「槽位」:`{ model, source, tier, token_quota, period, hours }`。
|
||||
|
||||
- `tier` 表示优先级档位:同一档的模型并排、共享该优先级;档位从上到下递减。
|
||||
- 同一模型可配置多个槽位(如 A 源低配 → B 源低配 → A 源高配 → B 源高配),按档位顺延。
|
||||
- `token_quota` > 0 时该槽位在重置周期内用满即顺延到下一槽位;`period` 支持
|
||||
`hour` / `week` / `month` / `nhour`(配合 `hours`),空 = 不限。
|
||||
- 同 tier 内按「偏好分」排序,失败槽位冷却后自动跳过;冷却 / 配额耗尽 / hard
|
||||
错误会顺延到下一 tier,全部失败时返回 503 并附上每档失败摘要。
|
||||
|
||||
### 冷却与半冷却探测(自愈调度)
|
||||
|
||||
槽位失败后进入指数退避冷却(5s → 10s → 20s …,上限 **5 分钟**)。冷却不是全时段
|
||||
硬闸:
|
||||
|
||||
- **前半段完全静默**:刚失败的槽位不接任何流量,避免对故障上游持续施压。
|
||||
- **后半段放行 1 个探测**:窗口过半后,同一 `(源, 模型)` 最多允许 **一个** 请求作为
|
||||
探测通过。探测槽数 = `clamp(max_concurrent / 10, 1, 2)`——`max_concurrent: 10`
|
||||
的源恰好放行 1 个探测请求。
|
||||
- **探测排在最后**:同 tier 内探测候选永远排在健康候选之后,只有没有正常槽可用时
|
||||
才承接真实流量,因此探测不会抢走可用容量。
|
||||
- **探测即真实请求**:成功则立即清零失败计数与冷却,槽位当场回到正常轮询——
|
||||
上游恢复/额度恢复后不必等满整个冷却窗口。失败则重开一个新窗口,下次探测顺延到
|
||||
新窗口的中点,不会连续重试。
|
||||
|
||||
分类冷却策略:
|
||||
|
||||
| 情况 | 冷却 | 说明 |
|
||||
|---|---|---|
|
||||
| 传输错误 / 5xx | 5s→10s→20s…,上限 5min | 指数退避 |
|
||||
| 401 / 403 凭据错误 | 10min | 换 key 后由下一次探测自动接回 |
|
||||
| 429 限流 | 30s 固定 | 「太快」不等于「坏了」,不进指数阶梯 |
|
||||
| 配额耗尽(`insufficient_quota` / `余额不足` 等) | 对齐配额窗口,上限 30min | **不进指数阶梯、不涨失败计数**,额度恢复即可用 |
|
||||
|
||||
WebUI「优先级」页的槽位健康标签会区分 `冷却` / `待探测` / `探测中`,悬停可看窗口
|
||||
起点、探测放行时刻与完全恢复时刻。
|
||||
|
||||
### 源(Source)与适配器(Adapter)的关系
|
||||
|
||||
- **源** 是到某个上游的连接描述:`name`、`base_url`、`api_key`、模型列表与优先级。
|
||||
- **适配器** 是协议转换逻辑(Lua 脚本):把统一 OpenAI 格式请求转成上游原生格式,
|
||||
再把上游响应/流式分块转回统一格式。
|
||||
- 一个适配器可被多个源复用(如 `openai.lua` 同时服务多个 OpenAI 兼容站点);
|
||||
同一个源也可以切换不同适配器(改 `adapter` 字段即可)。
|
||||
- 源决定“连谁、暴露哪些模型”,适配器决定“怎么对话”——二者在 `sources[]` 条目中
|
||||
通过 `adapter` 字段关联。
|
||||
|
||||
加载流程(`internal/core` 的 `Core` 负责装配):
|
||||
|
||||
1. 启动时 `lua.NewVM(adapter_dir)` 加载全部适配器:内置适配器(编译期 embed)+
|
||||
`adapter_dir` 下的同名覆盖文件;内置文件写在代码内,覆盖文件要求更高优先级。
|
||||
2. `config.Load` 读取 `config.yaml`,`config.NewStore(runtime_file)` 读 WebUI 改动的
|
||||
运行时源,二者按名称合并成完整源列表。
|
||||
3. `rebuildRegistry` 为每个源创建 `provider.Provider`(持有目标适配器),并按源的
|
||||
并发上限配置适配器 worker 池**上限**(不预分配,见下)。
|
||||
4. 请求进来时 `Registry` 按 `model` 路由到 Provider,Provider 调适配器
|
||||
`transform_request` → HTTP 发送 → `transform_response` / `transform_stream_chunk`。
|
||||
|
||||
WebUI 上的"新增/编辑源"、"上传 Lua 适配器"、"改 AUTO 优先级链"与"密钥模型范围"
|
||||
都即时生效(写入运行时文件或 `adapter_dir` 后重新装配,无需重启);直接编辑
|
||||
`config.yaml` / `adapter_dir` 下的文件则需要重启进程才会重新加载。
|
||||
|
||||
在 WebUI 删除源/适配器时:运行时(WebUI 创建)的源与 `.lua` 文件会真正移除;
|
||||
`config.yaml` 中定义的基源无法改写配置文件,采用删除标记隐藏(重启后仍隐藏),
|
||||
在 UI 里重新添加同名源即可恢复。
|
||||
|
||||
### 源模板(多 key 负载均衡)
|
||||
|
||||
多个上游 key 共享同一套配置(URL / 适配器 / 模型列表 / 并发 / rpm)时,逐一复制
|
||||
整段 source 太冗余。**源模板**保存除 `name` 和 `api_key` 以外的全部源字段,
|
||||
从模板一键创建多个仅 key 不同的源,天然支持多 key 负载均衡(每个展开源独立调度、
|
||||
独立冷却、独立健康状态)。
|
||||
|
||||
- WebUI「源」页右上角「**模板管理**」按钮:列出全部模板,可编辑 / 删除 / 新建。
|
||||
- 添加源弹窗右上角两个按钮:
|
||||
- **从模板创建** → 选模板 → 表单自动填充(name + key 仍需你填);
|
||||
- **存为模板** → 把当前表单非 key/name 字段存为新模板。
|
||||
- 模板持久化在 `runtime_file` 的 `source_templates` 字段,与运行时源一样热管理,
|
||||
改完即时生效。模板本身不含密钥,可以安全共享 / 纳管。
|
||||
|
||||
模板不会自己变成源——它只是「配方」。真正承担流量的是从模板创建出来的、带 key
|
||||
的具体源。
|
||||
|
||||
### 内存占用(实测与调优)
|
||||
|
||||
内存不是一个定数:它随**配置的源数量**增长(每个源一个 `http.Transport` 与连接池、
|
||||
一组健康状态),与运行时长无关。本机实测(Linux x86_64,12 核):
|
||||
|
||||
| 部署形态 | 启动 RSS | 稳态 RSS |
|
||||
|---|---|---|
|
||||
| 1 源 / 1 适配器(最小配置) | ~4 MB | ~10 MB |
|
||||
| 1 源 + 29 MB 历史审计日志 | ~19 MB | ~20 MB |
|
||||
| **16 源 / 13 适配器 / 59 模型(本机生产)** | ~28 MB | **~32–35 MB** |
|
||||
|
||||
> 历史参考:本项优化前同一生产配置为 **~105 MB**。降幅来自三处:审计日志的**原始记录**
|
||||
> 不再常驻内存(约 25 MB;统计聚合仍扫全量,但扫完即释放)、Lua 状态池不再单调增长、
|
||||
> 以及下面两个运行时开关。
|
||||
|
||||
内存构成(生产实例分段测量,`/proc/<pid>/smaps`):
|
||||
|
||||
| 区域 | RSS | 说明 |
|
||||
|---|---|---|
|
||||
| Go 堆 | ~14 MB | provider/registry/scheduler 结构 + 连接池缓冲 |
|
||||
| 其他匹名(线程栈 / LuaJIT chunk / runtime) | ~12 MB | 与线程数、已加载适配器数相关 |
|
||||
| 二进制 text+rodata | ~8 MB | 映射的可执行文件页(只读、可被内核回收) |
|
||||
| Go runtime 预留 | ~4 MB | VSZ 上看到的 2 GB+ 是地址空间预留,不占物理内存 |
|
||||
| 共享库 | ~3 MB | libc / libluajit / libm |
|
||||
|
||||
**两个推荐的部署开关**(只适用于环境变量,无需改代码):
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/llmsproxy.service
|
||||
Environment=GOGC=50
|
||||
Environment=MALLOC_ARENA_MAX=2
|
||||
```
|
||||
|
||||
- `MALLOC_ARENA_MAX=2`:LuaJIT 的分配走 cgo → glibc malloc,glibc 默认允许 `8×nproc`
|
||||
个 per-thread arena,每个碰到 malloc 的 OS 线程会占用一个(各约 1 MB,且**不归还给系统**)。
|
||||
实测从 8–15 个 arena(约 7–12 MB)降到 **0**。
|
||||
- `GOGC=50`:把 Go 堆增长目标减半。**单独使用无效**(省下的堆会立即被更多 glibc
|
||||
arena 吃掉,实测 20.3 → 21.5 MB 反而变大),必须与 `MALLOC_ARENA_MAX` 配合,
|
||||
两者同时开启才降 **~19%**。网关是 I/O 密集型(本机 9 小时仅消耗 1min10s CPU),
|
||||
多出的 GC 周期与它的空闲 CPU 相比可忽略。
|
||||
|
||||
> 桌面版(Electron)已在 `cmd/gui/main.js` 里默认为内嵌核心注入这两个开关;手动 export 同名
|
||||
> 环境变量可覆盖。如果你用自己的 systemd unit / 容器,建议照上面加上。
|
||||
|
||||
可选:`GOMEMLIMIT=48MiB` 作软上限,实测再省 ~1 MB;代价是逐近上限时 GC 转激进,
|
||||
温和场景不必开。
|
||||
|
||||
### thinking 预算
|
||||
|
||||
请求体带 `"thinking": {"enabled": false}` 时关闭推理;`"thinking": {"enabled": true,
|
||||
"budget_tokens": 4096}` 指定预算。内核只定义这个**中立**语义字段,映射到具体协议由
|
||||
适配器负责(Anthropic `thinking.budget_tokens`、Gemini `thinkingConfig.thinkingBudget`、
|
||||
DeepSeek `thinking.type = "disabled"`)。内核不再向适配器泄漏 `disable_thinking` /
|
||||
`extra_body` 这类私有字段。
|
||||
|
||||
### WebUI
|
||||
|
||||
内置管理界面(`GET /`),登录后可在浏览器完成:
|
||||
|
||||
- **状态页**:源在线状态(实时探测 + 悬停看错误)、模型/源/key 用量统计、请求记录,
|
||||
支持按时间范围导出 CSV;点击模型可生成 pin 到该模型的连接配置。
|
||||
- **对话页**:流式/非流式调试。
|
||||
- **密钥页**:创建/编辑网关 key,为每个 key 配模型范围(模型 + 源 + token 配额 + 周期),
|
||||
管理员管理全部 key,用户只看到自己的 key。每个模型砖块显示自己的配额徽标
|
||||
(如 `1.0K 77×·1h`,未设配额显示 `∞`),点开可编辑该模型的 token 配额、
|
||||
请求数配额与重置周期 —— 配额按模型独立生效,一个用满不影响同一 key 的其它模型。
|
||||
「我的密钥」页对用户展示本 key 的模型范围与各自配额。
|
||||
- **优先级页**:拖拽积木配置 AUTO 链档位。
|
||||
- **源页**:在线增删改上游源(API key 等敏感字段加密落盘)。
|
||||
- **适配器页**:上传 / 删除 Lua 适配器脚本。
|
||||
|
||||
改动写入 `runtime_file`(重启仍生效)。
|
||||
|
||||
## Lua 适配器协议
|
||||
|
||||
完整 API 见 **[Lua 适配器 API 文档](docs/lua-adapters.md)**([English](docs/lua-adapters-en.md))。
|
||||
|
||||
每个适配器是一个返回 table 的 Lua 脚本(`internal/lua/adapters/<name>.lua`),
|
||||
加载时可被 `adapter_dir` 下的同名脚本覆盖——**改 Lua 脚本无需重编译**,
|
||||
重启即生效;通过 WebUI 上传的适配器与在线编辑的源配置则即时生效。
|
||||
|
||||
```lua
|
||||
return {
|
||||
name = "mysrc",
|
||||
version = "1.0.0",
|
||||
endpoint = "/chat/completions", -- 上游路径(可被 source.endpoint 覆盖)
|
||||
headers = { ["X-Static"] = "v" }, -- 静态默认请求头(build_headers 缺省时使用)
|
||||
|
||||
-- 请求转换:把 OpenAI 格式 req 转成上游原生格式,返回字符串
|
||||
transform_request = function(raw_json) ... end,
|
||||
|
||||
-- 响应转换:把上游原生响应转成统一格式字符串
|
||||
-- { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
|
||||
transform_response = function(raw_json) ... end,
|
||||
|
||||
-- 流式分块转换:把上游 SSE data 转成 { content, done, ... },返回 "" 则跳过
|
||||
transform_stream_chunk = function(raw_chunk) ... end,
|
||||
|
||||
-- [可选] 动态请求头/签名钩子
|
||||
-- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
|
||||
build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
|
||||
|
||||
-- [可选] 出向序列化(契约 v2):上游不是 OpenAI 协议时接管写给客户端的字节。
|
||||
-- 不声明则沿用内核的 OpenAI 兜底;声明即完全取代,内核不再混合字段。
|
||||
serialize_response = function(result) return body_str end,
|
||||
serialize_open_preamble = function(result) return bytes end,
|
||||
serialize_chunk = function(delta) return bytes end,
|
||||
serialize_close = function(result) return bytes end,
|
||||
}
|
||||
```
|
||||
|
||||
内置信号辅助:`hmac_sha256_hex(key, data)`、`sha256_hex(data)`、`base64_encode(s)`、
|
||||
`tohex(s)`、`json.encode/decode`、`log(level, msg)`。
|
||||
|
||||
### 内置适配器
|
||||
|
||||
`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama` `kimicode`
|
||||
`qwen38` `sensenova` `trae`。
|
||||
|
||||
`anthropic`/`gemini`/`ollama` 适配器内置多模态转换(`image_url` → 各自上游格式);
|
||||
推理开关由中立的 `thinking` 字段表达,`deepseek` 适配器把它映射为
|
||||
`thinking.type = "disabled"`。
|
||||
|
||||
**kimicode** 是展示 `build_headers` 的样例:云端校验调用方 app,需要按
|
||||
`meta.app_secret` 对时间戳+URL+请求体哈希做 HMAC 签名并附 `X-App-Sign` 等头。
|
||||
配好 `sources[].meta.{app_id, app_secret, app_agent}` 即可。
|
||||
|
||||
## 桌面 GUI(Electron)
|
||||
|
||||
可选独立产物,面向内网桌面端用户:**内嵌 ModelRouter 核心**(Clash Verge 形态),
|
||||
一键启停、系统托盘常驻、开机自启、静默启动,内容区一比一内嵌完整 WebUI
|
||||
(状态/对话/密钥/优先级/源/适配器 六个页面,免登录)。服务器用户继续使用纯 Go 二进制。
|
||||
|
||||
### 两种安装包:Headless 与 Desktop
|
||||
|
||||
发布到 Release 的安装包分为两类,面向不同用户:
|
||||
|
||||
- **Headless(服务器版)**:纯核心二进制(无 Electron),配置驱动,适合服务器、
|
||||
容器、systemd 等无人值守场景。发布包含 `deb` + `rpm`(含 systemd unit、内存调优
|
||||
与示例配置)与 `tar.gz`(二进制 + `config.example.yaml` + `adapters/` + README),
|
||||
Windows 为 `.zip`。
|
||||
- **Desktop(桌面版)**:Electron GUI 安装包,内嵌核心,适合个人桌面日常使用。
|
||||
|
||||
### 选择 GUI 还是纯后端
|
||||
|
||||
| 场景 | 推荐 | 理由 |
|
||||
| ------ | ------ | ------ |
|
||||
| 服务器 / 内网网关 / 无人值守常驻 | **Headless**(单二进制) | 极轻量(二进制 ~8MB;RSS 随源数量,单源 ~10MB、16 源 ~32–35MB,见[内存占用](#内存占用实测与调优)),零依赖单进程,直接跑在 systemd/任意容器里,远程管理 |
|
||||
| 个人桌面日常使用 / 多设备内网共享 | **Desktop**(GUI) | 免登录内嵌 WebUI、系统托盘一键启停、开机自启、静默后台,适合不懂命令行的使用者 |
|
||||
| Windows 桌面 | **Desktop** | 纯后端在 Windows 上需自行注册服务,GUI 提供原生托盘/自启体验 |
|
||||
| CI 一键出三平台安装包 | **Desktop 打包脚本** | `make gui-*` 系列出 deb/AppImage/NSIS,可进发行流水线 |
|
||||
|
||||
两者完全同源:GUI 内嵌的就是纯后端的同一份 `llmsproxy` 核心(LuaJIT 版),
|
||||
配置/适配器格式完全一致,可随时互换。
|
||||
|
||||
### 构建与运行
|
||||
|
||||
```bash
|
||||
cd cmd/gui
|
||||
npm install
|
||||
make gui # 或 npm run dev —— 开发运行(需先 make build 生成 bin/llmsproxy)
|
||||
make gui-deb # 本机构建 deb(分享给其它 Linux 用户)
|
||||
make gui-dist # 构建 deb + rpm + AppImage(linux,rpm 需系统 rpmbuild)
|
||||
make core-dist # 构建核心 deb + rpm + tar.gz(服务器版,需 nfpm)
|
||||
make gui-win # 构建 Windows nsis 安装包(本机 mingw + wine)
|
||||
make gui-win-docker # 构建 Windows nsis 安装包(全 docker 自足,含 wine)
|
||||
make gui-dist-dir # 仅解包产物(调试用,不产安装包)
|
||||
```
|
||||
|
||||
产物在 `cmd/build/gui-dist/`:`ModelRouter-<ver>.AppImage`、
|
||||
`modelrouter-gui_<ver>_amd64.deb`、`ModelRouter Setup <ver>.exe`(win);
|
||||
核心包在 `cmd/build/dist/`:`llmsproxy_<ver>_amd64.deb`、`llmsproxy-<ver>.x86_64.rpm`。
|
||||
rpm 需系统 `rpmbuild`。每次打包后都会跑 `packaging/verify-dist.sh` 产物体检
|
||||
(大小下限门禁——1.3.0 曾因宿主 wine 损坏产出 264KB 的残缺 exe 却没有被拦住)。
|
||||
发布时会把产物重命名为 `ModelRouter-Desktop-*`(GUI)与 `ModelRouter-Headless-*`(纯后端)
|
||||
两类上传到 GitCode Release。
|
||||
|
||||
#### Windows 打包(docker 固化方案,推荐)
|
||||
|
||||
`make gui-win-docker` 一步完成:交叉编译核心 + NSIS 打包**全在 docker 内**,宿主机
|
||||
**无需 mingw、无需 wine、无需 node**。镜像内置 wine(含 i386 运行时),修复了
|
||||
此前宿主 wine 损坏(缺 syswow64 → `c0000135`)导致产出 264KB 残缺 exe 的问题。
|
||||
|
||||
- 构建镜像:`cmd/gui/docker/win-builder/Dockerfile`(`golang:1.25` + mingw-w64 +
|
||||
node + wine32/wine64 + 预编译 LuaJIT windows 版 `lua51.dll` 与 import lib)。
|
||||
LuaJIT 源码取自 gitcode 镜像(国内可达),github 兜底;Go 依赖走 `https://goproxy.cn`。
|
||||
- 一键发布:`cmd/gui/scripts/dist-win-docker.sh` —— 核心编译 + electron-builder
|
||||
NSIS 全部在容器内执行(`USE_SYSTEM_WINE=true` 走镜像内的 wine);electron-builder
|
||||
的 ~400MB 工具链缓存挂到 docker volume,二次构建增量复用。
|
||||
- 首次运行自动构建 `modelrouter/win-builder` 镜像(约 5–10 分钟);已存在则直接复用。
|
||||
- 打包完成后自动跑 `packaging/verify-dist.sh` 门禁:exe 低于 5MB / deb·rpm·7z 低于
|
||||
10MB 即失败——残缺安装包不可能再溜进 Release。
|
||||
|
||||
```bash
|
||||
# 全 Windows 发布(核心重编 + NSIS):
|
||||
make gui-win-docker
|
||||
```
|
||||
|
||||
### 特性
|
||||
|
||||
- **内嵌核心**:自动拉起捆绑的 `llmsproxy`(LuaJIT 版),配置/密钥/adapter 存于
|
||||
用户数据目录 `<userData>/profile/`,首次运行自动生成随机 admin key 写入 `keys`
|
||||
(非 seed)并自注入,WebUI 免登录、不会提示"更换初始密钥"。端口可在设置中更改(默认 8787)。
|
||||
- **系统托盘**:内核状态点、开机自启/静默启动开关、启停/重启内核、退出。
|
||||
- **静默启动**:`--silent` 参数或设置项,启动仅驻托盘不弹窗;首次运行始终显示窗口。
|
||||
- **开机自启**:Windows/macOS 用 `setLoginItemSettings`;Linux 写
|
||||
`~/.config/autostart/modelrouter-gui.desktop`,静默开启时自动附加 `--silent`。
|
||||
|
||||
## 目录
|
||||
|
||||
```
|
||||
cmd/llmsproxy # 入口(纯 Go 服务端)
|
||||
cmd/gui # Electron 桌面 GUI(内嵌核心 + 托盘 + 打包)
|
||||
internal/config # YAML 配置加载/校验
|
||||
internal/lua # LuaJIT VM + worker 池 + AdapterCache + 内置适配器 (embed)
|
||||
旧路径与旧环境变量在读取侧仍被接受,因此升级不会因为顺序问题立刻失效。
|
||||
`deploy.sh` 在检测到旧 `/etc/llmsproxy` 而新配置不存在时会**中止**,避免部署出
|
||||
一个「能启动但没有 key / 没有源」的空实例。
|
||||
|
||||
插件 SDK 同名调整:唯一公开入口是 `modelrouter`(`llmsproxy` 仅作兼容别名),
|
||||
详见 [docs/plugins.md](docs/plugins.md)。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```text
|
||||
cmd/modelrouter # 入口(纯 Go 服务端)
|
||||
cmd/gui # Electron 桌面壳(内嵌同一份核心)
|
||||
internal/config # 配置解析、加密存储、运行时文件
|
||||
internal/core # 调度、内核动作、状态与审计
|
||||
internal/lua # LuaJIT VM + 弹性 worker 池 + 内置适配器 (embed)
|
||||
internal/lua/plugins # 可选插件示例(不随二进制/发行包分发)
|
||||
internal/provider # Provider(HTTP) + Registry(路由)
|
||||
internal/gateway # OpenAI 兼容 HTTP 服务 + 鉴权 + SDK/流式 + WebUI
|
||||
internal/gateway # OpenAI 兼容 HTTP 服务 + 鉴权 + 流式 + WebUI
|
||||
internal/types # 统一格式 & OpenAI wire 类型
|
||||
packaging/ # deb/rpm/tar 打包、systemd unit、迁移脚本
|
||||
docs/ # 主题文档
|
||||
```
|
||||
|
||||
## 开发协作
|
||||
|
||||
- **Git 分支工作流**:主分支 / 特性分支 / 发布分支的规范见
|
||||
**[docs/git-workflow.md](docs/git-workflow.md)**([English](docs/git-workflow-en.md))。
|
||||
主线:`main` 永可部署 → 特性合入 `main` → 切 `release/vX.Y.Z` 打 tag 发布 →
|
||||
hotfix 提交发布分支并 cherry-pick 回 `main`。
|
||||
- **Lua 适配器协议**:见 **[docs/lua-adapters.md](docs/lua-adapters.md)**
|
||||
([English](docs/lua-adapters-en.md))。
|
||||
|
||||
## 测试
|
||||
## 开发与测试
|
||||
|
||||
```bash
|
||||
go test -tags luajit ./...
|
||||
go test -tags luajit ./... # 全量
|
||||
go test -tags luajit ./internal/lua # 单包(LuaJIT 相关改动必跑)
|
||||
```
|
||||
|
||||
覆盖:配置校验、适配器加载/变换、签名钩子、Gateway 鉴权、SDK 结算、SSE 流式、
|
||||
模型路由、多模态透传与 thinking 预算。
|
||||
- 分支与发布纪律见 [docs/git-workflow.md](docs/git-workflow.md)。
|
||||
- 插件放置在 `plugin_dir`,网关启动时**不会**创建、覆盖或删除任何插件文件;
|
||||
升级桌面端也不会动它们。
|
||||
|
||||
689
README_EN.md
689
README_EN.md
@ -2,625 +2,148 @@
|
||||
|
||||
> **中文**: [README.md](./README.md)
|
||||
>
|
||||
> **AI-assisted**: This project is developed with AI-assisted coding (code and docs are produced in collaboration with AI, reviewed by humans).
|
||||
> **AI-assisted**: code and documentation were produced with AI assistance and
|
||||
> reviewed by a human.
|
||||
|
||||
A lightweight, unified OpenAI-compatible LLM gateway for internal networks —
|
||||
configure once, and let multiple services (agents, SDKs, bots) share many
|
||||
upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama,
|
||||
KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mode,
|
||||
which routes to the best healthy upstream by configured priority.
|
||||
An intranet gateway for people with several LLM subscriptions: configure once and
|
||||
let multiple services share the upstream capacity. Pick a model explicitly, or let
|
||||
`AUTO` choose by priority among the sources that are currently healthy.
|
||||
|
||||
> **Lightweight, no recompile to add sources**: the gateway is a single Go
|
||||
> binary (8-12 MB, ~8 MB stripped; zero runtime dependencies). Adding or switching an upstream is
|
||||
> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua`
|
||||
> adapter — **no Go changes, no recompile**. Changes submitted through the
|
||||
> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua`
|
||||
> file under `adapter_dir` by hand requires a process restart.
|
||||
The gateway itself is a single Go binary (~10 MB, no runtime dependencies). Adding
|
||||
or switching an upstream is a `sources` entry in `config.yaml` (or the WebUI), or a
|
||||
dropped-in `.lua` adapter — **no Go changes, no rebuild**. Changes submitted through
|
||||
the WebUI take effect immediately; editing `config.yaml` or a `.lua` adapter file
|
||||
directly requires a restart.
|
||||
|
||||
## Overview
|
||||
## What it does
|
||||
|
||||
Exposes a standard `OpenAI Chat Completions` API (`/v1/chat/completions` +
|
||||
`/v1/models`) to your internal network, translating between multiple upstream
|
||||
protocols via **Lua adapters**. Supports one-shot calls, SSE streaming,
|
||||
**AUTO model routing**, multimodal passthrough, and tool calls.
|
||||
|
||||
Extracted and independently evolved from the multi-source LLM adapter layer of
|
||||
[HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent)
|
||||
(`internal/agent/api/provider.go` + `internal/lua/adapters/*`).
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-source**: any number of upstream sources in one process, routed by the
|
||||
request's `model`.
|
||||
- **AUTO mode**: with `default_model: AUTO`, scheduling follows the tiered AUTO
|
||||
chain saved on the Priority page (see below).
|
||||
- **Unified output**: every source speaks OpenAI format (incl.
|
||||
`reasoning_content`, `tool_calls`, `usage`).
|
||||
- **Image generation**: `POST /v1/images/generations`, routed to models with
|
||||
`kind: image`.
|
||||
- **Per-model quota**: each key gives every model its own token and request caps
|
||||
plus a reset period (hour/week/month/custom N hours). One model running out
|
||||
affects only that model — the key's other models keep working. Exhaustion
|
||||
answers 429 + `Retry-After` naming the model, so a client resumes when the
|
||||
window rolls over; admin keys are never capped.
|
||||
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
|
||||
Anthropic/Gemini/Ollama are translated automatically.
|
||||
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
|
||||
pool for safe concurrency. Pools are **elastic**: the sum of `max_concurrent`
|
||||
over an adapter's sources is a ceiling, not a preallocation — states are booted
|
||||
on demand and reclaimed when demand drops, so an idle gateway holds close to
|
||||
zero Lua states. The grow step follows the adapter's max concurrency
|
||||
(`clamp(ceil(max/8), 1, 8)`, and never exceeds the number of queued callers);
|
||||
the shrink step follows the live connection count
|
||||
(`clamp(ceil(slack/(1+in_use)), 1, slack)`), so an idle pool collapses in one
|
||||
round while a busy one gives up a single state at a time.
|
||||
- **On-demand request logs**: dashboard totals are computed from the **full**
|
||||
audit history (streamed once at startup and released — 29 MB / 221k lines in
|
||||
~260 ms), while the raw records are never held in memory: the dashboard loads
|
||||
one screen, scrolling pages the rest straight off disk, CSV export streams in
|
||||
O(1) memory, and leaving the page releases everything.
|
||||
- **Self-healing cooldown**: cooldown is capped at 5 minutes and, past the
|
||||
window's midpoint, exactly one probe request is allowed through; a recovered
|
||||
upstream (or a reset quota) returns to full rotation on that probe instead of
|
||||
waiting out the window.
|
||||
- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request.
|
||||
- **Lua adapter protocol**: each source mounts a `.lua` adapter with
|
||||
`transform_request` / `transform_response` / `transform_stream_chunk` — all
|
||||
protocol differences live in Lua, Go only schedules and proxies.
|
||||
- **Signature / header hooks**: adapters may define `build_headers(meta)` to
|
||||
inject/sign request headers before the HTTP call (e.g. KimiCode style
|
||||
app-validation), with helpers like `hmac_sha256_hex`, `sha256_hex`,
|
||||
`base64_encode`.
|
||||
- **WebUI**: built-in management page to view/add/edit sources & models, edit
|
||||
the AUTO chain, and manage per-key model scopes, persisted to a runtime file.
|
||||
- **Encrypted secrets at rest**: upstream `api_key`, custom header values and
|
||||
gateway keys are stored AES-256-GCM encrypted (`master.key` 0600 next to the
|
||||
runtime file, or `LLMS_PROXY_MASTER_KEY`).
|
||||
- **Live source probing**: the status page probes each source
|
||||
(`GET {base}/models`, falling back to a minimal chat call) without touching
|
||||
the scheduler's backoff state; errors are shown on hover.
|
||||
- **Auth**: the gateway validates client keys via `gateway_keys`; independent
|
||||
from each upstream's own key.
|
||||
- **Streaming**: SSE `chat.completion.chunk` with a role-first chunk and
|
||||
`[DONE]` terminator.
|
||||
- **One OpenAI-compatible API**: `/v1/chat/completions`, `/v1/models`,
|
||||
`/v1/images/generations`. Every source speaks the same shape outwards, including
|
||||
`reasoning_content`, `tool_calls` and `usage`.
|
||||
- **Many sources, one AUTO chain**: configure any number of upstreams; `AUTO`
|
||||
follows the priority tiers saved in the WebUI, fails over between them, cools
|
||||
down broken sources and probes them back to health at half-cool.
|
||||
- **Protocol differences live in Lua**: each source binds a `.lua` adapter that
|
||||
transforms requests, responses and stream chunks both ways, and can sign headers
|
||||
via `build_headers(meta)`. Vendor knowledge stays out of the Go kernel.
|
||||
- **Plugins are independent**: business extensions (channel login, billing, quota
|
||||
probing) are external Lua plugins loaded from `plugin_dir`. They are not compiled
|
||||
into the binary and not shipped in release archives, and they talk to the kernel
|
||||
only through the `modelrouter` SDK.
|
||||
- **Multi-tenant keys**: gateway keys are separate from upstream keys, and can be
|
||||
scoped to a model set, an AUTO chain and a quota.
|
||||
- **Encrypted at rest**: upstream keys, custom header values and gateway keys are
|
||||
sealed with AES-256-GCM; `master.key` is 0600, or injected via
|
||||
`MODELROUTER_MASTER_KEY`.
|
||||
- **Observability**: live source probing, HTTP access log, config-change audit,
|
||||
request and usage CSV export. Writes are kept off the request hot path.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
|
||||
./llmsproxy -config config.yaml # first run generates a default config and prints a random admin key
|
||||
# LuaJIT development libraries are required; -tags luajit is mandatory
|
||||
GOMODCACHE=... GOPROXY=off go build -tags luajit -o modelrouter ./cmd/modelrouter
|
||||
|
||||
# First run generates a default config and prints a random admin key
|
||||
./modelrouter -config config.yaml
|
||||
```
|
||||
|
||||
> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings).
|
||||
> You **must** build with `-tags luajit`; otherwise the built-in superset
|
||||
> gopher-lua path is used (behavior differs slightly).
|
||||
> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings). Without
|
||||
> `-tags luajit` the build falls back to gopher-lua, which behaves slightly
|
||||
> differently.
|
||||
|
||||
> **No config is shipped in the repo** (config files carry real keys). On
|
||||
> first run the binary generates a default config at the `-config` path: a
|
||||
> random admin key (printed to the startup log), loopback-only `127.0.0.1:8080`,
|
||||
> and a keyless source ready to chat. Rotate the admin key in the
|
||||
> WebUI after first login.
|
||||
> **No config in the repo**: the repository ships no config file (they contain
|
||||
> keys). The generated default binds `127.0.0.1:8080` only; rotate the admin key in
|
||||
> the WebUI after the first login.
|
||||
|
||||
```bash
|
||||
# no key -> 401
|
||||
# No key -> 401
|
||||
curl http://127.0.0.1:8080/v1/models
|
||||
|
||||
# one-shot ($KEY = the admin key printed at first startup)
|
||||
# Non-streaming ($KEY = the admin key printed at first start)
|
||||
curl -H "Authorization: Bearer $KEY" \
|
||||
-d '{"model":"AUTO","messages":[{"role":"user","content":"hi"}]}' \
|
||||
http://127.0.0.1:8080/v1/chat/completions
|
||||
|
||||
# streaming
|
||||
# Streaming
|
||||
curl -N -H "Authorization: Bearer $KEY" \
|
||||
-d '{"model":"AUTO","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
|
||||
http://127.0.0.1:8080/v1/chat/completions
|
||||
```
|
||||
|
||||
Any OpenAI SDK works: point `base_url` at the gateway, use any authorized
|
||||
gateway key as the API key.
|
||||
|
||||
## Configuration
|
||||
|
||||
Generated on first run at the `-config` path (default `./config.yaml`). Core fields:
|
||||
|
||||
```yaml
|
||||
listen: 127.0.0.1:8080 # bind address (keep internal/loopback)
|
||||
gateway_keys: [sk-gw-<random>] # initial admin key seed (random per install), migrated to the store on first start
|
||||
default_model: AUTO # when model is unroutable, follow the AUTO chain
|
||||
adapter_dir: adapters # Lua adapter dir; created+seeded if missing, read-only otherwise
|
||||
runtime_file: runtime.json # WebUI-edited sources/keys/AUTO chain persist here
|
||||
|
||||
sources:
|
||||
- name: deepseek
|
||||
base_url: https://api.deepseek.com
|
||||
api_key: sk-... # or api_key_env: SOME_ENV to read from an env var
|
||||
adapter: deepseek
|
||||
max_concurrent: 8
|
||||
models:
|
||||
- id: deepseek-v4-flash
|
||||
priority: 100 # YAML sources seed the AUTO chain on first start
|
||||
kind: chat
|
||||
- id: deepseek-v4-pro
|
||||
priority: 60
|
||||
kind: chat
|
||||
# static headers (take precedence over adapter defaults)
|
||||
headers: { X-Tenant: prod }
|
||||
# passthrough metadata for the Lua build_headers hook
|
||||
meta: { app_id: x, app_secret: y }
|
||||
temperature: 0.7
|
||||
max_tokens: 4096
|
||||
timeout: 120s # request timeout, default 120s
|
||||
```
|
||||
|
||||
Sensitive fields in the runtime file (`runtime_file`) are encrypted at rest:
|
||||
|
||||
- AES-256-GCM, stored as `enc:v1:<base64>`.
|
||||
- Master key source: env var `LLMS_PROXY_MASTER_KEY` (64 hex chars), else a
|
||||
`master.key` file next to the runtime file; if neither exists it is generated
|
||||
on first start (mode 0600).
|
||||
- Old plaintext files still load; the first UI save migrates the whole file to
|
||||
ciphertext.
|
||||
- Back up `master.key` with your config — losing it makes the secrets
|
||||
undecryptable. Do not commit it.
|
||||
|
||||
### Gateway keys (multi-key)
|
||||
|
||||
Gateway auth uses a "multi-key + role + model scope" model. Keys are persisted
|
||||
under the `keys` field of the runtime file (encrypted at rest):
|
||||
|
||||
- The `gateway_keys` config is only an **initial admin key seed** — it is
|
||||
migrated into the runtime store on first start and no longer drives auth.
|
||||
- **Important: after first start, replace the admin key via the WebUI Keys
|
||||
page.** The seed key is written in plaintext in `config.yaml`, so keeping it
|
||||
active is a security risk; create a new admin key, log in with it, then
|
||||
delete the seed key.
|
||||
- The WebUI **Keys page** creates/deletes keys. Each key has a role (`admin`
|
||||
manages everything, `user` sees only its own key) and an optional **model
|
||||
scope** (model + source + token quota + reset period). Each model brick shows
|
||||
its own budget badge (e.g. `1.0K 77×·1h`, `∞` when uncapped); clicking it
|
||||
edits that model's token quota, request quota and reset period. Quotas apply
|
||||
per model, so one model running out never blocks the key's others. The
|
||||
"My key" view shows a user its model scopes and their budgets.
|
||||
- Clients authenticate with any authorized key's plaintext as
|
||||
`Authorization: Bearer <key>`.
|
||||
- Deleting a key removes it from the store immediately.
|
||||
|
||||
#### Per-key usage quota
|
||||
|
||||
**Quotas are per model.** Each key's `models[]` list gives every model its own
|
||||
token budget and request budget. One model running out affects only that model
|
||||
— the key's other models keep working.
|
||||
|
||||
```yaml
|
||||
keys:
|
||||
- key: sk-gw-<hex>
|
||||
role: user
|
||||
name: agent-alice
|
||||
models:
|
||||
- model: deepseek-v4-flash
|
||||
token_quota: 1000000 # this key's token budget for this model
|
||||
req_quota: 20000 # requests within the window
|
||||
period: nhour # "" | hour | week | month | nhour
|
||||
hours: 6 # n-hour only
|
||||
- model: AUTO
|
||||
token_quota: 5000000 # AUTO is a quota entry like any other
|
||||
period: hour
|
||||
- model: kimi-k3 # no quota listed = unlimited
|
||||
```
|
||||
|
||||
- **There is deliberately no key-wide total.** A key-wide cap would make
|
||||
"move A's budget to B" a global reallocation; per-model budgets keep each
|
||||
model independently controllable, so it stays visible which model is
|
||||
actually consuming the spend.
|
||||
- `period`: empty = never resets (lifetime total); `hour` / `week` / `month` =
|
||||
fixed windows; `nhour` + `hours` = a custom hour count. **A misspelled
|
||||
period is rejected at write time** rather than silently becoming a
|
||||
never-resetting quota.
|
||||
- Quotas are isolated per key *and* per model within a key: one key exhausting
|
||||
`m1` never draws on another key's budget, and never blocks the same key's
|
||||
`m2`.
|
||||
- Usage counts chat, streaming and image requests, and survives a restart by
|
||||
replaying the audit log (40 days retained, covering the longest `month`
|
||||
window).
|
||||
- An exhausted quota returns **429 + `Retry-After`**
|
||||
(`rate_limit_exceeded`) and the message names the model that ran out, so a
|
||||
client resumes when the window rolls over; a model the key may not use stays
|
||||
403. **Admin keys are never capped** (quotas on their scopes are not
|
||||
enforced either), so a cap can never lock the operator out.
|
||||
- Buckets are whole unix hours, so a window frees up at most an hour late
|
||||
(deliberately freeing late rather than overspending).
|
||||
- `PUT /api/keys/{key}` submits quotas by submitting `models` — the caps are
|
||||
part of the scope, so there is no second budget that can drift out of sync
|
||||
with the model list. An explicit `0` lifts that model's cap.
|
||||
|
||||
##### Quota rejection vs capacity rejection
|
||||
|
||||
A quota is checked at the entry point and occupies no upstream slot, so it is a
|
||||
cheap rejection. Running out of capacity instead waits out the bounded
|
||||
`busyWait` before answering 503. Measured (capacity 8, 0.6 s/request, 100
|
||||
concurrent clients):
|
||||
|
||||
| Case | Status | Latency | Reaches upstream |
|
||||
|---|---|---|---|
|
||||
| Quota spent (`req_quota` 50, 100 concurrent) | `429 rate_limit_exceeded` + `Retry-After` | **19–21 ms** | no |
|
||||
| Capacity exhausted (same topology, no quota) | `503 upstream_error` | ~2.6 s (`busyWait`) | overflow: no |
|
||||
|
||||
So a client can tell "wait for the window to roll over" apart from "wait for
|
||||
upstream capacity" — the first only needs patience, the second usually means
|
||||
back off concurrency or switch sources.
|
||||
|
||||
##### What the quota costs
|
||||
|
||||
Buckets are kept per (key, model, whole unix hour) for 40 days. Measured on an
|
||||
AMD 7840HS:
|
||||
|
||||
- A window query scans the window, not the whole retention: 49 ns for 1 h,
|
||||
55 ns for 24 h, 3.9 µs for 30 d.
|
||||
- Recording one request: 283 ns, 3 allocations (unchanged from before this
|
||||
feature; the allocations come from the record ring buffer).
|
||||
- Memory: the production shape (7 keys x 8 models x 2 sources at full 40-day
|
||||
retention) costs about **3.7 MB**. The `source::model` bucket used by a
|
||||
source-pinned quota is created **lazily** — it is maintained only once some
|
||||
quota actually pins that source, because writing it on every record would
|
||||
cost an extra 18 MB at 20 keys x 8 models x 3 sources.
|
||||
The trade-off: usage recorded before a pinned quota was configured cannot be
|
||||
split by source afterwards, so **a pinned quota may under-count its first
|
||||
window**.
|
||||
|
||||
```
|
||||
HTTP/1.1 429 Too Many Requests
|
||||
Retry-After: 2100
|
||||
{"error":{"type":"rate_limit_exceeded","message":"token quota exceeded for \"deepseek-v4-flash\" (5000000/5000000)"}}
|
||||
```
|
||||
|
||||
### Model routing
|
||||
|
||||
`/v1/chat/completions` `model` resolution order:
|
||||
|
||||
1. `source/model` or `source:model` prefix → pinned source;
|
||||
2. exact match of a source's `model`;
|
||||
3. with `default_model: AUTO` → follow the AUTO chain from the Priority page
|
||||
(see below);
|
||||
4. otherwise the request errors.
|
||||
|
||||
Any OpenAI client can pin to a source by setting `model` to its
|
||||
`name/anything`; `AUTO` (or the gateway's `default_model: AUTO`) follows the
|
||||
tiered AUTO chain.
|
||||
|
||||
### AUTO chain (Priority page)
|
||||
|
||||
AUTO scheduling is driven **only** by the rules saved on the Priority page. The
|
||||
numeric `models[].priority` in source configs no longer participates in
|
||||
scheduling and is no longer shown. **Chat and image generation are two
|
||||
independent chains**, switched in the Priority page via the `Chat / Image` toggle:
|
||||
|
||||
- **Chat chain** (`auto` field): serves `POST /v1/chat/completions` with
|
||||
`model: AUTO`; only `kind: chat` models are honored (image slots are skipped).
|
||||
- **Image chain** (`auto_image` field): serves `POST /v1/images/generations` with
|
||||
`model: AUTO`; only `kind: image` models are honored. With no image chain
|
||||
configured, AUTO image generation falls back to legacy discovery (all sources
|
||||
exposing an image model, in registry order) for backward compatibility.
|
||||
|
||||
Each rule is one "slot": `{ model, source, tier, token_quota, period, hours }`.
|
||||
|
||||
- `tier` is the priority tier: models in the same tier sit side by side and
|
||||
share it; tiers run high → low.
|
||||
- The same model may appear in several slots (e.g. A low → B low → A high →
|
||||
B high) and is tried in tier order.
|
||||
- `token_quota` > 0 means the slot is skipped once its tokens within the reset
|
||||
window are exhausted; `period` supports `hour` / `week` / `month` / `nhour`
|
||||
(with `hours`); empty = unlimited.
|
||||
- Within a tier, slots run by preference score; cooling / quota-exhausted /
|
||||
hard-failed slots fall through to the next tier, and total failure returns
|
||||
503 with a per-tier summary.
|
||||
|
||||
### Sources vs. adapters
|
||||
|
||||
- A **source** describes a connection to an upstream: `name`, `base_url`,
|
||||
`api_key`, model list and priorities.
|
||||
- An **adapter** is the protocol translation logic (Lua script): converts the
|
||||
unified OpenAI request to the upstream's native format and back.
|
||||
- One adapter serves many sources (e.g. `openai.lua` for any OpenAI-compatible
|
||||
site); one source can switch adapters via its `adapter` field.
|
||||
- Sources decide *whom to talk to and which models to expose*; adapters decide
|
||||
*how to talk*. They are linked by the `adapter` field inside each `sources[]`
|
||||
entry.
|
||||
|
||||
Loading flow (assembled by `Core` in `internal/core`):
|
||||
|
||||
1. On startup `lua.NewVM(adapter_dir)` loads all adapters: built-in ones
|
||||
(embedded at compile time) + same-name override files under `adapter_dir`.
|
||||
2. `config.Load` reads `config.yaml`; `config.NewStore(runtime_file)` reads
|
||||
WebUI-edited runtime sources; both are merged by name.
|
||||
3. `rebuildRegistry` creates a `provider.Provider` per source (holding its
|
||||
adapter), sizing the adapter's worker pool from the source's concurrency
|
||||
limit.
|
||||
4. Incoming requests are routed by `Registry` to a provider, which calls the
|
||||
adapter's `transform_request` → HTTP call → `transform_response` /
|
||||
`transform_stream_chunk`.
|
||||
|
||||
"Add/Edit source", "Upload adapter", "AUTO chain edits" and "per-key model
|
||||
scopes" in the WebUI take effect immediately (written to the runtime file /
|
||||
`adapter_dir`, then reassembled — no restart). Directly editing `config.yaml`
|
||||
or files under `adapter_dir` requires a process restart.
|
||||
|
||||
Deleting a source/adapter in the WebUI truly removes runtime (UI-created)
|
||||
sources and `.lua` files; base sources defined in `config.yaml` cannot rewrite
|
||||
that file, so they are hidden via a deletion tombstone (still hidden after
|
||||
restart) and can be restored by re-adding the same name in the UI.
|
||||
|
||||
### Source templates (multi-key balancing)
|
||||
|
||||
When several upstream keys share one config (URL / adapter / model list /
|
||||
concurrency / rpm), copying the whole source block N times is wasteful. A
|
||||
**source template** stores every source field except `name` and `api_key`, so you
|
||||
create N key-only-different sources from one shared skeleton — natural multi-key
|
||||
load balancing (each expanded source schedules, cools down, and reports health
|
||||
independently).
|
||||
|
||||
- WebUI Sources page, top-right **Templates** button: list all templates with
|
||||
edit / delete / create.
|
||||
- Add-source dialog has two header buttons:
|
||||
- **From template** → pick a template → form auto-fills (name + key still
|
||||
yours to enter);
|
||||
- **As template** → save the current form's non-key/name fields as a template.
|
||||
- Templates persist in the runtime file's `source_templates` field alongside
|
||||
runtime sources (hot-managed, apply immediately). Templates hold no secret
|
||||
keys, so they're safe to share / version-control.
|
||||
|
||||
A template is just a recipe — it doesn't become a source by itself. Only the
|
||||
key-bearing sources created from it carry real traffic.
|
||||
|
||||
### Memory footprint (measured and tuned)
|
||||
|
||||
Memory is not a constant: it scales with the **number of configured sources**
|
||||
(each source owns an `http.Transport` with its connection pool plus per-model
|
||||
health state), not with uptime. Measured on this machine (Linux x86_64, 12 cores):
|
||||
|
||||
| Deployment shape | Startup RSS | Settled RSS |
|
||||
|---|---|---|
|
||||
| 1 source / 1 adapter (minimal) | ~4 MB | ~10 MB |
|
||||
| 1 source + a 29 MB audit history | ~19 MB | ~20 MB |
|
||||
| **16 sources / 13 adapters / 59 models (this host)** | ~28 MB | **~32-35 MB** |
|
||||
|
||||
> For reference, the same production config used **~105 MB** before this round of
|
||||
> work. The reduction comes from three places: raw audit records are no longer kept
|
||||
> resident (~25 MB — the aggregates still scan the full history, but the scan
|
||||
> releases as it goes), Lua state pools no longer grow monotonically, and the two
|
||||
> runtime knobs below.
|
||||
|
||||
Breakdown of the production instance (per-region, from `/proc/<pid>/smaps`):
|
||||
|
||||
| Region | RSS | Notes |
|
||||
|---|---|---|
|
||||
| Go heap | ~14 MB | provider/registry/scheduler state + connection-pool buffers |
|
||||
| Other anonymous (thread stacks / LuaJIT chunks / runtime) | ~12 MB | scales with thread count and loaded adapters |
|
||||
| Binary text+rodata | ~8 MB | mapped executable pages (read-only, reclaimable by the kernel) |
|
||||
| Go runtime reservations | ~4 MB | the 2 GB+ you see in VSZ is address space, not physical memory |
|
||||
| Shared libraries | ~3 MB | libc / libluajit / libm |
|
||||
|
||||
**Two recommended deployment knobs** (environment only, no code change):
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/llmsproxy.service
|
||||
Environment=GOGC=50
|
||||
Environment=MALLOC_ARENA_MAX=2
|
||||
```
|
||||
|
||||
- `MALLOC_ARENA_MAX=2`: LuaJIT allocates through cgo into glibc malloc, and glibc
|
||||
allows up to `8 x nproc` per-thread arenas. Every OS thread that touches malloc
|
||||
claims one (~1 MB each) and **never returns it to the OS**. Measured: 8-15
|
||||
arenas (7-12 MB) down to **0**.
|
||||
- `GOGC=50`: halves the Go heap growth target. **It does nothing on its own** —
|
||||
the heap it saves is immediately consumed by additional glibc arenas (measured
|
||||
20.3 -> 21.5 MB, i.e. slightly worse) — so it must be paired with
|
||||
`MALLOC_ARENA_MAX`. Together they cut settled RSS by **~19%**. The gateway is
|
||||
I/O bound (1min10s of CPU per 9 hours here), so the extra GC cycles are free.
|
||||
|
||||
> The desktop build (Electron) already injects both when spawning the embedded
|
||||
> core (`cmd/gui/main.js`); exporting either variable yourself overrides it. If you
|
||||
> write your own systemd unit or container spec, add them there.
|
||||
|
||||
Optional: `GOMEMLIMIT=48MiB` as a soft ceiling saves roughly 1 MB more, at the
|
||||
cost of GC turning aggressive as the limit approaches. Not worth it for a relaxed
|
||||
deployment.
|
||||
|
||||
### disable_thinking
|
||||
|
||||
With `"disable_thinking": true` in the request body, the gateway passes it to
|
||||
each adapter; the DeepSeek adapter maps it to `extra_body.thinking.type =
|
||||
"disabled"`, other sources follow their own protocol.
|
||||
|
||||
### WebUI
|
||||
|
||||
Built-in admin page at `GET /`; after login you can:
|
||||
|
||||
- **Status**: source online state (live probe, error on hover), per
|
||||
model/source/key usage stats, request records filtered by key, and CSV export
|
||||
over a time range; click a model to pin the connection snippet to it.
|
||||
- **Chat**: streaming / non-streaming debug.
|
||||
- **Keys**: create/edit gateway keys, set per-key model scopes (model + source +
|
||||
token quota + reset period); admins manage all keys, users see only their own.
|
||||
- **Priority**: drag blocks to build the tiered AUTO chain.
|
||||
- **Sources**: add/edit/delete upstream sources online (secrets stored
|
||||
encrypted).
|
||||
- **Adapters**: upload / delete Lua adapter scripts.
|
||||
|
||||
Changes persist to `runtime_file` (survive restarts).
|
||||
|
||||
## Lua adapter protocol
|
||||
|
||||
Full API: **[Lua adapter docs](docs/lua-adapters.md)**
|
||||
([中文](docs/lua-adapters-en.md)).
|
||||
|
||||
Each adapter is a Lua script returning a table
|
||||
(`internal/lua/adapters/<name>.lua`), overridable by a same-name file under
|
||||
`adapter_dir` — **no recompile needed**, restart to take effect; uploads via
|
||||
the WebUI take effect immediately.
|
||||
|
||||
```lua
|
||||
return {
|
||||
name = "mysrc",
|
||||
version = "1.0.0",
|
||||
endpoint = "/chat/completions", -- upstream path (source.endpoint overrides)
|
||||
headers = { ["X-Static"] = "v" }, -- static default headers (fallback)
|
||||
|
||||
-- request: convert unified OpenAI req -> upstream native format, return string
|
||||
transform_request = function(raw_json) ... end,
|
||||
|
||||
-- response: convert upstream response to unified format string
|
||||
-- { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
|
||||
transform_response = function(raw_json) ... end,
|
||||
|
||||
-- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips
|
||||
transform_stream_chunk = function(raw_chunk) ... end,
|
||||
|
||||
-- [optional] dynamic headers / signature hook
|
||||
-- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
|
||||
build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
|
||||
}
|
||||
```
|
||||
|
||||
Shared helpers: `hmac_sha256_hex(key, data)`, `sha256_hex(data)`,
|
||||
`base64_encode(s)`, `tohex(s)`, `json.encode/decode`, `log(level, msg)`.
|
||||
|
||||
### Built-in adapters
|
||||
|
||||
`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama`
|
||||
`kimicode` `qwen38` `sensenova` `trae`.
|
||||
|
||||
`anthropic`/`gemini`/`ollama` include multimodal conversion
|
||||
(`image_url` → their native format); with `disable_thinking` the `deepseek`
|
||||
adapter sets `extra_body.thinking.type` to `disabled`.
|
||||
|
||||
**kimicode** demonstrates `build_headers`: the cloud validates the calling
|
||||
app, so you HMAC-sign timestamp+URL+body with `meta.app_secret` and add
|
||||
`X-App-Sign`-style headers. Configure `sources[].meta.{app_id, app_secret,
|
||||
app_agent}`.
|
||||
|
||||
## Desktop GUI (Electron)
|
||||
|
||||
Optional standalone product for desktop users on your internal network: an
|
||||
**embedded ModelRouter core** (Clash Verge-style): one-click start/stop, system
|
||||
tray, autostart, silent launch, and a pixel-for-pixel embedded full WebUI
|
||||
(Status/Chat/Keys/Priority/Sources/Adapters — no login needed). Server users
|
||||
keep using the plain Go binary.
|
||||
|
||||
### Two installer kinds: Headless and Desktop
|
||||
|
||||
Release installers ship in two flavors for different audiences:
|
||||
|
||||
- **Headless (server)**: the plain core binary (no Electron), config-driven,
|
||||
for servers / containers / systemd / unattended runs. Ships as `deb` + `rpm`
|
||||
(systemd unit with memory tuning + example config) and `tar.gz` (binary +
|
||||
`config.example.yaml` + `adapters/` + README), Windows as `.zip`.
|
||||
- **Desktop (desktop app)**: the Electron GUI installer with an embedded core,
|
||||
for personal daily desktop use.
|
||||
|
||||
### GUI vs. plain backend
|
||||
|
||||
| Scenario | Pick | Why |
|
||||
| ---- | ---- | ---- |
|
||||
| Server / intranet gateway / unattended long-running | **Headless** (single binary) | ~8 MB binary; RSS scales with the number of sources (~10 MB for one, ~32-35 MB for 16 — see [Memory footprint](#memory-footprint-measured-and-tuned)), zero-dep single process — drop it into systemd or any container, remote admin |
|
||||
| Personal desktop daily use / multi-device intranet sharing | **Desktop** (GUI) | no-login embedded WebUI, tray one-click, autostart, silent background — for non-CLI users |
|
||||
| Windows desktop | **Desktop** | plain backend needs manual service registration; GUI ships native tray/autostart |
|
||||
| CI one-shot 3-platform installers | **Desktop packaging scripts** | `make gui-*` emits deb / AppImage / NSIS, drops straight into a release pipeline |
|
||||
|
||||
Both share the exact same `llmsproxy` core (LuaJIT build) — configs and
|
||||
adapters are fully compatible, freely interchangeable.
|
||||
|
||||
### Build & run
|
||||
Any OpenAI SDK works: point `base_url` at the gateway and use an authorized gateway
|
||||
key as `api_key`.
|
||||
|
||||
## Documentation
|
||||
|
||||
The detailed documentation is currently written in Chinese.
|
||||
|
||||
| Topic | Document |
|
||||
|---|---|
|
||||
| Implementation and the full feature list | [docs/features.md](docs/features.md) |
|
||||
| Configuration: `config.yaml`, model discovery, multiple keys | [docs/configuration.md](docs/configuration.md) |
|
||||
| Routing: model routing, AUTO chain, cooldown, sources vs adapters, source templates | [docs/routing.md](docs/routing.md) |
|
||||
| Adapters: Lua protocol and built-in adapters | [docs/adapters.md](docs/adapters.md) · [docs/lua-adapters-en.md](docs/lua-adapters-en.md) (English) |
|
||||
| Plugins: SDK contract, hooks, UI extensions, HTTP endpoints | [docs/plugins.md](docs/plugins.md) |
|
||||
| Desktop app: Headless / Desktop artifacts, build and run | [docs/gui.md](docs/gui.md) |
|
||||
| Performance: measured memory use, tuning, thinking budget | [docs/performance.md](docs/performance.md) |
|
||||
| WebUI usage | [docs/webui.md](docs/webui.md) |
|
||||
| Development: layout, tests, branch and release discipline | [docs/development.md](docs/development.md) · [docs/git-workflow-en.md](docs/git-workflow-en.md) (English) |
|
||||
| Deployment and migration | [docs/deployment.md](docs/deployment.md) |
|
||||
|
||||
## Renamed from llmsproxy
|
||||
|
||||
The product name has been ModelRouter for a while; the binary, config directory,
|
||||
service name and environment variables are now `modelrouter` as well:
|
||||
|
||||
| Before | After |
|
||||
|---|---|
|
||||
| `llmsproxy` (binary) | `modelrouter` |
|
||||
| `/etc/llmsproxy/` | `/etc/modelrouter/` |
|
||||
| `llmsproxy.service` | `modelrouter.service` |
|
||||
| `LLMSPROXY_*` | `MODELROUTER_*` |
|
||||
|
||||
Existing installs migrate with a dry-run-by-default script (copies, never moves,
|
||||
and can roll back):
|
||||
|
||||
```bash
|
||||
cd cmd/gui
|
||||
npm install
|
||||
make gui # or npm run dev — dev run (needs make build first for bin/llmsproxy)
|
||||
make gui-deb # local deb (share with other Linux users)
|
||||
make gui-dist # deb + rpm + AppImage (linux; rpm needs system rpmbuild)
|
||||
make core-dist # core deb + rpm + tar.gz (server edition; needs nfpm)
|
||||
make gui-win # Windows NSIS (host mingw + wine)
|
||||
make gui-win-docker # Windows NSIS, fully dockerized (wine in image; no host mingw/node/wine)
|
||||
sudo ./packaging/migrate-to-modelrouter.sh # preview
|
||||
sudo ./packaging/migrate-to-modelrouter.sh --apply # apply
|
||||
sudo ./packaging/migrate-to-modelrouter.sh --rollback --apply
|
||||
```
|
||||
|
||||
Artifacts land in `cmd/build/gui-dist/`: `ModelRouter-<ver>.AppImage`,
|
||||
`modelrouter-gui_<ver>_amd64.deb`, `ModelRouter Setup <ver>.exe` (win); core
|
||||
packages land in `cmd/build/dist/`: `llmsproxy_<ver>_amd64.deb`,
|
||||
`llmsproxy-<ver>.x86_64.rpm`. Both target families run
|
||||
`packaging/verify-dist.sh` afterwards — a size floor that rejects degenerate
|
||||
installers (1.3.0 shipped a 264 KB NSIS stub because the host wine was broken
|
||||
and nothing caught it). At release time artifacts are renamed into the
|
||||
`ModelRouter-Desktop-*` (GUI) and `ModelRouter-Headless-*` (plain backend)
|
||||
families before upload to GitCode Releases.
|
||||
Old paths and old environment variable names are still accepted on the read side,
|
||||
so an upgrade cannot fail merely because of ordering. `deploy.sh` refuses to run
|
||||
when it finds the old `/etc/llmsproxy` without a new config, because that would
|
||||
deploy a gateway with no key and no sources.
|
||||
|
||||
#### Windows packaging (dockerized, recommended)
|
||||
|
||||
`make gui-win-docker` does cross-compile + NSIS **entirely in docker**; the host
|
||||
needs **no mingw, no wine, no node**. Wine (with the i386 runtime) lives inside
|
||||
the image, fixing the broken-host-wine failure that produced the 264 KB stub
|
||||
installer.
|
||||
|
||||
- Image: `cmd/gui/docker/win-builder/Dockerfile` (`golang:1.25` + mingw-w64 +
|
||||
node + wine32/wine64 + prebuilt LuaJIT for Windows: `lua51.dll` + import lib).
|
||||
LuaJIT source comes from the gitcode mirror (reachable in CN) with github
|
||||
fallback; Go modules go through `https://goproxy.cn`.
|
||||
- One-shot release: `cmd/gui/scripts/dist-win-docker.sh` — core build + NSIS
|
||||
packaging run inside the container (`USE_SYSTEM_WINE=true` uses the image's
|
||||
wine); electron-builder's ~400 MB toolchain cache lives in a docker volume so
|
||||
later builds reuse it.
|
||||
- The `modelrouter/win-builder` image is auto-built on first run (~5-10 min)
|
||||
and reused afterwards.
|
||||
- After packaging, `packaging/verify-dist.sh` runs as a gate: an exe under
|
||||
5 MB (or a deb/rpm/7z under 10 MB) fails the build — a broken installer can
|
||||
never reach a Release again.
|
||||
|
||||
```bash
|
||||
# Full Windows release (rebuild core + NSIS):
|
||||
make gui-win-docker
|
||||
```
|
||||
|
||||
### Features
|
||||
|
||||
- **Embedded core**: auto-spawns the bundled `llmsproxy` (LuaJIT build);
|
||||
config/keys/adapters live in `<userData>/profile/`. First run generates a
|
||||
random admin key written into `keys` (not seed) and self-injects it — WebUI
|
||||
needs no login and never nags about changing the initial key. Port changeable
|
||||
in settings (default 8787).
|
||||
- **System tray**: core status dot, autostart/silent toggles, start/stop/restart
|
||||
core, quit.
|
||||
- **Silent launch**: `--silent` flag or setting — starts to the tray only;
|
||||
first run always shows the window.
|
||||
- **Autostart**: `setLoginItemSettings` on Windows/macOS; writes
|
||||
`~/.config/autostart/modelrouter-gui.desktop` on Linux, appending `--silent`
|
||||
when silent mode is on.
|
||||
The plugin SDK follows the same naming: the only public entry point is
|
||||
`modelrouter` (`llmsproxy` remains a compatibility alias). See
|
||||
[docs/plugins.md](docs/plugins.md).
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
cmd/llmsproxy # entry point
|
||||
internal/config # YAML config loading/validation
|
||||
internal/lua # LuaJIT VM + worker pool + AdapterCache + built-ins (embed)
|
||||
```text
|
||||
cmd/modelrouter # entry point (pure Go server)
|
||||
cmd/gui # Electron desktop shell (embeds the same core)
|
||||
internal/config # config parsing, sealed storage, runtime file
|
||||
internal/core # scheduling, kernel actions, state and audit
|
||||
internal/lua # LuaJIT VM + elastic worker pool + built-in adapters (embed)
|
||||
internal/lua/plugins # optional plugin examples (not shipped in binary/archives)
|
||||
internal/provider # Provider(HTTP) + Registry(routing)
|
||||
internal/gateway # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI
|
||||
internal/types # unified format & OpenAI wire types
|
||||
internal/gateway # OpenAI-compatible HTTP service + auth + streaming + WebUI
|
||||
internal/types # unified format and OpenAI wire types
|
||||
packaging/ # deb/rpm/tar packaging, systemd unit, migration script
|
||||
docs/ # topic documentation
|
||||
```
|
||||
|
||||
## Development & collaboration
|
||||
|
||||
- **Git branching workflow**: main / feature / release-branch conventions in
|
||||
**[docs/git-workflow-en.md](docs/git-workflow-en.md)**
|
||||
([中文](docs/git-workflow.md)). The line: `main` is always deployable →
|
||||
features merge into `main` → cut `release/vX.Y.Z` and tag → release → hotfixes
|
||||
land on the release branch and are cherry-picked back into `main`.
|
||||
- **Lua adapter protocol**: see
|
||||
**[docs/lua-adapters-en.md](docs/lua-adapters-en.md)**
|
||||
([中文](docs/lua-adapters.md)).
|
||||
|
||||
## Tests
|
||||
## Development and tests
|
||||
|
||||
```bash
|
||||
go test -tags luajit ./...
|
||||
go test -tags luajit ./... # everything
|
||||
go test -tags luajit ./internal/lua # one package (required for LuaJIT changes)
|
||||
```
|
||||
|
||||
Covers: config validation, adapter load/transform, signature hooks, gateway
|
||||
auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and
|
||||
disable_thinking.
|
||||
- Branch and release discipline: [docs/git-workflow-en.md](docs/git-workflow-en.md).
|
||||
- Plugins live in `plugin_dir`. The gateway never creates, overwrites or deletes
|
||||
plugin files, and a desktop upgrade never touches them either.
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
// ModelRouter Desktop GUI — main process
|
||||
// Clash-Verge-style: bundles and manages an embedded llmsproxy core instance
|
||||
// Desktop shell: bundles and manages an embedded modelrouter core instance
|
||||
// in a per-user profile dir (userData), auto-starts it, keeps a tray icon,
|
||||
// supports launch-at-login. Server-side users keep using the pure Go binary.
|
||||
const {
|
||||
@ -50,11 +50,27 @@ const CONFIG_FILE = path.join(PROFILE_DIR, "config.yaml");
|
||||
const PLUGINS_DIR = path.join(PROFILE_DIR, "plugins");
|
||||
// Packaged: extraResource at process.resourcesPath/bin (outside asar, spawnable).
|
||||
// Dev: cmd/gui/bin, populated by prepare-core.js.
|
||||
const CORE_EXE = path.join(
|
||||
// The embedded core is named after the product. Installs produced before the
|
||||
// llmsproxy -> modelrouter rename still carry the old file name inside their
|
||||
// profile, so resolution falls back to it instead of declaring the core missing.
|
||||
const CORE_BIN_DIR = path.join(
|
||||
app.isPackaged ? process.resourcesPath : __dirname,
|
||||
"bin",
|
||||
);
|
||||
const CORE_EXE = path.join(
|
||||
CORE_BIN_DIR,
|
||||
process.platform === "win32" ? "modelrouter.exe" : "modelrouter",
|
||||
);
|
||||
const LEGACY_CORE_EXE = path.join(
|
||||
CORE_BIN_DIR,
|
||||
process.platform === "win32" ? "llmsproxy.exe" : "llmsproxy",
|
||||
);
|
||||
|
||||
function resolveCoreExe() {
|
||||
if (fs.existsSync(CORE_EXE)) return CORE_EXE;
|
||||
if (fs.existsSync(LEGACY_CORE_EXE)) return LEGACY_CORE_EXE;
|
||||
return CORE_EXE;
|
||||
}
|
||||
// Packaged window icon: must be a REAL file on disk (extraResource), not an
|
||||
// asar path — Electron/GTK pulls BrowserWindow icons from the filesystem and
|
||||
// crashes (SIGSEGV) when given an asar virtual path on Linux.
|
||||
@ -163,7 +179,7 @@ function unsealViaCore() {
|
||||
let key = "";
|
||||
try {
|
||||
const out = execFileSync(
|
||||
CORE_EXE,
|
||||
resolveCoreExe(),
|
||||
["-config", CONFIG_FILE, "-show-secrets"],
|
||||
{
|
||||
encoding: "utf-8",
|
||||
@ -215,7 +231,7 @@ function coreStarted() {
|
||||
function writeDefaultEmbeddedConfig() {
|
||||
// Generate a self-contained embedded profile with a fresh random admin key.
|
||||
// No upstream sources are preconfigured — the desktop user adds them via
|
||||
// the WebUI's sources page (like Clash Verge's empty-by-default profile).
|
||||
// the WebUI's sources page (empty by default, like the WebUI profile).
|
||||
// If a config already exists we do NOT overwrite it (keep user's edits).
|
||||
if (fs.existsSync(CONFIG_FILE)) return;
|
||||
const key = "sk-gw-" + crypto.randomBytes(16).toString("hex");
|
||||
@ -272,8 +288,9 @@ function startCore() {
|
||||
console.log("core already running");
|
||||
return;
|
||||
}
|
||||
if (!fs.existsSync(CORE_EXE)) {
|
||||
console.error("embedded core binary missing:", CORE_EXE);
|
||||
const coreExe = resolveCoreExe();
|
||||
if (!fs.existsSync(coreExe)) {
|
||||
console.error("embedded core binary missing:", coreExe);
|
||||
return;
|
||||
}
|
||||
// ensure profile dir + config + local adapters exist
|
||||
@ -281,9 +298,8 @@ function startCore() {
|
||||
if (!fs.existsSync(CONFIG_FILE)) writeDefaultEmbeddedConfig();
|
||||
pluginSync.ensurePluginDirConfigured(CONFIG_FILE, PLUGINS_DIR, log);
|
||||
fs.mkdirSync(PLUGINS_DIR, { recursive: true });
|
||||
pluginSync.syncBundledPluginsOnUpgrade(PROFILE_DIR, PLUGINS_DIR, app.getVersion(), log);
|
||||
|
||||
coreProc = spawn(CORE_EXE, ["-config", CONFIG_FILE], {
|
||||
coreProc = spawn(coreExe, ["-config", CONFIG_FILE], {
|
||||
cwd: PROFILE_DIR,
|
||||
env: Object.assign({}, process.env, {
|
||||
LLMS_PROXY_GUI: "1",
|
||||
@ -686,7 +702,7 @@ function createWindow() {
|
||||
});
|
||||
}
|
||||
// Silent start: `--silent` CLI flag or settings.silentStart keeps the app
|
||||
// in the tray without showing the window (Clash-Verge style). First run
|
||||
// in the tray without showing the window (desktop-shell behaviour). First run
|
||||
// always shows the window so the user sees the dashboard.
|
||||
mainWindow.once("ready-to-show", () => {
|
||||
const firstRun = !fs.existsSync(
|
||||
@ -810,8 +826,8 @@ ipcMain.handle("core:state", () => ({
|
||||
port: settings.port || DEFAULT_PORT,
|
||||
profileDir: PROFILE_DIR,
|
||||
configFile: CONFIG_FILE,
|
||||
coreExe: CORE_EXE,
|
||||
coreExists: fs.existsSync(CORE_EXE),
|
||||
coreExe: resolveCoreExe(),
|
||||
coreExists: fs.existsSync(resolveCoreExe()),
|
||||
}));
|
||||
ipcMain.handle("core:start", () => {
|
||||
app.isQuitting = false;
|
||||
@ -883,7 +899,7 @@ if (gotLock) {
|
||||
createWindow();
|
||||
createTray();
|
||||
|
||||
// start embedded core if not already running (another instance of llmsproxy)
|
||||
// start embedded core if not already running (another instance of modelrouter)
|
||||
setTimeout(async () => {
|
||||
const p = await probeEmbedded();
|
||||
if (!p.ok) {
|
||||
|
||||
4
cmd/gui/package-lock.json
generated
4
cmd/gui/package-lock.json
generated
@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "modelrouter-gui",
|
||||
"version": "2.0.44",
|
||||
"version": "3.0.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "modelrouter-gui",
|
||||
"version": "2.0.44",
|
||||
"version": "3.0.0",
|
||||
"license": "Proprietary",
|
||||
"devDependencies": {
|
||||
"electron": "^33.0.0",
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "modelrouter-gui",
|
||||
"version": "2.0.44",
|
||||
"version": "3.0.0",
|
||||
"description": "ModelRouter Desktop — embedded ModelRouter core with tray",
|
||||
"author": "ModelRouter",
|
||||
"main": "main.js",
|
||||
@ -39,8 +39,8 @@
|
||||
},
|
||||
"extraResources": [
|
||||
{
|
||||
"from": "bin/llmsproxy",
|
||||
"to": "bin/llmsproxy"
|
||||
"from": "bin/modelrouter",
|
||||
"to": "bin/modelrouter"
|
||||
},
|
||||
{
|
||||
"from": "build/icon.png",
|
||||
@ -66,8 +66,8 @@
|
||||
"icon": "build/icon.ico",
|
||||
"extraResources": [
|
||||
{
|
||||
"from": "bin/llmsproxy.exe",
|
||||
"to": "bin/llmsproxy.exe"
|
||||
"from": "bin/modelrouter.exe",
|
||||
"to": "bin/modelrouter.exe"
|
||||
},
|
||||
{
|
||||
"from": "bin/lua51.dll",
|
||||
|
||||
@ -1,18 +1,11 @@
|
||||
// Plugin-tree maintenance for the embedded core (no Electron imports, so this
|
||||
// Plugin-tree maintenance for the desktop profile (no Electron imports, so this
|
||||
// is unit-testable with plain node).
|
||||
//
|
||||
// Two problems this solves, both invisible from the gateway side:
|
||||
// 1. The core deliberately does NOT default `plugin_dir` (an operator opts
|
||||
// in). A desktop profile without it runs with ZERO plugins — the channels
|
||||
// plugin (account login, credential vault, auto check-in) never appears.
|
||||
// 2. The core seeds bundled plugins only when the file is ABSENT, so a
|
||||
// desktop upgrade would keep the user's old channels.lua forever and
|
||||
// silently skip every channel fix. On version change the shell must drop
|
||||
// the built-in files so the new binary reseeds them.
|
||||
// The desktop may configure an operator-selected plugin directory, but it never
|
||||
// owns the files in that directory. In particular, upgrades must not seed,
|
||||
// replace, rename, or delete plugins. Plugin installation is an explicit
|
||||
// operator action (or an external package manager action).
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
|
||||
const BUILTIN_PLUGINS = ["billing.lua", "channels.lua", "quota.lua", "usage.lua"];
|
||||
|
||||
// ensurePluginDirConfigured back-fills `plugin_dir` into an existing config.
|
||||
// Appending a top-level YAML key is safe for the shapes this file can have
|
||||
@ -25,7 +18,7 @@ function ensurePluginDirConfigured(configFile, pluginsDir, log) {
|
||||
fs.writeFileSync(
|
||||
configFile,
|
||||
cur.replace(/\s*$/, "") +
|
||||
"\n# added by ModelRouter Desktop: enables the plugin tree (channels etc.)\n" +
|
||||
"\n# added by ModelRouter Desktop: enables the operator plugin directory\n" +
|
||||
"plugin_dir: " + JSON.stringify(pluginsDir) + "\n",
|
||||
"utf-8",
|
||||
);
|
||||
@ -37,46 +30,4 @@ function ensurePluginDirConfigured(configFile, pluginsDir, log) {
|
||||
}
|
||||
}
|
||||
|
||||
// syncBundledPluginsOnUpgrade drops built-in plugin files when the app version
|
||||
// changed, so the core reseeds them from the binary it shipped with. Returns
|
||||
// { changed, dropped:[...], kept: "prev-<ver>" }.
|
||||
function syncBundledPluginsOnUpgrade(profileDir, pluginsDir, version, log) {
|
||||
const stamp = path.join(profileDir, ".plugin-seed-version");
|
||||
let have = "";
|
||||
try {
|
||||
have = fs.readFileSync(stamp, "utf-8").trim();
|
||||
} catch (e) {
|
||||
/* first run */
|
||||
}
|
||||
if (have === version) return { changed: false, dropped: [] };
|
||||
const dropped = [];
|
||||
for (const name of BUILTIN_PLUGINS) {
|
||||
const f = path.join(pluginsDir, name);
|
||||
if (!fs.existsSync(f)) continue;
|
||||
try {
|
||||
if (have) fs.renameSync(f, f + ".prev-" + have);
|
||||
else fs.unlinkSync(f);
|
||||
dropped.push(name);
|
||||
} catch (e) {
|
||||
if (log) log("[gui] drop builtin plugin " + name + " failed: " + e.message);
|
||||
}
|
||||
}
|
||||
try {
|
||||
fs.writeFileSync(stamp, version + "\n", "utf-8");
|
||||
} catch (e) {
|
||||
if (log) log("[gui] write plugin seed stamp failed: " + e.message);
|
||||
}
|
||||
if (dropped.length && log) {
|
||||
log(
|
||||
"[gui] plugins refreshed for " + version + " (" + dropped.length +
|
||||
" builtin file(s) reseeded by core)",
|
||||
);
|
||||
}
|
||||
return { changed: true, dropped, kept: have ? "prev-" + have : "" };
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
BUILTIN_PLUGINS,
|
||||
ensurePluginDirConfigured,
|
||||
syncBundledPluginsOnUpgrade,
|
||||
};
|
||||
module.exports = { ensurePluginDirConfigured };
|
||||
|
||||
@ -1,26 +1,22 @@
|
||||
// Unit tests for the desktop plugin-tree maintenance (plain node, no Electron).
|
||||
// Run: node --test cmd/gui/plugin-sync.test.js
|
||||
// Unit tests for desktop plugin-directory configuration (plain node).
|
||||
const test = require("node:test");
|
||||
const assert = require("node:assert");
|
||||
const fs = require("fs");
|
||||
const os = require("os");
|
||||
const path = require("path");
|
||||
const { BUILTIN_PLUGINS, ensurePluginDirConfigured, syncBundledPluginsOnUpgrade } = require("./plugin-sync");
|
||||
const { ensurePluginDirConfigured } = require("./plugin-sync");
|
||||
|
||||
const tmp = () => fs.mkdtempSync(path.join(os.tmpdir(), "plugin-sync-"));
|
||||
|
||||
test("back-fills plugin_dir into an existing profile without it", () => {
|
||||
const dir = tmp();
|
||||
const cfg = path.join(dir, "config.yaml");
|
||||
fs.writeFileSync(cfg, 'listen: 127.0.0.1:8787\ndefault_model: AUTO\nkeys:\n - key: k\n role: admin\n');
|
||||
fs.writeFileSync(cfg, "listen: 127.0.0.1:8787\ndefault_model: AUTO\nkeys:\n - key: k\n role: admin\n");
|
||||
const plugins = path.join(dir, "plugins");
|
||||
assert.equal(ensurePluginDirConfigured(cfg, plugins, null), "added");
|
||||
const out = fs.readFileSync(cfg, "utf-8");
|
||||
assert.match(out, /^plugin_dir: "/m);
|
||||
assert.ok(out.includes(plugins.replace(/\\/g, "\\\\")) || out.includes(plugins), "path written");
|
||||
// the rest of the profile must survive untouched
|
||||
assert.match(out, /^listen: 127\.0\.0\.1:8787$/m);
|
||||
assert.match(out, /^ - key: k$/m);
|
||||
assert.match(out, /^listen: 127.0.0.1:8787$/m);
|
||||
});
|
||||
|
||||
test("leaves a profile that already sets plugin_dir alone", () => {
|
||||
@ -39,42 +35,14 @@ test("missing config is not created by the back-fill", () => {
|
||||
assert.equal(fs.existsSync(cfg), false);
|
||||
});
|
||||
|
||||
test("upgrade drops builtin plugins (keeping a backup) but not user plugins", () => {
|
||||
test("desktop upgrade never modifies operator plugin files", () => {
|
||||
const dir = tmp();
|
||||
const plugins = path.join(dir, "plugins");
|
||||
fs.mkdirSync(plugins, { recursive: true });
|
||||
for (const f of BUILTIN_PLUGINS) fs.writeFileSync(path.join(plugins, f), "OLD");
|
||||
fs.writeFileSync(path.join(plugins, "channels.lua"), "OPERATOR FILE");
|
||||
fs.writeFileSync(path.join(plugins, "my-own.lua"), "MINE");
|
||||
fs.writeFileSync(path.join(dir, ".plugin-seed-version"), "1.0.0\n");
|
||||
|
||||
const r = syncBundledPluginsOnUpgrade(dir, plugins, "2.0.42", null);
|
||||
assert.equal(r.changed, true);
|
||||
assert.deepEqual(r.dropped.sort(), [...BUILTIN_PLUGINS].sort());
|
||||
for (const f of BUILTIN_PLUGINS) {
|
||||
assert.equal(fs.existsSync(path.join(plugins, f)), false, f + " dropped");
|
||||
assert.equal(fs.readFileSync(path.join(plugins, f + ".prev-1.0.0"), "utf-8"), "OLD");
|
||||
}
|
||||
// There is deliberately no version stamp and no upgrade synchronizer.
|
||||
assert.equal(fs.readFileSync(path.join(plugins, "channels.lua"), "utf-8"), "OPERATOR FILE");
|
||||
assert.equal(fs.readFileSync(path.join(plugins, "my-own.lua"), "utf-8"), "MINE");
|
||||
assert.equal(fs.readFileSync(path.join(dir, ".plugin-seed-version"), "utf-8").trim(), "2.0.42");
|
||||
});
|
||||
|
||||
test("same version is a no-op (nothing dropped)", () => {
|
||||
const dir = tmp();
|
||||
const plugins = path.join(dir, "plugins");
|
||||
fs.mkdirSync(plugins, { recursive: true });
|
||||
fs.writeFileSync(path.join(plugins, "channels.lua"), "CURRENT");
|
||||
fs.writeFileSync(path.join(dir, ".plugin-seed-version"), "2.0.42\n");
|
||||
const r = syncBundledPluginsOnUpgrade(dir, plugins, "2.0.42", null);
|
||||
assert.equal(r.changed, false);
|
||||
assert.equal(fs.readFileSync(path.join(plugins, "channels.lua"), "utf-8"), "CURRENT");
|
||||
});
|
||||
|
||||
test("first run on a fresh profile just stamps the version", () => {
|
||||
const dir = tmp();
|
||||
const plugins = path.join(dir, "plugins");
|
||||
fs.mkdirSync(plugins, { recursive: true });
|
||||
const r = syncBundledPluginsOnUpgrade(dir, plugins, "2.0.42", null);
|
||||
assert.equal(r.changed, true);
|
||||
assert.deepEqual(r.dropped, []);
|
||||
assert.equal(fs.existsSync(path.join(dir, ".plugin-seed-version")), true);
|
||||
assert.equal(fs.readdirSync(dir).includes(".plugin-seed-version"), false);
|
||||
});
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
// ModelRouter Desktop GUI — renderer
|
||||
// Clash-Verge-style: the content area is a full-screen iframe loading the
|
||||
// Desktop shell: the content area is a full-screen iframe loading the
|
||||
// real WebUI 1:1 (status / chat / keys / priority / sources / adapters).
|
||||
// The shell only adds the titlebar, settings overlay and tray integration.
|
||||
const state = {
|
||||
|
||||
@ -83,7 +83,7 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- side rail: autostart / silent toggles (Clash-Verge style) -->
|
||||
<!-- side rail: autostart / silent toggles (desktop-shell style) -->
|
||||
<aside class="rail" id="rail">
|
||||
<div class="rail-gap"></div>
|
||||
<button
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
/* =====================================================================
|
||||
ModelRouter Desktop GUI — Clash-Verge-style shell
|
||||
ModelRouter Desktop GUI — desktop-shell-style shell
|
||||
Content area = full-screen iframe embedding the real WebUI 1:1.
|
||||
Palette mirrors the WebUI (sakura/frost/glassmorphism).
|
||||
===================================================================== */
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env node
|
||||
// Build the embedded llmsproxy core(s) for the GUI package.
|
||||
// Build the embedded modelrouter core(s) for the GUI package.
|
||||
// node prepare-core.js -> linux (native, luajit) + win (cross, best effort)
|
||||
// node prepare-core.js linux -> linux only
|
||||
// node prepare-core.js win -> win only
|
||||
@ -12,21 +12,21 @@ const binDir = path.join(guiDir, "bin");
|
||||
const projectRoot = path.join(guiDir, "..", "..");
|
||||
|
||||
function buildLinux() {
|
||||
const out = path.join(binDir, "llmsproxy");
|
||||
console.log("[prepare-core] building llmsproxy (linux/luajit)...");
|
||||
const out = path.join(binDir, "modelrouter");
|
||||
console.log("[prepare-core] building modelrouter (linux/luajit)...");
|
||||
fs.mkdirSync(binDir, { recursive: true });
|
||||
execFileSync(
|
||||
"go",
|
||||
["build", "-tags", "luajit", "-trimpath", "-o", out, "./cmd/llmsproxy"],
|
||||
["build", "-tags", "luajit", "-trimpath", "-o", out, "./cmd/modelrouter"],
|
||||
{ cwd: projectRoot, stdio: "inherit" },
|
||||
);
|
||||
return out;
|
||||
}
|
||||
|
||||
function buildWin() {
|
||||
const out = path.join(binDir, "llmsproxy.exe");
|
||||
const out = path.join(binDir, "modelrouter.exe");
|
||||
console.log(
|
||||
"[prepare-core] cross-building llmsproxy.exe (windows/amd64, luajit)...",
|
||||
"[prepare-core] cross-building modelrouter.exe (windows/amd64, luajit)...",
|
||||
);
|
||||
fs.mkdirSync(binDir, { recursive: true });
|
||||
const cc = process.env.MINGW_CC || "x86_64-w64-mingw32-gcc";
|
||||
@ -43,7 +43,7 @@ function buildWin() {
|
||||
}
|
||||
execFileSync(
|
||||
"go",
|
||||
["build", "-tags", "luajit", "-trimpath", "-o", out, "./cmd/llmsproxy"],
|
||||
["build", "-tags", "luajit", "-trimpath", "-o", out, "./cmd/modelrouter"],
|
||||
{
|
||||
cwd: projectRoot,
|
||||
stdio: "inherit",
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
// Command llmsproxy is a standalone gateway that exposes multiple upstream LLM
|
||||
// Command modelrouter is a standalone gateway that exposes multiple upstream LLM
|
||||
// sources behind a unified OpenAI-compatible HTTP API. Protocol differences are
|
||||
// handled by Lua adapters (per source), optionally signing outgoing requests.
|
||||
// It supports explicit model selection or AUTO routing by priority, image
|
||||
@ -16,9 +16,9 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/core"
|
||||
"llmsproxy/internal/gateway"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/core"
|
||||
"modelrouter/internal/gateway"
|
||||
)
|
||||
|
||||
func main() {
|
||||
@ -33,12 +33,12 @@ func main() {
|
||||
// error here instead of being silently created.
|
||||
if *checkOnly {
|
||||
if _, err := os.Stat(*cfgPath); err != nil {
|
||||
log.Fatalf("[llmsproxy] check: %v", err)
|
||||
log.Fatalf("[modelrouter] check: %v", err)
|
||||
}
|
||||
if _, err := config.Load(*cfgPath); err != nil {
|
||||
log.Fatalf("[llmsproxy] check: %v", err)
|
||||
log.Fatalf("[modelrouter] check: %v", err)
|
||||
}
|
||||
log.Printf("[llmsproxy] check: %s is valid", *cfgPath)
|
||||
log.Printf("[modelrouter] check: %s is valid", *cfgPath)
|
||||
return
|
||||
}
|
||||
|
||||
@ -48,34 +48,34 @@ func main() {
|
||||
// program prints a credential, and this path never logs to a file.
|
||||
if *showSecrets {
|
||||
if _, err := os.Stat(*cfgPath); err != nil {
|
||||
log.Fatalf("[llmsproxy] show-secrets: %v", err)
|
||||
log.Fatalf("[modelrouter] show-secrets: %v", err)
|
||||
}
|
||||
if err := config.PrintSecrets(*cfgPath); err != nil {
|
||||
log.Fatalf("[llmsproxy] show-secrets: %v", err)
|
||||
log.Fatalf("[modelrouter] show-secrets: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
created, err := config.EnsureDefault(*cfgPath)
|
||||
if err != nil {
|
||||
log.Fatalf("[llmsproxy] config: %v", err)
|
||||
log.Fatalf("[modelrouter] config: %v", err)
|
||||
}
|
||||
|
||||
c, err := core.New(*cfgPath)
|
||||
if err != nil {
|
||||
log.Fatalf("[llmsproxy] core: %v", err)
|
||||
log.Fatalf("[modelrouter] core: %v", err)
|
||||
}
|
||||
defer c.Close()
|
||||
|
||||
if created {
|
||||
for _, k := range c.GatewayKeys() {
|
||||
log.Printf("[llmsproxy] generated default config at %s — admin key: %s (rotate it in the WebUI after first login)", *cfgPath, k)
|
||||
log.Printf("[modelrouter] generated default config at %s — admin key: %s (rotate it in the WebUI after first login)", *cfgPath, k)
|
||||
}
|
||||
}
|
||||
|
||||
gw, err := gateway.New(c)
|
||||
if err != nil {
|
||||
log.Fatalf("[llmsproxy] gateway: %v", err)
|
||||
log.Fatalf("[modelrouter] gateway: %v", err)
|
||||
}
|
||||
|
||||
// Ops hygiene: surface the two most common footguns instead of silently
|
||||
@ -88,16 +88,16 @@ func main() {
|
||||
// (e.g. after rotating the starter key away) it reported "every request is
|
||||
// rejected" while seven working keys, one of them admin, were in service.
|
||||
if keys := c.ListKeys(); len(keys) == 0 {
|
||||
log.Printf("[llmsproxy] WARNING: no gateway key configured — every request will be rejected. Add one in the WebUI or set gateway_keys")
|
||||
log.Printf("[modelrouter] WARNING: no gateway key configured — every request will be rejected. Add one in the WebUI or set gateway_keys")
|
||||
} else {
|
||||
for _, k := range keys {
|
||||
if k.Key == "sk-gw-local-0001" || k.Key == "sk-local-0001" {
|
||||
log.Printf("[llmsproxy] WARNING: gateway key %q looks like the starter/example key — rotate it before exposing the gateway", k.Key)
|
||||
log.Printf("[modelrouter] WARNING: gateway key %q looks like the starter/example key — rotate it before exposing the gateway", k.Key)
|
||||
}
|
||||
}
|
||||
}
|
||||
if l := c.Listen(); strings.HasPrefix(l, "0.0.0.0:") || strings.HasPrefix(l, "::") {
|
||||
log.Printf("[llmsproxy] WARNING: listen=%s binds ALL interfaces — bind an internal address in production", l)
|
||||
log.Printf("[modelrouter] WARNING: listen=%s binds ALL interfaces — bind an internal address in production", l)
|
||||
}
|
||||
|
||||
srv := &http.Server{
|
||||
@ -108,22 +108,22 @@ func main() {
|
||||
go func() {
|
||||
cert, key := c.TLS()
|
||||
if cert != "" && key != "" {
|
||||
log.Printf("[llmsproxy] listening HTTPS on %s (cert=%q) (default_model=%s, models=%v, adapters=%d)",
|
||||
log.Printf("[modelrouter] listening HTTPS on %s (cert=%q) (default_model=%s, models=%v, adapters=%d)",
|
||||
c.Listen(), cert, c.DefaultModel(), c.Registry().ModelList(), len(c.ListAdapters()))
|
||||
if err := srv.ListenAndServeTLS(cert, key); err != nil && err != http.ErrServerClosed {
|
||||
log.Fatalf("[llmsproxy] server: %v", err)
|
||||
log.Fatalf("[modelrouter] server: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
log.Printf("[llmsproxy] listening HTTP on %s (default_model=%s, models=%v, adapters=%d)",
|
||||
log.Printf("[modelrouter] listening HTTP on %s (default_model=%s, models=%v, adapters=%d)",
|
||||
c.Listen(), c.DefaultModel(), c.Registry().ModelList(), len(c.ListAdapters()))
|
||||
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
log.Fatalf("[llmsproxy] server: %v", err)
|
||||
log.Fatalf("[modelrouter] server: %v", err)
|
||||
}
|
||||
}()
|
||||
|
||||
stop := make(chan os.Signal, 1)
|
||||
signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
|
||||
<-stop
|
||||
log.Printf("[llmsproxy] shutting down")
|
||||
log.Printf("[modelrouter] shutting down")
|
||||
}
|
||||
73
deploy.sh
73
deploy.sh
@ -1,10 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# ============================================================
|
||||
# llmsproxy 原子化滚动部署脚本
|
||||
# modelrouter 原子化滚动部署脚本
|
||||
#
|
||||
# 策略:
|
||||
# 1. /tmp (tmpfs) 构建,/usr/local/bin (ext4) 部署
|
||||
# 2. 构建后先 copy 到目标文件系统(.llmsproxy.tmp),
|
||||
# 2. 构建后先 copy 到目标文件系统(.modelrouter.tmp),
|
||||
# 再 rename(2) 做原子替换(同一文件系统内 mv 是原子的)
|
||||
# 3. runtime.json 不替换(保留运行时密钥、审计状态)
|
||||
# 4. systemctl restart → 服务中断仅在 Go 进程重启瞬间 (<500ms)
|
||||
@ -28,20 +28,21 @@
|
||||
set -euo pipefail
|
||||
|
||||
# ---------- 配置 ----------
|
||||
readonly TARGET_BIN="/usr/local/bin/llmsproxy"
|
||||
readonly TARGET_CONFIG="/etc/llmsproxy/config.yaml"
|
||||
readonly TARGET_ADAPTERS="/etc/llmsproxy/adapters"
|
||||
readonly TARGET_PLUGINS="/etc/llmsproxy/plugins"
|
||||
readonly REPO_DIR="/home/program/llmsproxy"
|
||||
readonly TARGET_BIN="/usr/local/bin/modelrouter"
|
||||
readonly TARGET_CONFIG="/etc/modelrouter/config.yaml"
|
||||
readonly TARGET_ADAPTERS="/etc/modelrouter/adapters"
|
||||
readonly TARGET_PLUGINS="/etc/modelrouter/plugins"
|
||||
# 仓库所在目录不等于产品名:这里保持真实路径,除非用 MODELROUTER_REPO_DIR 覆盖。
|
||||
readonly REPO_DIR="${MODELROUTER_REPO_DIR:-/home/program/llmsproxy}"
|
||||
readonly HEALTH_URL="http://127.0.0.1:8081/v1/models"
|
||||
readonly BIN_NAME="llmsproxy"
|
||||
readonly BIN_NAME="modelrouter"
|
||||
|
||||
# 构建目录。默认落在仓库下的构建工作区,而不是 /tmp:本机 /tmp 是 8.8G
|
||||
# tmpfs,实测被 Go 构建缓存占满后会让 go test/build 在链接阶段直接失败。
|
||||
# 可用 LLMSPROXY_BUILD_DIR 覆盖;同一目录也被测试用作 GOTMPDIR。
|
||||
# 可用 MODELROUTER_BUILD_DIR 覆盖;同一目录也被测试用作 GOTMPDIR。
|
||||
readonly BUILD_SUBDIR=".build-work"
|
||||
if [[ -n "${LLMSPROXY_BUILD_DIR:-}" ]]; then
|
||||
readonly TMP_DIR="${LLMSPROXY_BUILD_DIR}"
|
||||
if [[ -n "${MODELROUTER_BUILD_DIR:-}" ]]; then
|
||||
readonly TMP_DIR="${MODELROUTER_BUILD_DIR}"
|
||||
else
|
||||
readonly TMP_DIR="${REPO_DIR}/${BUILD_SUBDIR}"
|
||||
fi
|
||||
@ -59,7 +60,7 @@ usage() {
|
||||
cat <<'EOF'
|
||||
用法: ./deploy.sh [--config <file>]
|
||||
|
||||
--config <file> 投放这份配置到 /etc/llmsproxy/config.yaml(原子替换)。
|
||||
--config <file> 投放这份配置到 /etc/modelrouter/config.yaml(原子替换)。
|
||||
省略时不改动线上配置,只部署二进制与适配器。
|
||||
无论是否指定,重启前都会用新二进制 -check 预检线上配置。
|
||||
|
||||
@ -103,6 +104,17 @@ precheck() {
|
||||
command -v systemctl &>/dev/null || fail "未找到 systemctl"
|
||||
[[ $(id -u) -eq 0 ]] || fail "需要 root 权限"
|
||||
|
||||
# 改名保护:本脚本现在部署到 /etc/modelrouter。若机器上仍是改名前的
|
||||
# /etc/llmsproxy 且新路径尚未建立,直接继续会创建一份**空配置**并让新
|
||||
# 服务在没有 key/源的情况下起来 —— 看起来成功,实际网关拒掉所有请求。
|
||||
if [[ -d /etc/llmsproxy && ! -e "$TARGET_CONFIG" ]]; then
|
||||
fail "检测到改名前的部署 /etc/llmsproxy,但 $TARGET_CONFIG 不存在。
|
||||
先迁移(dry-run 默认,确认后加 --apply):
|
||||
sudo ./packaging/migrate-to-modelrouter.sh
|
||||
sudo ./packaging/migrate-to-modelrouter.sh --apply
|
||||
如需全新部署,先把旧目录移开:mv /etc/llmsproxy /etc/llmsproxy.bak"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_CONFIG_SRC" ]]; then
|
||||
[[ -f "$NEW_CONFIG_SRC" ]] || fail "待部署配置不存在: $NEW_CONFIG_SRC"
|
||||
# 同一文件比对:--config 指向线上文件本身时没有"投放"动作可做,
|
||||
@ -153,7 +165,7 @@ build() {
|
||||
-trimpath \
|
||||
-ldflags="-s -w" \
|
||||
-o "$TMP_DIR/bin/$BIN_NAME" \
|
||||
./cmd/llmsproxy
|
||||
./cmd/modelrouter
|
||||
|
||||
NEW_BIN="$TMP_DIR/bin/$BIN_NAME"
|
||||
NEW_SIZE=$(stat -c%s "$NEW_BIN")
|
||||
@ -254,7 +266,7 @@ verify_config() {
|
||||
|
||||
# ---------- 密钥可解封校验 ----------
|
||||
# config.yaml 里的凭据以 enc:v1: 密文落盘,解密钥匙是 master.key(或
|
||||
# LLMS_PROXY_MASTER_KEY)。一旦钥匙丢失/错位,配置在**启动时**才会报错,
|
||||
# MODELROUTER_MASTER_KEY)。一旦钥匙丢失/错位,配置在**启动时**才会报错,
|
||||
# 那时服务已经处于「二进制已替换 + 配置已替换」的半死状态,只能靠回滚。
|
||||
# 所以在替换任何东西之前先验:能不能用现有的钥匙解开现有的密文。
|
||||
#
|
||||
@ -279,21 +291,22 @@ verify_master_key() {
|
||||
[[ -n "$runtime_file" ]] || runtime_file="$(dirname "$TARGET_CONFIG")/runtime.json"
|
||||
key_file="$(dirname "$runtime_file")/master.key"
|
||||
|
||||
if [[ -n "${LLMS_PROXY_MASTER_KEY:-}" ]]; then
|
||||
log " 钥匙来源: LLMS_PROXY_MASTER_KEY(环境变量)"
|
||||
MASTER_KEY_ENV="${MODELROUTER_MASTER_KEY:-${LLMS_PROXY_MASTER_KEY:-}}"
|
||||
if [[ -n "$MASTER_KEY_ENV" ]]; then
|
||||
log " 钥匙来源: MODELROUTER_MASTER_KEY(环境变量)"
|
||||
# 走真实解密路径验证,而不是只检查格式
|
||||
if out=$("$TARGET_BIN" -show-secrets -config "$TARGET_CONFIG" 2>&1); then
|
||||
log "OK 凭据可解封($(echo "$out" | grep -cE '^(source|key) ') 项)"
|
||||
return 0
|
||||
fi
|
||||
err "凭据无法用 LLMS_PROXY_MASTER_KEY 解封,未替换任何文件:"
|
||||
err "凭据无法用 MODELROUTER_MASTER_KEY 解封,未替换任何文件:"
|
||||
printf '%s\n' "$out" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -f "$key_file" ]]; then
|
||||
err "配置含 $sealed 处密文,但钥匙文件不存在: $key_file"
|
||||
err " 没有它,部署后服务将无法启动。恢复 master.key 或设置 LLMS_PROXY_MASTER_KEY 后重试。"
|
||||
err " 没有它,部署后服务将无法启动。恢复 master.key 或设置 MODELROUTER_MASTER_KEY 后重试。"
|
||||
return 1
|
||||
fi
|
||||
log " 钥匙文件: $key_file ($(stat -c '%A' "$key_file" 2>/dev/null || echo '?'),$(wc -c < "$key_file") 字节)"
|
||||
@ -312,7 +325,7 @@ verify_master_key() {
|
||||
# ---------- 备份目录保留策略 ----------
|
||||
# 每次部署都新建一个 <prefix>.bak.<时间戳> 目录,而回滚只用到最近一次
|
||||
# ($BACKUP_BIN / $BACKUP_CONFIG 都是单文件覆盖)。多出来的目录从没人清理,
|
||||
# 实测线上已积累 71 个,/etc/llmsproxy 因此涨到 122M。
|
||||
# 实测线上已积累 71 个,/etc/modelrouter 因此涨到 122M。
|
||||
#
|
||||
# 保留最近 KEEP_BACKUPS 份足够回滚,同时给目录数设上限——否则一次误配置
|
||||
# 就可能在几秒内造出成百上千个目录。
|
||||
@ -327,7 +340,7 @@ KEEP_BACKUPS=5
|
||||
|
||||
prune_backups() {
|
||||
local prefix="$1"
|
||||
local base="/etc/llmsproxy"
|
||||
local base="/etc/modelrouter"
|
||||
[[ -d "$base" ]] || return 0
|
||||
# 防误用:前缀必须是已知的两个(避免有人传空串删掉整个目录)
|
||||
case "$prefix" in
|
||||
@ -393,7 +406,7 @@ sync_adapters() {
|
||||
[[ -d "$SRC_ADAPTERS" ]] || fail "内嵌适配器目录不存在: $SRC_ADAPTERS"
|
||||
|
||||
# 备份旧的适配器文件(含任何运行时覆盖版本,如 opencode.lua)
|
||||
BACKUP_DIR="/etc/llmsproxy/adapters.bak.$(date +%Y%m%d%H%M%S)"
|
||||
BACKUP_DIR="/etc/modelrouter/adapters.bak.$(date +%Y%m%d%H%M%S)"
|
||||
if [[ -d "$TARGET_ADAPTERS" ]]; then
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
cp -rf "$TARGET_ADAPTERS/"*.lua "$BACKUP_DIR/" 2>/dev/null || true
|
||||
@ -425,7 +438,7 @@ sync_adapters() {
|
||||
#
|
||||
# 早期实现是「只补缺,已存在一律不动」,看着安全,实际造成一个功能性缺口:
|
||||
# 内置插件本身的**修复无法下发**。v2.0.3/v2.0.4/v2.0.5 三个渠道修复就这样
|
||||
# 全部卡在仓库里 —— 二进制换了、/etc/llmsproxy/plugins/channels.lua 还是旧版,
|
||||
# 全部卡在仓库里 —— 二进制换了、/etc/modelrouter/plugins/channels.lua 还是旧版,
|
||||
# 生产行为毫无变化。内置源码的正确性由仓库保证,部署必须与二进制内容对齐。
|
||||
#
|
||||
# 「只补缺」原本想保护的是「运维改过 / 运行时新增的插件」,但那些都是**非内置
|
||||
@ -439,7 +452,7 @@ sync_plugins() {
|
||||
mkdir -p "$TARGET_PLUGINS"
|
||||
|
||||
# 备份将被覆盖/新增的内置插件(非内置名字不进入循环,天然不动)
|
||||
BACKUP_DIR="/etc/llmsproxy/plugins.bak.$(date +%Y%m%d%H%M%S)"
|
||||
BACKUP_DIR="/etc/modelrouter/plugins.bak.$(date +%Y%m%d%H%M%S)"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
local added=0 updated=0 same=0
|
||||
for src in "$SRC_PLUGINS"/*.lua; do
|
||||
@ -481,8 +494,8 @@ sync_plugins() {
|
||||
# ---------- 重启服务 ----------
|
||||
restart_service() {
|
||||
log "重启服务(中断 <500ms)"
|
||||
systemctl restart llmsproxy.service || fail "systemd restart 失败"
|
||||
log "OK llmsproxy.service restarted"
|
||||
systemctl restart modelrouter.service || fail "systemd restart 失败"
|
||||
log "OK modelrouter.service restarted"
|
||||
}
|
||||
|
||||
# ---------- 健康检查 ----------
|
||||
@ -505,7 +518,7 @@ healthcheck() {
|
||||
(( attempt++ ))
|
||||
done
|
||||
|
||||
warn "健康检查未通过,请检查服务状态: systemctl status llmsproxy.service"
|
||||
warn "健康检查未通过,请检查服务状态: systemctl status modelrouter.service"
|
||||
return 1
|
||||
}
|
||||
|
||||
@ -541,7 +554,7 @@ rollback() {
|
||||
|
||||
if (( restored_any == 0 )); then
|
||||
err "无任何回滚点,无法自动恢复"
|
||||
err "请手动检查: systemctl status llmsproxy.service && journalctl -u llmsproxy -n 50"
|
||||
err "请手动检查: systemctl status modelrouter.service && journalctl -u modelrouter -n 50"
|
||||
return 1
|
||||
fi
|
||||
|
||||
@ -555,9 +568,9 @@ rollback() {
|
||||
fi
|
||||
|
||||
log "重启服务(旧版本)"
|
||||
systemctl restart llmsproxy.service || {
|
||||
systemctl restart modelrouter.service || {
|
||||
err "重启失败,服务可能完全宕机"
|
||||
err "请手动检查: systemctl status llmsproxy.service"
|
||||
err "请手动检查: systemctl status modelrouter.service"
|
||||
return 1
|
||||
}
|
||||
|
||||
@ -592,7 +605,7 @@ print_summary() {
|
||||
# 于是坏配置在服务仍然健康时就被拦下,根本不会触发重启。
|
||||
main() {
|
||||
parse_args "$@"
|
||||
log "═══ llmsproxy 原子部署开始 ═══"
|
||||
log "═══ modelrouter 原子部署开始 ═══"
|
||||
precheck
|
||||
# 密钥校验放在 build/替换之前:它的失败后果最重(配置里的密文永久解不开),
|
||||
# 必须在动任何文件之前就挡住。此阶段用的是**当前线上二进制**,它已经
|
||||
|
||||
60
docs/adapters.md
Normal file
60
docs/adapters.md
Normal file
@ -0,0 +1,60 @@
|
||||
# 适配器协议
|
||||
|
||||
> 本文档由 README 拆分而来:Lua 适配器协议与内置适配器清单。
|
||||
> 编写适配器的完整参考见 [lua-adapters.md](lua-adapters.md)。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## Lua 适配器协议
|
||||
|
||||
完整 API 见 **[Lua 适配器 API 文档](docs/lua-adapters.md)**([English](docs/lua-adapters-en.md))。
|
||||
|
||||
每个适配器是一个返回 table 的 Lua 脚本(`internal/lua/adapters/<name>.lua`),
|
||||
加载时可被 `adapter_dir` 下的同名脚本覆盖——**改 Lua 脚本无需重编译**,
|
||||
重启即生效;通过 WebUI 上传的适配器与在线编辑的源配置则即时生效。
|
||||
|
||||
```lua
|
||||
return {
|
||||
name = "mysrc",
|
||||
version = "1.0.0",
|
||||
endpoint = "/chat/completions", -- 上游路径(可被 source.endpoint 覆盖)
|
||||
headers = { ["X-Static"] = "v" }, -- 静态默认请求头(build_headers 缺省时使用)
|
||||
|
||||
-- 请求转换:把 OpenAI 格式 req 转成上游原生格式,返回字符串
|
||||
transform_request = function(raw_json) ... end,
|
||||
|
||||
-- 响应转换:把上游原生响应转成统一格式字符串
|
||||
-- { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
|
||||
transform_response = function(raw_json) ... end,
|
||||
|
||||
-- 流式分块转换:把上游 SSE data 转成 { content, done, ... },返回 "" 则跳过
|
||||
transform_stream_chunk = function(raw_chunk) ... end,
|
||||
|
||||
-- [可选] 动态请求头/签名钩子
|
||||
-- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
|
||||
build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
|
||||
|
||||
-- [可选] 出向序列化(契约 v2):上游不是 OpenAI 协议时接管写给客户端的字节。
|
||||
-- 不声明则沿用内核的 OpenAI 兜底;声明即完全取代,内核不再混合字段。
|
||||
serialize_response = function(result) return body_str end,
|
||||
serialize_open_preamble = function(result) return bytes end,
|
||||
serialize_chunk = function(delta) return bytes end,
|
||||
serialize_close = function(result) return bytes end,
|
||||
}
|
||||
```
|
||||
|
||||
内置信号辅助:`hmac_sha256_hex(key, data)`、`sha256_hex(data)`、`base64_encode(s)`、
|
||||
`tohex(s)`、`json.encode/decode`、`log(level, msg)`。
|
||||
|
||||
### 内置适配器
|
||||
|
||||
`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama` `kimicode`
|
||||
`qwen38` `sensenova` `trae`。
|
||||
|
||||
`anthropic`/`gemini`/`ollama` 适配器内置多模态转换(`image_url` → 各自上游格式);
|
||||
推理开关由中立的 `thinking` 字段表达,`deepseek` 适配器把它映射为
|
||||
`thinking.type = "disabled"`。
|
||||
|
||||
**kimicode** 是展示 `build_headers` 的样例:云端校验调用方 app,需要按
|
||||
`meta.app_secret` 对时间戳+URL+请求体哈希做 HMAC 签名并附 `X-App-Sign` 等头。
|
||||
配好 `sources[].meta.{app_id, app_secret, app_agent}` 即可。
|
||||
|
||||
@ -389,10 +389,10 @@ flow 类型,不是"登录成功"。
|
||||
channels 插件用 `storeDir()` 决定存储位置,它有多级兜底:
|
||||
|
||||
```
|
||||
env LLMSPROXY_PLUGIN_DIR → settings.store_dir → 读 /etc/llmsproxy/config.yaml 的 plugin_dir → 硬编码
|
||||
env MODELROUTER_PLUGIN_DIR → settings.store_dir → 读 /etc/llmsproxy/config.yaml 的 plugin_dir → 硬编码
|
||||
```
|
||||
|
||||
测试台没设 `LLMSPROXY_PLUGIN_DIR`,**后两级直接命中生产**:
|
||||
测试台没设 `MODELROUTER_PLUGIN_DIR`,**后两级直接命中生产**:
|
||||
|
||||
```
|
||||
测试插件目录 = /tmp/TestXXX/002/plugins
|
||||
@ -404,7 +404,7 @@ storeDir 结果 = /etc/llmsproxy/plugins ← 探针实测输出
|
||||
`.channels-store.json` 从 1730 覆盖成 1986 字节、账号池清空(inode 都换了)。
|
||||
|
||||
只有 `internal/lua` 复现,`internal/core` 不复现 —— 因为 core 的测试台
|
||||
(`kernelTestCore`)设了 `LLMSPROXY_PLUGIN_DIR`。**这是隔离做得不一致**的代价。
|
||||
(`kernelTestCore`)设了 `MODELROUTER_PLUGIN_DIR`。**这是隔离做得不一致**的代价。
|
||||
|
||||
同类前科:曾把 26 个测试账号写进生产 store。两次同一类错误。
|
||||
|
||||
|
||||
183
docs/configuration.md
Normal file
183
docs/configuration.md
Normal file
@ -0,0 +1,183 @@
|
||||
# 配置
|
||||
|
||||
> 本文档由 README 拆分而来:网关配置、模型自动探测、多密钥。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## 配置文件
|
||||
|
||||
首次运行在 `-config` 路径自动生成默认配置(默认 `./config.yaml`),核心字段:
|
||||
|
||||
```yaml
|
||||
listen: 127.0.0.1:8080 # 网关监听地址(建议绑内网/回环)
|
||||
gateway_keys: [sk-gw-<随机>] # 初始 admin 密钥(每次安装随机生成,仅首启写入运行时存储用)
|
||||
default_model: AUTO # model 无法路由时自动按优先级链选源
|
||||
adapter_dir: adapters # Lua 适配目录;目录不存在时首启创建并 seed 内置,存在则只读
|
||||
runtime_file: runtime.json # WebUI 编辑的源/密钥/AUTO 链持久化到此文件
|
||||
|
||||
sources:
|
||||
- name: deepseek
|
||||
base_url: https://api.deepseek.com
|
||||
api_key: sk-... # 也可用 api_key_env: SOME_ENV 引用环境变量(不落盘明文)
|
||||
adapter: deepseek
|
||||
max_concurrent: 8
|
||||
models:
|
||||
- id: deepseek-v4-flash
|
||||
priority: 100 # YAML 源首次启动会 seed 进 AUTO 链的初值
|
||||
kind: chat
|
||||
- id: deepseek-v4-pro
|
||||
priority: 60
|
||||
kind: chat
|
||||
# 静态请求头(比适配器默认优先)
|
||||
headers: { X-Tenant: prod }
|
||||
# 传给 Lua build_headers 的透传元数据
|
||||
meta: { app_id: x, app_secret: y }
|
||||
temperature: 0.7
|
||||
max_tokens: 4096
|
||||
timeout: 120s # 请求超时,默认 120s
|
||||
```
|
||||
|
||||
|
||||
## 自动探测模型列表
|
||||
|
||||
不想手抄模型 id 时,只填 URL 和 key,让网关去扫:
|
||||
|
||||
```bash
|
||||
# 只探测,不写任何东西
|
||||
curl -X POST http://127.0.0.1:8080/api/sources/discover \
|
||||
-H "Authorization: Bearer sk-gw-…" -H "Content-Type: application/json" \
|
||||
-d '{"base_url":"https://api.vendor.com/v1","api_key":"sk-…"}'
|
||||
# → {"ok":true,"base_url":"…","endpoint":"","models":["gpt-x",…],
|
||||
# "image_models":["gpt-image-2"],"models_path":"/models","latency_ms":1387,…}
|
||||
|
||||
# 探测后直接落库
|
||||
curl -X POST http://127.0.0.1:8080/api/sources/discover/apply \
|
||||
-H "Authorization: Bearer sk-gw-…" -H "Content-Type: application/json" \
|
||||
-d '{"base_url":"https://api.vendor.com/v1","api_key":"sk-…","name":"vendor"}'
|
||||
```
|
||||
|
||||
WebUI 源编辑框里也有「自动探测模型」按钮:填好 URL 和 key 点一下,列表自动
|
||||
填好,仍需点保存才生效。
|
||||
|
||||
几点值得知道:
|
||||
|
||||
- **URL 形态不用猜。** 厂商对「`/v1` 放 base 还是放 path」没有共识,探测会
|
||||
依次尝试若干候选并返回答对的那个,同时给出归一化后的 `base_url` +
|
||||
`endpoint`(`endpoint` 平时留空,因为 openai adapter 自带 `/chat/completions`)。
|
||||
- **`image_endpoint` 是另一套规则。** 网关的图像默认路径是**绝对**的
|
||||
`/v1/images/generations`,而 chat 默认是相对的。所以 `base_url` 带 `/v1` 时
|
||||
必须显式给 `image_endpoint: /images/generations`,否则图像请求 404。
|
||||
探测会替你填好。
|
||||
- **图像模型只看上游自己声明的 modality**,绝不从模型名猜。`gpt-image-2`
|
||||
这类 id 也出现在同时提供对话服务的目录上,猜错会让 AUTO 链选中一个只支持
|
||||
图像生成的模型,其后每一层都被它拖垮。
|
||||
- **不会逐模型探活。** 34 个模型的目录跑一轮要花钱且耗时数分钟,`/models`
|
||||
是足够便宜的信号;模型是否真能应答留给正常请求去发现。
|
||||
- **对已存在的源,必须显式 `replace_models: true`** 才会覆盖模型列表。重新
|
||||
探测顺手抹掉为 AUTO 链手工调过的 priority,是最难排查的一类数据丢失。
|
||||
- **`api_key` 留空表示沿用已存值**,不会把凭据清空。
|
||||
|
||||
|
||||
`config.yaml`、`runtime.json`、`master.key`、`adapters/` 均不提交版本库(见
|
||||
`.gitignore`),请随备份带走——`master.key` 丢失后运行时密文无法解密。
|
||||
|
||||
运行文件生成在 `-config` 所在目录:`adapter_dir` / `runtime_file` 自动指向
|
||||
配置文件旁边的 `adapters/` 与 `runtime.json`,从任意工作目录启动都可用。
|
||||
|
||||
运行时文件(`runtime_file`)中的敏感字段自动加密:
|
||||
|
||||
- 加密算法 AES-256-GCM,格式 `enc:v1:<base64>`。
|
||||
- 主密钥来源:环境变量 `MODELROUTER_MASTER_KEY`(64 位 hex);否则读取
|
||||
`runtime_file` 同目录的 `master.key`;都不存在时首次启动自动生成 `master.key`(0600)。
|
||||
- 升级时旧明文文件自动兼容:首次运行正常读取,任何 UI 保存操作触发全文件加密迁移。
|
||||
- 注意:**master.key 丢失后密文无法解密**,请随配置一起备份;切勿提交到版本库。
|
||||
|
||||
## 网关密钥(多密钥)
|
||||
|
||||
网关鉴权采用「多密钥 + 角色 + 模型范围」架构,密钥持久化在 `runtime_file` 的
|
||||
`keys` 字段(加密存储):
|
||||
|
||||
- `gateway_keys` 配置只是**初始 admin 密钥种子**:首次启动迁移为运行时 admin
|
||||
key,之后不再参与鉴权管理。
|
||||
- **重要:首次启动后请在 WebUI「密钥」页更换管理员密钥**——初始密钥明文写在
|
||||
`config.yaml` 里(每次安装随机生成),继续使用存在被盗风险;用新密钥登录后删除初始密钥。
|
||||
- WebUI **密钥页**可创建/删除密钥;每个密钥可指定 `admin`(管理全部)或
|
||||
`user`(仅看自己的 key)角色,并配置**模型范围**(模型 + 源 + token 配额 +
|
||||
重置周期)。
|
||||
- 客户端用任意一个已授权的密钥明文作为 Bearer(`Authorization: Bearer <key>`)。
|
||||
- 删除密钥即从运行时存储移除,立即失效。
|
||||
|
||||
### 密钥用量配额
|
||||
|
||||
**配额按模型单独设置**:每个密钥的 `models[]` 里,每一条模型范围各自带一份
|
||||
token 配额与请求数配额。一个模型用满只影响该模型,同一密钥的其它模型照常工作。
|
||||
|
||||
```yaml
|
||||
keys:
|
||||
- key: sk-gw-<hex>
|
||||
role: user
|
||||
name: agent-alice
|
||||
models:
|
||||
- model: deepseek-v4-flash
|
||||
token_quota: 1000000 # 本周期内该 key 用这个模型的 token 预算
|
||||
req_quota: 20000 # 本周期内的请求次数
|
||||
period: nhour # "" | hour | week | month | nhour
|
||||
hours: 6 # 仅 nhour
|
||||
- model: AUTO
|
||||
token_quota: 5000000 # AUTO 也是一条独立配额
|
||||
period: day # ← 这种写法会被拒绝(词表只有 hour/week/month/nhour)
|
||||
- model: kimi-k3 # 未列配额 = 无限
|
||||
```
|
||||
|
||||
- **没有「整钥总配额」**:这是刻意的设计。整钥总额会让「把 A 模型的额度挪给
|
||||
B 模型」变成一次全局重分配;按模型独立计费则每个模型各自可控,运维可以
|
||||
看出哪个模型吃掉了预算。
|
||||
- `period` 词表:空 = 永不过期(累计总量),`hour` / `week` / `month` = 固定窗口,
|
||||
`nhour` + `hours` = 自定义小时数。**拼错的周期在写入时就被拒**,不会静默变成
|
||||
永不过期。
|
||||
- 配额严格按密钥隔离,且**同一密钥内按模型隔离**:A 密钥用满 `m1` 不会消耗
|
||||
B 密钥的额度,同一密钥的 `m2` 也不受影响。
|
||||
- 配额统计含聊天、流式、生图,跨重启从审计日志回放(保留 40 天,覆盖最长的
|
||||
month 窗口)。
|
||||
- 配额耗尽返回 **429 + `Retry-After`**(`rate_limit_exceeded`),消息里点名是哪个
|
||||
模型用满了,客户端可等窗口重置后自动恢复;模型越权才是 403。
|
||||
**admin 密钥永不受配额限制**(其 scope 上的配额也不强制),避免把管理员
|
||||
锁在门外。
|
||||
- 窗口用量按整点小时分桶统计,实际释放比配置窗口最多晚 1 小时(配额宁可晚释放
|
||||
也不超发)。
|
||||
- `PUT /api/keys/{key}` 提交 `models` 即同时提交它们的配额(配额就是 scope 的一部分,
|
||||
不存在会与模型列表脱节的第二份预算)。显式 `0` = 解除该模型的限制。
|
||||
|
||||
#### 配额拒绝 vs 容量拒绝:两种「拒绝」含义不同
|
||||
|
||||
配额在**入口**检查,不占用任何上游槽位,因此是廉价拒绝;容量不足则要走
|
||||
`busyWait` 有界等待后才返回 503。实测(容量 8、0.6s/请求、100 并发):
|
||||
|
||||
| 情况 | 状态码 | 延迟 | 是否打到上游 |
|
||||
|---|---|---|---|
|
||||
| 配额用尽(`req_quota` 50 → 100 并发) | `429 rate_limit_exceeded` + `Retry-After` | **19–21ms** | 否 |
|
||||
| 容量打满(同一拓扑,未限配额) | `503 upstream_error` | ~2.6s(`busyWait`) | 超出部分否 |
|
||||
|
||||
客户端因此可以区分「等窗口重置」与「等上游腾出容量」:前者只需等待,后者
|
||||
通常该降并发或换源。
|
||||
|
||||
#### 配额的开销
|
||||
|
||||
配额桶按 (密钥, 模型, 整点小时) 分桶保留 40 天,实测(AMD 7840HS):
|
||||
|
||||
- 窗口查询按窗口长度而非保留总量扫描:1h 窗口 49ns、24h 窗口 55ns、
|
||||
30d 窗口 3.9μs(此前全扫保留总量,960 桶时 5.9μs)。
|
||||
- 记录一条请求:283ns、3 分配(与引入配额前相同,分配来自 ring buffer)。
|
||||
- 内存:生产形态(7 密钥 × 8 模型 × 2 源 × 满 40 天 retention)约 **3.7MB**。
|
||||
按源 pin 的 `source::model` 桶**惰性创建**——只有当某条配额真的 pin 了
|
||||
某个源时才维护,否则每条记录多写一份桶,在 20 密钥 × 8 模型 × 3 源下会
|
||||
多占 18MB。
|
||||
代价:配置 pinned 配额之前的历史用量无法事后按源拆分,所以 **pinned 配额的
|
||||
首个窗口可能少算**。
|
||||
|
||||
```bash
|
||||
# 配额耗尽时客户端看到(点名了具体模型)
|
||||
HTTP/1.1 429 Too Many Requests
|
||||
Retry-After: 2100
|
||||
{"error":{"type":"rate_limit_exceeded","message":"token quota exceeded for \"deepseek-v4-flash\" (5000000/5000000)"}}
|
||||
```
|
||||
|
||||
137
docs/deployment.md
Normal file
137
docs/deployment.md
Normal file
@ -0,0 +1,137 @@
|
||||
# 部署与迁移
|
||||
|
||||
> 本文档面向运维:单机 systemd 部署、发行包安装、目录与权限、从 `llmsproxy`
|
||||
> 改名后的迁移与回滚。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## 1. 单机部署(推荐)
|
||||
|
||||
仓库根目录的 `deploy.sh` 负责:构建 → 预检配置 → 备份 → 原子替换二进制 →
|
||||
重启服务 → 后置校验。它**不**修改源码,也**不**假设全新机器。
|
||||
|
||||
```bash
|
||||
sudo ./deploy.sh # 默认读取仓库内 config.yaml
|
||||
sudo ./deploy.sh --config other.yaml # 投放并部署另一份配置
|
||||
```
|
||||
|
||||
部署目标(改名后):
|
||||
|
||||
| 项 | 路径 |
|
||||
|---|---|
|
||||
| 二进制 | `/usr/local/bin/modelrouter` |
|
||||
| 配置 | `/etc/modelrouter/config.yaml` |
|
||||
| 适配器 | `/etc/modelrouter/adapters/` |
|
||||
| 插件(可选) | `/etc/modelrouter/plugins/`(需在配置里设 `plugin_dir`) |
|
||||
| 主密钥 | `/etc/modelrouter/master.key`(0600) |
|
||||
| systemd unit | `/etc/systemd/system/modelrouter.service` |
|
||||
|
||||
`deploy.sh` 会在以下情况**中止**而不是"看起来成功":
|
||||
|
||||
- 部署后二进制起不来、或新配置 `-check` 不通过;
|
||||
- 凭据无法解封(`master.key` 丢失 / 与配置不匹配);
|
||||
- 检测到改名前的 `/etc/llmsproxy`,但新配置尚未建立(否则会部署出一个
|
||||
没有 key、没有源的空实例,所有请求都被拒)。
|
||||
|
||||
## 2. 发行包安装
|
||||
|
||||
三个产物由 `packaging/core-dist.sh` 生成到 `cmd/build/dist/`:
|
||||
|
||||
```text
|
||||
modelrouter_<ver>_amd64.deb
|
||||
modelrouter-<ver>.x86_64.rpm
|
||||
modelrouter-<ver>-linux-amd64.tar.gz
|
||||
```
|
||||
|
||||
deb / rpm 安装后:
|
||||
|
||||
```bash
|
||||
sudo systemctl enable --now modelrouter
|
||||
sudo journalctl -u modelrouter -f
|
||||
```
|
||||
|
||||
tar 包内只有:`modelrouter`、`config.example.yaml`、`modelrouter.service`、
|
||||
`migrate-to-modelrouter.sh`、`adapters/`。
|
||||
|
||||
**发行包不含业务插件**(`channels` / `billing` / `quota` / `usage` 等)。
|
||||
它们保留在仓库中作为可选扩展,需要时手工放入 `plugin_dir`,网关不会自动安装、
|
||||
覆盖或删除。这是有意的:插件属于运维资产,生命周期不与内核绑定。
|
||||
|
||||
首次启动会在 `/etc/modelrouter/config.yaml` 生成一份**随机 admin key** 的配置,
|
||||
key 打印在启动日志里。发行包不预置任何可用 key(历史上曾因此泄露过一个真实的
|
||||
`sk-gw-local-0001`)。
|
||||
|
||||
## 3. 目录与权限
|
||||
|
||||
- `/etc/modelrouter` 是唯一的读写根(systemd unit 用 `ProtectSystem=strict` +
|
||||
`ReadWritePaths=/etc/modelrouter` 固化这一点)。
|
||||
- `master.key` 必须 0600。当前 unit 以 root 运行:降权到普通用户需要先把
|
||||
`master.key` 交给该用户并统一目录属主,否则服务会"带着明文落盘继续跑",
|
||||
失败只留一行日志——这是独立变更,不要和加固混在一起。
|
||||
- `plugin_dir` 由运维维护;插件自带的状态文件(如 `.channels-store.json`)也在其中。
|
||||
|
||||
## 4. 从 llmsproxy 迁移
|
||||
|
||||
二进制、配置目录、服务名、环境变量已统一为 `modelrouter`:
|
||||
|
||||
| 旧 | 新 |
|
||||
|---|---|
|
||||
| `/usr/local/bin/llmsproxy` | `/usr/local/bin/modelrouter` |
|
||||
| `/etc/llmsproxy/` | `/etc/modelrouter/` |
|
||||
| `llmsproxy.service` | `modelrouter.service` |
|
||||
| `LLMSPROXY_PLUGIN_DIR` | `MODELROUTER_PLUGIN_DIR` |
|
||||
| `LLMSPROXY_XDISPLAY_BASE` | `MODELROUTER_XDISPLAY_BASE` |
|
||||
| `LLMS_PROXY_MASTER_KEY` | `MODELROUTER_MASTER_KEY` |
|
||||
|
||||
迁移脚本**默认 dry-run**,复制而非移动,旧目录永不删除:
|
||||
|
||||
```bash
|
||||
sudo ./packaging/migrate-to-modelrouter.sh # 预演(逐条打印将做什么)
|
||||
sudo ./packaging/migrate-to-modelrouter.sh --apply # 执行
|
||||
```
|
||||
|
||||
它做五件事:复制状态目录并改写配置里的旧绝对路径、复核 `master.key` 0600、
|
||||
安装新二进制(旧文件保留)、由旧 unit 生成新 unit(改写 ExecStart /
|
||||
WorkingDirectory / ReadWritePaths)、复制 `/usr/share` 资源;最后跑
|
||||
`modelrouter -check` 并 `systemctl daemon-reload`。
|
||||
|
||||
确认新服务可用后,再手工清理旧布局(脚本不会删):
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now llmsproxy.service
|
||||
sudo rm -f /etc/systemd/system/llmsproxy.service
|
||||
sudo rm -rf /usr/local/bin/llmsproxy /etc/llmsproxy /usr/share/llmsproxy
|
||||
```
|
||||
|
||||
回滚(不需要旧文件被删):
|
||||
|
||||
```bash
|
||||
sudo ./packaging/migrate-to-modelrouter.sh --rollback --apply
|
||||
```
|
||||
|
||||
**读取侧兼容**:旧配置路径与旧环境变量仍被接受,所以"先换二进制、后改 unit"
|
||||
这种顺序不会立刻失效。但写入永远落在新路径,迁移完成后不要两台布局混用。
|
||||
|
||||
## 5. 部署后验证清单
|
||||
|
||||
```bash
|
||||
# 1) 进程与监听
|
||||
systemctl status modelrouter --no-pager
|
||||
ss -ltnp | grep modelrouter
|
||||
|
||||
# 2) 鉴权生效(无 key 必须 401)
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
|
||||
|
||||
# 3) 带 key 能列出模型
|
||||
curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:8080/v1/models | head -c 400
|
||||
|
||||
# 4) 真实请求走通(证明 LuaJIT 适配器路径未被打断)
|
||||
curl -s -H "Authorization: Bearer $KEY" \
|
||||
-d '{"model":"AUTO","messages":[{"role":"user","content":"hi"}]}' \
|
||||
http://127.0.0.1:8080/v1/chat/completions | head -c 400
|
||||
|
||||
# 5) 仓库文件与生产文件一致(部署脚本的收尾检查)
|
||||
sha256sum /usr/local/bin/modelrouter build/modelrouter
|
||||
```
|
||||
|
||||
第 4 步失败但第 3 步成功,通常是适配器或上游凭据问题,不是部署问题:
|
||||
查看 `journalctl -u modelrouter -n 100`。
|
||||
34
docs/development.md
Normal file
34
docs/development.md
Normal file
@ -0,0 +1,34 @@
|
||||
# 开发与目录结构
|
||||
|
||||
> 本文档由 README 拆分而来:目录职责、开发协作、测试。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## 目录
|
||||
|
||||
```
|
||||
cmd/modelrouter # 入口(纯 Go 服务端)
|
||||
cmd/gui # Electron 桌面 GUI(内嵌核心 + 托盘 + 打包)
|
||||
internal/config # YAML 配置加载/校验
|
||||
internal/lua # LuaJIT VM + worker 池 + AdapterCache + 内置适配器 (embed)
|
||||
internal/provider # Provider(HTTP) + Registry(路由)
|
||||
internal/gateway # OpenAI 兼容 HTTP 服务 + 鉴权 + SDK/流式 + WebUI
|
||||
internal/types # 统一格式 & OpenAI wire 类型
|
||||
```
|
||||
|
||||
## 开发协作
|
||||
|
||||
- **Git 分支工作流**:主分支 / 特性分支 / 发布分支的规范见
|
||||
**[docs/git-workflow.md](docs/git-workflow.md)**([English](docs/git-workflow-en.md))。
|
||||
主线:`main` 永可部署 → 特性合入 `main` → 切 `release/vX.Y.Z` 打 tag 发布 →
|
||||
hotfix 提交发布分支并 cherry-pick 回 `main`。
|
||||
- **Lua 适配器协议**:见 **[docs/lua-adapters.md](docs/lua-adapters.md)**
|
||||
([English](docs/lua-adapters-en.md))。
|
||||
|
||||
## 测试
|
||||
|
||||
```bash
|
||||
go test -tags luajit ./...
|
||||
```
|
||||
|
||||
覆盖:配置校验、适配器加载/变换、签名钩子、Gateway 鉴权、SDK 结算、SSE 流式、
|
||||
模型路由、多模态透传与 thinking 预算。
|
||||
48
docs/features.md
Normal file
48
docs/features.md
Normal file
@ -0,0 +1,48 @@
|
||||
# 实现与完整特性
|
||||
|
||||
> 本文档由 README 拆分而来:架构来源与完整特性清单(README 只保留摘要)。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## 实现
|
||||
|
||||
统一 OpenAI 兼容网关:把多个上游 LLM 源(DeepSeek、Qijiar、OpenAI、Anthropic、Gemini、
|
||||
Groq、Mistral、Ollama、KimiCode…)通过 **Lua 适配器** 做协议转换,对内网暴露
|
||||
一个标准 `OpenAI Chat Completions` 接口(`/v1/chat/completions` + `/v1/models`),
|
||||
支持单次结算、SSE 流式、**AUTO 模型路由**与**多模态**透传。
|
||||
|
||||
从 [HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent) 的多源 LLM 适配层
|
||||
(`internal/agent/api/provider.go` + `internal/lua/adapters/*`)抽离而来并独立演进。
|
||||
|
||||
## 特性
|
||||
|
||||
- **多源**:一个进程内配置任意多个上游源,按请求的 `model` 自动路由。
|
||||
- **AUTO 模式**:`default_model: AUTO` 时按优先级页保存的 AUTO 链档位逐档调度并发请求。
|
||||
- **统一输出**:所有源对外都是 OpenAI 格式(含 `reasoning_content`、`tool_calls`、`usage`)。
|
||||
- **生图**:`POST /v1/images/generations`,`kind: image` 的模型独立路由。
|
||||
- **多模态**:`content` 数组(`image_url` 等)在多源间无损透传,Anthropic/Gemini/Ollama 自动转换。
|
||||
- **LuaJIT VM**:基于 golua 绑定的 LuaJIT,每个适配器独立 VM + worker 池,安全并发。
|
||||
池是 **弹性** 的:`Σ max_concurrent` 只是上限而非预分配,状态按需创建、空闲时自动回收,
|
||||
因此空载网关几乎不持有 Lua 状态(启动 0 个)。
|
||||
- 扩容步长由适配器最大并发决定:`clamp(ceil(max/8), 1, 8)`,且仅在**真实并发争用**
|
||||
(所有现有状态均已占用)时批量预热,顺序流量始终只用 1 个状态。
|
||||
- 缩容步长由当前连接数决定:`clamp(ceil(冗余/(1+占用)), 1, 冗余)`——无连接时一轮收到
|
||||
常驻下限,忙时每轮只释放 1 个,保护热路径。WebUI「适配器」页展示实时池指标。
|
||||
- **thinking 预算**:请求里的推理开关/预算归一为中立的 `thinking:{enabled,budget_tokens}`,
|
||||
适配器映射到各自协议(Anthropic `thinking.budget_tokens`、Gemini `thinkingConfig`、
|
||||
DeepSeek `thinking.type="disabled"`)。
|
||||
- **Lua 适配协议**:每个源挂一个 `.lua` 适配器,完成 `transform_request` /
|
||||
`transform_response` / `transform_stream_chunk` 双向转换,协议差异全在 Lua 层。
|
||||
- **签名 / 请求头钩子**:适配器可定义 `build_headers(meta)`,在 Go 发 HTTP 前
|
||||
动态注入/签名请求头——用于云端 API 校验调用方 app(如 KimiCode 只放行特定
|
||||
agent)。提供 `hmac_sha256_hex` / `sha256_hex` / `base64_encode` 等签名辅助。
|
||||
- **WebUI**:内置管理界面,可在线查看/新增/编辑上游源与模型、配置 AUTO 优先级链、
|
||||
管理密钥模型范围,写入运行时文件持久化。
|
||||
- **密钥加密存储**:运行时文件中的上游 `api_key`、自定义请求头值、网关 key 均以
|
||||
AES-256-GCM 加密落盘(`master.key` 独立 0600,或 `MODELROUTER_MASTER_KEY`)。
|
||||
- **实时源探测**:状态页探测各源可达性(`GET {base}/models`,失败回退最小请求),
|
||||
不污染正常调度的退避状态,错误信息在 UI 可悬停查看。
|
||||
- **鉴权**:网关自身用 `gateway_keys` 校验客户端 Bearer key;与上游各自的 key 相互独立。
|
||||
- **流式**:SSE `chat.completion.chunk`,含角色首包与 `[DONE]` 收尾。
|
||||
- **全链路审计**:HTTP 访问日志、登录/配置变更审计、请求记录 CSV 导出、密钥用量统计 CSV 导出
|
||||
- **AES-256-GCM 加密存储**:运行时文件敏感字段加密落盘,master.key 0600 权限,支持环境变量注入主密钥
|
||||
|
||||
86
docs/gui.md
Normal file
86
docs/gui.md
Normal file
@ -0,0 +1,86 @@
|
||||
# 桌面 GUI(Electron)
|
||||
|
||||
> 本文档由 README 拆分而来:Headless 与 Desktop 两种产物、构建与运行。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## 桌面 GUI(Electron)
|
||||
|
||||
可选独立产物,面向内网桌面端用户:**内嵌 ModelRouter 核心**(桌面壳形态),
|
||||
一键启停、系统托盘常驻、开机自启、静默启动,内容区一比一内嵌完整 WebUI
|
||||
(状态/对话/密钥/优先级/源/适配器 六个页面,免登录)。服务器用户继续使用纯 Go 二进制。
|
||||
|
||||
### 两种安装包:Headless 与 Desktop
|
||||
|
||||
发布到 Release 的安装包分为两类,面向不同用户:
|
||||
|
||||
- **Headless(服务器版)**:纯核心二进制(无 Electron),配置驱动,适合服务器、
|
||||
容器、systemd 等无人值守场景。发布包含 `deb` + `rpm`(含 systemd unit、内存调优
|
||||
与示例配置)与 `tar.gz`(二进制 + `config.example.yaml` + `adapters/` + README),
|
||||
Windows 为 `.zip`。
|
||||
- **Desktop(桌面版)**:Electron GUI 安装包,内嵌核心,适合个人桌面日常使用。
|
||||
|
||||
### 选择 GUI 还是纯后端
|
||||
|
||||
| 场景 | 推荐 | 理由 |
|
||||
| ------ | ------ | ------ |
|
||||
| 服务器 / 内网网关 / 无人值守常驻 | **Headless**(单二进制) | 极轻量(二进制 ~8MB;RSS 随源数量,单源 ~10MB、16 源 ~32–35MB,见[内存占用](#内存占用实测与调优)),零依赖单进程,直接跑在 systemd/任意容器里,远程管理 |
|
||||
| 个人桌面日常使用 / 多设备内网共享 | **Desktop**(GUI) | 免登录内嵌 WebUI、系统托盘一键启停、开机自启、静默后台,适合不懂命令行的使用者 |
|
||||
| Windows 桌面 | **Desktop** | 纯后端在 Windows 上需自行注册服务,GUI 提供原生托盘/自启体验 |
|
||||
| CI 一键出三平台安装包 | **Desktop 打包脚本** | `make gui-*` 系列出 deb/AppImage/NSIS,可进发行流水线 |
|
||||
|
||||
两者完全同源:GUI 内嵌的就是纯后端的同一份 `modelrouter` 核心(LuaJIT 版),
|
||||
配置/适配器格式完全一致,可随时互换。
|
||||
|
||||
### 构建与运行
|
||||
|
||||
```bash
|
||||
cd cmd/gui
|
||||
npm install
|
||||
make gui # 或 npm run dev —— 开发运行(需先 make build 生成 bin/modelrouter)
|
||||
make gui-deb # 本机构建 deb(分享给其它 Linux 用户)
|
||||
make gui-dist # 构建 deb + rpm + AppImage(linux,rpm 需系统 rpmbuild)
|
||||
make core-dist # 构建核心 deb + rpm + tar.gz(服务器版,需 nfpm)
|
||||
make gui-win # 构建 Windows nsis 安装包(本机 mingw + wine)
|
||||
make gui-win-docker # 构建 Windows nsis 安装包(全 docker 自足,含 wine)
|
||||
make gui-dist-dir # 仅解包产物(调试用,不产安装包)
|
||||
```
|
||||
|
||||
产物在 `cmd/build/gui-dist/`:`ModelRouter-<ver>.AppImage`、
|
||||
`modelrouter-gui_<ver>_amd64.deb`、`ModelRouter Setup <ver>.exe`(win);
|
||||
核心包在 `cmd/build/dist/`:`modelrouter_<ver>_amd64.deb`、`modelrouter-<ver>.x86_64.rpm`。
|
||||
rpm 需系统 `rpmbuild`。每次打包后都会跑 `packaging/verify-dist.sh` 产物体检
|
||||
(大小下限门禁——1.3.0 曾因宿主 wine 损坏产出 264KB 的残缺 exe 却没有被拦住)。
|
||||
发布时会把产物重命名为 `ModelRouter-Desktop-*`(GUI)与 `ModelRouter-Headless-*`(纯后端)
|
||||
两类上传到 GitCode Release。
|
||||
|
||||
#### Windows 打包(docker 固化方案,推荐)
|
||||
|
||||
`make gui-win-docker` 一步完成:交叉编译核心 + NSIS 打包**全在 docker 内**,宿主机
|
||||
**无需 mingw、无需 wine、无需 node**。镜像内置 wine(含 i386 运行时),修复了
|
||||
此前宿主 wine 损坏(缺 syswow64 → `c0000135`)导致产出 264KB 残缺 exe 的问题。
|
||||
|
||||
- 构建镜像:`cmd/gui/docker/win-builder/Dockerfile`(`golang:1.25` + mingw-w64 +
|
||||
node + wine32/wine64 + 预编译 LuaJIT windows 版 `lua51.dll` 与 import lib)。
|
||||
LuaJIT 源码取自 gitcode 镜像(国内可达),github 兜底;Go 依赖走 `https://goproxy.cn`。
|
||||
- 一键发布:`cmd/gui/scripts/dist-win-docker.sh` —— 核心编译 + electron-builder
|
||||
NSIS 全部在容器内执行(`USE_SYSTEM_WINE=true` 走镜像内的 wine);electron-builder
|
||||
的 ~400MB 工具链缓存挂到 docker volume,二次构建增量复用。
|
||||
- 首次运行自动构建 `modelrouter/win-builder` 镜像(约 5–10 分钟);已存在则直接复用。
|
||||
- 打包完成后自动跑 `packaging/verify-dist.sh` 门禁:exe 低于 5MB / deb·rpm·7z 低于
|
||||
10MB 即失败——残缺安装包不可能再溜进 Release。
|
||||
|
||||
```bash
|
||||
# 全 Windows 发布(核心重编 + NSIS):
|
||||
make gui-win-docker
|
||||
```
|
||||
|
||||
### 特性
|
||||
|
||||
- **内嵌核心**:自动拉起捆绑的 `modelrouter`(LuaJIT 版),配置/密钥/adapter 存于
|
||||
用户数据目录 `<userData>/profile/`,首次运行自动生成随机 admin key 写入 `keys`
|
||||
(非 seed)并自注入,WebUI 免登录、不会提示"更换初始密钥"。端口可在设置中更改(默认 8787)。
|
||||
- **系统托盘**:内核状态点、开机自启/静默启动开关、启停/重启内核、退出。
|
||||
- **静默启动**:`--silent` 参数或设置项,启动仅驻托盘不弹窗;首次运行始终显示窗口。
|
||||
- **开机自启**:Windows/macOS 用 `setLoginItemSettings`;Linux 写
|
||||
`~/.config/autostart/modelrouter-gui.desktop`,静默开启时自动附加 `--silent`。
|
||||
|
||||
60
docs/performance.md
Normal file
60
docs/performance.md
Normal file
@ -0,0 +1,60 @@
|
||||
# 性能与内存
|
||||
|
||||
> 本文档由 README 拆分而来:实测内存占用、调优手段、thinking 预算。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## 内存占用(实测与调优)
|
||||
|
||||
内存不是一个定数:它随**配置的源数量**增长(每个源一个 `http.Transport` 与连接池、
|
||||
一组健康状态),与运行时长无关。本机实测(Linux x86_64,12 核):
|
||||
|
||||
| 部署形态 | 启动 RSS | 稳态 RSS |
|
||||
|---|---|---|
|
||||
| 1 源 / 1 适配器(最小配置) | ~4 MB | ~10 MB |
|
||||
| 1 源 + 29 MB 历史审计日志 | ~19 MB | ~20 MB |
|
||||
| **16 源 / 13 适配器 / 59 模型(本机生产)** | ~28 MB | **~32–35 MB** |
|
||||
|
||||
> 历史参考:本项优化前同一生产配置为 **~105 MB**。降幅来自三处:审计日志的**原始记录**
|
||||
> 不再常驻内存(约 25 MB;统计聚合仍扫全量,但扫完即释放)、Lua 状态池不再单调增长、
|
||||
> 以及下面两个运行时开关。
|
||||
|
||||
内存构成(生产实例分段测量,`/proc/<pid>/smaps`):
|
||||
|
||||
| 区域 | RSS | 说明 |
|
||||
|---|---|---|
|
||||
| Go 堆 | ~14 MB | provider/registry/scheduler 结构 + 连接池缓冲 |
|
||||
| 其他匹名(线程栈 / LuaJIT chunk / runtime) | ~12 MB | 与线程数、已加载适配器数相关 |
|
||||
| 二进制 text+rodata | ~8 MB | 映射的可执行文件页(只读、可被内核回收) |
|
||||
| Go runtime 预留 | ~4 MB | VSZ 上看到的 2 GB+ 是地址空间预留,不占物理内存 |
|
||||
| 共享库 | ~3 MB | libc / libluajit / libm |
|
||||
|
||||
**两个推荐的部署开关**(只适用于环境变量,无需改代码):
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/modelrouter.service
|
||||
Environment=GOGC=50
|
||||
Environment=MALLOC_ARENA_MAX=2
|
||||
```
|
||||
|
||||
- `MALLOC_ARENA_MAX=2`:LuaJIT 的分配走 cgo → glibc malloc,glibc 默认允许 `8×nproc`
|
||||
个 per-thread arena,每个碰到 malloc 的 OS 线程会占用一个(各约 1 MB,且**不归还给系统**)。
|
||||
实测从 8–15 个 arena(约 7–12 MB)降到 **0**。
|
||||
- `GOGC=50`:把 Go 堆增长目标减半。**单独使用无效**(省下的堆会立即被更多 glibc
|
||||
arena 吃掉,实测 20.3 → 21.5 MB 反而变大),必须与 `MALLOC_ARENA_MAX` 配合,
|
||||
两者同时开启才降 **~19%**。网关是 I/O 密集型(本机 9 小时仅消耗 1min10s CPU),
|
||||
多出的 GC 周期与它的空闲 CPU 相比可忽略。
|
||||
|
||||
> 桌面版(Electron)已在 `cmd/gui/main.js` 里默认为内嵌核心注入这两个开关;手动 export 同名
|
||||
> 环境变量可覆盖。如果你用自己的 systemd unit / 容器,建议照上面加上。
|
||||
|
||||
可选:`GOMEMLIMIT=48MiB` 作软上限,实测再省 ~1 MB;代价是逐近上限时 GC 转激进,
|
||||
温和场景不必开。
|
||||
|
||||
## thinking 预算
|
||||
|
||||
请求体带 `"thinking": {"enabled": false}` 时关闭推理;`"thinking": {"enabled": true,
|
||||
"budget_tokens": 4096}` 指定预算。内核只定义这个**中立**语义字段,映射到具体协议由
|
||||
适配器负责(Anthropic `thinking.budget_tokens`、Gemini `thinkingConfig.thinkingBudget`、
|
||||
DeepSeek `thinking.type = "disabled"`)。内核不再向适配器泄漏 `disable_thinking` /
|
||||
`extra_body` 这类私有字段。
|
||||
|
||||
118
docs/plugins.md
118
docs/plugins.md
@ -15,6 +15,37 @@ ModelRouter 的插件是**单个 `.lua` 文件**,放在 `config.yaml` 的 `plu
|
||||
|
||||
---
|
||||
|
||||
## 0. 插件 SDK:唯一的公开接口
|
||||
|
||||
插件与网关之间**只有一个**公开接口表 `modelrouter`。它不是 Go 包的导出符号,
|
||||
而是内核在启动插件 VM 时注入的 Lua 表;插件通过它调用内核能力,永远不需要
|
||||
(也不应该)依赖任何 `__llmsproxy_*` 全局、Go 结构体或内部函数名。
|
||||
|
||||
```lua
|
||||
modelrouter.sdk_version -- 字符串,契约版本;SDK 形状变化才递增
|
||||
modelrouter.http(spec) -- 受出站白名单约束的 HTTP 请求
|
||||
modelrouter.secrets -- 本插件专用的凭据保险库(set/get/delete/keys)
|
||||
modelrouter.kernel -- 内核动作面(actions/call,受运维授权约束)
|
||||
modelrouter.adapters -- 安装/更新/删除本插件自己的适配器
|
||||
modelrouter.plugin_dir -- 本插件所在目录(内核给出的 ground truth)
|
||||
```
|
||||
|
||||
规则:
|
||||
|
||||
- **只依赖上表**。内核重命名内部全局、改实现或加守卫时,插件不受影响;
|
||||
这正是把六个散落的全局收进一张表的原因。
|
||||
- **能力可能缺失**。未授予的能力(例如出站 HTTP 全禁)在表里是 `nil` 而不是
|
||||
"存在但一调用就报错",插件应先判断再使用。
|
||||
- **`sdk_version` 用来做兼容分支**。增量字段不递增版本;字段删除或调用约定
|
||||
变化才递增。
|
||||
- **历史别名**:`llmsproxy`(旧产品名)与 `__llmsproxy_{http,secrets,kernel,
|
||||
adapters,plugin_dir}` 仍然指向同一批对象,只为已安装的插件继续可用而保留,
|
||||
已废弃;新插件使用 `modelrouter.*`。
|
||||
- 环境变量:插件目录优先读 `MODELROUTER_PLUGIN_DIR`,回退旧名
|
||||
`MODELROUTER_PLUGIN_DIR`。
|
||||
|
||||
---
|
||||
|
||||
## 1. 快速上手
|
||||
|
||||
一个最小的插件:
|
||||
@ -65,16 +96,14 @@ return plugin -- 必须返回一个 table
|
||||
core.New
|
||||
└─ lua.NewVM(adapter_dir).Start() 适配器状态
|
||||
└─ lua.NewPlugins(vm, plugin_dir)
|
||||
├─ SeedBundled() 仅当目录不存在时写入内置插件(目前是 billing)
|
||||
└─ LoadDir() 按文件名字典序逐个加载
|
||||
`plugin_dir` 中的 Lua 文件由网关逐个加载;安装包不携带业务插件,也不会自动创建或写入插件文件。首次启用时,先把需要的 `.lua` 插件放入该目录,再启动网关。
|
||||
```
|
||||
|
||||
**加载失败不影响网关启动。** 一个语法错误的插件会被记录在 `GET /api/plugins`
|
||||
的 `error` 字段里,永远不会被调用。这与适配器一致,但理由更强:插件是可选的
|
||||
第三方扩展,因为一个 `.lua` 打错字就让网关起不来是错误的取舍。
|
||||
|
||||
**目录一旦存在就是权威的。** 与适配器同规则:首启会 seed 内置插件,之后目录里
|
||||
的文件说了算,删除或编辑内置插件都是真实生效的操作。
|
||||
插件目录由运维独立管理。网关不会 seed、覆盖或删除其中的插件;目录缺失时只会跳过插件加载,不影响网关启动。
|
||||
|
||||
### 2.1 热更新
|
||||
|
||||
@ -294,7 +323,7 @@ PUT /api/plugins/{name}/state 替换状态(admin)
|
||||
"loaded": true
|
||||
}],
|
||||
"hook_errors": { "request_end": { "count": 3, "last_error": "billing: ..." } },
|
||||
"plugin_dir": "/etc/llmsproxy/plugins"
|
||||
"plugin_dir": "/etc/modelrouter/plugins"
|
||||
}
|
||||
```
|
||||
|
||||
@ -428,7 +457,7 @@ Cookie 鉴权都自然生效,无需暴露额外端口。
|
||||
-- 每小时的定时任务:轮换 token 并重写适配器
|
||||
function plugin.refresh_token()
|
||||
local tok = fetch_new_token() -- 插件自己的取 token 逻辑
|
||||
local name, err = __llmsproxy_adapters.install(
|
||||
local name, err = modelrouter.adapters.install(
|
||||
"myauth",
|
||||
'return { name = "myauth", endpoint = "/chat/completions",' ..
|
||||
' headers = { ["Authorization"] = "Bearer ' .. tok .. '" } }'
|
||||
@ -439,9 +468,9 @@ end
|
||||
|
||||
| 调用 | 参数 | 返回 |
|
||||
|---|---|---|
|
||||
| `__llmsproxy_adapters.install(name, source)` | 适配器名 + Lua 源码 | `(生效名, nil)` 或 `(nil, err)` |
|
||||
| `__llmsproxy_adapters.update(name, source)` | 同上(只限自己装的) | `(true, nil)` 或 `(nil, err)` |
|
||||
| `__llmsproxy_adapters.remove(name)` | 适配器名(只限自己装的) | `(true, nil)` 或 `(nil, err)` |
|
||||
| `modelrouter.adapters.install(name, source)` | 适配器名 + Lua 源码 | `(生效名, nil)` 或 `(nil, err)` |
|
||||
| `modelrouter.adapters.update(name, source)` | 同上(只限自己装的) | `(true, nil)` 或 `(nil, err)` |
|
||||
| `modelrouter.adapters.remove(name)` | 适配器名(只限自己装的) | `(true, nil)` 或 `(nil, err)` |
|
||||
|
||||
出错时不抛异常,而是返回 `(nil, err)`——与钩子/HTTP 处理器同一套约定,
|
||||
一个手滑的插件错误不会带走网关。
|
||||
@ -494,24 +523,66 @@ end
|
||||
|
||||
### 6.4 命名
|
||||
|
||||
- 全局 `__llmsproxy_plugin` 存放插件返回的 table,**不要占用**
|
||||
- 全局 `__llmsproxy_plugin` 是内核存放插件返回 table 的**内部位置**,不属于 SDK,**不要占用**
|
||||
- 插件间状态完全隔离,一个插件改不了另一个的全局
|
||||
|
||||
---
|
||||
|
||||
## 7. 内置计费插件
|
||||
## 7. Billing 插件示例
|
||||
|
||||
`plugins/billing.lua`,默认随首启 seed 进来,开箱可用。
|
||||
`internal/lua/plugins/billing.lua` 是仓库中的可选插件示例,不随核心安装包分发,也不会在启动时自动安装。需要使用计费功能时,应将该脚本单独复制到配置的 `plugin_dir` 后再重启网关。
|
||||
|
||||
### 7.1 计费维度
|
||||
### 7.0 分层:billing 完全在插件里,内核没有任何 billing 代码
|
||||
|
||||
| 维度 | 用途 | 优先级 |
|
||||
`billing` 是**插件自己的事**:规则文件、编译、计价、账目、规则编辑页都由
|
||||
`plugins/billing.lua` 负责。内核里没有 billing 的配置结构、端点、编译或迁移逻辑
|
||||
—— 它只提供所有插件通用的能力(加载 `.lua`、插件状态持久化、插件目录、
|
||||
`plugin.http` 路由、插件可用的 `io`/`os`)。
|
||||
|
||||
| 项 | 位置 |
|
||||
|---|---|
|
||||
| 规则文件 | `<plugin_dir>/billing.json`(插件自己读写;内核不知道这个文件) |
|
||||
| 计费账目 | 插件状态文件 `<plugin_dir>/.billing.state.json`(内核的通用状态持久化) |
|
||||
| 规则编辑 API | `/api/plugins/http/billing/rules`(GET/PUT/POST/DELETE,插件路由) |
|
||||
| 编译/计价 | 插件内 `compileRules()` / `priceFor()` / `costFor()` |
|
||||
|
||||
规则以**源名**(`source`)为键,而不是 URL:URL→源名的映射属于内核配置,插件
|
||||
不需要为了它和内核对接;编辑页在浏览器里,直接取 `/api/sources` 给运营选源名。
|
||||
`url: "*"` 只表示"全局模型价目表"(模型维度本来就不分源)。
|
||||
|
||||
> 历史:早期 `billing:` 写在 `config.yaml`、编译在 Go 的 `internal/billing` 里。
|
||||
> GitCode issue #1 的根因正是"Go 与 Lua 各写一半覆盖语义",组合起来把模型价
|
||||
> 静默清零。现在内核侧 billing 代码已全部删除,规则文件也由插件自持。
|
||||
|
||||
### 7.1 计费维度### 7.1 计费维度
|
||||
|
||||
| 维度 | 用途 | token 价覆盖顺序 |
|
||||
|---|---|---|
|
||||
| `sources` | 源的固定价(如按请求计费的转发服务) | 中 |
|
||||
| `models` | 模型的 per-token 价 | 高 |
|
||||
| `default` | 兜底价(通常全 0) | 最低 |
|
||||
| `models` | 模型的 per-token 价 | 中 |
|
||||
| `sources` | 源级价:固定价,或覆盖该源全部模型的 token 价 | 高 |
|
||||
| `keys` | 单个网关 key 的覆盖价 | **最高** |
|
||||
|
||||
**token 价**优先级:`keys` > `models` > `default`。
|
||||
**token 价**优先级:`keys` > `sources` > `models` > `default`。
|
||||
(实现顺序就是"后写覆盖前写":`default` → `models` → `sources` → `keys`;
|
||||
所以**源级 token 价会覆盖模型价**。这是有意的:运营要能给某个源整体改价。)
|
||||
|
||||
⚠️ 正因为 `sources` 会覆盖 `models`:**`mode: token` 的规则不产生任何源条目**。
|
||||
规则未列出的模型会进入未定价统计(可见),而不是静默计 0 —— 这是 issue #1 / PR #2
|
||||
的收口语义。只有 `free` / `subscription`("边际成本为零")会写有意为之的
|
||||
`{prompt:0, completion:0}` 零价条目。
|
||||
|
||||
规则写法:
|
||||
|
||||
```json
|
||||
{"active":"current","profiles":[{"id":"current","default":"unpriced","rules":[
|
||||
{"source":"localzen","mode":"free"},
|
||||
{"source":"codearts","mode":"token",
|
||||
"models":{"glm-5.3":{"prompt":"0.15","completion":"0.60"}}},
|
||||
{"url":"*","mode":"token","models":{"some-model":{"prompt":"1.0","completion":"2.0"}}}
|
||||
]}]}
|
||||
```
|
||||
|
||||
**`per_request` 固定价是叠加的**(不覆盖),所以一个生图模型可以既算 token 又收固定费。
|
||||
**`cache_discount`** 见 §7.7。
|
||||
|
||||
@ -599,6 +670,17 @@ curl -H "Authorization: Bearer $ADMIN_KEY" \
|
||||
|
||||
仪表盘上有 "Unpriced" 卡片。**这个数应该是 0**;不是 0 就去补价目。
|
||||
|
||||
**profile 级兜底**:`profiles[].default` 可设为 `free` 或 `unpriced`(默认)。
|
||||
它是插件运行时的策略(编译期写入 `prices.default_mode`):
|
||||
|
||||
| `default` | 没有单独规则的源 | `unpriced_reqs` |
|
||||
|---|---|---|
|
||||
| `unpriced`(省略时) | 零价,计入未定价债务 | +1 |
|
||||
| `free` | 零价,视为**刻意免费** | 不增加 |
|
||||
|
||||
显式 `mode: unpriced` 的源即使在 `default: free` 下也仍然计入("只统计不收费"
|
||||
的声明不能被兜底策略吞掉)。
|
||||
|
||||
注意"未定价"不等于"免费":这些请求的 `requests` / token 数**照常计入**
|
||||
`total` 与各维度,只有金额是 0。
|
||||
|
||||
@ -651,7 +733,7 @@ cached = cache_hit_tokens → 全价 × cache_discount
|
||||
插件的 `log(level, msg)` 输出到网关的日志:
|
||||
|
||||
```
|
||||
journalctl -u llmsproxy -f | grep -i plugin
|
||||
journalctl -u modelrouter -f | grep -i plugin
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
111
docs/routing.md
Normal file
111
docs/routing.md
Normal file
@ -0,0 +1,111 @@
|
||||
# 路由与调度
|
||||
|
||||
> 本文档由 README 拆分而来:模型路由、AUTO 链、冷却与半冷却、源与适配器、源模板。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## 模型路由
|
||||
|
||||
`/v1/chat/completions` 的 `model` 解析顺序:
|
||||
|
||||
1. `source/model` 或 `source:model` 前缀 → 指定源;
|
||||
2. 精确匹配某个源的 `model`;
|
||||
3. 配置 `default_model: AUTO` 时 → 走优先级页保存的 AUTO 链(见下);
|
||||
4. 否则回落(返回错误)。
|
||||
|
||||
任何 OpenAI 客户端,只要 `model` 设为某个源的 `name/任意名`,即可锁定走该源;
|
||||
设为 `AUTO`(或网关配了 `default_model: AUTO`)即自动按优先级链选源。
|
||||
|
||||
## AUTO 链(优先级页)
|
||||
|
||||
AUTO 调度**只**由优先级页保存的规则决定,源配置里的 `models[].priority` 数字不再
|
||||
参与调度、也不再显示。**聊天与生图是两套独立的链**,在优先级页面通过「聊天 / 生图」
|
||||
按钮切换编辑:
|
||||
|
||||
- **聊天链**(`runtime_file` 的 `auto` 字段):服务 `POST /v1/chat/completions`
|
||||
的 `model: AUTO` 请求;只接受 `kind: chat` 模型,配置在链里的生图槽位会被忽略。
|
||||
- **生图链**(`runtime_file` 的 `auto_image` 字段):服务 `POST /v1/images/generations`
|
||||
的 `model: AUTO` 请求;只接受 `kind: image` 模型。未配置生图链时,AUTO 生图回落
|
||||
到「所有含生图模型的源、按注册顺序」的旧行为,兼容旧配置。
|
||||
|
||||
每条规则 = 一个「槽位」:`{ model, source, tier, token_quota, period, hours }`。
|
||||
|
||||
- `tier` 表示优先级档位:同一档的模型并排、共享该优先级;档位从上到下递减。
|
||||
- 同一模型可配置多个槽位(如 A 源低配 → B 源低配 → A 源高配 → B 源高配),按档位顺延。
|
||||
- `token_quota` > 0 时该槽位在重置周期内用满即顺延到下一槽位;`period` 支持
|
||||
`hour` / `week` / `month` / `nhour`(配合 `hours`),空 = 不限。
|
||||
- 同 tier 内按「偏好分」排序,失败槽位冷却后自动跳过;冷却 / 配额耗尽 / hard
|
||||
错误会顺延到下一 tier,全部失败时返回 503 并附上每档失败摘要。
|
||||
|
||||
## 冷却与半冷却探测(自愈调度)
|
||||
|
||||
槽位失败后进入指数退避冷却(5s → 10s → 20s …,上限 **5 分钟**)。冷却不是全时段
|
||||
硬闸:
|
||||
|
||||
- **前半段完全静默**:刚失败的槽位不接任何流量,避免对故障上游持续施压。
|
||||
- **后半段放行 1 个探测**:窗口过半后,同一 `(源, 模型)` 最多允许 **一个** 请求作为
|
||||
探测通过。探测槽数 = `clamp(max_concurrent / 10, 1, 2)`——`max_concurrent: 10`
|
||||
的源恰好放行 1 个探测请求。
|
||||
- **探测排在最后**:同 tier 内探测候选永远排在健康候选之后,只有没有正常槽可用时
|
||||
才承接真实流量,因此探测不会抢走可用容量。
|
||||
- **探测即真实请求**:成功则立即清零失败计数与冷却,槽位当场回到正常轮询——
|
||||
上游恢复/额度恢复后不必等满整个冷却窗口。失败则重开一个新窗口,下次探测顺延到
|
||||
新窗口的中点,不会连续重试。
|
||||
|
||||
分类冷却策略:
|
||||
|
||||
| 情况 | 冷却 | 说明 |
|
||||
|---|---|---|
|
||||
| 传输错误 / 5xx | 5s→10s→20s…,上限 5min | 指数退避 |
|
||||
| 401 / 403 凭据错误 | 10min | 换 key 后由下一次探测自动接回 |
|
||||
| 429 限流 | 30s 固定 | 「太快」不等于「坏了」,不进指数阶梯 |
|
||||
| 配额耗尽(`insufficient_quota` / `余额不足` 等) | 对齐配额窗口,上限 30min | **不进指数阶梯、不涨失败计数**,额度恢复即可用 |
|
||||
|
||||
WebUI「优先级」页的槽位健康标签会区分 `冷却` / `待探测` / `探测中`,悬停可看窗口
|
||||
起点、探测放行时刻与完全恢复时刻。
|
||||
|
||||
## 源(Source)与适配器(Adapter)的关系
|
||||
|
||||
- **源** 是到某个上游的连接描述:`name`、`base_url`、`api_key`、模型列表与优先级。
|
||||
- **适配器** 是协议转换逻辑(Lua 脚本):把统一 OpenAI 格式请求转成上游原生格式,
|
||||
再把上游响应/流式分块转回统一格式。
|
||||
- 一个适配器可被多个源复用(如 `openai.lua` 同时服务多个 OpenAI 兼容站点);
|
||||
同一个源也可以切换不同适配器(改 `adapter` 字段即可)。
|
||||
- 源决定“连谁、暴露哪些模型”,适配器决定“怎么对话”——二者在 `sources[]` 条目中
|
||||
通过 `adapter` 字段关联。
|
||||
|
||||
加载流程(`internal/core` 的 `Core` 负责装配):
|
||||
|
||||
1. 启动时 `lua.NewVM(adapter_dir)` 加载全部适配器:内置适配器(编译期 embed)+
|
||||
`adapter_dir` 下的同名覆盖文件;内置文件写在代码内,覆盖文件要求更高优先级。
|
||||
2. `config.Load` 读取 `config.yaml`,`config.NewStore(runtime_file)` 读 WebUI 改动的
|
||||
运行时源,二者按名称合并成完整源列表。
|
||||
3. `rebuildRegistry` 为每个源创建 `provider.Provider`(持有目标适配器),并按源的
|
||||
并发上限配置适配器 worker 池**上限**(不预分配,见下)。
|
||||
4. 请求进来时 `Registry` 按 `model` 路由到 Provider,Provider 调适配器
|
||||
`transform_request` → HTTP 发送 → `transform_response` / `transform_stream_chunk`。
|
||||
|
||||
WebUI 上的"新增/编辑源"、"上传 Lua 适配器"、"改 AUTO 优先级链"与"密钥模型范围"
|
||||
都即时生效(写入运行时文件或 `adapter_dir` 后重新装配,无需重启);直接编辑
|
||||
`config.yaml` / `adapter_dir` 下的文件则需要重启进程才会重新加载。
|
||||
|
||||
在 WebUI 删除源/适配器时:运行时(WebUI 创建)的源与 `.lua` 文件会真正移除;
|
||||
`config.yaml` 中定义的基源无法改写配置文件,采用删除标记隐藏(重启后仍隐藏),
|
||||
在 UI 里重新添加同名源即可恢复。
|
||||
|
||||
## 源模板(多 key 负载均衡)
|
||||
|
||||
多个上游 key 共享同一套配置(URL / 适配器 / 模型列表 / 并发 / rpm)时,逐一复制
|
||||
整段 source 太冗余。**源模板**保存除 `name` 和 `api_key` 以外的全部源字段,
|
||||
从模板一键创建多个仅 key 不同的源,天然支持多 key 负载均衡(每个展开源独立调度、
|
||||
独立冷却、独立健康状态)。
|
||||
|
||||
- WebUI「源」页右上角「**模板管理**」按钮:列出全部模板,可编辑 / 删除 / 新建。
|
||||
- 添加源弹窗右上角两个按钮:
|
||||
- **从模板创建** → 选模板 → 表单自动填充(name + key 仍需你填);
|
||||
- **存为模板** → 把当前表单非 key/name 字段存为新模板。
|
||||
- 模板持久化在 `runtime_file` 的 `source_templates` 字段,与运行时源一样热管理,
|
||||
改完即时生效。模板本身不含密钥,可以安全共享 / 纳管。
|
||||
|
||||
模板不会自己变成源——它只是「配方」。真正承担流量的是从模板创建出来的、带 key
|
||||
的具体源。
|
||||
|
||||
23
docs/webui.md
Normal file
23
docs/webui.md
Normal file
@ -0,0 +1,23 @@
|
||||
# WebUI
|
||||
|
||||
> 本文档由 README 拆分而来。
|
||||
> 返回 [README](../README.md)。
|
||||
|
||||
## WebUI
|
||||
|
||||
内置管理界面(`GET /`),登录后可在浏览器完成:
|
||||
|
||||
- **状态页**:源在线状态(实时探测 + 悬停看错误)、模型/源/key 用量统计、请求记录,
|
||||
支持按时间范围导出 CSV;点击模型可生成 pin 到该模型的连接配置。
|
||||
- **对话页**:流式/非流式调试。
|
||||
- **密钥页**:创建/编辑网关 key,为每个 key 配模型范围(模型 + 源 + token 配额 + 周期),
|
||||
管理员管理全部 key,用户只看到自己的 key。每个模型砖块显示自己的配额徽标
|
||||
(如 `1.0K 77×·1h`,未设配额显示 `∞`),点开可编辑该模型的 token 配额、
|
||||
请求数配额与重置周期 —— 配额按模型独立生效,一个用满不影响同一 key 的其它模型。
|
||||
「我的密钥」页对用户展示本 key 的模型范围与各自配额。
|
||||
- **优先级页**:拖拽积木配置 AUTO 链档位。
|
||||
- **源页**:在线增删改上游源(API key 等敏感字段加密落盘)。
|
||||
- **适配器页**:上传 / 删除 Lua 适配器脚本。
|
||||
|
||||
改动写入 `runtime_file`(重启仍生效)。
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
// Package e2e runs the real llmsproxy binary against mocked upstreams over
|
||||
// Package e2e runs the real modelrouter binary against mocked upstreams over
|
||||
// real HTTP: startup, auth, chat, streaming SSE, image generation, and
|
||||
// AUTO failover when a higher-priority source fails.
|
||||
package e2e
|
||||
@ -100,15 +100,15 @@ type gatewayUnderTest struct {
|
||||
func buildBinary(t *testing.T) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
bin := filepath.Join(dir, "llmsproxy")
|
||||
bin := filepath.Join(dir, "modelrouter")
|
||||
if runtime.GOOS == "windows" {
|
||||
bin += ".exe" // go build writes the exact name; Windows needs the suffix to exec
|
||||
}
|
||||
out, err := exec.Command("go", "build", "-tags", "luajit", "-o", bin, "llmsproxy/cmd/llmsproxy").CombinedOutput()
|
||||
out, err := exec.Command("go", "build", "-tags", "luajit", "-o", bin, "modelrouter/cmd/modelrouter").CombinedOutput()
|
||||
if err != nil && runtime.GOOS == "windows" {
|
||||
// local Windows dev box may lack LuaJIT: fall back to the bundled
|
||||
// Lua runtime (the e2e adapters used here are passthrough-only)
|
||||
out, err = exec.Command("go", "build", "-o", bin, "llmsproxy/cmd/llmsproxy").CombinedOutput()
|
||||
out, err = exec.Command("go", "build", "-o", bin, "modelrouter/cmd/modelrouter").CombinedOutput()
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("build: %v\n%s", err, out)
|
||||
|
||||
@ -1,323 +0,0 @@
|
||||
package billing
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
)
|
||||
|
||||
// Compile turns one billing profile into the prices table the billing plugin
|
||||
// expects, resolving URL rules against the gateway's actual sources.
|
||||
//
|
||||
// WHY URL RULES NEED RESOLVING AT ALL: the operator declares pricing by URL
|
||||
// because that is what a provider's price list is keyed on, and because several
|
||||
// sources can point at the same URL. The plugin, however, looks up by
|
||||
// `payload.source` (a source NAME) and by model — it has no idea what URL a
|
||||
// request went to. So the URL match happens here, at compile time, where the
|
||||
// config's name->base_url mapping is known, and the result is expressed in the
|
||||
// dimensions the plugin already supports.
|
||||
//
|
||||
// Profiles are the "let the user choose" axis: the same URL can appear in
|
||||
// several profiles and switching recomputes this table, so a gateway can be
|
||||
// repriced without editing the plugin.
|
||||
func Compile(profile *config.BillingProfile, sources []config.Source) (map[string]interface{}, error) {
|
||||
return CompileOpts(profile, sources, false)
|
||||
}
|
||||
|
||||
// CompileOpts is Compile with an explicit policy for rules that match no
|
||||
// source.
|
||||
//
|
||||
// The default is STRICT and that is correct for startup: a profile with a
|
||||
// typo'd URL prices nothing, every request on it is recorded as unpriced, and
|
||||
// the bill silently comes out wrong. Failing the load is the right response.
|
||||
//
|
||||
// The rules editor needs the opposite. An operator editing a half-finished
|
||||
// profile — adding a rule before the source that will use it exists, or
|
||||
// renaming a URL while typing — must be able to save and then see the warning,
|
||||
// not be blocked by an error they can only resolve by guessing. Lenient mode
|
||||
// keeps the offending rule in the config and drops it from the compiled table,
|
||||
// so it is preserved as text and clearly reported, without pretending it
|
||||
// prices anything.
|
||||
func CompileOpts(profile *config.BillingProfile, sources []config.Source, lenient bool) (map[string]interface{}, error) {
|
||||
if profile == nil {
|
||||
return nil, fmt.Errorf("no billing profile")
|
||||
}
|
||||
prices := map[string]interface{}{
|
||||
"currency": "USD",
|
||||
"default": map[string]interface{}{"prompt": 0.0, "completion": 0.0, "per_request": 0.0},
|
||||
"sources": map[string]interface{}{},
|
||||
"models": map[string]interface{}{},
|
||||
"keys": map[string]interface{}{},
|
||||
}
|
||||
if profile.Default != "" {
|
||||
prices["default_mode"] = profile.Default
|
||||
}
|
||||
|
||||
// Currency is a profile-level statement; a rule may override it.
|
||||
for i := range profile.Rules {
|
||||
if c := profile.Rules[i].Currency; c != "" {
|
||||
prices["currency"] = c
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
matched := map[string]bool{}
|
||||
for i := range profile.Rules {
|
||||
rule := &profile.Rules[i]
|
||||
targets := matchSources(rule.URL, sources)
|
||||
if len(targets) == 0 && rule.URL != "*" {
|
||||
// A rule for a URL no source uses is almost always a typo (or a
|
||||
// source that was removed). Failing loudly beats a profile that
|
||||
// silently prices nothing — the whole reason this is a config file
|
||||
// instead of a hand-written JSON blob.
|
||||
if lenient {
|
||||
continue // kept in the config text, reported as a warning
|
||||
}
|
||||
return nil, fmt.Errorf("rule url %q matches no configured source base_url", rule.URL)
|
||||
}
|
||||
for _, srcName := range targets {
|
||||
if matched[srcName] {
|
||||
// First rule wins. Two rules matching one source is ambiguous,
|
||||
// and silently letting the later one win makes the file's
|
||||
// meaning depend on ordering the operator cannot see.
|
||||
continue
|
||||
}
|
||||
matched[srcName] = true
|
||||
if err := applyRule(prices, rule, srcName, sources); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sources no rule matched fall through to the profile default, which the
|
||||
// plugin applies via prices.default. Recording them explicitly means the
|
||||
// UI can say "this source is unpriced" instead of leaving the operator to
|
||||
// infer it from a zero.
|
||||
var unmatched []string
|
||||
for _, src := range sources {
|
||||
if !matched[src.Name] {
|
||||
unmatched = append(unmatched, src.Name)
|
||||
}
|
||||
}
|
||||
if len(unmatched) > 0 {
|
||||
prices["unmatched_sources"] = unmatched
|
||||
}
|
||||
return prices, nil
|
||||
}
|
||||
|
||||
// matchSources returns the source names whose base_url matches pattern.
|
||||
// "*" matches every source (used as a catch-all default rule).
|
||||
func matchSources(pattern string, sources []config.Source) []string {
|
||||
var out []string
|
||||
if pattern == "*" {
|
||||
for _, s := range sources {
|
||||
out = append(out, s.Name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
want := normalizeURL(pattern)
|
||||
for _, s := range sources {
|
||||
if normalizeURL(s.BaseURL) == want {
|
||||
out = append(out, s.Name)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// normalizeURL compares URLs the way an operator expects: trailing slashes and
|
||||
// case in the host are differences the provider's price list does not care
|
||||
// about, and requiring an exact byte match would make the config brittle.
|
||||
func normalizeURL(u string) string {
|
||||
u = strings.TrimSpace(u)
|
||||
u = strings.TrimRight(u, "/")
|
||||
return strings.ToLower(u)
|
||||
}
|
||||
|
||||
// applyRule writes one rule's pricing for one source into the prices table.
|
||||
func applyRule(prices map[string]interface{}, rule *config.BillingRule, srcName string, sources []config.Source) error {
|
||||
switch rule.Mode {
|
||||
case "free":
|
||||
// Explicitly priced at zero. This is NOT the same as unpriced: a source
|
||||
// the operator says is free must not appear in the unpriced warnings,
|
||||
// or those warnings become noise and stop being read.
|
||||
prices["sources"].(map[string]interface{})[srcName] = map[string]interface{}{
|
||||
"prompt": 0.0, "completion": 0.0, "per_request": 0.0,
|
||||
}
|
||||
return nil
|
||||
|
||||
case "subscription":
|
||||
// A fixed monthly commitment: the per-request MARGINAL cost is zero, and
|
||||
// the flat fee is reported separately. Spreading a monthly fee across
|
||||
// requests would invent a per-request number the provider never charges,
|
||||
// and it would change every time traffic did.
|
||||
prices["sources"].(map[string]interface{})[srcName] = map[string]interface{}{
|
||||
"prompt": 0.0, "completion": 0.0, "per_request": 0.0,
|
||||
}
|
||||
fixed, _ := prices["subscriptions"].(map[string]interface{})
|
||||
if fixed == nil {
|
||||
fixed = map[string]interface{}{}
|
||||
prices["subscriptions"] = fixed
|
||||
}
|
||||
cur := "USD"
|
||||
if rule.Currency != "" {
|
||||
cur = rule.Currency
|
||||
}
|
||||
fixed[srcName] = map[string]interface{}{
|
||||
"monthly": rule.Subscription, "currency": cur,
|
||||
}
|
||||
return nil
|
||||
|
||||
case "unpriced":
|
||||
// Deliberately left out of `sources` so the plugin's unpriced_models /
|
||||
// unpriced_reqs counters catch it. That is the point: a subscription
|
||||
// plan whose credits cannot be converted to tokens must be VISIBLE as
|
||||
// unbilled, not quietly estimated.
|
||||
return nil
|
||||
|
||||
case "token":
|
||||
if len(rule.Models) == 0 {
|
||||
return fmt.Errorf("rule for url %q: mode token requires at least one model", rule.URL)
|
||||
}
|
||||
models := prices["models"].(map[string]interface{})
|
||||
for modelID, t := range rule.Models {
|
||||
entry := map[string]interface{}{}
|
||||
prompt, err := usdPerM(t.Prompt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("model %q prompt: %w", modelID, err)
|
||||
}
|
||||
completion, err := usdPerM(t.Completion)
|
||||
if err != nil {
|
||||
return fmt.Errorf("model %q completion: %w", modelID, err)
|
||||
}
|
||||
entry["prompt"] = prompt
|
||||
entry["completion"] = completion
|
||||
if t.CacheDiscount != nil {
|
||||
entry["cache_discount"] = *t.CacheDiscount
|
||||
}
|
||||
// reasoning is optional and stays ABSENT (not nil, not 0) when the
|
||||
// rule does not set it, which is what makes the billing plugin skip
|
||||
// the repricing branch entirely. Writing an explicit 0 here would
|
||||
// mean "thinking is free" and silently under-bill every request
|
||||
// from a provider that bills it like ordinary output.
|
||||
if strings.TrimSpace(t.Reasoning) != "" {
|
||||
rp, err := usdPerM(t.Reasoning)
|
||||
if err != nil {
|
||||
return fmt.Errorf("model %q reasoning: %w", modelID, err)
|
||||
}
|
||||
entry["reasoning_price"] = rp
|
||||
}
|
||||
if lc := t.LongContext; lc != nil {
|
||||
if lc.Above <= 0 {
|
||||
return fmt.Errorf("model %q long_context: above must be > 0", modelID)
|
||||
}
|
||||
above, err := longContextRate(lc, prompt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("model %q long_context: %w", modelID, err)
|
||||
}
|
||||
entry["long_context"] = map[string]interface{}{
|
||||
"above": lc.Above,
|
||||
// Marginal, not blended. The plugin charges the tokens past the
|
||||
// threshold at this rate and the rest at the base rate, so the
|
||||
// threshold composes with a cache discount (which applies to the
|
||||
// cached portion) instead of forcing the plugin to guess how
|
||||
// the two interact.
|
||||
"prompt": above,
|
||||
}
|
||||
}
|
||||
if rule.Peak != nil {
|
||||
entry["peak"] = compilePeak(rule.Peak)
|
||||
}
|
||||
models[modelID] = entry
|
||||
}
|
||||
// A specific-URL token rule needs a source entry so a model served
|
||||
// from THIS url but absent from `rule.Models` is still recognised as
|
||||
// belonging to a priced provider.
|
||||
//
|
||||
// A WILDCARD rule ("*") must NOT get one. "*" is a model catalogue — it
|
||||
// says "these model ids have these prices, wherever they come from" —
|
||||
// and marking every source priced at {0,0} would make priceFor() set
|
||||
// priced=true for every model the catalogue does NOT list. Paid
|
||||
// sources (commandcode, alittokenplan, …) would then record $0 instead
|
||||
// of showing up in unpriced_reqs, which is precisely the silent
|
||||
// zero-billing failure this whole plugin is built to make visible.
|
||||
// The model lookup p.models[payload.model] already marks a listed
|
||||
// model priced on its own, so the wildcard loses nothing.
|
||||
if rule.URL != "*" {
|
||||
if _, ok := prices["sources"].(map[string]interface{})[srcName]; !ok {
|
||||
prices["sources"].(map[string]interface{})[srcName] = map[string]interface{}{
|
||||
"prompt": 0.0, "completion": 0.0, "per_request": 0.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("unknown mode %q", rule.Mode)
|
||||
}
|
||||
|
||||
// usdPerM converts a USD-per-million string into the per-token rate the plugin
|
||||
// expects. Delegates to config.ParseFloatUSDPerM so the DSL validator and the
|
||||
// compiler agree byte-for-byte on what a valid price is — a value that passes
|
||||
// Validate() but fails here (or vice versa) would be the worst kind of drift.
|
||||
func usdPerM(s string) (float64, error) {
|
||||
return config.ParseFloatUSDPerM(s)
|
||||
}
|
||||
|
||||
// compilePeak renders the peak window in the shape the plugin reads:
|
||||
// { multiplier, windows = { { days = {...}, hours = { {lo,hi}, ... } } } }.
|
||||
//
|
||||
// The plugin reads `days` / `hours` pairs. The first version of the deployed
|
||||
// price table used { start, end, weekdays } — a shape nothing reads — so peak
|
||||
// traffic was billed at off-peak rates with no error anywhere. Compiling from
|
||||
// typed config fields removes the chance of writing the wrong key names by hand.
|
||||
func compilePeak(p *config.BillingPeak) map[string]interface{} {
|
||||
win := map[string]interface{}{}
|
||||
if len(p.Weekdays) > 0 {
|
||||
days := make([]interface{}, 0, len(p.Weekdays))
|
||||
for _, d := range p.Weekdays {
|
||||
days = append(days, d)
|
||||
}
|
||||
win["days"] = days
|
||||
}
|
||||
if len(p.Hours) > 0 {
|
||||
hours := make([]interface{}, 0, len(p.Hours))
|
||||
for _, h := range p.Hours {
|
||||
hours = append(hours, []interface{}{h[0], h[1]})
|
||||
}
|
||||
win["hours"] = hours
|
||||
}
|
||||
out := map[string]interface{}{"multiplier": p.Multiplier}
|
||||
if len(win) > 0 {
|
||||
out["windows"] = []interface{}{win}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// longContextRate resolves the marginal per-token rate for the tokens ABOVE a
|
||||
// threshold, given the model's base per-million prompt price.
|
||||
//
|
||||
// Two ways to express it are supported because providers publish both: an
|
||||
// absolute rate (OpenAI quotes \$3.00/M past 200k) and a multiple (DeepSeek
|
||||
// quotes exactly 2x past 128k). Neither may be missing: a step with no rate is a
|
||||
// silent no-op, and an operator who wrote one meant something.
|
||||
func longContextRate(lc *config.BillingLongContext, basePerToken float64) (float64, error) {
|
||||
hasPrompt := strings.TrimSpace(lc.Prompt) != ""
|
||||
hasMult := lc.Multiplier > 0
|
||||
switch {
|
||||
case hasPrompt && hasMult:
|
||||
p, err := usdPerM(lc.Prompt)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
// Both given: the absolute rate wins, because it is the one the provider
|
||||
// actually quotes. Silently multiplying instead would produce a number
|
||||
// the operator never wrote.
|
||||
return p, nil
|
||||
case hasPrompt:
|
||||
return usdPerM(lc.Prompt)
|
||||
case hasMult:
|
||||
return basePerToken * lc.Multiplier, nil
|
||||
default:
|
||||
return 0, fmt.Errorf("needs either prompt or multiplier")
|
||||
}
|
||||
}
|
||||
@ -1,214 +0,0 @@
|
||||
package billing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
)
|
||||
|
||||
// The wildcard token rule must not blanket-mark every source as priced.
|
||||
//
|
||||
// Why this is a hazard and not a style question: priceFor() sets priced=true as
|
||||
// soon as the request's SOURCE has an entry in the price table, and that flag
|
||||
// is what keeps a request OUT of unpriced_reqs. A "*" rule is a model
|
||||
// catalogue — it prices specific model ids wherever they come from — but
|
||||
// compiling it into a {0,0,0} entry on all 21 sources made every model the
|
||||
// catalogue does not list look "priced at zero". Paid providers (commandcode,
|
||||
// alittokenplan, …) then stopped appearing in the unpriced warnings, which is
|
||||
// exactly the silent zero-billing failure the plugin exists to surface.
|
||||
func TestWildcardTokenRuleDoesNotMarkEverySourcePriced(t *testing.T) {
|
||||
sources := []config.Source{
|
||||
{Name: "free1", BaseURL: "https://free1.example.com/v1"},
|
||||
{Name: "paid1", BaseURL: "https://paid1.example.com/v1"},
|
||||
{Name: "paid2", BaseURL: "https://paid2.example.com/v1"},
|
||||
}
|
||||
profile := &config.BillingProfile{
|
||||
ID: "p",
|
||||
Rules: []config.BillingRule{
|
||||
{URL: "https://free1.example.com/v1", Mode: "free"},
|
||||
{
|
||||
URL: "*", Mode: "token",
|
||||
Models: map[string]config.BillingToken{
|
||||
"only-catalogued-model": {Prompt: "1.00", Completion: "2.00"},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
prices, err := Compile(profile, sources)
|
||||
if err != nil {
|
||||
t.Fatalf("compile: %v", err)
|
||||
}
|
||||
srcs := prices["sources"].(map[string]interface{})
|
||||
|
||||
// The explicitly free source keeps its entry (free is a deliberate 0).
|
||||
if _, ok := srcs["free1"]; !ok {
|
||||
t.Error("the explicit free rule lost its source entry")
|
||||
}
|
||||
// Paid sources must NOT be marked, or their unlisted models bill $0.
|
||||
for _, name := range []string{"paid1", "paid2"} {
|
||||
if _, ok := srcs[name]; ok {
|
||||
t.Errorf("wildcard token rule marked %q as priced at 0 — its unlisted "+
|
||||
"models will bill $0 instead of appearing as unpriced", name)
|
||||
}
|
||||
}
|
||||
// The catalogue itself must still be present, or the rule prices nothing.
|
||||
models := prices["models"].(map[string]interface{})
|
||||
if _, ok := models["only-catalogued-model"]; !ok {
|
||||
t.Error("the wildcard rule did not publish its model price")
|
||||
}
|
||||
}
|
||||
|
||||
// The inverse guard: a token rule for a SPECIFIC url still marks that source,
|
||||
// because it genuinely declares pricing for the models served from it.
|
||||
func TestSpecificURLTokenRuleStillMarksItsSource(t *testing.T) {
|
||||
sources := []config.Source{
|
||||
{Name: "mine", BaseURL: "https://mine.example.com/v1"},
|
||||
{Name: "other", BaseURL: "https://other.example.com/v1"},
|
||||
}
|
||||
profile := &config.BillingProfile{
|
||||
ID: "p",
|
||||
Rules: []config.BillingRule{{
|
||||
URL: "https://mine.example.com/v1", Mode: "token",
|
||||
Models: map[string]config.BillingToken{
|
||||
"m": {Prompt: "1.00", Completion: "2.00"},
|
||||
},
|
||||
}},
|
||||
}
|
||||
prices, err := Compile(profile, sources)
|
||||
if err != nil {
|
||||
t.Fatalf("compile: %v", err)
|
||||
}
|
||||
srcs := prices["sources"].(map[string]interface{})
|
||||
if _, ok := srcs["mine"]; !ok {
|
||||
t.Error("a specific-url token rule must still mark its own source priced")
|
||||
}
|
||||
if _, ok := srcs["other"]; ok {
|
||||
t.Error("an unrelated source was marked priced by a specific-url rule")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReasoningPriceIsOmittedUnlessSet pins the absence convention for the
|
||||
// thinking-token rate.
|
||||
//
|
||||
// A rule that says nothing about reasoning must produce a price entry with NO
|
||||
// reasoning_price key, so the billing plugin skips the repricing branch and
|
||||
// bills thinking at the ordinary output rate — which is what every provider
|
||||
// actually charges. Writing an explicit 0 instead would read as "thinking is
|
||||
// free" and silently under-bill every request from that model.
|
||||
func TestReasoningPriceIsOmittedUnlessSet(t *testing.T) {
|
||||
sources := []config.Source{{Name: "up", BaseURL: "https://up.example/v1"}}
|
||||
|
||||
// Case 1: no reasoning declared → the key must not appear.
|
||||
p1 := &config.BillingProfile{ID: "p", Rules: []config.BillingRule{{
|
||||
URL: "https://up.example/v1", Mode: "token",
|
||||
Models: map[string]config.BillingToken{
|
||||
"m": {Prompt: "1.00", Completion: "2.00"},
|
||||
},
|
||||
}}}
|
||||
prices, err := Compile(p1, sources)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entry := prices["models"].(map[string]interface{})["m"].(map[string]interface{})
|
||||
if _, present := entry["reasoning_price"]; present {
|
||||
t.Errorf("reasoning_price present (%v) without a rule; the plugin would treat 0 as free",
|
||||
entry["reasoning_price"])
|
||||
}
|
||||
|
||||
// Case 2: declared → present, and converted to per-token like every other rate.
|
||||
p2 := &config.BillingProfile{ID: "p", Rules: []config.BillingRule{{
|
||||
URL: "https://up.example/v1", Mode: "token",
|
||||
Models: map[string]config.BillingToken{
|
||||
"m": {Prompt: "1.00", Completion: "2.00", Reasoning: "6.00"},
|
||||
},
|
||||
}}}
|
||||
prices2, err := Compile(p2, sources)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e2 := prices2["models"].(map[string]interface{})["m"].(map[string]interface{})
|
||||
rp, ok := e2["reasoning_price"].(float64)
|
||||
if !ok {
|
||||
t.Fatalf("reasoning_price missing or wrong type: %#v", e2["reasoning_price"])
|
||||
}
|
||||
if rp != 6e-6 {
|
||||
t.Errorf("reasoning_price = %v, want 6e-6 (USD per TOKEN)", rp)
|
||||
}
|
||||
// A malformed rate must be rejected, not silently ignored: an operator who
|
||||
// typed "6 USD" expects either that value or an error, never a rule that
|
||||
// looks applied and bills nothing.
|
||||
p3 := &config.BillingProfile{ID: "p", Rules: []config.BillingRule{{
|
||||
URL: "https://up.example/v1", Mode: "token",
|
||||
Models: map[string]config.BillingToken{
|
||||
"m": {Prompt: "1.00", Completion: "2.00", Reasoning: "six dollars"},
|
||||
},
|
||||
}}}
|
||||
if _, err := Compile(p3, sources); err == nil {
|
||||
t.Error("a non-numeric reasoning rate must fail the compile")
|
||||
}
|
||||
}
|
||||
|
||||
// TestLongContextStepCompiles pins the long-context step's two accepted forms
|
||||
// and its rejection of a step that would silently do nothing.
|
||||
//
|
||||
// Providers publish both shapes — DeepSeek says "2x past 128k", OpenAI quotes
|
||||
// an absolute "$3.00/M past 200k" — so both must compile. Neither may be
|
||||
// missing: a step with no rate is a rule that LOOKS like it handles long
|
||||
// context and does not, which under-bills without any visible sign.
|
||||
func TestLongContextStepCompiles(t *testing.T) {
|
||||
sources := []config.Source{{Name: "up", BaseURL: "https://up.example/v1"}}
|
||||
|
||||
compile := func(lc config.BillingLongContext) (map[string]interface{}, error) {
|
||||
p := &config.BillingProfile{ID: "p", Rules: []config.BillingRule{{
|
||||
URL: "https://up.example/v1", Mode: "token",
|
||||
Models: map[string]config.BillingToken{
|
||||
"m": {Prompt: "1.00", Completion: "2.00", LongContext: &lc},
|
||||
},
|
||||
}}}
|
||||
prices, err := Compile(p, sources)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
e := prices["models"].(map[string]interface{})["m"].(map[string]interface{})
|
||||
return e["long_context"].(map[string]interface{}), nil
|
||||
}
|
||||
|
||||
// Absolute rate (OpenAI's published shape).
|
||||
lc1, err := compile(config.BillingLongContext{Above: 200000, Prompt: "3.00"})
|
||||
if err != nil {
|
||||
t.Fatalf("absolute long_context rate rejected: %v", err)
|
||||
}
|
||||
if lc1["above"] != 200000 || lc1["prompt"] != 3e-6 {
|
||||
t.Errorf("absolute step = %#v, want above=200000 prompt=3e-6", lc1)
|
||||
}
|
||||
|
||||
// Multiplier (DeepSeek's shape): 2x of the 1.00/M base.
|
||||
lc2, err := compile(config.BillingLongContext{Above: 128000, Multiplier: 2})
|
||||
if err != nil {
|
||||
t.Fatalf("multiplier long_context rejected: %v", err)
|
||||
}
|
||||
if lc2["prompt"] != 2e-6 {
|
||||
t.Errorf("multiplier step = %#v, want prompt=2e-6", lc2)
|
||||
}
|
||||
|
||||
// Both given: the absolute rate wins, because that is the number the
|
||||
// provider quotes. Multiplying instead would produce a figure the operator
|
||||
// never wrote.
|
||||
lc3, err := compile(config.BillingLongContext{Above: 128000, Prompt: "5.00", Multiplier: 2})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if lc3["prompt"] != 5e-6 {
|
||||
t.Errorf("both given = %#v, want the quoted absolute rate 5e-6", lc3)
|
||||
}
|
||||
|
||||
// Neither: a no-op step must fail, not compile into a rule that silently
|
||||
// bills long context at the base rate.
|
||||
if _, err := compile(config.BillingLongContext{Above: 128000}); err == nil {
|
||||
t.Error("a step with neither prompt nor multiplier must be rejected")
|
||||
}
|
||||
// A zero threshold would make every request "long".
|
||||
if _, err := compile(config.BillingLongContext{Above: 0, Multiplier: 2}); err == nil {
|
||||
t.Error("above=0 must be rejected")
|
||||
}
|
||||
}
|
||||
@ -7,7 +7,6 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@ -30,15 +29,6 @@ type Config struct {
|
||||
Auto []ModelScope `yaml:"auto,omitempty"` // AUTO 调度链规则(WebUI 优先级页编辑,chat)
|
||||
AutoImage []ModelScope `yaml:"auto_image,omitempty"` // AUTO 生图调度链规则(WebUI 优先级页·生图)
|
||||
Keys []GWKey `yaml:"keys,omitempty"` // 网关密钥(WebUI 密钥页管理)
|
||||
// BillingDSL declares per-URL pricing profiles for the billing plugin.
|
||||
//
|
||||
// Declarative, because hand-writing the prices JSON was error-prone in ways
|
||||
// that looked successful: the first PUT went out without the required
|
||||
// `prices` wrapper and silently REPLACED the accumulated totals with the
|
||||
// price table; the second used a peak-window shape the plugin does not read,
|
||||
// so peak traffic was billed at off-peak rates — no error, the bill just
|
||||
// quietly halved. A schema-checked config file catches both at load.
|
||||
BillingDSL *BillingDSL `yaml:"billing,omitempty"`
|
||||
// PluginVisibility overrides the audience a plugin declared for itself,
|
||||
// keyed by plugin name: "admin" hides a plugin (and its state) from user
|
||||
// keys, "all" widens one that shipped admin-only, "user,admin" names roles
|
||||
@ -95,7 +85,14 @@ type Source struct {
|
||||
// ProxyURL routes this source's HTTP(S) traffic through an explicit proxy
|
||||
// (e.g. http://127.0.0.1:7890). Empty = direct connection. Needed for
|
||||
// upstreams behind geo/IP blocks that only respond through a proxy.
|
||||
ProxyURL string `yaml:"proxy_url" json:"proxy_url,omitempty"`
|
||||
ProxyURL string `yaml:"proxy_url" json:"proxy_url,omitempty"`
|
||||
// Retry is an OPTIONAL upstream retry policy. It exists for upstreams that
|
||||
// answer a business-level rejection with a status code the kernel would
|
||||
// otherwise surface to the client — e.g. a per-account concurrency cap that
|
||||
// clears by itself after tens of seconds. The kernel stays vendor-agnostic:
|
||||
// the matcher is pure DATA (status list + response-body substrings), so no
|
||||
// provider name ever appears in core.
|
||||
Retry *RetryPolicy `yaml:"retry" json:"retry,omitempty"`
|
||||
Meta map[string]interface{} `yaml:"meta" json:"meta,omitempty"`
|
||||
Temperature float64 `yaml:"temperature" json:"temperature,omitempty"`
|
||||
MaxTokens int `yaml:"max_tokens" json:"max_tokens,omitempty"`
|
||||
@ -105,9 +102,66 @@ type Source struct {
|
||||
RPM int `yaml:"rpm" json:"rpm,omitempty"` // optional requests-per-minute cap (0 = unlimited)
|
||||
}
|
||||
|
||||
// RetryPolicy describes when a failed upstream attempt may be retried BEFORE
|
||||
// anything is returned to the client. All conditions are data:
|
||||
//
|
||||
// OnStatus - retry only when the HTTP status is one of these
|
||||
// OnBodyContains - ... AND the response body contains one of these substrings
|
||||
// (empty = status alone decides)
|
||||
// Delay - wait between attempts ("10s" etc.); parsed on demand
|
||||
// MaxAttempts - TOTAL attempts including the first (1 or 0 = no retry)
|
||||
//
|
||||
// Scope: the retry happens inside one gateway request, so a stream is only
|
||||
// retried before its first chunk reaches the client (the kernel already holds
|
||||
// the first chunk back), and a non-stream call is retried as a whole.
|
||||
type RetryPolicy struct {
|
||||
OnStatus []int `yaml:"on_status,omitempty" json:"on_status,omitempty"`
|
||||
OnBodyContains []string `yaml:"on_body_contains,omitempty" json:"on_body_contains,omitempty"`
|
||||
Delay string `yaml:"delay,omitempty" json:"delay,omitempty"`
|
||||
MaxAttempts int `yaml:"max_attempts,omitempty" json:"max_attempts,omitempty"`
|
||||
}
|
||||
|
||||
// ShouldRetry reports whether one failed attempt matches this policy.
|
||||
func (r *RetryPolicy) ShouldRetry(status int, body string) bool {
|
||||
if r == nil || r.MaxAttempts <= 1 {
|
||||
return false
|
||||
}
|
||||
statusOK := len(r.OnStatus) == 0
|
||||
for _, s := range r.OnStatus {
|
||||
if s == status {
|
||||
statusOK = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !statusOK {
|
||||
return false
|
||||
}
|
||||
if len(r.OnBodyContains) == 0 {
|
||||
return true
|
||||
}
|
||||
for _, sub := range r.OnBodyContains {
|
||||
if sub != "" && strings.Contains(body, sub) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// DelayDuration is the configured pause between attempts (default 10s).
|
||||
func (r *RetryPolicy) DelayDuration() time.Duration {
|
||||
if r == nil || r.Delay == "" {
|
||||
return 10 * time.Second
|
||||
}
|
||||
d, err := time.ParseDuration(r.Delay)
|
||||
if err != nil || d < 0 {
|
||||
return 10 * time.Second
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// Load reads and validates a config file. When the file does not exist yet a
|
||||
// default config is generated at that path first (first-run bootstrap), so a
|
||||
// fresh binary just works: `llmsproxy -config /path/to/config.yaml`.
|
||||
// fresh binary just works: `modelrouter -config /path/to/config.yaml`.
|
||||
func Load(path string) (*Config, error) {
|
||||
if _, err := EnsureDefault(path); err != nil {
|
||||
return nil, err
|
||||
@ -423,182 +477,6 @@ type GWKey struct {
|
||||
// chain" so a user can clear an override back to the global default.
|
||||
func (k GWKey) HasOwnAuto() bool { return len(k.Auto) > 0 }
|
||||
|
||||
// BillingDSL holds declarative per-URL pricing profiles for the billing plugin.
|
||||
//
|
||||
// Profiles are the "let the user pick" axis: the same upstream URL can be
|
||||
// declared under multiple profiles, and the operator chooses which is active.
|
||||
// Switching recomputes and re-injects the billing plugin's prices table.
|
||||
type BillingDSL struct {
|
||||
// Active is the id of the profile in effect. Empty => the first profile.
|
||||
Active string `yaml:"active,omitempty" json:"active,omitempty"`
|
||||
Profiles []BillingProfile `yaml:"profiles,omitempty" json:"profiles,omitempty"`
|
||||
}
|
||||
|
||||
// BillingProfile is one named set of per-URL pricing rules.
|
||||
type BillingProfile struct {
|
||||
ID string `yaml:"id" json:"id"`
|
||||
Label string `yaml:"label,omitempty" json:"label,omitempty"`
|
||||
// Default applies when no rule matches: "free" or "unpriced".
|
||||
Default string `yaml:"default,omitempty" json:"default,omitempty"`
|
||||
Rules []BillingRule `yaml:"rules,omitempty" json:"rules,omitempty"`
|
||||
}
|
||||
|
||||
// BillingRule declares the pricing for everything under one URL.
|
||||
type BillingRule struct {
|
||||
// URL matches a source's base_url. "*" = any unmatched URL.
|
||||
URL string `yaml:"url" json:"url"`
|
||||
// Mode is one of: free, token, subscription, unpriced.
|
||||
Mode string `yaml:"mode" json:"mode"`
|
||||
Currency string `yaml:"currency,omitempty" json:"currency,omitempty"`
|
||||
// Subscription is the flat monthly cost (for mode=subscription).
|
||||
Subscription float64 `yaml:"subscription,omitempty" json:"subscription,omitempty"`
|
||||
// Models declares per-token prices when mode=token.
|
||||
Models map[string]BillingToken `yaml:"models,omitempty" json:"models,omitempty"`
|
||||
// Free is true when mode=free (kept for explicitness).
|
||||
// Peak applies to ALL token-priced requests under this URL.
|
||||
Peak *BillingPeak `yaml:"peak,omitempty" json:"peak,omitempty"`
|
||||
}
|
||||
|
||||
// BillingToken is the per-model price.
|
||||
type BillingToken struct {
|
||||
Prompt string `yaml:"prompt" json:"prompt"` // USD per million tokens, e.g. "0.15"
|
||||
Completion string `yaml:"completion" json:"completion"` // USD per million tokens
|
||||
CacheDiscount *float64 `yaml:"cache_discount,omitempty" json:"cache_discount,omitempty"`
|
||||
// Reasoning is the USD-per-million rate for thinking tokens, for the rare
|
||||
// provider that charges them differently from ordinary output. Omit it and
|
||||
// reasoning is billed at Completion — which is what OpenAI, DeepSeek and
|
||||
// Anthropic all do, and what every source on this deployment reports.
|
||||
//
|
||||
// It is a rate for a SUBSET of completion tokens, never an extra charge: the
|
||||
// billing plugin applies only the difference between this and Completion, so
|
||||
// setting it to a multiple of Completion reprices the thinking share rather
|
||||
// than billing it twice.
|
||||
Reasoning string `yaml:"reasoning,omitempty" json:"reasoning,omitempty"`
|
||||
// LongContext is the prompt-price step above a token threshold. Nil = one
|
||||
// flat rate, which is what a rule means when it says nothing.
|
||||
LongContext *BillingLongContext `yaml:"long_context,omitempty" json:"long_context,omitempty"`
|
||||
}
|
||||
|
||||
// BillingPeak declares peak-hour pricing.
|
||||
type BillingPeak struct {
|
||||
Multiplier float64 `yaml:"multiplier" json:"multiplier"`
|
||||
// Weekdays: 1=Mon..5=Fri (Lua os.date !%w, 0=Sun)
|
||||
Weekdays []int `yaml:"weekdays,omitempty" json:"weekdays,omitempty"`
|
||||
// Hours: each pair is [lo,hi] inclusive
|
||||
Hours [][2]int `yaml:"hours,omitempty" json:"hours,omitempty"`
|
||||
}
|
||||
|
||||
// BillingLongContext is a prompt-price step for long inputs.
|
||||
//
|
||||
// Every major provider charges the input tokens ABOVE a threshold at a higher
|
||||
// rate — DeepSeek doubles past 128k, OpenAI and Anthropic step up as well. A
|
||||
// flat prompt price therefore under-bills exactly the traffic an agent workload
|
||||
// generates, and it under-bills it silently: the number is plausible, the bill
|
||||
// adds up, and nothing says the rate was wrong.
|
||||
//
|
||||
// The step applies to the PROMPT only. Output and cache reads are billed at
|
||||
// their own rates regardless of input length, which is how providers publish
|
||||
// it — the higher rate is for the extra context you made the model read, not for
|
||||
// what it wrote back.
|
||||
type BillingLongContext struct {
|
||||
// Above is the token count the step starts at. A request with exactly this
|
||||
// many prompt tokens is still at the base rate; the step covers the tokens
|
||||
// BEYOND it. Inclusive-below matches how the providers document it.
|
||||
Above int `yaml:"above" json:"above"`
|
||||
// Prompt is the USD-per-million rate for the tokens above Above.
|
||||
Prompt string `yaml:"prompt" json:"prompt"`
|
||||
// Multiplier is an alternative to Prompt: a factor applied to the base rate
|
||||
// (DeepSeek's published step is exactly 2x). When both are set Prompt wins,
|
||||
// and setting neither is a validation error rather than a silent no-op.
|
||||
Multiplier float64 `yaml:"multiplier,omitempty" json:"multiplier,omitempty"`
|
||||
}
|
||||
|
||||
// Validate rejects a BillingDSL that could not work as written.
|
||||
func (b *BillingDSL) Validate() error {
|
||||
if b == nil {
|
||||
return nil
|
||||
}
|
||||
ids := map[string]bool{}
|
||||
for i := range b.Profiles {
|
||||
p := &b.Profiles[i]
|
||||
if p.ID == "" {
|
||||
return fmt.Errorf("billing profile [%d] has no id", i)
|
||||
}
|
||||
if ids[p.ID] {
|
||||
return fmt.Errorf("billing profile id %q appears more than once", p.ID)
|
||||
}
|
||||
ids[p.ID] = true
|
||||
if p.Default != "" && p.Default != "free" && p.Default != "unpriced" {
|
||||
return fmt.Errorf("profile %q: default must be free or unpriced, got %q", p.ID, p.Default)
|
||||
}
|
||||
for j := range p.Rules {
|
||||
r := &p.Rules[j]
|
||||
if r.URL == "" {
|
||||
return fmt.Errorf("profile %q rule [%d]: url is required", p.ID, j)
|
||||
}
|
||||
switch r.Mode {
|
||||
case "free", "token", "subscription", "unpriced":
|
||||
default:
|
||||
return fmt.Errorf("profile %q rule [%d]: mode must be free/token/subscription/unpriced, got %q", p.ID, j, r.Mode)
|
||||
}
|
||||
if r.Mode == "token" {
|
||||
for m, t := range r.Models {
|
||||
if t.Prompt == "" || t.Completion == "" {
|
||||
return fmt.Errorf("profile %q model %q: prompt and completion are required for token mode", p.ID, m)
|
||||
}
|
||||
if _, err := ParseFloatUSDPerM(t.Prompt); err != nil {
|
||||
return fmt.Errorf("profile %q model %q prompt: %w", p.ID, m, err)
|
||||
}
|
||||
if _, err := ParseFloatUSDPerM(t.Completion); err != nil {
|
||||
return fmt.Errorf("profile %q model %q completion: %w", p.ID, m, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Active must resolve to a profile id.
|
||||
if b.Active != "" && !ids[b.Active] {
|
||||
return fmt.Errorf("billing.active = %q but no profile has that id", b.Active)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Resolve returns the profile with the given id; empty id (or an id that does
|
||||
// not exist) falls back to the FIRST profile. A config with profiles but no
|
||||
// active marker should mean "use the one I wrote first" rather than "price
|
||||
// nothing", which is what a nil here would silently do.
|
||||
func (b *BillingDSL) Resolve(id string) *BillingProfile {
|
||||
if b == nil || len(b.Profiles) == 0 {
|
||||
return nil
|
||||
}
|
||||
if id != "" {
|
||||
for i := range b.Profiles {
|
||||
if b.Profiles[i].ID == id {
|
||||
return &b.Profiles[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
return &b.Profiles[0]
|
||||
}
|
||||
|
||||
// parseFloatUSDPerM parses a USD-per-million price string. Returns the
|
||||
// per-single-token rate the plugin expects (price / 1e6).
|
||||
//
|
||||
// Kept as a string in the DSL rather than a float: "0.15" survives round-trips
|
||||
// through YAML editors, and a malformed value must fail VALIDATION (with the
|
||||
// file and field named) rather than parse as 0 and under-bill silently — which
|
||||
// is exactly the failure mode that motivated the DSL.
|
||||
func ParseFloatUSDPerM(s string) (float64, error) {
|
||||
f, err := strconv.ParseFloat(strings.TrimSpace(s), 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("not a number: %q", s)
|
||||
}
|
||||
if f < 0 {
|
||||
return 0, fmt.Errorf("negative price: %q", s)
|
||||
}
|
||||
return f / 1e6, nil
|
||||
}
|
||||
|
||||
// KeyQuota is retained only to carry a scope entry's caps through the admin
|
||||
// API. Quotas are per model, never per key: there is deliberately no key-wide
|
||||
// total, so exhausting one model's budget never blocks the others.
|
||||
|
||||
59
internal/config/retry_policy_test.go
Normal file
59
internal/config/retry_policy_test.go
Normal file
@ -0,0 +1,59 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ShouldRetry 是纯数据判据:状态码列表 + 响应体子串。它决定"这次失败值不值得
|
||||
// 在网关内部等一会儿重试"。厂商知识绝不能进内核,所以判据必须能用一个表格测清楚。
|
||||
func TestRetryPolicyShouldRetry(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
policy *RetryPolicy
|
||||
status int
|
||||
body string
|
||||
want bool
|
||||
}{
|
||||
{"nil policy", nil, 500, "anything", false},
|
||||
{"max_attempts=1 disables", &RetryPolicy{OnStatus: []int{400}, MaxAttempts: 1}, 400, "x", false},
|
||||
{"status matches, no body filter",
|
||||
&RetryPolicy{OnStatus: []int{429}, MaxAttempts: 3}, 429, "", true},
|
||||
{"status mismatch",
|
||||
&RetryPolicy{OnStatus: []int{429}, MaxAttempts: 3}, 400, "429", false},
|
||||
{"status + body match",
|
||||
&RetryPolicy{OnStatus: []int{400}, OnBodyContains: []string{"TM.00001041"}, MaxAttempts: 3},
|
||||
400, `{"error_code":"TM.00001041"}`, true},
|
||||
{"status matches but body does not",
|
||||
&RetryPolicy{OnStatus: []int{400}, OnBodyContains: []string{"TM.00001041"}, MaxAttempts: 3},
|
||||
400, `{"error_code":"TM.00001005"}`, false},
|
||||
{"empty substring never matches",
|
||||
&RetryPolicy{OnStatus: []int{400}, OnBodyContains: []string{""}, MaxAttempts: 3}, 400, "x", false},
|
||||
{"any of several body signals",
|
||||
&RetryPolicy{OnStatus: []int{400, 429}, OnBodyContains: []string{"TM.00001041", "TPM limit"}, MaxAttempts: 3},
|
||||
429, `{"message":"TPM limit exceeded"}`, true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := tc.policy.ShouldRetry(tc.status, tc.body); got != tc.want {
|
||||
t.Errorf("%s: ShouldRetry(%d, %q) = %v, want %v", tc.name, tc.status, tc.body, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryPolicyDelay(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
policy *RetryPolicy
|
||||
want time.Duration
|
||||
}{
|
||||
{"nil", nil, 10 * time.Second},
|
||||
{"explicit", &RetryPolicy{Delay: "250ms"}, 250 * time.Millisecond},
|
||||
{"garbage falls back", &RetryPolicy{Delay: "not-a-duration"}, 10 * time.Second},
|
||||
{"negative falls back", &RetryPolicy{Delay: "-5s"}, 10 * time.Second},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := tc.policy.DelayDuration(); got != tc.want {
|
||||
t.Errorf("%s: DelayDuration() = %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -27,8 +27,9 @@ func NewGatewayKey() (string, error) {
|
||||
|
||||
// SecretBox encrypts and decrypts sensitive values (upstream API keys,
|
||||
// custom header values, gateway keys) for persist-time protection. The
|
||||
// master key comes from the LLMS_PROXY_MASTER_KEY environment variable
|
||||
// (hex, 64 chars) or from a generated <runtime-dir>/master.key file.
|
||||
// master key comes from the MODELROUTER_MASTER_KEY environment variable
|
||||
// (hex, 64 chars; the pre-rename LLMS_PROXY_MASTER_KEY is still accepted) or
|
||||
// from a generated <runtime-dir>/master.key file.
|
||||
type SecretBox struct {
|
||||
aead cipher.AEAD
|
||||
}
|
||||
@ -37,10 +38,18 @@ type SecretBox struct {
|
||||
// file next to the runtime file. If neither exists, a fresh random key is
|
||||
// generated and written to master.key with mode 0600.
|
||||
func NewSecretBox(runtimeFile string) (*SecretBox, error) {
|
||||
if v := os.Getenv("LLMS_PROXY_MASTER_KEY"); v != "" {
|
||||
key, err := hex.DecodeString(strings.TrimSpace(v))
|
||||
// New name first, then the pre-rename name: an existing unit file that still
|
||||
// exports LLMS_PROXY_MASTER_KEY must keep starting after an upgrade.
|
||||
envKey := os.Getenv("MODELROUTER_MASTER_KEY")
|
||||
envName := "MODELROUTER_MASTER_KEY"
|
||||
if envKey == "" {
|
||||
envKey = os.Getenv("LLMS_PROXY_MASTER_KEY")
|
||||
envName = "LLMS_PROXY_MASTER_KEY"
|
||||
}
|
||||
if envKey != "" {
|
||||
key, err := hex.DecodeString(strings.TrimSpace(envKey))
|
||||
if err != nil || len(key) != 32 {
|
||||
return nil, fmt.Errorf("LLMS_PROXY_MASTER_KEY: expected 64 hex chars")
|
||||
return nil, fmt.Errorf("%s: expected 64 hex chars", envName)
|
||||
}
|
||||
return newBox(key)
|
||||
}
|
||||
|
||||
@ -6,7 +6,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// callback 平台的授权 URL 是最容易"看起来对、实际不可用"的地方:URL 能拼出来、
|
||||
|
||||
@ -9,7 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 三个 callback 平台的回调路径**各是注册值**,且校验规则不同:
|
||||
|
||||
@ -3,7 +3,7 @@ package core
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ 无凭据账号的批量清理:**只删没有凭据的**,有凭据的一律不动。
|
||||
|
||||
@ -8,7 +8,7 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ codearts 的模型目录必须从**两个**签名端点拉取并合并。
|
||||
|
||||
@ -8,7 +8,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// CodeArts 端到端:起真实回调服务器 → 模上游回 code → 走真实 STS 交换
|
||||
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// CodeArts 的 portal 拒绝 <10000 的回调端口(参考 listenOnCallbackPort 的
|
||||
|
||||
@ -7,7 +7,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 有道每日活动领取是**三步流程**,参考注释点名:无可用活动是**正常业务状态**,
|
||||
|
||||
@ -2,7 +2,7 @@ package core
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
@ -10,7 +10,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 有道登录闭环的判据:portal 会把授权重定向回 **本机 loopback 接收器**
|
||||
|
||||
@ -8,7 +8,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// loomyHarness:Loomy 微信扫码长轮询的测试台。
|
||||
|
||||
@ -8,7 +8,7 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 远端模型目录的判据:**远端为主 + 兜底补位**(照参考 reconcileWithFallback)。
|
||||
|
||||
@ -9,7 +9,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// channelsMoreTest:新引入的三条真实流程 + 两个边界流的行为对齐测试。
|
||||
|
||||
@ -5,12 +5,12 @@ import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ 插件加载时必须清掉上一进程留下的孤儿回调文件。
|
||||
//
|
||||
// 真实事故:生产 `/etc/llmsproxy/plugins/` 攒了 **18 个 callback-*.port**
|
||||
// 真实事故:生产 `/etc/modelrouter/plugins/` 攒了 **18 个 callback-*.port**
|
||||
// (对应的 .json 早已消失)。它们永远不会被清,因为:
|
||||
//
|
||||
// · loopbacks 追踪表只活在内存(plugin.state.loopbacks);
|
||||
@ -25,7 +25,7 @@ func TestPluginLoadSweepsOrphanCallbackFiles(t *testing.T) {
|
||||
if err := os.MkdirAll(pluginDir, 0755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("LLMSPROXY_PLUGIN_DIR", pluginDir)
|
||||
t.Setenv("MODELROUTER_PLUGIN_DIR", pluginDir)
|
||||
|
||||
// 摆上一轮的残留(模拟上一进程被强杀后的现场)
|
||||
orphans := []string{
|
||||
|
||||
56
internal/core/channels_platforms_logged_in_test.go
Normal file
56
internal/core/channels_platforms_logged_in_test.go
Normal file
@ -0,0 +1,56 @@
|
||||
package core
|
||||
|
||||
import "testing"
|
||||
|
||||
// `platforms` 的 logged_in 必须看**账号池**,不能只看旧单账号保险库键。
|
||||
//
|
||||
// 真实缺陷(2026-10-20 用户报「codearts 还是不可用」):
|
||||
// `list_platforms` 当时写的是 `credentialOf("channels:" .. id)` —— 那是旧版
|
||||
// 单账号的键;而现网凭据键是 `channels:<platform>:<accountId>`(一平台多账号)。
|
||||
// 于是**所有**通过账号池接入的平台都被判成未登录,UI 把渠道卡片渲染成"未接入",
|
||||
// 用户看到的正是"渠道不可用",而底层模型其实完全能用。
|
||||
//
|
||||
// 判据钉两层:池里的账号让 logged_in=true,且账号标识能被 UI 拿到。
|
||||
func TestPlatformsLoggedInFromAccountPool(t *testing.T) {
|
||||
_, ps := poolCore(t)
|
||||
if _, err := ps.Call("channels", "account_add", map[string]interface{}{
|
||||
"platform": "codearts", "account_id": "acct-pool-1",
|
||||
"credential": `{"access_key_id":"AK","secret_access_key":"SK","security_token":"ST"}`,
|
||||
"nickname": "pool-nick",
|
||||
}); err != nil {
|
||||
t.Fatalf("account_add: %v", err)
|
||||
}
|
||||
// list_platforms 是 HTTP 路由处理器(不在 plugin.exports 里),走探针取同一份数据
|
||||
m, err := ps.Call("channels", "__platforms_probe", map[string]interface{}{})
|
||||
if err != nil {
|
||||
t.Fatalf("__platforms_probe: %v", err)
|
||||
}
|
||||
w, _ := m.(map[string]interface{})
|
||||
j, _ := w["json"].(map[string]interface{})
|
||||
arr, _ := j["platforms"].([]interface{})
|
||||
if len(arr) == 0 {
|
||||
t.Fatalf("platforms 不应为空")
|
||||
}
|
||||
var got map[string]interface{}
|
||||
for _, raw := range arr {
|
||||
p, _ := raw.(map[string]interface{})
|
||||
if p["id"] == "codearts" {
|
||||
got = p
|
||||
}
|
||||
}
|
||||
if got == nil {
|
||||
t.Fatalf("platforms 里找不到 codearts")
|
||||
}
|
||||
if got["logged_in"] != true {
|
||||
t.Fatalf("账号池里有凭据的账号,logged_in 必须为 true;实得 %#v", got)
|
||||
}
|
||||
if got["account_id"] != "acct-pool-1" {
|
||||
t.Errorf("account_id 应回池里的账号 id;实得 %#v", got["account_id"])
|
||||
}
|
||||
if got["account"] != "pool-nick" {
|
||||
t.Errorf("account 应回池里的昵称;实得 %#v", got["account"])
|
||||
}
|
||||
if s, _ := got["accounts"].(float64); s < 1 {
|
||||
t.Errorf("accounts 计数应 >=1;实得 %#v", got["accounts"])
|
||||
}
|
||||
}
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// TestChannelsPluginEndToEnd 是「同类插件」的完整验收:
|
||||
|
||||
@ -9,7 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ 「为什么没有可用账号」必须区分**真的用尽**与**查不到**。
|
||||
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// qoderReason 调真实的 transform_error 钩子(不自己重算)。
|
||||
|
||||
@ -9,7 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// Qoder 活动领取有三条参考用**抓包 + 用户报障**定位过的「会误报」陷阱,
|
||||
|
||||
@ -8,7 +8,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// qoderHarness:Qoder PKCE 设备码轮的测试台。
|
||||
|
||||
@ -8,7 +8,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// raccoonHarness:Raccoon 内置二维码扫码登录的测试台。
|
||||
|
||||
@ -6,7 +6,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ 删除账号必须**真的把源撤掉** —— 且撤不掉时不能静默。
|
||||
@ -190,8 +190,14 @@ func TestChannelsDeclaresEveryKernelActionItCalls(t *testing.T) {
|
||||
|
||||
// ② 抽出实际调用的动作
|
||||
called := map[string]bool{}
|
||||
for _, m := range regexp.MustCompile(`__llmsproxy_kernel\.call\(\s*"([a-z_]+)"`).FindAllStringSubmatch(s, -1) {
|
||||
called[m[1]] = true
|
||||
// 插件只允许走公开 SDK(modelrouter.kernel);旧名 __llmsproxy_kernel 是
|
||||
// 兼容别名,新代码用它就说明插件又在依赖内核私有全局了。
|
||||
for _, m := range regexp.MustCompile(`\b(?:modelrouter|__llmsproxy_kernel)\.kernel\.call\(\s*"([a-z_]+)"|\b__llmsproxy_kernel\.call\(\s*"([a-z_]+)"`).FindAllStringSubmatch(s, -1) {
|
||||
name := m[1]
|
||||
if name == "" {
|
||||
name = m[2] // 旧名的分支
|
||||
}
|
||||
called[name] = true
|
||||
}
|
||||
if len(called) == 0 {
|
||||
t.Fatal("没扫描到任何内核调用(判据失效)")
|
||||
|
||||
@ -3,7 +3,7 @@ package core
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ account_reorder 必须**真的改变内核里的源顺序**。
|
||||
|
||||
@ -33,8 +33,8 @@ func TestAccountPoolStoreIsAtomicAndConsistent(t *testing.T) {
|
||||
storePath := filepath.Join(dir, ".channels-store.json")
|
||||
t.Logf("插件存储目录: %s", dir)
|
||||
// ⚠️ 必须在**测试目录**里,绝不能是生产路径
|
||||
if _, err := os.Stat("/etc/llmsproxy/plugins/.channels-store.json"); err == nil {
|
||||
if filepath.Clean(dir) == "/etc/llmsproxy/plugins" {
|
||||
if _, err := os.Stat("/etc/modelrouter/plugins/.channels-store.json"); err == nil {
|
||||
if filepath.Clean(dir) == "/etc/modelrouter/plugins" {
|
||||
t.Fatal("测试正在使用生产存储目录 —— 隔离失效(这会造成真实数据事故)")
|
||||
}
|
||||
}
|
||||
|
||||
@ -7,7 +7,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// TestChannelsStoreIsPluginOwned 是「插件自持持久化」的回归。
|
||||
@ -23,7 +23,7 @@ func TestChannelsStoreIsPluginOwned(t *testing.T) {
|
||||
// 固定插件目录:两次「重启」必须共用同一目录,否则测不出持久化。
|
||||
// 这也是防止测试写进生产插件目录的那道隔离(kernelTestCore 会 respect 它)。
|
||||
dir := t.TempDir()
|
||||
t.Setenv("LLMSPROXY_PLUGIN_DIR", dir)
|
||||
t.Setenv("MODELROUTER_PLUGIN_DIR", dir)
|
||||
load := func() *lua.Plugins {
|
||||
c := kernelTestCore(t)
|
||||
ps := c.Plugins()
|
||||
@ -128,6 +128,10 @@ func TestChannelsMigratesLegacyKernelState(t *testing.T) {
|
||||
|
||||
c := kernelTestCore(t)
|
||||
dir = c.Config().PluginDir
|
||||
// 网关不再自动创建 plugin_dir(插件不由内核安装),测试自己建。
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
legacyPath := filepath.Join(dir, ".channels.state.json")
|
||||
if err := os.WriteFile(legacyPath, b, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
@ -179,7 +183,7 @@ func TestChannelsMigrationKeepsLegacyOnWriteFailure(t *testing.T) {
|
||||
// (<dir>/.channels-store.json.tmp)**占成一个目录**,io.open(tmp,"w") 必失败。
|
||||
// 不用 chmod:测试以 root 跑,root 绕过权限位,chmod 拦不住。
|
||||
sub := t.TempDir()
|
||||
t.Setenv("LLMSPROXY_PLUGIN_DIR", sub)
|
||||
t.Setenv("MODELROUTER_PLUGIN_DIR", sub)
|
||||
|
||||
legacy := map[string]interface{}{
|
||||
"version": 1,
|
||||
|
||||
@ -11,8 +11,8 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// mustBox 构造一个真 SecretBox(与运行期同一实现,密钥落 runtime dir)。
|
||||
|
||||
@ -9,7 +9,7 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ TRAE 的 batch 模型目录解析:两个**真实缺陷**必须在移植时一起带上
|
||||
@ -120,23 +120,23 @@ func newTraeCatalogHarness(t *testing.T, onBody func(string)) *traeCatalogHarnes
|
||||
// ① solo_agent:glm-5.2 有档位;另有三个必被硬性过滤的条目
|
||||
`{"function":"solo_agent","config_info_list":[`+
|
||||
`{"config_name":"glm-5.2","usage":"chat_completion","config_switch":true,`+
|
||||
`"reasoning_effort_config":{"support_thinking":true,"options":["high","low"]}},`+
|
||||
`"reasoning_effort_config":{"support_thinking":true,"options":["high","low"]}}`+
|
||||
`]},`+
|
||||
// ② solo_work_lite 排在**后面**:glm-5.2 在这里**没有**档位配置。
|
||||
// 另三个"必被过滤"的条目也放这里 —— 生产里适配器恒发这条通道,
|
||||
// 硬性过滤的判据必须在这条通道上被验证到(放 solo_agent 会因为
|
||||
// admit_functions 策略先行剔除,测试就测不到过滤逻辑了)。
|
||||
`{"function":"solo_work_lite","config_info_list":[`+
|
||||
`{"config_name":"glm-5.2","usage":"chat_completion","config_switch":true},`+
|
||||
`{"config_name":"summary-model","usage":"summary","config_switch":true},`+
|
||||
`{"config_name":"disabled-model","usage":"chat_completion","config_switch":false},`+
|
||||
`{"config_name":"hidden-model","usage":"chat_completion","config_switch":true,"is_invisible_to_user":true},`+
|
||||
`{"config_name":"no-function-model","usage":"chat_completion","config_switch":true},`+
|
||||
// is_custom_model=true ⇒ 上游以流内 4001 拒绝("仅可见但不可调用")
|
||||
`{"config_name":"custom-flag-model","usage":"chat_completion","config_switch":true,`+
|
||||
`"display_config":{"is_custom_model":true}},`+
|
||||
// 对照组:名字带 custom_model_ 前缀但**标志为 false** ⇒ 必须**保留**
|
||||
// (参考点名:判据是标志的值,不是模型名)
|
||||
`{"config_name":"custom_model_prefix_but_ok","usage":"chat_completion","config_switch":true,`+
|
||||
`"display_config":{"is_custom_model":false}}`+
|
||||
`]},`+
|
||||
// ② solo_work_lite 排在**后面**:glm-5.2 在这里**没有**档位配置
|
||||
`{"function":"solo_work_lite","config_info_list":[`+
|
||||
`{"config_name":"glm-5.2","usage":"chat_completion","config_switch":true}`+
|
||||
`]},`+
|
||||
// ③ 白名单外通道:整组必须被丢弃
|
||||
`{"function":"inline_chat","config_info_list":[`+
|
||||
`{"config_name":"only-in-bad-channel","usage":"chat_completion","config_switch":true}`+
|
||||
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ TRAE 的**推理请求**必须用 Cloud-IDE-JWT 头族,不是 Bearer。
|
||||
|
||||
@ -6,7 +6,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// Trae 签到是最容易被"看起来对"骗过的地方:请求能发出去、上游可能也返回 200,
|
||||
|
||||
@ -6,7 +6,7 @@ import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 手工移植的确定性派生必须与参考**逐字节一致**,否则签名的设备身份是错的
|
||||
|
||||
@ -6,7 +6,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ TRAE ExchangeToken 的凭据在 **Result 里**,不在顶层。
|
||||
|
||||
@ -9,7 +9,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ TRAE 回调只带 code(PKCE 形态)时必须**明确报出**「暂不支持 PKCE」,
|
||||
|
||||
@ -10,7 +10,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// traeHarness:TRAE callback 流的测试台。假上游模拟 ExchangeToken;
|
||||
|
||||
@ -5,7 +5,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// TestChannelsVncStackEndToEnd 验证 VNC 兜底链路在**真插件**上跑通:
|
||||
|
||||
@ -6,7 +6,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 显示号泄漏是「测试单独跑绿、全量跑红」的根源,值得一条判据钉住它。
|
||||
@ -34,7 +34,7 @@ func TestXDisplaySelectionReclaimsStaleLocks(t *testing.T) {
|
||||
// (见 kernelTestCore);硬编码 90 段会让本判据造出**根本不会被扫描到**的
|
||||
// 陈旧锁,于是"没复用"这个断言永远失败 —— 那是判据与被测对象脱节。
|
||||
base := 300
|
||||
if v := os.Getenv("LLMSPROXY_XDISPLAY_BASE"); v != "" {
|
||||
if v := os.Getenv("MODELROUTER_XDISPLAY_BASE"); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil && n > 0 {
|
||||
base = n
|
||||
}
|
||||
|
||||
@ -7,7 +7,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// ★ ZCode 的身份块是 3012 准入的**唯一开关**。
|
||||
@ -269,7 +269,7 @@ func TestZcodeAdapterFailsLoudlyWithoutAnthropicBase(t *testing.T) {
|
||||
if err := os.MkdirAll(pluginDir, 0755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("LLMSPROXY_PLUGIN_DIR", pluginDir)
|
||||
t.Setenv("MODELROUTER_PLUGIN_DIR", pluginDir)
|
||||
|
||||
c := kernelTestCore(t)
|
||||
ps := c.Plugins()
|
||||
|
||||
@ -10,7 +10,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// zcodeHarness:ZCode 两段式服务端中介授权流的测试台。
|
||||
|
||||
@ -16,11 +16,10 @@ import (
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/billing"
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/lua"
|
||||
"llmsproxy/internal/provider"
|
||||
"llmsproxy/internal/scheduler"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/lua"
|
||||
"modelrouter/internal/provider"
|
||||
"modelrouter/internal/scheduler"
|
||||
)
|
||||
|
||||
// Core owns the running configuration and adapters. mu guards every read and
|
||||
@ -96,9 +95,6 @@ func NewFromConfig(cfg *config.Config) (*Core, error) {
|
||||
for name, aud := range cfg.PluginVisibility {
|
||||
c.plugins.SetPluginVisibility(name, aud)
|
||||
}
|
||||
if err := c.plugins.SeedBundled(); err != nil {
|
||||
log.Printf("[core] seed bundled plugins: %v", err)
|
||||
}
|
||||
if err := c.plugins.LoadDir(); err != nil {
|
||||
log.Printf("[core] plugin dir: %v", err)
|
||||
}
|
||||
@ -110,7 +106,6 @@ func NewFromConfig(cfg *config.Config) (*Core, error) {
|
||||
log.Printf("[core] plugin %v %v loaded (hooks=%v ui=%v)",
|
||||
p["name"], p["version"], p["hooks"], p["ui"])
|
||||
}
|
||||
c.applyBillingDSL()
|
||||
c.store = config.NewStore(cfg.RuntimeFile)
|
||||
// Share one box between the runtime store and config.yaml so a single
|
||||
// master.key seals both files. config.Load left the config holding
|
||||
@ -1010,63 +1005,6 @@ func (c *Core) Close() {
|
||||
}
|
||||
}
|
||||
|
||||
// applyBillingDSL compiles the configured billing profile and injects it into
|
||||
// the billing plugin.
|
||||
//
|
||||
// Configured pricing replaces the hand-written JSON blob that used to be PUT
|
||||
// through the state API. That path was used twice in production and both times
|
||||
// it "succeeded" while being wrong: once without the required `prices` wrapper
|
||||
// (silently wiping the accumulated totals), once with a peak-window shape the
|
||||
// plugin does not read (peak traffic billed at off-peak rates, no error). A
|
||||
// typed config with load-time validation turns both mistakes into startup
|
||||
// errors naming the file and field.
|
||||
//
|
||||
// A DSL with no profile, or one that fails to compile, leaves the plugin's
|
||||
// prices untouched: billing keeps running on whatever it had, and the problem
|
||||
// is logged rather than becoming a startup failure — the gateway must forward
|
||||
// even when its cost accounting is misconfigured.
|
||||
func (c *Core) applyBillingDSL() {
|
||||
ps := c.plugins
|
||||
if ps == nil || ps.Count() == 0 {
|
||||
return
|
||||
}
|
||||
// The billing plugin may simply not be installed; a DSL without it is a
|
||||
// no-op, not an error.
|
||||
var have bool
|
||||
for _, row := range ps.List() {
|
||||
if row["name"] == "billing" && row["loaded"] == true {
|
||||
have = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !have {
|
||||
return
|
||||
}
|
||||
dsl := c.cfg.BillingDSL
|
||||
if dsl == nil || len(dsl.Profiles) == 0 {
|
||||
return
|
||||
}
|
||||
if err := dsl.Validate(); err != nil {
|
||||
log.Printf("[billing] config invalid, profile not applied: %v", err)
|
||||
return
|
||||
}
|
||||
profile := dsl.Resolve(dsl.Active)
|
||||
if profile == nil {
|
||||
log.Printf("[billing] no billing profile resolved (active=%q)", dsl.Active)
|
||||
return
|
||||
}
|
||||
prices, err := billing.Compile(profile, c.cfg.Sources)
|
||||
if err != nil {
|
||||
log.Printf("[billing] profile %q not applied: %v", profile.ID, err)
|
||||
return
|
||||
}
|
||||
if err := ps.SetState("billing", map[string]interface{}{"prices": prices}); err != nil {
|
||||
log.Printf("[billing] applying profile %q: %v", profile.ID, err)
|
||||
return
|
||||
}
|
||||
log.Printf("[billing] profile %q applied (%d rules, %d sources)", profile.ID, len(profile.Rules), len(c.cfg.Sources))
|
||||
}
|
||||
|
||||
// ValidateScopeQuotas checks every scope entry's caps before they are stored.
|
||||
// A typo in a period must be rejected at write time rather than silently
|
||||
// becoming a never-resetting budget — the opposite of what was typed.
|
||||
|
||||
@ -5,7 +5,7 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// newTestConfig returns an in-memory config backed by temp files so
|
||||
|
||||
@ -8,7 +8,7 @@ import (
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// readBundledChannels 读内置 channels 插件源(供行为/变异测试共用)。
|
||||
|
||||
@ -6,8 +6,8 @@ import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 内核动作实现(R4-c)
|
||||
|
||||
@ -6,8 +6,8 @@ import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// 本文件验证 R4-c/R4-d 的闭环:**插件自己驱动外部源的链接认证,然后把源接进内核**。
|
||||
@ -22,12 +22,12 @@ func kernelTestCore(t *testing.T) *Core {
|
||||
dir := t.TempDir()
|
||||
pluginDir := filepath.Join(dir, "plugins")
|
||||
// 插件的自有存储位置由插件自己决定(读 config.yaml 的 plugin_dir,或
|
||||
// LLMSPROXY_PLUGIN_DIR 覆盖)。测试**必须**指向临时目录 —— 否则插件会读到
|
||||
// /etc/llmsproxy/config.yaml 并写进**生产插件目录**(真实事故:一次全量测试
|
||||
// MODELROUTER_PLUGIN_DIR 覆盖)。测试**必须**指向临时目录 —— 否则插件会读到
|
||||
// /etc/modelrouter/config.yaml 并写进**生产插件目录**(真实事故:一次全量测试
|
||||
// 把 26 个测试账号写进了生产 .channels-store.json)。
|
||||
// 调用方已显式设定时尊重它:多次「重启」要共用同一目录才能测持久化。
|
||||
if os.Getenv("LLMSPROXY_PLUGIN_DIR") == "" {
|
||||
t.Setenv("LLMSPROXY_PLUGIN_DIR", pluginDir)
|
||||
if os.Getenv("MODELROUTER_PLUGIN_DIR") == "" {
|
||||
t.Setenv("MODELROUTER_PLUGIN_DIR", pluginDir)
|
||||
}
|
||||
// ★ 显示器号与端口用**高段号**,避开生产。
|
||||
//
|
||||
@ -36,8 +36,8 @@ func kernelTestCore(t *testing.T) *Core {
|
||||
// 并去抢,结果 `Xvfb 未就绪(/tmp/.X11-unix/X91 未出现)`,而生产上正跑着
|
||||
// 一套浏览器栈。判据本身没错,错的是**两个 /tmp 互不可见**而号段重叠。
|
||||
// 故测试整体挪到 300+。
|
||||
if os.Getenv("LLMSPROXY_XDISPLAY_BASE") == "" {
|
||||
t.Setenv("LLMSPROXY_XDISPLAY_BASE", "300")
|
||||
if os.Getenv("MODELROUTER_XDISPLAY_BASE") == "" {
|
||||
t.Setenv("MODELROUTER_XDISPLAY_BASE", "300")
|
||||
}
|
||||
cfg := &config.Config{
|
||||
Path: filepath.Join(dir, "config.yaml"),
|
||||
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/lua"
|
||||
"modelrouter/internal/lua"
|
||||
)
|
||||
|
||||
// TestNoKernelActionDeadlocksFromLua 对**每个**内核动作做同一条断言:
|
||||
|
||||
@ -7,7 +7,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// AutoChainFor sits on the request hot path and reads keyAutoChains with no
|
||||
|
||||
@ -4,8 +4,8 @@ import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/scheduler"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/scheduler"
|
||||
)
|
||||
|
||||
// Per-key AUTO chains let an admin give one user their own scheduling chain
|
||||
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"net/url"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// The plugin outbound allowlist is built from the hosts of configured sources.
|
||||
|
||||
@ -6,7 +6,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// Startup must be idempotent, and every credential consumer must see PLAINTEXT.
|
||||
|
||||
@ -11,7 +11,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
type adapterPayload struct {
|
||||
@ -488,7 +488,7 @@ func (g *Gateway) handleStatsAPI(w http.ResponseWriter, r *http.Request) {
|
||||
if to == 0 {
|
||||
to = time.Now().UnixMilli()
|
||||
}
|
||||
csvHeaders(w, "llmsproxy-requests.csv")
|
||||
csvHeaders(w, "modelrouter-requests.csv")
|
||||
cw := csv.NewWriter(w)
|
||||
names := map[string]string{}
|
||||
for _, k := range g.core.ListKeys() {
|
||||
@ -537,7 +537,7 @@ func (g *Gateway) handleStatsAPI(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
if r.URL.Query().Get("export") == "keys-csv" {
|
||||
csvHeaders(w, "llmsproxy-keys.csv")
|
||||
csvHeaders(w, "modelrouter-keys.csv")
|
||||
cw := csv.NewWriter(w)
|
||||
_ = cw.Write([]string{"key", "key_name", "role", "models", "total_requests", "success_requests", "failed_requests", "prompt_tokens", "completion_tokens", "total_tokens", "avg_latency_ms", "max_latency_ms", "created_at"})
|
||||
keyFilter := exportKey(r)
|
||||
|
||||
@ -31,7 +31,7 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// apiV1Routes dispatches /api/v1/*. Kept separate from routes() so the original
|
||||
@ -144,7 +144,7 @@ func (g *Gateway) apiV1Index(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
index := map[string]interface{}{
|
||||
"api_version": "v1",
|
||||
"description": "llmsproxy management API. Authenticate with a gateway key: " +
|
||||
"description": "modelrouter management API. Authenticate with a gateway key: " +
|
||||
"'Authorization: Bearer <key>', '?api_key=<key>', or the gw_key cookie. " +
|
||||
"Read endpoints accept any role; mutations require an admin key.",
|
||||
"discovery": []ep{
|
||||
|
||||
@ -7,7 +7,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// v1Gateway builds a gateway with one source and two keys, and returns the
|
||||
|
||||
@ -1,282 +0,0 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
|
||||
"llmsproxy/internal/billing"
|
||||
"llmsproxy/internal/config"
|
||||
)
|
||||
|
||||
// Billing rules API.
|
||||
//
|
||||
// GET /api/plugins/billing/rules the active profile's rules + all profiles
|
||||
// PUT /api/plugins/billing/rules replace the whole DSL and re-inject prices
|
||||
// POST /api/plugins/billing/rules { profile, rule } append one rule
|
||||
// DELETE /api/plugins/billing/rules { profile, url } remove one rule
|
||||
//
|
||||
// Why a separate endpoint instead of the generic plugin state: prices are
|
||||
// plugin state, but they are also a durable, reviewable CONFIGURATION. Routing
|
||||
// them through /state would let an admin key PUT arbitrary plugin state and
|
||||
// silently reset the accumulated accounting. Here the server owns the shape:
|
||||
// it validates the DSL, writes it back to config.yaml, recompiles, and hands
|
||||
// the plugin its prices table. The operator edits rules; the plugin keeps
|
||||
// numbers. The two are never mixed in one payload.
|
||||
//
|
||||
// Editing means "what the operator typed is what config.yaml holds". A
|
||||
// billing mistake found next week must be traceable to a reviewable file, not
|
||||
// to a blob in the plugin's state sidecar.
|
||||
func (g *Gateway) handleBillingRules(w http.ResponseWriter, r *http.Request) {
|
||||
if reqRole(r.Context()) != "admin" {
|
||||
writeError(w, http.StatusForbidden, "forbidden", "admin role required")
|
||||
return
|
||||
}
|
||||
switch r.Method {
|
||||
case http.MethodGet:
|
||||
g.getBillingRules(w)
|
||||
case http.MethodPut:
|
||||
g.putBillingRules(w, r)
|
||||
case http.MethodPost:
|
||||
g.addBillingRule(w, r)
|
||||
case http.MethodDelete:
|
||||
g.delBillingRule(w, r)
|
||||
default:
|
||||
writeError(w, http.StatusMethodNotAllowed, "method_not_allowed", "use GET/PUT/POST/DELETE")
|
||||
}
|
||||
}
|
||||
|
||||
// billingRulesPayload is the UI-facing view: the profiles as declared, plus
|
||||
// which one is active and the source URLs they can match — the editor needs
|
||||
// the URL list to offer suggestions, and an operator typing a URL that matches
|
||||
// no source is exactly the silent-no-pricing failure this feature exists to
|
||||
// remove.
|
||||
type billingRulesPayload struct {
|
||||
DSL *config.BillingDSL `json:"billing"`
|
||||
Active string `json:"active"`
|
||||
URLs []string `json:"urls"`
|
||||
Warnings []string `json:"warnings"`
|
||||
}
|
||||
|
||||
func (g *Gateway) billingRulesPayload() billingRulesPayload {
|
||||
out := billingRulesPayload{}
|
||||
cfg := g.core.Config()
|
||||
if cfg != nil && cfg.BillingDSL != nil {
|
||||
// Copy so the response cannot be mutated back through the pointer.
|
||||
cp := *cfg.BillingDSL
|
||||
out.DSL = &cp
|
||||
if p := cp.Resolve(""); p != nil {
|
||||
out.Active = p.ID
|
||||
}
|
||||
}
|
||||
for _, s := range g.sourceURLs() {
|
||||
out.URLs = append(out.URLs, s)
|
||||
}
|
||||
out.Warnings = g.billingRuleWarnings()
|
||||
return out
|
||||
}
|
||||
|
||||
func (g *Gateway) sourceURLs() []string {
|
||||
cfg := g.core.Config()
|
||||
if cfg == nil {
|
||||
return nil
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
for _, s := range cfg.Sources {
|
||||
if s.BaseURL == "" || seen[s.BaseURL] {
|
||||
continue
|
||||
}
|
||||
seen[s.BaseURL] = true
|
||||
out = append(out, s.BaseURL)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// billingRuleWarnings reports rules that match no configured source. This is
|
||||
// the silent killer of hand-written price tables: the rule parses, validates,
|
||||
// and prices nothing at all, so every request on that URL lands in unpriced.
|
||||
// Saying so out loud is the difference between a five-second fix and an
|
||||
// afternoon wondering why the bill is zero.
|
||||
func (g *Gateway) billingRuleWarnings() []string {
|
||||
cfg := g.core.Config()
|
||||
if cfg == nil || cfg.BillingDSL == nil {
|
||||
return nil
|
||||
}
|
||||
urls := map[string]bool{}
|
||||
for _, u := range g.sourceURLs() {
|
||||
urls[u] = true
|
||||
}
|
||||
var warns []string
|
||||
for _, p := range cfg.BillingDSL.Profiles {
|
||||
for _, r := range p.Rules {
|
||||
if r.URL == "*" || urls[r.URL] {
|
||||
continue
|
||||
}
|
||||
warns = append(warns, "profile "+p.ID+": rule url "+r.URL+" matches no configured source")
|
||||
}
|
||||
}
|
||||
return warns
|
||||
}
|
||||
|
||||
func (g *Gateway) getBillingRules(w http.ResponseWriter) {
|
||||
writeJSON(w, http.StatusOK, g.billingRulesPayload())
|
||||
}
|
||||
|
||||
func (g *Gateway) putBillingRules(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Billing *config.BillingDSL `json:"billing"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "invalid json: "+err.Error())
|
||||
return
|
||||
}
|
||||
if body.Billing == nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "billing is required")
|
||||
return
|
||||
}
|
||||
g.applyBillingDSLUpdate(w, body.Billing)
|
||||
}
|
||||
|
||||
func (g *Gateway) addBillingRule(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Profile string `json:"profile"`
|
||||
Rule config.BillingRule `json:"rule"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "invalid json: "+err.Error())
|
||||
return
|
||||
}
|
||||
cfg := g.core.Config()
|
||||
if cfg == nil || cfg.BillingDSL == nil || len(cfg.BillingDSL.Profiles) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "no_profile", "no billing profiles configured; create one first")
|
||||
return
|
||||
}
|
||||
next := cloneBillingDSL(cfg.BillingDSL)
|
||||
idx := profileIndex(next, body.Profile)
|
||||
if idx < 0 {
|
||||
writeError(w, http.StatusBadRequest, "unknown_profile", "no such profile: "+body.Profile)
|
||||
return
|
||||
}
|
||||
p := &next.Profiles[idx]
|
||||
// Replace rather than append when the URL is already declared: two rules
|
||||
// for one URL would silently make the first unreachable (first match wins),
|
||||
// which is the exact ambiguity an editor should not be able to create.
|
||||
replaced := false
|
||||
for i := range p.Rules {
|
||||
if p.Rules[i].URL == body.Rule.URL {
|
||||
p.Rules[i] = body.Rule
|
||||
replaced = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !replaced {
|
||||
p.Rules = append(p.Rules, body.Rule)
|
||||
}
|
||||
g.applyBillingDSLUpdate(w, next)
|
||||
}
|
||||
|
||||
func (g *Gateway) delBillingRule(w http.ResponseWriter, r *http.Request) {
|
||||
var body struct {
|
||||
Profile string `json:"profile"`
|
||||
URL string `json:"url"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "invalid json: "+err.Error())
|
||||
return
|
||||
}
|
||||
cfg := g.core.Config()
|
||||
if cfg == nil || cfg.BillingDSL == nil {
|
||||
writeError(w, http.StatusBadRequest, "no_profile", "no billing profiles configured")
|
||||
return
|
||||
}
|
||||
next := cloneBillingDSL(cfg.BillingDSL)
|
||||
idx := profileIndex(next, body.Profile)
|
||||
if idx < 0 {
|
||||
writeError(w, http.StatusBadRequest, "unknown_profile", "no such profile: "+body.Profile)
|
||||
return
|
||||
}
|
||||
p := &next.Profiles[idx]
|
||||
out := p.Rules[:0]
|
||||
found := false
|
||||
for _, rule := range p.Rules {
|
||||
if rule.URL == body.URL {
|
||||
found = true
|
||||
continue
|
||||
}
|
||||
out = append(out, rule)
|
||||
}
|
||||
if !found {
|
||||
writeError(w, http.StatusNotFound, "not_found", "no rule for url "+body.URL)
|
||||
return
|
||||
}
|
||||
p.Rules = out
|
||||
g.applyBillingDSLUpdate(w, next)
|
||||
}
|
||||
|
||||
// applyBillingDSLUpdate is the single write path: validate, persist to
|
||||
// config.yaml, recompile, inject, and only then report success. If the config
|
||||
// cannot be written the change is NOT applied in memory either — a rules editor
|
||||
// that says "saved" and loses the edit on the next restart is worse than one
|
||||
// that refuses.
|
||||
func (g *Gateway) applyBillingDSLUpdate(w http.ResponseWriter, next *config.BillingDSL) {
|
||||
if err := next.Validate(); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_rules", err.Error())
|
||||
return
|
||||
}
|
||||
cfg := g.core.Config()
|
||||
if cfg == nil {
|
||||
writeError(w, http.StatusInternalServerError, "no_config", "no config loaded")
|
||||
return
|
||||
}
|
||||
// Assign BEFORE saving. The previous order saved first and assigned after,
|
||||
// so every rule edit reported success while writing a config.yaml with no
|
||||
// billing section at all — the operator's edit vanished on restart and
|
||||
// nothing in the API response said so.
|
||||
prev := cfg.BillingDSL
|
||||
cfg.BillingDSL = next
|
||||
|
||||
// Compile before persisting: a profile that cannot be turned into prices
|
||||
// must not reach the config file, or the next restart fails to load the
|
||||
// very rules the editor just accepted.
|
||||
prices, err := billing.CompileOpts(next.Resolve(next.Active), cfg.Sources, true)
|
||||
if err != nil {
|
||||
cfg.BillingDSL = prev // no half-applied state
|
||||
writeError(w, http.StatusBadRequest, "compile_failed", err.Error())
|
||||
return
|
||||
}
|
||||
if err := cfg.Save(); err != nil {
|
||||
cfg.BillingDSL = prev
|
||||
writeError(w, http.StatusInternalServerError, "persist_failed", err.Error())
|
||||
return
|
||||
}
|
||||
ps := g.core.Plugins()
|
||||
if ps != nil {
|
||||
if err := ps.SetState("billing", map[string]interface{}{"prices": prices}); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "plugin_error", err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
p := g.billingRulesPayload()
|
||||
writeJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"ok": true, "billing": p.DSL, "active": p.Active, "warnings": p.Warnings,
|
||||
})
|
||||
}
|
||||
|
||||
func cloneBillingDSL(d *config.BillingDSL) *config.BillingDSL {
|
||||
out := &config.BillingDSL{Active: d.Active}
|
||||
out.Profiles = make([]config.BillingProfile, len(d.Profiles))
|
||||
copy(out.Profiles, d.Profiles)
|
||||
return out
|
||||
}
|
||||
|
||||
func profileIndex(d *config.BillingDSL, id string) int {
|
||||
if id == "" {
|
||||
return -1
|
||||
}
|
||||
for i, p := range d.Profiles {
|
||||
if p.ID == id {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
@ -1,281 +0,0 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
)
|
||||
|
||||
// rulesGateway needs a real billing plugin (the API injects prices into it)
|
||||
// AND two sources whose base_urls the rules will match, so it cannot reuse
|
||||
// either existing helper: gatewayWithBilling has no sources, and
|
||||
// newTestGateway loads no plugins.
|
||||
func rulesGateway(t *testing.T) *Gateway {
|
||||
t.Helper()
|
||||
g := gatewayWithBilling(t)
|
||||
cfg := g.core.Config()
|
||||
cfg.Sources = []config.Source{
|
||||
{Name: "localzen", BaseURL: "https://free.example.com/v1"},
|
||||
{Name: "commandcode", BaseURL: "https://api.commandcode.ai/v1"},
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
func rulesDo(t *testing.T, g *Gateway, method, body string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
return doReq(t, g, method, "/api/plugins/billing/rules", body)
|
||||
}
|
||||
|
||||
// TestBillingRulesCRUD covers the whole editor loop against the real HTTP
|
||||
// surface: read, add, replace-by-URL, delete. The replace-by-URL case is the
|
||||
// one worth pinning — two rules for one URL would make the first unreachable
|
||||
// (first match wins) without any error, which is exactly the kind of silent
|
||||
// ambiguity an editor must not be able to create.
|
||||
func TestBillingRulesCRUD(t *testing.T) {
|
||||
g := rulesGateway(t)
|
||||
seed := `{"billing":{"active":"p1","profiles":[{"id":"p1","rules":[
|
||||
{"url":"https://free.example.com/v1","mode":"free"}]}]}}`
|
||||
if rr := rulesDo(t, g, http.MethodPut, seed); rr.Code != http.StatusOK {
|
||||
t.Fatalf("seed rules = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
// Add a token-priced rule for the second URL.
|
||||
add := `{"profile":"p1","rule":{"url":"https://api.commandcode.ai/v1","mode":"token",
|
||||
"models":{"deepseek-v4.1-flash":{"prompt":"0.15","completion":"0.60"}}}}`
|
||||
if rr := rulesDo(t, g, http.MethodPost, add); rr.Code != http.StatusOK {
|
||||
t.Fatalf("add rule = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var got billingRulesPayload
|
||||
if err := json.Unmarshal(rulesDo(t, g, http.MethodGet, "").Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.DSL.Profiles[0].Rules) != 2 {
|
||||
t.Fatalf("after add there are %d rules, want 2", len(got.DSL.Profiles[0].Rules))
|
||||
}
|
||||
|
||||
// Adding the SAME url again must replace, not append.
|
||||
if rr := rulesDo(t, g, http.MethodPost, `{"profile":"p1","rule":{"url":"https://api.commandcode.ai/v1","mode":"free"}}`); rr.Code != http.StatusOK {
|
||||
t.Fatalf("re-add same url = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if err := json.Unmarshal(rulesDo(t, g, http.MethodGet, "").Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := len(got.DSL.Profiles[0].Rules); n != 2 {
|
||||
t.Fatalf("re-adding an existing url appended instead of replacing: %d rules", n)
|
||||
}
|
||||
var mode string
|
||||
for _, r := range got.DSL.Profiles[0].Rules {
|
||||
if r.URL == "https://api.commandcode.ai/v1" {
|
||||
mode = r.Mode
|
||||
}
|
||||
}
|
||||
if mode != "free" {
|
||||
t.Errorf("re-added rule mode = %q, want free (the replacement must win)", mode)
|
||||
}
|
||||
|
||||
// Delete by URL.
|
||||
if rr := rulesDo(t, g, http.MethodDelete, `{"profile":"p1","url":"https://api.commandcode.ai/v1"}`); rr.Code != http.StatusOK {
|
||||
t.Fatalf("delete rule = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if err := json.Unmarshal(rulesDo(t, g, http.MethodGet, "").Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.DSL.Profiles[0].Rules) != 1 {
|
||||
t.Errorf("after delete there are %d rules, want 1", len(got.DSL.Profiles[0].Rules))
|
||||
}
|
||||
}
|
||||
|
||||
// TestBillingRulesRejectUnknownURLWithWarning is the silent-failure guard: a
|
||||
// rule whose URL matches no source prices nothing, so every request on it
|
||||
// falls into unpriced. The API must say so instead of accepting it silently.
|
||||
func TestBillingRulesRejectUnknownURLWithWarning(t *testing.T) {
|
||||
g := rulesGateway(t)
|
||||
if rr := rulesDo(t, g, http.MethodPut, `{"billing":{"active":"p","profiles":[{"id":"p",
|
||||
"rules":[{"url":"https://typo.example.com/v1","mode":"free"}]}]}}`); rr.Code != http.StatusOK {
|
||||
t.Fatalf("seed = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var got billingRulesPayload
|
||||
if err := json.Unmarshal(rulesDo(t, g, http.MethodGet, "").Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Warnings) == 0 {
|
||||
t.Fatal("a rule matching no configured source was accepted with no warning — " +
|
||||
"it will price nothing and every request will be recorded as unpriced")
|
||||
}
|
||||
if !strings.Contains(got.Warnings[0], "typo.example.com") {
|
||||
t.Errorf("warning does not name the offending url: %v", got.Warnings)
|
||||
}
|
||||
// The editor needs the real URL list to offer suggestions.
|
||||
if len(got.URLs) != 2 {
|
||||
t.Errorf("urls offered to the editor = %v, want the 2 configured base_urls", got.URLs)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBillingRulesInvalidIsRejectedNotPersisted checks the write path refuses
|
||||
// bad input and leaves the previous rules in place — an editor that clears the
|
||||
// table on a typo loses the price table.
|
||||
func TestBillingRulesInvalidIsRejectedNotPersisted(t *testing.T) {
|
||||
g := rulesGateway(t)
|
||||
seed := `{"billing":{"active":"p","profiles":[{"id":"p","rules":[
|
||||
{"url":"https://free.example.com/v1","mode":"free"}]}]}}`
|
||||
if rr := rulesDo(t, g, http.MethodPut, seed); rr.Code != http.StatusOK {
|
||||
t.Fatalf("seed = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
// An unknown mode must be refused.
|
||||
if rr := rulesDo(t, g, http.MethodPost, `{"profile":"p","rule":{"url":"https://x/v1","mode":"banana"}}`); rr.Code != http.StatusBadRequest {
|
||||
t.Errorf("invalid mode accepted with %d, want 400", rr.Code)
|
||||
}
|
||||
var got billingRulesPayload
|
||||
if err := json.Unmarshal(rulesDo(t, g, http.MethodGet, "").Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.DSL.Profiles[0].Rules) != 1 {
|
||||
t.Errorf("a rejected rule was persisted anyway: %d rules", len(got.DSL.Profiles[0].Rules))
|
||||
}
|
||||
// An unparseable price must be refused too, not silently priced at zero.
|
||||
if rr := rulesDo(t, g, http.MethodPost, `{"profile":"p","rule":{"url":"https://y/v1","mode":"token",
|
||||
"models":{"m":{"prompt":"free","completion":"0.6"}}}}`); rr.Code != http.StatusBadRequest {
|
||||
t.Errorf("non-numeric price accepted with %d, want 400", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBillingRulesPersistAndReachThePlugin is the point of the whole
|
||||
// endpoint, and mutation verification showed the earlier CRUD tests missed it
|
||||
// twice: dropping cfg.Save() and dropping the SetState injection both left
|
||||
// every test green. Both failures are invisible in a GET — the response echoes
|
||||
// the in-memory rules either way — so this test checks the two consequences
|
||||
// that actually matter:
|
||||
//
|
||||
// 1. persistence: the rule must be in the config file on disk, because an
|
||||
// edit that evaporates on restart is a lie the UI told the operator;
|
||||
// 2. effect: the billing plugin must now PRICE the source (its published
|
||||
// prices table has a non-zero entry), because a rule that is stored but
|
||||
// never injected prices nothing at all.
|
||||
func TestBillingRulesPersistAndReachThePlugin(t *testing.T) {
|
||||
g := rulesGateway(t)
|
||||
cfg := g.core.Config()
|
||||
|
||||
seed := `{"billing":{"active":"p","profiles":[{"id":"p","rules":[
|
||||
{"url":"https://free.example.com/v1","mode":"free"},
|
||||
{"url":"https://api.commandcode.ai/v1","mode":"token",
|
||||
"models":{"deepseek-v4.1-flash":{"prompt":"0.15","completion":"0.60"}}}]}]}}`
|
||||
if rr := rulesDo(t, g, http.MethodPut, seed); rr.Code != http.StatusOK {
|
||||
t.Fatalf("seed = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
// 1. persisted to the file the process loaded from.
|
||||
raw, err := os.ReadFile(cfg.Path)
|
||||
if err != nil {
|
||||
t.Fatalf("read config: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(raw), "commandcode.ai") {
|
||||
t.Errorf("the rule is not in config.yaml on disk — an edit that does not "+
|
||||
"survive a restart:\n%s", raw)
|
||||
}
|
||||
reloaded, err := config.Load(cfg.Path)
|
||||
if err != nil {
|
||||
t.Fatalf("reload saved config: %v", err)
|
||||
}
|
||||
if reloaded.BillingDSL == nil || len(reloaded.BillingDSL.Profiles) == 0 {
|
||||
t.Fatalf("saved config has no billing DSL: %s", raw)
|
||||
}
|
||||
// A round-trip through YAML must not lose the price strings.
|
||||
found := false
|
||||
for _, r := range reloaded.BillingDSL.Profiles[0].Rules {
|
||||
if strings.Contains(r.URL, "commandcode") {
|
||||
found = true
|
||||
if m, ok := r.Models["deepseek-v4.1-flash"]; !ok || m.Prompt != "0.15" {
|
||||
t.Errorf("price did not survive the YAML round-trip: %+v", m)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("the token rule is missing after reload:\n%s", raw)
|
||||
}
|
||||
|
||||
// 2. reached the plugin. Prices live in a separate Lua field from `state`,
|
||||
// and the GET /state endpoint returns the published state together with
|
||||
// the prices sidecar — which is exactly what the plugin's own UI reads.
|
||||
state := g.core.Plugins().State("billing")
|
||||
if state == nil {
|
||||
t.Fatalf("billing plugin published no state")
|
||||
}
|
||||
rr := doReq(t, g, http.MethodGet, "/api/plugins/billing/state", "")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("GET state = %d: %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var envelope struct {
|
||||
Prices map[string]interface{} `json:"prices"`
|
||||
}
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &envelope); err != nil {
|
||||
t.Fatalf("decode state envelope: %v", err)
|
||||
}
|
||||
prices := envelope.Prices
|
||||
if prices == nil {
|
||||
t.Fatalf("state payload carries no prices")
|
||||
}
|
||||
sources, ok := prices["sources"].(map[string]interface{})
|
||||
if !ok {
|
||||
t.Fatalf("prices has no sources map: %#v", prices)
|
||||
}
|
||||
cc, ok := sources["commandcode"]
|
||||
if !ok {
|
||||
t.Fatalf("the rule was saved but never injected: prices has %v, "+
|
||||
"no commandcode entry. Every request on it stays unpriced.", keysOf(sources))
|
||||
}
|
||||
if cc == nil {
|
||||
t.Error("commandcode price entry is nil")
|
||||
}
|
||||
// The free rule must price localzen, and the two must not be conflated.
|
||||
if _, ok := sources["localzen"]; !ok {
|
||||
t.Errorf("the free rule did not reach the plugin: %v", keysOf(sources))
|
||||
}
|
||||
// The injected price must be per-token, i.e. 0.15 USD/M => 1.5e-7.
|
||||
// Asserting the exact magnitude catches a unit error (per-million vs
|
||||
// per-token), which would be off by a factor of a million and still look
|
||||
// like "a number".
|
||||
models, ok := cc.(map[string]interface{})["models"].(map[string]interface{})
|
||||
if ok {
|
||||
if m, ok := models["deepseek-v4.1-flash"].(map[string]interface{}); ok {
|
||||
p, _ := m["prompt"].(float64)
|
||||
if p <= 0 || p > 1e-6 {
|
||||
t.Errorf("injected prompt price = %v, want per-token (~1.5e-7); "+
|
||||
"a per-million value here would over-bill by 1e6x", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func keysOf(m map[string]interface{}) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// TestBillingRulesRequireAdmin keeps price rewriting behind the admin role,
|
||||
// like every other plugin write: prices are configuration, not user data.
|
||||
func TestBillingRulesRequireAdmin(t *testing.T) {
|
||||
g := rulesGateway(t)
|
||||
rec, err := g.core.CreateKey("viewer", "user", nil, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
userKey := rec.Key
|
||||
for _, m := range []string{http.MethodPut, http.MethodPost, http.MethodDelete} {
|
||||
req, _ := http.NewRequest(m, "/api/plugins/billing/rules", strings.NewReader(`{}`))
|
||||
req.Header.Set("Authorization", "Bearer "+userKey)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if rr := newRecorderFor(t, g, req); rr.Code != http.StatusForbidden {
|
||||
t.Errorf("user %s rules = %d, want 403", m, rr.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -13,11 +13,11 @@ import (
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/lua"
|
||||
"llmsproxy/internal/provider"
|
||||
"llmsproxy/internal/scheduler"
|
||||
"llmsproxy/internal/types"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/lua"
|
||||
"modelrouter/internal/provider"
|
||||
"modelrouter/internal/scheduler"
|
||||
"modelrouter/internal/types"
|
||||
)
|
||||
|
||||
// chatRequest mirrors the OpenAI chat completions request the gateway accepts.
|
||||
|
||||
@ -3,7 +3,7 @@ package gateway
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/types"
|
||||
"modelrouter/internal/types"
|
||||
)
|
||||
|
||||
// TestMergeUsageKeepsZeroCacheDetails locks the streaming counterpart of the
|
||||
|
||||
@ -3,7 +3,7 @@ package gateway
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/types"
|
||||
"modelrouter/internal/types"
|
||||
)
|
||||
|
||||
// TestRecordChatUsageKeepsReasoningTokens pins the reasoning-token plumbing.
|
||||
|
||||
@ -11,7 +11,7 @@ import (
|
||||
|
||||
// deploy.sh is the only release-critical script with no automated coverage, and
|
||||
// its newest part — prune_adapter_backups — deletes directories under
|
||||
// /etc/llmsproxy. This exercises the function in a throwaway directory with the
|
||||
// /etc/modelrouter. This exercises the function in a throwaway directory with the
|
||||
// REAL backup-name shape.
|
||||
//
|
||||
// The shape matters and got this wrong twice: the guard is
|
||||
@ -45,7 +45,7 @@ func TestDeployPruneAdapterBackups(t *testing.T) {
|
||||
|
||||
// Redirect ONLY the base path. The logic under test must stay verbatim.
|
||||
body = strings.Replace(body,
|
||||
`local base="/etc/llmsproxy"`,
|
||||
`local base="/etc/modelrouter"`,
|
||||
`local base="`+base+`"`, 1)
|
||||
// KEEP_BACKUPS must come from deploy.sh itself, NOT from a literal
|
||||
// here: hardcoding 5 in the shim overrode whatever the script configured, so
|
||||
@ -207,7 +207,7 @@ func TestDeployBacksUpTheLiveFileBeforeOverwriting(t *testing.T) {
|
||||
//
|
||||
// 事实:backup 清理最初只处理 `adapters.bak.*`,而 `sync_plugins` 同样每次部署
|
||||
// 建一个 `plugins.bak.<时间戳>` —— 于是 adapters 的堆积被修好,plugins 继续涨。
|
||||
// 实测线上 plugins.bak 积累到 **36 个**、/etc/llmsproxy 涨到 192M。
|
||||
// 实测线上 plugins.bak 积累到 **36 个**、/etc/modelrouter 涨到 192M。
|
||||
//
|
||||
// 这是「两处只修一处」的老形状。判据钉住:prune_all_backups 必须同时裁两个前缀,
|
||||
// 且新增任何 .bak.<时间戳> 前缀时都要在这里补上(否则又是一个泄漏点)。
|
||||
@ -264,7 +264,7 @@ func TestDeployPruneCoversBothBackupPrefixes(t *testing.T) {
|
||||
k := strings.Index(src, "prune_backups()")
|
||||
kj := strings.Index(src[k:], "\n}\n")
|
||||
fnBody := src[k : k+kj+3]
|
||||
fnBody = strings.Replace(fnBody, `local base="/etc/llmsproxy"`, `local base="`+base+`"`, 1)
|
||||
fnBody = strings.Replace(fnBody, `local base="/etc/modelrouter"`, `local base="`+base+`"`, 1)
|
||||
keepRe := regexp.MustCompile(`(?m)^KEEP_BACKUPS=(\d+)`)
|
||||
km := keepRe.FindStringSubmatch(src)
|
||||
if km == nil {
|
||||
|
||||
@ -11,7 +11,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// Model discovery: turn "here is a URL and a key" into a configured source.
|
||||
|
||||
@ -5,7 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@ -9,7 +9,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// upstream that answers /v1/models with the real captured catalog
|
||||
|
||||
@ -126,7 +126,12 @@ func TestRealMutants(t *testing.T) {
|
||||
},
|
||||
{
|
||||
name: "nested metadata no longer read",
|
||||
from: "\tcopyFields(m.Metadata) // nested first",
|
||||
// 锚点必须与 discover.go 逐字节一致:此前这里是 6 个空格,源码已
|
||||
// 变成 2 个空格,于是这条变异永远"找不到锚点"——变异campaign 每次
|
||||
// 都以 anchor not found 失败,而它本该验证"嵌套 metadata 被丢掉"能被
|
||||
// 测试抓住。锚点漂移 = 这条变异从未真正跑过(和 issue #1 里
|
||||
// `cc["models"]` 恒 false 的测试跳过是同一种失效:断言写了但没执行)。
|
||||
from: "\tcopyFields(m.Metadata) // nested first",
|
||||
to: "\tcopyFields(json.RawMessage(nil)) // mutant: nested metadata dropped",
|
||||
},
|
||||
{
|
||||
|
||||
@ -14,8 +14,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/core"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/core"
|
||||
)
|
||||
|
||||
func mockUpstream() *httptest.Server {
|
||||
|
||||
@ -9,8 +9,8 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/core"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/core"
|
||||
)
|
||||
|
||||
// Per-key AUTO chain admin API. The behaviours pinned here are the ones that
|
||||
|
||||
@ -9,8 +9,8 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/core"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/core"
|
||||
)
|
||||
|
||||
// adminGateway builds a gateway whose admin key can call /api/keys.
|
||||
|
||||
@ -4,7 +4,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"modelrouter/internal/config"
|
||||
)
|
||||
|
||||
// ---- data layer: per-key isolation ----
|
||||
|
||||
@ -12,8 +12,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/core"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/core"
|
||||
)
|
||||
|
||||
// quotaGateway builds a gateway with two user keys that differ only in their
|
||||
|
||||
@ -6,8 +6,8 @@ import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"llmsproxy/internal/config"
|
||||
"llmsproxy/internal/scheduler"
|
||||
"modelrouter/internal/config"
|
||||
"modelrouter/internal/scheduler"
|
||||
)
|
||||
|
||||
// handleKeysAPI manages gateway keys: GET /api/keys (admin: all keys),
|
||||
|
||||
@ -10,9 +10,9 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"llmsproxy/internal/provider"
|
||||
"llmsproxy/internal/scheduler"
|
||||
"llmsproxy/internal/types"
|
||||
"modelrouter/internal/provider"
|
||||
"modelrouter/internal/scheduler"
|
||||
"modelrouter/internal/types"
|
||||
)
|
||||
|
||||
// handleMessages serves the Anthropic Messages protocol (POST /v1/messages).
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user