diff --git a/README.md b/README.md index e231ba8..d9c5178 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,13 @@ # ModelRouter +> **English**: [README_EN.md](./README_EN.md) + 面向多 llms 订阅者,部署在内网,实现一次配置多个服务共同使用的效果。支持指定模型或 auto 模式,按照配置的优先级选择可用模型提供服务。 > **轻量且增源无需重编译**:网关本体是单 Go 二进制(约 10MB,零运行时依赖)。新增/切换上游 > 只需在 `config.yaml`(或 WebUI)加一个 `sources` 条目或挂一个 `.lua` 适配器——**不改 Go、 -> 不重编译、不重启**。适配器 Lua 脚本热加载,协议差异全部隔离在 Lua 层,Go 只负责调度与透传。 +> 不重编译**。源与适配器的改动经 WebUI 提交时即时生效(热更新);直接编辑 +> `config.yaml` 或 `adapter_dir` 下的 `.lua` 文件则需要重启进程生效。 ## 实现 @@ -106,6 +109,31 @@ sources: 任何 OpenAI 客户端,只要 `model` 设为某个源的 `name/任意名`,即可锁定走该源; 设为 `AUTO`(或网关配了 `default_model: AUTO`)即自动按优先级选源。 +### 源(Source)与适配器(Adapter)的关系 + +- **源** 是到某个上游的连接描述:`name`、`base_url`、`api_key`、模型列表与优先级。 +- **适配器** 是协议转换逻辑(Lua 脚本):把统一 OpenAI 格式请求转成上游原生格式, + 再把上游响应/流式分块转回统一格式。 +- 一个适配器可被多个源复用(如 `openai.lua` 同时服务多个 OpenAI 兼容站点); + 同一个源也可以切换不同适配器(改 `adapter` 字段即可)。 +- 源决定“连谁、暴露哪些模型”,适配器决定“怎么对话”——二者在 `sources[]` 条目中 + 通过 `adapter` 字段关联。 + +加载流程(`internal/core` 的 `Core` 负责装配): + +1. 启动时 `lua.NewVM(adapter_dir)` 加载全部适配器:内置适配器(编译期 embed)+ + `adapter_dir` 下的同名覆盖文件;内置文件写在代码内,覆盖文件要求更高优先级。 +2. `config.Load` 读取 `config.yaml`,`config.NewStore(runtime_file)` 读 WebUI 改动的 + 运行时源,二者按名称合并成完整源列表。 +3. `rebuildRegistry` 为每个源创建 `provider.Provider`(持有目标适配器),并按源的 + 并发上限配置适配器 worker 池大小。 +4. 请求进来时 `Registry` 按 `model` 路由到 Provider,Provider 调适配器 + `transform_request` → HTTP 发送 → `transform_response` / `transform_stream_chunk`。 + +WebUI 上的"新增/编辑源"与"上传 Lua 适配器"都即时生效(写入运行时文件或 +`adapter_dir` 后重新装配,无需重启);直接编辑 `config.yaml` / `adapter_dir` +下的文件则需要重启进程才会重新加载。 + ### disable_thinking 请求体带 `"disable_thinking": true` 时,网关透传给各适配器;DeepSeek 适配器将其 @@ -118,9 +146,11 @@ sources: ## Lua 适配器协议 +完整 API 见 **[Lua 适配器 API 文档](docs/lua-adapters.md)**([English](docs/lua-adapters-en.md))。 + 每个适配器是一个返回 table 的 Lua 脚本(`internal/lua/adapters/.lua`), -加载时可被 `adapter_dir` 下的同名脚本覆盖——**改 Lua 脚本同样无需重编译**, -重启即生效(甚至可通过 WebUI 在线编辑源配置)。 +加载时可被 `adapter_dir` 下的同名脚本覆盖——**改 Lua 脚本无需重编译**, +重启即生效;通过 WebUI 上传的适配器与在线编辑的源配置则即时生效。 ```lua return { diff --git a/README_EN.md b/README_EN.md new file mode 100644 index 0000000..5e15dca --- /dev/null +++ b/README_EN.md @@ -0,0 +1,242 @@ +# ModelRouter + +> **中文**: [README.md](./README.md) + +A lightweight, unified OpenAI-compatible LLM gateway for internal networks — +configure once, and let multiple services (agents, SDKs, bots) share many +upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama, +KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mode, +which routes to the best healthy upstream by configured priority. + +> **Lightweight, no recompile to add sources**: the gateway is a single Go +> binary (~10 MB, zero runtime dependencies). Adding or switching an upstream is +> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua` +> adapter — **no Go changes, no recompile**. Changes submitted through the +> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua` +> file under `adapter_dir` by hand requires a process restart. + +## Overview + +Exposes a standard `OpenAI Chat Completions` API (`/v1/chat/completions` + +`/v1/models`) to your internal network, translating between multiple upstream +protocols via **Lua adapters**. Supports one-shot calls, SSE streaming, +**AUTO model routing**, multimodal passthrough, and tool calls. + +Extracted and independently evolved from the multi-source LLM adapter layer of +[HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent) +(`internal/agent/api/provider.go` + `internal/lua/adapters/*`). + +## Features + +- **Multi-source**: any number of upstream sources in one process, routed by the + request's `model`. +- **AUTO mode**: with `default_model: AUTO`, the highest-priority available + source model wins. +- **Unified output**: every source speaks OpenAI format (incl. + `reasoning_content`, `tool_calls`, `usage`). +- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly; + Anthropic/Gemini/Ollama are translated automatically. +- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker + pool for safe concurrency. +- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request. +- **Lua adapter protocol**: each source mounts a `.lua` adapter with + `transform_request` / `transform_response` / `transform_stream_chunk` — all + protocol differences live in Lua, Go only schedules and proxies. +- **Signature / header hooks**: adapters may define `build_headers(meta)` to + inject/sign request headers before the HTTP call (e.g. KimiCode style + app-validation), with helpers like `hmac_sha256_hex`, `sha256_hex`, + `base64_encode`. +- **WebUI**: built-in management page to view/add/edit sources & models, + persisted to a runtime file. +- **Auth**: the gateway validates client keys via `gateway_keys`; independent + from each upstream's own key. +- **Streaming**: SSE `chat.completion.chunk` with a role-first chunk and + `[DONE]` terminator. + +## Quick start + +```bash +cp config.example.yaml config.yaml # edit your sources & keys +GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy +./llmsproxy -config config.yaml +``` + +> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings). +> You **must** build with `-tags luajit`; otherwise the built-in superset +> gopher-lua path is used (behavior differs slightly). + +```bash +# no key -> 401 +curl http://127.0.0.1:8080/v1/models + +# one-shot +curl -H "Authorization: Bearer sk-gw-local-0001" \ + -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}]}' \ + http://127.0.0.1:8080/v1/chat/completions + +# streaming +curl -N -H "Authorization: Bearer sk-gw-local-0001" \ + -d '{"model":"deepseek-v4-flash","stream":true,"messages":[{"role":"user","content":"hi"}]}' \ + http://127.0.0.1:8080/v1/chat/completions +``` + +Any OpenAI SDK works: point `base_url` at the gateway, use one of `gateway_keys` +as the API key. + +## Configuration + +See [`config.example.yaml`](config.example.yaml). Core fields: + +```yaml +listen: 127.0.0.1:8080 # bind address (keep internal/loopback) +gateway_keys: [sk-gw-0001] # keys clients use; empty = no auth +default_model: AUTO # when model is unroutable, pick source by priority +adapter_dir: adapters # Lua adapter dir; built-ins written on first start +runtime_file: runtime.json # WebUI-edited sources persist here + +sources: + - name: deepseek + base_url: https://api.deepseek.com + api_key: sk-... + adapter: deepseek + max_concurrent: 8 + models: + - id: deepseek-v4-flash + priority: 100 # higher -> preferred by AUTO + kind: chat + - id: deepseek-v4-pro + priority: 60 + kind: chat + # static headers (take precedence over adapter defaults) + headers: { X-Tenant: prod } + # passthrough metadata for the Lua build_headers hook + meta: { app_id: x, app_secret: y } + temperature: 0.7 + max_tokens: 4096 + timeout: 120s # request timeout, default 120s +``` + +### Model routing + +`/v1/chat/completions` `model` resolution order: + +1. `source/model` or `source:model` prefix → pinned source; +2. exact match of a source's `model`; +3. with `default_model: AUTO` → highest-`priority` healthy source model; +4. otherwise fall back to `default_source`. + +Any OpenAI client can pin to a source by setting `model` to its +`name/anything`; `AUTO` (or the gateway's `default_model: AUTO`) picks the +healthy source by priority. + +### Sources vs. adapters + +- A **source** describes a connection to an upstream: `name`, `base_url`, + `api_key`, model list and priorities. +- An **adapter** is the protocol translation logic (Lua script): converts the + unified OpenAI request to the upstream's native format and back. +- One adapter serves many sources (e.g. `openai.lua` for any OpenAI-compatible + site); one source can switch adapters via its `adapter` field. +- Sources decide *whom to talk to and which models to expose*; adapters decide + *how to talk*. They are linked by the `adapter` field inside each `sources[]` + entry. + +Loading flow (assembled by `Core` in `internal/core`): + +1. On startup `lua.NewVM(adapter_dir)` loads all adapters: built-in ones + (embedded at compile time) + same-name override files under `adapter_dir`. +2. `config.Load` reads `config.yaml`; `config.NewStore(runtime_file)` reads + WebUI-edited runtime sources; both are merged by name. +3. `rebuildRegistry` creates a `provider.Provider` per source (holding its + adapter), sizing the adapter's worker pool from the source's concurrency + limit. +4. Incoming requests are routed by `Registry` to a provider, which calls the + adapter's `transform_request` → HTTP call → `transform_response` / + `transform_stream_chunk`. + +"Add/Edit source" and "Upload adapter" in the WebUI take effect immediately +(written to the runtime file / `adapter_dir`, then reassembled — no restart). +Directly editing `config.yaml` or files under `adapter_dir` requires a process +restart. + +### disable_thinking + +With `"disable_thinking": true` in the request body, the gateway passes it to +each adapter; the DeepSeek adapter maps it to `extra_body.thinking.type = +"disabled"`, other sources follow their own protocol. + +### WebUI + +Built-in admin page at `GET /`; after login you can view/edit sources and +models in the browser, persisted to `runtime_file` (survives restarts). + +## Lua adapter protocol + +Full API: **[Lua adapter docs](docs/lua-adapters.md)** +([中文](docs/lua-adapters-en.md)). + +Each adapter is a Lua script returning a table +(`internal/lua/adapters/.lua`), overridable by a same-name file under +`adapter_dir` — **no recompile needed**, restart to take effect; uploads via +the WebUI take effect immediately. + +```lua +return { + name = "mysrc", + version = "1.0.0", + endpoint = "/chat/completions", -- upstream path (source.endpoint overrides) + headers = { ["X-Static"] = "v" }, -- static default headers (fallback) + + -- request: convert unified OpenAI req -> upstream native format, return string + transform_request = function(raw_json) ... end, + + -- response: convert upstream response to unified format string + -- { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] } + transform_response = function(raw_json) ... end, + + -- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips + transform_stream_chunk = function(raw_chunk) ... end, + + -- [optional] dynamic headers / signature hook + -- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } } + build_headers = function(meta) return { ["X-App-Sign"] = sign } end, +} +``` + +Shared helpers: `hmac_sha256_hex(key, data)`, `sha256_hex(data)`, +`base64_encode(s)`, `tohex(s)`, `json.encode/decode`, `log(level, msg)`. + +### Built-in adapters + +`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama` +`kimicode`. + +`anthropic`/`gemini`/`ollama` include multimodal conversion +(`image_url` → their native format); with `disable_thinking` the `deepseek` +adapter sets `extra_body.thinking.type` to `disabled`. + +**kimicode** demonstrates `build_headers`: the cloud validates the calling +app, so you HMAC-sign timestamp+URL+body with `meta.app_secret` and add +`X-App-Sign`-style headers. Configure `sources[].meta.{app_id, app_secret, +app_agent}`. + +## Layout + +``` +cmd/llmsproxy # entry point +internal/config # YAML config loading/validation +internal/lua # LuaJIT VM + worker pool + AdapterCache + built-ins (embed) +internal/provider # Provider(HTTP) + Registry(routing) +internal/gateway # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI +internal/types # unified format & OpenAI wire types +``` + +## Tests + +```bash +go test -tags luajit ./... +``` + +Covers: config validation, adapter load/transform, signature hooks, gateway +auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and +disable_thinking. \ No newline at end of file diff --git a/docs/lua-adapters-en.md b/docs/lua-adapters-en.md new file mode 100644 index 0000000..bc62f84 --- /dev/null +++ b/docs/lua-adapters-en.md @@ -0,0 +1,271 @@ +# Lua Adapter API + +> **中文**: [lua-adapters.md](./lua-adapters.md) + +Every upstream source is mounted with a `.lua` adapter that **transforms** the unified +OpenAI-format request into the upstream's native format, and converts upstream +responses / stream chunks back into the unified format. The Go layer only handles +scheduling, concurrency and pass-through — so **adding an adapter or adapting a new +protocol never requires recompiling Go**. + +Adapters live in two equivalent places: +- Bundled: `internal/lua/adapters/.lua` (embedded at build time) +- Override: a same-named script in the configured `adapter_dir` (takes precedence) + +> **When are they loaded?** All adapters are loaded once at startup by +> `lua.NewVM(adapter_dir)`. Adapters uploaded via the WebUI +> (`Core.UploadAdapter` → `vm.LoadAdapter`) and sources edited online take +> effect immediately, without a restart. Directly editing a `.lua` file under +> `adapter_dir` requires a process restart to reload. + +## Table of Contents + +1. [Script Structure](#1-script-structure) +2. [Static Fields](#2-static-fields) +3. [Transform Hooks](#3-transform-hooks) +4. [Optional Hooks](#4-optional-hooks) +5. [Built-in Helper Functions](#5-built-in-helper-functions) +6. [meta Contract](#6-meta-contract) +7. [A Complete Minimal Adapter](#7-a-complete-minimal-adapter) +8. [Multimodal & disable_thinking](#8-multimodal--disable_thinking) + +--- + +## 1. Script Structure + +A script is a Lua file that returns a table — it **must** `return` a table: + +```lua +local adapter = {} +adapter.name = "mysrc" +adapter.version = "1.0.0" +-- ... fields and functions ... +return adapter +``` + +Scripts are executed by LuaJIT (via golua bindings, cgo). Each adapter owns an +**independent VM + worker pool**; adapters never interfere with each other. No mutable +state may be shared across workers (`log` to stdout is the only side effect). + +## 2. Static Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | yes | Adapter name, shown in the WebUI | +| `version` | string | no | Version, shown in the WebUI | +| `endpoint` | string | no | Upstream path, default `/chat/completions`; overridable by `source.endpoint` / `source.image_endpoint` | +| `headers` | table | no | Static default request headers; used as fallback when no `build_headers` hook is defined | + +These fields are extracted statically at load time (compile-once); reading them never +occupies a pooled worker. + +## 3. Transform Hooks + +### `transform_request(raw_body) -> string` + +Input: the unified OpenAI-format JSON string built by the Go layer (the +`/v1/chat/completions` request body). +Output: the request body string to send upstream. + +```lua +function adapter.transform_request(raw_body) + local ok, req = pcall(json.decode, raw_body) + if not ok then return raw_body end + req.model = "upstream-model-name" -- rewrite the model name + req.stream = req.stream or false + return json.encode(req) +end +``` + +Contract: +- The returned string is POSTed verbatim to `base_url + (source.endpoint or adapter.endpoint)`. +- Internal gateway fields such as `disable_thinking` and `extra_body` should be cleaned + up by the transform (`openai.lua` deletes both). + +### `transform_response(raw_body) -> string` + +Input: the raw JSON string of a non-streaming upstream response. +Output: a JSON string in the unified format. + +Unified format fields: + +| Field | Type | Description | +|-------|------|-------------| +| `content` | string | Response text | +| `finish_reason` | string | `stop` / `length` / `tool_calls` etc. | +| `reasoning_content` | string (optional) | Reasoning text (DeepSeek etc.) | +| `token_usage` | table | `{ prompt, completion, total }` (tokens) | +| `tool_calls` | array (optional) | Tool calls: `{ id, type, name, arguments }`; `arguments` is a **decoded** table | + +```lua +function adapter.transform_response(raw_body) + local ok, resp = pcall(json.decode, raw_body) + if not ok or resp == nil then return raw_body end + local unified = { + content = "", + finish_reason = "", + token_usage = { prompt = 0, completion = 0, total = 0 } + } + if type(resp.choices) == "table" and #resp.choices > 0 then + local ch = resp.choices[1] + unified.content = ch.message.content or "" + unified.finish_reason = ch.finish_reason or "" + if ch.message.reasoning_content then + unified.reasoning_content = ch.message.reasoning_content + end + end + return json.encode(unified) +end +``` + +### `transform_stream_chunk(raw_chunk) -> string` + +Input: the raw JSON string of each `data:` line in the SSE stream (without the +`data:` prefix). +Output: a unified chunk JSON string; **returning `""` skips that chunk**. + +Unified chunk format: + +| Field | Type | Description | +|-------|------|-------------| +| `content` | string | Incremental text for this chunk (may be `""`) | +| `done` | boolean (optional) | `true` ends the stream (when `finish_reason` appears) | + +```lua +function adapter.transform_stream_chunk(raw_chunk) + local ok, chunk = pcall(json.decode, raw_chunk) + if not ok then return "" end + if not chunk.choices or #chunk.choices == 0 then return "" end + local delta = chunk.choices[1].delta or {} + local fr = chunk.choices[1].finish_reason + return json.encode({ + content = delta.content or "", + done = (fr ~= nil) + }) +end +``` + +## 4. Optional Hooks + +### `build_headers(meta) -> table` + +Dynamically generate / sign request headers (e.g. KimiCode's HMAC signature). If the +script does not define this function, the Go layer falls back to the static +`adapter.headers` (or `source.headers`). + +```lua +function adapter.build_headers(meta) + local msg = meta.method .. meta.url .. meta.body + return { + ["X-App-Sign"] = hmac_sha256_hex(meta.source.meta.app_secret, msg), + ["X-Timestamp"] = meta.timestamp, + } +end +``` + +Returning a non-table errors; returning `{}` means no custom headers (no fallback to +static headers). + +## 5. Built-in Helper Functions + +Global functions shared by all adapters (injected by Go): + +| Function | Description | +|----------|-------------| +| `json.encode(v)` | Lua value → JSON string; returns `"null"` on failure | +| `json.decode(s)` | JSON string → Lua value; returns `nil` on failure | +| `hmac_sha256_hex(key, data)` | HMAC-SHA256, lowercase hex string | +| `sha256_hex(data)` | SHA-256, lowercase hex string | +| `base64_encode(s)` | Standard Base64 encoding | +| `tohex(s)` | Bytes → lowercase hex string | +| `log(level, msg)` | Prints `[adapter/] ` to stdout | + +Note: scripts run under LuaJIT with the full standard library (`string`/`table`/`pcall` +etc.); `os`/`io` are not exposed (sandbox semantics). + +## 6. meta Contract + +The meta structure shared by `build_headers(meta)` and the request transforms: + +| Field | Description | +|-------|-------------| +| `meta.url` | Full request URL | +| `meta.method` | HTTP method (usually `POST`) | +| `meta.body` | Request body string | +| `meta.api_key` | The source's `api_key` | +| `meta.timestamp` | Request timestamp | +| `meta.source` | table: `{ name, meta = { ... } }` — the source's `meta` field (e.g. `app_secret`) | + +## 7. A Complete Minimal Adapter + +```lua +local adapter = {} + +adapter.name = "mysrc" +adapter.version = "1.0.0" +adapter.endpoint = "/chat/completions" +adapter.headers = { ["X-Tenant"] = "prod" } + +function adapter.transform_request(raw_body) + local ok, req = pcall(json.decode, raw_body) + if not ok then return raw_body end + req.disable_thinking = nil + req.extra_body = nil + return json.encode(req) +end + +function adapter.transform_response(raw_body) + local ok, resp = pcall(json.decode, raw_body) + if not ok or resp == nil then return raw_body end + local unified = { + content = "", + finish_reason = "", + token_usage = { prompt = 0, completion = 0, total = 0 } + } + if type(resp.choices) == "table" and #resp.choices > 0 then + local ch = resp.choices[1] + unified.content = ch.message.content or "" + unified.finish_reason = ch.finish_reason or "" + end + return json.encode(unified) +end + +function adapter.transform_stream_chunk(raw_chunk) + local ok, chunk = pcall(json.decode, raw_chunk) + if not ok then return "" end + if not chunk.choices or #chunk.choices == 0 then return "" end + return json.encode({ + content = (chunk.choices[1].delta or {}).content or "", + done = (chunk.choices[1].finish_reason ~= nil) + }) +end + +return adapter +``` + +Corresponding `config.yaml` source: + +```yaml +sources: + - name: mysrc + base_url: https://upstream.example.com + api_key: sk-xxx + adapter: mysrc # or omit = bundled adapter with the same name + endpoint: /chat/completions + models: + - id: my-model + priority: 50 + kind: chat +``` + +## 8. Multimodal & disable_thinking + +- **Multimodal**: if the request `content` is an array + (`[{type:"text"...},{type:"image_url"...}]`), the Go layer passes it through + verbatim to the adapter. Sources whose protocol doesn't support it (Anthropic / + Gemini / Ollama) must convert inside `transform_request`; the bundled + `anthropic.lua` / `gemini.lua` / `ollama.lua` already do. +- **disable_thinking**: the gateway passes `disable_thinking:true` from the request + body into `transform_request`. The DeepSeek adapter maps it to + `extra_body.thinking = { type = "disabled" }` and cleans up its own field; other + adapters handle it per their protocol (implement or ignore as needed). diff --git a/docs/lua-adapters.md b/docs/lua-adapters.md new file mode 100644 index 0000000..6bd14c3 --- /dev/null +++ b/docs/lua-adapters.md @@ -0,0 +1,257 @@ +# Lua 适配器 API 文档 + +> **English**: [lua-adapters-en.md](./lua-adapters-en.md) + +每个上游源(source)挂载一个 `.lua` 适配器,负责把统一 OpenAI 格式请求**转换**为上游原生 +格式,并把上游响应/流式分块**转换**回统一格式。Go 层只负责调度、并发与透传——因此**新增 +适配器、适配新协议,无需重编译 Go**。 + +适配器放两份(同构): +- 内置:`internal/lua/adapters/.lua`(编译期 embed) +- 可覆盖:配置 `adapter_dir` 目录下的同名脚本(优先级更高) + +> **加载时机**:启动时由 `lua.NewVM(adapter_dir)` 一次性加载全部适配器;通过 WebUI +> 上传的适配器(`Core.UploadAdapter` → `vm.LoadAdapter`)与在线编辑的源配置即时生效, +> 无需重启。直接修改 `adapter_dir` 下的 `.lua` 文件需要重启进程才会重新加载。 + +## 目录 + +1. [脚本结构](#1-脚本结构) +2. [静态字段](#2-静态字段) +3. [转换钩子](#3-转换钩子) +4. [可选钩子](#4-可选钩子) +5. [内置辅助函数](#5-内置辅助函数) +6. [meta 约定](#6-meta-约定) +7. [一个完整的最小适配器](#7-一个完整的最小适配器) +8. [多模态与 disable_thinking](#8-多模态与-disable_thinking) + +--- + +## 1. 脚本结构 + +脚本是一个返回 table 的 Lua 文件,必须 `return` 一个表: + +```lua +local adapter = {} +adapter.name = "mysrc" +adapter.version = "1.0.0" +-- ... 字段与函数 ... +return adapter +``` + +脚本由 LuaJIT 解析(golua 绑定,cgo)。每个适配器拥有**独立 VM + worker 池**,适配器之间 +互不干扰;脚本中不允许跨 worker 共享可变状态(`log` 到 stdout 是唯一副作用)。 + +## 2. 静态字段 + +| 字段 | 类型 | 必填 | 说明 | +|------|------|------|------| +| `name` | string | 是 | 适配器名,用于 WebUI 展示 | +| `version` | string | 否 | 版本号,用于 WebUI 展示 | +| `endpoint` | string | 否 | 上游请求路径,默认 `/chat/completions`;可被 `source.endpoint` / `source.image_endpoint` 覆盖 | +| `headers` | table | 否 | 静态默认请求头;若未定义 `build_headers` 钩子则作为请求头回退 | + +这些字段在加载时静态提取(compile-once),之后读它们不会占用池内 worker。 + +## 3. 转换钩子 + +### `transform_request(raw_body) -> string` + +入参:Go 层构造的统一 OpenAI 格式 JSON 字符串(`/v1/chat/completions` 请求体)。 +返回:发送给上游的请求体字符串。 + +```lua +function adapter.transform_request(raw_body) + local ok, req = pcall(json.decode, raw_body) + if not ok then return raw_body end + req.model = "upstream-model-name" -- 改写模型名 + req.stream = req.stream or false + return json.encode(req) +end +``` + +约定: +- 返回的字符串将被原样 POST 到 `base_url + (source.endpoint or adapter.endpoint)`。 +- 请求体中包含的 `disable_thinking`、`extra_body` 等网关内部字段,转换时应自行清理 + (`openai.lua` 即删掉这两个字段)。 + +### `transform_response(raw_body) -> string` + +入参:上游非流式响应的原始 JSON 字符串。返回:统一格式 JSON 字符串。 + +统一格式字段: + +| 字段 | 类型 | 说明 | +|------|------|------| +| `content` | string | 回复正文 | +| `finish_reason` | string | `stop` / `length` / `tool_calls` 等 | +| `reasoning_content` | string(可选) | 推理内容(DeepSeek 等) | +| `token_usage` | table | `{ prompt, completion, total }`(tokens) | +| `tool_calls` | array(可选) | 工具调用:`{ id, type, name, arguments }`,`arguments` 为**已解码**的 table | + +```lua +function adapter.transform_response(raw_body) + local ok, resp = pcall(json.decode, raw_body) + if not ok or resp == nil then return raw_body end + local unified = { + content = "", + finish_reason = "", + token_usage = { prompt = 0, completion = 0, total = 0 } + } + if type(resp.choices) == "table" and #resp.choices > 0 then + local ch = resp.choices[1] + unified.content = ch.message.content or "" + unified.finish_reason = ch.finish_reason or "" + if ch.message.reasoning_content then + unified.reasoning_content = ch.message.reasoning_content + end + end + return json.encode(unified) +end +``` + +### `transform_stream_chunk(raw_chunk) -> string` + +入参:SSE 流中每条 `data:` 的原始 JSON 字符串(不含 `data:` 前缀)。 +返回:统一分块 JSON;**返回 `""` 表示跳过该 chunk**。 + +统一分块格式: + +| 字段 | 类型 | 说明 | +|------|------|------| +| `content` | string | 本次增量文本(可为 `""`) | +| `done` | boolean(可选) | `true` 表示流结束(对应 `finish_reason` 出现) | + +```lua +function adapter.transform_stream_chunk(raw_chunk) + local ok, chunk = pcall(json.decode, raw_chunk) + if not ok then return "" end + if not chunk.choices or #chunk.choices == 0 then return "" end + local delta = chunk.choices[1].delta or {} + local fr = chunk.choices[1].finish_reason + return json.encode({ + content = delta.content or "", + done = (fr ~= nil) + }) +end +``` + +## 4. 可选钩子 + +### `build_headers(meta) -> table` + +动态生成/签名请求头(如 KimiCode 的 HMAC 签名)。若脚本未定义此函数,Go 层回退使用静态 +`adapter.headers`(或 `source.headers`)。 + +```lua +function adapter.build_headers(meta) + local msg = meta.method .. meta.url .. meta.body + return { + ["X-App-Sign"] = hmac_sha256_hex(meta.source.meta.app_secret, msg), + ["X-Timestamp"] = meta.timestamp, + } +end +``` + +返回非 table 会报错;返回 `{}` 表示无自定义头(不会回退静态头)。 + +## 5. 内置辅助函数 + +所有适配器共享的全局函数(由 Go 注入): + +| 函数 | 说明 | +|------|------| +| `json.encode(v)` | Lua 值 → JSON 字符串;失败返回 `"null"` | +| `json.decode(s)` | JSON 字符串 → Lua 值;失败返回 `nil` | +| `hmac_sha256_hex(key, data)` | HMAC-SHA256,十六进制小写字符串 | +| `sha256_hex(data)` | SHA-256,十六进制小写字符串 | +| `base64_encode(s)` | 标准 Base64 编码 | +| `tohex(s)` | 字节 → 十六进制小写字符串 | +| `log(level, msg)` | 打印 `[adapter/] ` 到 stdout | + +另注:脚本由 LuaJIT 执行,标准库(`string`/`table`/`pcall` 等)完整可用,`os`/`io` 未暴露 +(保持沙箱语义)。 + +## 6. meta 约定 + +`build_headers(meta)` 与请求转换共享的 meta 结构: + +| 字段 | 说明 | +|------|------| +| `meta.url` | 完整请求 URL | +| `meta.method` | HTTP 方法(通常 `POST`) | +| `meta.body` | 请求体字符串 | +| `meta.api_key` | 该源的 `api_key` | +| `meta.timestamp` | 请求时间戳 | +| `meta.source` | table:`{ name, meta = { ... } }`,即 source 的 `meta` 字段(如 `app_secret`) | + +## 7. 一个完整的最小适配器 + +```lua +local adapter = {} + +adapter.name = "mysrc" +adapter.version = "1.0.0" +adapter.endpoint = "/chat/completions" +adapter.headers = { ["X-Tenant"] = "prod" } + +function adapter.transform_request(raw_body) + local ok, req = pcall(json.decode, raw_body) + if not ok then return raw_body end + req.disable_thinking = nil + req.extra_body = nil + return json.encode(req) +end + +function adapter.transform_response(raw_body) + local ok, resp = pcall(json.decode, raw_body) + if not ok or resp == nil then return raw_body end + local unified = { + content = "", + finish_reason = "", + token_usage = { prompt = 0, completion = 0, total = 0 } + } + if type(resp.choices) == "table" and #resp.choices > 0 then + local ch = resp.choices[1] + unified.content = ch.message.content or "" + unified.finish_reason = ch.finish_reason or "" + end + return json.encode(unified) +end + +function adapter.transform_stream_chunk(raw_chunk) + local ok, chunk = pcall(json.decode, raw_chunk) + if not ok then return "" end + if not chunk.choices or #chunk.choices == 0 then return "" end + return json.encode({ + content = (chunk.choices[1].delta or {}).content or "", + done = (chunk.choices[1].finish_reason ~= nil) + }) +end + +return adapter +``` + +对应的 `config.yaml` source: + +```yaml +sources: + - name: mysrc + base_url: https://upstream.example.com + api_key: sk-xxx + adapter: mysrc # 或省略 = 内置同名适配器 + endpoint: /chat/completions + models: + - id: my-model + priority: 50 + kind: chat +``` + +## 8. 多模态与 disable_thinking + +- **多模态**:请求的 `content` 若是数组(`[{type:"text"...},{type:"image_url"...}]`),Go 层 + 原样透传给适配器。协议不支持的源(Anthropic/Gemini/Ollama)需在 + `transform_request` 内转换;内置 `anthropic.lua`/`gemini.lua`/`ollama.lua` 已实现。 +- **disable_thinking**:网关将请求体中的 `disable_thinking:true` 透传进 + `transform_request`。DeepSeek 适配器将其映射为 `extra_body.thinking = { type = "disabled" }` + 并清理自身字段;其余适配器按各自协议处理(可自行实现或忽略)。 diff --git a/internal/gateway/chat.go b/internal/gateway/chat.go index 7b1d3e5..311e406 100644 --- a/internal/gateway/chat.go +++ b/internal/gateway/chat.go @@ -44,10 +44,10 @@ type ChatChoice struct { } type RespMessage struct { - Role string `json:"role"` - Content string `json:"content"` - ReasoningContent string `json:"reasoning_content,omitempty"` - ToolCalls []types.ToolCall `json:"tool_calls,omitempty"` + Role string `json:"role,omitempty"` + Content string `json:"content"` + ReasoningContent string `json:"reasoning_content,omitempty"` + ToolCalls json.RawMessage `json:"tool_calls,omitempty"` } type ChatChunk struct { @@ -77,13 +77,49 @@ func isAuto(m string) bool { } // resolveCands picks the ordered candidate providers for a requested model. -func (g *Gateway) resolveCands(model string) ([]*provider.Provider, string) { - if model == "" || isAuto(model) { +// toolCalling requests are anchored: they resolve to exactly one provider +// (highest-priority available) so a tool-call round never switches models. +func (g *Gateway) resolveCands(req *chatRequest) ([]*provider.Provider, string) { + model := req.Model + if model == "" { + model = g.core.DefaultModel() + } + cands, effective := g.resolveByModel(model) + if !toolRequest(req) { + return cands, effective + } + // tool-call request: pin to one provider (no AUTO fallback across models) + if len(cands) == 0 { + return nil, effective + } + first := cands[0] + eff := first.ModelFor(model) + if eff == "" { + eff = firstModel(first) + } + return []*provider.Provider{first}, eff +} + +func (g *Gateway) resolveByModel(model string) ([]*provider.Provider, string) { + if isAuto(model) { return g.core.Registry().Resolve("AUTO"), "" } return g.core.Registry().Resolve(model), model } +// toolRequest reports whether the request participates in a tool-call round. +func toolRequest(req *chatRequest) bool { + if len(req.Tools) > 0 || req.ToolChoice != nil { + return true + } + for _, m := range req.Messages { + if m.Role == "tool" || len(m.ToolCalls) > 0 { + return true + } + } + return false +} + func (g *Gateway) handleChat(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { writeError(w, http.StatusMethodNotAllowed, "method_not_allowed", "use POST") @@ -102,7 +138,7 @@ func (g *Gateway) handleChat(w http.ResponseWriter, r *http.Request) { if model == "" { model = g.core.DefaultModel() } - cands, effective := g.resolveCands(model) + cands, effective := g.resolveCands(&req) if len(cands) == 0 { writeError(w, http.StatusServiceUnavailable, "no_provider", "no LLM source configured") return @@ -159,6 +195,31 @@ func imageOnly(cands []*provider.Provider) []*provider.Provider { return out } +// toolCallsWire converts unified tool calls to the OpenAI wire format: +// tool_calls:[{id,type,function:{name,arguments:StringJSON}}]. Clients expect +// arguments to be a JSON string, not an object. +func toolCallsWire(tcs []types.ToolCall) json.RawMessage { + wire := make([]map[string]interface{}, 0, len(tcs)) + for _, tc := range tcs { + args := "{}" + if tc.Arguments != nil { + if b, err := json.Marshal(tc.Arguments); err == nil { + args = string(b) + } + } + wire = append(wire, map[string]interface{}{ + "id": tc.ID, + "type": tc.Type, + "function": map[string]interface{}{ + "name": tc.Name, + "arguments": args, + }, + }) + } + b, _ := json.Marshal(wire) + return b +} + func (g *Gateway) singleChat(w http.ResponseWriter, ctx context.Context, cands []*provider.Provider, req *types.ChatRequest, effective string) { resp, err := g.core.Scheduler().Chat(ctx, scheduler.FromRegistry(cands), req) if err != nil { @@ -170,7 +231,7 @@ func (g *Gateway) singleChat(w http.ResponseWriter, ctx context.Context, cands [ msg.ReasoningContent = resp.ReasoningContent } if len(resp.ToolCalls) > 0 { - msg.ToolCalls = resp.ToolCalls + msg.ToolCalls = toolCallsWire(resp.ToolCalls) } out := ChatCompletion{ ID: newID(), @@ -223,7 +284,7 @@ func (g *Gateway) streamChat(w http.ResponseWriter, ctx context.Context, cands [ chunk := ChatChunk{ ID: id, Object: "chat.completion.chunk", Created: created, Model: effective, } - delta := RespMessage{Content: ck.Content} + delta := RespMessage{Role: "assistant", Content: ck.Content} if ck.ReasoningContent != "" { delta.ReasoningContent = ck.ReasoningContent } @@ -269,7 +330,7 @@ func (g *Gateway) handleImage(w http.ResponseWriter, r *http.Request) { if model == "" { model = g.core.DefaultModel() } - cands, _ := g.resolveCands(model) + cands, _ := g.resolveByModel(model) cands = imageOnly(cands) if len(cands) == 0 { writeError(w, http.StatusServiceUnavailable, "no_provider", "no image source configured") diff --git a/internal/gateway/ui/index.html b/internal/gateway/ui/index.html index 06b8d17..c1ed534 100644 --- a/internal/gateway/ui/index.html +++ b/internal/gateway/ui/index.html @@ -155,6 +155,50 @@ html[data-theme="dark"] .dropzone.dragover, html[data-theme="dark"] .dropzone:ho .empty { color:var(--muted); text-align:center; padding:24px 0; } #modal-wrap { position:fixed; inset:0; background:rgba(15,22,44,.45); display:flex; align-items:flex-start; justify-content:center; overflow:auto; padding:48px 20px; z-index:50; } +.twrap { overflow-x:auto; -webkit-overflow-scrolling:touch; } + +/* ---------- responsive / narrow screens ---------- */ +@media (max-width: 900px) { + header { padding:12px 16px; } + nav { padding:12px 16px 0; overflow-x:auto; } + nav button { padding:7px 12px; white-space:nowrap; } + main { padding:16px 16px 40px; } + .card { padding:16px; } +} +@media (max-width: 640px) { + header { gap:8px; padding:10px 12px; } + .brand h1 { font-size:15px; } + .brand .sub { display:none; } + .hd-actions button { padding:5px 9px; } + nav { gap:4px; padding:10px 12px 0; } + nav button { padding:6px 10px; font-size:13px; } + main { padding:12px 12px 32px; } + .card { padding:13px; border-radius:12px; margin-bottom:14px; } + .card h2 { font-size:13px; } + th,td { padding:8px 10px; } + .row { flex-direction:column; gap:0; } + .model-row { flex-wrap:wrap; } + .model-row input { flex:1 1 120px; } + .model-row select { flex:0 0 auto; } + .tab-chat { height:calc(100vh - 150px); } + .msg { gap:7px; } + .avatar { width:24px; height:24px; font-size:11px; } + .bubble { max-width:90%; padding:8px 11px; font-size:13px; } + .chat-log { padding:14px 12px 6px; gap:14px; } + .chat-tools { flex-wrap:wrap; gap:8px; } + .chat-tools .tl { display:none; } + .chat-tools select { flex:1 1 auto; min-width:0; } + .chat-box { gap:7px; } + .chat-box .sendbtn { padding:10px 13px; } + .attach-btn { width:38px; height:38px; } + .chat-box textarea { font-size:13.5px; } + .chat-composer { padding:8px 9px 10px; } + #modal-wrap { padding:14px 10px; } + .dropzone { padding:18px 14px; } + #toast { left:12px; right:12px; bottom:12px; text-align:center; } + pre.configbox { font-size:11.5px; padding:11px; } + .att img { height:52px; max-width:90px; } +} diff --git a/internal/lua/adapters/anthropic.lua b/internal/lua/adapters/anthropic.lua index 7b2ec02..acf018e 100644 --- a/internal/lua/adapters/anthropic.lua +++ b/internal/lua/adapters/anthropic.lua @@ -103,12 +103,41 @@ function adapter.transform_stream_chunk(raw_chunk) if chunk.type == "message_delta" then return json.encode({ content = "", done = (chunk.delta and chunk.delta.stop_reason ~= nil) }) end + if chunk.type == "content_block_start" and chunk.content_block + and chunk.content_block.type == "tool_use" then + -- first fragment of a tool call: emit index + id + name, empty args + return json.encode({ + content = "", done = false, + tool_calls = { { + index = chunk.index or 0, + id = chunk.content_block.id or "", + type = "function", + ["function"] = { name = chunk.content_block.name or "", arguments = "" } + } } + }) + end if chunk.type == "content_block_delta" and chunk.delta then + if chunk.delta.type == "input_json_delta" then + -- incremental JSON fragment; clients accumulate across chunks + local unified = { content = "", done = false, tool_calls = { { + index = chunk.index or 0, + id = "", + type = "function", + ["function"] = { name = "", arguments = chunk.delta.partial_json or "" } + } } } + return json.encode(unified) + end + if chunk.delta.type == "thinking_delta" and chunk.delta.thinking then + return json.encode({ content = "", done = false, reasoning_content = chunk.delta.thinking }) + end return json.encode({ content = chunk.delta.text or "", done = false }) end if chunk.type == "message_stop" then return json.encode({ content = "", done = true }) end + if chunk.type == "content_block_stop" then + return json.encode({ content = "", done = false }) + end return "" end diff --git a/internal/lua/adapters/deepseek.lua b/internal/lua/adapters/deepseek.lua index de6652a..3344ce1 100644 --- a/internal/lua/adapters/deepseek.lua +++ b/internal/lua/adapters/deepseek.lua @@ -20,6 +20,18 @@ function adapter.transform_request(raw_body) req.extra_body.thinking = { type = "disabled" } end req.disable_thinking = nil + + -- V4 thinking 模式要求:带 tool_calls 的 assistant 消息必须回传 reasoning_content。 + -- OpenAI 兼容客户端不会发该字段,补空串即可通过校验。 + if req.messages then + for _, msg in ipairs(req.messages) do + if msg.role == "assistant" and msg.tool_calls and msg.tool_calls[1] then + if msg.reasoning_content == nil then + msg.reasoning_content = "" + end + end + end + end return json.encode(req) end @@ -73,10 +85,18 @@ function adapter.transform_stream_chunk(raw_chunk) if not chunk.choices or #chunk.choices == 0 then return "" end local delta = chunk.choices[1].delta or {} local fr = chunk.choices[1].finish_reason - return json.encode({ + + local unified = { content = delta.content or "", done = (fr ~= nil) - }) + } + if delta.reasoning_content then + unified.reasoning_content = delta.reasoning_content + end + if delta.tool_calls then + unified.tool_calls = delta.tool_calls + end + return json.encode(unified) end return adapter diff --git a/internal/lua/adapters/gemini.lua b/internal/lua/adapters/gemini.lua index d4ddd54..c6d190a 100644 --- a/internal/lua/adapters/gemini.lua +++ b/internal/lua/adapters/gemini.lua @@ -93,16 +93,31 @@ function adapter.transform_stream_chunk(raw_chunk) if not chunk.candidates or #chunk.candidates == 0 then return "" end local cand = chunk.candidates[1] - local content = "" + local unified = { content = "", done = (cand.finishReason ~= nil) } + local reasoning = "" + local tools = {} if cand.content and cand.content.parts then for _, part in ipairs(cand.content.parts) do - content = content .. (part.text or "") + if part.text then + unified.content = (unified.content or "") .. part.text + elseif part.reasoning_content then + reasoning = reasoning .. part.reasoning_content + elseif part.functionCall then + table.insert(tools, { + index = #tools, + id = part.functionCall.id or ("call_" .. #tools), + type = "function", + ["function"] = { + name = part.functionCall.name or "", + arguments = part.functionCall.args or "{}" + } + }) + end end end - return json.encode({ - content = content, - done = (cand.finishReason ~= nil) - }) + if reasoning ~= "" then unified.reasoning_content = reasoning end + if #tools > 0 then unified.tool_calls = tools end + return json.encode(unified) end return adapter diff --git a/internal/lua/adapters/github.lua b/internal/lua/adapters/github.lua index c27103b..baa2e0d 100644 --- a/internal/lua/adapters/github.lua +++ b/internal/lua/adapters/github.lua @@ -66,10 +66,18 @@ function adapter.transform_stream_chunk(raw_chunk) if not chunk.choices or #chunk.choices == 0 then return "" end local delta = chunk.choices[1].delta or {} local fr = chunk.choices[1].finish_reason - return json.encode({ + + local unified = { content = delta.content or "", done = (fr ~= nil) - }) + } + if delta.reasoning_content then + unified.reasoning_content = delta.reasoning_content + end + if delta.tool_calls then + unified.tool_calls = delta.tool_calls + end + return json.encode(unified) end return adapter diff --git a/internal/lua/adapters/groq.lua b/internal/lua/adapters/groq.lua index 3d66033..d235cd4 100644 --- a/internal/lua/adapters/groq.lua +++ b/internal/lua/adapters/groq.lua @@ -65,10 +65,18 @@ function adapter.transform_stream_chunk(raw_chunk) if not chunk.choices or #chunk.choices == 0 then return "" end local delta = chunk.choices[1].delta or {} local fr = chunk.choices[1].finish_reason - return json.encode({ + + local unified = { content = delta.content or "", done = (fr ~= nil) - }) + } + if delta.reasoning_content then + unified.reasoning_content = delta.reasoning_content + end + if delta.tool_calls then + unified.tool_calls = delta.tool_calls + end + return json.encode(unified) end return adapter diff --git a/internal/lua/adapters/kimicode.lua b/internal/lua/adapters/kimicode.lua index 064a460..0cb5cfc 100644 --- a/internal/lua/adapters/kimicode.lua +++ b/internal/lua/adapters/kimicode.lua @@ -98,10 +98,18 @@ function adapter.transform_stream_chunk(raw_chunk) if not chunk.choices or #chunk.choices == 0 then return "" end local delta = chunk.choices[1].delta or {} local fr = chunk.choices[1].finish_reason - return json.encode({ + + local unified = { content = delta.content or "", done = (fr ~= nil) - }) + } + if delta.reasoning_content then + unified.reasoning_content = delta.reasoning_content + end + if delta.tool_calls then + unified.tool_calls = delta.tool_calls + end + return json.encode(unified) end return adapter \ No newline at end of file diff --git a/internal/lua/adapters/mistral.lua b/internal/lua/adapters/mistral.lua index 763da2a..10598cb 100644 --- a/internal/lua/adapters/mistral.lua +++ b/internal/lua/adapters/mistral.lua @@ -65,10 +65,18 @@ function adapter.transform_stream_chunk(raw_chunk) if not chunk.choices or #chunk.choices == 0 then return "" end local delta = chunk.choices[1].delta or {} local fr = chunk.choices[1].finish_reason - return json.encode({ + + local unified = { content = delta.content or "", done = (fr ~= nil) - }) + } + if delta.reasoning_content then + unified.reasoning_content = delta.reasoning_content + end + if delta.tool_calls then + unified.tool_calls = delta.tool_calls + end + return json.encode(unified) end return adapter diff --git a/internal/lua/adapters/ollama.lua b/internal/lua/adapters/ollama.lua index 8cb92b9..4e13d71 100644 --- a/internal/lua/adapters/ollama.lua +++ b/internal/lua/adapters/ollama.lua @@ -72,10 +72,29 @@ function adapter.transform_stream_chunk(raw_chunk) if not ok then return "" end if not chunk.message then return "" end - return json.encode({ + local unified = { content = chunk.message.content or "", done = chunk.done or false - }) + } + if chunk.message.reasoning_content then + unified.reasoning_content = chunk.message.reasoning_content + end + if chunk.message.tool_calls then + local tools = {} + for _, tc in ipairs(chunk.message.tool_calls) do + table.insert(tools, { + index = #tools, + id = tc.id or ("call_" .. #tools), + type = "function", + ["function"] = { + name = tc["function"] and tc["function"].name or "", + arguments = tc["function"] and (tc["function"].arguments or "{}") or "{}" + } + }) + end + unified.tool_calls = tools + end + return json.encode(unified) end return adapter diff --git a/internal/lua/adapters/openai.lua b/internal/lua/adapters/openai.lua index 648c9b6..dcf9030 100644 --- a/internal/lua/adapters/openai.lua +++ b/internal/lua/adapters/openai.lua @@ -71,10 +71,18 @@ function adapter.transform_stream_chunk(raw_chunk) local delta = chunk.choices[1].delta or {} local fr = chunk.choices[1].finish_reason - return json.encode({ + local unified = { content = delta.content or "", done = (fr ~= nil) - }) + } + if delta.reasoning_content then + unified.reasoning_content = delta.reasoning_content + end + if delta.tool_calls then + -- pass raw streaming fragments through; OpenAI clients accumulate index+id+name+arguments + unified.tool_calls = delta.tool_calls + end + return json.encode(unified) end return adapter diff --git a/internal/provider/provider.go b/internal/provider/provider.go index c8e56af..1988157 100644 --- a/internal/provider/provider.go +++ b/internal/provider/provider.go @@ -100,6 +100,43 @@ func (p *Provider) ModelByID(id string) *config.Model { return nil } +// ModelFor resolves the model name this provider should send upstream. +// If the requested model is not owned by this provider (e.g. an AUTO chain +// fallback), it returns this provider's highest-priority chat model instead. +func (p *Provider) ModelFor(reqModel string) string { + if reqModel == "" || isAutoID(reqModel) { + return p.bestChatModel() + } + if p.ModelByID(reqModel) != nil { + return reqModel + } + return p.bestChatModel() +} + +// bestChatModel returns the highest-priority chat-kind model of this source. +func (p *Provider) bestChatModel() string { + bestID, bestPrio := "", -1 + for _, m := range p.cfg.Models { + if m.Kind != "" && m.Kind != "chat" { + continue + } + if m.Priority > bestPrio { + bestPrio = m.Priority + bestID = m.ID + } + } + if bestID == "" && len(p.cfg.Models) > 0 { + bestID = p.cfg.Models[0].ID + } + return bestID +} + +// IsAutoID reports whether s is an AUTO routing placeholder. +func isAutoID(s string) bool { + s = strings.TrimSpace(s) + return s == "" || strings.EqualFold(s, "AUTO") +} + // Endpoint resolves the upstream chat path. func (p *Provider) Endpoint() string { if p.cfg.Endpoint != "" { diff --git a/internal/scheduler/scheduler.go b/internal/scheduler/scheduler.go index f4361c6..6a89324 100644 --- a/internal/scheduler/scheduler.go +++ b/internal/scheduler/scheduler.go @@ -28,6 +28,7 @@ func New(maxRetries int) *Scheduler { type Provider interface { Name() string Available() bool + ModelFor(reqModel string) string Chat(ctx context.Context, req *types.ChatRequest) (*types.UnifiedResponse, error) ChatStream(ctx context.Context, req *types.ChatRequest) (<-chan types.UnifiedChunk, error) Image(ctx context.Context, req *types.ImageGenRequest) (*types.UnifiedResponse, error) @@ -42,13 +43,18 @@ func FromRegistry(ps []*provider.Provider) []Provider { return out } -// Chat runs a chat request across cands, falling back on failure. +// Chat runs a chat request across cands, falling back on failure. Each +// candidate receives a request pinned to its own model (ModelFor), so an AUTO +// chain fallback switches the model id per provider instead of reusing the +// first candidate's model name. func (s *Scheduler) Chat(ctx context.Context, cands []Provider, req *types.ChatRequest) (*types.UnifiedResponse, error) { attempts := s.MaxRetries + 1 var lastErr error for i := 0; i < attempts && i < len(cands); i++ { p := cands[i] - resp, err := p.Chat(ctx, req) + r := *req + r.Model = p.ModelFor(req.Model) + resp, err := p.Chat(ctx, &r) if ctx.Err() != nil { return nil, ctx.Err() } @@ -68,13 +74,16 @@ func (s *Scheduler) Chat(ctx context.Context, cands []Provider, req *types.ChatR return nil, lastErr } -// ChatStream runs a streaming chat across cands, falling back early on connect errors. +// ChatStream runs a streaming chat across cands, falling back early on connect +// errors. The request model is pinned per candidate like Chat. func (s *Scheduler) ChatStream(ctx context.Context, cands []Provider, req *types.ChatRequest) (<-chan types.UnifiedChunk, error) { attempts := s.MaxRetries + 1 var lastErr error for i := 0; i < attempts && i < len(cands); i++ { p := cands[i] - resp, err := p.ChatStream(ctx, req) + r := *req + r.Model = p.ModelFor(req.Model) + resp, err := p.ChatStream(ctx, &r) if err == nil { return resp, nil } diff --git a/internal/types/types.go b/internal/types/types.go index b0f78ea..1f07c2b 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -47,7 +47,7 @@ type ChatMessage struct { Content json.RawMessage `json:"content,omitempty"` ReasoningContent string `json:"reasoning_content,omitempty"` ToolCallID string `json:"tool_call_id,omitempty"` - ToolCalls []ToolCall `json:"tool_calls,omitempty"` + ToolCalls json.RawMessage `json:"tool_calls,omitempty"` } func StringContent(s string) json.RawMessage { b, _ := json.Marshal(s); return b } @@ -100,11 +100,14 @@ type ImageGenResponse struct { // ---- Unified streaming chunk produced by adapters ---- +// UnifiedChunk is one streamed delta. ToolCalls carries the raw upstream +// streaming tool_calls array (incremental fragments with an index field), which +// OpenAI-compatible clients accumulate themselves. type UnifiedChunk struct { - Content string `json:"content"` - Done bool `json:"done"` - ToolCalls []ToolCall `json:"tool_calls,omitempty"` - ReasoningContent string `json:"reasoning_content,omitempty"` + Content string `json:"content"` + Done bool `json:"done"` + ToolCalls json.RawMessage `json:"tool_calls,omitempty"` + ReasoningContent string `json:"reasoning_content,omitempty"` } // Meta passed to Lua build_headers hook