jianf/ModelRouter
1
0
Fork 0
mirror of https://gitcode.com/JianFeeeee/ModelRouter.git synced 2026-11-12 20:27:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: README 拆分为主题文档 + 中性化竞品表述

Browse Source 
README 从 642 行降到 131 行，只保留简介、核心能力、最小快速开始、文档索引、
改名说明、目录结构与测试入口。其余内容迁到 docs/，中英文索引一一对应：

  docs/features.md       实现与完整特性清单（原 README 的"实现/特性"）
  docs/configuration.md  配置文件、自动探测模型、多密钥与配额
  docs/routing.md        模型路由、AUTO 链、冷却与半冷却、源/适配器、源模板
  docs/performance.md    实测内存占用与调优、thinking 预算
  docs/webui.md          WebUI 使用
  docs/adapters.md       Lua 适配器协议与内置适配器清单
  docs/gui.md            桌面端 Headless/Desktop 两种产物与构建
  docs/development.md    目录职责、开发协作、测试
  docs/deployment.md     部署、发行包、目录权限、llmsproxy 迁移与验证清单

中性化：README / README_EN / GUI 注释里的 `Clash Verge` 类比全部改为
「桌面壳 / desktop shell」，仓库内（源码与文档）已无该表述。
This commit is contained in:
JianFeeeee
2026-10-26 22:28:24 +08:00
parent 688dce88d1
commit f7e926ccb7
14 changed files with 933 additions and 1178 deletions
Download Patch File Download Diff File Expand all files Collapse all files

689
README_EN.md
View File

@ -2,625 +2,148 @@
> **中文**: [README.md](./README.md)
>
> **AI-assisted**: This project is developed with AI-assisted coding (code and docs are produced in collaboration with AI, reviewed by humans).
> **AI-assisted**: code and documentation were produced with AI assistance and
> reviewed by a human.
A lightweight, unified OpenAI-compatible LLM gateway for internal networks —
configure once, and let multiple services (agents, SDKs, bots) share many
upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama,
KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mode,
which routes to the best healthy upstream by configured priority.
An intranet gateway for people with several LLM subscriptions: configure once and
let multiple services share the upstream capacity. Pick a model explicitly, or let
`AUTO` choose by priority among the sources that are currently healthy.
> **Lightweight, no recompile to add sources**: the gateway is a single Go
> binary (8-12 MB, ~8 MB stripped; zero runtime dependencies). Adding or switching an upstream is
> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua`
> adapter — **no Go changes, no recompile**. Changes submitted through the
> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua`
> file under `adapter_dir` by hand requires a process restart.
The gateway itself is a single Go binary (~10 MB, no runtime dependencies). Adding
or switching an upstream is a `sources` entry in `config.yaml` (or the WebUI), or a
dropped-in `.lua` adapter — **no Go changes, no rebuild**. Changes submitted through
the WebUI take effect immediately; editing `config.yaml` or a `.lua` adapter file
directly requires a restart.
## Overview
## What it does
Exposes a standard `OpenAI Chat Completions` API (`/v1/chat/completions` +
`/v1/models`) to your internal network, translating between multiple upstream
protocols via **Lua adapters**. Supports one-shot calls, SSE streaming,
**AUTO model routing**, multimodal passthrough, and tool calls.
Extracted and independently evolved from the multi-source LLM adapter layer of
[HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent)
(`internal/agent/api/provider.go` + `internal/lua/adapters/*`).
## Features
- **Multi-source**: any number of upstream sources in one process, routed by the
request's `model`.
- **AUTO mode**: with `default_model: AUTO`, scheduling follows the tiered AUTO
chain saved on the Priority page (see below).
- **Unified output**: every source speaks OpenAI format (incl.
`reasoning_content`, `tool_calls`, `usage`).
- **Image generation**: `POST /v1/images/generations`, routed to models with
`kind: image`.
- **Per-model quota**: each key gives every model its own token and request caps
plus a reset period (hour/week/month/custom N hours). One model running out
affects only that model — the key's other models keep working. Exhaustion
answers 429 + `Retry-After` naming the model, so a client resumes when the
window rolls over; admin keys are never capped.
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
Anthropic/Gemini/Ollama are translated automatically.
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
pool for safe concurrency. Pools are **elastic**: the sum of `max_concurrent`
over an adapter's sources is a ceiling, not a preallocation — states are booted
on demand and reclaimed when demand drops, so an idle gateway holds close to
zero Lua states. The grow step follows the adapter's max concurrency
(`clamp(ceil(max/8), 1, 8)`, and never exceeds the number of queued callers);
the shrink step follows the live connection count
(`clamp(ceil(slack/(1+in_use)), 1, slack)`), so an idle pool collapses in one
round while a busy one gives up a single state at a time.
- **On-demand request logs**: dashboard totals are computed from the **full**
audit history (streamed once at startup and released — 29 MB / 221k lines in
~260 ms), while the raw records are never held in memory: the dashboard loads
one screen, scrolling pages the rest straight off disk, CSV export streams in
O(1) memory, and leaving the page releases everything.
- **Self-healing cooldown**: cooldown is capped at 5 minutes and, past the
window's midpoint, exactly one probe request is allowed through; a recovered
upstream (or a reset quota) returns to full rotation on that probe instead of
waiting out the window.
- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request.
- **Lua adapter protocol**: each source mounts a `.lua` adapter with
`transform_request` / `transform_response` / `transform_stream_chunk` — all
protocol differences live in Lua, Go only schedules and proxies.
- **Signature / header hooks**: adapters may define `build_headers(meta)` to
inject/sign request headers before the HTTP call (e.g. KimiCode style
app-validation), with helpers like `hmac_sha256_hex`, `sha256_hex`,
`base64_encode`.
- **WebUI**: built-in management page to view/add/edit sources & models, edit
the AUTO chain, and manage per-key model scopes, persisted to a runtime file.
- **Encrypted secrets at rest**: upstream `api_key`, custom header values and
gateway keys are stored AES-256-GCM encrypted (`master.key` 0600 next to the
runtime file, or `LLMS_PROXY_MASTER_KEY`).
- **Live source probing**: the status page probes each source
(`GET {base}/models`, falling back to a minimal chat call) without touching
the scheduler's backoff state; errors are shown on hover.
- **Auth**: the gateway validates client keys via `gateway_keys`; independent
from each upstream's own key.
- **Streaming**: SSE `chat.completion.chunk` with a role-first chunk and
`[DONE]` terminator.
- **One OpenAI-compatible API**: `/v1/chat/completions`, `/v1/models`,
`/v1/images/generations`. Every source speaks the same shape outwards, including
`reasoning_content`, `tool_calls` and `usage`.
- **Many sources, one AUTO chain**: configure any number of upstreams; `AUTO`
follows the priority tiers saved in the WebUI, fails over between them, cools
down broken sources and probes them back to health at half-cool.
- **Protocol differences live in Lua**: each source binds a `.lua` adapter that
transforms requests, responses and stream chunks both ways, and can sign headers
via `build_headers(meta)`. Vendor knowledge stays out of the Go kernel.
- **Plugins are independent**: business extensions (channel login, billing, quota
probing) are external Lua plugins loaded from `plugin_dir`. They are not compiled
into the binary and not shipped in release archives, and they talk to the kernel
only through the `modelrouter` SDK.
- **Multi-tenant keys**: gateway keys are separate from upstream keys, and can be
scoped to a model set, an AUTO chain and a quota.
- **Encrypted at rest**: upstream keys, custom header values and gateway keys are
sealed with AES-256-GCM; `master.key` is 0600, or injected via
`MODELROUTER_MASTER_KEY`.
- **Observability**: live source probing, HTTP access log, config-change audit,
request and usage CSV export. Writes are kept off the request hot path.
## Quick start
```bash
GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
./llmsproxy -config config.yaml # first run generates a default config and prints a random admin key
# LuaJIT development libraries are required; -tags luajit is mandatory
GOMODCACHE=... GOPROXY=off go build -tags luajit -o modelrouter ./cmd/modelrouter
# First run generates a default config and prints a random admin key
./modelrouter -config config.yaml
```
> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings).
> You **must** build with `-tags luajit`; otherwise the built-in superset
> gopher-lua path is used (behavior differs slightly).
> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings). Without
> `-tags luajit` the build falls back to gopher-lua, which behaves slightly
> differently.
> **No config is shipped in the repo** (config files carry real keys). On
> first run the binary generates a default config at the `-config` path: a
> random admin key (printed to the startup log), loopback-only `127.0.0.1:8080`,
> and a keyless source ready to chat. Rotate the admin key in the
> WebUI after first login.
> **No config in the repo**: the repository ships no config file (they contain
> keys). The generated default binds `127.0.0.1:8080` only; rotate the admin key in
> the WebUI after the first login.
```bash
# no key -> 401
# No key -> 401
curl http://127.0.0.1:8080/v1/models
# one-shot ($KEY = the admin key printed at first startup)
# Non-streaming ($KEY = the admin key printed at first start)
curl -H "Authorization: Bearer $KEY" \
-d '{"model":"AUTO","messages":[{"role":"user","content":"hi"}]}' \
http://127.0.0.1:8080/v1/chat/completions
# streaming
# Streaming
curl -N -H "Authorization: Bearer $KEY" \
-d '{"model":"AUTO","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
http://127.0.0.1:8080/v1/chat/completions
```
Any OpenAI SDK works: point `base_url` at the gateway, use any authorized
gateway key as the API key.
## Configuration
Generated on first run at the `-config` path (default `./config.yaml`). Core fields:
```yaml
listen: 127.0.0.1:8080 # bind address (keep internal/loopback)
gateway_keys: [sk-gw-<random>] # initial admin key seed (random per install), migrated to the store on first start
default_model: AUTO # when model is unroutable, follow the AUTO chain
adapter_dir: adapters # Lua adapter dir; created+seeded if missing, read-only otherwise
runtime_file: runtime.json # WebUI-edited sources/keys/AUTO chain persist here
sources:
- name: deepseek
base_url: https://api.deepseek.com
api_key: sk-... # or api_key_env: SOME_ENV to read from an env var
adapter: deepseek
max_concurrent: 8
models:
- id: deepseek-v4-flash
priority: 100 # YAML sources seed the AUTO chain on first start
kind: chat
- id: deepseek-v4-pro
priority: 60
kind: chat
# static headers (take precedence over adapter defaults)
headers: { X-Tenant: prod }
# passthrough metadata for the Lua build_headers hook
meta: { app_id: x, app_secret: y }
temperature: 0.7
max_tokens: 4096
timeout: 120s # request timeout, default 120s
```
Sensitive fields in the runtime file (`runtime_file`) are encrypted at rest:
- AES-256-GCM, stored as `enc:v1:<base64>`.
- Master key source: env var `LLMS_PROXY_MASTER_KEY` (64 hex chars), else a
`master.key` file next to the runtime file; if neither exists it is generated
on first start (mode 0600).
- Old plaintext files still load; the first UI save migrates the whole file to
ciphertext.
- Back up `master.key` with your config — losing it makes the secrets
undecryptable. Do not commit it.
### Gateway keys (multi-key)
Gateway auth uses a "multi-key + role + model scope" model. Keys are persisted
under the `keys` field of the runtime file (encrypted at rest):
- The `gateway_keys` config is only an **initial admin key seed** — it is
migrated into the runtime store on first start and no longer drives auth.
- **Important: after first start, replace the admin key via the WebUI Keys
page.** The seed key is written in plaintext in `config.yaml`, so keeping it
active is a security risk; create a new admin key, log in with it, then
delete the seed key.
- The WebUI **Keys page** creates/deletes keys. Each key has a role (`admin`
manages everything, `user` sees only its own key) and an optional **model
scope** (model + source + token quota + reset period). Each model brick shows
its own budget badge (e.g. `1.0K 77×·1h`, `∞` when uncapped); clicking it
edits that model's token quota, request quota and reset period. Quotas apply
per model, so one model running out never blocks the key's others. The
"My key" view shows a user its model scopes and their budgets.
- Clients authenticate with any authorized key's plaintext as
`Authorization: Bearer <key>`.
- Deleting a key removes it from the store immediately.
#### Per-key usage quota
**Quotas are per model.** Each key's `models[]` list gives every model its own
token budget and request budget. One model running out affects only that model
— the key's other models keep working.
```yaml
keys:
- key: sk-gw-<hex>
role: user
name: agent-alice
models:
- model: deepseek-v4-flash
token_quota: 1000000 # this key's token budget for this model
req_quota: 20000 # requests within the window
period: nhour # "" | hour | week | month | nhour
hours: 6 # n-hour only
- model: AUTO
token_quota: 5000000 # AUTO is a quota entry like any other
period: hour
- model: kimi-k3 # no quota listed = unlimited
```
- **There is deliberately no key-wide total.** A key-wide cap would make
"move A's budget to B" a global reallocation; per-model budgets keep each
model independently controllable, so it stays visible which model is
actually consuming the spend.
- `period`: empty = never resets (lifetime total); `hour` / `week` / `month` =
fixed windows; `nhour` + `hours` = a custom hour count. **A misspelled
period is rejected at write time** rather than silently becoming a
never-resetting quota.
- Quotas are isolated per key *and* per model within a key: one key exhausting
`m1` never draws on another key's budget, and never blocks the same key's
`m2`.
- Usage counts chat, streaming and image requests, and survives a restart by
replaying the audit log (40 days retained, covering the longest `month`
window).
- An exhausted quota returns **429 + `Retry-After`**
(`rate_limit_exceeded`) and the message names the model that ran out, so a
client resumes when the window rolls over; a model the key may not use stays
403. **Admin keys are never capped** (quotas on their scopes are not
enforced either), so a cap can never lock the operator out.
- Buckets are whole unix hours, so a window frees up at most an hour late
(deliberately freeing late rather than overspending).
- `PUT /api/keys/{key}` submits quotas by submitting `models` — the caps are
part of the scope, so there is no second budget that can drift out of sync
with the model list. An explicit `0` lifts that model's cap.
##### Quota rejection vs capacity rejection
A quota is checked at the entry point and occupies no upstream slot, so it is a
cheap rejection. Running out of capacity instead waits out the bounded
`busyWait` before answering 503. Measured (capacity 8, 0.6 s/request, 100
concurrent clients):
| Case | Status | Latency | Reaches upstream |
|---|---|---|---|
| Quota spent (`req_quota` 50, 100 concurrent) | `429 rate_limit_exceeded` + `Retry-After` | **19–21 ms** | no |
| Capacity exhausted (same topology, no quota) | `503 upstream_error` | ~2.6 s (`busyWait`) | overflow: no |
So a client can tell "wait for the window to roll over" apart from "wait for
upstream capacity" — the first only needs patience, the second usually means
back off concurrency or switch sources.
##### What the quota costs
Buckets are kept per (key, model, whole unix hour) for 40 days. Measured on an
AMD 7840HS:
- A window query scans the window, not the whole retention: 49 ns for 1 h,
55 ns for 24 h, 3.9 µs for 30 d.
- Recording one request: 283 ns, 3 allocations (unchanged from before this
feature; the allocations come from the record ring buffer).
- Memory: the production shape (7 keys x 8 models x 2 sources at full 40-day
retention) costs about **3.7 MB**. The `source::model` bucket used by a
source-pinned quota is created **lazily** — it is maintained only once some
quota actually pins that source, because writing it on every record would
cost an extra 18 MB at 20 keys x 8 models x 3 sources.
The trade-off: usage recorded before a pinned quota was configured cannot be
split by source afterwards, so **a pinned quota may under-count its first
window**.
```
HTTP/1.1 429 Too Many Requests
Retry-After: 2100
{"error":{"type":"rate_limit_exceeded","message":"token quota exceeded for \"deepseek-v4-flash\" (5000000/5000000)"}}
```
### Model routing
`/v1/chat/completions` `model` resolution order:
1. `source/model` or `source:model` prefix → pinned source;
2. exact match of a source's `model`;
3. with `default_model: AUTO` → follow the AUTO chain from the Priority page
(see below);
4. otherwise the request errors.
Any OpenAI client can pin to a source by setting `model` to its
`name/anything`; `AUTO` (or the gateway's `default_model: AUTO`) follows the
tiered AUTO chain.
### AUTO chain (Priority page)
AUTO scheduling is driven **only** by the rules saved on the Priority page. The
numeric `models[].priority` in source configs no longer participates in
scheduling and is no longer shown. **Chat and image generation are two
independent chains**, switched in the Priority page via the `Chat / Image` toggle:
- **Chat chain** (`auto` field): serves `POST /v1/chat/completions` with
`model: AUTO`; only `kind: chat` models are honored (image slots are skipped).
- **Image chain** (`auto_image` field): serves `POST /v1/images/generations` with
`model: AUTO`; only `kind: image` models are honored. With no image chain
configured, AUTO image generation falls back to legacy discovery (all sources
exposing an image model, in registry order) for backward compatibility.
Each rule is one "slot": `{ model, source, tier, token_quota, period, hours }`.
- `tier` is the priority tier: models in the same tier sit side by side and
share it; tiers run high → low.
- The same model may appear in several slots (e.g. A low → B low → A high →
B high) and is tried in tier order.
- `token_quota` > 0 means the slot is skipped once its tokens within the reset
window are exhausted; `period` supports `hour` / `week` / `month` / `nhour`
(with `hours`); empty = unlimited.
- Within a tier, slots run by preference score; cooling / quota-exhausted /
hard-failed slots fall through to the next tier, and total failure returns
503 with a per-tier summary.
### Sources vs. adapters
- A **source** describes a connection to an upstream: `name`, `base_url`,
`api_key`, model list and priorities.
- An **adapter** is the protocol translation logic (Lua script): converts the
unified OpenAI request to the upstream's native format and back.
- One adapter serves many sources (e.g. `openai.lua` for any OpenAI-compatible
site); one source can switch adapters via its `adapter` field.
- Sources decide *whom to talk to and which models to expose*; adapters decide
*how to talk*. They are linked by the `adapter` field inside each `sources[]`
entry.
Loading flow (assembled by `Core` in `internal/core`):
1. On startup `lua.NewVM(adapter_dir)` loads all adapters: built-in ones
(embedded at compile time) + same-name override files under `adapter_dir`.
2. `config.Load` reads `config.yaml`; `config.NewStore(runtime_file)` reads
WebUI-edited runtime sources; both are merged by name.
3. `rebuildRegistry` creates a `provider.Provider` per source (holding its
adapter), sizing the adapter's worker pool from the source's concurrency
limit.
4. Incoming requests are routed by `Registry` to a provider, which calls the
adapter's `transform_request` → HTTP call → `transform_response` /
`transform_stream_chunk`.
"Add/Edit source", "Upload adapter", "AUTO chain edits" and "per-key model
scopes" in the WebUI take effect immediately (written to the runtime file /
`adapter_dir`, then reassembled — no restart). Directly editing `config.yaml`
or files under `adapter_dir` requires a process restart.
Deleting a source/adapter in the WebUI truly removes runtime (UI-created)
sources and `.lua` files; base sources defined in `config.yaml` cannot rewrite
that file, so they are hidden via a deletion tombstone (still hidden after
restart) and can be restored by re-adding the same name in the UI.
### Source templates (multi-key balancing)
When several upstream keys share one config (URL / adapter / model list /
concurrency / rpm), copying the whole source block N times is wasteful. A
**source template** stores every source field except `name` and `api_key`, so you
create N key-only-different sources from one shared skeleton — natural multi-key
load balancing (each expanded source schedules, cools down, and reports health
independently).
- WebUI Sources page, top-right **Templates** button: list all templates with
edit / delete / create.
- Add-source dialog has two header buttons:
- **From template** → pick a template → form auto-fills (name + key still
yours to enter);
- **As template** → save the current form's non-key/name fields as a template.
- Templates persist in the runtime file's `source_templates` field alongside
runtime sources (hot-managed, apply immediately). Templates hold no secret
keys, so they're safe to share / version-control.
A template is just a recipe — it doesn't become a source by itself. Only the
key-bearing sources created from it carry real traffic.
### Memory footprint (measured and tuned)
Memory is not a constant: it scales with the **number of configured sources**
(each source owns an `http.Transport` with its connection pool plus per-model
health state), not with uptime. Measured on this machine (Linux x86_64, 12 cores):
| Deployment shape | Startup RSS | Settled RSS |
|---|---|---|
| 1 source / 1 adapter (minimal) | ~4 MB | ~10 MB |
| 1 source + a 29 MB audit history | ~19 MB | ~20 MB |
| **16 sources / 13 adapters / 59 models (this host)** | ~28 MB | **~32-35 MB** |
> For reference, the same production config used **~105 MB** before this round of
> work. The reduction comes from three places: raw audit records are no longer kept
> resident (~25 MB — the aggregates still scan the full history, but the scan
> releases as it goes), Lua state pools no longer grow monotonically, and the two
> runtime knobs below.
Breakdown of the production instance (per-region, from `/proc/<pid>/smaps`):
| Region | RSS | Notes |
|---|---|---|
| Go heap | ~14 MB | provider/registry/scheduler state + connection-pool buffers |
| Other anonymous (thread stacks / LuaJIT chunks / runtime) | ~12 MB | scales with thread count and loaded adapters |
| Binary text+rodata | ~8 MB | mapped executable pages (read-only, reclaimable by the kernel) |
| Go runtime reservations | ~4 MB | the 2 GB+ you see in VSZ is address space, not physical memory |
| Shared libraries | ~3 MB | libc / libluajit / libm |
**Two recommended deployment knobs** (environment only, no code change):
```ini
# /etc/systemd/system/llmsproxy.service
Environment=GOGC=50
Environment=MALLOC_ARENA_MAX=2
```
- `MALLOC_ARENA_MAX=2`: LuaJIT allocates through cgo into glibc malloc, and glibc
allows up to `8 x nproc` per-thread arenas. Every OS thread that touches malloc
claims one (~1 MB each) and **never returns it to the OS**. Measured: 8-15
arenas (7-12 MB) down to **0**.
- `GOGC=50`: halves the Go heap growth target. **It does nothing on its own** —
the heap it saves is immediately consumed by additional glibc arenas (measured
20.3 -> 21.5 MB, i.e. slightly worse) — so it must be paired with
`MALLOC_ARENA_MAX`. Together they cut settled RSS by **~19%**. The gateway is
I/O bound (1min10s of CPU per 9 hours here), so the extra GC cycles are free.
> The desktop build (Electron) already injects both when spawning the embedded
> core (`cmd/gui/main.js`); exporting either variable yourself overrides it. If you
> write your own systemd unit or container spec, add them there.
Optional: `GOMEMLIMIT=48MiB` as a soft ceiling saves roughly 1 MB more, at the
cost of GC turning aggressive as the limit approaches. Not worth it for a relaxed
deployment.
### disable_thinking
With `"disable_thinking": true` in the request body, the gateway passes it to
each adapter; the DeepSeek adapter maps it to `extra_body.thinking.type =
"disabled"`, other sources follow their own protocol.
### WebUI
Built-in admin page at `GET /`; after login you can:
- **Status**: source online state (live probe, error on hover), per
model/source/key usage stats, request records filtered by key, and CSV export
over a time range; click a model to pin the connection snippet to it.
- **Chat**: streaming / non-streaming debug.
- **Keys**: create/edit gateway keys, set per-key model scopes (model + source +
token quota + reset period); admins manage all keys, users see only their own.
- **Priority**: drag blocks to build the tiered AUTO chain.
- **Sources**: add/edit/delete upstream sources online (secrets stored
encrypted).
- **Adapters**: upload / delete Lua adapter scripts.
Changes persist to `runtime_file` (survive restarts).
## Lua adapter protocol
Full API: **[Lua adapter docs](docs/lua-adapters.md)**
([中文](docs/lua-adapters-en.md)).
Each adapter is a Lua script returning a table
(`internal/lua/adapters/<name>.lua`), overridable by a same-name file under
`adapter_dir` — **no recompile needed**, restart to take effect; uploads via
the WebUI take effect immediately.
```lua
return {
name = "mysrc",
version = "1.0.0",
endpoint = "/chat/completions", -- upstream path (source.endpoint overrides)
headers = { ["X-Static"] = "v" }, -- static default headers (fallback)
-- request: convert unified OpenAI req -> upstream native format, return string
transform_request = function(raw_json) ... end,
-- response: convert upstream response to unified format string
-- { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
transform_response = function(raw_json) ... end,
-- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips
transform_stream_chunk = function(raw_chunk) ... end,
-- [optional] dynamic headers / signature hook
-- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
}
```
Shared helpers: `hmac_sha256_hex(key, data)`, `sha256_hex(data)`,
`base64_encode(s)`, `tohex(s)`, `json.encode/decode`, `log(level, msg)`.
### Built-in adapters
`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama`
`kimicode` `qwen38` `sensenova` `trae`.
`anthropic`/`gemini`/`ollama` include multimodal conversion
(`image_url` → their native format); with `disable_thinking` the `deepseek`
adapter sets `extra_body.thinking.type` to `disabled`.
**kimicode** demonstrates `build_headers`: the cloud validates the calling
app, so you HMAC-sign timestamp+URL+body with `meta.app_secret` and add
`X-App-Sign`-style headers. Configure `sources[].meta.{app_id, app_secret,
app_agent}`.
## Desktop GUI (Electron)
Optional standalone product for desktop users on your internal network: an
**embedded ModelRouter core** (Clash Verge-style): one-click start/stop, system
tray, autostart, silent launch, and a pixel-for-pixel embedded full WebUI
(Status/Chat/Keys/Priority/Sources/Adapters — no login needed). Server users
keep using the plain Go binary.
### Two installer kinds: Headless and Desktop
Release installers ship in two flavors for different audiences:
- **Headless (server)**: the plain core binary (no Electron), config-driven,
for servers / containers / systemd / unattended runs. Ships as `deb` + `rpm`
(systemd unit with memory tuning + example config) and `tar.gz` (binary +
`config.example.yaml` + `adapters/` + README), Windows as `.zip`.
- **Desktop (desktop app)**: the Electron GUI installer with an embedded core,
for personal daily desktop use.
### GUI vs. plain backend
| Scenario | Pick | Why |
| ---- | ---- | ---- |
| Server / intranet gateway / unattended long-running | **Headless** (single binary) | ~8 MB binary; RSS scales with the number of sources (~10 MB for one, ~32-35 MB for 16 — see [Memory footprint](#memory-footprint-measured-and-tuned)), zero-dep single process — drop it into systemd or any container, remote admin |
| Personal desktop daily use / multi-device intranet sharing | **Desktop** (GUI) | no-login embedded WebUI, tray one-click, autostart, silent background — for non-CLI users |
| Windows desktop | **Desktop** | plain backend needs manual service registration; GUI ships native tray/autostart |
| CI one-shot 3-platform installers | **Desktop packaging scripts** | `make gui-*` emits deb / AppImage / NSIS, drops straight into a release pipeline |
Both share the exact same `llmsproxy` core (LuaJIT build) — configs and
adapters are fully compatible, freely interchangeable.
### Build & run
Any OpenAI SDK works: point `base_url` at the gateway and use an authorized gateway
key as `api_key`.
## Documentation
The detailed documentation is currently written in Chinese.
| Topic | Document |
|---|---|
| Implementation and the full feature list | [docs/features.md](docs/features.md) |
| Configuration: `config.yaml`, model discovery, multiple keys | [docs/configuration.md](docs/configuration.md) |
| Routing: model routing, AUTO chain, cooldown, sources vs adapters, source templates | [docs/routing.md](docs/routing.md) |
| Adapters: Lua protocol and built-in adapters | [docs/adapters.md](docs/adapters.md) · [docs/lua-adapters-en.md](docs/lua-adapters-en.md) (English) |
| Plugins: SDK contract, hooks, UI extensions, HTTP endpoints | [docs/plugins.md](docs/plugins.md) |
| Desktop app: Headless / Desktop artifacts, build and run | [docs/gui.md](docs/gui.md) |
| Performance: measured memory use, tuning, thinking budget | [docs/performance.md](docs/performance.md) |
| WebUI usage | [docs/webui.md](docs/webui.md) |
| Development: layout, tests, branch and release discipline | [docs/development.md](docs/development.md) · [docs/git-workflow-en.md](docs/git-workflow-en.md) (English) |
| Deployment and migration | [docs/deployment.md](docs/deployment.md) |
## Renamed from llmsproxy
The product name has been ModelRouter for a while; the binary, config directory,
service name and environment variables are now `modelrouter` as well:
| Before | After |
|---|---|
| `llmsproxy` (binary) | `modelrouter` |
| `/etc/llmsproxy/` | `/etc/modelrouter/` |
| `llmsproxy.service` | `modelrouter.service` |
| `LLMSPROXY_*` | `MODELROUTER_*` |
Existing installs migrate with a dry-run-by-default script (copies, never moves,
and can roll back):
```bash
cd cmd/gui
npm install
make gui # or npm run dev — dev run (needs make build first for bin/llmsproxy)
make gui-deb # local deb (share with other Linux users)
make gui-dist # deb + rpm + AppImage (linux; rpm needs system rpmbuild)
make core-dist # core deb + rpm + tar.gz (server edition; needs nfpm)
make gui-win # Windows NSIS (host mingw + wine)
make gui-win-docker # Windows NSIS, fully dockerized (wine in image; no host mingw/node/wine)
sudo ./packaging/migrate-to-modelrouter.sh # preview
sudo ./packaging/migrate-to-modelrouter.sh --apply # apply
sudo ./packaging/migrate-to-modelrouter.sh --rollback --apply
```
Artifacts land in `cmd/build/gui-dist/`: `ModelRouter-<ver>.AppImage`,
`modelrouter-gui_<ver>_amd64.deb`, `ModelRouter Setup <ver>.exe` (win); core
packages land in `cmd/build/dist/`: `llmsproxy_<ver>_amd64.deb`,
`llmsproxy-<ver>.x86_64.rpm`. Both target families run
`packaging/verify-dist.sh` afterwards — a size floor that rejects degenerate
installers (1.3.0 shipped a 264 KB NSIS stub because the host wine was broken
and nothing caught it). At release time artifacts are renamed into the
`ModelRouter-Desktop-*` (GUI) and `ModelRouter-Headless-*` (plain backend)
families before upload to GitCode Releases.
Old paths and old environment variable names are still accepted on the read side,
so an upgrade cannot fail merely because of ordering. `deploy.sh` refuses to run
when it finds the old `/etc/llmsproxy` without a new config, because that would
deploy a gateway with no key and no sources.
#### Windows packaging (dockerized, recommended)
`make gui-win-docker` does cross-compile + NSIS **entirely in docker**; the host
needs **no mingw, no wine, no node**. Wine (with the i386 runtime) lives inside
the image, fixing the broken-host-wine failure that produced the 264 KB stub
installer.
- Image: `cmd/gui/docker/win-builder/Dockerfile` (`golang:1.25` + mingw-w64 +
node + wine32/wine64 + prebuilt LuaJIT for Windows: `lua51.dll` + import lib).
LuaJIT source comes from the gitcode mirror (reachable in CN) with github
fallback; Go modules go through `https://goproxy.cn`.
- One-shot release: `cmd/gui/scripts/dist-win-docker.sh` — core build + NSIS
packaging run inside the container (`USE_SYSTEM_WINE=true` uses the image's
wine); electron-builder's ~400 MB toolchain cache lives in a docker volume so
later builds reuse it.
- The `modelrouter/win-builder` image is auto-built on first run (~5-10 min)
and reused afterwards.
- After packaging, `packaging/verify-dist.sh` runs as a gate: an exe under
5 MB (or a deb/rpm/7z under 10 MB) fails the build — a broken installer can
never reach a Release again.
```bash
# Full Windows release (rebuild core + NSIS):
make gui-win-docker
```
### Features
- **Embedded core**: auto-spawns the bundled `llmsproxy` (LuaJIT build);
config/keys/adapters live in `<userData>/profile/`. First run generates a
random admin key written into `keys` (not seed) and self-injects it — WebUI
needs no login and never nags about changing the initial key. Port changeable
in settings (default 8787).
- **System tray**: core status dot, autostart/silent toggles, start/stop/restart
core, quit.
- **Silent launch**: `--silent` flag or setting — starts to the tray only;
first run always shows the window.
- **Autostart**: `setLoginItemSettings` on Windows/macOS; writes
`~/.config/autostart/modelrouter-gui.desktop` on Linux, appending `--silent`
when silent mode is on.
The plugin SDK follows the same naming: the only public entry point is
`modelrouter` (`llmsproxy` remains a compatibility alias). See
[docs/plugins.md](docs/plugins.md).
## Layout
```
cmd/llmsproxy # entry point
internal/config # YAML config loading/validation
internal/lua # LuaJIT VM + worker pool + AdapterCache + built-ins (embed)
```text
cmd/modelrouter # entry point (pure Go server)
cmd/gui # Electron desktop shell (embeds the same core)
internal/config # config parsing, sealed storage, runtime file
internal/core # scheduling, kernel actions, state and audit
internal/lua # LuaJIT VM + elastic worker pool + built-in adapters (embed)
internal/lua/plugins # optional plugin examples (not shipped in binary/archives)
internal/provider # Provider(HTTP) + Registry(routing)
internal/gateway # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI
internal/types # unified format & OpenAI wire types
internal/gateway # OpenAI-compatible HTTP service + auth + streaming + WebUI
internal/types # unified format and OpenAI wire types
packaging/ # deb/rpm/tar packaging, systemd unit, migration script
docs/ # topic documentation
```
## Development & collaboration
- **Git branching workflow**: main / feature / release-branch conventions in
**[docs/git-workflow-en.md](docs/git-workflow-en.md)**
([中文](docs/git-workflow.md)). The line: `main` is always deployable →
features merge into `main` → cut `release/vX.Y.Z` and tag → release → hotfixes
land on the release branch and are cherry-picked back into `main`.
- **Lua adapter protocol**: see
**[docs/lua-adapters-en.md](docs/lua-adapters-en.md)**
([中文](docs/lua-adapters.md)).
## Tests
## Development and tests
```bash
go test -tags luajit ./...
go test -tags luajit ./... # everything
go test -tags luajit ./internal/lua # one package (required for LuaJIT changes)
```
Covers: config validation, adapter load/transform, signature hooks, gateway
auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and
disable_thinking.
- Branch and release discipline: [docs/git-workflow-en.md](docs/git-workflow-en.md).
- Plugins live in `plugin_dir`. The gateway never creates, overwrites or deletes
plugin files, and a desktop upgrade never touches them either.