ModelRouter/README_EN.md

# ModelRouter

> **中文**: [README.md](./README.md)
>
> **AI-assisted**: This project is developed with AI-assisted coding (code and docs are produced in collaboration with AI, reviewed by humans).

A lightweight, unified OpenAI-compatible LLM gateway for internal networks —
configure once, and let multiple services (agents, SDKs, bots) share many
upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama,
KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mode,
which routes to the best healthy upstream by configured priority.

> **Lightweight, no recompile to add sources**: the gateway is a single Go
> binary (8-12 MB, ~8 MB stripped; zero runtime dependencies). Adding or switching an upstream is
> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua`
> adapter — **no Go changes, no recompile**. Changes submitted through the
> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua`
> file under `adapter_dir` by hand requires a process restart.

## Overview

Exposes a standard `OpenAI Chat Completions` API (`/v1/chat/completions` +
`/v1/models`) to your internal network, translating between multiple upstream
protocols via **Lua adapters**. Supports one-shot calls, SSE streaming,
**AUTO model routing**, multimodal passthrough, and tool calls.

Extracted and independently evolved from the multi-source LLM adapter layer of
[HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent)
(`internal/agent/api/provider.go` + `internal/lua/adapters/*`).

## Features

- **Multi-source**: any number of upstream sources in one process, routed by the
  request's `model`.
- **AUTO mode**: with `default_model: AUTO`, scheduling follows the tiered AUTO
  chain saved on the Priority page (see below).
- **Unified output**: every source speaks OpenAI format (incl.
  `reasoning_content`, `tool_calls`, `usage`).
- **Image generation**: `POST /v1/images/generations`, routed to models with
  `kind: image`.
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
  Anthropic/Gemini/Ollama are translated automatically.
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
  pool for safe concurrency. Pools are **elastic**: the sum of `max_concurrent`
  over an adapter's sources is a ceiling, not a preallocation — states are booted
  on demand and reclaimed when demand drops, so an idle gateway holds close to
  zero Lua states. The grow step follows the adapter's max concurrency
  (`clamp(ceil(max/8), 1, 8)`, and never exceeds the number of queued callers);
  the shrink step follows the live connection count
  (`clamp(ceil(slack/(1+in_use)), 1, slack)`), so an idle pool collapses in one
  round while a busy one gives up a single state at a time.
- **On-demand request logs**: dashboard totals are computed from the **full**
  audit history (streamed once at startup and released — 29 MB / 221k lines in
  ~260 ms), while the raw records are never held in memory: the dashboard loads
  one screen, scrolling pages the rest straight off disk, CSV export streams in
  O(1) memory, and leaving the page releases everything.
- **Self-healing cooldown**: cooldown is capped at 5 minutes and, past the
  window's midpoint, exactly one probe request is allowed through; a recovered
  upstream (or a reset quota) returns to full rotation on that probe instead of
  waiting out the window.
- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request.
- **Lua adapter protocol**: each source mounts a `.lua` adapter with
  `transform_request` / `transform_response` / `transform_stream_chunk` — all
  protocol differences live in Lua, Go only schedules and proxies.
- **Signature / header hooks**: adapters may define `build_headers(meta)` to
  inject/sign request headers before the HTTP call (e.g. KimiCode style
  app-validation), with helpers like `hmac_sha256_hex`, `sha256_hex`,
  `base64_encode`.
- **WebUI**: built-in management page to view/add/edit sources & models, edit
  the AUTO chain, and manage per-key model scopes, persisted to a runtime file.
- **Encrypted secrets at rest**: upstream `api_key`, custom header values and
  gateway keys are stored AES-256-GCM encrypted (`master.key` 0600 next to the
  runtime file, or `LLMS_PROXY_MASTER_KEY`).
- **Live source probing**: the status page probes each source
  (`GET {base}/models`, falling back to a minimal chat call) without touching
  the scheduler's backoff state; errors are shown on hover.
- **Auth**: the gateway validates client keys via `gateway_keys`; independent
  from each upstream's own key.
- **Streaming**: SSE `chat.completion.chunk` with a role-first chunk and
  `[DONE]` terminator.

## Quick start

```bash
GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
./llmsproxy -config config.yaml    # first run generates a default config and prints a random admin key
```

> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings).
> You **must** build with `-tags luajit`; otherwise the built-in superset
> gopher-lua path is used (behavior differs slightly).

> **No config is shipped in the repo** (config files carry real keys). On
> first run the binary generates a default config at the `-config` path: a
> random admin key (printed to the startup log), loopback-only `127.0.0.1:8080`,
> and a keyless source ready to chat. Rotate the admin key in the
> WebUI after first login.

```bash
# no key -> 401
curl http://127.0.0.1:8080/v1/models

# one-shot ($KEY = the admin key printed at first startup)
curl -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

# streaming
curl -N -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions
```

Any OpenAI SDK works: point `base_url` at the gateway, use any authorized
gateway key as the API key.

## Configuration

Generated on first run at the `-config` path (default `./config.yaml`). Core fields:

```yaml
listen: 127.0.0.1:8080        # bind address (keep internal/loopback)
gateway_keys: [sk-gw-<random>] # initial admin key seed (random per install), migrated to the store on first start
default_model: AUTO           # when model is unroutable, follow the AUTO chain
adapter_dir: adapters         # Lua adapter dir; created+seeded if missing, read-only otherwise
runtime_file: runtime.json    # WebUI-edited sources/keys/AUTO chain persist here

sources:
  - name: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-...            # or api_key_env: SOME_ENV to read from an env var
    adapter: deepseek
    max_concurrent: 8
    models:
      - id: deepseek-v4-flash
        priority: 100          # YAML sources seed the AUTO chain on first start
        kind: chat
      - id: deepseek-v4-pro
        priority: 60
        kind: chat
    # static headers (take precedence over adapter defaults)
    headers: { X-Tenant: prod }
    # passthrough metadata for the Lua build_headers hook
    meta: { app_id: x, app_secret: y }
    temperature: 0.7
    max_tokens: 4096
    timeout: 120s              # request timeout, default 120s
```

Sensitive fields in the runtime file (`runtime_file`) are encrypted at rest:

- AES-256-GCM, stored as `enc:v1:<base64>`.
- Master key source: env var `LLMS_PROXY_MASTER_KEY` (64 hex chars), else a
  `master.key` file next to the runtime file; if neither exists it is generated
  on first start (mode 0600).
- Old plaintext files still load; the first UI save migrates the whole file to
  ciphertext.
- Back up `master.key` with your config — losing it makes the secrets
  undecryptable. Do not commit it.

### Gateway keys (multi-key)

Gateway auth uses a "multi-key + role + model scope" model. Keys are persisted
under the `keys` field of the runtime file (encrypted at rest):

- The `gateway_keys` config is only an **initial admin key seed** — it is
  migrated into the runtime store on first start and no longer drives auth.
- **Important: after first start, replace the admin key via the WebUI Keys
  page.** The seed key is written in plaintext in `config.yaml`, so keeping it
  active is a security risk; create a new admin key, log in with it, then
  delete the seed key.
- The WebUI **Keys page** creates/deletes keys. Each key has a role (`admin`
  manages everything, `user` sees only its own key) and an optional **model
  scope** (model + source + token quota + reset period).
- Clients authenticate with any authorized key's plaintext as
  `Authorization: Bearer <key>`.
- Deleting a key removes it from the store immediately.

### Model routing

`/v1/chat/completions` `model` resolution order:

1. `source/model` or `source:model` prefix → pinned source;
2. exact match of a source's `model`;
3. with `default_model: AUTO` → follow the AUTO chain from the Priority page
   (see below);
4. otherwise the request errors.

Any OpenAI client can pin to a source by setting `model` to its
`name/anything`; `AUTO` (or the gateway's `default_model: AUTO`) follows the
tiered AUTO chain.

### AUTO chain (Priority page)

AUTO scheduling is driven **only** by the rules saved on the Priority page. The
numeric `models[].priority` in source configs no longer participates in
scheduling and is no longer shown. **Chat and image generation are two
independent chains**, switched in the Priority page via the `Chat / Image` toggle:

- **Chat chain** (`auto` field): serves `POST /v1/chat/completions` with
  `model: AUTO`; only `kind: chat` models are honored (image slots are skipped).
- **Image chain** (`auto_image` field): serves `POST /v1/images/generations` with
  `model: AUTO`; only `kind: image` models are honored. With no image chain
  configured, AUTO image generation falls back to legacy discovery (all sources
  exposing an image model, in registry order) for backward compatibility.

Each rule is one "slot": `{ model, source, tier, token_quota, period, hours }`.

- `tier` is the priority tier: models in the same tier sit side by side and
  share it; tiers run high → low.
- The same model may appear in several slots (e.g. A low → B low → A high →
  B high) and is tried in tier order.
- `token_quota` > 0 means the slot is skipped once its tokens within the reset
  window are exhausted; `period` supports `hour` / `week` / `month` / `nhour`
  (with `hours`); empty = unlimited.
- Within a tier, slots run by preference score; cooling / quota-exhausted /
  hard-failed slots fall through to the next tier, and total failure returns
  503 with a per-tier summary.

### Sources vs. adapters

- A **source** describes a connection to an upstream: `name`, `base_url`,
  `api_key`, model list and priorities.
- An **adapter** is the protocol translation logic (Lua script): converts the
  unified OpenAI request to the upstream's native format and back.
- One adapter serves many sources (e.g. `openai.lua` for any OpenAI-compatible
  site); one source can switch adapters via its `adapter` field.
- Sources decide *whom to talk to and which models to expose*; adapters decide
  *how to talk*. They are linked by the `adapter` field inside each `sources[]`
  entry.

Loading flow (assembled by `Core` in `internal/core`):

1. On startup `lua.NewVM(adapter_dir)` loads all adapters: built-in ones
   (embedded at compile time) + same-name override files under `adapter_dir`.
2. `config.Load` reads `config.yaml`; `config.NewStore(runtime_file)` reads
   WebUI-edited runtime sources; both are merged by name.
3. `rebuildRegistry` creates a `provider.Provider` per source (holding its
   adapter), sizing the adapter's worker pool from the source's concurrency
   limit.
4. Incoming requests are routed by `Registry` to a provider, which calls the
   adapter's `transform_request` → HTTP call → `transform_response` /
   `transform_stream_chunk`.

"Add/Edit source", "Upload adapter", "AUTO chain edits" and "per-key model
scopes" in the WebUI take effect immediately (written to the runtime file /
`adapter_dir`, then reassembled — no restart). Directly editing `config.yaml`
or files under `adapter_dir` requires a process restart.

Deleting a source/adapter in the WebUI truly removes runtime (UI-created)
sources and `.lua` files; base sources defined in `config.yaml` cannot rewrite
that file, so they are hidden via a deletion tombstone (still hidden after
restart) and can be restored by re-adding the same name in the UI.

### Source templates (multi-key balancing)

When several upstream keys share one config (URL / adapter / model list /
concurrency / rpm), copying the whole source block N times is wasteful. A
**source template** stores every source field except `name` and `api_key`, so you
create N key-only-different sources from one shared skeleton — natural multi-key
load balancing (each expanded source schedules, cools down, and reports health
independently).

- WebUI Sources page, top-right **Templates** button: list all templates with
  edit / delete / create.
- Add-source dialog has two header buttons:
  - **From template** → pick a template → form auto-fills (name + key still
    yours to enter);
  - **As template** → save the current form's non-key/name fields as a template.
- Templates persist in the runtime file's `source_templates` field alongside
  runtime sources (hot-managed, apply immediately). Templates hold no secret
  keys, so they're safe to share / version-control.

A template is just a recipe — it doesn't become a source by itself. Only the
key-bearing sources created from it carry real traffic.

### Memory footprint (measured and tuned)

Memory is not a constant: it scales with the **number of configured sources**
(each source owns an `http.Transport` with its connection pool plus per-model
health state), not with uptime. Measured on this machine (Linux x86_64, 12 cores):

| Deployment shape | Startup RSS | Settled RSS |
|---|---|---|
| 1 source / 1 adapter (minimal) | ~4 MB | ~10 MB |
| 1 source + a 29 MB audit history | ~19 MB | ~20 MB |
| **16 sources / 13 adapters / 59 models (this host)** | ~28 MB | **~32-35 MB** |

> For reference, the same production config used **~105 MB** before this round of
> work. The reduction comes from three places: raw audit records are no longer kept
> resident (~25 MB — the aggregates still scan the full history, but the scan
> releases as it goes), Lua state pools no longer grow monotonically, and the two
> runtime knobs below.

Breakdown of the production instance (per-region, from `/proc/<pid>/smaps`):

| Region | RSS | Notes |
|---|---|---|
| Go heap | ~14 MB | provider/registry/scheduler state + connection-pool buffers |
| Other anonymous (thread stacks / LuaJIT chunks / runtime) | ~12 MB | scales with thread count and loaded adapters |
| Binary text+rodata | ~8 MB | mapped executable pages (read-only, reclaimable by the kernel) |
| Go runtime reservations | ~4 MB | the 2 GB+ you see in VSZ is address space, not physical memory |
| Shared libraries | ~3 MB | libc / libluajit / libm |

**Two recommended deployment knobs** (environment only, no code change):

```ini
# /etc/systemd/system/llmsproxy.service
Environment=GOGC=50
Environment=MALLOC_ARENA_MAX=2
```

- `MALLOC_ARENA_MAX=2`: LuaJIT allocates through cgo into glibc malloc, and glibc
  allows up to `8 x nproc` per-thread arenas. Every OS thread that touches malloc
  claims one (~1 MB each) and **never returns it to the OS**. Measured: 8-15
  arenas (7-12 MB) down to **0**.
- `GOGC=50`: halves the Go heap growth target. **It does nothing on its own** —
  the heap it saves is immediately consumed by additional glibc arenas (measured
  20.3 -> 21.5 MB, i.e. slightly worse) — so it must be paired with
  `MALLOC_ARENA_MAX`. Together they cut settled RSS by **~19%**. The gateway is
  I/O bound (1min10s of CPU per 9 hours here), so the extra GC cycles are free.

> The desktop build (Electron) already injects both when spawning the embedded
> core (`cmd/gui/main.js`); exporting either variable yourself overrides it. If you
> write your own systemd unit or container spec, add them there.

Optional: `GOMEMLIMIT=48MiB` as a soft ceiling saves roughly 1 MB more, at the
cost of GC turning aggressive as the limit approaches. Not worth it for a relaxed
deployment.

### disable_thinking

With `"disable_thinking": true` in the request body, the gateway passes it to
each adapter; the DeepSeek adapter maps it to `extra_body.thinking.type =
"disabled"`, other sources follow their own protocol.

### WebUI

Built-in admin page at `GET /`; after login you can:

- **Status**: source online state (live probe, error on hover), per
  model/source/key usage stats, request records filtered by key, and CSV export
  over a time range; click a model to pin the connection snippet to it.
- **Chat**: streaming / non-streaming debug.
- **Keys**: create/edit gateway keys, set per-key model scopes (model + source +
  token quota + reset period); admins manage all keys, users see only their own.
- **Priority**: drag blocks to build the tiered AUTO chain.
- **Sources**: add/edit/delete upstream sources online (secrets stored
  encrypted).
- **Adapters**: upload / delete Lua adapter scripts.

Changes persist to `runtime_file` (survive restarts).

## Lua adapter protocol

Full API: **[Lua adapter docs](docs/lua-adapters.md)**
([中文](docs/lua-adapters-en.md)).

Each adapter is a Lua script returning a table
(`internal/lua/adapters/<name>.lua`), overridable by a same-name file under
`adapter_dir` — **no recompile needed**, restart to take effect; uploads via
the WebUI take effect immediately.

```lua
return {
  name = "mysrc",
  version = "1.0.0",
  endpoint = "/chat/completions",        -- upstream path (source.endpoint overrides)
  headers = { ["X-Static"] = "v" },      -- static default headers (fallback)

  -- request: convert unified OpenAI req -> upstream native format, return string
  transform_request = function(raw_json) ... end,

  -- response: convert upstream response to unified format string
  --   { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
  transform_response = function(raw_json) ... end,

  -- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips
  transform_stream_chunk = function(raw_chunk) ... end,

  -- [optional] dynamic headers / signature hook
  -- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
  build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
}
```

Shared helpers: `hmac_sha256_hex(key, data)`, `sha256_hex(data)`,
`base64_encode(s)`, `tohex(s)`, `json.encode/decode`, `log(level, msg)`.

### Built-in adapters

`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama`
`kimicode`.

`anthropic`/`gemini`/`ollama` include multimodal conversion
(`image_url` → their native format); with `disable_thinking` the `deepseek`
adapter sets `extra_body.thinking.type` to `disabled`.

**kimicode** demonstrates `build_headers`: the cloud validates the calling
app, so you HMAC-sign timestamp+URL+body with `meta.app_secret` and add
`X-App-Sign`-style headers. Configure `sources[].meta.{app_id, app_secret,
app_agent}`.

## Desktop GUI (Electron)

Optional standalone product for desktop users on your internal network: an
**embedded ModelRouter core** (Clash Verge-style): one-click start/stop, system
tray, autostart, silent launch, and a pixel-for-pixel embedded full WebUI
(Status/Chat/Keys/Priority/Sources/Adapters — no login needed). Server users
keep using the plain Go binary.

### Two installer kinds: Headless and Desktop

Release installers ship in two flavors for different audiences:

- **Headless (server)**: the plain core binary (no Electron), config-driven,
  for servers / containers / systemd / unattended runs. Ships as `deb` + `rpm`
  (systemd unit with memory tuning + example config) and `tar.gz` (binary +
  `config.example.yaml` + `adapters/` + README), Windows as `.zip`.
- **Desktop (desktop app)**: the Electron GUI installer with an embedded core,
  for personal daily desktop use.

### GUI vs. plain backend

| Scenario | Pick | Why |
| ---- | ---- | ---- |
| Server / intranet gateway / unattended long-running | **Headless** (single binary) | ~8 MB binary; RSS scales with the number of sources (~10 MB for one, ~32-35 MB for 16 — see [Memory footprint](#memory-footprint-measured-and-tuned)), zero-dep single process — drop it into systemd or any container, remote admin |
| Personal desktop daily use / multi-device intranet sharing | **Desktop** (GUI) | no-login embedded WebUI, tray one-click, autostart, silent background — for non-CLI users |
| Windows desktop | **Desktop** | plain backend needs manual service registration; GUI ships native tray/autostart |
| CI one-shot 3-platform installers | **Desktop packaging scripts** | `make gui-*` emits deb / AppImage / NSIS, drops straight into a release pipeline |

Both share the exact same `llmsproxy` core (LuaJIT build) — configs and
adapters are fully compatible, freely interchangeable.

### Build & run

```bash
cd cmd/gui
npm install
make gui            # or npm run dev — dev run (needs make build first for bin/llmsproxy)
make gui-deb        # local deb (share with other Linux users)
make gui-dist       # deb + rpm + AppImage (linux; rpm needs system rpmbuild)
make core-dist      # core deb + rpm + tar.gz (server edition; needs nfpm)
make gui-win        # Windows NSIS (host mingw + wine)
make gui-win-docker # Windows NSIS, fully dockerized (wine in image; no host mingw/node/wine)
```

Artifacts land in `cmd/build/gui-dist/`: `ModelRouter-<ver>.AppImage`,
`modelrouter-gui_<ver>_amd64.deb`, `ModelRouter Setup <ver>.exe` (win); core
packages land in `cmd/build/dist/`: `llmsproxy_<ver>_amd64.deb`,
`llmsproxy-<ver>.x86_64.rpm`. Both target families run
`packaging/verify-dist.sh` afterwards — a size floor that rejects degenerate
installers (1.3.0 shipped a 264 KB NSIS stub because the host wine was broken
and nothing caught it). At release time artifacts are renamed into the
`ModelRouter-Desktop-*` (GUI) and `ModelRouter-Headless-*` (plain backend)
families before upload to GitCode Releases.

#### Windows packaging (dockerized, recommended)

`make gui-win-docker` does cross-compile + NSIS **entirely in docker**; the host
needs **no mingw, no wine, no node**. Wine (with the i386 runtime) lives inside
 the image, fixing the broken-host-wine failure that produced the 264 KB stub
installer.

- Image: `cmd/gui/docker/win-builder/Dockerfile` (`golang:1.25` + mingw-w64 +
  node + wine32/wine64 + prebuilt LuaJIT for Windows: `lua51.dll` + import lib).
  LuaJIT source comes from the gitcode mirror (reachable in CN) with github
  fallback; Go modules go through `https://goproxy.cn`.
- One-shot release: `cmd/gui/scripts/dist-win-docker.sh` — core build + NSIS
  packaging run inside the container (`USE_SYSTEM_WINE=true` uses the image's
  wine); electron-builder's ~400 MB toolchain cache lives in a docker volume so
  later builds reuse it.
- The `modelrouter/win-builder` image is auto-built on first run (~5-10 min)
  and reused afterwards.
- After packaging, `packaging/verify-dist.sh` runs as a gate: an exe under
  5 MB (or a deb/rpm/7z under 10 MB) fails the build — a broken installer can
  never reach a Release again.

```bash
# Full Windows release (rebuild core + NSIS):
make gui-win-docker
```

### Features

- **Embedded core**: auto-spawns the bundled `llmsproxy` (LuaJIT build);
  config/keys/adapters live in `<userData>/profile/`. First run generates a
  random admin key written into `keys` (not seed) and self-injects it — WebUI
  needs no login and never nags about changing the initial key. Port changeable
  in settings (default 8787).
- **System tray**: core status dot, autostart/silent toggles, start/stop/restart
  core, quit.
- **Silent launch**: `--silent` flag or setting — starts to the tray only;
  first run always shows the window.
- **Autostart**: `setLoginItemSettings` on Windows/macOS; writes
  `~/.config/autostart/modelrouter-gui.desktop` on Linux, appending `--silent`
  when silent mode is on.

## Layout

```
cmd/llmsproxy            # entry point
internal/config          # YAML config loading/validation
internal/lua             # LuaJIT VM + worker pool + AdapterCache + built-ins (embed)
internal/provider        # Provider(HTTP) + Registry(routing)
internal/gateway         # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI
internal/types           # unified format & OpenAI wire types
```

## Development & collaboration

- **Git branching workflow**: main / feature / release-branch conventions in
  **[docs/git-workflow-en.md](docs/git-workflow-en.md)**
  ([中文](docs/git-workflow.md)). The line: `main` is always deployable →
  features merge into `main` → cut `release/vX.Y.Z` and tag → release → hotfixes
  land on the release branch and are cherry-picked back into `main`.
- **Lua adapter protocol**: see
  **[docs/lua-adapters-en.md](docs/lua-adapters-en.md)**
  ([中文](docs/lua-adapters.md)).

## Tests

```bash
go test -tags luajit ./...
```

Covers: config validation, adapter load/transform, signature hooks, gateway
auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and
disable_thinking.