ModelRouter/README_EN.md

# ModelRouter

> **中文**: [README.md](./README.md)

A lightweight, unified OpenAI-compatible LLM gateway for internal networks —
configure once, and let multiple services (agents, SDKs, bots) share many
upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama,
KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mode,
which routes to the best healthy upstream by configured priority.

> **Lightweight, no recompile to add sources**: the gateway is a single Go
> binary (~10 MB, zero runtime dependencies). Adding or switching an upstream is
> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua`
> adapter — **no Go changes, no recompile**. Changes submitted through the
> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua`
> file under `adapter_dir` by hand requires a process restart.

## Overview

Exposes a standard `OpenAI Chat Completions` API (`/v1/chat/completions` +
`/v1/models`) to your internal network, translating between multiple upstream
protocols via **Lua adapters**. Supports one-shot calls, SSE streaming,
**AUTO model routing**, multimodal passthrough, and tool calls.

Extracted and independently evolved from the multi-source LLM adapter layer of
[HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent)
(`internal/agent/api/provider.go` + `internal/lua/adapters/*`).

## Features

- **Multi-source**: any number of upstream sources in one process, routed by the
  request's `model`.
- **AUTO mode**: with `default_model: AUTO`, scheduling follows the tiered AUTO
  chain saved on the Priority page (see below).
- **Unified output**: every source speaks OpenAI format (incl.
  `reasoning_content`, `tool_calls`, `usage`).
- **Image generation**: `POST /v1/images/generations`, routed to models with
  `kind: image`.
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
  Anthropic/Gemini/Ollama are translated automatically.
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
  pool for safe concurrency.
- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request.
- **Lua adapter protocol**: each source mounts a `.lua` adapter with
  `transform_request` / `transform_response` / `transform_stream_chunk` — all
  protocol differences live in Lua, Go only schedules and proxies.
- **Signature / header hooks**: adapters may define `build_headers(meta)` to
  inject/sign request headers before the HTTP call (e.g. KimiCode style
  app-validation), with helpers like `hmac_sha256_hex`, `sha256_hex`,
  `base64_encode`.
- **WebUI**: built-in management page to view/add/edit sources & models, edit
  the AUTO chain, and manage per-key model scopes, persisted to a runtime file.
- **Encrypted secrets at rest**: upstream `api_key`, custom header values and
  gateway keys are stored AES-256-GCM encrypted (`master.key` 0600 next to the
  runtime file, or `LLMS_PROXY_MASTER_KEY`).
- **Live source probing**: the status page probes each source
  (`GET {base}/models`, falling back to a minimal chat call) without touching
  the scheduler's backoff state; errors are shown on hover.
- **Auth**: the gateway validates client keys via `gateway_keys`; independent
  from each upstream's own key.
- **Streaming**: SSE `chat.completion.chunk` with a role-first chunk and
  `[DONE]` terminator.

## Quick start

```bash
cp config.example.yaml config.yaml   # edit your sources & keys
GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
./llmsproxy -config config.yaml
```

> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings).
> You **must** build with `-tags luajit`; otherwise the built-in superset
> gopher-lua path is used (behavior differs slightly).

```bash
# no key -> 401
curl http://127.0.0.1:8080/v1/models

# one-shot
curl -H "Authorization: Bearer sk-gw-local-0001" \
     -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

# streaming
curl -N -H "Authorization: Bearer sk-gw-local-0001" \
     -d '{"model":"deepseek-v4-flash","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions
```

Any OpenAI SDK works: point `base_url` at the gateway, use any authorized
gateway key as the API key.

## Configuration

See [`config.example.yaml`](config.example.yaml). Core fields:

```yaml
listen: 127.0.0.1:8080        # bind address (keep internal/loopback)
gateway_keys: [sk-gw-0001]    # initial admin key seed, migrated to the store on first start
default_model: AUTO           # when model is unroutable, follow the AUTO chain
adapter_dir: adapters         # Lua adapter dir; created+seeded if missing, read-only otherwise
runtime_file: runtime.json    # WebUI-edited sources/keys/AUTO chain persist here

sources:
  - name: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-...            # or api_key_env: SOME_ENV to read from an env var
    adapter: deepseek
    max_concurrent: 8
    models:
      - id: deepseek-v4-flash
        priority: 100          # YAML sources seed the AUTO chain on first start
        kind: chat
      - id: deepseek-v4-pro
        priority: 60
        kind: chat
    # static headers (take precedence over adapter defaults)
    headers: { X-Tenant: prod }
    # passthrough metadata for the Lua build_headers hook
    meta: { app_id: x, app_secret: y }
    temperature: 0.7
    max_tokens: 4096
    timeout: 120s              # request timeout, default 120s
```

Sensitive fields in the runtime file (`runtime_file`) are encrypted at rest:

- AES-256-GCM, stored as `enc:v1:<base64>`.
- Master key source: env var `LLMS_PROXY_MASTER_KEY` (64 hex chars), else a
  `master.key` file next to the runtime file; if neither exists it is generated
  on first start (mode 0600).
- Old plaintext files still load; the first UI save migrates the whole file to
  ciphertext.
- Back up `master.key` with your config — losing it makes the secrets
  undecryptable. Do not commit it.

### Gateway keys (multi-key)

Gateway auth uses a "multi-key + role + model scope" model. Keys are persisted
under the `keys` field of the runtime file (encrypted at rest):

- The `gateway_keys` config is only an **initial admin key seed** — it is
  migrated into the runtime store on first start and no longer drives auth.
- **Important: after first start, replace the admin key via the WebUI Keys
  page.** The seed key is written in plaintext in `config.yaml`, so keeping it
  active is a security risk; create a new admin key, log in with it, then
  delete the seed key.
- The WebUI **Keys page** creates/deletes keys. Each key has a role (`admin`
  manages everything, `user` sees only its own key) and an optional **model
  scope** (model + source + token quota + reset period).
- Clients authenticate with any authorized key's plaintext as
  `Authorization: Bearer <key>`.
- Deleting a key removes it from the store immediately.

### Model routing

`/v1/chat/completions` `model` resolution order:

1. `source/model` or `source:model` prefix → pinned source;
2. exact match of a source's `model`;
3. with `default_model: AUTO` → follow the AUTO chain from the Priority page
   (see below);
4. otherwise the request errors.

Any OpenAI client can pin to a source by setting `model` to its
`name/anything`; `AUTO` (or the gateway's `default_model: AUTO`) follows the
tiered AUTO chain.

### AUTO chain (Priority page)

AUTO scheduling is driven **only** by the rules saved on the Priority page
(persisted as the `auto` field of the runtime file). The numeric
`models[].priority` in source configs no longer participates in scheduling and
is no longer shown.

- Each rule is one "slot": `{ model, source, tier, token_quota, period, hours }`.
- `tier` is the priority tier: models in the same tier sit side by side and
  share it; tiers run high → low.
- The same model may appear in several slots (e.g. A low → B low → A high →
  B high) and is tried in tier order.
- `token_quota` > 0 means the slot is skipped once its tokens within the reset
  window are exhausted; `period` supports `hour` / `week` / `month` / `nhour`
  (with `hours`); empty = unlimited.
- Image models (`kind: image`) are kept out of the chain and are served by the
  separate `POST /v1/images/generations` path.

### Sources vs. adapters

- A **source** describes a connection to an upstream: `name`, `base_url`,
  `api_key`, model list and priorities.
- An **adapter** is the protocol translation logic (Lua script): converts the
  unified OpenAI request to the upstream's native format and back.
- One adapter serves many sources (e.g. `openai.lua` for any OpenAI-compatible
  site); one source can switch adapters via its `adapter` field.
- Sources decide *whom to talk to and which models to expose*; adapters decide
  *how to talk*. They are linked by the `adapter` field inside each `sources[]`
  entry.

Loading flow (assembled by `Core` in `internal/core`):

1. On startup `lua.NewVM(adapter_dir)` loads all adapters: built-in ones
   (embedded at compile time) + same-name override files under `adapter_dir`.
2. `config.Load` reads `config.yaml`; `config.NewStore(runtime_file)` reads
   WebUI-edited runtime sources; both are merged by name.
3. `rebuildRegistry` creates a `provider.Provider` per source (holding its
   adapter), sizing the adapter's worker pool from the source's concurrency
   limit.
4. Incoming requests are routed by `Registry` to a provider, which calls the
   adapter's `transform_request` → HTTP call → `transform_response` /
   `transform_stream_chunk`.

"Add/Edit source", "Upload adapter", "AUTO chain edits" and "per-key model
scopes" in the WebUI take effect immediately (written to the runtime file /
`adapter_dir`, then reassembled — no restart). Directly editing `config.yaml`
or files under `adapter_dir` requires a process restart.

Deleting a source/adapter in the WebUI truly removes runtime (UI-created)
sources and `.lua` files; base sources defined in `config.yaml` cannot rewrite
that file, so they are hidden via a deletion tombstone (still hidden after
restart) and can be restored by re-adding the same name in the UI.

### disable_thinking

With `"disable_thinking": true` in the request body, the gateway passes it to
each adapter; the DeepSeek adapter maps it to `extra_body.thinking.type =
"disabled"`, other sources follow their own protocol.

### WebUI

Built-in admin page at `GET /`; after login you can:

- **Status**: source online state (live probe, error on hover), per
  model/source/key usage stats, request records filtered by key, and CSV export
  over a time range; click a model to pin the connection snippet to it.
- **Chat**: streaming / non-streaming debug.
- **Keys**: create/edit gateway keys, set per-key model scopes (model + source +
  token quota + reset period); admins manage all keys, users see only their own.
- **Priority**: drag blocks to build the tiered AUTO chain.
- **Sources**: add/edit/delete upstream sources online (secrets stored
  encrypted).
- **Adapters**: upload / delete Lua adapter scripts.

Changes persist to `runtime_file` (survive restarts).

## Lua adapter protocol

Full API: **[Lua adapter docs](docs/lua-adapters.md)**
([中文](docs/lua-adapters-en.md)).

Each adapter is a Lua script returning a table
(`internal/lua/adapters/<name>.lua`), overridable by a same-name file under
`adapter_dir` — **no recompile needed**, restart to take effect; uploads via
the WebUI take effect immediately.

```lua
return {
  name = "mysrc",
  version = "1.0.0",
  endpoint = "/chat/completions",        -- upstream path (source.endpoint overrides)
  headers = { ["X-Static"] = "v" },      -- static default headers (fallback)

  -- request: convert unified OpenAI req -> upstream native format, return string
  transform_request = function(raw_json) ... end,

  -- response: convert upstream response to unified format string
  --   { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
  transform_response = function(raw_json) ... end,

  -- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips
  transform_stream_chunk = function(raw_chunk) ... end,

  -- [optional] dynamic headers / signature hook
  -- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
  build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
}
```

Shared helpers: `hmac_sha256_hex(key, data)`, `sha256_hex(data)`,
`base64_encode(s)`, `tohex(s)`, `json.encode/decode`, `log(level, msg)`.

### Built-in adapters

`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama`
`kimicode`.

`anthropic`/`gemini`/`ollama` include multimodal conversion
(`image_url` → their native format); with `disable_thinking` the `deepseek`
adapter sets `extra_body.thinking.type` to `disabled`.

**kimicode** demonstrates `build_headers`: the cloud validates the calling
app, so you HMAC-sign timestamp+URL+body with `meta.app_secret` and add
`X-App-Sign`-style headers. Configure `sources[].meta.{app_id, app_secret,
app_agent}`.

## Layout

```
cmd/llmsproxy            # entry point
internal/config          # YAML config loading/validation
internal/lua             # LuaJIT VM + worker pool + AdapterCache + built-ins (embed)
internal/provider        # Provider(HTTP) + Registry(routing)
internal/gateway         # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI
internal/types           # unified format & OpenAI wire types
```

## Tests

```bash
go test -tags luajit ./...
```

Covers: config validation, adapter load/transform, signature hooks, gateway
auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and
disable_thinking.