ModelRouter/README_EN.md

# ModelRouter

> **中文**: [README.md](./README.md)

A lightweight, unified OpenAI-compatible LLM gateway for internal networks —
configure once, and let multiple services (agents, SDKs, bots) share many
upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama,
KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mode,
which routes to the best healthy upstream by configured priority.

> **Lightweight, no recompile to add sources**: the gateway is a single Go
> binary (~10 MB, zero runtime dependencies). Adding or switching an upstream is
> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua`
> adapter — **no Go changes, no recompile**. Changes submitted through the
> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua`
> file under `adapter_dir` by hand requires a process restart.

## Overview

Exposes a standard `OpenAI Chat Completions` API (`/v1/chat/completions` +
`/v1/models`) to your internal network, translating between multiple upstream
protocols via **Lua adapters**. Supports one-shot calls, SSE streaming,
**AUTO model routing**, multimodal passthrough, and tool calls.

Extracted and independently evolved from the multi-source LLM adapter layer of
[HomeAgent](https://gitcode.com/JianFeeeee/HomeAgent)
(`internal/agent/api/provider.go` + `internal/lua/adapters/*`).

## Features

- **Multi-source**: any number of upstream sources in one process, routed by the
  request's `model`.
- **AUTO mode**: with `default_model: AUTO`, the highest-priority available
  source model wins.
- **Unified output**: every source speaks OpenAI format (incl.
  `reasoning_content`, `tool_calls`, `usage`).
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
  Anthropic/Gemini/Ollama are translated automatically.
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
  pool for safe concurrency.
- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request.
- **Lua adapter protocol**: each source mounts a `.lua` adapter with
  `transform_request` / `transform_response` / `transform_stream_chunk` — all
  protocol differences live in Lua, Go only schedules and proxies.
- **Signature / header hooks**: adapters may define `build_headers(meta)` to
  inject/sign request headers before the HTTP call (e.g. KimiCode style
  app-validation), with helpers like `hmac_sha256_hex`, `sha256_hex`,
  `base64_encode`.
- **WebUI**: built-in management page to view/add/edit sources & models,
  persisted to a runtime file.
- **Auth**: the gateway validates client keys via `gateway_keys`; independent
  from each upstream's own key.
- **Streaming**: SSE `chat.completion.chunk` with a role-first chunk and
  `[DONE]` terminator.

## Quick start

```bash
cp config.example.yaml config.yaml   # edit your sources & keys
GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
./llmsproxy -config config.yaml
```

> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings).
> You **must** build with `-tags luajit`; otherwise the built-in superset
> gopher-lua path is used (behavior differs slightly).

```bash
# no key -> 401
curl http://127.0.0.1:8080/v1/models

# one-shot
curl -H "Authorization: Bearer sk-gw-local-0001" \
     -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

# streaming
curl -N -H "Authorization: Bearer sk-gw-local-0001" \
     -d '{"model":"deepseek-v4-flash","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions
```

Any OpenAI SDK works: point `base_url` at the gateway, use one of `gateway_keys`
as the API key.

## Configuration

See [`config.example.yaml`](config.example.yaml). Core fields:

```yaml
listen: 127.0.0.1:8080        # bind address (keep internal/loopback)
gateway_keys: [sk-gw-0001]    # keys clients use; empty = no auth
default_model: AUTO           # when model is unroutable, pick source by priority
adapter_dir: adapters         # Lua adapter dir; built-ins written on first start
runtime_file: runtime.json    # WebUI-edited sources persist here

sources:
  - name: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-...
    adapter: deepseek
    max_concurrent: 8
    models:
      - id: deepseek-v4-flash
        priority: 100          # higher -> preferred by AUTO
        kind: chat
      - id: deepseek-v4-pro
        priority: 60
        kind: chat
    # static headers (take precedence over adapter defaults)
    headers: { X-Tenant: prod }
    # passthrough metadata for the Lua build_headers hook
    meta: { app_id: x, app_secret: y }
    temperature: 0.7
    max_tokens: 4096
    timeout: 120s              # request timeout, default 120s
```

### Model routing

`/v1/chat/completions` `model` resolution order:

1. `source/model` or `source:model` prefix → pinned source;
2. exact match of a source's `model`;
3. with `default_model: AUTO` → highest-`priority` healthy source model;
4. otherwise fall back to `default_source`.

Any OpenAI client can pin to a source by setting `model` to its
`name/anything`; `AUTO` (or the gateway's `default_model: AUTO`) picks the
healthy source by priority.

### Sources vs. adapters

- A **source** describes a connection to an upstream: `name`, `base_url`,
  `api_key`, model list and priorities.
- An **adapter** is the protocol translation logic (Lua script): converts the
  unified OpenAI request to the upstream's native format and back.
- One adapter serves many sources (e.g. `openai.lua` for any OpenAI-compatible
  site); one source can switch adapters via its `adapter` field.
- Sources decide *whom to talk to and which models to expose*; adapters decide
  *how to talk*. They are linked by the `adapter` field inside each `sources[]`
  entry.

Loading flow (assembled by `Core` in `internal/core`):

1. On startup `lua.NewVM(adapter_dir)` loads all adapters: built-in ones
   (embedded at compile time) + same-name override files under `adapter_dir`.
2. `config.Load` reads `config.yaml`; `config.NewStore(runtime_file)` reads
   WebUI-edited runtime sources; both are merged by name.
3. `rebuildRegistry` creates a `provider.Provider` per source (holding its
   adapter), sizing the adapter's worker pool from the source's concurrency
   limit.
4. Incoming requests are routed by `Registry` to a provider, which calls the
   adapter's `transform_request` → HTTP call → `transform_response` /
   `transform_stream_chunk`.

"Add/Edit source" and "Upload adapter" in the WebUI take effect immediately
(written to the runtime file / `adapter_dir`, then reassembled — no restart).
Directly editing `config.yaml` or files under `adapter_dir` requires a process
restart.

### disable_thinking

With `"disable_thinking": true` in the request body, the gateway passes it to
each adapter; the DeepSeek adapter maps it to `extra_body.thinking.type =
"disabled"`, other sources follow their own protocol.

### WebUI

Built-in admin page at `GET /`; after login you can view/edit sources and
models in the browser, persisted to `runtime_file` (survives restarts).

## Lua adapter protocol

Full API: **[Lua adapter docs](docs/lua-adapters.md)**
([中文](docs/lua-adapters-en.md)).

Each adapter is a Lua script returning a table
(`internal/lua/adapters/<name>.lua`), overridable by a same-name file under
`adapter_dir` — **no recompile needed**, restart to take effect; uploads via
the WebUI take effect immediately.

```lua
return {
  name = "mysrc",
  version = "1.0.0",
  endpoint = "/chat/completions",        -- upstream path (source.endpoint overrides)
  headers = { ["X-Static"] = "v" },      -- static default headers (fallback)

  -- request: convert unified OpenAI req -> upstream native format, return string
  transform_request = function(raw_json) ... end,

  -- response: convert upstream response to unified format string
  --   { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
  transform_response = function(raw_json) ... end,

  -- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips
  transform_stream_chunk = function(raw_chunk) ... end,

  -- [optional] dynamic headers / signature hook
  -- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
  build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
}
```

Shared helpers: `hmac_sha256_hex(key, data)`, `sha256_hex(data)`,
`base64_encode(s)`, `tohex(s)`, `json.encode/decode`, `log(level, msg)`.

### Built-in adapters

`openai` `deepseek` `anthropic` `gemini` `github` `groq` `mistral` `ollama`
`kimicode`.

`anthropic`/`gemini`/`ollama` include multimodal conversion
(`image_url` → their native format); with `disable_thinking` the `deepseek`
adapter sets `extra_body.thinking.type` to `disabled`.

**kimicode** demonstrates `build_headers`: the cloud validates the calling
app, so you HMAC-sign timestamp+URL+body with `meta.app_secret` and add
`X-App-Sign`-style headers. Configure `sources[].meta.{app_id, app_secret,
app_agent}`.

## Layout

```
cmd/llmsproxy            # entry point
internal/config          # YAML config loading/validation
internal/lua             # LuaJIT VM + worker pool + AdapterCache + built-ins (embed)
internal/provider        # Provider(HTTP) + Registry(routing)
internal/gateway         # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI
internal/types           # unified format & OpenAI wire types
```

## Tests

```bash
go test -tags luajit ./...
```

Covers: config validation, adapter load/transform, signature hooks, gateway
auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and
disable_thinking.