ModelRouter

中文: README.md

AI-assisted: This project is developed with AI-assisted coding (code and docs are produced in collaboration with AI, reviewed by humans).

A lightweight, unified OpenAI-compatible LLM gateway for internal networks — configure once, and let multiple services (agents, SDKs, bots) share many upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama, KimiCode…) behind a single endpoint. Pick a specific model or use AUTO mode, which routes to the best healthy upstream by configured priority.

Lightweight, no recompile to add sources: the gateway is a single Go binary (8-12 MB, ~8 MB stripped; zero runtime dependencies). Adding or switching an upstream is just a sources entry in config.yaml (or via the WebUI) or a .lua adapter — no Go changes, no recompile. Changes submitted through the WebUI take effect immediately (hot reload); editing config.yaml or a .lua file under adapter_dir by hand requires a process restart.

Overview

Exposes a standard OpenAI Chat Completions API (/v1/chat/completions + /v1/models) to your internal network, translating between multiple upstream protocols via Lua adapters. Supports one-shot calls, SSE streaming, AUTO model routing, multimodal passthrough, and tool calls.

Extracted and independently evolved from the multi-source LLM adapter layer of HomeAgent (internal/agent/api/provider.go + internal/lua/adapters/*).

Features

• Multi-source: any number of upstream sources in one process, routed by the request's model.
• AUTO mode: with default_model: AUTO, scheduling follows the tiered AUTO chain saved on the Priority page (see below).
• Unified output: every source speaks OpenAI format (incl. reasoning_content, tool_calls, usage).
• Image generation: POST /v1/images/generations, routed to models with kind: image.
• Multimodal: content arrays (image_url etc.) pass through losslessly; Anthropic/Gemini/Ollama are translated automatically.
• LuaJIT VM: golua-binding LuaJIT; each adapter has its own VM + worker pool for safe concurrency. Pools are elastic: the sum of max_concurrent over an adapter's sources is a ceiling, not a preallocation — states are booted on demand and reclaimed when demand drops, so an idle gateway holds close to zero Lua states. The grow step follows the adapter's max concurrency (clamp(ceil(max/8), 1, 8), and never exceeds the number of queued callers); the shrink step follows the live connection count (clamp(ceil(slack/(1+in_use)), 1, slack)), so an idle pool collapses in one round while a busy one gives up a single state at a time.
• On-demand request logs: dashboard totals are computed from the full audit history (streamed once at startup and released — 29 MB / 221k lines in ~260 ms), while the raw records are never held in memory: the dashboard loads one screen, scrolling pages the rest straight off disk, CSV export streams in O(1) memory, and leaving the page releases everything.
• Self-healing cooldown: cooldown is capped at 5 minutes and, past the window's midpoint, exactly one probe request is allowed through; a recovered upstream (or a reset quota) returns to full rotation on that probe instead of waiting out the window.
• disable_thinking: disable_thinking: true toggles reasoning per-request.
• Lua adapter protocol: each source mounts a .lua adapter with transform_request / transform_response / transform_stream_chunk — all protocol differences live in Lua, Go only schedules and proxies.
• Signature / header hooks: adapters may define build_headers(meta) to inject/sign request headers before the HTTP call (e.g. KimiCode style app-validation), with helpers like hmac_sha256_hex, sha256_hex, base64_encode.
• WebUI: built-in management page to view/add/edit sources & models, edit the AUTO chain, and manage per-key model scopes, persisted to a runtime file.
• Encrypted secrets at rest: upstream api_key, custom header values and gateway keys are stored AES-256-GCM encrypted (master.key 0600 next to the runtime file, or LLMS_PROXY_MASTER_KEY).
• Live source probing: the status page probes each source (GET {base}/models, falling back to a minimal chat call) without touching the scheduler's backoff state; errors are shown on hover.
• Auth: the gateway validates client keys via gateway_keys; independent from each upstream's own key.
• Streaming: SSE chat.completion.chunk with a role-first chunk and [DONE] terminator.

Quick start

GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
./llmsproxy -config config.yaml    # first run generates a default config and prints a random admin key

Depends on golua (LuaJIT bindings). You must build with -tags luajit; otherwise the built-in superset gopher-lua path is used (behavior differs slightly).

No config is shipped in the repo (config files carry real keys). On first run the binary generates a default config at the -config path: a random admin key (printed to the startup log), loopback-only 127.0.0.1:8080, and a keyless source ready to chat. Rotate the admin key in the WebUI after first login.

# no key -> 401
curl http://127.0.0.1:8080/v1/models

# one-shot ($KEY = the admin key printed at first startup)
curl -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

# streaming
curl -N -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

Any OpenAI SDK works: point base_url at the gateway, use any authorized gateway key as the API key.

Configuration

Generated on first run at the -config path (default ./config.yaml). Core fields:

listen: 127.0.0.1:8080        # bind address (keep internal/loopback)
gateway_keys: [sk-gw-<random>] # initial admin key seed (random per install), migrated to the store on first start
default_model: AUTO           # when model is unroutable, follow the AUTO chain
adapter_dir: adapters         # Lua adapter dir; created+seeded if missing, read-only otherwise
runtime_file: runtime.json    # WebUI-edited sources/keys/AUTO chain persist here

sources:
  - name: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-...            # or api_key_env: SOME_ENV to read from an env var
    adapter: deepseek
    max_concurrent: 8
    models:
      - id: deepseek-v4-flash
        priority: 100          # YAML sources seed the AUTO chain on first start
        kind: chat
      - id: deepseek-v4-pro
        priority: 60
        kind: chat
    # static headers (take precedence over adapter defaults)
    headers: { X-Tenant: prod }
    # passthrough metadata for the Lua build_headers hook
    meta: { app_id: x, app_secret: y }
    temperature: 0.7
    max_tokens: 4096
    timeout: 120s              # request timeout, default 120s

Sensitive fields in the runtime file (runtime_file) are encrypted at rest:

• AES-256-GCM, stored as enc:v1:<base64>.
• Master key source: env var LLMS_PROXY_MASTER_KEY (64 hex chars), else a master.key file next to the runtime file; if neither exists it is generated on first start (mode 0600).
• Old plaintext files still load; the first UI save migrates the whole file to ciphertext.
• Back up master.key with your config — losing it makes the secrets undecryptable. Do not commit it.

Gateway keys (multi-key)

Gateway auth uses a "multi-key + role + model scope" model. Keys are persisted under the keys field of the runtime file (encrypted at rest):

• The gateway_keys config is only an initial admin key seed — it is migrated into the runtime store on first start and no longer drives auth.
• Important: after first start, replace the admin key via the WebUI Keys page. The seed key is written in plaintext in config.yaml, so keeping it active is a security risk; create a new admin key, log in with it, then delete the seed key.
• The WebUI Keys page creates/deletes keys. Each key has a role (admin manages everything, user sees only its own key) and an optional model scope (model + source + token quota + reset period).
• Clients authenticate with any authorized key's plaintext as Authorization: Bearer <key>.
• Deleting a key removes it from the store immediately.

Model routing

/v1/chat/completions model resolution order:

1. source/model or source:model prefix → pinned source;
2. exact match of a source's model;
3. with default_model: AUTO → follow the AUTO chain from the Priority page (see below);
4. otherwise the request errors.

Any OpenAI client can pin to a source by setting model to its name/anything; AUTO (or the gateway's default_model: AUTO) follows the tiered AUTO chain.

AUTO chain (Priority page)

AUTO scheduling is driven only by the rules saved on the Priority page. The numeric models[].priority in source configs no longer participates in scheduling and is no longer shown. Chat and image generation are two independent chains, switched in the Priority page via the Chat / Image toggle:

• Chat chain (auto field): serves POST /v1/chat/completions with model: AUTO; only kind: chat models are honored (image slots are skipped).
• Image chain (auto_image field): serves POST /v1/images/generations with model: AUTO; only kind: image models are honored. With no image chain configured, AUTO image generation falls back to legacy discovery (all sources exposing an image model, in registry order) for backward compatibility.

Each rule is one "slot": { model, source, tier, token_quota, period, hours }.

• tier is the priority tier: models in the same tier sit side by side and share it; tiers run high → low.
• The same model may appear in several slots (e.g. A low → B low → A high → B high) and is tried in tier order.
• token_quota > 0 means the slot is skipped once its tokens within the reset window are exhausted; period supports hour / week / month / nhour (with hours); empty = unlimited.
• Within a tier, slots run by preference score; cooling / quota-exhausted / hard-failed slots fall through to the next tier, and total failure returns 503 with a per-tier summary.

Sources vs. adapters

• A source describes a connection to an upstream: name, base_url, api_key, model list and priorities.
• An adapter is the protocol translation logic (Lua script): converts the unified OpenAI request to the upstream's native format and back.
• One adapter serves many sources (e.g. openai.lua for any OpenAI-compatible site); one source can switch adapters via its adapter field.
• Sources decide whom to talk to and which models to expose; adapters decide how to talk. They are linked by the adapter field inside each sources[] entry.

Loading flow (assembled by Core in internal/core):

1. On startup lua.NewVM(adapter_dir) loads all adapters: built-in ones (embedded at compile time) + same-name override files under adapter_dir.
2. config.Load reads config.yaml; config.NewStore(runtime_file) reads WebUI-edited runtime sources; both are merged by name.
3. rebuildRegistry creates a provider.Provider per source (holding its adapter), sizing the adapter's worker pool from the source's concurrency limit.
4. Incoming requests are routed by Registry to a provider, which calls the adapter's transform_request → HTTP call → transform_response / transform_stream_chunk.

"Add/Edit source", "Upload adapter", "AUTO chain edits" and "per-key model scopes" in the WebUI take effect immediately (written to the runtime file / adapter_dir, then reassembled — no restart). Directly editing config.yaml or files under adapter_dir requires a process restart.

Deleting a source/adapter in the WebUI truly removes runtime (UI-created) sources and .lua files; base sources defined in config.yaml cannot rewrite that file, so they are hidden via a deletion tombstone (still hidden after restart) and can be restored by re-adding the same name in the UI.

Source templates (multi-key balancing)

When several upstream keys share one config (URL / adapter / model list / concurrency / rpm), copying the whole source block N times is wasteful. A source template stores every source field except name and api_key, so you create N key-only-different sources from one shared skeleton — natural multi-key load balancing (each expanded source schedules, cools down, and reports health independently).

• WebUI Sources page, top-right Templates button: list all templates with edit / delete / create.
• Add-source dialog has two header buttons:
  • From template → pick a template → form auto-fills (name + key still yours to enter);
  • As template → save the current form's non-key/name fields as a template.
• Templates persist in the runtime file's source_templates field alongside runtime sources (hot-managed, apply immediately). Templates hold no secret keys, so they're safe to share / version-control.

A template is just a recipe — it doesn't become a source by itself. Only the key-bearing sources created from it carry real traffic.

Memory footprint (measured and tuned)

Memory is not a constant: it scales with the number of configured sources (each source owns an http.Transport with its connection pool plus per-model health state), not with uptime. Measured on this machine (Linux x86_64, 12 cores):

Deployment shape	Startup RSS	Settled RSS
1 source / 1 adapter (minimal)	~4 MB	~10 MB
1 source + a 29 MB audit history	~19 MB	~20 MB
16 sources / 13 adapters / 59 models (this host)	~28 MB	~32-35 MB

For reference, the same production config used ~105 MB before this round of work. The reduction comes from three places: raw audit records are no longer kept resident (~25 MB — the aggregates still scan the full history, but the scan releases as it goes), Lua state pools no longer grow monotonically, and the two runtime knobs below.

Breakdown of the production instance (per-region, from /proc/<pid>/smaps):

Region	RSS	Notes
Go heap	~14 MB	provider/registry/scheduler state + connection-pool buffers
Other anonymous (thread stacks / LuaJIT chunks / runtime)	~12 MB	scales with thread count and loaded adapters
Binary text+rodata	~8 MB	mapped executable pages (read-only, reclaimable by the kernel)
Go runtime reservations	~4 MB	the 2 GB+ you see in VSZ is address space, not physical memory
Shared libraries	~3 MB	libc / libluajit / libm

Two recommended deployment knobs (environment only, no code change):

# /etc/systemd/system/llmsproxy.service
Environment=GOGC=50
Environment=MALLOC_ARENA_MAX=2

• MALLOC_ARENA_MAX=2: LuaJIT allocates through cgo into glibc malloc, and glibc allows up to 8 x nproc per-thread arenas. Every OS thread that touches malloc claims one (~1 MB each) and never returns it to the OS. Measured: 8-15 arenas (7-12 MB) down to 0.
• GOGC=50: halves the Go heap growth target. It does nothing on its own — the heap it saves is immediately consumed by additional glibc arenas (measured 20.3 -> 21.5 MB, i.e. slightly worse) — so it must be paired with MALLOC_ARENA_MAX. Together they cut settled RSS by ~19%. The gateway is I/O bound (1min10s of CPU per 9 hours here), so the extra GC cycles are free.

The desktop build (Electron) already injects both when spawning the embedded core (cmd/gui/main.js); exporting either variable yourself overrides it. If you write your own systemd unit or container spec, add them there.

Optional: GOMEMLIMIT=48MiB as a soft ceiling saves roughly 1 MB more, at the cost of GC turning aggressive as the limit approaches. Not worth it for a relaxed deployment.

disable_thinking

With "disable_thinking": true in the request body, the gateway passes it to each adapter; the DeepSeek adapter maps it to extra_body.thinking.type = "disabled", other sources follow their own protocol.

WebUI

Built-in admin page at GET /; after login you can:

• Status: source online state (live probe, error on hover), per model/source/key usage stats, request records filtered by key, and CSV export over a time range; click a model to pin the connection snippet to it.
• Chat: streaming / non-streaming debug.
• Keys: create/edit gateway keys, set per-key model scopes (model + source + token quota + reset period); admins manage all keys, users see only their own.
• Priority: drag blocks to build the tiered AUTO chain.
• Sources: add/edit/delete upstream sources online (secrets stored encrypted).
• Adapters: upload / delete Lua adapter scripts.

Changes persist to runtime_file (survive restarts).

Lua adapter protocol

Full API: Lua adapter docs (中文).

Each adapter is a Lua script returning a table (internal/lua/adapters/<name>.lua), overridable by a same-name file under adapter_dir — no recompile needed, restart to take effect; uploads via the WebUI take effect immediately.

return {
  name = "mysrc",
  version = "1.0.0",
  endpoint = "/chat/completions",        -- upstream path (source.endpoint overrides)
  headers = { ["X-Static"] = "v" },      -- static default headers (fallback)

  -- request: convert unified OpenAI req -> upstream native format, return string
  transform_request = function(raw_json) ... end,

  -- response: convert upstream response to unified format string
  --   { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
  transform_response = function(raw_json) ... end,

  -- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips
  transform_stream_chunk = function(raw_chunk) ... end,

  -- [optional] dynamic headers / signature hook
  -- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
  build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
}

Shared helpers: hmac_sha256_hex(key, data), sha256_hex(data), base64_encode(s), tohex(s), json.encode/decode, log(level, msg).

Built-in adapters

openai deepseek anthropic gemini github groq mistral ollama kimicode.

anthropic/gemini/ollama include multimodal conversion (image_url → their native format); with disable_thinking the deepseek adapter sets extra_body.thinking.type to disabled.

kimicode demonstrates build_headers: the cloud validates the calling app, so you HMAC-sign timestamp+URL+body with meta.app_secret and add X-App-Sign-style headers. Configure sources[].meta.{app_id, app_secret, app_agent}.

Desktop GUI (Electron)

Optional standalone product for desktop users on your internal network: an embedded ModelRouter core (Clash Verge-style): one-click start/stop, system tray, autostart, silent launch, and a pixel-for-pixel embedded full WebUI (Status/Chat/Keys/Priority/Sources/Adapters — no login needed). Server users keep using the plain Go binary.

Two installer kinds: Headless and Desktop

Release installers ship in two flavors for different audiences:

• Headless (server): the plain core binary (no Electron), config-driven, for servers / containers / systemd / unattended runs. Ships as deb + rpm (systemd unit with memory tuning + example config) and tar.gz (binary + config.example.yaml + adapters/ + README), Windows as .zip.
• Desktop (desktop app): the Electron GUI installer with an embedded core, for personal daily desktop use.

GUI vs. plain backend

Scenario	Pick	Why
Server / intranet gateway / unattended long-running	Headless (single binary)	~8 MB binary; RSS scales with the number of sources (~10 MB for one, ~32-35 MB for 16 — see Memory footprint), zero-dep single process — drop it into systemd or any container, remote admin
Personal desktop daily use / multi-device intranet sharing	Desktop (GUI)	no-login embedded WebUI, tray one-click, autostart, silent background — for non-CLI users
Windows desktop	Desktop	plain backend needs manual service registration; GUI ships native tray/autostart
CI one-shot 3-platform installers	Desktop packaging scripts	make gui-* emits deb / AppImage / NSIS, drops straight into a release pipeline

Both share the exact same llmsproxy core (LuaJIT build) — configs and adapters are fully compatible, freely interchangeable.

Build & run

cd cmd/gui
npm install
make gui            # or npm run dev — dev run (needs make build first for bin/llmsproxy)
make gui-deb        # local deb (share with other Linux users)
make gui-dist       # deb + rpm + AppImage (linux; rpm needs system rpmbuild)
make core-dist      # core deb + rpm + tar.gz (server edition; needs nfpm)
make gui-win        # Windows NSIS (host mingw + wine)
make gui-win-docker # Windows NSIS, fully dockerized (wine in image; no host mingw/node/wine)

Artifacts land in cmd/build/gui-dist/: ModelRouter-<ver>.AppImage, modelrouter-gui_<ver>_amd64.deb, ModelRouter Setup <ver>.exe (win); core packages land in cmd/build/dist/: llmsproxy_<ver>_amd64.deb, llmsproxy-<ver>.x86_64.rpm. Both target families run packaging/verify-dist.sh afterwards — a size floor that rejects degenerate installers (1.3.0 shipped a 264 KB NSIS stub because the host wine was broken and nothing caught it). At release time artifacts are renamed into the ModelRouter-Desktop-* (GUI) and ModelRouter-Headless-* (plain backend) families before upload to GitCode Releases.

Windows packaging (dockerized, recommended)

make gui-win-docker does cross-compile + NSIS entirely in docker; the host needs no mingw, no wine, no node. Wine (with the i386 runtime) lives inside the image, fixing the broken-host-wine failure that produced the 264 KB stub installer.

• Image: cmd/gui/docker/win-builder/Dockerfile (golang:1.25 + mingw-w64 + node + wine32/wine64 + prebuilt LuaJIT for Windows: lua51.dll + import lib). LuaJIT source comes from the gitcode mirror (reachable in CN) with github fallback; Go modules go through https://goproxy.cn.
• One-shot release: cmd/gui/scripts/dist-win-docker.sh — core build + NSIS packaging run inside the container (USE_SYSTEM_WINE=true uses the image's wine); electron-builder's ~400 MB toolchain cache lives in a docker volume so later builds reuse it.
• The modelrouter/win-builder image is auto-built on first run (~5-10 min) and reused afterwards.
• After packaging, packaging/verify-dist.sh runs as a gate: an exe under 5 MB (or a deb/rpm/7z under 10 MB) fails the build — a broken installer can never reach a Release again.

# Full Windows release (rebuild core + NSIS):
make gui-win-docker

Features

• Embedded core: auto-spawns the bundled llmsproxy (LuaJIT build); config/keys/adapters live in <userData>/profile/. First run generates a random admin key written into keys (not seed) and self-injects it — WebUI needs no login and never nags about changing the initial key. Port changeable in settings (default 8787).
• System tray: core status dot, autostart/silent toggles, start/stop/restart core, quit.
• Silent launch: --silent flag or setting — starts to the tray only; first run always shows the window.
• Autostart: setLoginItemSettings on Windows/macOS; writes ~/.config/autostart/modelrouter-gui.desktop on Linux, appending --silent when silent mode is on.

Layout

cmd/llmsproxy            # entry point
internal/config          # YAML config loading/validation
internal/lua             # LuaJIT VM + worker pool + AdapterCache + built-ins (embed)
internal/provider        # Provider(HTTP) + Registry(routing)
internal/gateway         # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI
internal/types           # unified format & OpenAI wire types

Development & collaboration

• Git branching workflow: main / feature / release-branch conventions in docs/git-workflow-en.md (中文). The line: main is always deployable → features merge into main → cut release/vX.Y.Z and tag → release → hotfixes land on the release branch and are cherry-picked back into main.
• Lua adapter protocol: see docs/lua-adapters-en.md (中文).

Tests

go test -tags luajit ./...

Covers: config validation, adapter load/transform, signature hooks, gateway auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and disable_thinking.