ModelRouter/README_EN.md

ModelRouter

中文: README.md

A lightweight, unified OpenAI-compatible LLM gateway for internal networks — configure once, and let multiple services (agents, SDKs, bots) share many upstreams (DeepSeek, Qijiar, OpenAI, Anthropic, Gemini, Groq, Mistral, Ollama, KimiCode…) behind a single endpoint. Pick a specific model or use AUTO mode, which routes to the best healthy upstream by configured priority.

Lightweight, no recompile to add sources: the gateway is a single Go binary (~10 MB, zero runtime dependencies). Adding or switching an upstream is just a sources entry in config.yaml (or via the WebUI) or a .lua adapter — no Go changes, no recompile. Changes submitted through the WebUI take effect immediately (hot reload); editing config.yaml or a .lua file under adapter_dir by hand requires a process restart.

Overview

Exposes a standard OpenAI Chat Completions API (/v1/chat/completions + /v1/models) to your internal network, translating between multiple upstream protocols via Lua adapters. Supports one-shot calls, SSE streaming, AUTO model routing, multimodal passthrough, and tool calls.

Extracted and independently evolved from the multi-source LLM adapter layer of HomeAgent (internal/agent/api/provider.go + internal/lua/adapters/*).

Features

• Multi-source: any number of upstream sources in one process, routed by the request's model.
• AUTO mode: with default_model: AUTO, the highest-priority available source model wins.
• Unified output: every source speaks OpenAI format (incl. reasoning_content, tool_calls, usage).
• Multimodal: content arrays (image_url etc.) pass through losslessly; Anthropic/Gemini/Ollama are translated automatically.
• LuaJIT VM: golua-binding LuaJIT; each adapter has its own VM + worker pool for safe concurrency.
• disable_thinking: disable_thinking: true toggles reasoning per-request.
• Lua adapter protocol: each source mounts a .lua adapter with transform_request / transform_response / transform_stream_chunk — all protocol differences live in Lua, Go only schedules and proxies.
• Signature / header hooks: adapters may define build_headers(meta) to inject/sign request headers before the HTTP call (e.g. KimiCode style app-validation), with helpers like hmac_sha256_hex, sha256_hex, base64_encode.
• WebUI: built-in management page to view/add/edit sources & models, persisted to a runtime file.
• Auth: the gateway validates client keys via gateway_keys; independent from each upstream's own key.
• Streaming: SSE chat.completion.chunk with a role-first chunk and [DONE] terminator.

Quick start

cp config.example.yaml config.yaml   # edit your sources & keys
GOMODCACHE=... GOPROXY=off go build -tags luajit -o llmsproxy ./cmd/llmsproxy
./llmsproxy -config config.yaml

Depends on golua (LuaJIT bindings). You must build with -tags luajit; otherwise the built-in superset gopher-lua path is used (behavior differs slightly).

# no key -> 401
curl http://127.0.0.1:8080/v1/models

# one-shot
curl -H "Authorization: Bearer sk-gw-local-0001" \
     -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

# streaming
curl -N -H "Authorization: Bearer sk-gw-local-0001" \
     -d '{"model":"deepseek-v4-flash","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

Any OpenAI SDK works: point base_url at the gateway, use one of gateway_keys as the API key.

Configuration

See config.example.yaml. Core fields:

listen: 127.0.0.1:8080        # bind address (keep internal/loopback)
gateway_keys: [sk-gw-0001]    # keys clients use; empty = no auth
default_model: AUTO           # when model is unroutable, pick source by priority
adapter_dir: adapters         # Lua adapter dir; built-ins written on first start
runtime_file: runtime.json    # WebUI-edited sources persist here

sources:
  - name: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-...
    adapter: deepseek
    max_concurrent: 8
    models:
      - id: deepseek-v4-flash
        priority: 100          # higher -> preferred by AUTO
        kind: chat
      - id: deepseek-v4-pro
        priority: 60
        kind: chat
    # static headers (take precedence over adapter defaults)
    headers: { X-Tenant: prod }
    # passthrough metadata for the Lua build_headers hook
    meta: { app_id: x, app_secret: y }
    temperature: 0.7
    max_tokens: 4096
    timeout: 120s              # request timeout, default 120s

Model routing

/v1/chat/completions model resolution order:

1. source/model or source:model prefix → pinned source;
2. exact match of a source's model;
3. with default_model: AUTO → highest-priority healthy source model;
4. otherwise fall back to default_source.

Any OpenAI client can pin to a source by setting model to its name/anything; AUTO (or the gateway's default_model: AUTO) picks the healthy source by priority.

Sources vs. adapters

• A source describes a connection to an upstream: name, base_url, api_key, model list and priorities.
• An adapter is the protocol translation logic (Lua script): converts the unified OpenAI request to the upstream's native format and back.
• One adapter serves many sources (e.g. openai.lua for any OpenAI-compatible site); one source can switch adapters via its adapter field.
• Sources decide whom to talk to and which models to expose; adapters decide how to talk. They are linked by the adapter field inside each sources[] entry.

Loading flow (assembled by Core in internal/core):

1. On startup lua.NewVM(adapter_dir) loads all adapters: built-in ones (embedded at compile time) + same-name override files under adapter_dir.
2. config.Load reads config.yaml; config.NewStore(runtime_file) reads WebUI-edited runtime sources; both are merged by name.
3. rebuildRegistry creates a provider.Provider per source (holding its adapter), sizing the adapter's worker pool from the source's concurrency limit.
4. Incoming requests are routed by Registry to a provider, which calls the adapter's transform_request → HTTP call → transform_response / transform_stream_chunk.

"Add/Edit source" and "Upload adapter" in the WebUI take effect immediately (written to the runtime file / adapter_dir, then reassembled — no restart). Directly editing config.yaml or files under adapter_dir requires a process restart.

disable_thinking

With "disable_thinking": true in the request body, the gateway passes it to each adapter; the DeepSeek adapter maps it to extra_body.thinking.type = "disabled", other sources follow their own protocol.

WebUI

Built-in admin page at GET /; after login you can view/edit sources and models in the browser, persisted to runtime_file (survives restarts).

Lua adapter protocol

Full API: Lua adapter docs (中文).

Each adapter is a Lua script returning a table (internal/lua/adapters/<name>.lua), overridable by a same-name file under adapter_dir — no recompile needed, restart to take effect; uploads via the WebUI take effect immediately.

return {
  name = "mysrc",
  version = "1.0.0",
  endpoint = "/chat/completions",        -- upstream path (source.endpoint overrides)
  headers = { ["X-Static"] = "v" },      -- static default headers (fallback)

  -- request: convert unified OpenAI req -> upstream native format, return string
  transform_request = function(raw_json) ... end,

  -- response: convert upstream response to unified format string
  --   { content, reasoning_content, finish_reason, token_usage{...}, tool_calls[{...}] }
  transform_response = function(raw_json) ... end,

  -- streaming chunk: convert upstream SSE data to { content, done, ... }; "" skips
  transform_stream_chunk = function(raw_chunk) ... end,

  -- [optional] dynamic headers / signature hook
  -- meta = { url, method, body, api_key, timestamp, source={ name, meta={...} } }
  build_headers = function(meta) return { ["X-App-Sign"] = sign } end,
}

Shared helpers: hmac_sha256_hex(key, data), sha256_hex(data), base64_encode(s), tohex(s), json.encode/decode, log(level, msg).

Built-in adapters

openai deepseek anthropic gemini github groq mistral ollama kimicode.

anthropic/gemini/ollama include multimodal conversion (image_url → their native format); with disable_thinking the deepseek adapter sets extra_body.thinking.type to disabled.

kimicode demonstrates build_headers: the cloud validates the calling app, so you HMAC-sign timestamp+URL+body with meta.app_secret and add X-App-Sign-style headers. Configure sources[].meta.{app_id, app_secret, app_agent}.

Layout

cmd/llmsproxy            # entry point
internal/config          # YAML config loading/validation
internal/lua             # LuaJIT VM + worker pool + AdapterCache + built-ins (embed)
internal/provider        # Provider(HTTP) + Registry(routing)
internal/gateway         # OpenAI-compatible HTTP + auth + SDK/streaming + WebUI
internal/types           # unified format & OpenAI wire types

Tests

go test -tags luajit ./...

Covers: config validation, adapter load/transform, signature hooks, gateway auth, SDK round trip, SSE streaming, model routing, multimodal passthrough and disable_thinking.