ModelRouter/README_EN.md

ModelRouter

中文: README.md

AI-assisted: code and documentation were produced with AI assistance and reviewed by a human.

An intranet gateway for people with several LLM subscriptions: configure once and let multiple services share the upstream capacity. Pick a model explicitly, or let AUTO choose by priority among the sources that are currently healthy.

The gateway itself is a single Go binary (~10 MB, no runtime dependencies). Adding or switching an upstream is a sources entry in config.yaml (or the WebUI), or a dropped-in .lua adapter — no Go changes, no rebuild. Changes submitted through the WebUI take effect immediately; editing config.yaml or a .lua adapter file directly requires a restart.

What it does

• One OpenAI-compatible API: /v1/chat/completions, /v1/models, /v1/images/generations. Every source speaks the same shape outwards, including reasoning_content, tool_calls and usage.
• Many sources, one AUTO chain: configure any number of upstreams; AUTO follows the priority tiers saved in the WebUI, fails over between them, cools down broken sources and probes them back to health at half-cool.
• Protocol differences live in Lua: each source binds a .lua adapter that transforms requests, responses and stream chunks both ways, and can sign headers via build_headers(meta). Vendor knowledge stays out of the Go kernel.
• Plugins are independent: business extensions (channel login, billing, quota probing) are external Lua plugins loaded from plugin_dir. They are not compiled into the binary and not shipped in release archives, and they talk to the kernel only through the modelrouter SDK.
• Multi-tenant keys: gateway keys are separate from upstream keys, and can be scoped to a model set, an AUTO chain and a quota.
• Encrypted at rest: upstream keys, custom header values and gateway keys are sealed with AES-256-GCM; master.key is 0600, or injected via MODELROUTER_MASTER_KEY.
• Observability: live source probing, HTTP access log, config-change audit, request and usage CSV export. Writes are kept off the request hot path.

Quick start

# LuaJIT development libraries are required; -tags luajit is mandatory
GOMODCACHE=... GOPROXY=off go build -tags luajit -o modelrouter ./cmd/modelrouter

# First run generates a default config and prints a random admin key
./modelrouter -config config.yaml

Depends on golua (LuaJIT bindings). Without -tags luajit the build falls back to gopher-lua, which behaves slightly differently.

No config in the repo: the repository ships no config file (they contain keys). The generated default binds 127.0.0.1:8080 only; rotate the admin key in the WebUI after the first login.

# No key -> 401
curl http://127.0.0.1:8080/v1/models

# Non-streaming ($KEY = the admin key printed at first start)
curl -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

# Streaming
curl -N -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

Any OpenAI SDK works: point base_url at the gateway and use an authorized gateway key as api_key.

Documentation

The detailed documentation is currently written in Chinese.

Topic	Document
Implementation and the full feature list	docs/features.md
Configuration: config.yaml, model discovery, multiple keys	docs/configuration.md
Routing: model routing, AUTO chain, cooldown, sources vs adapters, source templates	docs/routing.md
Adapters: Lua protocol and built-in adapters	docs/adapters.md · docs/lua-adapters-en.md (English)
Plugins: SDK contract, hooks, UI extensions, HTTP endpoints	docs/plugins.md
Desktop app: Headless / Desktop artifacts, build and run	docs/gui.md
Performance: measured memory use, tuning, thinking budget	docs/performance.md
WebUI usage	docs/webui.md
Development: layout, tests, branch and release discipline	docs/development.md · docs/git-workflow-en.md (English)
Deployment and migration	docs/deployment.md

Renamed from llmsproxy

The product name has been ModelRouter for a while; the binary, config directory, service name and environment variables are now modelrouter as well:

Before	After
llmsproxy (binary)	modelrouter
/etc/llmsproxy/	/etc/modelrouter/
llmsproxy.service	modelrouter.service
LLMSPROXY_*	MODELROUTER_*

Existing installs migrate with a dry-run-by-default script (copies, never moves, and can roll back):

sudo ./packaging/migrate-to-modelrouter.sh              # preview
sudo ./packaging/migrate-to-modelrouter.sh --apply      # apply
sudo ./packaging/migrate-to-modelrouter.sh --rollback --apply

Old paths and old environment variable names are still accepted on the read side, so an upgrade cannot fail merely because of ordering. deploy.sh refuses to run when it finds the old /etc/llmsproxy without a new config, because that would deploy a gateway with no key and no sources.

The plugin SDK follows the same naming: the only public entry point is modelrouter (llmsproxy remains a compatibility alias). See docs/plugins.md.

Layout

cmd/modelrouter          # entry point (pure Go server)
cmd/gui                  # Electron desktop shell (embeds the same core)
internal/config          # config parsing, sealed storage, runtime file
internal/core            # scheduling, kernel actions, state and audit
internal/lua             # LuaJIT VM + elastic worker pool + built-in adapters (embed)
internal/lua/plugins     # optional plugin examples (not shipped in binary/archives)
internal/provider        # Provider(HTTP) + Registry(routing)
internal/gateway         # OpenAI-compatible HTTP service + auth + streaming + WebUI
internal/types           # unified format and OpenAI wire types
packaging/               # deb/rpm/tar packaging, systemd unit, migration script
docs/                    # topic documentation

Development and tests

go test -tags luajit ./...           # everything
go test -tags luajit ./internal/lua  # one package (required for LuaJIT changes)

• Branch and release discipline: docs/git-workflow-en.md.
• Plugins live in plugin_dir. The gateway never creates, overwrites or deletes plugin files, and a desktop upgrade never touches them either.