ModelRouter/README_EN.md

# ModelRouter

> **Repository**: <https://atomgit.com/HeNanMU-EEA/ModelRouter>
>
> **中文**: [README.md](./README.md)
>
> **AI-assisted**: code and documentation were produced with AI assistance and
> reviewed by a human.

An intranet gateway for people with several LLM subscriptions: configure once and
let multiple services share the upstream capacity. Pick a model explicitly, or let
`AUTO` choose by priority among the sources that are currently healthy.

The gateway itself is a single Go binary (~10 MB, no runtime dependencies). Adding
or switching an upstream is a `sources` entry in `config.yaml` (or the WebUI), or a
dropped-in `.lua` adapter — **no Go changes, no rebuild**. Changes submitted through
the WebUI take effect immediately; editing `config.yaml` or a `.lua` adapter file
directly requires a restart.

## What it does

- **One OpenAI-compatible API**: `/v1/chat/completions`, `/v1/models`,
  `/v1/images/generations`. Every source speaks the same shape outwards, including
  `reasoning_content`, `tool_calls` and `usage`.
- **Many sources, one AUTO chain**: configure any number of upstreams; `AUTO`
  follows the priority tiers saved in the WebUI, fails over between them, cools
  down broken sources and probes them back to health at half-cool.
- **Protocol differences live in Lua**: each source binds a `.lua` adapter that
  transforms requests, responses and stream chunks both ways, and can sign headers
  via `build_headers(meta)`. Vendor knowledge stays out of the Go kernel.
- **Plugins are independent**: business extensions (channel login, billing, quota
  probing) are external Lua plugins loaded from `plugin_dir`. They are not compiled
  into the binary and not shipped in release archives, and they talk to the kernel
  only through the `modelrouter` SDK.
- **Multi-tenant keys**: gateway keys are separate from upstream keys, and can be
  scoped to a model set, an AUTO chain and a quota.
- **Encrypted at rest**: upstream keys, custom header values and gateway keys are
  sealed with AES-256-GCM; `master.key` is 0600, or injected via
  `MODELROUTER_MASTER_KEY`.
- **Observability**: live source probing, HTTP access log, config-change audit,
  request and usage CSV export. Writes are kept off the request hot path.

## Quick start

```bash
# LuaJIT development libraries are required; -tags luajit is mandatory
GOMODCACHE=... GOPROXY=off go build -tags luajit -o modelrouter ./cmd/modelrouter

# First run generates a default config and prints a random admin key
./modelrouter -config config.yaml
```

> Depends on [golua](https://github.com/aarzilli/golua) (LuaJIT bindings). Without
> `-tags luajit` the build falls back to gopher-lua, which behaves slightly
> differently.

> **No config in the repo**: the repository ships no config file (they contain
> keys). The generated default binds `127.0.0.1:8080` only; rotate the admin key in
> the WebUI after the first login.

```bash
# No key -> 401
curl http://127.0.0.1:8080/v1/models

# Non-streaming ($KEY = the admin key printed at first start)
curl -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions

# Streaming
curl -N -H "Authorization: Bearer $KEY" \
     -d '{"model":"AUTO","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
     http://127.0.0.1:8080/v1/chat/completions
```

Any OpenAI SDK works: point `base_url` at the gateway and use an authorized gateway
key as `api_key`.

## Documentation

The detailed documentation is currently written in Chinese.

| Topic | Document |
|---|---|
| Implementation and the full feature list | [docs/features.md](docs/features.md) |
| Configuration: `config.yaml`, model discovery, multiple keys | [docs/configuration.md](docs/configuration.md) |
| Routing: model routing, AUTO chain, cooldown, sources vs adapters, source templates | [docs/routing.md](docs/routing.md) |
| Adapters: Lua protocol and built-in adapters | [docs/adapters.md](docs/adapters.md) · [docs/lua-adapters-en.md](docs/lua-adapters-en.md) (English) |
| Plugins: SDK contract, hooks, UI extensions, HTTP endpoints | [docs/plugins.md](docs/plugins.md) |
| Desktop app: Headless / Desktop artifacts, build and run | [docs/gui.md](docs/gui.md) |
| Performance: measured memory use, tuning, thinking budget | [docs/performance.md](docs/performance.md) |
| WebUI usage | [docs/webui.md](docs/webui.md) |
| Development: layout, tests, branch and release discipline | [docs/development.md](docs/development.md) · [docs/git-workflow-en.md](docs/git-workflow-en.md) (English) |
| Deployment and migration | [docs/deployment.md](docs/deployment.md) |

## Renamed from llmsproxy

The product name has been ModelRouter for a while; the binary, config directory,
service name and environment variables are now `modelrouter` as well:

| Before | After |
|---|---|
| `llmsproxy` (binary) | `modelrouter` |
| `/etc/llmsproxy/` | `/etc/modelrouter/` |
| `llmsproxy.service` | `modelrouter.service` |
| `LLMSPROXY_*` | `MODELROUTER_*` |

Existing installs migrate with a dry-run-by-default script (copies, never moves,
and can roll back):

```bash
sudo ./packaging/migrate-to-modelrouter.sh              # preview
sudo ./packaging/migrate-to-modelrouter.sh --apply      # apply
sudo ./packaging/migrate-to-modelrouter.sh --rollback --apply
```

Old paths and old environment variable names are still accepted on the read side,
so an upgrade cannot fail merely because of ordering. `deploy.sh` refuses to run
when it finds the old `/etc/llmsproxy` without a new config, because that would
deploy a gateway with no key and no sources.

The plugin SDK follows the same naming: the only public entry point is
`modelrouter` (`llmsproxy` remains a compatibility alias). See
[docs/plugins.md](docs/plugins.md).

## Layout

```text
cmd/modelrouter          # entry point (pure Go server)
cmd/gui                  # Electron desktop shell (embeds the same core)
internal/config          # config parsing, sealed storage, runtime file
internal/core            # scheduling, kernel actions, state and audit
internal/lua             # LuaJIT VM + elastic worker pool + built-in adapters (embed)
internal/lua/plugins     # optional plugin examples (not shipped in binary/archives)
internal/provider        # Provider(HTTP) + Registry(routing)
internal/gateway         # OpenAI-compatible HTTP service + auth + streaming + WebUI
internal/types           # unified format and OpenAI wire types
packaging/               # deb/rpm/tar packaging, systemd unit, migration script
docs/                    # topic documentation
```

## Development and tests

```bash
go test -tags luajit ./...           # everything
go test -tags luajit ./internal/lua  # one package (required for LuaJIT changes)
```

- Branch and release discipline: [docs/git-workflow-en.md](docs/git-workflow-en.md).
- Plugins live in `plugin_dir`. The gateway never creates, overwrites or deletes
  plugin files, and a desktop upgrade never touches them either.