mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-09-19 16:39:15 +00:00
docs: replace invented memory figures with measured ones, ship the tuning knobs
The README claimed "~15 MB RSS" and, after the log-loading work, "~10 MB idle /
~19 MB with a 29 MB audit log". Those were TEST-INSTANCE numbers: one mock source
and one adapter. The real production config on this host (16 sources, 13
adapters, 59 models) sits at ~37-42 MB, and sat at ~105 MB before this series.
Quoting the single-source figure as the headline was misleading.
Both READMEs now state that memory scales with the number of configured sources
rather than with uptime, give a three-row measurement table (1 source / 1 source
with a 29 MB audit history / the 16-source production instance), and break the
production RSS down per region (Go heap, thread stacks + LuaJIT, mapped binary,
Go reservations, shared libs) so an operator can tell which part their own
deployment will grow.
Two runtime knobs are documented and now shipped by default in the desktop
build's core spawn (cmd/gui/main.js, overridable by exporting either variable):
* MALLOC_ARENA_MAX=2 — LuaJIT allocates through cgo into glibc malloc, and
glibc keeps up to 8*nproc per-thread arenas of ~1 MB that are never returned.
Measured 8-15 arenas (7-12 MB) -> 0.
* GOGC=50 — halves the Go heap target. Documented explicitly as useless ALONE
(measured 20.3 -> 21.5 MB, i.e. worse, because the saved heap is eaten by
more glibc arenas); only the pair cuts settled RSS, by ~19%.
Also corrects the binary size (8-12 MB, ~8 MB after the deploy script's -s -w)
and adds the elastic-pool / on-demand-log / self-healing-cooldown bullets that
README.md already had to README_EN.md.
This commit is contained in:
52
README.md
52
README.md
@ -15,8 +15,8 @@
|
|||||||
|
|
||||||
### 极致轻量
|
### 极致轻量
|
||||||
|
|
||||||
- **单二进制**:编译后约 10MB,零运行时依赖(仅依赖系统 libc),部署即用
|
- **单二进制**:编译后 8~12MB(`-s -w` strip 后约 8MB),零运行时依赖(仅依赖系统 libc),部署即用
|
||||||
- **极低内存占用**:空闲 ~10MB RSS,带 29MB 历史审计日志启动仅 ~19MB,满载并发 100+ 请求时峰值 < 100MB
|
- **低内存占用**:与源数量相关,非与运行时长相关——单源 ~10MB、16 源生产实例 ~40MB([实测分解与调优](#内存占用实测与调优))
|
||||||
- **日志按需加载**:审计日志不常驻内存——默认只加载首屏,下滚自动分页,CSV 导出流式写出(O(1) 内存),离页即释放
|
- **日志按需加载**:审计日志不常驻内存——默认只加载首屏,下滚自动分页,CSV 导出流式写出(O(1) 内存),离页即释放
|
||||||
- **零运行时依赖**:纯 Go + LuaJIT 静态链接,无需安装 Python/Node/Java 等运行时
|
- **零运行时依赖**:纯 Go + LuaJIT 静态链接,无需安装 Python/Node/Java 等运行时
|
||||||
- **启动极快**:冷启动 < 200ms,热重载配置 < 10ms
|
- **启动极快**:冷启动 < 200ms,热重载配置 < 10ms
|
||||||
@ -284,6 +284,52 @@ WebUI 上的"新增/编辑源"、"上传 Lua 适配器"、"改 AUTO 优先级链
|
|||||||
模板不会自己变成源——它只是「配方」。真正承担流量的是从模板创建出来的、带 key
|
模板不会自己变成源——它只是「配方」。真正承担流量的是从模板创建出来的、带 key
|
||||||
的具体源。
|
的具体源。
|
||||||
|
|
||||||
|
### 内存占用(实测与调优)
|
||||||
|
|
||||||
|
内存不是一个定数:它随**配置的源数量**增长(每个源一个 `http.Transport` 与连接池、
|
||||||
|
一组健康状态),与运行时长无关。本机实测(Linux x86_64,12 核):
|
||||||
|
|
||||||
|
| 部署形态 | 启动 RSS | 稳态 RSS |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 源 / 1 适配器(最小配置) | ~4 MB | ~10 MB |
|
||||||
|
| 1 源 + 29 MB 历史审计日志 | ~19 MB | ~20 MB |
|
||||||
|
| **16 源 / 13 适配器 / 59 模型(本机生产)** | ~28 MB | **~37–42 MB** |
|
||||||
|
|
||||||
|
> 历史参考:本项优化前同一生产配置为 **~105 MB**。降幅来自三处:审计日志不再全量回放
|
||||||
|
> (约 25 MB)、Lua 状态池不再单调增长、以及下面两个运行时开关。
|
||||||
|
|
||||||
|
内存构成(生产实例分段测量,`/proc/<pid>/smaps`):
|
||||||
|
|
||||||
|
| 区域 | RSS | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| Go 堆 | ~14 MB | provider/registry/scheduler 结构 + 连接池缓冲 |
|
||||||
|
| 其他匹名(线程栈 / LuaJIT chunk / runtime) | ~12 MB | 与线程数、已加载适配器数相关 |
|
||||||
|
| 二进制 text+rodata | ~8 MB | 映射的可执行文件页(只读、可被内核回收) |
|
||||||
|
| Go runtime 预留 | ~4 MB | VSZ 上看到的 2 GB+ 是地址空间预留,不占物理内存 |
|
||||||
|
| 共享库 | ~3 MB | libc / libluajit / libm |
|
||||||
|
|
||||||
|
**两个推荐的部署开关**(只适用于环境变量,无需改代码):
|
||||||
|
|
||||||
|
```ini
|
||||||
|
# /etc/systemd/system/llmsproxy.service
|
||||||
|
Environment=GOGC=50
|
||||||
|
Environment=MALLOC_ARENA_MAX=2
|
||||||
|
```
|
||||||
|
|
||||||
|
- `MALLOC_ARENA_MAX=2`:LuaJIT 的分配走 cgo → glibc malloc,glibc 默认允许 `8×nproc`
|
||||||
|
个 per-thread arena,每个碰到 malloc 的 OS 线程会占用一个(各约 1 MB,且**不归还给系统**)。
|
||||||
|
实测从 8–15 个 arena(约 7–12 MB)降到 **0**。
|
||||||
|
- `GOGC=50`:把 Go 堆增长目标减半。**单独使用无效**(省下的堆会立即被更多 glibc
|
||||||
|
arena 吃掉,实测 20.3 → 21.5 MB 反而变大),必须与 `MALLOC_ARENA_MAX` 配合,
|
||||||
|
两者同时开启才降 **~19%**。网关是 I/O 密集型(本机 9 小时仅消耗 1min10s CPU),
|
||||||
|
多出的 GC 周期与它的空闲 CPU 相比可忽略。
|
||||||
|
|
||||||
|
> 桌面版(Electron)已在 `cmd/gui/main.js` 里默认为内嵌核心注入这两个开关;手动 export 同名
|
||||||
|
> 环境变量可覆盖。如果你用自己的 systemd unit / 容器,建议照上面加上。
|
||||||
|
|
||||||
|
可选:`GOMEMLIMIT=48MiB` 作软上限,实测再省 ~1 MB;代价是逐近上限时 GC 转激进,
|
||||||
|
温和场景不必开。
|
||||||
|
|
||||||
### disable_thinking
|
### disable_thinking
|
||||||
|
|
||||||
请求体带 `"disable_thinking": true` 时,网关透传给各适配器;DeepSeek 适配器将其
|
请求体带 `"disable_thinking": true` 时,网关透传给各适配器;DeepSeek 适配器将其
|
||||||
@ -369,7 +415,7 @@ return {
|
|||||||
|
|
||||||
| 场景 | 推荐 | 理由 |
|
| 场景 | 推荐 | 理由 |
|
||||||
| ------ | ------ | ------ |
|
| ------ | ------ | ------ |
|
||||||
| 服务器 / 内网网关 / 无人值守常驻 | **Headless**(单二进制) | 极轻量(~10MB、~15MB RSS),零依赖单进程,直接跑在 systemd/任意容器里,远程管理 |
|
| 服务器 / 内网网关 / 无人值守常驻 | **Headless**(单二进制) | 极轻量(二进制 ~8MB;RSS 随源数量,单源 ~10MB、16 源 ~40MB,见[内存占用](#内存占用实测与调优)),零依赖单进程,直接跑在 systemd/任意容器里,远程管理 |
|
||||||
| 个人桌面日常使用 / 多设备内网共享 | **Desktop**(GUI) | 免登录内嵌 WebUI、系统托盘一键启停、开机自启、静默后台,适合不懂命令行的使用者 |
|
| 个人桌面日常使用 / 多设备内网共享 | **Desktop**(GUI) | 免登录内嵌 WebUI、系统托盘一键启停、开机自启、静默后台,适合不懂命令行的使用者 |
|
||||||
| Windows 桌面 | **Desktop** | 纯后端在 Windows 上需自行注册服务,GUI 提供原生托盘/自启体验 |
|
| Windows 桌面 | **Desktop** | 纯后端在 Windows 上需自行注册服务,GUI 提供原生托盘/自启体验 |
|
||||||
| CI 一键出三平台安装包 | **Desktop 打包脚本** | `make gui-*` 系列出 deb/AppImage/NSIS,可进发行流水线 |
|
| CI 一键出三平台安装包 | **Desktop 打包脚本** | `make gui-*` 系列出 deb/AppImage/NSIS,可进发行流水线 |
|
||||||
|
|||||||
73
README_EN.md
73
README_EN.md
@ -11,7 +11,7 @@ KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mod
|
|||||||
which routes to the best healthy upstream by configured priority.
|
which routes to the best healthy upstream by configured priority.
|
||||||
|
|
||||||
> **Lightweight, no recompile to add sources**: the gateway is a single Go
|
> **Lightweight, no recompile to add sources**: the gateway is a single Go
|
||||||
> binary (~10 MB, zero runtime dependencies). Adding or switching an upstream is
|
> binary (8-12 MB, ~8 MB stripped; zero runtime dependencies). Adding or switching an upstream is
|
||||||
> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua`
|
> just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua`
|
||||||
> adapter — **no Go changes, no recompile**. Changes submitted through the
|
> adapter — **no Go changes, no recompile**. Changes submitted through the
|
||||||
> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua`
|
> WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua`
|
||||||
@ -41,7 +41,21 @@ Extracted and independently evolved from the multi-source LLM adapter layer of
|
|||||||
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
|
- **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly;
|
||||||
Anthropic/Gemini/Ollama are translated automatically.
|
Anthropic/Gemini/Ollama are translated automatically.
|
||||||
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
|
- **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker
|
||||||
pool for safe concurrency.
|
pool for safe concurrency. Pools are **elastic**: the sum of `max_concurrent`
|
||||||
|
over an adapter's sources is a ceiling, not a preallocation — states are booted
|
||||||
|
on demand and reclaimed when demand drops, so an idle gateway holds close to
|
||||||
|
zero Lua states. The grow step follows the adapter's max concurrency
|
||||||
|
(`clamp(ceil(max/8), 1, 8)`, and never exceeds the number of queued callers);
|
||||||
|
the shrink step follows the live connection count
|
||||||
|
(`clamp(ceil(slack/(1+in_use)), 1, slack)`), so an idle pool collapses in one
|
||||||
|
round while a busy one gives up a single state at a time.
|
||||||
|
- **On-demand request logs**: the audit log is never held in memory — the
|
||||||
|
dashboard loads one screen, scrolling pages the rest straight off disk, CSV
|
||||||
|
export streams in O(1) memory, and leaving the page releases everything.
|
||||||
|
- **Self-healing cooldown**: cooldown is capped at 5 minutes and, past the
|
||||||
|
window's midpoint, exactly one probe request is allowed through; a recovered
|
||||||
|
upstream (or a reset quota) returns to full rotation on that probe instead of
|
||||||
|
waiting out the window.
|
||||||
- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request.
|
- **disable_thinking**: `disable_thinking: true` toggles reasoning per-request.
|
||||||
- **Lua adapter protocol**: each source mounts a `.lua` adapter with
|
- **Lua adapter protocol**: each source mounts a `.lua` adapter with
|
||||||
`transform_request` / `transform_response` / `transform_stream_chunk` — all
|
`transform_request` / `transform_response` / `transform_stream_chunk` — all
|
||||||
@ -258,6 +272,59 @@ independently).
|
|||||||
A template is just a recipe — it doesn't become a source by itself. Only the
|
A template is just a recipe — it doesn't become a source by itself. Only the
|
||||||
key-bearing sources created from it carry real traffic.
|
key-bearing sources created from it carry real traffic.
|
||||||
|
|
||||||
|
### Memory footprint (measured and tuned)
|
||||||
|
|
||||||
|
Memory is not a constant: it scales with the **number of configured sources**
|
||||||
|
(each source owns an `http.Transport` with its connection pool plus per-model
|
||||||
|
health state), not with uptime. Measured on this machine (Linux x86_64, 12 cores):
|
||||||
|
|
||||||
|
| Deployment shape | Startup RSS | Settled RSS |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 source / 1 adapter (minimal) | ~4 MB | ~10 MB |
|
||||||
|
| 1 source + a 29 MB audit history | ~19 MB | ~20 MB |
|
||||||
|
| **16 sources / 13 adapters / 59 models (this host)** | ~28 MB | **~37-42 MB** |
|
||||||
|
|
||||||
|
> For reference, the same production config used **~105 MB** before this round of
|
||||||
|
> work. The reduction comes from three places: the audit log is no longer replayed
|
||||||
|
> in full (~25 MB), Lua state pools no longer grow monotonically, and the two
|
||||||
|
> runtime knobs below.
|
||||||
|
|
||||||
|
Breakdown of the production instance (per-region, from `/proc/<pid>/smaps`):
|
||||||
|
|
||||||
|
| Region | RSS | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Go heap | ~14 MB | provider/registry/scheduler state + connection-pool buffers |
|
||||||
|
| Other anonymous (thread stacks / LuaJIT chunks / runtime) | ~12 MB | scales with thread count and loaded adapters |
|
||||||
|
| Binary text+rodata | ~8 MB | mapped executable pages (read-only, reclaimable by the kernel) |
|
||||||
|
| Go runtime reservations | ~4 MB | the 2 GB+ you see in VSZ is address space, not physical memory |
|
||||||
|
| Shared libraries | ~3 MB | libc / libluajit / libm |
|
||||||
|
|
||||||
|
**Two recommended deployment knobs** (environment only, no code change):
|
||||||
|
|
||||||
|
```ini
|
||||||
|
# /etc/systemd/system/llmsproxy.service
|
||||||
|
Environment=GOGC=50
|
||||||
|
Environment=MALLOC_ARENA_MAX=2
|
||||||
|
```
|
||||||
|
|
||||||
|
- `MALLOC_ARENA_MAX=2`: LuaJIT allocates through cgo into glibc malloc, and glibc
|
||||||
|
allows up to `8 x nproc` per-thread arenas. Every OS thread that touches malloc
|
||||||
|
claims one (~1 MB each) and **never returns it to the OS**. Measured: 8-15
|
||||||
|
arenas (7-12 MB) down to **0**.
|
||||||
|
- `GOGC=50`: halves the Go heap growth target. **It does nothing on its own** —
|
||||||
|
the heap it saves is immediately consumed by additional glibc arenas (measured
|
||||||
|
20.3 -> 21.5 MB, i.e. slightly worse) — so it must be paired with
|
||||||
|
`MALLOC_ARENA_MAX`. Together they cut settled RSS by **~19%**. The gateway is
|
||||||
|
I/O bound (1min10s of CPU per 9 hours here), so the extra GC cycles are free.
|
||||||
|
|
||||||
|
> The desktop build (Electron) already injects both when spawning the embedded
|
||||||
|
> core (`cmd/gui/main.js`); exporting either variable yourself overrides it. If you
|
||||||
|
> write your own systemd unit or container spec, add them there.
|
||||||
|
|
||||||
|
Optional: `GOMEMLIMIT=48MiB` as a soft ceiling saves roughly 1 MB more, at the
|
||||||
|
cost of GC turning aggressive as the limit approaches. Not worth it for a relaxed
|
||||||
|
deployment.
|
||||||
|
|
||||||
### disable_thinking
|
### disable_thinking
|
||||||
|
|
||||||
With `"disable_thinking": true` in the request body, the gateway passes it to
|
With `"disable_thinking": true` in the request body, the gateway passes it to
|
||||||
@ -354,7 +421,7 @@ Release installers ship in two flavors for different audiences:
|
|||||||
|
|
||||||
| Scenario | Pick | Why |
|
| Scenario | Pick | Why |
|
||||||
| ---- | ---- | ---- |
|
| ---- | ---- | ---- |
|
||||||
| Server / intranet gateway / unattended long-running | **Headless** (single binary) | ~10 MB, ~15 MB RSS, zero-dep single process — drop it into systemd or any container, remote admin |
|
| Server / intranet gateway / unattended long-running | **Headless** (single binary) | ~8 MB binary; RSS scales with the number of sources (~10 MB for one, ~40 MB for 16 — see [Memory footprint](#memory-footprint-measured-and-tuned)), zero-dep single process — drop it into systemd or any container, remote admin |
|
||||||
| Personal desktop daily use / multi-device intranet sharing | **Desktop** (GUI) | no-login embedded WebUI, tray one-click, autostart, silent background — for non-CLI users |
|
| Personal desktop daily use / multi-device intranet sharing | **Desktop** (GUI) | no-login embedded WebUI, tray one-click, autostart, silent background — for non-CLI users |
|
||||||
| Windows desktop | **Desktop** | plain backend needs manual service registration; GUI ships native tray/autostart |
|
| Windows desktop | **Desktop** | plain backend needs manual service registration; GUI ships native tray/autostart |
|
||||||
| CI one-shot 3-platform installers | **Desktop packaging scripts** | `make gui-*` emits deb / AppImage / NSIS, drops straight into a release pipeline |
|
| CI one-shot 3-platform installers | **Desktop packaging scripts** | `make gui-*` emits deb / AppImage / NSIS, drops straight into a release pipeline |
|
||||||
|
|||||||
@ -198,6 +198,17 @@ function startCore() {
|
|||||||
cwd: PROFILE_DIR,
|
cwd: PROFILE_DIR,
|
||||||
env: Object.assign({}, process.env, {
|
env: Object.assign({}, process.env, {
|
||||||
LLMS_PROXY_GUI: "1",
|
LLMS_PROXY_GUI: "1",
|
||||||
|
// Memory tuning, measured on a 16-source deployment (see README 内存占用):
|
||||||
|
// MALLOC_ARENA_MAX caps glibc per-thread malloc arenas. LuaJIT allocates
|
||||||
|
// through cgo -> glibc malloc, and glibc defaults to 8*nproc arenas, so
|
||||||
|
// every OS thread touching malloc reserves its own ~1 MB arena that is
|
||||||
|
// never returned to the OS.
|
||||||
|
// GOGC halves the Go heap growth target. On its own it does not help (the
|
||||||
|
// saved heap is immediately eaten by extra glibc arenas); only the pair
|
||||||
|
// lowers settled RSS. The core is I/O bound, so the extra GC is free.
|
||||||
|
// Both are overridable: a user who exports them keeps their own values.
|
||||||
|
GOGC: process.env.GOGC || "50",
|
||||||
|
MALLOC_ARENA_MAX: process.env.MALLOC_ARENA_MAX || "2",
|
||||||
}),
|
}),
|
||||||
stdio: ["ignore", "pipe", "pipe"],
|
stdio: ["ignore", "pipe", "pipe"],
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user