diff --git a/README.md b/README.md index 4942c50..6105860 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,8 @@ ### 极致轻量 -- **单二进制**:编译后约 10MB,零运行时依赖(仅依赖系统 libc),部署即用 -- **极低内存占用**:空闲 ~10MB RSS,带 29MB 历史审计日志启动仅 ~19MB,满载并发 100+ 请求时峰值 < 100MB +- **单二进制**:编译后 8~12MB(`-s -w` strip 后约 8MB),零运行时依赖(仅依赖系统 libc),部署即用 +- **低内存占用**:与源数量相关,非与运行时长相关——单源 ~10MB、16 源生产实例 ~40MB([实测分解与调优](#内存占用实测与调优)) - **日志按需加载**:审计日志不常驻内存——默认只加载首屏,下滚自动分页,CSV 导出流式写出(O(1) 内存),离页即释放 - **零运行时依赖**:纯 Go + LuaJIT 静态链接,无需安装 Python/Node/Java 等运行时 - **启动极快**:冷启动 < 200ms,热重载配置 < 10ms @@ -284,6 +284,52 @@ WebUI 上的"新增/编辑源"、"上传 Lua 适配器"、"改 AUTO 优先级链 模板不会自己变成源——它只是「配方」。真正承担流量的是从模板创建出来的、带 key 的具体源。 +### 内存占用(实测与调优) + +内存不是一个定数:它随**配置的源数量**增长(每个源一个 `http.Transport` 与连接池、 +一组健康状态),与运行时长无关。本机实测(Linux x86_64,12 核): + +| 部署形态 | 启动 RSS | 稳态 RSS | +|---|---|---| +| 1 源 / 1 适配器(最小配置) | ~4 MB | ~10 MB | +| 1 源 + 29 MB 历史审计日志 | ~19 MB | ~20 MB | +| **16 源 / 13 适配器 / 59 模型(本机生产)** | ~28 MB | **~37–42 MB** | + +> 历史参考:本项优化前同一生产配置为 **~105 MB**。降幅来自三处:审计日志不再全量回放 +> (约 25 MB)、Lua 状态池不再单调增长、以及下面两个运行时开关。 + +内存构成(生产实例分段测量,`/proc//smaps`): + +| 区域 | RSS | 说明 | +|---|---|---| +| Go 堆 | ~14 MB | provider/registry/scheduler 结构 + 连接池缓冲 | +| 其他匹名(线程栈 / LuaJIT chunk / runtime) | ~12 MB | 与线程数、已加载适配器数相关 | +| 二进制 text+rodata | ~8 MB | 映射的可执行文件页(只读、可被内核回收) | +| Go runtime 预留 | ~4 MB | VSZ 上看到的 2 GB+ 是地址空间预留,不占物理内存 | +| 共享库 | ~3 MB | libc / libluajit / libm | + +**两个推荐的部署开关**(只适用于环境变量,无需改代码): + +```ini +# /etc/systemd/system/llmsproxy.service +Environment=GOGC=50 +Environment=MALLOC_ARENA_MAX=2 +``` + +- `MALLOC_ARENA_MAX=2`:LuaJIT 的分配走 cgo → glibc malloc,glibc 默认允许 `8×nproc` + 个 per-thread arena,每个碰到 malloc 的 OS 线程会占用一个(各约 1 MB,且**不归还给系统**)。 + 实测从 8–15 个 arena(约 7–12 MB)降到 **0**。 +- `GOGC=50`:把 Go 堆增长目标减半。**单独使用无效**(省下的堆会立即被更多 glibc + arena 吃掉,实测 20.3 → 21.5 MB 反而变大),必须与 `MALLOC_ARENA_MAX` 配合, + 两者同时开启才降 **~19%**。网关是 I/O 密集型(本机 9 小时仅消耗 1min10s CPU), + 多出的 GC 周期与它的空闲 CPU 相比可忽略。 + +> 桌面版(Electron)已在 `cmd/gui/main.js` 里默认为内嵌核心注入这两个开关;手动 export 同名 +> 环境变量可覆盖。如果你用自己的 systemd unit / 容器,建议照上面加上。 + +可选:`GOMEMLIMIT=48MiB` 作软上限,实测再省 ~1 MB;代价是逐近上限时 GC 转激进, +温和场景不必开。 + ### disable_thinking 请求体带 `"disable_thinking": true` 时,网关透传给各适配器;DeepSeek 适配器将其 @@ -369,7 +415,7 @@ return { | 场景 | 推荐 | 理由 | | ------ | ------ | ------ | -| 服务器 / 内网网关 / 无人值守常驻 | **Headless**(单二进制) | 极轻量(~10MB、~15MB RSS),零依赖单进程,直接跑在 systemd/任意容器里,远程管理 | +| 服务器 / 内网网关 / 无人值守常驻 | **Headless**(单二进制) | 极轻量(二进制 ~8MB;RSS 随源数量,单源 ~10MB、16 源 ~40MB,见[内存占用](#内存占用实测与调优)),零依赖单进程,直接跑在 systemd/任意容器里,远程管理 | | 个人桌面日常使用 / 多设备内网共享 | **Desktop**(GUI) | 免登录内嵌 WebUI、系统托盘一键启停、开机自启、静默后台,适合不懂命令行的使用者 | | Windows 桌面 | **Desktop** | 纯后端在 Windows 上需自行注册服务,GUI 提供原生托盘/自启体验 | | CI 一键出三平台安装包 | **Desktop 打包脚本** | `make gui-*` 系列出 deb/AppImage/NSIS,可进发行流水线 | diff --git a/README_EN.md b/README_EN.md index 88faa89..2bf2cda 100644 --- a/README_EN.md +++ b/README_EN.md @@ -11,7 +11,7 @@ KimiCode…) behind a single endpoint. Pick a specific model or use **AUTO** mod which routes to the best healthy upstream by configured priority. > **Lightweight, no recompile to add sources**: the gateway is a single Go -> binary (~10 MB, zero runtime dependencies). Adding or switching an upstream is +> binary (8-12 MB, ~8 MB stripped; zero runtime dependencies). Adding or switching an upstream is > just a `sources` entry in `config.yaml` (or via the WebUI) or a `.lua` > adapter — **no Go changes, no recompile**. Changes submitted through the > WebUI take effect immediately (hot reload); editing `config.yaml` or a `.lua` @@ -41,7 +41,21 @@ Extracted and independently evolved from the multi-source LLM adapter layer of - **Multimodal**: `content` arrays (`image_url` etc.) pass through losslessly; Anthropic/Gemini/Ollama are translated automatically. - **LuaJIT VM**: golua-binding LuaJIT; each adapter has its own VM + worker - pool for safe concurrency. + pool for safe concurrency. Pools are **elastic**: the sum of `max_concurrent` + over an adapter's sources is a ceiling, not a preallocation — states are booted + on demand and reclaimed when demand drops, so an idle gateway holds close to + zero Lua states. The grow step follows the adapter's max concurrency + (`clamp(ceil(max/8), 1, 8)`, and never exceeds the number of queued callers); + the shrink step follows the live connection count + (`clamp(ceil(slack/(1+in_use)), 1, slack)`), so an idle pool collapses in one + round while a busy one gives up a single state at a time. +- **On-demand request logs**: the audit log is never held in memory — the + dashboard loads one screen, scrolling pages the rest straight off disk, CSV + export streams in O(1) memory, and leaving the page releases everything. +- **Self-healing cooldown**: cooldown is capped at 5 minutes and, past the + window's midpoint, exactly one probe request is allowed through; a recovered + upstream (or a reset quota) returns to full rotation on that probe instead of + waiting out the window. - **disable_thinking**: `disable_thinking: true` toggles reasoning per-request. - **Lua adapter protocol**: each source mounts a `.lua` adapter with `transform_request` / `transform_response` / `transform_stream_chunk` — all @@ -258,6 +272,59 @@ independently). A template is just a recipe — it doesn't become a source by itself. Only the key-bearing sources created from it carry real traffic. +### Memory footprint (measured and tuned) + +Memory is not a constant: it scales with the **number of configured sources** +(each source owns an `http.Transport` with its connection pool plus per-model +health state), not with uptime. Measured on this machine (Linux x86_64, 12 cores): + +| Deployment shape | Startup RSS | Settled RSS | +|---|---|---| +| 1 source / 1 adapter (minimal) | ~4 MB | ~10 MB | +| 1 source + a 29 MB audit history | ~19 MB | ~20 MB | +| **16 sources / 13 adapters / 59 models (this host)** | ~28 MB | **~37-42 MB** | + +> For reference, the same production config used **~105 MB** before this round of +> work. The reduction comes from three places: the audit log is no longer replayed +> in full (~25 MB), Lua state pools no longer grow monotonically, and the two +> runtime knobs below. + +Breakdown of the production instance (per-region, from `/proc//smaps`): + +| Region | RSS | Notes | +|---|---|---| +| Go heap | ~14 MB | provider/registry/scheduler state + connection-pool buffers | +| Other anonymous (thread stacks / LuaJIT chunks / runtime) | ~12 MB | scales with thread count and loaded adapters | +| Binary text+rodata | ~8 MB | mapped executable pages (read-only, reclaimable by the kernel) | +| Go runtime reservations | ~4 MB | the 2 GB+ you see in VSZ is address space, not physical memory | +| Shared libraries | ~3 MB | libc / libluajit / libm | + +**Two recommended deployment knobs** (environment only, no code change): + +```ini +# /etc/systemd/system/llmsproxy.service +Environment=GOGC=50 +Environment=MALLOC_ARENA_MAX=2 +``` + +- `MALLOC_ARENA_MAX=2`: LuaJIT allocates through cgo into glibc malloc, and glibc + allows up to `8 x nproc` per-thread arenas. Every OS thread that touches malloc + claims one (~1 MB each) and **never returns it to the OS**. Measured: 8-15 + arenas (7-12 MB) down to **0**. +- `GOGC=50`: halves the Go heap growth target. **It does nothing on its own** — + the heap it saves is immediately consumed by additional glibc arenas (measured + 20.3 -> 21.5 MB, i.e. slightly worse) — so it must be paired with + `MALLOC_ARENA_MAX`. Together they cut settled RSS by **~19%**. The gateway is + I/O bound (1min10s of CPU per 9 hours here), so the extra GC cycles are free. + +> The desktop build (Electron) already injects both when spawning the embedded +> core (`cmd/gui/main.js`); exporting either variable yourself overrides it. If you +> write your own systemd unit or container spec, add them there. + +Optional: `GOMEMLIMIT=48MiB` as a soft ceiling saves roughly 1 MB more, at the +cost of GC turning aggressive as the limit approaches. Not worth it for a relaxed +deployment. + ### disable_thinking With `"disable_thinking": true` in the request body, the gateway passes it to @@ -354,7 +421,7 @@ Release installers ship in two flavors for different audiences: | Scenario | Pick | Why | | ---- | ---- | ---- | -| Server / intranet gateway / unattended long-running | **Headless** (single binary) | ~10 MB, ~15 MB RSS, zero-dep single process — drop it into systemd or any container, remote admin | +| Server / intranet gateway / unattended long-running | **Headless** (single binary) | ~8 MB binary; RSS scales with the number of sources (~10 MB for one, ~40 MB for 16 — see [Memory footprint](#memory-footprint-measured-and-tuned)), zero-dep single process — drop it into systemd or any container, remote admin | | Personal desktop daily use / multi-device intranet sharing | **Desktop** (GUI) | no-login embedded WebUI, tray one-click, autostart, silent background — for non-CLI users | | Windows desktop | **Desktop** | plain backend needs manual service registration; GUI ships native tray/autostart | | CI one-shot 3-platform installers | **Desktop packaging scripts** | `make gui-*` emits deb / AppImage / NSIS, drops straight into a release pipeline | diff --git a/cmd/gui/main.js b/cmd/gui/main.js index f306794..172be6d 100644 --- a/cmd/gui/main.js +++ b/cmd/gui/main.js @@ -198,6 +198,17 @@ function startCore() { cwd: PROFILE_DIR, env: Object.assign({}, process.env, { LLMS_PROXY_GUI: "1", + // Memory tuning, measured on a 16-source deployment (see README 内存占用): + // MALLOC_ARENA_MAX caps glibc per-thread malloc arenas. LuaJIT allocates + // through cgo -> glibc malloc, and glibc defaults to 8*nproc arenas, so + // every OS thread touching malloc reserves its own ~1 MB arena that is + // never returned to the OS. + // GOGC halves the Go heap growth target. On its own it does not help (the + // saved heap is immediately eaten by extra glibc arenas); only the pair + // lowers settled RSS. The core is I/O bound, so the extra GC is free. + // Both are overridable: a user who exports them keeps their own values. + GOGC: process.env.GOGC || "50", + MALLOC_ARENA_MAX: process.env.MALLOC_ARENA_MAX || "2", }), stdio: ["ignore", "pipe", "pipe"], });