fix(gateway): aggregate the full audit history, repair record paging

Two problems reported after the on-demand log work landed.

1. Dashboard totals were wrong. LoadAudit only replayed the last 4 MB of the
   audit file, so requests/tokens/per-key rows reflected a window instead of all
   time — a regression in reported numbers, not just in presentation.

   The aggregates are now built by streaming EVERY audit file (oldest first, so
   the hourly quota buckets keep their intended trailing window) and keeping
   nothing per record: aggregate maps are keyed by key/model/source, so their
   size is bounded by cardinality. Measured on the production host: 29 MB /
   221k lines / 37k requests in ~260 ms at startup.

   What stays bounded is the RAW-record ring: a fixed-size reqRing keeps only the
   newest maxRecs records, so the ~25 MB that used to be spent appending every
   record into a slice is still saved. auditReplayBytes is gone, and
   replayPartial now means "an audit file could not be read", which is the only
   remaining way for the totals to be incomplete.

2. Scrolling to the bottom stopped loading more records. Two independent causes:

   * paintRecords rebuilt the entire table on every 5s poll whenever the row
     count did not exceed the first screen — the "is a paged view live?" test
     compared row counts and matched exactly on the first refresh — wiping loaded
     pages and resetting scroll position.
   * IntersectionObserver only fires on TRANSITIONS. With a short list, or after a
     page whose rows all duplicated the first screen, the sentinel stayed visible
     and never fired again.

   paintRecords now builds once (recsState.built) and later polls PREPEND only
   genuinely new rows; attachRecsObserver adds a scroll-position fallback;
   fillRecordsViewport loads until the list actually overflows; and
   loadMoreRecords chains (bounded) when a page yields no new rows, since the
   first fetch necessarily overlaps the first screen.

TestUIRecordsPagingWiring pins all four mechanisms structurally, since none of
them is reachable from Go. Test names/comments referring to bounded replay are
updated to describe the bounded RING instead, and both READMEs now state that
totals come from the full history while records are paged.
This commit is contained in:
JianFeeeee
2026-08-30 09:29:42 +08:00
parent 2378bc00ba
commit 3ddae41f0c
6 changed files with 409 additions and 143 deletions

View File

@ -17,7 +17,7 @@
- **单二进制**:编译后 8~12MB`-s -w` strip 后约 8MB零运行时依赖仅依赖系统 libc部署即用
- **低内存占用**:与源数量相关,非与运行时长相关——单源 ~10MB、16 源生产实例 ~40MB[实测分解与调优](#内存占用实测与调优)
- **日志按需加载**审计日志不常驻内存——默认只加载首屏下滚自动分页CSV 导出流式写出O(1) 内存),离页即释放
- **日志按需加载**统计数字来自**全量**审计日志启动时流式扫一遍即释放29MB/22 万行约 260ms但原始记录不常驻内存——默认只加载首屏下滚自动分页CSV 导出流式写出O(1) 内存),离页即释放
- **零运行时依赖**:纯 Go + LuaJIT 静态链接,无需安装 Python/Node/Java 等运行时
- **启动极快**:冷启动 < 200ms热重载配置 < 10ms
@ -295,8 +295,9 @@ WebUI 上的"新增/编辑源"、"上传 Lua 适配器"、"改 AUTO 优先级链
| 1 源 + 29 MB 历史审计日志 | ~19 MB | ~20 MB |
| **16 源 / 13 适配器 / 59 模型(本机生产)** | ~28 MB | **~3742 MB** |
> 历史参考:本项优化前同一生产配置为 **~105 MB**。降幅来自三处:审计日志不再全量回放
> (约 25 MB、Lua 状态池不再单调增长、以及下面两个运行时开关。
> 历史参考:本项优化前同一生产配置为 **~105 MB**。降幅来自三处:审计日志的**原始记录**
> 不再常驻内存(约 25 MB;统计聚合仍扫全量,但扫完即释放、Lua 状态池不再单调增长、
> 以及下面两个运行时开关。
内存构成(生产实例分段测量,`/proc/<pid>/smaps`