fix(gateway): aggregate the full audit history, repair record paging

Two problems reported after the on-demand log work landed.

1. Dashboard totals were wrong. LoadAudit only replayed the last 4 MB of the
   audit file, so requests/tokens/per-key rows reflected a window instead of all
   time — a regression in reported numbers, not just in presentation.

   The aggregates are now built by streaming EVERY audit file (oldest first, so
   the hourly quota buckets keep their intended trailing window) and keeping
   nothing per record: aggregate maps are keyed by key/model/source, so their
   size is bounded by cardinality. Measured on the production host: 29 MB /
   221k lines / 37k requests in ~260 ms at startup.

   What stays bounded is the RAW-record ring: a fixed-size reqRing keeps only the
   newest maxRecs records, so the ~25 MB that used to be spent appending every
   record into a slice is still saved. auditReplayBytes is gone, and
   replayPartial now means "an audit file could not be read", which is the only
   remaining way for the totals to be incomplete.

2. Scrolling to the bottom stopped loading more records. Two independent causes:

   * paintRecords rebuilt the entire table on every 5s poll whenever the row
     count did not exceed the first screen — the "is a paged view live?" test
     compared row counts and matched exactly on the first refresh — wiping loaded
     pages and resetting scroll position.
   * IntersectionObserver only fires on TRANSITIONS. With a short list, or after a
     page whose rows all duplicated the first screen, the sentinel stayed visible
     and never fired again.

   paintRecords now builds once (recsState.built) and later polls PREPEND only
   genuinely new rows; attachRecsObserver adds a scroll-position fallback;
   fillRecordsViewport loads until the list actually overflows; and
   loadMoreRecords chains (bounded) when a page yields no new rows, since the
   first fetch necessarily overlaps the first screen.

TestUIRecordsPagingWiring pins all four mechanisms structurally, since none of
them is reachable from Go. Test names/comments referring to bounded replay are
updated to describe the bounded RING instead, and both READMEs now state that
totals come from the full history while records are paged.

README.md

@@ -17,7 +17,7 @@
 
 - **单二进制**：编译后 8~12MB（`-s -w` strip 后约 8MB），零运行时依赖（仅依赖系统 libc），部署即用
 - **低内存占用**：与源数量相关，非与运行时长相关——单源 ~10MB、16 源生产实例 ~40MB（[实测分解与调优](#内存占用实测与调优)）
-- **日志按需加载**：审计日志不常驻内存——默认只加载首屏，下滚自动分页，CSV 导出流式写出（O(1) 内存），离页即释放
+- **日志按需加载**：统计数字来自**全量**审计日志（启动时流式扫一遍即释放，29MB/22 万行约 260ms），但原始记录不常驻内存——默认只加载首屏，下滚自动分页，CSV 导出流式写出（O(1) 内存），离页即释放
 - **零运行时依赖**：纯 Go + LuaJIT 静态链接，无需安装 Python/Node/Java 等运行时
 - **启动极快**：冷启动 < 200ms，热重载配置 < 10ms
 
@@ -295,8 +295,9 @@ WebUI 上的"新增/编辑源"、"上传 Lua 适配器"、"改 AUTO 优先级链
 | 1 源 + 29 MB 历史审计日志 | ~19 MB | ~20 MB |
 | **16 源 / 13 适配器 / 59 模型（本机生产）** | ~28 MB | **~37–42 MB** |
 
-> 历史参考：本项优化前同一生产配置为 **~105 MB**。降幅来自三处：审计日志不再全量回放
-> （约 25 MB）、Lua 状态池不再单调增长、以及下面两个运行时开关。
+> 历史参考：本项优化前同一生产配置为 **~105 MB**。降幅来自三处：审计日志的**原始记录**
+> 不再常驻内存（约 25 MB；统计聚合仍扫全量，但扫完即释放）、Lua 状态池不再单调增长、
+> 以及下面两个运行时开关。
 
 内存构成（生产实例分段测量，`/proc/<pid>/smaps`）：
 

README_EN.md

@@ -49,9 +49,11 @@ Extracted and independently evolved from the multi-source LLM adapter layer of
   the shrink step follows the live connection count
   (`clamp(ceil(slack/(1+in_use)), 1, slack)`), so an idle pool collapses in one
   round while a busy one gives up a single state at a time.
-- **On-demand request logs**: the audit log is never held in memory — the
-  dashboard loads one screen, scrolling pages the rest straight off disk, CSV
-  export streams in O(1) memory, and leaving the page releases everything.
+- **On-demand request logs**: dashboard totals are computed from the **full**
+  audit history (streamed once at startup and released — 29 MB / 221k lines in
+  ~260 ms), while the raw records are never held in memory: the dashboard loads
+  one screen, scrolling pages the rest straight off disk, CSV export streams in
+  O(1) memory, and leaving the page releases everything.
 - **Self-healing cooldown**: cooldown is capped at 5 minutes and, past the
   window's midpoint, exactly one probe request is allowed through; a recovered
   upstream (or a reset quota) returns to full rotation on that probe instead of
@@ -285,8 +287,9 @@ health state), not with uptime. Measured on this machine (Linux x86_64, 12 cores
 | **16 sources / 13 adapters / 59 models (this host)** | ~28 MB | **~37-42 MB** |
 
 > For reference, the same production config used **~105 MB** before this round of
-> work. The reduction comes from three places: the audit log is no longer replayed
-> in full (~25 MB), Lua state pools no longer grow monotonically, and the two
+> work. The reduction comes from three places: raw audit records are no longer kept
+> resident (~25 MB — the aggregates still scan the full history, but the scan
+> releases as it goes), Lua state pools no longer grow monotonically, and the two
 > runtime knobs below.
 
 Breakdown of the production instance (per-region, from `/proc/<pid>/smaps`):

internal/gateway/stats.go

@@ -6,6 +6,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
+	"log"
 	"os"
 	"path/filepath"
 	"sort"
@@ -100,14 +101,6 @@ var (
 	auditKeepOld           = 16
 )
 
-// auditReplayBytes bounds how much of the audit tail is replayed at startup.
-// Replaying the WHOLE file used to cost ~25 MB of resident memory for a 29 MB
-// audit log, which dominated the process RSS. Only the recent window belongs in
-// memory: older records stay on disk and are paged in on demand by AuditPage /
-// StreamAuditRecords, while long-term totals come from the aggregate snapshot
-// instead of a full replay.
-var auditReplayBytes int64 = 4 << 20
-
 // defaultRingSize is how many recent request records stay resident. It has to
 // cover two consumers: the status page's 5-minute source windows
 // (SourceRecent/SourceAverages) and the first screen of the records table.
@@ -175,68 +168,132 @@ func incStatus(a *Stat, name string, r Req) {
 	}
 }
 
-// LoadAudit primes the in-memory state from the audit file. Only the last
-// auditReplayBytes are replayed: the ring buffer and the recent-window
-// aggregates need the tail, and paging older records is what AuditPage and
-// StreamAuditRecords are for. This keeps startup memory proportional to the
-// replay window instead of to the (unbounded) audit file.
+// LoadAudit primes the in-memory state from the audit file.
+//
+// The aggregates (totals, per-key/model/source rows, status counts and the
+// hourly quota buckets) are built from the FULL history: every audit file is
+// streamed oldest-first so the dashboard shows real all-time numbers rather than
+// whatever happened to fit in a replay window. This is affordable because the
+// scan keeps nothing per record — aggregate maps are keyed by key/model/source,
+// so their size is bounded by cardinality, not by request count. Measured on the
+// production host: 29 MB / 221k lines / 37k requests in ~260 ms.
+//
+// The raw-record ring is what stays bounded: only the newest maxRecs records are
+// retained, and everything older is paged from disk on demand by AuditPage /
+// StreamAuditRecords. The old behaviour — appending EVERY record into a slice
+// and truncating at the end — is what cost ~25 MB of resident memory.
 func (s *Stats) LoadAudit(path string) {
 	s.mu.Lock()
 	s.auditPath = path
 	s.mu.Unlock()
 
-	f, err := os.Open(path)
-	if err != nil {
-		return
-	}
-	defer f.Close()
-	fi, err := f.Stat()
-	if err != nil {
-		return
-	}
-	offset := int64(0)
-	partial := false
-	if fi.Size() > auditReplayBytes {
-		offset = fi.Size() - auditReplayBytes
-		partial = true // the first line is very likely cut in half
-	}
-	if _, err := f.Seek(offset, io.SeekStart); err != nil {
-		return
+	// Oldest-first: the hourly-bucket retention prunes relative to the newest
+	// hour seen so far, so replaying in chronological order keeps exactly the
+	// intended trailing window.
+	chain := s.auditChain()
+	files := make([]string, 0, len(chain))
+	for i := len(chain) - 1; i >= 0; i-- {
+		files = append(files, chain[i])
 	}
 
+	ring := newReqRing(s.ringSize())
+	scanned := 0
+	s.mu.Lock()
+	for _, p := range files {
+		n, err := scanAuditFile(p, func(r Req) {
+			s.aggregateLocked(r)
+			ring.push(r)
+		})
+		scanned += n
+		if err != nil {
+			// A truncated or unreadable tail is not fatal: keep whatever was
+			// aggregated and mark the numbers as incomplete.
+			s.replayPartial = true
+		}
+	}
+	s.recs = ring.slice()
+	s.mu.Unlock()
+
+	if scanned > 0 {
+		log.Printf("[stats] replayed %d audit records from %d file(s) for aggregates; keeping the newest %d in memory",
+			scanned, len(files), len(s.recs))
+	}
+}
+
+func (s *Stats) ringSize() int {
 	s.mu.Lock()
 	defer s.mu.Unlock()
-	var recs []Req
+	if s.maxRecs <= 0 {
+		return defaultRingSize
+	}
+	return s.maxRecs
+}
+
+// scanAuditFile streams one audit file, invoking fn for every request row, and
+// returns how many request rows it saw. Access/event rows and malformed lines
+// are skipped.
+func scanAuditFile(path string, fn func(Req)) (int, error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return 0, err
+	}
+	defer f.Close()
 	sc := bufio.NewScanner(f)
 	// tolerate long error summaries / oversized junk lines
 	sc.Buffer(make([]byte, 64*1024), 16*1024*1024)
-	first := true
+	n := 0
 	for sc.Scan() {
-		if first {
-			first = false
-			if partial {
-				continue // drop the truncated first record
-			}
-		}
 		var r Req
 		if json.Unmarshal(sc.Bytes(), &r) != nil || r.Type == "" {
-			continue // access/event rows (obj) and malformed lines are no requests
-		}
-		s.aggregateLocked(r)
-		recs = append(recs, r)
-		if len(recs) > s.maxRecs {
-			// keep the replay bounded in memory as well as on disk: aggregates
-			// already absorbed the dropped record
-			recs = recs[len(recs)-s.maxRecs:]
+			continue
 		}
+		fn(r)
+		n++
 	}
-	s.recs = recs
-	s.replayPartial = partial
+	return n, sc.Err()
 }
 
-// ReplayPartial reports whether the resident aggregates were built from a
-// bounded tail of the audit file rather than its full history, so the UI can say
-// so instead of implying the numbers are all-time totals.
+// reqRing keeps the newest n records seen, in chronological order, without
+// growing with the number of records pushed through it.
+type reqRing struct {
+	buf   []Req
+	next  int
+	full  bool
+	limit int
+}
+
+func newReqRing(n int) *reqRing {
+	if n <= 0 {
+		n = defaultRingSize
+	}
+	return &reqRing{buf: make([]Req, n), limit: n}
+}
+
+func (r *reqRing) push(rec Req) {
+	r.buf[r.next] = rec
+	r.next++
+	if r.next == r.limit {
+		r.next = 0
+		r.full = true
+	}
+}
+
+// slice returns the retained records oldest-first.
+func (r *reqRing) slice() []Req {
+	if !r.full {
+		out := make([]Req, r.next)
+		copy(out, r.buf[:r.next])
+		return out
+	}
+	out := make([]Req, 0, r.limit)
+	out = append(out, r.buf[r.next:]...)
+	out = append(out, r.buf[:r.next]...)
+	return out
+}
+
+// ReplayPartial reports whether the resident aggregates are known to be
+// incomplete (an audit file could not be read in full). Under normal operation
+// the aggregates cover the entire audit history, so this is false.
 func (s *Stats) ReplayPartial() bool {
 	s.mu.Lock()
 	defer s.mu.Unlock()

internal/gateway/stats_test.go

@@ -179,50 +179,110 @@ func writeAuditFile(t *testing.T, path string, n int, key string, startMs int64)
 	}
 }
 
-// TestLoadAuditBoundedTail is the startup-memory fix: only the tail of a large
-// audit file is replayed, and the ring stays capped.
-func TestLoadAuditBoundedTail(t *testing.T) {
+// TestLoadAuditAggregatesFullHistory is the correctness half of the log work:
+// the dashboard's totals must cover EVERY audit record, however large the file,
+// while the in-memory record ring stays bounded.
+func TestLoadAuditAggregatesFullHistory(t *testing.T) {
 	dir := t.TempDir()
 	path := filepath.Join(dir, "audit.jsonl")
-	writeAuditFile(t, path, 20000, "k1", 1_700_000_000_000)
-
-	fi, err := os.Stat(path)
-	if err != nil {
-		t.Fatal(err)
-	}
-	orig := auditReplayBytes
-	auditReplayBytes = 64 << 10 // 64 KB tail out of a much larger file
-	defer func() { auditReplayBytes = orig }()
-	if fi.Size() <= auditReplayBytes {
-		t.Fatalf("fixture too small (%d bytes) to exercise bounded replay", fi.Size())
-	}
+	const n = 20000
+	writeAuditFile(t, path, n, "k1", 1_700_000_000_000)
 
 	s := NewStats(0)
 	s.LoadAudit(path)
-	if !s.ReplayPartial() {
-		t.Fatal("a truncated replay must be reported as partial")
+	if s.ReplayPartial() {
+		t.Fatal("a readable history must not be reported as partial")
 	}
+
 	snap := s.Snapshot(0, "")
+	total, _ := snap["total"].(Stat)
+	if total.Reqs != n {
+		t.Fatalf("total reqs = %d, want %d (aggregates must cover the whole file)", total.Reqs, n)
+	}
+	if total.Tokens != int64(n)*2 { // 1 prompt + 1 completion per row
+		t.Fatalf("total tokens = %d, want %d", total.Tokens, n*2)
+	}
+
+	// ...while the resident ring is capped and the snapshot ships one screen
 	recs, _ := snap["records"].([]Req)
 	if len(recs) == 0 {
-		t.Fatal("bounded replay must still load the recent window")
+		t.Fatal("the newest records must stay resident")
 	}
 	if len(recs) > firstScreenRecords {
 		t.Fatalf("snapshot shipped %d records, must be capped at the first screen (%d)",
 			len(recs), firstScreenRecords)
 	}
-	// the aggregates must come from the tail only, not the whole file
-	total, _ := snap["total"].(Stat)
-	if total.Reqs == 0 {
-		t.Fatal("tail replay must feed the aggregates")
+	s.mu.Lock()
+	ring := len(s.recs)
+	s.mu.Unlock()
+	if ring > defaultRingSize {
+		t.Fatalf("resident ring holds %d records, must be capped at %d", ring, defaultRingSize)
 	}
-	if total.Reqs >= 20000 {
-		t.Fatalf("aggregates replayed the whole file (%d reqs); replay must be bounded", total.Reqs)
+
+	// the ring must hold the NEWEST records, not the first ones scanned
+	newest := recs[len(recs)-1]
+	wantNewest := int64(1_700_000_000_000) + int64(n-1)*1000
+	if newest.Time != wantNewest {
+		t.Fatalf("newest resident record time = %d, want %d", newest.Time, wantNewest)
 	}
 }
 
-// TestLoadAuditFullFileWhenSmall: a file inside the replay window is loaded
-// completely and is not flagged partial.
+// TestLoadAuditAggregatesAcrossRotatedFiles: history split over rotated .old
+// files must still add up, and must be replayed oldest-first so the hourly
+// quota buckets keep the intended trailing window.
+func TestLoadAuditAggregatesAcrossRotatedFiles(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "audit.jsonl")
+	writeAuditFile(t, path, 40, "k1", 1_700_000_500_000)            // newest
+	writeAuditFile(t, path+".100.old", 40, "k1", 1_700_000_300_000) // older
+	writeAuditFile(t, path+".050.old", 40, "k1", 1_700_000_100_000) // oldest
+
+	s := NewStats(0)
+	s.LoadAudit(path)
+	snap := s.Snapshot(0, "")
+	total, _ := snap["total"].(Stat)
+	if total.Reqs != 120 {
+		t.Fatalf("total reqs = %d, want 120 across three files", total.Reqs)
+	}
+	recs, _ := snap["records"].([]Req)
+	if len(recs) == 0 {
+		t.Fatal("expected resident records")
+	}
+	// oldest-first replay + newest-last ring => the last resident record is the
+	// newest one in the live file
+	if got := recs[len(recs)-1].Time; got != 1_700_000_500_000+39*1000 {
+		t.Fatalf("newest resident record time = %d, want the live file's last row", got)
+	}
+}
+
+// TestReqRingKeepsNewest pins the ring semantics used by the full-history scan:
+// bounded size, newest retained, oldest-first output.
+func TestReqRingKeepsNewest(t *testing.T) {
+	r := newReqRing(3)
+	for i := 1; i <= 7; i++ {
+		r.push(Req{Time: int64(i)})
+	}
+	got := r.slice()
+	if len(got) != 3 {
+		t.Fatalf("ring len = %d, want 3", len(got))
+	}
+	for i, want := range []int64{5, 6, 7} {
+		if got[i].Time != want {
+			t.Fatalf("ring[%d] = %d, want %d (oldest-first, newest retained)", i, got[i].Time, want)
+		}
+	}
+
+	// a partially filled ring returns exactly what was pushed
+	r2 := newReqRing(5)
+	r2.push(Req{Time: 1})
+	r2.push(Req{Time: 2})
+	if got := r2.slice(); len(got) != 2 || got[0].Time != 1 || got[1].Time != 2 {
+		t.Fatalf("partial ring = %+v", got)
+	}
+}
+
+// TestLoadAuditFullFileWhenSmall: a small file is loaded completely and is not
+// flagged partial.
 func TestLoadAuditFullFileWhenSmall(t *testing.T) {
 	path := filepath.Join(t.TempDir(), "audit.jsonl")
 	writeAuditFile(t, path, 50, "k1", 1_700_000_000_000)
@@ -238,10 +298,10 @@ func TestLoadAuditFullFileWhenSmall(t *testing.T) {
 	}
 }
 
-// TestQuotaWindowsSurviveBoundedReplay is the regression guard the plan calls
-// out: shrinking the ring must not break token-quota accounting, which reads the
-// hour buckets rather than the ring.
-func TestQuotaWindowsSurviveBoundedReplay(t *testing.T) {
+// TestQuotaWindowsSurviveBoundedRing is the regression guard the plan calls out:
+// capping the resident record ring must not break token-quota accounting, which
+// reads the hour buckets rather than the ring.
+func TestQuotaWindowsSurviveBoundedRing(t *testing.T) {
 	s := NewStats(10) // deliberately tiny ring
 	now := time.Now().UnixMilli()
 	for i := 0; i < 100; i++ {

internal/gateway/ui/index.html

@@ -927,7 +927,7 @@
           recsRotated: "审计日志已轮转，已从最新记录重新加载",
           recsNewest: "回到最新",
           recsPartial:
-            "统计基于最近一段审计日志（启动时只回放尾部）；完整历史请导出 CSV",
+            "部分审计日志无法读取，统计可能不完整；完整历史请导出 CSV",
           thTokens: "Tokens",
           noUsage: "暂无用量记录",
         },
@@ -1153,7 +1153,7 @@
           recsRotated: "The audit log rotated; reloaded from the newest record",
           recsNewest: "Back to newest",
           recsPartial:
-            "Aggregates cover the recent audit tail replayed at startup; export CSV for the full history",
+            "Some audit files could not be read, so these totals may be incomplete; export CSV for the full history",
           thTokens: "Tokens",
           noUsage: "No usage data yet",
         },
@@ -1406,7 +1406,9 @@
         cursor: "", // next-page cursor
         hasMore: false,
         loading: false,
+        built: false, // table DOM exists (polls prepend instead of rebuilding)
         observer: null, // IntersectionObserver on the bottom sentinel
+        onScroll: null, // scroll fallback (observers only fire on transitions)
         abort: null, // AbortController for the in-flight page fetch
         keyNames: {},
       };
@@ -1418,6 +1420,10 @@
           recsState.observer.disconnect();
           recsState.observer = null;
         }
+        const root = $("#tb-recs");
+        if (root && recsState.onScroll)
+          root.removeEventListener("scroll", recsState.onScroll);
+        recsState.onScroll = null;
         if (recsState.abort) {
           recsState.abort.abort();
           recsState.abort = null;
@@ -1426,11 +1432,9 @@
         recsState.cursor = "";
         recsState.hasMore = false;
         recsState.loading = false;
+        recsState.built = false;
         recsState.keyNames = {};
-        if (clearDom) {
-          const el = $("#tb-recs");
-          if (el) el.innerHTML = "";
-        }
+        if (clearDom && root) root.innerHTML = "";
       }
 
       // ---- Source status column (refreshable) ----
@@ -2101,36 +2105,95 @@
     <th class="num">${t("thPrompt")}</th><th class="num">${t("thCompl")}</th><th class="num">${t("thCache") || "缓存"}</th><th class="num">${t("thLatMs")}</th></tr>`;
       }
 
-      // paintRecords renders the FIRST SCREEN from the dashboard poll. Rows are
-      // newest-first. Once the user has scrolled (extra pages loaded) the poll no
-      // longer rebuilds the table, otherwise every refresh would throw the loaded
-      // history away.
+      // paintRecords renders the record table from the dashboard poll.
+      //
+      // It builds the table ONCE. Later polls only PREPEND records that arrived
+      // since the last paint: rebuilding on every 5s refresh used to wipe the
+      // pages the user had scrolled in and reset the scroll position (which made
+      // paging look broken), because the "is the paged view live?" check compared
+      // row counts and matched exactly on the first refresh.
       function paintRecords(records, keyNames) {
         const el = $("#tb-recs");
         if (!el) return;
         recsState.keyNames = keyNames || {};
-        const first = (records || []).slice().reverse(); // API sends oldest-first
-        if (recsState.rows.length > first.length) {
-          // paged view is live: only refresh the key names, keep the loaded rows
+        const incoming = (records || []).slice().reverse(); // API sends oldest-first
+
+        if (!recsState.built) {
+          if (!incoming.length) {
+            recsState.rows = [];
+            el.innerHTML = `<div class="muted">${t("noUsage")}</div>`;
+            return;
+          }
+          recsState.rows = incoming;
+          recsState.cursor = "";
+          recsState.hasMore = true; // probed on the first page fetch
+          el.innerHTML =
+            `<table id="recs-tbl">${recHeadHtml()}<tbody id="recs-body">` +
+            incoming.map((r) => recRowHtml(r, recsState.keyNames)).join("") +
+            `</tbody></table><div id="recs-sentinel" class="recs-sentinel"></div>`;
+          recsState.built = true;
+          attachRecsObserver();
+          fillRecordsViewport();
           return;
         }
-        if (!first.length) {
-          recsState.rows = [];
-          el.innerHTML = `<div class="muted">${t("noUsage")}</div>`;
-          return;
+
+        // Already built: splice in anything newer than our newest row.
+        const body = $("#recs-body");
+        if (!body || !incoming.length) return;
+        const seen = new Set(recsState.rows.map(recId));
+        const fresh = incoming.filter((r) => !seen.has(recId(r)));
+        if (!fresh.length) return;
+        body.insertAdjacentHTML(
+          "afterbegin",
+          fresh.map((r) => recRowHtml(r, recsState.keyNames)).join(""),
+        );
+        recsState.rows = fresh.concat(recsState.rows);
+        trimRecordsDom(body);
+      }
+
+      // recId identifies a record for de-duplication across the snapshot and the
+      // paged endpoint (both can return the same row).
+      function recId(r) {
+        return (
+          r.time + "|" + r.key + "|" + r.model + "|" + r.type + "|" + r.status
+        );
+      }
+
+      // trimRecordsDom enforces the sliding window: drop the oldest rendered rows
+      // (bottom of the table) so a long scroll cannot grow the DOM without bound.
+      function trimRecordsDom(body) {
+        const over = recsState.rows.length - RECS_MAX_DOM;
+        if (over <= 0) return;
+        for (let i = 0; i < over && body.lastElementChild; i++)
+          body.removeChild(body.lastElementChild);
+        recsState.rows = recsState.rows.slice(0, RECS_MAX_DOM);
+      }
+
+      // fillRecordsViewport loads pages until the list actually overflows its
+      // container. Without this, a short table leaves the sentinel permanently
+      // visible: IntersectionObserver only fires on TRANSITIONS, so it would
+      // never fire again and scrolling could never trigger a load.
+      async function fillRecordsViewport() {
+        const el = $("#tb-recs");
+        if (!el) return;
+        let guard = 0;
+        while (
+          recsState.built &&
+          recsState.hasMore !== false &&
+          el.scrollHeight <= el.clientHeight + 8 &&
+          guard++ < 5
+        ) {
+          const before = recsState.rows.length;
+          await loadMoreRecords();
+          if (recsState.rows.length === before) break; // no progress: stop
         }
-        recsState.rows = first;
-        recsState.cursor = "";
-        recsState.hasMore = true; // probed on the first scroll
-        el.innerHTML =
-          `<table id="recs-tbl">${recHeadHtml()}<tbody id="recs-body">` +
-          first.map((r) => recRowHtml(r, recsState.keyNames)).join("") +
-          `</tbody></table><div id="recs-sentinel" class="recs-sentinel"></div>`;
-        attachRecsObserver();
       }
 
       // attachRecsObserver watches a sentinel below the last row; entering the
-      // viewport loads the next page. The observer is torn down by
+      // viewport loads the next page. A scroll listener backs the observer up:
+      // IntersectionObserver only fires on transitions, so a sentinel that is
+      // already visible (short list, or a page that added no new rows) would
+      // otherwise never trigger another load. Both are torn down by
       // releaseRecords when the view goes away.
       function attachRecsObserver() {
         if (recsState.observer) {
@@ -2139,26 +2202,39 @@
         }
         const root = $("#tb-recs");
         const sentinel = $("#recs-sentinel");
-        if (!root || !sentinel || typeof IntersectionObserver === "undefined")
-          return;
-        recsState.observer = new IntersectionObserver(
-          (entries) => {
-            if (entries.some((e) => e.isIntersecting)) loadMoreRecords();
-          },
-          { root, rootMargin: "120px" },
-        );
-        recsState.observer.observe(sentinel);
+        if (!root || !sentinel) return;
+        if (typeof IntersectionObserver !== "undefined") {
+          recsState.observer = new IntersectionObserver(
+            (entries) => {
+              if (entries.some((e) => e.isIntersecting)) loadMoreRecords();
+            },
+            { root, rootMargin: "120px" },
+          );
+          recsState.observer.observe(sentinel);
+        }
+        if (recsState.onScroll) root.removeEventListener("scroll", recsState.onScroll);
+        recsState.onScroll = () => {
+          if (root.scrollTop + root.clientHeight >= root.scrollHeight - 120)
+            loadMoreRecords();
+        };
+        root.addEventListener("scroll", recsState.onScroll, { passive: true });
       }
 
       // loadMoreRecords fetches the next page and appends it. Concurrent calls
       // are ignored, and a stale response (view released meanwhile) is dropped.
-      async function loadMoreRecords() {
+      //
+      // A page that yields no NEW rows (the first fetch overlaps the dashboard's
+      // first screen) chains straight into the following page, bounded by
+      // maxChain, so the user never has to scroll twice for one page of data.
+      async function loadMoreRecords(depth) {
+        const maxChain = 3;
         if (recsState.loading || recsState.hasMore === false) return;
         recsState.loading = true;
         const sentinel = $("#recs-sentinel");
         if (sentinel) sentinel.textContent = t("recsLoading");
         const ctrl = new AbortController();
         recsState.abort = ctrl;
+        let added = 0;
         try {
           let q =
             "/api/stats/records?limit=" +
@@ -2168,7 +2244,7 @@
             q += "&before=" + encodeURIComponent(recsState.cursor);
           const page = await api(q, { signal: ctrl.signal });
           if (recsState.abort !== ctrl) return; // released while in flight
-          appendRecords(page);
+          added = appendRecords(page);
         } catch (e) {
           if (e && e.name === "AbortError") return;
           console.error("[records]", e);
@@ -2180,39 +2256,35 @@
           if (s)
             s.textContent = recsState.hasMore === false ? t("recsEnd") : "";
         }
+        if (added === 0 && recsState.hasMore && (depth || 0) < maxChain)
+          await loadMoreRecords((depth || 0) + 1);
       }
 
       // appendRecords merges one page into the table, skipping records already on
-      // screen (the first page overlaps the dashboard's first screen) and
-      // trimming the oldest DOM rows past RECS_MAX_DOM so the table cannot grow
-      // without bound during a long scroll.
+      // screen and trimming the oldest DOM rows past RECS_MAX_DOM.
+      //
+      // The first page fetch necessarily overlaps the dashboard's first screen, so
+      // it can legitimately yield ZERO new rows. When that happens the cursor has
+      // still advanced, so we immediately pull the next page instead of waiting
+      // for another scroll event that may never come.
       function appendRecords(page) {
         const body = $("#recs-body");
-        if (!body) return;
+        if (!body) return 0;
         if (page.key_names) recsState.keyNames = page.key_names;
-        const seen = new Set(
-          recsState.rows.map((r) => r.time + "|" + r.key + "|" + r.model),
-        );
-        const fresh = (page.records || []).filter(
-          (r) => !seen.has(r.time + "|" + r.key + "|" + r.model),
-        );
+        const seen = new Set(recsState.rows.map(recId));
+        const fresh = (page.records || []).filter((r) => !seen.has(recId(r)));
         if (fresh.length) {
           body.insertAdjacentHTML(
             "beforeend",
             fresh.map((r) => recRowHtml(r, recsState.keyNames)).join(""),
           );
           recsState.rows = recsState.rows.concat(fresh);
-          // sliding window: drop the oldest rendered rows (bottom of the table)
-          const over = recsState.rows.length - RECS_MAX_DOM;
-          if (over > 0) {
-            for (let i = 0; i < over && body.lastElementChild; i++)
-              body.removeChild(body.lastElementChild);
-            recsState.rows = recsState.rows.slice(0, RECS_MAX_DOM);
-          }
+          trimRecordsDom(body);
         }
         recsState.cursor = page.next_cursor || "";
         recsState.hasMore = !!page.has_more && !!page.next_cursor;
         if (page.rotated) toast(t("recsRotated"));
+        return fresh.length;
       }
 
       // cacheCell renders the per-request cache-hit column: a percentage when

internal/gateway/ui_contract_test.go

@@ -176,3 +176,76 @@ func oneLine(s string) string {
 	}
 	return s
 }
+
+// TestUIRecordsPagingWiring guards the record table's paging plumbing. Two
+// separate bugs made "scroll to the bottom" silently stop loading:
+//
+//  1. paintRecords rebuilt the whole table on every 5s poll whenever the row
+//     count did not exceed the first screen, wiping loaded pages and resetting
+//     scroll position.
+//  2. IntersectionObserver only fires on TRANSITIONS. With a short list — or a
+//     page whose rows all duplicated the first screen — the sentinel stayed
+//     visible and never fired again.
+//
+// The fixes are a build-once flag, a scroll-position fallback, an initial
+// viewport fill, and chaining when a page yields no new rows. None of that is
+// reachable from Go, so the wiring is asserted structurally.
+func TestUIRecordsPagingWiring(t *testing.T) {
+	src := uiSource(t)
+
+	paint, ok := jsFunctionBody(src, "paintRecords")
+	if !ok {
+		t.Fatal("paintRecords() not found")
+	}
+	if !strings.Contains(paint, "recsState.built") {
+		t.Error("paintRecords must build the table once (recsState.built) instead of rebuilding on every poll")
+	}
+	if !strings.Contains(paint, "afterbegin") {
+		t.Error("paintRecords must PREPEND newer rows on later polls, not rebuild the table")
+	}
+	if !strings.Contains(paint, "fillRecordsViewport") {
+		t.Error("paintRecords must fill the viewport so the sentinel can transition")
+	}
+
+	attach, ok := jsFunctionBody(src, "attachRecsObserver")
+	if !ok {
+		t.Fatal("attachRecsObserver() not found")
+	}
+	if !strings.Contains(attach, "IntersectionObserver") {
+		t.Error("attachRecsObserver must still use IntersectionObserver")
+	}
+	if !strings.Contains(attach, `addEventListener("scroll"`) {
+		t.Error("attachRecsObserver needs a scroll fallback: an already-visible sentinel never fires again")
+	}
+
+	load, ok := jsFunctionBody(src, "loadMoreRecords")
+	if !ok {
+		t.Fatal("loadMoreRecords() not found")
+	}
+	if !strings.Contains(load, "maxChain") {
+		t.Error("loadMoreRecords must chain when a page adds no new rows (the first page overlaps the first screen)")
+	}
+	if !strings.Contains(load, "signal: ctrl.signal") {
+		t.Error("loadMoreRecords must remain abortable")
+	}
+
+	fill, ok := jsFunctionBody(src, "fillRecordsViewport")
+	if !ok {
+		t.Fatal("fillRecordsViewport() not found")
+	}
+	for _, want := range []string{"scrollHeight", "clientHeight"} {
+		if !strings.Contains(fill, want) {
+			t.Errorf("fillRecordsViewport must compare %s to decide whether the list overflows", want)
+		}
+	}
+
+	rel, ok := jsFunctionBody(src, "releaseRecords")
+	if !ok {
+		t.Fatal("releaseRecords() not found")
+	}
+	for _, want := range []string{"observer.disconnect()", "removeEventListener", "abort()", "recsState.built = false"} {
+		if !strings.Contains(rel, want) {
+			t.Errorf("releaseRecords must clean up %s", want)
+		}
+	}
+}