jianf/ModelRouter
1
0
Fork 0
mirror of https://gitcode.com/JianFeeeee/ModelRouter.git synced 2026-09-24 10:58:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(gateway): load request logs on demand instead of holding them in memory

Browse Source 
Startup RSS on this deployment was 56 MB with a 29 MB audit log and ~10 MB
without one: LoadAudit() json-unmarshalled the ENTIRE file into the aggregates
and kept a 10000-entry ring of raw records. Two more paths had the same shape —
AuditRecords() materialized a whole export window into a []Req before sorting
it, and a dashboard poll serialized the full ring so the browser could render
300 rows of it.

The audit file is now the source of truth and memory only holds the live
window:

  * LoadAudit replays only the last auditReplayBytes (4 MB) and drops the
    truncated first line; the ring default drops 10000 -> 500, which still
    covers both of its consumers (the status page's 5-minute SourceRecent /
    SourceAverages windows and the first screen of the records table).
    replayPartial is exported so the UI can say the totals cover a window
    rather than all time. Token-quota accounting is unaffected: it reads the
    modelHour buckets, not the ring (pinned by a test).
  * AuditPage(cursor, limit, key) pages records straight off disk, reading the
    newest file backwards in 64 KB chunks and returning as soon as the page is
    full. The cursor is "<file>:<offset>" and walks into rotated .old files;
    a cursor whose file rotated away reports rotated=true so the client can
    reset instead of silently skipping records. No state is cached between
    requests and the file handle is closed before responding, so "release when
    the user leaves the page" is guaranteed by never retaining anything.
  * StreamAuditRecords(from,to,key,fn) replaces the accumulate-then-sort export
    path; the CSV handler writes rows as they are read and flushes every 1000,
    and a write error (client gone) aborts the walk. Export memory is O(1)
    regardless of the window. AuditRecords is kept as a test-only wrapper.
  * Snapshot ships one screen (firstScreenRecords=100) by default; aggregates
    are untouched.
  * Audit rotation 64 MB x 10 -> 16 MB x 16: same 256 MB total budget, but a
    smaller newest file keeps the first reverse page cheap.

New route: GET /api/stats/records?before=&limit=&key= (non-admins are pinned to
their own key by exportKey). /api/status additionally reports adapter_pools for
admins.

Measured with production's 29 MB audit copied to the test instance: startup RSS
19.0 MB (was 56 MB); scrolling 10 pages (1000 records) +0.7 MB; exporting the
full history (36441 rows / 4.4 MB CSV) +0.1 MB with no residual growth.
This commit is contained in:
JianFeeeee
2026-08-30 08:05:54 +08:00
parent df9aeed5fb
commit d42c02b15d
5 changed files with 979 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
internal/gateway/server.go
View File

@ -49,12 +49,12 @@ type loginFail struct {
}
const (
loginMaxFails = 5 // failures before lockout kicks in
loginLockBase = 30 // first lockout: 30s
loginLockCap = 30 * 60 // repeated lockouts cap at 30min
loginDecay = 15 * 60 // counters decay after 15min of quiet
loginGlobalMax = 100 // global failures per minute across all IPs
loginGlobalWindow = 60 // global window length (sec)
loginMaxFails = 5 // failures before lockout kicks in
loginLockBase = 30 // first lockout: 30s
loginLockCap = 30 * 60 // repeated lockouts cap at 30min
loginDecay = 15 * 60 // counters decay after 15min of quiet
loginGlobalMax = 100 // global failures per minute across all IPs
loginGlobalWindow = 60 // global window length (sec)
)
// loginAllow checks (and records) a login attempt for ip. It returns false
@ -226,6 +226,8 @@ func (g *Gateway) routes(w http.ResponseWriter, r *http.Request) {
g.handleStatusAPI(w, r)
case r.URL.Path == "/api/status/reset":
g.handleResetHealth(w, r)
case r.URL.Path == "/api/stats/records":
g.handleStatsRecordsAPI(w, r)
case r.URL.Path == "/api/stats" || strings.HasPrefix(r.URL.Path, "/api/stats/"):
g.handleStatsAPI(w, r)
case r.URL.Path == "/api/keys" || strings.HasPrefix(r.URL.Path, "/api/keys/"):
@ -591,6 +593,10 @@ func (g *Gateway) handleStatusAPI(w http.ResponseWriter, r *http.Request) {
}
resp["sources"] = sts
resp["adapters"] = g.core.ListAdapters()
// Elastic Lua pool sizing: created/idle/in_use vs the ceiling, plus the
// current grow/shrink steps, so the algorithm is inspectable in the UI
// instead of being a black box.
resp["adapter_pools"] = g.core.VM().PoolStats()
}
writeJSON(w, http.StatusOK, resp)
}