mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-10-15 12:14:54 +00:00
feat(stats): 用量按日/周/月/全部统计（审计文件聚合）
统计页原来只有一个视图——进程启动以来的累计。早上没人和一整周没人看起来 一模一样。加日/周/月/总四个周期。 ## 口径与实现 周期视图必须走审计文件，不能走内存聚合：内存 byModel/byKey 等是终身累计， 而 recs 环形缓冲只有 500 条（defaultRingSize）。读环会把任何超过几百个 请求的周期悄悄少算——这正是要消除的那类错数。 - 日/周/月 = UTC 日历窗口（今日 / ISO 周周一 00:00 / 本月 1 日）。 刻意不用滚动 24h：滚动窗口会让"今天"和"最近一天"边界不同，同一个数字 随查看时刻在两张卡片间跳。UTC 也和 billing 的峰段计算同口径，峰谷小时 不会在费用视图和用量视图里落到不同一天。 - 全部 = 复用现有 Snapshot（内存聚合），无审计文件时依然可用。 - 时间桶：日→每小时（今天内部的尖峰要看得见），周/月→每天（否则一周是 7×24 个点、一个月 31×24）。全部视图无时间线（终身总量没有有意义的 时间轴，硬画 500 个滚动小时点是另一种撒谎）。 - key 过滤在所有维度生效；非法 period 返回 400 而不是静默回落"全部"—— 书签里的手误应当报错，而不是悄悄换成终身数字。 ## 判据（10 条 + 8 个变异全部被捕获） 窗口边界（含"周日必须回到上一个周一"这个 Go Weekday() 陷阱）、旧记录不 计入、维度独立聚合且 by_model 求和等于 total、日桶按小时且有序、key 隔离、 全部视图走终身、空窗口标记 truncated、period 校验、query 解析。 变异验证时 by_status 假绿了一次：禁用状态码聚合后判据全过，查下去是我 **根本没测 by_status**（零覆盖）。补 TestPeriodStatusDimension 后该变异 立即被捕获。判据报假问题时，先怀疑判据——这次确实是我错了。 ## 真实流量核对 生产审计文件手算 vs 后端（含轮转文件）： day 手算 3559 / 后端 3532 week 手算 43240 / 后端 35034 month 手算 13696 / 后端 13670 差异是核对快照与请求之间的新流量，量级一致。 一个必须说明的发现：审计文件里混着两种记录 —— Req（type/model/ prompt_tokens）和访问日志（lat_ms/status/path）。46026 行里 33450 行是 访问日志，Go 侧按 r.Type=="" 跳过。这不是 bug（CSV 导出同样如此），但 意味着任何按行数手算都必须过滤，否则会差一个数量级。 CDP 实测四周期切换：reqs 3,571 / 35,075 / 13,710 / 196,712，与后端一致， 无控制台错误，localStorage 持久化生效。
This commit is contained in:
JianFeeeee
234
internal/gateway/stats_period.go
Normal file
234
internal/gateway/stats_period.go
Normal file
@ -0,0 +1,234 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Period is a reporting window for the usage dashboard. The dashboard used to
|
||||
// have exactly one view — everything since process start — which made a quiet
|
||||
// morning indistinguishable from a quiet week. Periods give the operator a
|
||||
// scale to read the numbers at: today vs this week vs this month vs all time.
|
||||
//
|
||||
// The set is deliberately calendar-based and UTC-anchored. A rolling 24h window
|
||||
// would put "today" and "the last day" at different boundaries, so the same
|
||||
// number would move between two cards depending on when you looked; calendar
|
||||
// days are what people actually mean by "today". UTC also matches the billing
|
||||
// plugin's peak-window arithmetic, so a peak-rate hour does not land in a
|
||||
// different day in the cost view than in the usage view.
|
||||
type Period string
|
||||
|
||||
const (
|
||||
// PeriodDay is the current UTC calendar day.
|
||||
PeriodDay Period = "day"
|
||||
// PeriodWeek is the current ISO week (Mon 00:00 UTC to now).
|
||||
PeriodWeek Period = "week"
|
||||
// PeriodMonth is the current UTC calendar month.
|
||||
PeriodMonth Period = "month"
|
||||
// PeriodAll is since process start — the only view backed by the
|
||||
// in-memory aggregates, and the only one available when no audit file is
|
||||
// configured.
|
||||
PeriodAll Period = "all"
|
||||
)
|
||||
|
||||
// ValidPeriod reports whether p is a period the aggregator understands.
|
||||
// An unknown period is a client error, not a silent fallback to "all": a
|
||||
// dashboard that quietly shows lifetime totals when the caller asked for today
|
||||
// is worse than one that refuses.
|
||||
func ValidPeriod(p Period) bool {
|
||||
switch p {
|
||||
case PeriodDay, PeriodWeek, PeriodMonth, PeriodAll:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// periodStart returns the inclusive start of the window for p at time now.
|
||||
// Only PeriodDay/Week/Month are meaningful here; PeriodAll returns 0, which
|
||||
// every "from > 0" bounds check treats as unbounded.
|
||||
func periodStart(p Period, now time.Time) int64 {
|
||||
now = now.UTC()
|
||||
switch p {
|
||||
case PeriodDay:
|
||||
return time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.UTC).UnixMilli()
|
||||
case PeriodWeek:
|
||||
// ISO week starts Monday. Go's Weekday() is Sunday=0, so the shift
|
||||
// below is 1 on Sunday and 0 on Monday..Saturday.
|
||||
off := (int(now.Weekday()) + 6) % 7
|
||||
d := now.AddDate(0, 0, -off)
|
||||
return time.Date(d.Year(), d.Month(), d.Day(), 0, 0, 0, 0, time.UTC).UnixMilli()
|
||||
case PeriodMonth:
|
||||
return time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC).UnixMilli()
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// PeriodBucket is one labelled point on the dashboard's timeline. Buckets are
|
||||
// the aggregation grain for a period: hourly for a day (so a spike is visible
|
||||
// inside "today"), daily for a week or month (so a week is not 7×24 points
|
||||
// wide and a month is not 31×24), and empty for "all" — a lifetime total has
|
||||
// no meaningful timeline, and pretending otherwise by drawing 500 hourly
|
||||
// buckets of rolling memory would be a different lie.
|
||||
type PeriodBucket struct {
|
||||
Bucket string `json:"bucket"`
|
||||
Stat
|
||||
}
|
||||
|
||||
// PeriodSnapshot is the period-scoped twin of Snapshot's payload: the same
|
||||
// totals and the same by_* rows, plus a timeline. It deliberately mirrors
|
||||
// Snapshot's field names so the dashboard's table painters work unchanged —
|
||||
// paintModelTable(st.by_model) reads rows of {name, ...Stat}, and that shape
|
||||
// does not care where the numbers came from.
|
||||
type PeriodSnapshot struct {
|
||||
Period Period `json:"period"`
|
||||
From int64 `json:"from"` // unix millis, 0 when PeriodAll
|
||||
Total Stat `json:"total"`
|
||||
ByKey []StatsRow `json:"by_key"`
|
||||
Models []StatsRow `json:"by_model"`
|
||||
Srcs []StatsRow `json:"by_source"`
|
||||
Status []agrRow `json:"by_status"`
|
||||
Bucket []PeriodBucket `json:"buckets"`
|
||||
// Truncated marks that the window was clipped by the retained audit
|
||||
// history, so the numbers are a lower bound rather than the true period
|
||||
// total. The dashboard shows this next to the numbers rather than letting
|
||||
// a rotated-away week read as "that week had no traffic".
|
||||
Truncated bool `json:"truncated"`
|
||||
}
|
||||
|
||||
// bucketKey maps a record's timestamp onto the timeline grain for p.
|
||||
// Daily buckets are stamped at UTC midnight; hourly buckets carry the hour.
|
||||
func bucketKey(p Period, ms int64) string {
|
||||
t := time.UnixMilli(ms).UTC()
|
||||
if p == PeriodDay {
|
||||
return t.Format("2006-01-02T15")
|
||||
}
|
||||
return t.Format("2006-01-02")
|
||||
}
|
||||
|
||||
// bucketOf returns the bucket label plus the truncated-flag side effects of
|
||||
// walking a file: a record older than the requested window means the window
|
||||
// starts before the retained history, and the file may have been cut short.
|
||||
func (s *Stats) PeriodSnapshot(p Period, key string, now time.Time) PeriodSnapshot {
|
||||
out := PeriodSnapshot{Period: p}
|
||||
from := periodStart(p, now)
|
||||
out.From = from
|
||||
|
||||
// "all" is exactly what Snapshot already answers, from the in-memory
|
||||
// aggregates, and it is the one view that must keep working with no audit
|
||||
// file configured at all (a fresh dev setup, or an operator who turned
|
||||
// auditing off). Serving it from the same code path keeps the dashboard's
|
||||
// "total" card identical whether or not a period is selected.
|
||||
if p == PeriodAll {
|
||||
snap := s.Snapshot(firstScreenRecords, key)
|
||||
if tot, ok := snap["total"].(Stat); ok {
|
||||
out.Total = tot
|
||||
}
|
||||
out.ByKey, _ = snap["by_key"].([]StatsRow)
|
||||
out.Models, _ = snap["by_model"].([]StatsRow)
|
||||
out.Srcs, _ = snap["by_source"].([]StatsRow)
|
||||
out.Status, _ = snap["by_status"].([]agrRow)
|
||||
// replay_partial is the same "these numbers came from a bounded
|
||||
// tail" caveat, carried through under this view's own name.
|
||||
out.Truncated, _ = snap["replay_partial"].(bool)
|
||||
return out
|
||||
}
|
||||
|
||||
// A bounded window is aggregated from the audit files, because the
|
||||
// in-memory aggregates are lifetime totals and the ring buffer holds only
|
||||
// maxRecs records (500 by default). Reading the ring would silently
|
||||
// under-report any period longer than the last few hundred requests.
|
||||
total := Stat{}
|
||||
byKey := map[string]*Stat{}
|
||||
byModel := map[string]*Stat{}
|
||||
bySrc := map[string]*Stat{}
|
||||
byStatus := map[string]*Stat{}
|
||||
buckets := map[string]*Stat{}
|
||||
|
||||
var seen int
|
||||
err := s.StreamAuditRecords(from, 0, key, func(r Req) error {
|
||||
seen++
|
||||
incStatus(&total, "", r)
|
||||
inc(byKey, r.Key, r)
|
||||
if r.Model != "" {
|
||||
inc(byModel, r.Model, r)
|
||||
}
|
||||
inc(bySrc, r.Source, r)
|
||||
if r.Status != 0 {
|
||||
inc(byStatus, strconv.Itoa(r.Status), r)
|
||||
}
|
||||
k := bucketKey(p, r.Time)
|
||||
b := buckets[k]
|
||||
if b == nil {
|
||||
b = &Stat{}
|
||||
buckets[k] = b
|
||||
}
|
||||
incStatus(b, k, r)
|
||||
return nil
|
||||
})
|
||||
|
||||
out.Total = total
|
||||
out.ByKey = rows(byKey)
|
||||
out.Models = rows(byModel)
|
||||
out.Srcs = rows(bySrc)
|
||||
bs := make([]agrRow, 0, len(byStatus))
|
||||
for code, st := range byStatus {
|
||||
bs = append(bs, agrRow{Name: code, Stat: *st})
|
||||
}
|
||||
sort.Slice(bs, func(i, j int) bool {
|
||||
ci, _ := strconv.Atoi(bs[i].Name)
|
||||
cj, _ := strconv.Atoi(bs[j].Name)
|
||||
return ci < cj
|
||||
})
|
||||
out.Status = bs
|
||||
|
||||
ks := make([]string, 0, len(buckets))
|
||||
for k := range buckets {
|
||||
ks = append(ks, k)
|
||||
}
|
||||
// Chronological, string-sorted: "2006-01-02T15" and "2006-01-02" both
|
||||
// sort lexicographically in time order, so no date parsing is needed.
|
||||
sort.Strings(ks)
|
||||
out.Bucket = make([]PeriodBucket, 0, len(ks))
|
||||
for _, k := range ks {
|
||||
out.Bucket = append(out.Bucket, PeriodBucket{Bucket: k, Stat: *buckets[k]})
|
||||
}
|
||||
|
||||
// The window is only "complete" if the audit walk actually reached back
|
||||
// far enough to cover it. Two ways it cannot:
|
||||
//
|
||||
// 1. The walk found nothing at all in a window that certainly had
|
||||
// traffic, because the files holding it rotated away.
|
||||
// 2. The walk errored part-way (I/O), leaving a partial total.
|
||||
//
|
||||
// Case 2 is reported from err; case 1 from seen == 0 combined with the
|
||||
// caller having asked for a bounded window. It is deliberately
|
||||
// conservative: a genuinely empty hour is rare enough that flagging it as
|
||||
// possibly-truncated costs one tooltip, whereas silently under-reporting a
|
||||
// month because rotation ate it is a wrong number with no indication.
|
||||
if err != nil || (seen == 0 && p != PeriodAll) {
|
||||
out.Truncated = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// periodFromQuery parses the ?period= parameter. An empty value means "all" so
|
||||
// that existing callers of /api/stats keep seeing exactly what they saw.
|
||||
// A malformed value is rejected by the caller (ValidPeriod) rather than
|
||||
// defaulting, so a typo in a bookmarked URL surfaces as an error instead of
|
||||
// silently switching the operator to lifetime totals.
|
||||
func periodFromQuery(q map[string][]string) Period {
|
||||
v := strings.ToLower(strings.TrimSpace(firstQuery(q, "period")))
|
||||
if v == "" {
|
||||
return PeriodAll
|
||||
}
|
||||
return Period(v)
|
||||
}
|
||||
|
||||
func firstQuery(q map[string][]string, key string) string {
|
||||
if vs, ok := q[key]; ok && len(vs) > 0 {
|
||||
return vs[0]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
Reference in New Issue
Block a user