mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-10-03 23:54:06 +00:00
feat(gateway): per-key 用量配额(token + 请求数)与重置周期
问题:密钥控制只能限制模型范围。实测发现三个缺陷,其中前两个让
per-model token_quota 在真实链路上从未生效:
1. 桶键不含 key。scopeTokens 调 WindowTokens(model, source, win),
桶键是 model / source::model,与调用方无关。实测两把 key 各用
1000 token,窗口报 2000 —— A key 的额度被 B key 消耗。
2. 无 source pin 的桶永远是空的。真实记录 Source 总被填上,桶键存成
"deepseek::m1",而无 pin 的查询找 "m1" —— 读到 0,永远 < quota,
配额形同虚设。实测 WindowTokens("m1","",1h)=0 而 pinned=2000。
3. AUTO scope 走 KeyTokens(key),是全时段累计、永不重置。实测 30 天
前的 200 token 仍计入 1 小时配额(报 210 而非 10)。配了
period: hour 也不会每小时归零。
生产 5 把 user key 全是 token_quota: 0,所以前两条一直没暴露。
改动:
- Stats 新增 per-key 小时桶 keyModelHour(key → model → hour)与
keyHour(key 总量)、keyReqHour(请求数),retention 40 天,与既有
modelHour 对齐以覆盖最长的 month 窗口;LoadAudit 走 aggregateLocked,
所以窗口用量跨重启存活。modelHour 保持 key-blind:它服务的是 AUTO
槽位配额(限制整个网关对某槽位的消耗),语义不同,不应被 per-key
改造污染。
- 每个请求写两份模型桶:裸 model 与 source::model。无 pin 的 scope
条目读前者,有 pin 的读后者。
- GWKey 新增 TokenQuota / ReqQuota / Period / Hours:整钥配额,
跨该 key 所有模型共享一份预算;ReqQuota 覆盖持续请求量(源上的
RPM 只管突发)。
- 配额耗尽返回 429 + Retry-After(rate_limit_exceeded),而不是 403:
403 让客户端以为这把 key 永远不能用该模型,直接放弃;429 + 等待
才能在窗口重置后自动恢复。模型越权仍是 403。
- admin key 永不受配额限制 —— 否则操作者会把自己锁在门外。
- 周期词表在写入时校验,拼错的 period 被拒绝而不是静默当成永不过期
(那与操作者输入的意图正好相反)。
- PUT /api/keys 的配额字段是指针:省略=保留原值,显式 0=解除限制。
否则只改模型范围就会悄悄清空预算。
判据 3 个文件 24 例,9 个变异全部被抓:key 隔离、pin 桶缺失、
AUTO 周期、key-blind 退化、429→403、admin 被限、PUT 清空配额、
Validate 失效、pinned 桶缺失。前三个变异最初漏网 —— 判据只测了
Stats 层没测接线,补了走真实 HTTP 的接线层与 API 层判据后抓住。
端到端验证:真实进程 + 加密配置往返,配额字段与 enc:v1 密钥均正常。
This commit is contained in:
@ -84,6 +84,18 @@ type Stats struct {
|
||||
auditPath string
|
||||
replayPartial bool // aggregates built from a bounded audit tail
|
||||
modelHour map[string]map[int64]int64 // model -> unix-hour bucket -> tokens
|
||||
// keyModelHour buckets the same tokens as modelHour but keyed by
|
||||
// (gateway key id, model) so per-key model quotas are isolated from each
|
||||
// other. modelHour stays key-blind on purpose: it backs the AUTO slot
|
||||
// quota, which is a gateway-wide limit on a slot, not a per-key one.
|
||||
keyModelHour map[string]map[string]map[int64]int64 // key -> model -> unix-hour -> tokens
|
||||
// keyHour buckets a key's total tokens per unix hour, backing the
|
||||
// whole-key quota (all models of one key share one budget).
|
||||
keyHour map[string]map[int64]int64
|
||||
// keyReqHour buckets a key's request count per unix hour, backing the
|
||||
// key-wide request quota. Counted for every request type (chat, stream,
|
||||
// image) including failed ones, so a failing client cannot loop for free.
|
||||
keyReqHour map[string]map[int64]int64
|
||||
}
|
||||
|
||||
const hourSec = 3600
|
||||
@ -116,14 +128,17 @@ func NewStats(maxRecords int) *Stats {
|
||||
maxRecords = defaultRingSize
|
||||
}
|
||||
return &Stats{
|
||||
byKey: map[string]*Stat{},
|
||||
byModel: map[string]*Stat{},
|
||||
bySrc: map[string]*Stat{},
|
||||
byKeyModel: map[string]map[string]*Stat{},
|
||||
byKeySrc: map[string]map[string]*Stat{},
|
||||
byStatus: map[int]*Stat{},
|
||||
modelHour: map[string]map[int64]int64{},
|
||||
maxRecs: maxRecords,
|
||||
byKey: map[string]*Stat{},
|
||||
byModel: map[string]*Stat{},
|
||||
bySrc: map[string]*Stat{},
|
||||
byKeyModel: map[string]map[string]*Stat{},
|
||||
byKeySrc: map[string]map[string]*Stat{},
|
||||
byStatus: map[int]*Stat{},
|
||||
modelHour: map[string]map[int64]int64{},
|
||||
keyModelHour: map[string]map[string]map[int64]int64{},
|
||||
keyHour: map[string]map[int64]int64{},
|
||||
keyReqHour: map[string]map[int64]int64{},
|
||||
maxRecs: maxRecords,
|
||||
}
|
||||
}
|
||||
|
||||
@ -318,6 +333,9 @@ func (s *Stats) Record(r Req) {
|
||||
// aggregateLocked folds r into every aggregate row and the quota window
|
||||
// bucket. Caller must hold s.mu.
|
||||
func (s *Stats) aggregateLocked(r Req) {
|
||||
if r.Key != "" {
|
||||
s.addKeyReqLocked(r.Key, (r.Time/1000)/hourSec, 1)
|
||||
}
|
||||
inc(s.byKey, r.Key, r)
|
||||
if r.Model != "" {
|
||||
inc(s.byModel, r.Model, r)
|
||||
@ -362,6 +380,12 @@ func (s *Stats) aggregateLocked(r Req) {
|
||||
s.modelHour[key] = hm
|
||||
}
|
||||
hm[h] += tok
|
||||
// per-key buckets: same token split, but isolated per key so one
|
||||
// key's quota cannot be drained by another key's traffic
|
||||
if r.Key != "" {
|
||||
s.addKeyTokenLocked(r.Key, r.Model, r.Source, h, tok)
|
||||
s.addKeyHourLocked(r.Key, h, tok)
|
||||
}
|
||||
// retention: 24*40 = 960 hourly buckets ≈ 40 days of history (covers
|
||||
// the longest "month" quota window)
|
||||
if len(hm) > 24*40 {
|
||||
@ -374,6 +398,152 @@ func (s *Stats) aggregateLocked(r Req) {
|
||||
}
|
||||
}
|
||||
|
||||
// quotaRetentionHours is how much hourly history the per-key quota buckets
|
||||
// keep. It matches the modelHour retention (40 days) so the longest "month"
|
||||
// window is fully covered after a restart, and it is deliberately applied per
|
||||
// key so an idle key's buckets are reclaimed instead of pinning memory.
|
||||
const quotaRetentionHours = 24 * 40
|
||||
|
||||
// addKeyTokenLocked adds tok to one key's (model[, source]) hourly bucket.
|
||||
//
|
||||
// It writes BOTH an unpinned and a pinned bucket. Every recorded request
|
||||
// carries a resolved source, so the bucket key would otherwise always be
|
||||
// "source::model" and a quota without a source pin would read an empty bucket
|
||||
// and never trip. The unpinned bucket is the one a scope entry normally looks
|
||||
// up; the pinned one serves entries that pin a source.
|
||||
func (s *Stats) addKeyTokenLocked(key, model, source string, h, tok int64) {
|
||||
if model == "" {
|
||||
return
|
||||
}
|
||||
byModel := s.keyModelHour[key]
|
||||
if byModel == nil {
|
||||
byModel = map[string]map[int64]int64{}
|
||||
s.keyModelHour[key] = byModel
|
||||
}
|
||||
// Write the bare-model bucket plus, when a source is known, the pinned
|
||||
// one. The unpinned bucket is the aggregate the scope lookup normally
|
||||
// reads; the pinned bucket serves entries that pin a source.
|
||||
keys := [2]string{model, model}
|
||||
n := 1
|
||||
if source != "" {
|
||||
if pinned := source + "::" + model; pinned != model {
|
||||
keys[1] = pinned
|
||||
n = 2
|
||||
}
|
||||
}
|
||||
for i := 0; i < n; i++ {
|
||||
hm := byModel[keys[i]]
|
||||
if hm == nil {
|
||||
hm = map[int64]int64{}
|
||||
byModel[keys[i]] = hm
|
||||
}
|
||||
hm[h] += tok
|
||||
if len(hm) > quotaRetentionHours {
|
||||
for k := range hm {
|
||||
if k < h-quotaRetentionHours {
|
||||
delete(hm, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// addKeyHourLocked adds tok to one key's all-model hourly total.
|
||||
func (s *Stats) addKeyHourLocked(key string, h, tok int64) {
|
||||
hm := s.keyHour[key]
|
||||
if hm == nil {
|
||||
hm = map[int64]int64{}
|
||||
s.keyHour[key] = hm
|
||||
}
|
||||
hm[h] += tok
|
||||
if len(hm) > quotaRetentionHours {
|
||||
for k := range hm {
|
||||
if k < h-quotaRetentionHours {
|
||||
delete(hm, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// KeyWindowReqs returns how many requests one key issued within sec seconds;
|
||||
// sec <= 0 means all retained history.
|
||||
func (s *Stats) KeyWindowReqs(key string, sec int64) int64 {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return sumBuckets(s.keyReqHour[key], time.Now().Unix(), sec)
|
||||
}
|
||||
|
||||
// addKeyReqLocked adds n requests to a key's hourly count bucket.
|
||||
func (s *Stats) addKeyReqLocked(key string, h, n int64) {
|
||||
hm := s.keyReqHour[key]
|
||||
if hm == nil {
|
||||
hm = map[int64]int64{}
|
||||
s.keyReqHour[key] = hm
|
||||
}
|
||||
hm[h] += n
|
||||
if len(hm) > quotaRetentionHours {
|
||||
for k := range hm {
|
||||
if k < h-quotaRetentionHours {
|
||||
delete(hm, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sumBuckets totals the hourly buckets at or after the cutoff given in unix
|
||||
// seconds. sec <= 0 means "all retained history" (no reset).
|
||||
func sumBuckets(hm map[int64]int64, now, sec int64) int64 {
|
||||
if len(hm) == 0 {
|
||||
return 0
|
||||
}
|
||||
if sec <= 0 {
|
||||
var total int64
|
||||
for _, v := range hm {
|
||||
total += v
|
||||
}
|
||||
return total
|
||||
}
|
||||
cut := now - sec
|
||||
var total int64
|
||||
for h, v := range hm {
|
||||
if h*hourSec >= cut {
|
||||
total += v
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// KeyWindowModelTokens returns the tokens one key consumed for one model
|
||||
// (optionally pinned to a source) within sec seconds; sec <= 0 means all
|
||||
// retained history. Unlike WindowTokens this is isolated per key, which is
|
||||
// what a per-key model quota needs.
|
||||
//
|
||||
// A source pin matches the pinned bucket exactly. Without a pin the call
|
||||
// totals the key's bare-model bucket, which counts traffic on every source —
|
||||
// a quota on "this model" should not stop applying just because the request
|
||||
// happened to be served by a different upstream.
|
||||
func (s *Stats) KeyWindowModelTokens(key, model, source string, sec int64) int64 {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
byModel := s.keyModelHour[key]
|
||||
if len(byModel) == 0 {
|
||||
return 0
|
||||
}
|
||||
mk := model
|
||||
if source != "" {
|
||||
mk = source + "::" + model
|
||||
}
|
||||
return sumBuckets(byModel[mk], time.Now().Unix(), sec)
|
||||
}
|
||||
|
||||
// KeyWindowTokens returns one key's total tokens across all models within sec
|
||||
// seconds; sec <= 0 means all retained history. It backs the whole-key quota.
|
||||
func (s *Stats) KeyWindowTokens(key string, sec int64) int64 {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return sumBuckets(s.keyHour[key], time.Now().Unix(), sec)
|
||||
}
|
||||
|
||||
// rotateAuditLocked renames the audit file to <path>.<unix>.old once it
|
||||
// exceeds auditRotateBytes and prunes old files beyond auditKeepOld, keeping
|
||||
// the newest ones. Caller must hold s.mu.
|
||||
@ -464,6 +634,27 @@ func AutoPeriodSeconds(period string, hours int64) int64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
// AutoSecondsToReset returns how many seconds remain until the quota window
|
||||
// rolls over, for the Retry-After header on a 429. Buckets are whole unix
|
||||
// hours, so the value is rounded up to the next hour boundary and never
|
||||
// exceeds one window. 0 means "unknown" (never resets, or no period set), in
|
||||
// which case the caller should not advertise a retry time.
|
||||
func AutoSecondsToReset(period string, hours int64) int64 {
|
||||
sec := AutoPeriodSeconds(period, hours)
|
||||
if sec <= 0 {
|
||||
return 0
|
||||
}
|
||||
now := time.Now().Unix()
|
||||
// window = the last sec seconds, bucketed by whole hours; the oldest
|
||||
// bucket still inside the window expires at its own hour boundary
|
||||
elapsed := now % hourSec
|
||||
left := sec - elapsed
|
||||
if left <= 0 || left > sec {
|
||||
left = sec
|
||||
}
|
||||
return left
|
||||
}
|
||||
|
||||
// WindowTokens returns the tokens consumed for one model (optionally pinned
|
||||
// to a single source) within the window; sec <= 0 means all time. Buckets are
|
||||
// whole unix hours, so a sliding window overcounts by up to one hour — an
|
||||
|
||||
Reference in New Issue
Block a user