jianf/ModelRouter
1
0
Fork 0
mirror of https://gitcode.com/JianFeeeee/ModelRouter.git synced 2026-09-20 08:57:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(lua): reclaim burst leftovers while traffic continues

Browse Source 
Production after the first deploy showed the openai pool stuck at 9 states with
peak_in_use=1: a startup burst grew it, and then it never shrank again. The
shrink grace counter was reset by every CHECKOUT, so on a gateway that always
has a request in flight the counter never reached shrinkGraceRounds and the
burst's leftover states were pinned indefinitely — the same monotonic-growth
behaviour this series set out to remove, just with an extra step.

Grace is now reset by GROWTH (a miss that had to boot a state), which is the
actual signal that capacity is short. Ordinary sequential traffic no longer
defers reclaim, while two consecutive quiet-ish rounds are still required so a
gap between two bursts does not tear the pool down.

TestCheckoutResetsGrace asserted the old behaviour and is replaced by
TestGrowthResetsGrace (sequential traffic must NOT defer, growth must) plus
TestBurstLeftoverIsReclaimedUnderSteadyTraffic, which reproduces the production
shape: 9 concurrent holds, then one request per janitor round, and the pool must
still fall back to the resident floor.
This commit is contained in:
JianFeeeee
2026-08-30 08:13:34 +08:00
parent 3e27f4db24
commit 25d8bd8632
2 changed files with 79 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
internal/lua/vm.go
View File

@ -126,8 +126,11 @@ const (
// shrinkInterval is how often the VM janitor reclaims idle states.
shrinkInterval = 30 * time.Second
// shrinkGraceRounds is how many consecutive janitor rounds must see slack
// before anything is released, so a gap between requests is not mistaken
// for the end of a load period.
// before anything is released. It guards against tearing a pool down between
// two bursts — NOT against ordinary traffic: a pool serving one request at a
// time still has reclaimable slack, and requiring "no traffic at all" would
// pin a burst's leftover states forever on any busy gateway. The counter is
// therefore reset by GROWTH (real contention), not by a mere checkout.
shrinkGraceRounds = 2
)
@ -360,6 +363,9 @@ func (p *adapterPool) acquire() (*worker, error) {
// sequential caller reusing one warm state.
contended := p.created > 0 && p.inUse >= p.created
p.created++
// Growing is the signal that capacity is genuinely short, so any
// pending shrink decision is stale.
p.idleRounds = 0
extra := p.growPlanLocked(contended)
p.mu.Unlock()
if extra > 0 {
@ -391,8 +397,6 @@ func (p *adapterPool) checkoutLocked() {
if p.inUse > p.peakInUse {
p.peakInUse = p.inUse
}
// Live demand invalidates any pending shrink decision.
p.idleRounds = 0
}
func (p *adapterPool) release(w *worker) {