fix(scheduler): rebalance the pref score so metered sources aren't written off silently

Report: an allowance-metered source (sensenova) had its deepseek model sitting at
pref -10 while the WebUI showed zero failures. All three numbers were accurate,
and they exposed three compounding problems.

1. Quota exhaustion cost the SAME score as a real failure. RecordQuotaExhausted
   intentionally avoids failCount (an exhausted allowance is not a fault), so the
   UI showed fails=0 / cooling=false — yet it deducted the full prefFailStep (5).
   For a metered source, running out of budget is an everyday event, so the score
   drifted deep negative with no visible cause. Now quota and 429 events cost
   prefQuotaStep (1): the cooldown already keeps the slot out of rotation until
   the window resets, the score only needs a mild preference for slots with budget.

2. Recovery was 5:1 asymmetric. A failure cost -5 but a success only +1, so a
   slot at -10 needed ten consecutive successes just to reach neutral — which it
   could never get, because a low score makes the scheduler not pick it in the
   first place (starvation). Success now rewards prefSuccessStep (2): recovery
   from -10 needs five successes, while a real failure still outweighs one.

3. No idle decay. A penalised slot kept its negative score forever once it stopped
   being selected. Pref() now applies lazy decay: after prefDecayAfter (2 min) of
   no outcome, the score drifts one step back toward 0 per interval (never past
   0, never touches positive scores). Applied in Pref() and TryProbe(), so a
   naturally-recovered idle slot is schedulable again without needing a probe.

The failure penalty itself is unchanged (prefFailStep=5), so genuinely broken
upstreams are still marked as clearly worse than healthy ones.

Tests: quota penalty lighter than failure; 5 quota resets stay well above the
floor with failCount untouched; 429 is quota-class; recovery from -10 needs <=5;
idle decay rehabilitates a written-off slot, stops at 0, and never drags a
positive score; decay repeatedly lifts a slot off the prefMin floor. Updated the
two pre-existing tests that asserted the old -5/129 +1 values.

internal/provider/provider.go

@@ -34,9 +34,14 @@ import (
 // cooldown always expires and any success resets the state, so a fixed
 // upstream recovers on its own.
 type ModelState struct {
-	pref          atomic.Int64 // +1 per success / -5 per failure, clamped
+	pref          atomic.Int64 // +prefSuccessStep per success / -prefFailStep per failure, clamped
 	failCount     atomic.Int64
 	cooldownUntil atomic.Int64 // unix seconds; 0 = schedulable
+	// prefTouched is the unix second pref was last moved by an outcome. It is
+	// the reference point for decay: a negative score with no traffic since
+	// prefDecayAfter drifts one step toward 0 per elapsed interval, so a slot
+	// cannot stay written off forever just because nothing selected it.
+	prefTouched atomic.Int64
 	// cooldownFrom is the unix second the current cooldown window opened. It
 	// is the reference point for the half-cooldown probe gate: probing is
 	// allowed only in the second half of [cooldownFrom, cooldownUntil), so a
@@ -50,10 +55,35 @@ type ModelState struct {
 }
 
 const (
+	// prefFailStep is the penalty for a REAL failure (5xx, transport error,
+	// unusable adapter output): something is wrong with the upstream or the
+	// model, so two of them are enough to push a slot to the back of its tier.
 	prefFailStep = 5
-	prefMin      = -20
-	prefMax      = 20
-	backoffBase  = 5 * time.Second
+	// prefQuotaStep is the penalty for running out of allowance (quota/TPM
+	// exhausted). Deliberately much smaller than prefFailStep: an exhausted
+	// quota means "no budget right now", not "this model is bad", and for
+	// allowance-metered sources it is an everyday event rather than a fault.
+	// Charging it the full failure penalty made such sources drift to a deeply
+	// negative score with ZERO visible failures - RecordQuotaExhausted
+	// intentionally does not touch failCount, so the WebUI showed fails=0 and
+	// cooling=false while the score sat at -10.
+	prefQuotaStep = 1
+	// prefSuccessStep is the reward for a successful request. Recovery used to
+	// be +1 against a -5 penalty, a 5:1 asymmetry that made a slot at -10 need
+	// ten consecutive successes just to reach neutral. Rewarding +2 keeps
+	// failures meaningful (a real failure still costs more than one success
+	// earns) while letting a recovered slot return to normal rotation quickly.
+	prefSuccessStep = 2
+	prefMin         = -20
+	prefMax         = 20
+	// prefDecayAfter is how long a slot may sit at a negative score without any
+	// traffic before the score drifts back toward neutral. Without this a slot
+	// that was penalised and then simply not selected again (the score itself
+	// makes it unattractive) stays penalised indefinitely: it has no way to earn
+	// the successes that would rehabilitate it. Applied lazily on read, so there
+	// are no timers.
+	prefDecayAfter = 2 * time.Minute
+	backoffBase    = 5 * time.Second
 	// backoffCap bounds the exponential schedule. It used to be 30min, which
 	// in practice meant "a slot whose upstream blipped is unusable for half an
 	// hour even though its quota recovered in seconds". Combined with the
@@ -142,8 +172,55 @@ func (s *ModelState) RecordFailure(auth bool) {
 	if auth {
 		pen *= 2
 	}
-	s.pref.Add(-pen)
+	s.addPref(-pen)
+}
+
+// addPref moves the preference score and stamps prefTouched, which restarts the
+// decay clock. Every outcome-driven score change must go through here so decay
+// never fires on a slot that is actively being scored.
+func (s *ModelState) addPref(delta int64) {
+	s.pref.Add(delta)
 	clampPref(&s.pref, prefMin, prefMax)
+	s.prefTouched.Store(time.Now().Unix())
+}
+
+// decayPref drifts a negative score back toward neutral once a slot has been
+// idle for prefDecayAfter. This closes a starvation loop: a penalised slot is
+// unattractive, so it stops being selected, so it never earns the successes that
+// would rehabilitate it, so it stays penalised. One step per elapsed interval,
+// never past 0, and positive scores are left alone (a proven-good slot should
+// not be dragged down for being idle).
+//
+// Called lazily from Pref(); no timers and no goroutines.
+func (s *ModelState) decayPref() {
+	cur := s.pref.Load()
+	if cur >= 0 {
+		return
+	}
+	last := s.prefTouched.Load()
+	if last <= 0 {
+		// no reference point yet: start the clock instead of decaying blindly
+		s.prefTouched.CompareAndSwap(0, time.Now().Unix())
+		return
+	}
+	step := int64(prefDecayAfter / time.Second)
+	if step <= 0 {
+		return
+	}
+	elapsed := time.Now().Unix() - last
+	if elapsed < step {
+		return
+	}
+	steps := elapsed / step
+	target := cur + steps
+	if target > 0 {
+		target = 0
+	}
+	// Advance prefTouched by exactly the consumed intervals so leftover time
+	// still counts toward the next step.
+	if s.pref.CompareAndSwap(cur, target) {
+		s.prefTouched.Store(last + steps*step)
+	}
 }
 
 // openCooldown starts a fresh cooldown window of length cd. Recording both
@@ -164,8 +241,10 @@ func (s *ModelState) openCooldown(cd time.Duration) {
 func (s *ModelState) RecordRateLimit() {
 	s.failCount.Add(1)
 	s.openCooldown(rateLimitCooldown)
-	s.pref.Add(-prefFailStep)
-	clampPref(&s.pref, prefMin, prefMax)
+	// A 429 is "too fast", not "broken": charge it like an allowance event, not
+	// like a fault, or a throttled-but-healthy source sinks purely for being
+	// popular.
+	s.addPref(-prefQuotaStep)
 }
 
 // RecordQuotaExhausted marks the model as out of allowance until its quota
@@ -185,20 +264,24 @@ func (s *ModelState) RecordQuotaExhausted(resetIn time.Duration) {
 	// No failCount bump: an exhausted quota is not an error streak, and
 	// letting it inflate failCount would make the NEXT real failure jump
 	// straight to the capped backoff.
-	s.pref.Add(-prefFailStep)
-	clampPref(&s.pref, prefMin, prefMax)
+	//
+	// The score penalty is prefQuotaStep, not prefFailStep, for the same
+	// reason: cooldown alone already keeps the slot out of rotation until the
+	// window resets, so the score only needs to express a mild preference for
+	// slots with budget left. Charging the full failure penalty here is what
+	// drove allowance-metered sources to -10 and beyond with fails=0.
+	s.addPref(-prefQuotaStep)
 }
 
-// RecordSuccess resets the failure counter and cooldown and bumps the
-// preference score by one. A single success is enough to lift a slot off the
-// prefMin floor (prefMin+1 > prefMin), which is what lets one successful probe
-// return a written-off slot to normal rotation.
+// RecordSuccess resets the failure counter and cooldown and rewards the
+// preference score. A single success lifts a slot off the prefMin floor, which
+// is what lets one successful probe return a written-off slot to normal
+// rotation.
 func (s *ModelState) RecordSuccess() {
 	s.failCount.Store(0)
 	s.cooldownUntil.Store(0)
 	s.cooldownFrom.Store(0)
-	s.pref.Add(1)
-	clampPref(&s.pref, prefMin, prefMax)
+	s.addPref(prefSuccessStep)
 }
 
 func (s *ModelState) reset() {
@@ -207,6 +290,7 @@ func (s *ModelState) reset() {
 	s.cooldownFrom.Store(0)
 	s.probeInFlight.Store(false)
 	s.pref.Store(0)
+	s.prefTouched.Store(0)
 }
 
 // TryProbe attempts to claim the single probe permit for a model that normal
@@ -225,6 +309,7 @@ func (s *ModelState) TryProbe() bool {
 	now := time.Now().Unix()
 	until := s.cooldownUntil.Load()
 	if until <= now {
+		s.decayPref() // a naturally-recovered slot needs no probe
 		if s.pref.Load() > prefMin {
 			return false // schedulable normally, no probe needed
 		}
@@ -251,8 +336,13 @@ func (s *ModelState) Probing() bool { return s.probeInFlight.Load() }
 // when the model is not cooling.
 func (s *ModelState) CooldownFrom() int64 { return s.cooldownFrom.Load() }
 
-// Pref is the current preference score (higher = preferred).
-func (s *ModelState) Pref() int64 { return s.pref.Load() }
+// Pref is the current preference score (higher = preferred). Reading it also
+// applies idle decay, so a slot penalised long ago and never selected since
+// drifts back toward neutral instead of being written off permanently.
+func (s *ModelState) Pref() int64 {
+	s.decayPref()
+	return s.pref.Load()
+}
 
 // FailCount is the number of consecutive failures.
 func (s *ModelState) FailCount() int64 { return s.failCount.Load() }