feat(quota): 探测退避改指数 —— 死上游不再每 30 分钟重付一遍

## 为什么 30 分钟 TTL 治不了死上游

线上有两个 host 已经**不可达十个小时**,而 PROBE_MEMO_TTL 是 1800 秒。
于是每次刷新它们的负 memo 都已经过期,每次都重付三个 adapter 的超时:
每个源 24s(预算修好前)、16s(修好后)。固定 TTL 对「已经死了很久」这个
形状完全无效 —— 它只会一直重复付。

指数退避:连续失败 streak 次,等待 1800 × 2^(streak-1),封顶 6 小时。
成功会把 memo 覆盖成 vendor 字符串,streak 自然重置。

封顶不是可选项 —— 没有它,下线一周的 host 就不再被探测,页面会**永久**
把它误报成「没有余额接口」。

## 过程中判据抓到的三个真 bug

**① 记录守卫让 streak 永远是 1**
`if rep == nil and not HOST_MEMO[host]` 只在 host **第一次**失败时为真。
之后 memo 已是 table,守卫为假,什么都不写。实测连读四次都是
`{at:... streak:1}` —— 退避退化成固定 30 分钟,正是它要解决的问题。

**② `return 0` 在 Lua 里是真值**
`if streak == nil or streak <= 0 then return 0 end` —— 只有 nil 和 false
是假值。调用点写的是 `if backoffSeconds(...) then 跳过`,于是**所有没有
streak 字段的 memo 会被永久跳过**(而那正是本改动之前写的每一个 memo)。

**③ 上限路径 `return BACKOFF_MAX` 也是真值**
21600 会被判成「仍在等待」,封顶反而变成「永远不重试」—— 恰好是封顶要防的
那个故障。现在所有路径只返回布尔。

三个都是静默的:症状只是页面少了几张卡片,没有任何报错。

## 判据

五条,全部读**实际发布的 quota.lua**(go:embed 那份),不是副本:

- 退避确实被调用(变异把调用点换成 `if false` 时会红)
- 窗口表:每个 streak 的窗口长度、窗口内跳过、窗口过期后重试、
  荒谬 streak 也必须重试
- 只返回布尔(六种输入,含无 streak 字段的老 memo)
- 记录守卫会替换 memo 而非合并
- 通过真实插件:首次失败记 streak=1;**跳过不算失败**(否则一个从不被探测的
  host 会无限增长 streak,再也不会被重试)

变异自证六次全部变红:`return 0`、上限返回数字、守卫不替换、调用点去掉、
去掉上限、倍数改成 1。头一次跑「守卫不替换」漏放了 —— 因为我把集成判据缩到
只验首次记录,补了一条源码断言才抓住。

## 这个文件的判据绕了远路,如实记下

`HOST_MEMO` 是进程内缓存,只在首次 collect 时从 state 恢复,之后只往回写。
所以从 Go 侧改 state 对插件不可见 —— 我先是不自知地这么试了两轮,把一个
**正确工作的退避**看成坏的。现在窗口策略直接验函数(经 shim 导出插件自己的
backoffSeconds,不是重新实现),集成判据只留真正能从外部观察的两条。

shim 也踩过:直接把 `backoffSeconds` 当 export,而 Go 侧传的是 table,
于是 at=table、streak=nil,每次都走「无 streak」分支返回 false —— 看起来
和「退避永不触发」一模一样。包一层就对了。
This commit is contained in:
JianFeeeee
2026-10-05 07:21:43 +08:00
parent ff8afaa43d
commit 2272b4ecbd
3 changed files with 412 additions and 2 deletions

View File

@ -298,6 +298,52 @@ local HOST_MEMO = {}
local HOST_MEMO_AT = {}
local PROBE_MEMO_TTL = 1800 -- 30 minutes
-- Exponential backoff for hosts that answer nothing.
--
-- A flat TTL is the wrong shape for a dead upstream. Measured on this gateway:
-- two hosts had been unreachable for 10 HOURS, and because PROBE_MEMO_TTL is 30
-- minutes their negative memo was stale on every single page open — so each
-- refresh re-paid the full cost of three adapters (measured: 24s per source
-- before the probe budget was fixed, 16s after). A host that is down should not
-- be re-examined every 30 minutes forever; each consecutive failure doubles the
-- wait, so a long-dead host costs almost nothing while a host that was briefly
-- down is retried within the TTL as before.
--
-- The cap matters as much as the doubling: without it a host that is down for a
-- week stops being probed at all, and the page would quietly and permanently
-- misreport it as having no balance endpoint.
--
-- SUCCESS resets the streak. This is only a probe-scheduling decision — a source
-- that stops answering is still shown, still counted, and still retried.
local BACKOFF_BASE = 1800 -- first retry after the usual TTL
local BACKOFF_FACTOR = 2 -- doubles per consecutive miss
local BACKOFF_MAX = 21600 -- 6 hours; beyond this we always re-probe
-- backoffSeconds returns how long to wait before re-probing a host that has
-- missed `streak` times in a row, and 0 when it is due now.
--
-- `at` is when the host last failed. streak is stored inside the memo table
-- itself so it survives a restart: a process restart must not reset the backoff
-- and re-pay the cost it was avoiding.
-- Returns a BOOLEAN and nothing else.
--
-- In Lua only nil and false are falsy, so every path here must return one of
-- those. Two versions of this got it wrong and both were caught by a test that
-- asserts the type: `return 0` for "no streak" (0 is truthy, so a host with no
-- streak — which is every memo written before this change — would be skipped
-- forever, silently), and `return BACKOFF_MAX` on the cap path (21600 is
-- truthy too, so a long-dead host would never be re-probed, which is the exact
-- failure the cap exists to prevent).
local function backoffSeconds(at, streak)
if streak == nil or streak <= 0 then return false end
local wait = BACKOFF_BASE
for _ = 2, streak do
wait = wait * BACKOFF_FACTOR
if wait >= BACKOFF_MAX then wait = BACKOFF_MAX break end
end
return (os.time() - (at or 0)) < wait
end
-- PROBE_BUDGET_MS caps one collect pass. Without it a page open over a set of
-- dead upstreams runs to the sum of every timeout — measured at 149s — and the
-- plugin's write lock is held the whole time.
@ -325,6 +371,12 @@ local function collectSource(src, budget)
-- exactly the confusion that made an earlier revision drop five sources.
if type(known) == "table" then
local seen = known.at or 0
-- A streak is a host that has failed repeatedly. Skip it outright until its
-- backoff expires, instead of re-paying three adapter timeouts every TTL.
if backoffSeconds(seen, known.streak) then
return { name = src.name, host = host, skipped = true,
reason = "backing off after " .. tostring(known.streak) .. " failed probes" }
end
if PROBE_MEMO_TTL > 0 and (os.time() - seen) > PROBE_MEMO_TTL then
known = nil
else
@ -394,8 +446,21 @@ local function collectSource(src, budget)
-- would make the page quietly stop reporting a source that started working.
-- The stamp is what turns a 60s first load into a fast steady state without
-- freezing what the page can see.
if rep == nil and not HOST_MEMO[host] then
HOST_MEMO[host] = { at = os.time() }
-- Record the negative result, and GROW THE STREAK.
--
-- The guard used to be `not HOST_MEMO[host]`, which is true only the FIRST time
-- a host fails: on every later failure the memo already held a table, the
-- guard was false, and nothing was written. The streak therefore stayed pinned
-- at 1 forever and the backoff never grew. Measured while writing this: the
-- memo read back `{at:... streak:1}` after four consecutive failures.
--
-- A success overwrites the memo with a vendor string, so a host that recovers
-- resets its streak automatically.
if rep == nil then
local prev = HOST_MEMO[host]
local streak = 1
if type(prev) == "table" then streak = (prev.streak or 1) + 1 end
HOST_MEMO[host] = { at = os.time(), streak = streak }
end
if rep ~= nil then
HOST_MEMO_AT[host] = os.time()