mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-10-06 15:37:42 +00:00
fix(plugins): 插件 Lua 报错不再拖垮网关(生产事故修复)
## 事故
13:37 部署后线上 6 次 SIGSEGV 崩溃循环,8081 完全不可用,用户报大量
connect error。崩溃点固定在 internal/lua/plugins.go:invoke → L.Call →
golua StackTrace 里的 lua_getinfo。
## 根因(不是并发/GC/锁)
golua 的 callEx 在**任何** pcall 失败后无条件执行 L.StackTrace(),而
StackTrace 调 lua_getinfo,这个 LuaJIT 构建在栈够深时(带 AUTO 链轨迹的
request_end payload 正好够深)直接段错误。这是 C 层信号,Go 无法 recover,
所以一个插件的脚本错误就能带走整个进程和所有在途请求。
触发错误来自我上一轮加的 billing 日级维度:
add(bucket(bucket(bucket(s.by_day_src, dk), payload.source)), ...)
三个 bucket( 只对应两个 ),最外层 bucket() 只收到一个参数,k=nil,于是
billing.lua:141 `tbl[k] = b` 抛 "table index is nil",**每个请求都抛**。
同时还有第二个 bug:中间层用了 bucket()(返回 emptyBucket,含 cost/requests
字段)当作嵌套容器,结构也是错的。改为 dayMap() 返回纯表。
## 修法
1. billing.lua:修正括号,多层容器改用 dayMap()。
2. **pcall 守卫**(真正的架构修复):在 setupGlobals 里注册
__llmsproxy_call_hook,钩子改为经它调用。
function __llmsproxy_call_hook(fn, payload)
local ok, res = pcall(fn, payload)
if not ok then return nil, tostring(res) end
return res, nil
end
Lua 侧 pcall 在 golua 看到非零 pcall 状态之前就拦下错误,C 栈回溯路径
永远进不去。错误变成普通返回值 (nil, msg),Go 侧记进 hook_errors 并跳过
——"插件出错不影响请求转发"这条承诺对脚本错误也终于成立,而不只是对 Go panic。
## 这同时修掉了那个查了很久的间歇崩溃
同一个机制解释了此前 8/20 复现、却查不出根因的 SIGSEGV(怀疑过 janitor 竞态、
GC、LuaJIT 全局状态、VM 释放时序,全部排除)。实测对比:
TestBillingPrecedence 修复前 8/20 崩溃 → 修复后 0/20
并发建 16 个 VM 的探针 修复前 3/3 崩溃 → 修复后 0/6
全量 ./... 连跑 5 次全绿
那些崩溃本来就是一个 Lua 钩子错误在栈深时炸掉 StackTrace,时机随机所以看着
像并发问题。
## 判据
TestHookThatRaisesDoesNotCrashTheProcess:装一个每请求必崩的插件,连打 50 次,
断言进程存活 + 错误被记录 + 同状态里健康的 billing 插件照常工作。
3 个变异(守卫不 pcall / 守卫名写错 / 守卫未注册)全部被捕获,其中第一个直接
让 SIGSEGV 重现,说明守卫就是唯一防线。
## 线上验证
往生产插件目录放一个每请求必然报错的插件,连打 30 个真实流式请求:
30× HTTP 200,SIGSEGV 0 次
hook_errors 记录 count=44 且指名 zbroken-test(可观测)
billing 照常累计(2999 请求 / $0.5668)
测试插件已移除。
回滚点:/usr/local/bin/llmsproxy.bak-real-<TS>、billing.lua.bak-real-<TS>。
This commit is contained in:
@ -993,9 +993,47 @@ func restorePcall(L *golua.State) {
|
||||
}
|
||||
}
|
||||
|
||||
// hookGuardName is the Lua global that wraps every plugin hook call.
|
||||
const hookGuardName = "__llmsproxy_call_hook"
|
||||
|
||||
// hookGuardSrc defines the wrapper. It exists because of a hard constraint in
|
||||
// the binding: golua's callEx calls L.StackTrace() on ANY pcall error, and
|
||||
// StackTrace() calls lua_getinfo, which SIGSEGVs in this LuaJIT build once the
|
||||
// stack is deep enough (a request_end payload with the AUTO chain trace does
|
||||
// it). Go cannot recover from a C-level signal, so a single Lua mistake inside
|
||||
// one plugin killed the whole gateway — the production outage this was written
|
||||
// for. Repeated crashes proved the boundary is not theoretical.
|
||||
//
|
||||
// pcall INSIDE Lua catches the error before golua ever sees a non-zero
|
||||
// pcall status, so the C stack-trace path is never entered. The failure comes
|
||||
// back as an ordinary (nil, message) pair, which the Go side records in
|
||||
// hook_errors and moves on from — the documented contract that "a broken plugin
|
||||
// must not affect request forwarding" finally holds for script errors too, not
|
||||
// just for Go panics.
|
||||
//
|
||||
// returns: (result, errorMessage) — both nil/"" on success.
|
||||
const hookGuardSrc = `
|
||||
function ` + hookGuardName + `(fn, payload)
|
||||
local ok, res = pcall(fn, payload)
|
||||
if not ok then
|
||||
return nil, tostring(res)
|
||||
end
|
||||
return res, nil
|
||||
end`
|
||||
|
||||
func registerHookGuard(L *golua.State) {
|
||||
if err := L.DoString(hookGuardSrc); err != nil {
|
||||
// Nothing useful to do here beyond leaving the global absent: invoke()
|
||||
// checks for it and falls back to a direct call, which still works for
|
||||
// correct plugins (only their ERRORS stop being survivable).
|
||||
L.Pop(1)
|
||||
}
|
||||
}
|
||||
|
||||
func setupGlobals(L *golua.State) {
|
||||
restorePcall(L)
|
||||
buildJSONTable(L)
|
||||
registerHookGuard(L)
|
||||
|
||||
registerFn(L, "hmac_sha256_hex", func(L *golua.State) int {
|
||||
key := L.ToString(1)
|
||||
|
||||
Reference in New Issue
Block a user