mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-10-05 07:02:29 +00:00
d9652f479a
## 事故
13:37 部署后线上 6 次 SIGSEGV 崩溃循环，8081 完全不可用，用户报大量
connect error。崩溃点固定在 internal/lua/plugins.go:invoke → L.Call →
golua StackTrace 里的 lua_getinfo。
## 根因（不是并发/GC/锁）
golua 的 callEx 在**任何** pcall 失败后无条件执行 L.StackTrace()，而
StackTrace 调 lua_getinfo，这个 LuaJIT 构建在栈够深时（带 AUTO 链轨迹的
request_end payload 正好够深）直接段错误。这是 C 层信号，Go 无法 recover，
所以一个插件的脚本错误就能带走整个进程和所有在途请求。
触发错误来自我上一轮加的 billing 日级维度：
add(bucket(bucket(bucket(s.by_day_src, dk), payload.source)), ...)
三个 bucket( 只对应两个 )，最外层 bucket() 只收到一个参数，k=nil，于是
billing.lua:141 `tbl[k] = b` 抛 "table index is nil"，**每个请求都抛**。
同时还有第二个 bug：中间层用了 bucket()（返回 emptyBucket，含 cost/requests
字段）当作嵌套容器，结构也是错的。改为 dayMap() 返回纯表。
## 修法
1. billing.lua：修正括号，多层容器改用 dayMap()。
2. **pcall 守卫**（真正的架构修复）：在 setupGlobals 里注册
__llmsproxy_call_hook，钩子改为经它调用。
function __llmsproxy_call_hook(fn, payload)
local ok, res = pcall(fn, payload)
if not ok then return nil, tostring(res) end
return res, nil
end
Lua 侧 pcall 在 golua 看到非零 pcall 状态之前就拦下错误，C 栈回溯路径
永远进不去。错误变成普通返回值 (nil, msg)，Go 侧记进 hook_errors 并跳过
——"插件出错不影响请求转发"这条承诺对脚本错误也终于成立，而不只是对 Go panic。
## 这同时修掉了那个查了很久的间歇崩溃
同一个机制解释了此前 8/20 复现、却查不出根因的 SIGSEGV（怀疑过 janitor 竞态、
GC、LuaJIT 全局状态、VM 释放时序，全部排除）。实测对比：
TestBillingPrecedence 修复前 8/20 崩溃 → 修复后 0/20
并发建 16 个 VM 的探针 修复前 3/3 崩溃 → 修复后 0/6
全量 ./... 连跑 5 次全绿
那些崩溃本来就是一个 Lua 钩子错误在栈深时炸掉 StackTrace，时机随机所以看着
像并发问题。
## 判据
TestHookThatRaisesDoesNotCrashTheProcess：装一个每请求必崩的插件，连打 50 次，
断言进程存活 + 错误被记录 + 同状态里健康的 billing 插件照常工作。
3 个变异（守卫不 pcall / 守卫名写错 / 守卫未注册）全部被捕获，其中第一个直接
让 SIGSEGV 重现，说明守卫就是唯一防线。
## 线上验证
往生产插件目录放一个每请求必然报错的插件，连打 30 个真实流式请求：
30× HTTP 200，SIGSEGV 0 次
hook_errors 记录 count=44 且指名 zbroken-test（可观测）
billing 照常累计（2999 请求 / $0.5668）
测试插件已移除。
回滚点：/usr/local/bin/llmsproxy.bak-real-<TS>、billing.lua.bak-real-<TS>。
77 lines
2.7 KiB
Go
77 lines
2.7 KiB
Go
package lua
|
|
|
|
import (
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// A plugin hook that RAISES must not take the process down. This is the
|
|
// production outage: a Lua error anywhere in a hook made golua's callEx call
|
|
// L.StackTrace(), which calls lua_getinfo and SIGSEGVs on a deep enough stack —
|
|
// a C-level signal Go cannot recover from, so one buggy plugin killed the whole
|
|
// gateway and took every in-flight request with it.
|
|
//
|
|
// The fix routes hook calls through a Lua-side pcall guard, so the error comes
|
|
// back as an ordinary return value. This test fires hooks that raise on purpose
|
|
// and asserts three things: the process survives, the failure is RECORDED, and
|
|
// a well-behaved plugin on the same state keeps working afterwards (the error
|
|
// must not poison the Lua state).
|
|
func TestHookThatRaisesDoesNotCrashTheProcess(t *testing.T) {
|
|
ps, pdir := billingVM(t)
|
|
boom := `
|
|
local plugin = {}
|
|
plugin.name = "boom"
|
|
plugin.version = "0.1"
|
|
function plugin.request_end(payload)
|
|
-- Raise on a table index, the exact shape of the billing bug that caused the
|
|
-- outage. Deliberately NOT a syntax error: this must load fine and fail only
|
|
-- when invoked.
|
|
local x = nil
|
|
return x.field
|
|
end
|
|
return plugin`
|
|
if err := os.WriteFile(filepath.Join(pdir, "boom.lua"), []byte(boom), 0644); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if err := ps.LoadSource("boom", boom); err != nil {
|
|
t.Fatalf("load boom: %v", err)
|
|
}
|
|
|
|
payload := map[string]interface{}{
|
|
"model": "m", "source": "s", "ok": true,
|
|
"prompt_tokens": 100, "completion_tokens": 10, "time": 1750000000000,
|
|
}
|
|
// Fire many times: a single call could pass by luck, but if the error ever
|
|
// escapes into golua's C path the process dies and this test never returns.
|
|
for i := 0; i < 50; i++ {
|
|
ps.Fire(StageRequestEnd, payload)
|
|
}
|
|
// Reaching this line at all is the primary assertion.
|
|
|
|
errs := ps.HookErrors()
|
|
end, ok := errs[string(StageRequestEnd)]
|
|
if !ok {
|
|
t.Fatal("a raising hook left no record — failures must be observable, not swallowed")
|
|
}
|
|
if end["count"] == nil || end["count"].(int) == 0 {
|
|
t.Error("hook error count is zero despite 50 raising calls")
|
|
}
|
|
msg, _ := end["last_error"].(string)
|
|
if !strings.Contains(msg, "boom") {
|
|
t.Errorf("last_error does not name the offending plugin: %q", msg)
|
|
}
|
|
|
|
// The billing plugin shares the same Plugins registry and must still work:
|
|
// one broken plugin may not disable the others.
|
|
st, _ := ps.State("billing").(map[string]interface{})
|
|
if st == nil || st["total"] == nil {
|
|
t.Fatalf("the healthy plugin stopped working after another plugin raised")
|
|
}
|
|
tot, _ := st["total"].(map[string]interface{})
|
|
if tot == nil || tot["requests"] == nil || tot["requests"].(float64) == 0 {
|
|
t.Errorf("billing recorded no requests after the raising plugin ran: %v", st["total"])
|
|
}
|
|
}
|