mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-11-12 04:07:25 +00:00
fix(lua): 定时任务改跑在插件自己的 VM 上 —— 纠正 Phase 4 的双 VM 错误
用户指出的设计错误:**一个插件 = 一个 Lua VM = 一个调度器**。而 Phase 4
(4d1407b)给定时任务开了一个**私有 VM**,于是:
定时任务 refresh_token 写 plugin.state.token → 写进私有 VM 的副本
请求钩子 读 plugin.state.token → 读主 VM 的值
轮换"成功"了,对实际流量完全无效;而且私有 VM 绕过状态持久化,token 连
磁盘都到不了。这直接违背这个功能存在的理由。
## 修正
PluginRuntime 不再持有 Lua 状态。任务通过 p.state + p.mu 运行 —— 与请求
钩子**同一个 VM、同一把锁**:
- 任务写入立即可见于钩子(TestTaskWriteIsVisibleToHooks 锁死这一点)
- 任务写入经 markDirtyLocked 走同一套持久化
- 跨插件仍并行(每插件各自 VM、各自锁)
## 由此产生的行为(有意为之,非缺陷)
- 某插件的长任务会排在**它自己**的钩子前面(同一 Lua 栈,一次一个调用者)
- 卡死处置随之改变:不再"抛弃 VM 重建"(那会丢掉累计状态——计费总额、
签到历史,正是插件存在的意义),而是**停用该插件**:
- checkWedged 超过 wedgedAfter(10min) 置 disarmed
- Fire 对 disarmed 插件快速失败并记 hook_errors(否则请求会排在永不释放
的锁后面永久挂起——比停用更糟)
- Unload 检测 InFlight:有任务在跑时不取 p.mu(会永久挂起)、不 Close
状态(Lua C 层 use-after-free 直接 abort)。泄漏这一个状态是有界的,
比崩溃好。
- maxAbandons/抛弃计数因此删除:没有"重生"了,熔断由 disarmed 承担
## 测试(10 个)
主断言从"任务跑过"改为"**任务的写入对请求路径可见**"—— 旧实现会在这里
失败。另含:无触发自执行/无 schedule 无运行时/解析/at 下次触发/错误隔离/
卸载停 goroutine/卡死停用+钩子快失败/健康任务不误判/8 插件并发各自调度。
This commit is contained in:
@ -245,12 +245,14 @@ type Plugin struct {
|
||||
persisted bool
|
||||
|
||||
// Schedule is the periodic work this plugin declared, and Runtime is the
|
||||
// long-lived environment that runs it (private VM + kernel ticker).
|
||||
// scheduler that runs it on THIS plugin's own VM (p.state) under p.mu — the
|
||||
// same state and the same lock a request hook uses. That is what makes a
|
||||
// token rotated by a task immediately visible to inference traffic.
|
||||
//
|
||||
// Runtime is nil when the plugin declared nothing — the common case stays
|
||||
// exactly as cheap as before: one shared state, no extra VM, no goroutine.
|
||||
// The point of it is auto check-in / token rotation: work that must happen
|
||||
// with no browser open and nobody clicking anything.
|
||||
// exactly as cheap as before: one VM, no extra goroutine. The point of the
|
||||
// schedule is auto check-in / token rotation: work that must happen with no
|
||||
// browser open and nobody clicking anything.
|
||||
Schedule []ScheduledTask
|
||||
Runtime *PluginRuntime
|
||||
}
|
||||
@ -708,8 +710,8 @@ func (ps *Plugins) LoadSource(name, code string) error {
|
||||
|
||||
// ---- schedule ----
|
||||
//
|
||||
// A plugin that declares `schedule` gets a LONG-RUNNING environment: a
|
||||
// private VM plus a kernel-driven ticker, so its periodic work (auto
|
||||
// A plugin that declares `schedule` gets a kernel-driven scheduler running
|
||||
// ON ITS OWN VM (the same p.state a hook uses), so its periodic work (auto
|
||||
// check-in, token rotation, quota polling) happens with no browser open and
|
||||
// no operator clicking anything. Without this the only periodic work a
|
||||
// plugin could do was a front-end poll, which is exactly the "nobody is
|
||||
@ -720,9 +722,9 @@ func (ps *Plugins) LoadSource(name, code string) error {
|
||||
p.Schedule = parseSchedule(w.L).tasks
|
||||
w.L.SetTop(0)
|
||||
if len(p.Schedule) > 0 {
|
||||
p.Runtime = ps.startRuntime(p.Info.Name, code, scheduleSpec{tasks: p.Schedule})
|
||||
p.Runtime = ps.startRuntime(p, scheduleSpec{tasks: p.Schedule})
|
||||
if p.Runtime == nil {
|
||||
// The private VM failed to boot. That must NOT silently disable the
|
||||
// The scheduler could not start. That must NOT silently disable the
|
||||
// plugin — hooks still work — but the operator needs to know their
|
||||
// scheduled work will never run.
|
||||
log.Printf("[scheduler] plugin %s: declared %d task(s) but no runtime could be started",
|
||||
@ -1792,13 +1794,29 @@ func (ps *Plugins) Unload(name string) error {
|
||||
p := ps.plugins[idx]
|
||||
ps.plugins = append(ps.plugins[:idx], ps.plugins[idx+1:]...)
|
||||
ps.mu.Unlock()
|
||||
// Stop the scheduler BEFORE closing the state: a task in flight holds that
|
||||
// state, and closing it underneath would leave the task writing into a
|
||||
// closed Lua state — a crash rather than a clean unload.
|
||||
// Stop the scheduler BEFORE touching the state. Stop() closes the loop and
|
||||
// waits for it, so after it returns no NEW task can be dispatched — the set
|
||||
// of in-flight tasks is now frozen, which is what makes the check below
|
||||
// meaningful rather than racy.
|
||||
inFlight := false
|
||||
if p.Runtime != nil {
|
||||
p.Runtime.Stop()
|
||||
inFlight = p.Runtime.InFlight()
|
||||
p.Runtime = nil
|
||||
}
|
||||
// A task holds p.mu for the whole duration of its Lua call. If one is still
|
||||
// in flight we must NOT take p.mu (a wedged task never releases it, so Lock
|
||||
// would hang the unload forever) and must NOT close the state underneath it
|
||||
// (that is a use-after-free in the Lua C layer, which aborts the process
|
||||
// instead of returning an error). Leaking this one state is the bounded,
|
||||
// non-crashing outcome; the operator's action is to remove the file and
|
||||
// restart, which is what a plugin wedged by its own task requires anyway.
|
||||
if inFlight {
|
||||
log.Printf("[scheduler] plugin %s: unloaded while a task was still running; "+
|
||||
"its Lua state is left open to avoid a use-after-free", name)
|
||||
ps.rebuild()
|
||||
return nil
|
||||
}
|
||||
p.mu.Lock()
|
||||
if p.state != nil && p.state.L != nil {
|
||||
p.state.L.Close()
|
||||
@ -1863,6 +1881,15 @@ func (ps *Plugins) Fire(stage Stage, payload map[string]interface{}) map[string]
|
||||
if p == nil || p.LoadError != "" {
|
||||
continue
|
||||
}
|
||||
// A plugin disarmed by an overrunning task still holds its own lock (the
|
||||
// task never returned), so dispatching to it would block THIS request
|
||||
// behind a lock that is never released. Skip it and record why: the
|
||||
// operator needs to see that inference traffic is being dropped and which
|
||||
// plugin is responsible, not watch requests hang.
|
||||
if p.Runtime != nil && p.Runtime.Disarmed() {
|
||||
ps.hookErr.note(stage, p.Info.Name+": "+errPluginWedged.Error())
|
||||
continue
|
||||
}
|
||||
targets = append(targets, target{p: p, hc: hc})
|
||||
}
|
||||
if len(targets) == 0 {
|
||||
|
||||
Reference in New Issue
Block a user