mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-11-07 09:47:23 +00:00
feat(plugins): 分片调用 —— 让阻塞型导出函数真正并行
## 问题
quota 页面 20 个源，刷新要 29 秒。页面用 `Promise.all` 并发发 20 个请求，
**看起来是并发的，实际不是**。
`Plugins.Call` 持 `p.mu.Lock()`，锁一直持有到 `L.Call` 返回 —— **包含 Lua 函数
内部的全部网络等待**。20 个请求到了网关后排队，串行累加。
生产实测：
单源探测（无竞争） 1.4 – 3.3 s
20 源 Promise.all 29 s 墙钟
串行理论值 ≈ 29 s ← 与实测吻合
代码里其实已经知道这个问题：quota.lua 第 104 行写着「the whole page shares ONE
Lua state under a write lock」，并为此把超时从 10s 压到 4s。但那只把 29s 变成
约 12s，没有解决「刷新不可用」。
而当初改成并发的理由（页面注释）是错的 —— 它假设并发能绕开串行。锁让它退化
回串行了。
## 不能去掉锁
一个 Lua VM 只有一个栈。两个并发调用会破坏栈。锁是必要的，答案只能是**让每个
并发调用有自己的 VM，跑同一份插件源码**。
先量，再写实现（8 分片 × 300ms 本地 HTTP 端点）：
共享 VM，8 次串行 2.407 s
8 个独立 VM 并行 0.304 s → 7.9x
顺带一个测量教训：这个 spike 第一版用**忙等**而不是网络 I/O，报出 2.3x。
那是量到了 12 核机器上的 CPU 争抢（负载 8.95），和锁无关。插件做的是网络等待，
spike 就必须做网络等待 —— 和上一条提交里 os.clock 那个错误是同一类。
## 实现
- `callOn(w, fn, args)`：从 `Call` 里抽出的「在某 VM 上调一个导出函数」。`Call`
从 66 行缩到 18 行，仍负责查找、export 白名单和共享 VM 的锁。
- `shardPool`：私有 VM 池，上限 32。开机约 0.1ms，池化省不了什么但省两次刷新
的重复开销。
- `ShardCall(name, fn, argsList)`：结果与输入**按位置对齐**。
- `POST /api/plugins/{name}/fanout/{fn}`：`{"args": [...]}`，上限 64 条。
- `pluginAPI.fanout()`：页面一次请求。
- quota.lua 改用 fanout。
## 分片的硬约束（写进注释，不是靠约定）
分片 VM **不能碰 PLUGIN.state** —— state 在 VM 里，每个分片一份副本，billing
的累加会各分叉，saver 写出去的是随机某一份。分片只能用于只读、幂等的工作。
所以 `callOn` 既不查 export 白名单（`Call` 那层才是安全边界）也不落盘。
## 一个分片失败不拖垮整批
一个上游连不上，不该让 20 张卡片变成一个错误。失败的分片在自己的位置留
null，其余照常返回。**但全部分片失败时报错** —— 那不是「某个上游挂了」，
是结构性问题，报成功会把它藏在满屏红卡片后面。
## 顺手修掉一个既有缺陷：热重载是坏的
写分片源码来源时，判据直接把它顶了出来：`LoadSource` 无条件 append，而
`findLocked` 返回**第一个**匹配。实测：
LoadDir 后 1 条 → LoadSource 后 2 条
[0] version=1 [1] version=2
find 返回 version=1
也就是说，**编辑插件并保存看起来什么都没发生** —— 旧版本一直被返回，新版本
躺在旁边不生效。修法是先 Unload 再加载。这不是为分片做的，是分片正好照到了
这条路径，已单独补判据 `TestReloadReplacesRatherThanAppends`。
## 判据
七条：并行性（有界而非计时断言，8×200ms 串行是 1.6s，超过 1.0s 即红）、
位置对齐、未导出与未知插件被拒、单个失败不拖批、全失败报错、用已安装源码而
非磁盘、以及热重载替换而非追加。
全量 9 包通过。
Co-Authored-By: ModelRouter <noreply@modelrouter.dev>
This commit is contained in:
JianFeeeee
210
internal/lua/plugin_shard.go
Normal file
210
internal/lua/plugin_shard.go
Normal file
@ -0,0 +1,210 @@
|
||||
package lua
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// ShardCall runs one exported function across N private VMs and returns the
|
||||
// results in the order the calls were given.
|
||||
//
|
||||
// Why this exists, measured on a 20-source gateway:
|
||||
//
|
||||
// one probe, no contention 1.4 – 3.3 s
|
||||
// 20 probes via Promise.all 29 s wall clock
|
||||
// the same 20 through one VM the SUM of all 20
|
||||
//
|
||||
// The page was written to fire its probes concurrently, and the browser really
|
||||
// did send twenty requests at once. They serialised anyway, in Plugins.Call,
|
||||
// which holds the plugin mutex for the whole call — including the network wait
|
||||
// inside the Lua function. A mutex around one Lua VM is not optional: the VM has
|
||||
// a single stack, so two concurrent calls would corrupt it. The fix cannot be
|
||||
// "drop the lock".
|
||||
//
|
||||
// What it can be: give each concurrent call its OWN VM, running the SAME plugin
|
||||
// source. Measured with a 300ms local endpoint, 8 shards:
|
||||
//
|
||||
// shared VM, 8 in sequence 2.407 s
|
||||
// 8 independent VMs 0.304 s → 7.9x
|
||||
//
|
||||
// A busy-wait version of the same spike reported only 2.3x, which was measuring
|
||||
// core contention on a loaded box rather than the lock. Network I/O is what the
|
||||
// plugin actually does, so network I/O is what the spike has to use.
|
||||
//
|
||||
// Why a private VM per call rather than a pool: a pool with a bounded size
|
||||
// still queues, and the queue is exactly the symptom being fixed. Booting a VM
|
||||
// is ~0.1ms, so a pool would buy nothing a fresh VM does not.
|
||||
//
|
||||
// What a shard VM must NOT do:
|
||||
//
|
||||
// - Touch PLUGIN.state. State lives in the Lua VM, and every shard would hold
|
||||
// its own copy of it. The billing plugin's totals would diverge per shard
|
||||
// and the saver would write whichever copy it happened to see. A shard is
|
||||
// therefore for READ-ONLY, idempotent work: a probe, a query, a computation.
|
||||
// - Register hooks. Hooks are the shared VM's job; a shard that registered one
|
||||
// would simply never be asked, which is a confusing way to lose it.
|
||||
//
|
||||
// The plugin decides whether a function is shardable by being listed. There is
|
||||
// no inference: a function that mutates state and is mistakenly listed would
|
||||
// write to a copy that is thrown away, silently.
|
||||
type shardPool struct {
|
||||
ps *Plugins
|
||||
|
||||
mu sync.Mutex
|
||||
idle []*worker
|
||||
live int
|
||||
peak int
|
||||
limit int
|
||||
}
|
||||
|
||||
// defaultShardLimit caps how many VMs a single shardCall may create.
|
||||
//
|
||||
// It is a memory bound, not a concurrency policy. Each VM costs roughly a
|
||||
// megabyte of Lua state for a plugin this size, and a page that fires twenty
|
||||
// probes should not be able to make the gateway allocate twenty VMs on demand
|
||||
// with no ceiling. Twenty sources is the measured real case, so 32 leaves
|
||||
// headroom without being unbounded.
|
||||
const defaultShardLimit = 32
|
||||
|
||||
func newShardPool(ps *Plugins) *shardPool {
|
||||
return &shardPool{ps: ps, limit: defaultShardLimit}
|
||||
}
|
||||
|
||||
// ShardCall is the Plugins-level entry point.
|
||||
func (ps *Plugins) ShardCall(name, fn string, argsList []map[string]interface{}) ([]interface{}, error) {
|
||||
return ps.shards.ShardCall(name, fn, argsList)
|
||||
}
|
||||
|
||||
// ShardCall runs fn once per element of argsList, each on its own VM, and
|
||||
// returns results positionally aligned with argsList.
|
||||
//
|
||||
// A shard that fails does not fail the batch. Its slot gets an error value and
|
||||
// the others still return — the alternative is one unreachable upstream
|
||||
// turning a twenty-card page into a single error, which is the behaviour this
|
||||
// whole change exists to remove.
|
||||
func (sp *shardPool) ShardCall(name, fn string, argsList []map[string]interface{}) ([]interface{}, error) {
|
||||
if len(argsList) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
p := sp.ps.find(name)
|
||||
if p == nil {
|
||||
return nil, fmt.Errorf("plugin %s not found", name)
|
||||
}
|
||||
if !pluginExportAllowed(p, fn) {
|
||||
return nil, fmt.Errorf("plugin %s does not export %q", name, fn)
|
||||
}
|
||||
// The source comes from the SHARED plugin's own record, so a shard runs
|
||||
// exactly the code that is serving requests right now — including a version
|
||||
// installed through the API, which never touches a file. Reading ps.dir
|
||||
// instead would shard the previous version of an edited plugin.
|
||||
code := p.script
|
||||
if code == "" {
|
||||
return nil, fmt.Errorf("plugin %s has no source to shard", name)
|
||||
}
|
||||
|
||||
out := make([]interface{}, len(argsList))
|
||||
errs := make([]error, len(argsList))
|
||||
var wg sync.WaitGroup
|
||||
for i := range argsList {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
w, release, err := sp.acquire(code, name)
|
||||
if err != nil {
|
||||
errs[i] = err
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
res, err := callOn(w, fn, argsList[i])
|
||||
if err != nil {
|
||||
errs[i] = fmt.Errorf("%s: %w", name, err)
|
||||
return
|
||||
}
|
||||
out[i] = res
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
var failed int
|
||||
for _, err := range errs {
|
||||
if err != nil {
|
||||
failed++
|
||||
}
|
||||
}
|
||||
if failed == len(argsList) {
|
||||
// Every shard failed. That is not "one upstream is down" — something is
|
||||
// structurally wrong, and reporting success with N error values would
|
||||
// hide it behind a page full of red cards.
|
||||
return nil, fmt.Errorf("all %d shards of %s.%s failed: %v", len(argsList), name, fn, errs[0])
|
||||
}
|
||||
if failed > 0 {
|
||||
if sp.ps.logf != nil {
|
||||
sp.ps.logf("[plugins] %s.%s: %d/%d shards failed", name, fn, failed, len(argsList))
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// acquire takes an idle VM or boots one, respecting the limit.
|
||||
func (sp *shardPool) acquire(code, name string) (*worker, func(), error) {
|
||||
sp.mu.Lock()
|
||||
if n := len(sp.idle); n > 0 {
|
||||
w := sp.idle[n-1]
|
||||
sp.idle = sp.idle[:n-1]
|
||||
sp.mu.Unlock()
|
||||
return w, func() { sp.release(w) }, nil
|
||||
}
|
||||
if sp.limit > 0 && sp.live >= sp.limit {
|
||||
sp.mu.Unlock()
|
||||
return nil, nil, fmt.Errorf("shard limit %d reached", sp.limit)
|
||||
}
|
||||
sp.live++
|
||||
sp.mu.Unlock()
|
||||
|
||||
w, err := bootPluginState(code, name, sp.ps.outboundAllowlist())
|
||||
if err != nil {
|
||||
sp.mu.Lock()
|
||||
sp.live--
|
||||
sp.mu.Unlock()
|
||||
return nil, nil, err
|
||||
}
|
||||
return w, func() { sp.release(w) }, nil
|
||||
}
|
||||
|
||||
func (sp *shardPool) release(w *worker) {
|
||||
if w == nil || w.L == nil {
|
||||
return
|
||||
}
|
||||
sp.mu.Lock()
|
||||
// Keep a small idle set: booting is cheap but not free, and a page that
|
||||
// refreshes twice in a row should not pay for two rounds of VMs.
|
||||
if len(sp.idle) < sp.limit {
|
||||
sp.idle = append(sp.idle, w)
|
||||
sp.mu.Unlock()
|
||||
return
|
||||
}
|
||||
sp.live--
|
||||
sp.mu.Unlock()
|
||||
w.L.Close()
|
||||
}
|
||||
|
||||
// Shutdown closes every pooled VM. Called from Plugins.Close.
|
||||
func (sp *shardPool) Shutdown() {
|
||||
if sp == nil {
|
||||
return
|
||||
}
|
||||
sp.mu.Lock()
|
||||
idle := sp.idle
|
||||
sp.idle = nil
|
||||
sp.live = 0
|
||||
sp.mu.Unlock()
|
||||
for _, w := range idle {
|
||||
if w != nil && w.L != nil {
|
||||
w.L.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A single-request fan-out does not belong in this file: it needs the HTTP layer
|
||||
// (routing, auth, JSON encoding) and would be one thin wrapper over ShardCall.
|
||||
// The gateway route will call ShardCall directly with the sources it parsed.
|
||||
Reference in New Issue
Block a user