mirror of
https://gitcode.com/JianFeeeee/webui4frpc.git
synced 2026-10-15 12:14:37 +00:00
按用户要求重做:不再由 leader 单独调度,而是**每个节点对自己的转发跑同一条
规则**,超了就自己撤销一条、把任务丢回环上,由环原有的「最闲者认领」决定它
落到哪台。
## 为什么必须是 owner 自己动手(不是洁癖,是实测逼出来的)
真机探针(跑完已删)显示,真实 revoke 之后:
TopologyOwner after revoke = "n1" ← 条目还在(只标记不删除)
n1 Forwards=3 score=310.0 ← 负载根本没降
after step: revoking=true publishing=false
⇒ 一节点的负载**只有它自己的拓扑条目消失时才会下降**,而条目是本地簿记,没有任何
对端可以代它写。leader 从外面只能看见失衡、无法让失衡成为事实——它必须伸手去改
另一个节点的簿记,而这正是这个环到处都在避免的跨节点写(revoke / restart 定向往
owner 也是同一个理由)。
⇒ 而且旧实现**根本跑不起来**:phase 1 判据是「条目消失」,可 mark-don't-remove
让它永远不消失,于是每 120s 打一条 `revoking … timed out; abandoning`,
**phase 2 永不执行,一条转发都不会迁**。开 `W4F_BALANCE_MODE=on` 只会刷日志。
## 规则(替代原来的 ratio 阈值)
当 Forwards >= 2 且 Forwards > ceil(总数/存活节点数) 时,撤销一条
- **自限**:3 条 / 3 台 → owner 2 > 1,卸一条 → 1,`1 > 1` 不成 → 停。
没有上界的话 owner 会把转发**全部卸光**。
- **不会卸空**:低于下限(2)永不撤销。
- 平摊值向上取整(3/3=1,不是 0),否则 1 条 / 3 台的节点也算「超了」。
- **不再用 mem/net 参与判据**:实测 `.30` 空闲时 netPct 能到 55%,Score 会高于
有转发的节点——用 Score 会把纯网络抖动当成负载。
- 节奏用**每节点冷却**(默认 600s)而不是「失衡持续多久」窗口:规则本身已经要求
真实失衡,没有需要平滑的瞬态,只有需要限制的抖动频率。
## 为什么必须是新的 Release 任务类型,不能复用 Revoke
用户停止(Revoke)**保留拓扑条目**作为 disabled 标志的载体;均衡释放必须**删除
条目**,因为 (a) 负载由自己的条目数算出,条目不删负载不降;(b) 每个节点的
duplicate-guard 会拒绝「已有 owner 的转发」的创建任务——保留条目会让**所有节点
都拒领**,转发被永久搁死在刚放弃它的那台机器上。
Release 因此:`Handler.Release`(停 worker,但**不标记 stopped**)→ 删条目 →
`SubmitTask` 以**启用状态**重新发布。它需要自己的 Handler 方法,不能回落到
Revoke:Revoke 会顺手把 link 标成 stopped,而这条转发即将在别的节点上跑。
## 顺带修掉两个既有缺口(实测确认)
1. **`submitMigrate` 发布不出去**:`OnToken` 的 localPending 重合并只保留「从未
publish 过」的本地 pending,而只有 `Add*` 系列会记 `published`。引擎里直接写
`PendingTasks` 的任务会在下一次 adopt 时被静默丢弃。
⇒ 新 `ReleaseTask` 走 `AddPending`。
(`SubmitRestart` 有同样的毛病,但它不在本次范围,未改动——见提交说明末尾。)
2. `Release` 需要绕过 duplicate-claim 守卫(同 Restart),否则「owner 本来就持有
这条转发」这一前提会让每条 Release 都被丢掉。
## 测试(17 条,全绿)
规则:平摊值取整、超了才撤、撤到平摊就停、单节点不撤、低于下限不撤、不撤已停
转发、候选选择确定。
机制:Revoke 留载体 vs Release 删条目(这是两种语义的核心差异)、Release 只由
owner 执行且不消费他人的任务、owner 释放后环的普通认领把它接走。
模式:dry-run 不动任何东西、off 真的什么都不做、未知模式回落 dry-run、
冷却限制抖动频率、确认超时清理。
只读:Report 不变更任何状态、能解释「超了但不动」的原因、均衡节点给出真实答案。
**双向验证 3 条**(改坏→变红→还原→变绿):
- 去掉 Release 的 duplicate-guard 例外 → `the owner released 0 times, want 1`
- 去掉「不超过平摊值」上界 → `2 owned against share 2 must not release`
- Release 改成只标记不删除 → `n1 should now own 2, got 3`
go build / go vet / go test ./... 全绿(go clean -testcache 后逐包实跑)。
## 未做 / 待办
- `SubmitRestart` 的同类发布缺口未修(同样会被 adopt 丢弃),需要单独确认它
在实机上是否真的丢过 restart。
- UI 尚未跟进新报告结构(下一步)。
206 lines
7.6 KiB
Go
206 lines
7.6 KiB
Go
package cluster
|
|
|
|
import (
|
|
"context"
|
|
"testing"
|
|
|
|
"webui4frpc/internal/store"
|
|
)
|
|
|
|
type fakeHandler struct {
|
|
load Load
|
|
claim func(ctx context.Context, tk *Task) error
|
|
revoke func(ctx context.Context, tk *Task) error
|
|
restart func(ctx context.Context, tk *Task) error
|
|
release func(ctx context.Context, tk *Task) error
|
|
}
|
|
|
|
func (h *fakeHandler) Release(ctx context.Context, tk *Task) error {
|
|
if h.release != nil {
|
|
return h.release(ctx, tk)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (h *fakeHandler) Revoke(ctx context.Context, tk *Task) error {
|
|
if h.revoke != nil {
|
|
return h.revoke(ctx, tk)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (h *fakeHandler) Restart(ctx context.Context, tk *Task) error {
|
|
if h.restart != nil {
|
|
return h.restart(ctx, tk)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (h *fakeHandler) RuntimeLoad() Load { return h.load }
|
|
func (h *fakeHandler) Claim(ctx context.Context, tk *Task) error {
|
|
if h.claim != nil {
|
|
return h.claim(ctx, tk)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// sendNull is a no-op sender (single-node ring).
|
|
func sendNull(ctx context.Context, next string, tk *Token) error { return nil }
|
|
|
|
// TestSingleNodeCycle: leader starts ring, processes phases locally, and the
|
|
// task gets claimed (lowest load = only node).
|
|
func TestSingleNodeCycle(t *testing.T) {
|
|
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", []string{"0.71.0"},
|
|
&fakeHandler{load: Load{MemPct: 30, NetPct: 30}}, sendNull, "n1:7500", true, "")
|
|
eng.StartRing(context.Background())
|
|
|
|
// single node: token stays local; no successor so nothing travels.
|
|
if eng.State().Nodes[0].ID != "n1" {
|
|
t.Fatalf("nodes = %+v", eng.State().Nodes)
|
|
}
|
|
}
|
|
|
|
// TestTwoNodesPhaseCollect: leader n1 sends to n2; n1 collects both after n2
|
|
// stamps; then phase flips and second round syncs.
|
|
func TestTwoNodesSingleRound(t *testing.T) {
|
|
eng2 := NewEngine("n2", "n2:7500", "u", "p", "0.71.0", nil,
|
|
&fakeHandler{load: Load{MemPct: 10, NetPct: 10}}, sendNull, "n2:7500", false, "")
|
|
eng2.state.UpsertNode(Node{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true, Load: Load{MemPct: 50, NetPct: 50}})
|
|
|
|
// n1 sends a token carrying the cluster picture; single-round processing:
|
|
// the receiving node adopts it and appends its own info in one pass.
|
|
tk := &Token{Cycle: 1, State: State{
|
|
Nodes: []Node{
|
|
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true},
|
|
{ID: "n2", Addr: "n2:7500", Alive: true},
|
|
},
|
|
}}
|
|
|
|
out, err := eng2.OnToken(context.Background(), tk)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if eng2.State().Find("n2") < 0 {
|
|
t.Fatal("n2 should be present in adopted state after single-round")
|
|
}
|
|
if out == nil {
|
|
t.Fatal("nil token after processing")
|
|
}
|
|
}
|
|
|
|
// TestLeaderRoundTripNoResurrection: when the leader submits a task and the
|
|
// token completes a full round (leader→n2 claims→n3→leader), the consumed task
|
|
// MUST NOT be resurrected on the leader's next OnToken, and the topology must
|
|
// still attribute the forward to the single claimer (n2).
|
|
//
|
|
// This holds NOT via a published-set mark on StartRing, but via Go map
|
|
// reference semantics: StartRing sets tk.State = e.state, so the token's
|
|
// PendingTasks map IS the leader's own map. When n2 calls ClaimPending it
|
|
// deletes from that shared map — the deletion is visible to the leader too.
|
|
// By the time the token returns, the claimed task is already gone from the
|
|
// leader's e.state.PendingTasks, so the localPending re-merge has nothing to
|
|
// re-inject. Single token + remove-on-claim = single claim (plan §M6).
|
|
func TestLeaderRoundTripNoResurrection(t *testing.T) {
|
|
// 3-node ring: n1 (leader+submitter), n2 (lowest, claims), n3 (idle).
|
|
var sent *Token
|
|
sendCap := func(ctx context.Context, next string, tk *Token) error {
|
|
sent = tk
|
|
return nil
|
|
}
|
|
n1 := NewEngine("n1", "n1:7500", "u", "p", "v", nil,
|
|
&fakeHandler{load: Load{MemPct: 50, NetPct: 50}}, sendCap, "n1:7500", true, "")
|
|
// n2/n3 carry a lower stored load than n1's NewEngine default ({10,10})
|
|
// so LowestAlive picks n2 (first among the tied low nodes) as the claimer.
|
|
n1.state.UpsertNode(Node{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{MemPct: 1, NetPct: 1}})
|
|
n1.state.UpsertNode(Node{ID: "n3", Addr: "n3:7500", Alive: true, Load: Load{MemPct: 1, NetPct: 1}})
|
|
|
|
task := n1.SubmitTask(store.Local{Name: "l1"}, store.Remote{Name: "r1"}, store.Link{RemotePort: 100})
|
|
if task == nil {
|
|
t.Fatal("submit returned nil")
|
|
}
|
|
n1.StartRing(context.Background())
|
|
if sent == nil {
|
|
t.Fatal("StartRing did not send a token")
|
|
}
|
|
|
|
// n2 receives, claims t1 (lowest load), establishes topology.
|
|
n2 := NewEngine("n2", "n2:7500", "u", "p", "v", nil,
|
|
&fakeHandler{load: Load{MemPct: 10, NetPct: 10}}, sendCap, "n2:7500", false, "")
|
|
out2, err := n2.OnToken(context.Background(), sent)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if got := len(n2.state.TopologyList()); got != 1 {
|
|
t.Fatalf("n2 should own 1 forward, topo=%+v", n2.state.TopologyList())
|
|
}
|
|
if len(n2.state.PendingList()) != 0 {
|
|
t.Fatalf("pending should be empty after n2 claim, got %+v", n2.state.PendingList())
|
|
}
|
|
|
|
// n3 receives, nothing to claim, forwards.
|
|
n3 := NewEngine("n3", "n3:7500", "u", "p", "v", nil,
|
|
&fakeHandler{load: Load{MemPct: 10, NetPct: 10}}, sendCap, "n3:7500", false, "")
|
|
out3, err := n3.OnToken(context.Background(), out2)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
|
|
// Token returns to n1. The consumed task t1 MUST NOT be resurrected.
|
|
if _, err := n1.OnToken(context.Background(), out3); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if got := len(n1.state.PendingList()); got != 0 {
|
|
t.Fatalf("n1 resurrected a consumed task: pending=%+v (single-token + shared-map must prevent this)",
|
|
n1.state.PendingList())
|
|
}
|
|
// Topology still attributes t1 to n2 (not overwritten by a re-claim).
|
|
topo := n1.state.TopologyList()
|
|
if len(topo) != 1 || topo[0].OwnerID != "n2" {
|
|
t.Fatalf("topology should be t1@n2, got %+v", topo)
|
|
}
|
|
}
|
|
|
|
// TestClaimGuardDropsDuplicateForward: when a node (lowest load) receives a
|
|
// pending task whose forward is ALREADY in the topology owned by another node
|
|
// (a resurrected/stale copy), the claim path must drop it WITHOUT spawning a
|
|
// worker or overwriting the topology. Without the guard a second worker for
|
|
// the same forward would be spawned and orphaned (the topology entry is keyed
|
|
// by task id, so AddTopology would silently overwrite the owner).
|
|
func TestClaimGuardDropsDuplicateForward(t *testing.T) {
|
|
// n1 is lowest load and would otherwise claim; the fake Claim hook fails
|
|
// the test if ever called.
|
|
n1 := NewEngine("n1", "n1:7500", "u", "p", "v", nil,
|
|
&fakeHandler{
|
|
load: Load{MemPct: 10, NetPct: 10},
|
|
claim: func(ctx context.Context, tk *Task) error {
|
|
t.Fatalf("Claim must not be called for an already-owned forward: %s", tk.ID)
|
|
return nil
|
|
},
|
|
}, sendNull, "n1:7500", true, "")
|
|
|
|
dup := &Task{ID: "t1", Local: store.Local{Name: "l1"}, Remote: store.Remote{Name: "r1"}, Link: store.Link{RemotePort: 100}}
|
|
tk := &Token{Cycle: 1, State: State{
|
|
Nodes: []Node{
|
|
{ID: "n1", Addr: "n1:7500", Alive: true, Load: Load{MemPct: 10, NetPct: 10}},
|
|
{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{MemPct: 50, NetPct: 50}},
|
|
},
|
|
PendingTasks: map[string]*Task{"t1": dup},
|
|
Topology: map[string]*TopoEntry{"t1": {
|
|
TaskID: "t1", OwnerID: "n2", Local: store.Local{Name: "l1"},
|
|
Remote: store.Remote{Name: "r1"}, Link: store.Link{RemotePort: 100}, Active: true,
|
|
}},
|
|
}}
|
|
if _, err := n1.OnToken(context.Background(), tk); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
// Pending drained (the stale task was consumed/dropped, not left to ride).
|
|
if got := len(n1.state.PendingList()); got != 0 {
|
|
t.Fatalf("stale task should be dropped, pending=%+v", n1.state.PendingList())
|
|
}
|
|
// Topology untouched: still owned by n2.
|
|
topo := n1.state.TopologyList()
|
|
if len(topo) != 1 || topo[0].OwnerID != "n2" {
|
|
t.Fatalf("topology should remain t1@n2, got %+v", topo)
|
|
}
|
|
}
|