Files
webui4frpc/internal/cluster/balance_test.go
JianFeeeee c69fb2caba feat(cluster): 负载均衡改为每节点自主「先撤销本地转发,再丢回环」
按用户要求重做:不再由 leader 单独调度,而是**每个节点对自己的转发跑同一条
规则**,超了就自己撤销一条、把任务丢回环上,由环原有的「最闲者认领」决定它
落到哪台。

## 为什么必须是 owner 自己动手(不是洁癖,是实测逼出来的)

真机探针(跑完已删)显示,真实 revoke 之后:

    TopologyOwner after revoke = "n1"     ← 条目还在(只标记不删除)
    n1 Forwards=3  score=310.0            ← 负载根本没降
    after step: revoking=true publishing=false

⇒ 一节点的负载**只有它自己的拓扑条目消失时才会下降**,而条目是本地簿记,没有任何
对端可以代它写。leader 从外面只能看见失衡、无法让失衡成为事实——它必须伸手去改
另一个节点的簿记,而这正是这个环到处都在避免的跨节点写(revoke / restart 定向往
owner 也是同一个理由)。

⇒ 而且旧实现**根本跑不起来**:phase 1 判据是「条目消失」,可 mark-don't-remove
让它永远不消失,于是每 120s 打一条 `revoking … timed out; abandoning`,
**phase 2 永不执行,一条转发都不会迁**。开 `W4F_BALANCE_MODE=on` 只会刷日志。

## 规则(替代原来的 ratio 阈值)

    当 Forwards >= 2 且 Forwards > ceil(总数/存活节点数) 时,撤销一条

- **自限**:3 条 / 3 台 → owner 2 > 1,卸一条 → 1,`1 > 1` 不成 → 停。
  没有上界的话 owner 会把转发**全部卸光**。
- **不会卸空**:低于下限(2)永不撤销。
- 平摊值向上取整(3/3=1,不是 0),否则 1 条 / 3 台的节点也算「超了」。
- **不再用 mem/net 参与判据**:实测 `.30` 空闲时 netPct 能到 55%,Score 会高于
  有转发的节点——用 Score 会把纯网络抖动当成负载。
- 节奏用**每节点冷却**(默认 600s)而不是「失衡持续多久」窗口:规则本身已经要求
  真实失衡,没有需要平滑的瞬态,只有需要限制的抖动频率。

## 为什么必须是新的 Release 任务类型,不能复用 Revoke

用户停止(Revoke)**保留拓扑条目**作为 disabled 标志的载体;均衡释放必须**删除
条目**,因为 (a) 负载由自己的条目数算出,条目不删负载不降;(b) 每个节点的
duplicate-guard 会拒绝「已有 owner 的转发」的创建任务——保留条目会让**所有节点
都拒领**,转发被永久搁死在刚放弃它的那台机器上。

Release 因此:`Handler.Release`(停 worker,但**不标记 stopped**)→ 删条目 →
`SubmitTask` 以**启用状态**重新发布。它需要自己的 Handler 方法,不能回落到
Revoke:Revoke 会顺手把 link 标成 stopped,而这条转发即将在别的节点上跑。

## 顺带修掉两个既有缺口(实测确认)

1. **`submitMigrate` 发布不出去**:`OnToken` 的 localPending 重合并只保留「从未
   publish 过」的本地 pending,而只有 `Add*` 系列会记 `published`。引擎里直接写
   `PendingTasks` 的任务会在下一次 adopt 时被静默丢弃。
   ⇒ 新 `ReleaseTask` 走 `AddPending`。
   (`SubmitRestart` 有同样的毛病,但它不在本次范围,未改动——见提交说明末尾。)
2. `Release` 需要绕过 duplicate-claim 守卫(同 Restart),否则「owner 本来就持有
   这条转发」这一前提会让每条 Release 都被丢掉。

## 测试(17 条,全绿)

规则:平摊值取整、超了才撤、撤到平摊就停、单节点不撤、低于下限不撤、不撤已停
转发、候选选择确定。
机制:Revoke 留载体 vs Release 删条目(这是两种语义的核心差异)、Release 只由
owner 执行且不消费他人的任务、owner 释放后环的普通认领把它接走。
模式:dry-run 不动任何东西、off 真的什么都不做、未知模式回落 dry-run、
冷却限制抖动频率、确认超时清理。
只读:Report 不变更任何状态、能解释「超了但不动」的原因、均衡节点给出真实答案。

**双向验证 3 条**(改坏→变红→还原→变绿):
- 去掉 Release 的 duplicate-guard 例外 → `the owner released 0 times, want 1`
- 去掉「不超过平摊值」上界 → `2 owned against share 2 must not release`
- Release 改成只标记不删除 → `n1 should now own 2, got 3`

go build / go vet / go test ./... 全绿(go clean -testcache 后逐包实跑)。

## 未做 / 待办

- `SubmitRestart` 的同类发布缺口未修(同样会被 adopt 丢弃),需要单独确认它
  在实机上是否真的丢过 restart。
- UI 尚未跟进新报告结构(下一步)。
2026-10-06 14:03:03 +08:00

622 lines
22 KiB
Go

package cluster
import (
"context"
"encoding/json"
"strconv"
"strings"
"testing"
"time"
"webui4frpc/internal/store"
)
// mustState round-trips the seed through JSON so the engine under test owns its
// own copy of Nodes/Topology. AdoptState assigns the state by value, but Nodes is
// a slice: handing one seed to several engines would alias one backing array and
// make a mutation on one node visible on the others — a test artefact that
// previously produced a false green here.
func mustState(t *testing.T, s State) *State {
t.Helper()
b, err := json.Marshal(s)
if err != nil {
t.Fatal(err)
}
var out State
if err := json.Unmarshal(b, &out); err != nil {
t.Fatal(err)
}
e := newTestEngine("probe", true)
e.AdoptState(out)
return e.State()
}
// migrateRing builds a 3-node ring where n1 owns every forward, plus n2/n3 idle.
func migrateRing(t *testing.T, ownCount int) (*Engine, *Engine, *Engine) {
t.Helper()
mk := func(id string, leader bool) *Engine {
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5}}
return NewEngine(id, id+":7500", "u", "p", "0.1.3", nil, h,
func(ctx context.Context, next string, tk *Token) error { return nil },
id+":7500", leader, "")
}
n1, n2, n3 := mk("n1", true), mk("n2", false), mk("n3", false)
seed := State{
LeaderID: "n1",
Nodes: []Node{
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true, Load: Load{Forwards: ownCount, MemPct: 5}},
{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{MemPct: 5}},
{ID: "n3", Addr: "n3:7500", Alive: true, Load: Load{MemPct: 5}},
},
Topology: map[string]*TopoEntry{},
PendingTasks: map[string]*Task{},
}
for _, e := range []*Engine{n1, n2, n3} {
var own State
b, _ := json.Marshal(seed)
if err := json.Unmarshal(b, &own); err != nil {
t.Fatal(err)
}
e.AdoptState(own)
}
// n1 claims `ownCount` forwards.
for i := 0; i < ownCount; i++ {
n1.state.AddTopology(&Task{
ID: "f" + string(rune('a'+i)),
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
Link: store.Link{RemotePort: 100 + i},
}, "n1")
}
var snap State
b, _ := json.Marshal(n1.state)
_ = json.Unmarshal(b, &snap)
for _, e := range []*Engine{n1, n2, n3} {
e.AdoptState(snap)
}
return n1, n2, n3
}
// ---- helpers ----
// balanceRing builds a 3-node ring with `owned` forwards on n1 (the evaluation
// node) and `others` on each of the other two, so a test can set up any
// distribution directly. The replicated Nodes[].Load is made consistent with the
// topology, because that is what a real token round leaves behind.
func balanceRing(t *testing.T, owned, others int) (*Engine, *Engine, *Engine) {
t.Helper()
mk := func(id string, leader bool) *Engine {
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5}}
return NewEngine(id, id+":7500", "u", "p", "0.1.3", nil, h,
func(ctx context.Context, next string, tk *Token) error { return nil },
id+":7500", leader, "")
}
n1, n2, n3 := mk("n1", true), mk("n2", false), mk("n3", false)
seed := State{
LeaderID: "n1",
Nodes: []Node{
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true, Load: Load{Forwards: owned, MemPct: 5}},
{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{Forwards: others, MemPct: 5}},
{ID: "n3", Addr: "n3:7500", Alive: true, Load: Load{Forwards: others, MemPct: 5}},
},
Topology: map[string]*TopoEntry{},
PendingTasks: map[string]*Task{},
}
for _, e := range []*Engine{n1, n2, n3} {
var own State
b, _ := json.Marshal(seed)
if err := json.Unmarshal(b, &own); err != nil {
t.Fatal(err)
}
e.AdoptState(own)
}
// Build the whole topology in n1's state, then broadcast it. Adding each
// node's forwards to its OWN state and then re-adopting from n1 would discard
// the others (one state is the source of the broadcast), which is how an
// earlier version of this helper silently produced a 1-1-1 split regardless
// of the arguments.
add := func(owner, prefix string, count, basePort int) {
for i := 0; i < count; i++ {
n1.state.AddTopology(&Task{
ID: prefix + strconv.Itoa(i),
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
Link: store.Link{RemotePort: basePort + i},
}, owner)
}
}
add("n1", "f", owned, 100)
add("n2", "g", others, 200)
add("n3", "h", others, 300)
var snap State
b, _ := json.Marshal(n1.state)
_ = json.Unmarshal(b, &snap)
for _, e := range []*Engine{n1, n2, n3} {
e.AdoptState(snap)
}
return n1, n2, n3
}
// ---- the rule ----
// TestShareIsTheEvenSplit pins the arithmetic, including the ceiling. An even
// split of 3 over 3 is 1 (not 1.0 floored to 0): with a floor, a node holding
// one of three forwards would still be "over its share" and shed it.
func TestShareIsTheEvenSplit(t *testing.T) {
for _, tc := range []struct{ total, alive, want int }{
{3, 3, 1}, {4, 3, 2}, {6, 2, 3}, {1, 3, 1}, {0, 3, 0}, {7, 3, 3},
} {
got := (tc.total + tc.alive - 1) / tc.alive
if got != tc.want {
t.Errorf("share(%d over %d) = %d, want %d", tc.total, tc.alive, got, tc.want)
}
}
}
// TestOwnerReleasesWhenOverItsShare: 3 forwards on n1, none elsewhere → n1 holds
// 3 against a share of 1 and must give one up.
func TestOwnerReleasesWhenOverItsShare(t *testing.T) {
n1, _, _ := balanceRing(t, 3, 0)
b := NewBalancer(n1, 0)
pick, p, blocked := b.assess(1000)
if pick == nil {
t.Fatalf("n1 holds 3 of 3 forwards and should release one (blocked=%q)", blocked)
}
if blocked != "" {
t.Errorf("nothing should block the first release, got %q", blocked)
}
if p.mine != 3 || p.share != 1 || p.total != 3 || p.alive != 3 {
t.Errorf("pool = %+v, want mine=3 share=1 total=3 alive=3", p)
}
if pick.OwnerID != "n1" {
t.Errorf("must release a forward n1 owns, got owner %q", pick.OwnerID)
}
}
// TestReleasingStopsAtTheShare is the bound that keeps the rule from emptying a
// node: after shedding one, n1 holds 1 of 3 over 3 nodes, which is no longer more
// than the share, and it must stop.
func TestReleasingStopsAtTheShare(t *testing.T) {
n1, _, _ := balanceRing(t, 1, 1) // even 1-1-1 split
b := NewBalancer(n1, 0)
pick, p, blocked := b.assess(1000)
if pick != nil {
t.Errorf("an even 1-1-1 split must not release anything, got %s:%d (blocked=%q)",
pick.Local.Name, pick.Link.RemotePort, blocked)
}
if p.mine != 1 || p.share != 1 {
t.Errorf("pool = %+v, want mine=1 share=1", p)
}
}
// TestSingleNodeNeverReleases: a standalone node is the whole cluster, so its
// share is everything it holds. Without this the node would shed forwards onto
// nobody (they would sit unclaimed until it re-claimed them).
func TestSingleNodeNeverReleases(t *testing.T) {
e := newTestEngine("solo", true)
e.state.Nodes = []Node{{ID: "solo", Addr: "solo:7500", Alive: true, IsLeader: true}}
for i := 0; i < 5; i++ {
e.state.AddTopology(&Task{
ID: "f" + strconv.Itoa(i), Local: store.Local{Name: "web"},
Remote: store.Remote{Name: "frps1"}, Link: store.Link{RemotePort: 100 + i},
}, "solo")
}
b := NewBalancer(e, 0)
if pick, _, blocked := b.assess(1000); pick != nil {
t.Errorf("a single-node ring must not release; got blocked=%q", blocked)
}
}
// TestBelowFloorNeverReleases: two forwards on a node is the floor. A node
// holding exactly two at an even split of two must keep both — one release per
// node per pass is the granularity, and shedding below the floor would empty it.
func TestBelowFloorNeverReleases(t *testing.T) {
n1, _, _ := balanceRing(t, 2, 2) // 6 total over 3 nodes → share 2
b := NewBalancer(n1, 0)
if pick, p, blocked := b.assess(1000); pick != nil {
t.Errorf("2 owned against share %d must not release (blocked=%q, mine=%d)",
p.share, blocked, p.mine)
}
}
// TestNeverReleasesAStoppedForward: a stopped forward's entry is the carrier of
// the user's stop decision. Releasing it would hand another node a STOPPED
// forward as if it were work to run.
func TestNeverReleasesAStoppedForward(t *testing.T) {
n1, _, _ := balanceRing(t, 3, 0)
// Stop two of n1's three; the only live one must be chosen.
for _, te := range n1.state.ForwardsOwnedBy("n1") {
if te.Link.RemotePort == 100 || te.Link.RemotePort == 101 {
n1.state.UpdateTopologyDisabled("web", "frps1", te.Link.RemotePort, true)
}
}
b := NewBalancer(n1, 0)
pick, _, _ := b.assess(1000)
if pick == nil {
t.Fatal("n1 still owns one live forward and is over its share")
}
if pick.Link.RemotePort != 102 {
t.Errorf("released port %d; a stopped forward (100/101) must never be chosen",
pick.Link.RemotePort)
}
// And with every forward stopped there is nothing to release.
n1.state.UpdateTopologyDisabled("web", "frps1", 102, true)
if pick, _, blocked := b.assess(1000); pick != nil {
t.Errorf("all forwards stopped: nothing may be released (blocked=%q)", blocked)
}
}
// TestCandidateChoiceIsDeterministic: the same state must pick the same forward,
// so the decision is explainable and a re-evaluation does not churn a different
// connection each tick.
func TestCandidateChoiceIsDeterministic(t *testing.T) {
n1, _, _ := balanceRing(t, 4, 0)
a := releaseCandidate(n1.State(), "n1")
c := releaseCandidate(n1.State(), "n1")
if a == nil || c == nil {
t.Fatal("expected a candidate")
}
if a.TaskID != c.TaskID || a.Link.RemotePort != c.Link.RemotePort {
t.Errorf("candidate changed between identical evaluations: %s:%d vs %s:%d",
a.TaskID, a.Link.RemotePort, c.TaskID, c.Link.RemotePort)
}
if a.Link.RemotePort != 100 {
t.Errorf("expected the lowest port (100), got %d", a.Link.RemotePort)
}
}
// ---- the runtime signal that makes this safe ----
// TestRevokeLeavesACarrierAndReleaseRemovesIt is the reason the release path has
// to be a distinct task kind.
//
// A user stop keeps the topology entry as the carrier of the disabled flag, so
// the entry is STILL THERE and still attributed to the owner. A rebalance is the
// opposite: the entry must go, because (a) the node's load is derived from
// owned entries so it cannot fall while the entry exists, and (b) every node's
// duplicate guard refuses a creation task for a forward that already has an owner
// — so the re-published task would be dropped by everyone and the forward
// stranded on the node that gave it up.
func TestRevokeLeavesACarrierAndReleaseRemovesIt(t *testing.T) {
n1, _, _ := balanceRing(t, 3, 0)
// A stop marks the entry and keeps it.
n1.state.UpdateTopologyDisabled("web", "frps1", 100, true)
tk := &Task{Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
Link: store.Link{RemotePort: 100}}
if owner := n1.state.TopologyOwner(tk); owner != "n1" {
t.Fatalf("a stop must keep the entry as the flag's carrier, owner=%q", owner)
}
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 3 {
t.Fatalf("a stop must not change the owner's forward count, got %d", got)
}
// A release removes the entry (leaving the other two alone).
if !n1.RemoveTopologyEntry("web", "frps1", 100) {
t.Fatal("release must remove the entry")
}
if owner := n1.state.TopologyOwner(tk); owner != "" {
t.Errorf("after a release the forward must have no owner, got %q", owner)
}
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 2 {
t.Errorf("a release must reduce the owner's count, got %d", got)
}
if got := len(n1.state.TopologyList()); got != 2 {
t.Errorf("the other forwards must be untouched, got %d entries", got)
}
}
// TestReleaseTaskExecutesOnlyOnTheOwner: like a revoke, a release is
// owner-directed. Executing it elsewhere would be an attempt to move somebody
// else's load, which cannot work — the owner's own count is what must fall.
func TestReleaseTaskExecutesOnlyOnTheOwner(t *testing.T) {
n1, n2, _ := balanceRing(t, 3, 0)
released := map[string]int{}
for _, e := range []*Engine{n1, n2} {
h := e.Handler.(*fakeHandler)
h.release = func(ctx context.Context, tk *Task) error {
released[e.ID]++
return nil
}
}
tk := n1.ReleaseTask(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
store.Link{RemotePort: 100})
if tk == nil {
t.Fatal("ReleaseTask returned nil")
}
// Each Engine keeps its OWN state, so a task published on n1 is not visible
// to n2 until a token round carries it. Broadcast it the way the ring does.
var carry State
cb, _ := json.Marshal(n1.state)
_ = json.Unmarshal(cb, &carry)
n2.AdoptState(carry)
n2.state.PendingTasks[tk.ID] = tk
// n2 is the lightest node and does NOT own the forward: it must leave the
// task alone.
if err := n2.runCommands(context.Background(), &Token{}); err != nil {
t.Fatalf("n2 runCommands: %v", err)
}
if released["n2"] != 0 {
t.Errorf("n2 released %d forward(s); a release is owner-directed only", released["n2"])
}
if _, still := n2.state.PendingTasks[tk.ID]; !still {
t.Error("n2 consumed a release it does not own; it must ride the token to the owner")
}
// n1 owns it: it must release, drop the entry, and re-publish the forward.
if err := n1.runCommands(context.Background(), &Token{}); err != nil {
t.Fatalf("n1 runCommands: %v", err)
}
if released["n1"] != 1 {
t.Fatalf("the owner released %d times, want 1", released["n1"])
}
if owner := n1.state.TopologyOwner(&Task{
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
Link: store.Link{RemotePort: 100},
}); owner != "" {
t.Errorf("the released forward still has owner %q", owner)
}
// And it is back in the ring as an ordinary creation task.
var republished *Task
for _, p := range n1.state.PendingList() {
if !p.Release && !p.Revoke && !p.Restart &&
p.Link.RemotePort == 100 && p.Local.Name == "web" {
republished = p
}
}
if republished == nil {
t.Fatal("the released forward was not re-published as an unclaimed task")
}
if republished.Link.Disabled {
t.Error("the re-published forward must be ENABLED, not marked stopped")
}
}
// ---- dry-run / off ----
// TestDryRunReleasesNothing: dry-run is the default, and its whole value is that
// it can run on a live cluster without disturbing anything.
func TestDryRunReleasesNothing(t *testing.T) {
n1, _, _ := balanceRing(t, 3, 0)
b := NewBalancer(n1, 0)
b.SetMode(BalanceDryRun)
msg := b.Step(1000)
if !strings.Contains(msg, "would release") {
t.Fatalf("dry-run should report what it would do, got %q", msg)
}
if b.InFlight() {
t.Error("dry-run started a release")
}
for _, tk := range n1.state.PendingList() {
if tk.Release {
t.Error("dry-run published a release task")
}
}
if got := len(n1.state.TopologyList()); got != 3 {
t.Errorf("dry-run changed the topology: %d entries, want 3", got)
}
// A second identical tick must not repeat the same line (log hygiene).
if again := b.Step(1010); again != "" {
t.Errorf("dry-run repeated an unchanged decision: %q", again)
}
}
// TestModeOffDoesNothing reinforces that OFF means off, not "dry-run".
func TestModeOffDoesNothing(t *testing.T) {
n1, _, _ := balanceRing(t, 3, 0)
b := NewBalancer(n1, 0)
b.SetMode(BalanceOff)
if msg := b.Step(1000); msg != "" {
t.Errorf("mode off reported %q", msg)
}
if b.InFlight() {
t.Error("mode off started a release")
}
// A typo must not enable releasing.
b.SetMode("yes-please")
if b.Mode() != BalanceDryRun {
t.Errorf("an unknown mode must fall back to dry-run, got %q", b.Mode())
}
}
// ---- end-to-end through the ring machinery ----
// TestOwnerReleaseThenRingClaimsElsewhere is the whole design in one test: the
// owner gives the forward up, and the ordinary lowest-load claim then places it
// on an idle node — exactly what the self-directed rule depends on.
func TestOwnerReleaseThenRingClaimsElsewhere(t *testing.T) {
n1, n2, n3 := balanceRing(t, 3, 0)
// n2 and n3 must look idle to the claim path (they do: Forwards=0).
claimed := map[string]int{}
for _, e := range []*Engine{n1, n2, n3} {
h := e.Handler.(*fakeHandler)
h.claim = func(ctx context.Context, tk *Task) error {
claimed[e.ID]++
return nil
}
}
b := NewBalancer(n1, 0)
b.SetMode(BalanceOn)
// n1 is over its share → it releases.
msg := b.Step(1000)
if !strings.Contains(msg, "releasing") {
t.Fatalf("expected a release, got %q", msg)
}
if err := n1.runCommands(context.Background(), &Token{}); err != nil {
t.Fatalf("n1 runCommands: %v", err)
}
// Find the task the owner re-published for the forward it gave up.
var tk *Task
for _, p := range n1.state.PendingList() {
if !p.Release && p.Link.RemotePort == 100 && p.Local.Name == "web" {
tk = p
}
}
if tk == nil {
t.Fatal("the owner did not re-publish the forward it released")
}
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 2 {
t.Fatalf("n1 should now own 2, got %d", got)
}
// The forward is unclaimed; the lowest node claims it. n2 is the lightest
// (Forwards=0 and the same mem/net), so it should take it. The task reaches
// n2 on the token — each engine holds its own state, so carry it over the way
// a real round does.
var carried State
cb, _ := json.Marshal(n1.state)
_ = json.Unmarshal(cb, &carried)
n2.AdoptState(carried)
n2.state.PendingTasks[tk.ID] = tk
if err := n2.runCommands(context.Background(), &Token{}); err != nil {
t.Fatalf("n2 runCommands: %v", err)
}
if claimed["n2"] == 0 {
t.Fatal("the idle node did not claim the released forward")
}
// And n1 no longer owns it.
for _, te := range n1.state.ForwardsOwnedBy("n1") {
if te.Link.RemotePort == 100 {
t.Error("n1 still owns the forward it released")
}
}
// The balancer sees the release land and stops holding the release state.
if msg := b.Step(1100); !strings.Contains(msg, "gave up") {
t.Errorf("expected the release to be confirmed, got %q", msg)
}
if b.InFlight() {
t.Error("the release should be complete")
}
}
// TestCooldownBoundsTheChurn: a release costs a live connection, so one node may
// not shed again until the cooldown elapses — even if it is still over its share.
func TestCooldownBoundsTheChurn(t *testing.T) {
n1, _, _ := balanceRing(t, 4, 0)
b := NewBalancer(n1, 600) // 10 minutes
b.SetMode(BalanceOn)
if msg := b.Step(1000); !strings.Contains(msg, "releasing") {
t.Fatalf("first release expected, got %q", msg)
}
n1.RemoveTopologyEntry("web", "frps1", 100) // land it
if msg := b.Step(1010); !strings.Contains(msg, "gave up") {
t.Fatalf("expected confirmation, got %q", msg)
}
// Still over its share (3 of 4 over 3 → share 2), but inside the cooldown.
if pick, _, blocked := b.assess(1020); pick == nil || blocked != "cooldown" {
t.Errorf("expected the second release to be blocked by cooldown, got blocked=%q", blocked)
}
if b.InFlight() {
t.Error("cooldown must not leave a release in flight")
}
// Past the cooldown it may act again.
if pick, _, blocked := b.assess(1000 + 601); pick == nil || blocked != "" {
t.Errorf("past the cooldown the release should be allowed, got blocked=%q", blocked)
}
}
// TestReleaseDeadlineClearsAStuckConfirmation: if the release never lands (leader
// change, ring join), the node must not wedge forever.
func TestReleaseDeadlineClearsAStuckConfirmation(t *testing.T) {
n1, _, _ := balanceRing(t, 3, 0)
b := NewBalancer(n1, 0)
b.SetMode(BalanceOn)
_ = b.Step(1000) // release published, entry NOT removed (nothing executed it)
if !b.InFlight() {
t.Fatal("precondition: a release should be in flight")
}
if msg := b.Step(1000 + releaseDeadlineSec + 1); msg != "" {
t.Errorf("a timed-out release should clear silently, got %q", msg)
}
if b.InFlight() {
t.Error("the release state should have been cleared after the deadline")
}
}
// ---- read-only report ----
// TestReportDoesNotMutate — polling the endpoint must not start, delay or cancel
// a release. A read-only surface that perturbs what it reports is worse than none.
func TestReportDoesNotMutate(t *testing.T) {
n1, _, _ := balanceRing(t, 3, 0)
b := NewBalancer(n1, 600)
b.SetMode(BalanceOn)
for i := 0; i < 5; i++ {
rep := b.Report()
if rep.SelfForwards != 3 {
t.Fatalf("report changed the node's forward count: %d", rep.SelfForwards)
}
if rep.Mode != BalanceOn {
t.Fatalf("report changed the mode: %q", rep.Mode)
}
}
if b.InFlight() || b.lastReleaseAt != 0 {
t.Error("polling the report started a release")
}
if rep := b.Report(); rep.WouldRelease == nil {
t.Error("the report should still surface the pending decision")
}
}
// TestReportExplainsWhyNothingHappens pins the field that answers the question
// the log cannot: an over-share node that is holding back must say why.
func TestReportExplainsWhyNothingHappens(t *testing.T) {
n1, _, _ := balanceRing(t, 4, 0)
b := NewBalancer(n1, 600)
b.SetMode(BalanceOn)
rep := b.Report()
if !rep.SelfOverShare {
t.Fatalf("4 of 4 forwards over 3 nodes should be over the share (share=%d)", rep.Share)
}
if rep.WouldRelease == nil {
t.Fatal("an over-share node must surface the forward it would give up")
}
if rep.BlockedBy != "" {
t.Errorf("nothing should block the first release, got %q", rep.BlockedBy)
}
// Inside the cooldown the same node reports the block. Report() reads the
// real clock, so set lastReleaseAt relative to now rather than to a fake one.
b.lastReleaseAt = time.Now().Unix() - 10
n1.RemoveTopologyEntry("web", "frps1", 100)
rep = b.Report()
if rep.BlockedBy != "cooldown" {
t.Errorf("after a release the node should report a cooldown block, got %q", rep.BlockedBy)
}
}
// TestReportOnBalancedNodeIsQuiet: the healthy case must be a real answer, not an
// empty one — otherwise "balanced" and "balancing is switched off" look identical.
func TestReportOnBalancedNodeIsQuiet(t *testing.T) {
n1, _, _ := balanceRing(t, 1, 1)
b := NewBalancer(n1, 0)
b.SetMode(BalanceOn)
rep := b.Report()
if rep.WouldRelease != nil {
t.Errorf("a balanced node reported a release: %+v", rep.WouldRelease)
}
if rep.SelfOverShare {
t.Error("1 of 3 over 3 nodes is not over the share")
}
if rep.TotalForwards != 3 || rep.AliveNodes != 3 || rep.Share != 1 {
t.Errorf("report pool = total %d alive %d share %d, want 3/3/1",
rep.TotalForwards, rep.AliveNodes, rep.Share)
}
}