fix(cluster): startup reconcile 不该等 leader,否则非 leader 永远不重建自己的 worker

前两个提交修好分裂与冻结后,滚动重启实测又暴露第三个缺口:

    .106 重入成功 → adopt-state(topology 里 .106 是 gitea_ssh 的 owner)
                  → 但 .106 没有任何 frpc worker,且日志里从无 reconcile

拓扑说 .106 拥有该转发,进程却不存在 —— 拓扑与实际不一致。

## 根因:代码与自己注释矛盾

启动 reconcile 的注释原文就是「every node (leader or not) checks its OWN topology
entries and re-claims any whose per-forward frpc worker is absent」,但代码第一行是

    for { if ring.IsLeader() { break }; sleep 2s }

即必须先当上 leader 才执行。reconcile 做的是**本节点**的活(重建自己拥有的
worker),而三分之一个环里每台都拥有转发 —— 于是所有非 leader 节点永远等不到,
worker 永远不重建。节点可以「已回到集群、拓扑里写着它 own 某转发、却没有任何进程」
这种自相矛盾的状态。

## 修法

新增 Engine.HasSelfInTopology():当前拓扑是否指名本节点为某条 active 转发
的 owner。reconcile 改等这个条件(30s 上限后无论如何继续,保证孤立节点也能推进),
不再等 leadership。领导者与 reconcile 本就无关:follower 也 own 转发。

## 测试

TestHasSelfInTopology:用**两节点**环,转发由非 leader 的 n2 持有,断言非 leader
仍然报告 true。

★ 这条测试第一版是**假绿**:单节点环里唯一的成员必然是 leader,两种实现结果相同,
加回 IsLeader 门控后测试仍全绿(已实测)。改成两节点、明确断言
`precondition: n2 must not be the leader` 之后,加回门控如期变红:

    a non-leader that owns a forward reports false — the startup reconcile would
    skip it and leave its worker down

go build / go vet / go test ./... 全绿,gofmt 干净。
This commit is contained in:
JianFeeeee
2026-10-05 17:54:29 +08:00
parent bfd7265405
commit fc4cbaa726
3 changed files with 76 additions and 6 deletions

View File

@ -561,17 +561,32 @@ func main() {
// LocalOnly forwards are never in the topology and are handled by the
// auto-start loop above.
go func() {
for {
if ring.IsLeader() {
break
}
// No leader gate here. The reconcile body is per-NODE work: it re-claims
// the forwards THIS node owns, whose worker processes died with the
// restart. Every member owns forwards (a 3-node ring splits them), so
// gating on leadership meant non-leaders waited forever and never
// rebuilt their own workers — a node could come back up "in the cluster"
// while owning a forward with no process behind it. Observed live: after a
// rolling restart a non-leader rejoined, adopted state listing itself as
// the owner of a forward, and never started it, because it sat waiting to
// become leader.
//
// What actually needs settling is the ring, not the leadership: wait for
// the topology to show us (with a bounded fallback so a lone node that
// rejoined a ring whose leader is unreachable still makes progress).
deadline := time.After(30 * time.Second)
for !ring.HasSelfInTopology() {
select {
case <-time.After(2 * time.Second):
case <-deadline:
log.Printf("ring[%s] startup reconcile: self not in topology after 30s, proceeding anyway", ring.ID)
case <-time.After(500 * time.Millisecond):
continue
case <-clusterCtx.Done():
return
}
break
}
// Let the ring stabilize after leader election / rejoin.
// Let the ring stabilize after election / rejoin before acting on it.
time.Sleep(3 * time.Second)
snap := ring.Snapshot()
reclaimed := 0