mirror of
https://gitcode.com/JianFeeeee/webui4frpc.git
synced 2026-10-15 12:14:37 +00:00
feat(cluster): 负载均衡迁移 —— 定向投递 + 两阶段提交
承接 875e580（只落决策逻辑）。本提交补齐运行时：周期检查、迟滞判定、
两阶段迁移，以及让迁移任务**定向**到指定节点。
## 为什么必须是定向的（核心安全点）
loadSnapshot() 的 Forwards 来自 ForwardsOwnedBy，所以旧 owner 一旦被摘掉条目，
它立刻变成最闲节点。而普通创建任务是"任何最闲者认领"—— 于是它会把刚迁走的
转发**当场抢回**，迁移变成"只重启一次 worker"的空操作。
故 Task 新增 Migrate + Target 字段，runCommands 的选择循环照 Restart 的范式
处理：Target == 自己才认领，否则 `continue` 让任务随环前进。
## 为什么必须两阶段（不能一次发 revoke + submit）
1. 旧 owner 变最闲后会把未定向的任务抢回（上面那条）；
2. PendingList 按 `t<ID>` **字符串**排序（`t10 < t5`），同轮内两个任务的先后
不可控。
Balancer 因此是显式状态机：phase1 发布 revoke 并等待旧条目真正消失，确认后才
发布 phase2 的定向 migrate。两个超时兜底：revoke 超时则放弃；migrate 超时
（target 没认领）则退回**无定向** SubmitTask，让"任何最闲者"接手 —— 转发不会
被丢，只是延迟。这与环本身的行为一致（实测：只要还有更闲的节点，pending 任务
最终会被认领；只有"全员都更忙"时才滞留）。
## 两个被测试逼出来的设计修正
- **绝对下限 minForwards=2**：只有"超均值倍数"在小集群失真——3 节点 1 条转发时
owner Score≈105 而两台空闲≈5，20 倍于均值会触发迁移，但那是完全正常的分布，
迁它只会断一条活跃连接。倍数负责看形状，下限负责"有没有东西值得搬"。
- **Migrate 也要绕过 duplicate-claim 守卫**，否则旧条目还在时目标节点会拒绝
认领（这条守卫正是为了防止同一 remotePort 注册两次 → frps 报
"proxy already exists"）。
## 测试（本项目第六次假绿的教训）
新增 6 条，全绿。做双向验证时发现
TestMigrateTaskIsAddressedNotClaimedByLightest **删掉定向分支后仍然 PASS**，
查因：`loadSnapshot()每次 OnToken 都用 Handler.RuntimeLoad() 刷新 mem/net，
我手工设的 State.Nodes[].Load 被静默覆盖 ⇒ n2/n3 始终平手 ⇒ "最闲者"恰好
就是 target，定向性从未被验证。
改为**用三个 Handler 的不同 RuntimeLoad 制造真实差异**（n3 最闲、n2 是较重的
target）后，删掉定向分支如期变红：
n3 (the LIGHTEST node) claimed 1 migrate task(s); the task is addressed
to n2 and must ignore load entirely
⇒ 这条纪律再次生效：测试必须制造出能让错误实现暴露的**不对称**，否则"恰好通过"
比"失败"更危险。
go build / go vet / go test ./... 全绿，gofmt 干净。
This commit is contained in:
JianFeeeee
@ -1,5 +1,12 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"log"
|
||||
"strconv"
|
||||
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// fwdKey identifies a forward for balancing purposes.
|
||||
type fwdKey struct {
|
||||
local, remote string
|
||||
@ -20,11 +27,23 @@ type fwdKey struct {
|
||||
// BalanceTracker) can pass its bookkeeping through without changing this
|
||||
// signature, but this function itself is stateless about time: it answers
|
||||
// "given the current snapshot, what (if anything) should move right now".
|
||||
// minForwards is an absolute floor: never migrate unless the overloaded node
|
||||
// holds at least this many forwards.
|
||||
//
|
||||
// The ratio test alone is misleading on small rings. With 3 nodes and a single
|
||||
// forward, the owner scores ~105 (Forwards*100) and the two idle nodes ~5, so the
|
||||
// owner sits at 20x the mean and trips any ratio — yet one forward on one node is
|
||||
// the perfectly correct distribution, there is nothing to gain by moving it, and
|
||||
// the move would only drop a live connection. The ratio detects *shape*; this
|
||||
// floor says there must be something worth moving.
|
||||
const minForwards = 2
|
||||
|
||||
func planBalance(s *State, minOverRatio float64) (*TopoEntry, string, bool) {
|
||||
// Candidate destinations: alive, non-leader-free (any), scored by Load.
|
||||
type cand struct {
|
||||
id string
|
||||
score float64
|
||||
id string
|
||||
score float64
|
||||
forwards int
|
||||
}
|
||||
var dests []cand
|
||||
total := 0
|
||||
@ -33,7 +52,7 @@ func planBalance(s *State, minOverRatio float64) (*TopoEntry, string, bool) {
|
||||
if !n.Alive {
|
||||
continue
|
||||
}
|
||||
dests = append(dests, cand{n.ID, n.Load.Score()})
|
||||
dests = append(dests, cand{n.ID, n.Load.Score(), n.Load.Forwards})
|
||||
total += n.Load.Forwards
|
||||
}
|
||||
if len(dests) < 2 {
|
||||
@ -42,15 +61,17 @@ func planBalance(s *State, minOverRatio float64) (*TopoEntry, string, bool) {
|
||||
avg := float64(total) / float64(len(dests))
|
||||
threshold := avg * minOverRatio
|
||||
|
||||
// Overloaded owners, worst first.
|
||||
// Overloaded owners, worst first. The absolute floor is applied here so a node
|
||||
// holding one forward in a three-node ring is never considered hot.
|
||||
type over struct {
|
||||
id string
|
||||
score float64
|
||||
id string
|
||||
score float64
|
||||
forwards int
|
||||
}
|
||||
var overs []over
|
||||
for _, d := range dests {
|
||||
if d.score > threshold {
|
||||
overs = append(overs, over{d.id, d.score})
|
||||
if d.score > threshold && d.forwards >= minForwards {
|
||||
overs = append(overs, over{d.id, d.score, d.forwards})
|
||||
}
|
||||
}
|
||||
if len(overs) == 0 {
|
||||
@ -99,3 +120,176 @@ func planBalance(s *State, minOverRatio float64) (*TopoEntry, string, bool) {
|
||||
}
|
||||
return pick, dest.id, true
|
||||
}
|
||||
|
||||
// Balancer moves forwards off an overloaded node, one at a time.
|
||||
//
|
||||
// It is a TWO-PHASE state machine, and the split is a safety requirement rather
|
||||
// than a stylistic choice:
|
||||
//
|
||||
// phase 1 (revoking): ask the current owner to stop. Wait until the topology
|
||||
// entry is actually gone before publishing anything.
|
||||
// phase 2 (publishing): hand the forward to a NAMED destination.
|
||||
//
|
||||
// Publishing both at once does not work, for two independent reasons:
|
||||
//
|
||||
// 1. loadSnapshot() derives Forwards from ForwardsOwnedBy, so the moment the old
|
||||
// entry is dropped the old owner becomes the lightest node. A plain creation
|
||||
// task is claimed by "whoever is lightest" — so the node we just took the
|
||||
// forward FROM would immediately claim it back and the migration would be a
|
||||
// no-op that merely restarts the worker.
|
||||
// 2. PendingList sorts by task id as a STRING (t10 < t5), so the revoke and the
|
||||
// submit could execute in either order within one pass.
|
||||
//
|
||||
// Migrate tasks solve (1) by carrying an explicit Target; this type solves (2) by
|
||||
// refusing to publish phase 2 before phase 1 has completed.
|
||||
type Balancer struct {
|
||||
engine *Engine
|
||||
|
||||
// ratio is how far above the average a node's Score must be, relative to the
|
||||
// lightest node, before anything moves.
|
||||
ratio float64
|
||||
// sustainedSec is how long an imbalance must persist before we act. This is
|
||||
// what stops a transient CPU/network spike from restarting workers.
|
||||
sustainedSec int64
|
||||
|
||||
// overSince records when the currently-overloaded node first crossed the
|
||||
// threshold. Zero means "not currently over".
|
||||
overSince map[string]int64
|
||||
// pending migration awaiting phase 1 (the forward to revoke).
|
||||
revoking *pendingMigration
|
||||
// migration awaiting phase 2 (the forward to publish, and where to).
|
||||
publishing *pendingMigration
|
||||
}
|
||||
|
||||
type pendingMigration struct {
|
||||
local string
|
||||
remote string
|
||||
port int
|
||||
fromOwner string
|
||||
toOwner string
|
||||
deadlineAt int64 // give up (re-publish for anyone) after this unix second
|
||||
}
|
||||
|
||||
// NewBalancer builds a balancer. ratio <= 1 or sustainedSec < 0 are clamped to
|
||||
// sane minimums so a bad configuration cannot turn into a migration storm.
|
||||
func NewBalancer(e *Engine, ratio float64, sustainedSec int64) *Balancer {
|
||||
if ratio <= 1 {
|
||||
ratio = 2
|
||||
}
|
||||
if sustainedSec < 0 {
|
||||
sustainedSec = 0
|
||||
}
|
||||
return &Balancer{
|
||||
engine: e, ratio: ratio, sustainedSec: sustainedSec,
|
||||
overSince: map[string]int64{},
|
||||
}
|
||||
}
|
||||
|
||||
// InFlight reports whether a migration is currently between phases.
|
||||
func (b *Balancer) InFlight() bool {
|
||||
return b.revoking != nil || b.publishing != nil
|
||||
}
|
||||
|
||||
// Step advances the state machine by one cycle. now is the current unix second.
|
||||
// It returns a short human-readable description when something happened.
|
||||
func (b *Balancer) Step(now int64) string {
|
||||
// --- phase 2: did phase 1 finish? ---
|
||||
if b.revoking != nil {
|
||||
if b.engine.state.TopologyOwner(&Task{
|
||||
Local: store.Local{Name: b.revoking.local},
|
||||
Remote: store.Remote{Name: b.revoking.remote},
|
||||
Link: store.Link{RemotePort: b.revoking.port},
|
||||
}) != "" {
|
||||
// Still owned: the revoke has not completed yet. Wait, unless it has
|
||||
// been far too long, in which case something is wrong with the ring.
|
||||
if now < b.revoking.deadlineAt {
|
||||
return ""
|
||||
}
|
||||
log.Printf("ring[%s] migrate revoke for %s→%s:%d timed out; abandoning",
|
||||
b.engine.ID, b.revoking.local, b.revoking.remote, b.revoking.port)
|
||||
b.revoking = nil
|
||||
return ""
|
||||
}
|
||||
// Entry gone — the old worker is stopped. Publish for the named target.
|
||||
m := b.revoking
|
||||
b.revoking = nil
|
||||
b.publishing = m
|
||||
b.publishMigration()
|
||||
return "revoke complete, migrating " + m.local + ":" + strconv.Itoa(m.port) + " → " + m.toOwner
|
||||
}
|
||||
|
||||
// --- phase 1: wait for an in-flight publish, or start a new migration ---
|
||||
if b.publishing != nil {
|
||||
// The migrate task carries itself in the ring; nothing to do but let it
|
||||
// ride. Clear once the forward is back in the topology.
|
||||
if b.engine.state.TopologyOwner(&Task{
|
||||
Local: store.Local{Name: b.publishing.local},
|
||||
Remote: store.Remote{Name: b.publishing.remote},
|
||||
Link: store.Link{RemotePort: b.publishing.port},
|
||||
}) == b.publishing.toOwner {
|
||||
done := b.publishing
|
||||
b.publishing = nil
|
||||
return "migrated " + done.local + ":" + strconv.Itoa(done.port) + " to " + done.toOwner
|
||||
}
|
||||
if now >= b.publishing.deadlineAt {
|
||||
// Nobody claimed it. Submit an UNTARGETED task so whichever node is
|
||||
// lightest picks it up — the ring keeps the forward alive rather than
|
||||
// stranding it, which is the same guarantee the rest of the protocol
|
||||
// relies on.
|
||||
m := b.publishing
|
||||
b.publishing = nil
|
||||
b.engine.SubmitTask(store.Local{Name: m.local}, store.Remote{Name: m.remote},
|
||||
store.Link{RemotePort: m.port})
|
||||
return "migrate target did not claim " + m.local + ":" + strconv.Itoa(m.port) + "; re-opened for any node"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// --- nothing in flight: look for work ---
|
||||
pick, dest, hot := b.engine.findBalanceCandidate(b.ratio)
|
||||
if hot != "" {
|
||||
if _, seen := b.overSince[hot]; !seen {
|
||||
b.overSince[hot] = now
|
||||
}
|
||||
} else {
|
||||
b.overSince = map[string]int64{}
|
||||
}
|
||||
if pick == nil || dest == "" {
|
||||
return ""
|
||||
}
|
||||
if since, ok := b.overSince[pick.OwnerID]; !ok || now-since < int64(b.sustainedSec) {
|
||||
return ""
|
||||
}
|
||||
// Phase 1: publish the revoke and remember what we are waiting for.
|
||||
m := &pendingMigration{
|
||||
local: pick.Local.Name, remote: pick.Remote.Name, port: pick.Link.RemotePort,
|
||||
fromOwner: pick.OwnerID, toOwner: dest,
|
||||
// Generous deadline: a token round is ~2s, and the owner may be busy.
|
||||
deadlineAt: now + 120,
|
||||
}
|
||||
b.revoking = m
|
||||
b.engine.RevokeTask(pick.Local, pick.Remote, pick.Link)
|
||||
// The revoke task must be able to reach the owner even though the balancer
|
||||
// runs on the leader: it travels in the ring like any other command.
|
||||
return "revoking " + m.local + ":" + strconv.Itoa(m.port) + " from " + m.fromOwner
|
||||
}
|
||||
|
||||
func (b *Balancer) publishMigration() {
|
||||
m := b.publishing
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
b.engine.submitMigrate(store.Local{Name: m.local}, store.Remote{Name: m.remote},
|
||||
store.Link{RemotePort: m.port}, m.toOwner)
|
||||
}
|
||||
|
||||
// findBalanceCandidate exposes planBalance to the engine's live state.
|
||||
func (e *Engine) findBalanceCandidate(ratio float64) (*TopoEntry, string, string) {
|
||||
pick, dest, ok := planBalance(&e.state, ratio)
|
||||
if !ok {
|
||||
return nil, "", ""
|
||||
}
|
||||
// Report the overloaded node even when there is nothing movable, so the
|
||||
// hysteresis timer can start for a node whose forwards are all stopped.
|
||||
return pick, dest, pick.OwnerID
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user