mirror of
https://gitcode.com/JianFeeeee/webui4frpc.git
synced 2026-10-15 12:14:37 +00:00
feat(cluster): 负载均衡改为每节点自主「先撤销本地转发，再丢回环」
按用户要求重做：不再由 leader 单独调度，而是**每个节点对自己的转发跑同一条
规则**，超了就自己撤销一条、把任务丢回环上，由环原有的「最闲者认领」决定它
落到哪台。
## 为什么必须是 owner 自己动手（不是洁癖，是实测逼出来的）
真机探针（跑完已删）显示，真实 revoke 之后：
TopologyOwner after revoke = "n1" ← 条目还在（只标记不删除）
n1 Forwards=3 score=310.0 ← 负载根本没降
after step: revoking=true publishing=false
⇒ 一节点的负载**只有它自己的拓扑条目消失时才会下降**，而条目是本地簿记，没有任何
对端可以代它写。leader 从外面只能看见失衡、无法让失衡成为事实——它必须伸手去改
另一个节点的簿记，而这正是这个环到处都在避免的跨节点写（revoke / restart 定向往
owner 也是同一个理由）。
⇒ 而且旧实现**根本跑不起来**：phase 1 判据是「条目消失」，可 mark-don't-remove
让它永远不消失，于是每 120s 打一条 `revoking … timed out; abandoning`，
**phase 2 永不执行，一条转发都不会迁**。开 `W4F_BALANCE_MODE=on` 只会刷日志。
## 规则（替代原来的 ratio 阈值）
当 Forwards >= 2 且 Forwards > ceil(总数/存活节点数) 时，撤销一条
- **自限**：3 条 / 3 台 → owner 2 > 1，卸一条 → 1，`1 > 1` 不成 → 停。
没有上界的话 owner 会把转发**全部卸光**。
- **不会卸空**：低于下限（2）永不撤销。
- 平摊值向上取整（3/3=1，不是 0），否则 1 条 / 3 台的节点也算「超了」。
- **不再用 mem/net 参与判据**：实测 `.30` 空闲时 netPct 能到 55%，Score 会高于
有转发的节点——用 Score 会把纯网络抖动当成负载。
- 节奏用**每节点冷却**（默认 600s）而不是「失衡持续多久」窗口：规则本身已经要求
真实失衡，没有需要平滑的瞬态，只有需要限制的抖动频率。
## 为什么必须是新的 Release 任务类型，不能复用 Revoke
用户停止（Revoke）**保留拓扑条目**作为 disabled 标志的载体；均衡释放必须**删除
条目**，因为 (a) 负载由自己的条目数算出，条目不删负载不降；(b) 每个节点的
duplicate-guard 会拒绝「已有 owner 的转发」的创建任务——保留条目会让**所有节点
都拒领**，转发被永久搁死在刚放弃它的那台机器上。
Release 因此：`Handler.Release`（停 worker，但**不标记 stopped**）→ 删条目 →
`SubmitTask` 以**启用状态**重新发布。它需要自己的 Handler 方法，不能回落到
Revoke：Revoke 会顺手把 link 标成 stopped，而这条转发即将在别的节点上跑。
## 顺带修掉两个既有缺口（实测确认）
1. **`submitMigrate` 发布不出去**：`OnToken` 的 localPending 重合并只保留「从未
publish 过」的本地 pending，而只有 `Add*` 系列会记 `published`。引擎里直接写
`PendingTasks` 的任务会在下一次 adopt 时被静默丢弃。
⇒ 新 `ReleaseTask` 走 `AddPending`。
（`SubmitRestart` 有同样的毛病，但它不在本次范围，未改动——见提交说明末尾。）
2. `Release` 需要绕过 duplicate-claim 守卫（同 Restart），否则「owner 本来就持有
这条转发」这一前提会让每条 Release 都被丢掉。
## 测试（17 条，全绿）
规则：平摊值取整、超了才撤、撤到平摊就停、单节点不撤、低于下限不撤、不撤已停
转发、候选选择确定。
机制：Revoke 留载体 vs Release 删条目（这是两种语义的核心差异）、Release 只由
owner 执行且不消费他人的任务、owner 释放后环的普通认领把它接走。
模式：dry-run 不动任何东西、off 真的什么都不做、未知模式回落 dry-run、
冷却限制抖动频率、确认超时清理。
只读：Report 不变更任何状态、能解释「超了但不动」的原因、均衡节点给出真实答案。
**双向验证 3 条**（改坏→变红→还原→变绿）：
- 去掉 Release 的 duplicate-guard 例外 → `the owner released 0 times, want 1`
- 去掉「不超过平摊值」上界 → `2 owned against share 2 must not release`
- Release 改成只标记不删除 → `n1 should now own 2, got 3`
go build / go vet / go test ./... 全绿（go clean -testcache 后逐包实跑）。
## 未做 / 待办
- `SubmitRestart` 的同类发布缺口未修（同样会被 adopt 丢弃），需要单独确认它
在实机上是否真的丢过 restart。
- UI 尚未跟进新报告结构（下一步）。
This commit is contained in:
JianFeeeee
@ -3,321 +3,316 @@ package cluster
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"sort"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// fwdKey identifies a forward for balancing purposes.
|
||||
type fwdKey struct {
|
||||
local, remote string
|
||||
port int
|
||||
// The rebalance rule, deliberately simple and self-limiting:
|
||||
//
|
||||
// release one forward when Forwards >= minForwardsToShare AND Forwards > share
|
||||
//
|
||||
// where share = ceil(totalForwards / aliveNodes) — the even split.
|
||||
//
|
||||
// Every node runs this check about its OWN forwards. That is the whole point:
|
||||
// a node's load only actually falls when its own topology entry goes away, so
|
||||
// only the owner can produce the state change that makes the ring hand the
|
||||
// forward to somebody else. A coordinator on another machine can see the
|
||||
// imbalance but cannot make it true; it would have to reach in and mutate the
|
||||
// owner's bookkeeping, which is exactly the sort of cross-node write this ring
|
||||
// avoids everywhere else (revoke and restart are owner-directed for the same
|
||||
// reason).
|
||||
//
|
||||
// The rule is self-limiting in both directions, so it needs no upper bound on
|
||||
// how many forwards a node may shed:
|
||||
//
|
||||
// - It stops at the average. 3 forwards over 3 nodes: the owner holds 2 > 1,
|
||||
// releases one, now holds 1 which is not > 1, and stops. Without the "> share"
|
||||
// bound an owner would shed EVERY forward it has.
|
||||
// - It cannot empty a node. A node holding fewer than minForwardsToShare never
|
||||
// releases, so there is always something left to serve.
|
||||
//
|
||||
// A release costs a live connection: stopping the worker drops whatever was
|
||||
// flowing through the forward, and the new owner re-dials the local service.
|
||||
// Because the rule above already requires a real imbalance, pacing is handled by
|
||||
// a per-node cooldown (cooldownSec) rather than a "how long has this lasted"
|
||||
// window — there is no transient to smooth over, only a churn rate to bound.
|
||||
const (
|
||||
// minForwardsToShare: never release below this many owned forwards.
|
||||
minForwardsToShare = 2
|
||||
// balanceCooldownSec: minimum gap between two releases by one node. One
|
||||
// release per 10 minutes per node is a trickle; it is what stops a cluster
|
||||
// whose loads oscillate around the split point from restarting workers in a
|
||||
// loop.
|
||||
balanceCooldownSec = 600
|
||||
// releaseDeadlineSec: how long to wait for a published release to be
|
||||
// executed by its owner (us) before giving up on confirming it. A token
|
||||
// round is ~2s, so this is generous; it exists so a release lost to a
|
||||
// leader change or a joined ring cannot wedge the node forever.
|
||||
releaseDeadlineSec = 120
|
||||
)
|
||||
|
||||
// Balance modes.
|
||||
const (
|
||||
BalanceOff = "off"
|
||||
BalanceDryRun = "dry-run"
|
||||
BalanceOn = "on"
|
||||
)
|
||||
|
||||
// pool is the replicated view the rule works from.
|
||||
type pool struct {
|
||||
alive int // nodes that can take a forward
|
||||
mine int // forwards THIS node owns (live count, not the replicated one)
|
||||
total int // forwards across the ring
|
||||
share int // ceil(total/alive) — the even split
|
||||
}
|
||||
|
||||
// planBalance decides whether one forward should move off an overloaded owner.
|
||||
//
|
||||
// The rule is deliberately conservative:
|
||||
// - never touch a stopped forward (disabled) — moving it would resurrect work
|
||||
// the user explicitly turned off;
|
||||
// - never touch a forward whose owner is not actually above the threshold;
|
||||
// - never move more than one per pass, so a transient load spike cannot cause a
|
||||
// burst of worker restarts (each migration drops live connections).
|
||||
//
|
||||
// It returns the chosen forward and the node that should take it.
|
||||
// now / sustainedSec are accepted so the caller-side hysteresis (which lives in
|
||||
// BalanceTracker) can pass its bookkeeping through without changing this
|
||||
// signature, but this function itself is stateless about time: it answers
|
||||
// "given the current snapshot, what (if anything) should move right now".
|
||||
// minForwards is an absolute floor: never migrate unless the overloaded node
|
||||
// holds at least this many forwards.
|
||||
//
|
||||
// The ratio test alone is misleading on small rings. With 3 nodes and a single
|
||||
// forward, the owner scores ~105 (Forwards*100) and the two idle nodes ~5, so the
|
||||
// owner sits at 20x the mean and trips any ratio — yet one forward on one node is
|
||||
// the perfectly correct distribution, there is nothing to gain by moving it, and
|
||||
// the move would only drop a live connection. The ratio detects *shape*; this
|
||||
// floor says there must be something worth moving.
|
||||
const minForwards = 2
|
||||
// releaseState is one release this node has published and is waiting to see
|
||||
// leave its own topology.
|
||||
type releaseState struct {
|
||||
local, remote string
|
||||
port int
|
||||
startedAt int64
|
||||
}
|
||||
|
||||
func planBalance(s *State, minOverRatio float64) (*TopoEntry, string, bool) {
|
||||
// Candidate destinations: alive, non-leader-free (any), scored by Load.
|
||||
type cand struct {
|
||||
id string
|
||||
score float64
|
||||
forwards int
|
||||
// Balancer lets each node shed its own share of the cluster's forwards.
|
||||
//
|
||||
// It publishes only ONE kind of thing: a release of a forward it owns. There is
|
||||
// no targeted hand-off and no coordinator, because "drop it and let the ring's
|
||||
// normal lowest-load claim pick it up" is already the ring's only mechanism for
|
||||
// moving a forward — and it is the mechanism that, unlike a coordinated move,
|
||||
// cannot be undone by the node that gave the forward up (the claim path runs
|
||||
// after the node's load has already fallen).
|
||||
type Balancer struct {
|
||||
engine *Engine
|
||||
mode string
|
||||
|
||||
// cooldownSec is the minimum gap between two releases by this node.
|
||||
cooldownSec int64
|
||||
// releasing is the forward this node gave up and is waiting to see leave.
|
||||
// While it is set, no second release is started: one at a time keeps the
|
||||
// connection churn bounded and the log readable.
|
||||
releasing *releaseState
|
||||
// lastReleaseAt is when the previous release was published (0 = never).
|
||||
lastReleaseAt int64
|
||||
// reported remembers the last dry-run candidate so the observe-only mode
|
||||
// logs a change of mind rather than the same line every tick.
|
||||
reported string
|
||||
}
|
||||
|
||||
// NewBalancer builds a balancer. It starts in observe-only mode: releasing a
|
||||
// forward drops a live connection, so acting on the ring is opt-in.
|
||||
func NewBalancer(e *Engine, cooldownSec int64) *Balancer {
|
||||
b := &Balancer{engine: e, mode: BalanceDryRun, cooldownSec: cooldownSec}
|
||||
if b.cooldownSec <= 0 {
|
||||
b.cooldownSec = balanceCooldownSec
|
||||
}
|
||||
var dests []cand
|
||||
total := 0
|
||||
for i := range s.Nodes {
|
||||
n := &s.Nodes[i]
|
||||
return b
|
||||
}
|
||||
|
||||
// SetMode selects off / dry-run / on. An unrecognised value is treated as
|
||||
// dry-run — a typo must not be able to turn releasing on.
|
||||
func (b *Balancer) SetMode(m string) {
|
||||
switch m {
|
||||
case BalanceOff, BalanceDryRun, BalanceOn:
|
||||
b.mode = m
|
||||
default:
|
||||
if m != "" {
|
||||
log.Printf("ring[%s] unknown balance mode %q; using %s", b.engine.ID, m, BalanceDryRun)
|
||||
}
|
||||
b.mode = BalanceDryRun
|
||||
}
|
||||
}
|
||||
|
||||
// Mode returns the active mode.
|
||||
func (b *Balancer) Mode() string { return b.mode }
|
||||
|
||||
// SetCooldown overrides the per-node release cooldown.
|
||||
func (b *Balancer) SetCooldown(sec int64) {
|
||||
if sec > 0 {
|
||||
b.cooldownSec = sec
|
||||
}
|
||||
}
|
||||
|
||||
// InFlight reports whether a release is awaiting confirmation.
|
||||
func (b *Balancer) InFlight() bool { return b.releasing != nil }
|
||||
|
||||
// pool computes the ring-wide view this node derives for itself.
|
||||
//
|
||||
// Our own count is taken LIVE from the topology rather than from the replicated
|
||||
// Nodes[].Load, because Nodes[].Load is only refreshed when a token arrives and
|
||||
// the decision must see the effect of the release immediately — otherwise a node
|
||||
// would keep believing it is over its share and shed another forward each round.
|
||||
func (b *Balancer) pool() pool {
|
||||
st := &b.engine.state
|
||||
p := pool{mine: len(st.ForwardsOwnedBy(b.engine.ID))}
|
||||
for i := range st.Nodes {
|
||||
n := &st.Nodes[i]
|
||||
if !n.Alive {
|
||||
continue
|
||||
}
|
||||
dests = append(dests, cand{n.ID, n.Load.Score(), n.Load.Forwards})
|
||||
total += n.Load.Forwards
|
||||
}
|
||||
if len(dests) < 2 {
|
||||
return nil, "", false // nothing to balance between
|
||||
}
|
||||
avg := float64(total) / float64(len(dests))
|
||||
threshold := avg * minOverRatio
|
||||
|
||||
// Overloaded owners, worst first. The absolute floor is applied here so a node
|
||||
// holding one forward in a three-node ring is never considered hot.
|
||||
type over struct {
|
||||
id string
|
||||
score float64
|
||||
forwards int
|
||||
}
|
||||
var overs []over
|
||||
for _, d := range dests {
|
||||
if d.score > threshold && d.forwards >= minForwards {
|
||||
overs = append(overs, over{d.id, d.score, d.forwards})
|
||||
}
|
||||
}
|
||||
if len(overs) == 0 {
|
||||
return nil, "", false
|
||||
}
|
||||
// pick the worst (stable: compare score then id so ties are deterministic)
|
||||
worstIdx := 0
|
||||
for i := 1; i < len(overs); i++ {
|
||||
if overs[i].score > overs[worstIdx].score ||
|
||||
(overs[i].score == overs[worstIdx].score && overs[i].id < overs[worstIdx].id) {
|
||||
worstIdx = i
|
||||
}
|
||||
}
|
||||
worst := overs[worstIdx]
|
||||
|
||||
// Lightest node overall is the destination, provided it is meaningfully
|
||||
// lighter than the worst owner.
|
||||
lightestIdx := 0
|
||||
for i := 1; i < len(dests); i++ {
|
||||
if dests[i].score < dests[lightestIdx].score ||
|
||||
(dests[i].score == dests[lightestIdx].score && dests[i].id < dests[lightestIdx].id) {
|
||||
lightestIdx = i
|
||||
}
|
||||
}
|
||||
dest := dests[lightestIdx]
|
||||
if dest.id == worst.id {
|
||||
return nil, "", false // the worst owner IS the lightest — nothing to do
|
||||
}
|
||||
// Require a real gap; migrating for a 1% difference just churns connections.
|
||||
if worst.score-dest.score < 100 {
|
||||
return nil, "", false // less than one forward's worth of difference
|
||||
}
|
||||
|
||||
// Choose which forward to move: the cheapest to give up. Skip disabled ones.
|
||||
var pick *TopoEntry
|
||||
for _, te := range s.TopologyList() {
|
||||
if te.OwnerID != worst.id || !te.Active || te.Link.Disabled {
|
||||
p.alive++
|
||||
if n.ID == b.engine.ID {
|
||||
p.total += p.mine
|
||||
continue
|
||||
}
|
||||
if pick == nil || te.Link.RemotePort < pick.Link.RemotePort {
|
||||
pick = te
|
||||
}
|
||||
p.total += n.Load.Forwards
|
||||
}
|
||||
if p.alive > 0 {
|
||||
p.share = (p.total + p.alive - 1) / p.alive
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// assess answers "should this node give one forward up, and which one?" together
|
||||
// with the facts behind the answer, so the read-only report can explain a node
|
||||
// that is over its share but not acting. It never mutates anything.
|
||||
//
|
||||
// blocked is a short reason: "single-node", "below-min", "balanced",
|
||||
// "nothing-active", "cooldown" or "in-flight".
|
||||
func (b *Balancer) assess(now int64) (pick *TopoEntry, p pool, blocked string) {
|
||||
p = b.pool()
|
||||
if !b.engine.IsMember() {
|
||||
return nil, p, "single-node"
|
||||
}
|
||||
if p.mine < minForwardsToShare {
|
||||
return nil, p, "below-min"
|
||||
}
|
||||
if p.mine <= p.share {
|
||||
return nil, p, "balanced"
|
||||
}
|
||||
pick = releaseCandidate(&b.engine.state, b.engine.ID)
|
||||
if pick == nil {
|
||||
return nil, "", false // everything it owns is stopped
|
||||
return nil, p, "nothing-active"
|
||||
}
|
||||
return pick, dest.id, true
|
||||
}
|
||||
|
||||
// Balancer moves forwards off an overloaded node, one at a time.
|
||||
//
|
||||
// It is a TWO-PHASE state machine, and the split is a safety requirement rather
|
||||
// than a stylistic choice:
|
||||
//
|
||||
// phase 1 (revoking): ask the current owner to stop. Wait until the topology
|
||||
// entry is actually gone before publishing anything.
|
||||
// phase 2 (publishing): hand the forward to a NAMED destination.
|
||||
//
|
||||
// Publishing both at once does not work, for two independent reasons:
|
||||
//
|
||||
// 1. loadSnapshot() derives Forwards from ForwardsOwnedBy, so the moment the old
|
||||
// entry is dropped the old owner becomes the lightest node. A plain creation
|
||||
// task is claimed by "whoever is lightest" — so the node we just took the
|
||||
// forward FROM would immediately claim it back and the migration would be a
|
||||
// no-op that merely restarts the worker.
|
||||
// 2. PendingList sorts by task id as a STRING (t10 < t5), so the revoke and the
|
||||
// submit could execute in either order within one pass.
|
||||
//
|
||||
// Migrate tasks solve (1) by carrying an explicit Target; this type solves (2) by
|
||||
// refusing to publish phase 2 before phase 1 has completed.
|
||||
type Balancer struct {
|
||||
engine *Engine
|
||||
|
||||
// ratio is how far above the average a node's Score must be, relative to the
|
||||
// lightest node, before anything moves.
|
||||
ratio float64
|
||||
// sustainedSec is how long an imbalance must persist before we act. This is
|
||||
// what stops a transient CPU/network spike from restarting workers.
|
||||
sustainedSec int64
|
||||
|
||||
// dryRun records the mode set at construction so Report can surface it.
|
||||
// SetDryRun updates it; main.go sets it once from W4F_BALANCE_MODE.
|
||||
dryRun bool
|
||||
// overSince records when the currently-overloaded node first crossed the
|
||||
// threshold. Zero means "not currently over".
|
||||
overSince map[string]int64
|
||||
// pending migration awaiting phase 1 (the forward to revoke).
|
||||
revoking *pendingMigration
|
||||
// migration awaiting phase 2 (the forward to publish, and where to).
|
||||
publishing *pendingMigration
|
||||
}
|
||||
|
||||
type pendingMigration struct {
|
||||
local string
|
||||
remote string
|
||||
port int
|
||||
fromOwner string
|
||||
toOwner string
|
||||
deadlineAt int64 // give up (re-publish for anyone) after this unix second
|
||||
}
|
||||
|
||||
// NewBalancer builds a balancer. ratio <= 1 or sustainedSec < 0 are clamped to
|
||||
// sane minimums so a bad configuration cannot turn into a migration storm.
|
||||
func NewBalancer(e *Engine, ratio float64, sustainedSec int64) *Balancer {
|
||||
if ratio <= 1 {
|
||||
ratio = 2
|
||||
// The guards below do not change the verdict ("this node is over its
|
||||
// share"), only whether now is the moment; the report shows both.
|
||||
if b.releasing != nil {
|
||||
return pick, p, "in-flight"
|
||||
}
|
||||
if sustainedSec < 0 {
|
||||
sustainedSec = 0
|
||||
}
|
||||
return &Balancer{
|
||||
engine: e, ratio: ratio, sustainedSec: sustainedSec,
|
||||
dryRun: true, // safe default: report, never move
|
||||
overSince: map[string]int64{},
|
||||
if b.lastReleaseAt > 0 && now-b.lastReleaseAt < b.cooldownSec {
|
||||
return pick, p, "cooldown"
|
||||
}
|
||||
return pick, p, ""
|
||||
}
|
||||
|
||||
// SetDryRun selects the mode. Dry-run (the default) only reports.
|
||||
func (b *Balancer) SetDryRun(v bool) { b.dryRun = v }
|
||||
|
||||
// DryRun reports whether the balancer is in observe-only mode.
|
||||
func (b *Balancer) DryRun() bool { return b.dryRun }
|
||||
|
||||
// InFlight reports whether a migration is currently between phases.
|
||||
func (b *Balancer) InFlight() bool {
|
||||
return b.revoking != nil || b.publishing != nil
|
||||
// releaseCandidate picks which of this node's forwards to give up.
|
||||
//
|
||||
// Deterministic (lowest remote port, then task id) so the choice is explainable
|
||||
// and repeatable: a node that re-evaluates the same state picks the same
|
||||
// forward. Stopped forwards are skipped — their entry is the carrier of a user's
|
||||
// stop decision, and releasing one would hand a STOPPED forward to another node
|
||||
// as if it were work to run.
|
||||
func releaseCandidate(s *State, owner string) *TopoEntry {
|
||||
var mine []*TopoEntry
|
||||
for _, te := range s.TopologyList() {
|
||||
if te.OwnerID != owner || !te.Active || te.Link.Disabled {
|
||||
continue
|
||||
}
|
||||
mine = append(mine, te)
|
||||
}
|
||||
if len(mine) == 0 {
|
||||
return nil
|
||||
}
|
||||
sort.Slice(mine, func(i, j int) bool {
|
||||
if mine[i].Link.RemotePort != mine[j].Link.RemotePort {
|
||||
return mine[i].Link.RemotePort < mine[j].Link.RemotePort
|
||||
}
|
||||
return mine[i].TaskID < mine[j].TaskID
|
||||
})
|
||||
return mine[0]
|
||||
}
|
||||
|
||||
// Step advances the state machine by one cycle. now is the current unix second.
|
||||
// It returns a short human-readable description when something happened.
|
||||
// landed reports whether the forward this node released has actually left it.
|
||||
//
|
||||
// The test is "no longer owned by me", NOT "no entry exists": the ring's normal
|
||||
// claim picks the forward up within a round or two, so an existence test would
|
||||
// see the new owner's entry and wait forever for a state that has already
|
||||
// happened.
|
||||
func (b *Balancer) landed() bool {
|
||||
m := b.releasing
|
||||
if m == nil {
|
||||
return true
|
||||
}
|
||||
owner := b.engine.state.TopologyOwner(&Task{
|
||||
Local: store.Local{Name: m.local},
|
||||
Remote: store.Remote{Name: m.remote},
|
||||
Link: store.Link{RemotePort: m.port},
|
||||
})
|
||||
return owner != b.engine.ID
|
||||
}
|
||||
|
||||
// Step advances the balancer by one cycle and returns a human-readable
|
||||
// description when it did (or, in dry-run, would have done) something.
|
||||
func (b *Balancer) Step(now int64) string {
|
||||
// --- phase 2: did phase 1 finish? ---
|
||||
if b.revoking != nil {
|
||||
if b.engine.state.TopologyOwner(&Task{
|
||||
Local: store.Local{Name: b.revoking.local},
|
||||
Remote: store.Remote{Name: b.revoking.remote},
|
||||
Link: store.Link{RemotePort: b.revoking.port},
|
||||
}) != "" {
|
||||
// Still owned: the revoke has not completed yet. Wait, unless it has
|
||||
// been far too long, in which case something is wrong with the ring.
|
||||
if now < b.revoking.deadlineAt {
|
||||
return ""
|
||||
}
|
||||
log.Printf("ring[%s] migrate revoke for %s→%s:%d timed out; abandoning",
|
||||
b.engine.ID, b.revoking.local, b.revoking.remote, b.revoking.port)
|
||||
b.revoking = nil
|
||||
if b.releasing != nil {
|
||||
if b.landed() {
|
||||
done := b.releasing
|
||||
b.releasing = nil
|
||||
return "gave up " + done.local + ":" + strconv.Itoa(done.port) + " for rebalancing"
|
||||
}
|
||||
if now-b.releasing.startedAt > releaseDeadlineSec {
|
||||
log.Printf("ring[%s] release of %s:%d not confirmed within %ds; clearing (forward is still ours)",
|
||||
b.engine.ID, b.releasing.local, b.releasing.port, releaseDeadlineSec)
|
||||
b.releasing = nil
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
pick, p, blocked := b.assess(now)
|
||||
if pick == nil {
|
||||
b.reported = ""
|
||||
return ""
|
||||
}
|
||||
desc := pick.Local.Name + ":" + strconv.Itoa(pick.Link.RemotePort) +
|
||||
" (" + strconv.Itoa(p.mine) + " owned > " + strconv.Itoa(p.share) + " share of " +
|
||||
strconv.Itoa(p.total) + " over " + strconv.Itoa(p.alive) + " nodes)"
|
||||
|
||||
switch b.mode {
|
||||
case BalanceOff:
|
||||
return ""
|
||||
case BalanceDryRun:
|
||||
// Log a change of mind, not the same line every tick.
|
||||
if b.reported == desc {
|
||||
return ""
|
||||
}
|
||||
// Entry gone — the old worker is stopped. Publish for the named target.
|
||||
m := b.revoking
|
||||
b.revoking = nil
|
||||
b.publishing = m
|
||||
b.publishMigration()
|
||||
return "revoke complete, migrating " + m.local + ":" + strconv.Itoa(m.port) + " → " + m.toOwner
|
||||
b.reported = desc
|
||||
if blocked != "" {
|
||||
return "would release " + desc + " [blocked: " + blocked + "]"
|
||||
}
|
||||
return "would release " + desc
|
||||
}
|
||||
|
||||
// --- phase 1: wait for an in-flight publish, or start a new migration ---
|
||||
if b.publishing != nil {
|
||||
// The migrate task carries itself in the ring; nothing to do but let it
|
||||
// ride. Clear once the forward is back in the topology.
|
||||
if b.engine.state.TopologyOwner(&Task{
|
||||
Local: store.Local{Name: b.publishing.local},
|
||||
Remote: store.Remote{Name: b.publishing.remote},
|
||||
Link: store.Link{RemotePort: b.publishing.port},
|
||||
}) == b.publishing.toOwner {
|
||||
done := b.publishing
|
||||
b.publishing = nil
|
||||
return "migrated " + done.local + ":" + strconv.Itoa(done.port) + " to " + done.toOwner
|
||||
}
|
||||
if now >= b.publishing.deadlineAt {
|
||||
// Nobody claimed it. Submit an UNTARGETED task so whichever node is
|
||||
// lightest picks it up — the ring keeps the forward alive rather than
|
||||
// stranding it, which is the same guarantee the rest of the protocol
|
||||
// relies on.
|
||||
m := b.publishing
|
||||
b.publishing = nil
|
||||
b.engine.SubmitTask(store.Local{Name: m.local}, store.Remote{Name: m.remote},
|
||||
store.Link{RemotePort: m.port})
|
||||
return "migrate target did not claim " + m.local + ":" + strconv.Itoa(m.port) + "; re-opened for any node"
|
||||
}
|
||||
// Mode is ON. Blocked reasons still stop us.
|
||||
if blocked != "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
// --- nothing in flight: look for work ---
|
||||
pick, dest, hot := b.engine.findBalanceCandidate(b.ratio)
|
||||
if hot != "" {
|
||||
if _, seen := b.overSince[hot]; !seen {
|
||||
b.overSince[hot] = now
|
||||
}
|
||||
} else {
|
||||
b.overSince = map[string]int64{}
|
||||
}
|
||||
if pick == nil || dest == "" {
|
||||
t := b.engine.ReleaseTask(pick.Local, pick.Remote, pick.Link)
|
||||
if t == nil {
|
||||
return ""
|
||||
}
|
||||
if since, ok := b.overSince[pick.OwnerID]; !ok || now-since < int64(b.sustainedSec) {
|
||||
return ""
|
||||
b.releasing = &releaseState{
|
||||
local: pick.Local.Name, remote: pick.Remote.Name,
|
||||
port: pick.Link.RemotePort, startedAt: now,
|
||||
}
|
||||
// Phase 1: publish the revoke and remember what we are waiting for.
|
||||
m := &pendingMigration{
|
||||
local: pick.Local.Name, remote: pick.Remote.Name, port: pick.Link.RemotePort,
|
||||
fromOwner: pick.OwnerID, toOwner: dest,
|
||||
// Generous deadline: a token round is ~2s, and the owner may be busy.
|
||||
deadlineAt: now + 120,
|
||||
}
|
||||
b.revoking = m
|
||||
b.engine.RevokeTask(pick.Local, pick.Remote, pick.Link)
|
||||
// The revoke task must be able to reach the owner even though the balancer
|
||||
// runs on the leader: it travels in the ring like any other command.
|
||||
return "revoking " + m.local + ":" + strconv.Itoa(m.port) + " from " + m.fromOwner
|
||||
b.lastReleaseAt = now
|
||||
b.reported = ""
|
||||
return "releasing " + desc
|
||||
}
|
||||
|
||||
func (b *Balancer) publishMigration() {
|
||||
m := b.publishing
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
b.engine.submitMigrate(store.Local{Name: m.local}, store.Remote{Name: m.remote},
|
||||
store.Link{RemotePort: m.port}, m.toOwner)
|
||||
}
|
||||
|
||||
// findBalanceCandidate exposes planBalance to the engine's live state.
|
||||
func (e *Engine) findBalanceCandidate(ratio float64) (*TopoEntry, string, string) {
|
||||
pick, dest, ok := planBalance(&e.state, ratio)
|
||||
if !ok {
|
||||
return nil, "", ""
|
||||
}
|
||||
// Report the overloaded node even when there is nothing movable, so the
|
||||
// hysteresis timer can start for a node whose forwards are all stopped.
|
||||
return pick, dest, pick.OwnerID
|
||||
}
|
||||
|
||||
// WatchBalance drives the balancer on a ticker until ctx is cancelled.
|
||||
// Watch drives Step on a ticker until ctx is cancelled.
|
||||
//
|
||||
// Only the LEADER runs this. Balancing is a cluster-wide decision (it moves
|
||||
// forwards between members), and letting every member decide independently would
|
||||
// mean several migrations racing for the same forward — the ring is a shared
|
||||
// token, so two nodes must not both publish a revoke for the same entry.
|
||||
// It runs on EVERY node, not just the leader. Balancing here is not a cluster
|
||||
// decision being delegated to one node — it is each node applying the same
|
||||
// replicated rule to its own forwards. Two nodes never contend for the same
|
||||
// forward because a node can only release what it owns.
|
||||
//
|
||||
// dryRun makes the balancer report what it WOULD do without publishing anything.
|
||||
// That is the mode to use first on a live cluster: it answers "is my real data
|
||||
// going to trip the threshold, and what would it move?" before anything on the
|
||||
// ring is disturbed.
|
||||
func (b *Balancer) WatchBalance(ctx context.Context, interval time.Duration, dryRun bool) {
|
||||
// The first check happens after one interval rather than immediately: a node
|
||||
// that has just booted adopts the ring's topology and its own load in the first
|
||||
// token round, and deciding before that would act on a half-known pool.
|
||||
func (b *Balancer) Watch(ctx context.Context, interval time.Duration) {
|
||||
tick := time.NewTicker(interval)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
@ -326,69 +321,46 @@ func (b *Balancer) WatchBalance(ctx context.Context, interval time.Duration, dry
|
||||
return
|
||||
case <-tick.C:
|
||||
}
|
||||
if !b.engine.IsLeader() {
|
||||
continue
|
||||
}
|
||||
now := time.Now().Unix()
|
||||
if dryRun {
|
||||
if msg := b.stepDryRun(now); msg != "" {
|
||||
log.Printf("ring[%s] balance (dry-run): %s", b.engine.ID, msg)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if msg := b.Step(now); msg != "" {
|
||||
if msg := b.Step(time.Now().Unix()); msg != "" {
|
||||
log.Printf("ring[%s] balance: %s", b.engine.ID, msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// stepDryRun reports the migration the balancer would start right now, without
|
||||
// publishing a revoke. Hysteresis is still applied so the report reflects the
|
||||
// same sustained-imbalance rule the real path uses.
|
||||
func (b *Balancer) stepDryRun(now int64) string {
|
||||
if b.InFlight() {
|
||||
return ""
|
||||
}
|
||||
pick, dest, hot := b.engine.findBalanceCandidate(b.ratio)
|
||||
if hot == "" {
|
||||
b.overSince = map[string]int64{}
|
||||
return ""
|
||||
}
|
||||
if _, seen := b.overSince[hot]; !seen {
|
||||
b.overSince[hot] = now
|
||||
return ""
|
||||
}
|
||||
if pick == nil || dest == "" {
|
||||
return ""
|
||||
}
|
||||
if since, ok := b.overSince[pick.OwnerID]; !ok || now-since < int64(b.sustainedSec) {
|
||||
return ""
|
||||
}
|
||||
return "would migrate " + pick.Local.Name + ":" + strconv.Itoa(pick.Link.RemotePort) +
|
||||
" from " + pick.OwnerID + " to " + dest
|
||||
}
|
||||
// ---- read-only report for the API/UI ----
|
||||
|
||||
// BalanceReport is a read-only view of what the balancer sees and would do.
|
||||
// BalanceReport is what this node thinks about balancing right now.
|
||||
//
|
||||
// It exists because the balancer's decision is otherwise only visible in the
|
||||
// log, and only when an imbalance exists AND the hysteresis window has elapsed.
|
||||
// An operator asking "is this cluster balanced?" gets silence in exactly the
|
||||
// healthy case, which is indistinguishable from "balancing is switched off".
|
||||
// It is per-node by nature: every member has its own answer, and the answer that
|
||||
// matters is the one on the node doing (or refusing to do) the releasing. The
|
||||
// peer list is included because "am I over my share?" is only meaningful next to
|
||||
// what everyone else holds.
|
||||
type BalanceReport struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
DryRun bool `json:"dryRun"`
|
||||
Ratio float64 `json:"ratio"`
|
||||
SustainSec int64 `json:"sustainSec"`
|
||||
// Only the leader acts; on a follower these stay false/empty.
|
||||
IsLeader bool `json:"isLeader"`
|
||||
// Current per-node scores.
|
||||
Mode string `json:"mode"` // off | dry-run | on
|
||||
Enabled bool `json:"enabled"` // mode != off
|
||||
DryRun bool `json:"dryRun"` // mode != on
|
||||
IsMember bool `json:"isMember"`
|
||||
Self string `json:"self"`
|
||||
|
||||
SelfForwards int `json:"selfForwards"`
|
||||
Share int `json:"share"`
|
||||
TotalForwards int `json:"totalForwards"`
|
||||
AliveNodes int `json:"aliveNodes"`
|
||||
SelfOverShare bool `json:"selfOverShare"`
|
||||
|
||||
// WouldRelease is what this node would give up (present whenever it is over
|
||||
// its share, including while blocked — see BlockedBy).
|
||||
WouldRelease *BalanceReleasePlan `json:"wouldRelease,omitempty"`
|
||||
// Releasing is the release this node has published and is awaiting.
|
||||
Releasing *BalanceReleasePlan `json:"releasing,omitempty"`
|
||||
// BlockedBy explains why an over-share node is not acting yet.
|
||||
BlockedBy string `json:"blockedBy,omitempty"`
|
||||
|
||||
CooldownSec int64 `json:"cooldownSec"`
|
||||
LastReleaseAt int64 `json:"lastReleaseAt,omitempty"`
|
||||
|
||||
// Peers: every member's replicated load, for context.
|
||||
Nodes []BalanceNodeReport `json:"nodes"`
|
||||
// Overloaded (above ratio x mean) and sustained past the window, if any.
|
||||
OverloadedSince int64 `json:"overloadedSince,omitempty"`
|
||||
// The migration that would happen on the next Step, if any.
|
||||
WouldMigrate *BalanceMigrationPlan `json:"wouldMigrate,omitempty"`
|
||||
// Set when a migration is between phases.
|
||||
InFlight *BalanceMigrationPlan `json:"inFlight,omitempty"`
|
||||
}
|
||||
|
||||
type BalanceNodeReport struct {
|
||||
@ -401,24 +373,47 @@ type BalanceNodeReport struct {
|
||||
Score float64 `json:"score"`
|
||||
}
|
||||
|
||||
type BalanceMigrationPlan struct {
|
||||
Local string `json:"local"`
|
||||
Remote string `json:"remote"`
|
||||
Port int `json:"remotePort"`
|
||||
From string `json:"from"`
|
||||
To string `json:"to"`
|
||||
type BalanceReleasePlan struct {
|
||||
Local string `json:"local"`
|
||||
Remote string `json:"remote"`
|
||||
RemotePort int `json:"remotePort"`
|
||||
Owner string `json:"owner"`
|
||||
}
|
||||
|
||||
// Report renders the balancer's current view. It never mutates anything —
|
||||
// note that it does NOT touch overSince, so calling it cannot start or reset
|
||||
// the hysteresis timer that the real Step path uses.
|
||||
func (b *Balancer) Report(now int64) BalanceReport {
|
||||
// Report renders the current view. It never mutates anything — in particular it
|
||||
// does NOT touch lastReleaseAt or the release state, so an operator polling this
|
||||
// cannot start, delay or cancel a release. (A read-only endpoint that perturbs
|
||||
// the behaviour it reports is worse than no endpoint.)
|
||||
func (b *Balancer) Report() BalanceReport {
|
||||
now := time.Now().Unix()
|
||||
rep := BalanceReport{
|
||||
Enabled: !b.dryRun,
|
||||
DryRun: b.dryRun,
|
||||
Ratio: b.ratio,
|
||||
SustainSec: int64(b.sustainedSec),
|
||||
IsLeader: b.engine.IsLeader(),
|
||||
Mode: b.mode,
|
||||
Enabled: b.mode != BalanceOff,
|
||||
DryRun: b.mode != BalanceOn,
|
||||
IsMember: b.engine.IsMember(),
|
||||
Self: b.engine.ID,
|
||||
CooldownSec: b.cooldownSec,
|
||||
}
|
||||
p := b.pool()
|
||||
rep.SelfForwards, rep.Share = p.mine, p.share
|
||||
rep.TotalForwards, rep.AliveNodes = p.total, p.alive
|
||||
rep.SelfOverShare = p.mine > p.share
|
||||
|
||||
if pick, _, blocked := b.assess(now); pick != nil {
|
||||
rep.WouldRelease = &BalanceReleasePlan{
|
||||
Local: pick.Local.Name, Remote: pick.Remote.Name,
|
||||
RemotePort: pick.Link.RemotePort, Owner: pick.OwnerID,
|
||||
}
|
||||
rep.BlockedBy = blocked
|
||||
}
|
||||
if m := b.releasing; m != nil {
|
||||
rep.Releasing = &BalanceReleasePlan{
|
||||
Local: m.local, Remote: m.remote, RemotePort: m.port, Owner: b.engine.ID,
|
||||
}
|
||||
rep.BlockedBy = "in-flight"
|
||||
}
|
||||
if b.lastReleaseAt > 0 {
|
||||
rep.LastReleaseAt = b.lastReleaseAt
|
||||
}
|
||||
for i := range b.engine.state.Nodes {
|
||||
n := &b.engine.state.Nodes[i]
|
||||
@ -428,46 +423,5 @@ func (b *Balancer) Report(now int64) BalanceReport {
|
||||
Score: n.Load.Score(),
|
||||
})
|
||||
}
|
||||
if m := b.revoking; m != nil {
|
||||
rep.InFlight = &BalanceMigrationPlan{
|
||||
Local: m.local, Remote: m.remote, Port: m.port, From: m.fromOwner, To: m.toOwner,
|
||||
}
|
||||
}
|
||||
if m := b.publishing; m != nil {
|
||||
rep.InFlight = &BalanceMigrationPlan{
|
||||
Local: m.local, Remote: m.remote, Port: m.port, From: m.fromOwner, To: m.toOwner,
|
||||
}
|
||||
}
|
||||
// Only a leader would act, so only report a plan there.
|
||||
if rep.IsLeader {
|
||||
pick, dest, hot := b.engine.findBalanceCandidate(b.ratio)
|
||||
if hot != "" {
|
||||
rep.OverloadedSince = b.overSince[hot]
|
||||
}
|
||||
if pick != nil && dest != "" {
|
||||
plan := &BalanceMigrationPlan{
|
||||
Local: pick.Local.Name, Remote: pick.Remote.Name,
|
||||
Port: pick.Link.RemotePort, From: pick.OwnerID, To: dest,
|
||||
}
|
||||
// surface it as "would migrate" only once the window is satisfied
|
||||
if since, ok := b.overSince[pick.OwnerID]; ok && now-since >= int64(b.sustainedSec) {
|
||||
rep.WouldMigrate = plan
|
||||
}
|
||||
}
|
||||
}
|
||||
return rep
|
||||
}
|
||||
|
||||
// SetRatioAndSustain updates the thresholds after construction, so main.go can
|
||||
// create the balancer early (the HTTP handler needs it) and apply the environment
|
||||
// configuration once it has parsed it.
|
||||
func (b *Balancer) SetRatioAndSustain(ratio float64, sustainedSec int64) {
|
||||
if ratio <= 1 {
|
||||
ratio = 2
|
||||
}
|
||||
if sustainedSec < 0 {
|
||||
sustainedSec = 0
|
||||
}
|
||||
b.ratio = ratio
|
||||
b.sustainedSec = sustainedSec
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user