jianf/webui4frpc
1
0
Fork 0
mirror of https://gitcode.com/JianFeeeee/webui4frpc.git synced 2026-10-15 12:14:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(cluster): 负载均衡改为每节点自主「先撤销本地转发，再丢回环」

Browse Source 
按用户要求重做：不再由 leader 单独调度，而是**每个节点对自己的转发跑同一条
规则**，超了就自己撤销一条、把任务丢回环上，由环原有的「最闲者认领」决定它
落到哪台。

## 为什么必须是 owner 自己动手（不是洁癖，是实测逼出来的）

真机探针（跑完已删）显示，真实 revoke 之后：

    TopologyOwner after revoke = "n1"     ← 条目还在（只标记不删除）
    n1 Forwards=3  score=310.0            ← 负载根本没降
    after step: revoking=true publishing=false

⇒ 一节点的负载**只有它自己的拓扑条目消失时才会下降**，而条目是本地簿记，没有任何
对端可以代它写。leader 从外面只能看见失衡、无法让失衡成为事实——它必须伸手去改
另一个节点的簿记，而这正是这个环到处都在避免的跨节点写（revoke / restart 定向往
owner 也是同一个理由）。

⇒ 而且旧实现**根本跑不起来**：phase 1 判据是「条目消失」，可 mark-don't-remove
让它永远不消失，于是每 120s 打一条 `revoking … timed out; abandoning`，
**phase 2 永不执行，一条转发都不会迁**。开 `W4F_BALANCE_MODE=on` 只会刷日志。

## 规则（替代原来的 ratio 阈值）

    当 Forwards >= 2 且 Forwards > ceil(总数/存活节点数) 时，撤销一条

- **自限**：3 条 / 3 台 → owner 2 > 1，卸一条 → 1，`1 > 1` 不成 → 停。
  没有上界的话 owner 会把转发**全部卸光**。
- **不会卸空**：低于下限（2）永不撤销。
- 平摊值向上取整（3/3=1，不是 0），否则 1 条 / 3 台的节点也算「超了」。
- **不再用 mem/net 参与判据**：实测 `.30` 空闲时 netPct 能到 55%，Score 会高于
  有转发的节点——用 Score 会把纯网络抖动当成负载。
- 节奏用**每节点冷却**（默认 600s）而不是「失衡持续多久」窗口：规则本身已经要求
  真实失衡，没有需要平滑的瞬态，只有需要限制的抖动频率。

## 为什么必须是新的 Release 任务类型，不能复用 Revoke

用户停止（Revoke）**保留拓扑条目**作为 disabled 标志的载体；均衡释放必须**删除
条目**，因为 (a) 负载由自己的条目数算出，条目不删负载不降；(b) 每个节点的
duplicate-guard 会拒绝「已有 owner 的转发」的创建任务——保留条目会让**所有节点
都拒领**，转发被永久搁死在刚放弃它的那台机器上。

Release 因此：`Handler.Release`（停 worker，但**不标记 stopped**）→ 删条目 →
`SubmitTask` 以**启用状态**重新发布。它需要自己的 Handler 方法，不能回落到
Revoke：Revoke 会顺手把 link 标成 stopped，而这条转发即将在别的节点上跑。

## 顺带修掉两个既有缺口（实测确认）

1. **`submitMigrate` 发布不出去**：`OnToken` 的 localPending 重合并只保留「从未
   publish 过」的本地 pending，而只有 `Add*` 系列会记 `published`。引擎里直接写
   `PendingTasks` 的任务会在下一次 adopt 时被静默丢弃。
   ⇒ 新 `ReleaseTask` 走 `AddPending`。
   （`SubmitRestart` 有同样的毛病，但它不在本次范围，未改动——见提交说明末尾。）
2. `Release` 需要绕过 duplicate-claim 守卫（同 Restart），否则「owner 本来就持有
   这条转发」这一前提会让每条 Release 都被丢掉。

## 测试（17 条，全绿）

规则：平摊值取整、超了才撤、撤到平摊就停、单节点不撤、低于下限不撤、不撤已停
转发、候选选择确定。
机制：Revoke 留载体 vs Release 删条目（这是两种语义的核心差异）、Release 只由
owner 执行且不消费他人的任务、owner 释放后环的普通认领把它接走。
模式：dry-run 不动任何东西、off 真的什么都不做、未知模式回落 dry-run、
冷却限制抖动频率、确认超时清理。
只读：Report 不变更任何状态、能解释「超了但不动」的原因、均衡节点给出真实答案。

**双向验证 3 条**（改坏→变红→还原→变绿）：
- 去掉 Release 的 duplicate-guard 例外 → `the owner released 0 times, want 1`
- 去掉「不超过平摊值」上界 → `2 owned against share 2 must not release`
- Release 改成只标记不删除 → `n1 should now own 2, got 3`

go build / go vet / go test ./... 全绿（go clean -testcache 后逐包实跑）。

## 未做 / 待办

- `SubmitRestart` 的同类发布缺口未修（同样会被 adopt 丢弃），需要单独确认它
  在实机上是否真的丢过 restart。
- UI 尚未跟进新报告结构（下一步）。
This commit is contained in:
JianFeeeee
2026-10-06 14:03:03 +08:00
parent b40b53b40b
commit c69fb2caba
8 changed files with 989 additions and 860 deletions
Download Patch File Download Diff File Expand all files Collapse all files

117
internal/cluster/ring_engine.go
View File

@ -24,6 +24,12 @@ type Handler interface {
// creation path (idempotency guard) and the duplicate-claim guard refuse to
// re-create an existing forward.
Restart(ctx context.Context, tk *Task) error
// Release stops this node's worker for a forward it is giving up for
// rebalancing, WITHOUT marking the link stopped. The forward is about to run
// on another node, so a stopped flag would be wrong: every node's duplicate
// guard reads it and would then refuse to claim the re-published task,
// stranding the forward on the node that just gave it up.
Release(ctx context.Context, tk *Task) error
RuntimeLoad() Load
}
@ -406,16 +412,24 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
}
continue // owned elsewhere — ride to the owner
}
if t.Migrate {
// Addressed to one node. The whole point is that "whoever is
// lightest" is the WRONG answer here: the node we just revoked the
// forward from is the one that became lightest, and it would claim
// the task straight back.
if t.Target == e.ID {
if t.Release {
// Owner-directed, like a revoke: only the node holding the forward
// may give it up. A release executed anywhere else would be an
// attempt to move somebody else's load, which cannot work — the
// owner's own count is what has to fall, and its topology entry is
// local bookkeeping no peer may write.
//
// A release whose forward is already gone is an idempotent no-op
// consumed by the lowest node, so it cannot ride forever if its
// owner departed first.
if owner := e.state.TopologyOwner(t); owner == e.ID {
target = t
break
} else if owner == "" && selfIsLowest {
target = t
break
}
continue // ride to the target
continue // owned elsewhere — ride to the owner
}
if t.Restart {
// Owner-directed, exactly like a revoke: only the node holding the
@ -549,7 +563,11 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
// exists precisely to re-activate a forward whose entry is still present
// (a stop keeps the entry as the flag's carrier), so applying the guard
// would swallow every re-enable and leave the forward stopped forever.
if !claimed.Restart && !claimed.Migrate {
// A RELEASE is the second deliberate exception, for the same reason as a
// restart: it is addressed to the node that ALREADY owns the forward, so
// the entry is still there by definition. Applying the guard would drop
// every release and a node could never shed anything.
if !claimed.Restart && !claimed.Release {
if owner := e.state.TopologyOwner(claimed); owner != "" {
log.Printf("ring[%s] drop duplicate task %s: %s→%s:%d already owned by %s",
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
@ -557,33 +575,46 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
continue
}
}
if claimed.Migrate {
// We are the designated destination. The forward must NOT already be
// here: the two-phase migration guarantees the old entry was gone before
// this task was published. If one somehow exists, drop the task rather
// than creating a second topology entry for the same forward.
if owner := e.state.TopologyOwner(claimed); owner != "" {
log.Printf("ring[%s] drop migrate %s: already owned by %s", e.ID, claimed.ID, owner)
if claimed.Release {
// Only reached when this node owns the forward (or it is already gone
// and we are the fallback consumer — see the selection loop).
owner := e.state.TopologyOwner(claimed)
if owner != e.ID {
// The forward vanished before we got here; nothing to give up.
continue
}
// Stop our worker. Deliberately NOT Handler.Revoke: that marks the
// link stopped, and the forward is about to run elsewhere — a
// "stopped" flag would make every node treat the re-published task as
// a duplicate and refuse to claim it.
if e.Handler != nil {
if err := e.Handler.Claim(ctx, claimed); err != nil {
// Put it back so another cycle can retry — the forward is
// currently owned by nobody, so losing this task would strand it.
e.state.PendingTasks[claimed.ID] = claimed
log.Printf("ring[%s] migrate %s failed: %v", e.ID, claimed.ID, err)
break
if err := e.Handler.Release(ctx, claimed); err != nil {
log.Printf("ring[%s] release %s: %v", e.ID, claimed.ID, err)
}
}
e.state.AddTopology(claimed, e.ID)
// Drop the entry so the forward has no owner. This is what makes the
// load actually fall (ForwardsOwnedBy counts entries) and what lets a
// fresh creation task pass the duplicate guard above.
e.RemoveTopologyEntry(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort)
// Hand it back to the ring as an ordinary unclaimed forward. It is
// published by THIS node so it rides the very token we are holding —
// no coordinator, no addressed hand-off.
link := claimed.Link
link.Disabled = false
if t := e.SubmitTask(claimed.Local, claimed.Remote, link); t != nil {
log.Printf("ring[%s] released %s: %s→%s:%d back to the ring as %s",
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
claimed.Link.RemotePort, t.ID)
} else {
log.Printf("ring[%s] released %s but the re-publish was deduped",
e.ID, claimed.ID)
}
if e.Log != nil {
_, _ = e.Log.Append(e.ID, LogForwardAdd, map[string]any{
"taskId": claimed.ID, "local": claimed.Local.Name, "remote": claimed.Remote.Name,
"migratedFrom": e.state.TopologyOwner(claimed),
_, _ = e.Log.Append(e.ID, LogForwardStop, map[string]any{
"taskId": claimed.ID, "local": claimed.Local.Name,
"remote": claimed.Remote.Name, "released": true,
})
}
log.Printf("ring[%s] migrated in %s: %s→%s:%d", e.ID, claimed.ID,
claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort)
continue
}
if claimed.Restart {
@ -1000,27 +1031,23 @@ func (e *Engine) SubmitRestart(local store.Local, remote store.Remote, link stor
return t
}
// submitMigrate publishes a load-balancing task addressed to one specific node.
// ReleaseTask publishes a rebalancing task asking the OWNER of this forward to
// give it up. The task rides the token like any other command; it is not
// addressed to a destination, because the ring's normal lowest-load claim
// decides where the forward goes next.
//
// It is addressed on purpose. A plain creation task is claimed by "whichever node
// is currently lightest", and after a revoke the node we just took the forward
// from IS the lightest — so an untargeted task would be claimed straight back by
// the old owner and the migration would achieve nothing but a worker restart.
func (e *Engine) submitMigrate(local store.Local, remote store.Remote, link store.Link, target string) *Task {
// It goes through AddPending rather than writing PendingTasks directly. That is
// not cosmetic: OnToken rebuilds local-only pending commands from the ids it
// never published, and only the Add* helpers record them — a task injected by
// hand was silently dropped at the next state adoption, so it never reached the
// ring at all.
func (e *Engine) ReleaseTask(local store.Local, remote store.Remote, link store.Link) *Task {
link.Disabled = false
t := &Task{
ID: e.state.NextTaskID(),
Local: local,
Remote: remote,
Link: link,
Created: time.Now().Unix(),
Migrate: true,
Target: target,
t := e.state.AddPending(local, remote, link)
if t == nil {
return nil
}
if e.state.PendingTasks == nil {
e.state.PendingTasks = map[string]*Task{}
}
e.state.PendingTasks[t.ID] = t
t.Release = true
return t
}