mirror of
https://gitcode.com/JianFeeeee/webui4frpc.git
synced 2026-10-15 12:14:37 +00:00
Compare commits
28 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 21eb52df24 | |||
| 1a850b4846 | |||
| c90d4f3791 | |||
| 790eaa5ed1 | |||
| c69fb2caba | |||
| b40b53b40b | |||
| d347d2a648 | |||
| 8176478269 | |||
| 45a6a76621 | |||
| 7d5bca5b72 | |||
| 875e5805eb | |||
| 084ef62b32 | |||
| 57d4d14d28 | |||
| fc4cbaa726 | |||
| bfd7265405 | |||
| 2dd32648e6 | |||
| 50c7729779 | |||
| 918ca5d5ba | |||
| c8ab2584a8 | |||
| 8ab52b238a | |||
| 5cdc052d01 | |||
| 1c835425de | |||
| 041cc04dd6 | |||
| 46e8bc3703 | |||
| f4fb964734 | |||
| 64d3359981 | |||
| 0559cc55bd | |||
| 609f0df517 |
83
README.md
83
README.md
@ -32,6 +32,14 @@
|
||||
- 主负载信号:当前节点拥有的转发数(权重 ×100)
|
||||
- 次级信号:**NIC 网络接口利用率**(从 `/proc/net/dev` 增量 + `/sys/class/net/*/speed` 计算,排除了 lo/docker/veth 等虚拟接口)
|
||||
- 同转发数节点间由内存占用 + 网络饱和度打破平局
|
||||
- **负载均衡(自导式)**:每个节点对**自己的**转发跑同一条规则 —— 持有数超过
|
||||
`ceil(总转发数 / 存活节点数)` 且至少 2 条时,自行撤销一条、把任务丢回环上,
|
||||
再由环按「最闲者优先」重新认领
|
||||
- **没有中心调度**:一节点的负载只有它自己的拓扑条目消失时才会下降,而条目是
|
||||
本地簿记,没有对端能代它写,所以只能由持有者自己动手
|
||||
- 规则自限:撤到平摊值即停(否则会把转发卸光),且低于下限永不撤、不会卸空节点
|
||||
- 每节点默认 10 分钟冷却,限制抖动频率;默认 **dry-run**(只报告不释放)
|
||||
- 集群页每节点一个开关:关闭 / 观察模式 / 启用,可持久化
|
||||
- **per-forward 独立进程**:每条转发拥有独立的 frpc 配置文件和 frpc 进程(worker 键 = `local~remote~port`),单条转发故障不影响兄弟,消除旧模型的多节点 `proxy already exists` 冲突
|
||||
- **容错自愈**:leader / 非 leader 节点宕机 → 自动检测 → 环跳过死亡节点、任务重挂;leader 宕机 → 上邻居晋升为新 leader
|
||||
- **宕机自动重联**:节点宕机重启后通过缓存的集群拓扑(对端地址 + 准入密钥)自动重联,显式脱离集群的节点不自动重联
|
||||
@ -79,7 +87,7 @@ cd web && npm run build && rm -rf internal/httpapi/dist && cp -r web/dist intern
|
||||
### Linux (Debian / Ubuntu)
|
||||
|
||||
```bash
|
||||
sudo dpkg -i webui4frpc-0.1.1-linux-amd64.deb # 或 linux-arm64.deb
|
||||
sudo dpkg -i webui4frpc-0.1.4-linux-amd64.deb # 或 linux-arm64.deb
|
||||
sudo nano /etc/webui4frpc/webui4frpc.env # 改掉默认密码
|
||||
sudo systemctl restart webui4frpc
|
||||
```
|
||||
@ -87,7 +95,7 @@ sudo systemctl restart webui4frpc
|
||||
### Linux (RHEL / Fedora / openEuler / 默认 / 龙蜥)
|
||||
|
||||
```bash
|
||||
sudo rpm -i webui4frpc-0.1.1-linux-x86_64.rpm
|
||||
sudo rpm -i webui4frpc-0.1.4-linux-x86_64.rpm
|
||||
sudo vi /etc/webui4frpc/webui4frpc.env
|
||||
sudo systemctl enable --now webui4frpc
|
||||
```
|
||||
@ -97,7 +105,7 @@ sudo systemctl enable --now webui4frpc
|
||||
|
||||
### Windows
|
||||
|
||||
双击 `webui4frpc-0.1.1-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
||||
双击 `webui4frpc-0.1.4-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
||||
安装向导会询问监听地址与管理员凭据,完成后:
|
||||
|
||||
- 程序装到 `C:\Program Files\webui4frpc`
|
||||
@ -107,8 +115,8 @@ sudo systemctl enable --now webui4frpc
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
sudo installer -pkg webui4frpc-0.1.1-darwin-arm64.pkg -target / # Apple Silicon
|
||||
sudo installer -pkg webui4frpc-0.1.1-darwin-amd64.pkg -target / # Intel
|
||||
sudo installer -pkg webui4frpc-0.1.4-darwin-arm64.pkg -target / # Apple Silicon
|
||||
sudo installer -pkg webui4frpc-0.1.4-darwin-amd64.pkg -target / # Intel
|
||||
```
|
||||
|
||||
或直接双击 `.pkg` 走图形安装器。安装后 launchd 开机自启,配置在
|
||||
@ -126,7 +134,7 @@ sudo launchctl kickstart -k system/com.jianfeeeee.webui4frpc
|
||||
不想装服务只想跑一下,用对应的 `.tar.gz` / `.zip`,解压即用:
|
||||
|
||||
```bash
|
||||
tar xzf webui4frpc-0.1.1-linux-amd64.tar.gz
|
||||
tar xzf webui4frpc-0.1.4-linux-amd64.tar.gz
|
||||
./webui4frpc -addr 127.0.0.1:7500 -user admin -password 你的密码 -workdir ./data
|
||||
```
|
||||
|
||||
@ -160,6 +168,19 @@ sha256sum -c SHA256SUMS --ignore-missing
|
||||
|
||||
环境变量 `W4F_HOST` 可覆盖节点对外可达的主机名(Docker DNS / 通配监听场景用)。
|
||||
|
||||
### 负载均衡参数
|
||||
|
||||
每节点独立生效;集群页的开关**优先于**这里的默认值(在页面上选过就以页面为准,重启不丢)。
|
||||
|
||||
| 环境变量 | 默认 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `W4F_BALANCE_MODE` | `dry-run` | `off` 不参与 / `dry-run` 只报告 / `on` 真正释放 |
|
||||
| `W4F_BALANCE_COOLDOWN` | `600` | 同一节点两次释放之间的最小秒数 |
|
||||
| `W4F_BALANCE_INTERVAL` | `10` | 检查间隔(秒) |
|
||||
|
||||
**释放会停掉该转发的 frpc worker,其上的在途连接会断一次**,由新节点认领后恢复;
|
||||
因此默认是 `dry-run`。集群页的开关走 write 级权限。
|
||||
|
||||
### 启动行为
|
||||
|
||||
节点启动时的集群引导优先级:
|
||||
@ -191,10 +212,10 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
||||
│ ├── render 渲染 frpc JSON 配置(per-forward 单 proxy + 高级参数 + LB/健康检查)
|
||||
│ ├── process worker 进程管理(spawn/stop/restart/自愈/日志轮转,worker 键 = local~remote~port)
|
||||
│ ├── install 一键下载官方 frpc(GitHub Releases,含解压安全防护)
|
||||
│ ├── cluster 令牌环协议(ring engine / leader 选举 / 容错 / 宕机重联 / 二进制分发 / NIC 负载采样)
|
||||
│ ├── cluster 令牌环协议(ring engine / leader 选举 / 容错 / 宕机重联 / 二进制分发 / NIC 负载采样 / 自导式负载均衡)
|
||||
│ └── httpapi REST API + 静态资源(auth/session / canvas / forwards / cluster / users / audit 导出)
|
||||
└── web/ Vue 3 SPA(Element Plus + VueFlow 画布 + Pinia)
|
||||
└── src/views/ 状态总览 / 连接配置(画布)/ 集群 / 设置 / 账号与密钥
|
||||
└── src/views/ 状态总览 / 连接配置(画布)/ 集群(含负载均衡开关)/ 设置 / 账号与密钥
|
||||
```
|
||||
|
||||
## API
|
||||
@ -220,6 +241,8 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
||||
| | GET | `/binary/status` | read | frpc 路径 |
|
||||
| | POST | `/binary/install` | write | 安装 frpc |
|
||||
| **集群** | GET | `/cluster/ring` | read | 令牌环快照 |
|
||||
| | GET | `/cluster/balance` | read | 本节点负载均衡视图(只读,不变更状态) |
|
||||
| | POST | `/cluster/balance/mode` | write | 切换本节点负载均衡模式(持久化) |
|
||||
| | POST | `/cluster/create` | write | 创建独立集群 |
|
||||
| | POST | `/cluster/join-ring` | write | 加入集群(本节点发起) |
|
||||
| | POST | `/cluster/join` | write | 接收新节点加入 |
|
||||
@ -254,8 +277,45 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
||||
- **任务分发**:在任意节点画布上配置 `localOnly=false` 的转发并保存 → 画布差异判断生成命令 → 令牌携带 → 负载最低节点摘取 → 拉起 frpc worker
|
||||
- **容错**:节点宕机 → 自动检测 → 环自愈(跳过死亡节点 / leader failover);重启 → 自动重联
|
||||
- **退出集群**:在集群页「退出集群」→ 本节点脱离为 standalone,不自动重联
|
||||
- **负载均衡**:见下节
|
||||
|
||||
集群页展示:环拓扑(self/leader 标记)、令牌轮次、节点负载、待办任务、活跃转发拓扑(含 owner 归属)、增量事件日志。
|
||||
集群页展示:环拓扑(self/leader 标记)、令牌轮次、节点负载、待办任务、活跃转发拓扑(含 owner 归属)、负载均衡(本机份额与释放计划)、增量事件日志。
|
||||
|
||||
### 负载均衡(自导式释放)
|
||||
|
||||
**为什么不是 leader 统一调度**:一节点的负载信号是它持有的转发数,而这个数只在
|
||||
**它自己的拓扑条目消失**时才下降 —— 条目是本地簿记。所以 leader 从外面只能*看见*
|
||||
失衡,无法*消除*它;要消除就得伸手改另一个节点的簿记,而这正是这个环到处都在避免
|
||||
的跨节点写(撤销、重启同样都是定向往 owner)。
|
||||
|
||||
于是规则由每个节点对自己的转发本地执行:
|
||||
|
||||
```
|
||||
当 持有转发数 >= 2 且 持有转发数 > ceil(总转发数 / 存活节点数)
|
||||
则 撤销其中一条 → 删除条目 → 以启用状态重新发布到环 → 由最闲节点认领
|
||||
```
|
||||
|
||||
- **自限**:撤到平摊值即停。3 条分 3 台时,持有 2 条的那台卸一条后变成 1 条,
|
||||
`1 > 1` 不成立 → 停手。没有这个上界,owner 会把转发全部卸光。
|
||||
- **不会卸空**:低于下限 2 条永不撤销。
|
||||
- **不碰已停用的转发**:停用条目的 entry 是用户「停用」决定的载体,释放它等于把
|
||||
一条**已停**的转发当活干交给别人。
|
||||
- **释放 ≠ 停止**:用户停止保留条目并标记链路为停用;均衡释放**删除条目**且链路保持
|
||||
启用。两者不能混用 —— 复用停止的话,每个节点的重复认领守卫都会拒绝这条已是「有主」
|
||||
的转发,它会被永久搁死在刚放弃它的那台机器上。
|
||||
|
||||
三种模式(集群页每节点一个开关,也可用 `W4F_BALANCE_MODE` 设默认值):
|
||||
|
||||
| 模式 | 行为 |
|
||||
| --- | --- |
|
||||
| `off` | 不参与均衡 |
|
||||
| `dry-run` | 只报告「将释放哪条」,不动任何东西(默认) |
|
||||
| `on` | 真正释放 |
|
||||
|
||||
只读接口 `GET /cluster/balance` 返回本节点视角:`selfForwards` / `share` /
|
||||
`totalForwards` / `aliveNodes` / `selfOverShare`、`wouldRelease`、`releasing`、
|
||||
`blockedBy`(为何暂不动的机器原因)、`mode` / `modeSource` / `canControl`。
|
||||
该接口**不改变任何状态** —— 反复轮询不会启动、延迟或取消一次释放。
|
||||
|
||||
## 测试
|
||||
|
||||
@ -263,6 +323,9 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
||||
# 后端
|
||||
go test ./...
|
||||
|
||||
# 单包(负载均衡规则与状态机)
|
||||
go test ./internal/cluster/ -run TestShare -v
|
||||
|
||||
# 前端类型检查
|
||||
cd web && npm run type-check
|
||||
cd web && npm run build
|
||||
```
|
||||
|
||||
@ -15,6 +15,7 @@ import (
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
@ -175,13 +176,41 @@ func main() {
|
||||
return err
|
||||
}
|
||||
// A re-claim after a forward-centric stop leaves the link flagged
|
||||
// disabled (by RevokeFn); clear it so renderRemote renders the
|
||||
// proxy back in. No-op for a fresh claim.
|
||||
_ = st.SetLinkDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, false)
|
||||
// disabled (by RevokeFn). Do NOT clear that flag here: the ring
|
||||
// re-claims automatically (startup reconcile re-spawns any owned
|
||||
// forward whose worker is missing), so clearing it on claim made
|
||||
// "stopped" un-durable — every restart resurrected a forward the user
|
||||
// had explicitly stopped, and it then failed forever against a local
|
||||
// service that was intentionally not running.
|
||||
//
|
||||
// The flag is now owned by the two user-facing actions:
|
||||
// startForward → SetLinkDisabled(false) before submitting the task
|
||||
// stopForward → SetLinkDisabled(true) before revoking it
|
||||
// Claim only READS it to decide whether to bring the worker up. A fresh
|
||||
// claim of a link with no row still starts, because the lookup miss
|
||||
// below is treated as "not disabled".
|
||||
disabled := false
|
||||
if existing, found, err := st.LinkByTriple(tk.Link.Local, tk.Link.Remote, tk.Link.RemotePort); err != nil {
|
||||
return err
|
||||
} else if found {
|
||||
disabled = existing.Disabled
|
||||
}
|
||||
// Start (or restart) the per-forward worker for exactly this link.
|
||||
// Each forward has its own frpc process (keyed by the forward
|
||||
// triple); restarting only this key leaves sibling forwards'
|
||||
// processes untouched.
|
||||
if disabled {
|
||||
// Mark the entry stopped rather than deleting it. The entry is the
|
||||
// carrier that keeps the flag travelling around the ring, and
|
||||
// UpdateTopologyDisabled also clears Active — which is what makes
|
||||
// this entry inert for OfflineReassign() and the startup
|
||||
// reconcile, so it is not resurrected later.
|
||||
if ring != nil {
|
||||
ring.UpdateTopologyDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, true)
|
||||
}
|
||||
log.Printf("ring[%s] claim %s skipped: %s→%s:%d is disabled", selfID, tk.ID, tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
return nil
|
||||
}
|
||||
if tk.Remote.Enabled {
|
||||
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
if _, has := pm.Status(key); has {
|
||||
@ -201,13 +230,118 @@ func main() {
|
||||
// stop/start: it wiped sibling forwards sharing the local or remote.
|
||||
RevokeFn: func(ctx context.Context, tk *cluster.Task) error {
|
||||
_ = st.SetLinkDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, true)
|
||||
// The links table is node-local: only the node that received the
|
||||
// stop request has the flag persisted, while the node that OWNS the
|
||||
// forward is the one holding the worker (and is often a different
|
||||
// machine). Trusting only the local row meant the owner's copy could
|
||||
// still read disabled=false. The task now carries the flag, so use it
|
||||
// as the authority and write it here.
|
||||
ln, found, err := st.LinkByTriple(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
switch {
|
||||
case err != nil:
|
||||
return err
|
||||
case found:
|
||||
ln.Disabled = true
|
||||
// Persist via the same rewrite path the canvas uses, so the flag
|
||||
// survives a restart on THIS node too.
|
||||
links, err := st.ListLinks()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range links {
|
||||
if links[i].Local == ln.Local && links[i].Remote == ln.Remote && links[i].RemotePort == ln.RemotePort {
|
||||
links[i].Disabled = true
|
||||
}
|
||||
}
|
||||
if err := st.ReplaceLinks(links); err != nil {
|
||||
return err
|
||||
}
|
||||
case tk.Link.Disabled:
|
||||
// No local row yet (this node never claimed the forward) but the
|
||||
// task asserts the stop — record it so a later claim on this node
|
||||
// cannot resurrect the forward.
|
||||
loc, rem := tk.Local, tk.Remote
|
||||
links, err := st.ListLinks()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
links = append(links, store.Link{
|
||||
Local: loc.Name, Remote: rem.Name, RemotePort: tk.Link.RemotePort,
|
||||
OffsetX: tk.Link.OffsetX, OffsetY: tk.Link.OffsetY,
|
||||
Group: tk.Link.Group, Disabled: true,
|
||||
})
|
||||
if err := st.ReplaceLinks(links); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
if _, running := pm.Status(key); running {
|
||||
_ = pm.Stop(key)
|
||||
}
|
||||
// Mark the entry stopped instead of deleting it: the entry carries
|
||||
// the flag around the ring, and clearing Active keeps it inert for
|
||||
// the rebalancing paths (OfflineReassign skips inactive entries, and
|
||||
// the startup reconcile skips disabled forwards), so the worker is
|
||||
// not re-spawned on the next restart. Deleting it here is what used
|
||||
// to leave the stop with no carrier at all.
|
||||
if ring != nil {
|
||||
ring.UpdateTopologyDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, true)
|
||||
}
|
||||
log.Printf("ring[%s] revoked task %s: %s→%s:%d", selfID, tk.ID, tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
return nil
|
||||
},
|
||||
// ReleaseFn stops this forward's worker for a REBALANCE, leaving the
|
||||
// link ENABLED. This must not reuse RevokeFn: that marks the link
|
||||
// stopped, and the released forward is about to run on another node —
|
||||
// a "stopped" flag would make the claim path treat the re-published
|
||||
// task as a disabled forward and refuse to start a worker for it,
|
||||
// leaving the forward dead everywhere.
|
||||
ReleaseFn: func(ctx context.Context, tk *cluster.Task) error {
|
||||
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
if _, running := pm.Status(key); running {
|
||||
_ = pm.Stop(key)
|
||||
}
|
||||
log.Printf("ring[%s] released worker %s (rebalancing): %s→%s:%d", selfID, key,
|
||||
tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
return nil
|
||||
},
|
||||
// RestartFn brings an already-claimed forward back after a stop. It
|
||||
// must NOT run the claim bookkeeping (ClaimFn), which appends links and
|
||||
// writes topology state: the forward is already established, and a stop
|
||||
// deliberately keeps its topology entry, so all that is needed is to
|
||||
// clear the flag and respawn the worker. The engine has already
|
||||
// re-marked the topology entry enabled before calling this.
|
||||
RestartFn: func(ctx context.Context, tk *cluster.Task) error {
|
||||
// Clear the stop on this node's own store, using the same path the
|
||||
// start handler uses, so the local copy cannot disagree with the
|
||||
// ring and block a later restart.
|
||||
if ln, found, err := st.LinkByTriple(tk.Link.Local, tk.Link.Remote, tk.Link.RemotePort); err != nil {
|
||||
return err
|
||||
} else if found && ln.Disabled {
|
||||
links, err := st.ListLinks()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range links {
|
||||
if links[i].Local == ln.Local && links[i].Remote == ln.Remote && links[i].RemotePort == ln.RemotePort {
|
||||
links[i].Disabled = false
|
||||
}
|
||||
}
|
||||
if err := st.ReplaceLinks(links); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if tk.Remote.Enabled {
|
||||
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
if _, has := pm.Status(key); has {
|
||||
_ = pm.Restart(key)
|
||||
} else {
|
||||
_ = pm.Start(key)
|
||||
}
|
||||
}
|
||||
log.Printf("ring[%s] restarted %s: %s→%s:%d", selfID, tk.ID, tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||
return nil
|
||||
},
|
||||
},
|
||||
ringTransport.SendTo(func(nodeID string) string {
|
||||
if ring == nil {
|
||||
@ -232,10 +366,24 @@ func main() {
|
||||
// left does NOT auto-rejoin.
|
||||
ring.SetPeerPersist(func(peersJSON string) error { return st.SetClusterPeers(peersJSON) })
|
||||
|
||||
// Re-apply local store group labels onto the ring topology after each
|
||||
// state adoption. Without this, group changes made via HTTP handlers
|
||||
// Re-apply local store state onto the ring topology after each state
|
||||
// adoption, and learn the ring's disabled decisions back into the store.
|
||||
//
|
||||
// Without the group half, group changes made via HTTP handlers
|
||||
// (POST /forwards/assign) are overwritten by the next e.state = tk.State
|
||||
// and never propagate to other nodes.
|
||||
// and never propagate.
|
||||
//
|
||||
// The disabled half makes the RING authoritative for stop/start. That is the
|
||||
// only source all members can agree on: the links table is a per-node copy, so
|
||||
// treating it as authoritative let two nodes disagree indefinitely (observed
|
||||
// live: one node reading disabled=1 while the owner still read 0 and kept its
|
||||
// worker running). This is a write-through, not a listener — it runs on every
|
||||
// adoption, so any peer's decision lands here within a token cycle.
|
||||
//
|
||||
// Ordering matters in one direction only: the engine has already re-applied
|
||||
// this node's not-yet-confirmed local decisions onto the adopted state (see
|
||||
// reconcileLocalDisabled), so reading the ring here can never clobber an
|
||||
// action the user just took on this node.
|
||||
ring.SetTopologySync(func() {
|
||||
links, err := st.ListLinks()
|
||||
if err != nil {
|
||||
@ -244,8 +392,23 @@ func main() {
|
||||
for _, ln := range links {
|
||||
ring.UpdateTopologyGroup(ln.Local, ln.Remote, ln.RemotePort, ln.Group)
|
||||
}
|
||||
for _, te := range ring.State().TopologyList() {
|
||||
disabled, known := ring.TopologyDisabled(te.Local.Name, te.Remote.Name, te.Link.RemotePort)
|
||||
if !known {
|
||||
continue
|
||||
}
|
||||
if ln, found, err := st.LinkByTriple(te.Local.Name, te.Remote.Name, te.Link.RemotePort); err == nil && found && ln.Disabled == disabled {
|
||||
continue // store already agrees with the ring
|
||||
}
|
||||
_ = st.ReconcileLinkDisabled(te.Local.Name, te.Remote.Name, te.Link.RemotePort, disabled)
|
||||
}
|
||||
})
|
||||
|
||||
// Load balancer is created here (before the Handler) so the HTTP layer can
|
||||
// expose its read-only report. Mode/parameters are applied further down, once
|
||||
// the W4F_BALANCE_* environment has been parsed.
|
||||
balancer := cluster.NewBalancer(ring, 600)
|
||||
|
||||
h := &httpapi.Handler{
|
||||
Store: st,
|
||||
Process: pm,
|
||||
@ -254,6 +417,9 @@ func main() {
|
||||
Sessions: httpapi.NewSessionStore(),
|
||||
Cluster: reg,
|
||||
Ring: ring,
|
||||
// Balancer is created above, before the environment that configures it is
|
||||
// parsed, because the HTTP layer needs the pointer.
|
||||
Balancer: balancer,
|
||||
// SelfAddr must be reachable by peers. 0.0.0.0/empty on a real host
|
||||
// would break peer reconnect, so fall back to the hostname (container
|
||||
// name inside docker is a routable addr on the compose network).
|
||||
@ -411,6 +577,65 @@ func main() {
|
||||
go ring.WatchTokenLoss(clusterCtx)
|
||||
go ring.WatchLeader(clusterCtx)
|
||||
|
||||
// Load balancing — SELF-DIRECTED, and dry-run by default.
|
||||
//
|
||||
// Every node runs this check against its OWN forwards. There is no
|
||||
// coordinator: a node's load only actually falls when its own topology entry
|
||||
// goes away, so only the owner can produce the state change that makes the
|
||||
// ring hand a forward to somebody else. The node gives a forward up and
|
||||
// re-publishes it as an ordinary unclaimed task; the ring's normal
|
||||
// lowest-load claim then places it. (The earlier design had the leader pick a
|
||||
// victim and a destination — but it can only see the imbalance, not resolve
|
||||
// it, so it had to reach into another node's bookkeeping to make its decision
|
||||
// true.)
|
||||
//
|
||||
// A release stops a worker and drops a live connection, so acting on the ring
|
||||
// is opt-in:
|
||||
//
|
||||
// W4F_BALANCE_MODE=dry-run report the decision, release nothing (default)
|
||||
// W4F_BALANCE_MODE=on actually release
|
||||
// W4F_BALANCE_MODE=off do nothing at all
|
||||
// W4F_BALANCE_COOLDOWN=600 min seconds between two releases by one node
|
||||
// W4F_BALANCE_INTERVAL=10 check every N seconds
|
||||
balMode := strings.ToLower(strings.TrimSpace(os.Getenv("W4F_BALANCE_MODE")))
|
||||
if balMode == "" {
|
||||
balMode = cluster.BalanceDryRun
|
||||
}
|
||||
balInterval := time.Duration(envFloat("W4F_BALANCE_INTERVAL", 10)) * time.Second
|
||||
if balInterval < time.Second {
|
||||
balInterval = 10 * time.Second
|
||||
}
|
||||
balCooldown := int64(envFloat("W4F_BALANCE_COOLDOWN", 600))
|
||||
balancer.SetCooldown(balCooldown)
|
||||
balancer.SetMode(balMode)
|
||||
// A mode stored from the cluster page wins over the environment, so flipping
|
||||
// the switch is not silently reverted by the next restart. SetMode is applied
|
||||
// first so a stored value only has to be a no-op when the two agree.
|
||||
balancer.SetModePersistence(
|
||||
func() (string, bool) {
|
||||
m, ok, err := st.BalanceMode()
|
||||
if err != nil {
|
||||
log.Printf("ring[%s] read stored balance mode: %v", selfID, err)
|
||||
return "", false
|
||||
}
|
||||
return m, ok
|
||||
},
|
||||
func(m string) error {
|
||||
s, err := st.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.BalanceMode = m
|
||||
return st.UpdateSettings(s)
|
||||
},
|
||||
)
|
||||
// Report the EFFECTIVE mode, not balMode: a stored value may have overridden
|
||||
// the environment, and a startup line naming the wrong mode is exactly how an
|
||||
// operator ends up believing releasing is off while it is on.
|
||||
log.Printf("ring[%s] load balancing %s (source=%s cooldown=%ds interval=%s, self-directed)",
|
||||
selfID, strings.ToUpper(balancer.Mode()), balancer.ModeSource(), balCooldown, balInterval)
|
||||
go balancer.Watch(clusterCtx, balInterval)
|
||||
|
||||
// Startup reconcile: every node (leader or not) checks its own topology
|
||||
// entries and re-claims any whose per-forward frpc worker process is
|
||||
// absent — the topology was recovered via AdoptState but the worker
|
||||
@ -419,17 +644,32 @@ func main() {
|
||||
// LocalOnly forwards are never in the topology and are handled by the
|
||||
// auto-start loop above.
|
||||
go func() {
|
||||
for {
|
||||
if ring.IsLeader() {
|
||||
break
|
||||
}
|
||||
// No leader gate here. The reconcile body is per-NODE work: it re-claims
|
||||
// the forwards THIS node owns, whose worker processes died with the
|
||||
// restart. Every member owns forwards (a 3-node ring splits them), so
|
||||
// gating on leadership meant non-leaders waited forever and never
|
||||
// rebuilt their own workers — a node could come back up "in the cluster"
|
||||
// while owning a forward with no process behind it. Observed live: after a
|
||||
// rolling restart a non-leader rejoined, adopted state listing itself as
|
||||
// the owner of a forward, and never started it, because it sat waiting to
|
||||
// become leader.
|
||||
//
|
||||
// What actually needs settling is the ring, not the leadership: wait for
|
||||
// the topology to show us (with a bounded fallback so a lone node that
|
||||
// rejoined a ring whose leader is unreachable still makes progress).
|
||||
deadline := time.After(30 * time.Second)
|
||||
for !ring.HasSelfInTopology() {
|
||||
select {
|
||||
case <-time.After(2 * time.Second):
|
||||
case <-deadline:
|
||||
log.Printf("ring[%s] startup reconcile: self not in topology after 30s, proceeding anyway", ring.ID)
|
||||
case <-time.After(500 * time.Millisecond):
|
||||
continue
|
||||
case <-clusterCtx.Done():
|
||||
return
|
||||
}
|
||||
break
|
||||
}
|
||||
// Let the ring stabilize after leader election / rejoin.
|
||||
// Let the ring stabilize after election / rejoin before acting on it.
|
||||
time.Sleep(3 * time.Second)
|
||||
snap := ring.Snapshot()
|
||||
reclaimed := 0
|
||||
@ -437,6 +677,16 @@ func main() {
|
||||
if t.OwnerID != selfID {
|
||||
continue
|
||||
}
|
||||
// A forward the user stopped must stay stopped: skip it here so a
|
||||
// restart does not re-spawn its worker. The ClaimFn enforces the
|
||||
// same rule (belt and braces — this also avoids a pointless
|
||||
// claim→skip round trip per disabled forward on every boot).
|
||||
if ln, found, err := st.LinkByTriple(t.Local.Name, t.Remote.Name, t.Link.RemotePort); err != nil {
|
||||
log.Printf("ring[%s] reconcile: lookup %s→%s:%d: %v", ring.ID, t.Local.Name, t.Remote.Name, t.Link.RemotePort, err)
|
||||
continue
|
||||
} else if found && ln.Disabled {
|
||||
continue
|
||||
}
|
||||
key := process.WorkerKey(t.Local.Name, t.Remote.Name, t.Link.RemotePort)
|
||||
if _, has := pm.Status(key); has {
|
||||
continue // worker already running
|
||||
@ -568,6 +818,22 @@ func retryJoinCluster(ctx context.Context, ring *cluster.Engine, targetAddr, joi
|
||||
}
|
||||
}
|
||||
|
||||
// envFloat reads a numeric environment variable, returning def when unset or
|
||||
// unparseable. A malformed value must not silently change behaviour, so it falls
|
||||
// back to the documented default rather than 0.
|
||||
func envFloat(key string, def float64) float64 {
|
||||
v := strings.TrimSpace(os.Getenv(key))
|
||||
if v == "" {
|
||||
return def
|
||||
}
|
||||
f, err := strconv.ParseFloat(v, 64)
|
||||
if err != nil {
|
||||
log.Printf("env %s=%q is not a number; using %v", key, v, def)
|
||||
return def
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// retryRejoinCached is the crash/restart recovery path: parse the persisted
|
||||
// peer list (JSON of [{addr,key},...]) and try to rejoin via each peer in
|
||||
// order. The first peer that accepts wins. Retry the whole list every 10s
|
||||
@ -600,6 +866,20 @@ func retryRejoinCached(ctx context.Context, ring *cluster.Engine, peersJSON stri
|
||||
cancel()
|
||||
if err == nil {
|
||||
log.Printf("ring[%s] rejoined cluster via cached peer %s", ring.ID, p.Addr)
|
||||
// A rejoined node must be able to drive the ring itself. The
|
||||
// normal case is that the existing leader keeps circulating the
|
||||
// token, but if WE are the leader (typical after a rolling
|
||||
// restart where the leader came back last, or the leader died
|
||||
// while we were down) nobody else will: StartRing is a no-op on
|
||||
// non-leaders, and the ring sits still with every task stranded
|
||||
// in pending. Observed live: the leader restarted, rejoined fine,
|
||||
// and then never forwarded a token — cycle frozen, topology
|
||||
// empty, pending=3.
|
||||
//
|
||||
// StartRing is safe to call unconditionally: it returns
|
||||
// immediately unless this node leads, and it throttles itself
|
||||
// against RoundDelay so a redundant call cannot double-send.
|
||||
go ring.StartRing(ctx)
|
||||
return
|
||||
}
|
||||
log.Printf("ring[%s] rejoin %s failed: %v", ring.ID, p.Addr, err)
|
||||
|
||||
145
cmd/webui4frpc/rejoin_ring_test.go
Normal file
145
cmd/webui4frpc/rejoin_ring_test.go
Normal file
@ -0,0 +1,145 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"webui4frpc/internal/cluster"
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// fakeJoinPeer stands in for a healthy cluster member: it answers the rejoin
|
||||
// POST and records it, so the rejoin path can be exercised end to end.
|
||||
type fakeJoinPeer struct {
|
||||
mu sync.Mutex
|
||||
joined int
|
||||
stateFn func() cluster.State
|
||||
}
|
||||
|
||||
func (p *fakeJoinPeer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
p.mu.Lock()
|
||||
p.joined++
|
||||
p.mu.Unlock()
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"state": p.stateFn()})
|
||||
}
|
||||
|
||||
func (p *fakeJoinPeer) joins() int {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
return p.joined
|
||||
}
|
||||
|
||||
// TestRejoinThenStartRingCoversTheRejoinPath is the regression test for the
|
||||
// frozen ring, and it deliberately drives retryRejoinCached itself rather than
|
||||
// calling StartRing directly — a test that calls the fix's own helper proves
|
||||
// nothing about whether the fix is wired up (verified: such a test stayed green
|
||||
// with the fix reverted).
|
||||
//
|
||||
// The failure, observed live on the three-node cluster: the leader restarted,
|
||||
// rejoined successfully, logged "startup reconcile: reclaimed=0 submitted=3",
|
||||
// and then never forwarded a token. cycle froze, topology stayed empty and the
|
||||
// three forwards sat in pending forever, because rejoin returned immediately
|
||||
// and only a leader may call StartRing.
|
||||
func TestRejoinThenStartRingCoversTheRejoinPath(t *testing.T) {
|
||||
selfID := "192.168.2.30:7500"
|
||||
|
||||
// The peer hands back a ring in which SELF is the leader — the situation a
|
||||
// rolling restart produces when the leader comes back last.
|
||||
var peerState cluster.State
|
||||
peer := &fakeJoinPeer{stateFn: func() cluster.State { return peerState }}
|
||||
|
||||
// Bind on loopback so the engine's hardcoded peer address is reachable.
|
||||
// We therefore advertise 127.0.0.1 in the ring while keeping the same IDs.
|
||||
ln := httptest.NewServer(peer)
|
||||
defer ln.Close()
|
||||
|
||||
peerState = cluster.State{
|
||||
LeaderID: selfID,
|
||||
Nodes: []cluster.Node{
|
||||
{ID: selfID, Addr: selfID, Alive: true, IsLeader: true},
|
||||
{ID: "peer:7500", Addr: "peer:7500", Alive: true},
|
||||
},
|
||||
Topology: map[string]*cluster.TopoEntry{},
|
||||
PendingTasks: map[string]*cluster.Task{
|
||||
"t1": {ID: "t1", Local: store.Local{Name: "web"},
|
||||
Remote: store.Remote{Name: "frps1", IP: "1.2.3.4", Port: 7000, Enabled: true},
|
||||
Link: store.Link{RemotePort: 18081}},
|
||||
},
|
||||
}
|
||||
|
||||
// Redirect the peer's advertised address at the transport level: the engine
|
||||
// dials whatever the ring says, so give the ring the test server's address.
|
||||
// JoinRing builds "http://<addr>/api/manager/cluster/join" itself, so the
|
||||
// cached entry must be a bare host:port, not a URL.
|
||||
realAddr := strings.TrimPrefix(ln.URL, "http://")
|
||||
for i := range peerState.Nodes {
|
||||
if peerState.Nodes[i].ID == "peer:7500" {
|
||||
peerState.Nodes[i].Addr = realAddr
|
||||
}
|
||||
}
|
||||
|
||||
var mu sync.Mutex
|
||||
var emitted int
|
||||
eng := cluster.NewEngine(selfID, selfID, "u", "p", "0.1.2", nil,
|
||||
&cluster.AppHandler{ClaimFn: func(ctx context.Context, tk *cluster.Task) error { return nil }},
|
||||
func(ctx context.Context, next string, tk *cluster.Token) error {
|
||||
if tk != nil {
|
||||
mu.Lock()
|
||||
emitted++
|
||||
mu.Unlock()
|
||||
}
|
||||
return nil
|
||||
},
|
||||
selfID, true, "")
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
// The rejoin path, as main() runs it. Cached peers point at the test server.
|
||||
peersJSON, err := json.Marshal([]map[string]string{{"addr": realAddr, "key": "peerkey"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
retryRejoinCached(ctx, eng, string(peersJSON))
|
||||
}()
|
||||
|
||||
// retryRejoinCached returns once rejoin succeeds; give the goroutine a moment.
|
||||
for i := 0; i < 100; i++ {
|
||||
if peer.joins() > 0 {
|
||||
break
|
||||
}
|
||||
sleepShort()
|
||||
}
|
||||
if peer.joins() == 0 {
|
||||
t.Fatal("the rejoin never reached the peer — the test setup is wrong")
|
||||
}
|
||||
<-done
|
||||
|
||||
// Rejoin succeeded. The engine must now consider itself the leader...
|
||||
if !eng.IsLeader() {
|
||||
t.Fatalf("precondition broken: rejoined node does not consider itself leader")
|
||||
}
|
||||
sleepShort()
|
||||
|
||||
// ...and it must have restarted the ring: a token has to be emitted, or the
|
||||
// cluster sits frozen with its forwards stranded in pending.
|
||||
mu.Lock()
|
||||
got := emitted
|
||||
mu.Unlock()
|
||||
if got == 0 {
|
||||
t.Fatal("after a successful rejoin the leader emitted NO token: the ring is frozen " +
|
||||
"and pending forwards are never claimed (the live failure this guards)")
|
||||
}
|
||||
}
|
||||
func sleepShort() { time.Sleep(10 * time.Millisecond) }
|
||||
199
docs/how-it-works.md
Normal file
199
docs/how-it-works.md
Normal file
@ -0,0 +1,199 @@
|
||||
# webui4frpc 内部机制(虚拟 frps 实证)
|
||||
|
||||
> 本文所有结论均由**隔离的虚拟 frps 环境**实测得出,不依赖生产集群。
|
||||
> 复现脚本见 `scripts/probe-with-virtual-frps.sh`。
|
||||
>
|
||||
> 探测时间:2026-10-02 · frp 版本 0.61.1 · 二进制 md5 三机一致(5c4c23d5…)
|
||||
|
||||
---
|
||||
|
||||
## 1. 数据流:DB 是唯一真源,configs/ 是派生产物
|
||||
|
||||
```
|
||||
API (HTTP)
|
||||
→ manager.db (SQLite) ← 真源
|
||||
↓ Render(forward) 渲染
|
||||
→ data/configs/<name>.json ← 派生产物
|
||||
↓ exec: frpc -c <cfg>
|
||||
→ frpc worker 进程
|
||||
```
|
||||
|
||||
**推论(已在生产验证)**:手改 `configs/*.json` 会被下一次 DB 写回覆盖。
|
||||
配置变更必须走 API(`PUT /api/manager/canvas` 或 `/remotes`)。
|
||||
|
||||
`configs/` 目录的文件名格式为 `<local>~<remote>~<remotePort>.json`。
|
||||
只有**实跑的那一份**会被 frpc `-c` 引用;同名不同端口的历史文件会残留。
|
||||
|
||||
---
|
||||
|
||||
## 2. 集群不复制配置,只复制环状态
|
||||
|
||||
`internal/cluster/token_transport.go` 在环上传的东西:
|
||||
|
||||
```go
|
||||
type Token struct { // ring.go:106
|
||||
Cycle int64
|
||||
State State // 节点拓扑 / leader / log delta
|
||||
Log []LogEntry
|
||||
}
|
||||
```
|
||||
|
||||
这是一个**领导者选举 + 拓扑状态**的载体,**不包含** remotes / locals / links。
|
||||
`sessions are not replicated across the ring`(`session.go:32`)——
|
||||
每个节点有自己的 `manager.db`。
|
||||
|
||||
**推论**:
|
||||
- 令牌环 peer 认证用的是 webui Basic 凭据(`SetBasicAuth(User, Pass)`),
|
||||
与 frps `auth.token` 是两个独立密钥
|
||||
|
||||
### 配置为什么会同步到新节点
|
||||
|
||||
`Token.State` 里除了环状态,还带 `Topology map[string]*TopoEntry`,
|
||||
而 **`TopoEntry` 携带完整配置**(`ring.go:76`):
|
||||
|
||||
```go
|
||||
type TopoEntry struct {
|
||||
TaskID string
|
||||
OwnerID string
|
||||
Local store.Local // 完整 local 配置
|
||||
Remote store.Remote // 完整 remote 配置(含 token)
|
||||
Link store.Link
|
||||
Active bool
|
||||
}
|
||||
```
|
||||
|
||||
owner 节点 claim 任务时,配置**直接写进本地 DB**(`cmd/webui4frpc/main.go:147`):
|
||||
|
||||
```go
|
||||
ClaimFn: func(ctx, tk) error {
|
||||
st.UpsertLocal(tk.Local)
|
||||
st.UpsertRemote(tk.Remote)
|
||||
... st.ReplaceLinks(links) // 去重后替换
|
||||
// 然后 spawn worker
|
||||
}
|
||||
```
|
||||
|
||||
读侧对应 `handleCanvasGet`(`handlers.go:22`)的合并:把拓扑里本节点没有的
|
||||
local/remote/link 补进画布,让每台都看到**全集群视图**。这是 read-time merge,
|
||||
**不回写 SQLite**(避免触发 per-forward stop/start)。
|
||||
|
||||
### 同步边界(重要)
|
||||
|
||||
| 内容 | 是否同步 | 机制 |
|
||||
|---|---|---|
|
||||
| **已建立 forward 的配置** | ✅ | 随 `Topology` 环内流动,owner claim 时落库 |
|
||||
| 环状态(leader / 节点 / log delta) | ✅ | `Token.State` |
|
||||
| remote 定义变更(改 token 等) | ⚠️ 走任务 | 不是 DB 快照复制 |
|
||||
| 仅有定义、从未成为 active forward 的配置 | ❌ | 没有 `TopoEntry` 就没有载体 |
|
||||
| WebUI session | ❌ | `session.go:32` 明确不复制 |
|
||||
|
||||
**推论**:
|
||||
- "在一台改、集群自动同步"对**已建立的 forward** 成立——靠 `Topology` + `ClaimFn`
|
||||
- 改 frps remote 的 token 会随任务下发到 owner,不必逐台手改
|
||||
- 但只存在于某台 DB、且无对应 active forward 的配置不会自动过去
|
||||
- 这也是本项目 frps token 轮换的正确路径:走 API 提交任务,而非逐台改文件
|
||||
|
||||
---
|
||||
|
||||
## 3. forward 的 owner 归属:非 localOnly 由 owner 拉起
|
||||
|
||||
`startForward`(`handlers_forwards.go:62`)的两条分支:
|
||||
|
||||
| forward 类型 | 谁执行 | 机制 |
|
||||
|---|---|---|
|
||||
| `localOnly = true` | **本节点直接** `Process.Start(key)` | 无需环 |
|
||||
| `localOnly = false` | **拓扑 owner 节点** | `Ring.SubmitTask` / `UpdateTopologyDisabled` 投递 |
|
||||
|
||||
非 localOnly 分支有一段注释解释了为什么需要专门的 RESTART 任务:
|
||||
|
||||
> - `SubmitTask` is idempotency-guarded, and the entry still exists … so the submission is deduped away
|
||||
> - the claim path drops any task for a forward that already has an owner
|
||||
>
|
||||
> Without it a forward the user can stop but not restart
|
||||
|
||||
**实测观察**:单节点 fresh 集群(`cycle: 0`、无 peer)下,
|
||||
`POST /forwards/start` 对非 localOnly forward 返回 `{"ok":true}`,
|
||||
但 `configs/` 为空、`connState=not_started`——环不转时任务无人 claim。
|
||||
单节点环境应使用 `localOnly: true` 的 forward,或加入多节点集群。
|
||||
**这是设计行为,不是缺陷。**
|
||||
|
||||
---
|
||||
|
||||
## 4. 三个易踩的坑
|
||||
|
||||
### 4.1 `binaryPath` 为空 → worker 静默不启动
|
||||
|
||||
`main.go:89` 的解析顺序:
|
||||
|
||||
```go
|
||||
BinaryPath: func() string {
|
||||
if s, _ := st.Settings(); s.BinaryPath != "" { return s.BinaryPath }
|
||||
return fallbackBin
|
||||
}
|
||||
```
|
||||
|
||||
全新 workdir(未装过 frpc)`binaryPath` 为空,回退到 `fallbackBin`。
|
||||
若该路径也不存在,`spawn()` 报错,但 **`POST /forwards/start` 仍返回 `{"ok":true}`**
|
||||
(错误只落在 worker 日志里)。排障时先查:
|
||||
|
||||
```bash
|
||||
GET /api/manager/binary/status # {"binaryPath": "..."}
|
||||
```
|
||||
|
||||
### 4.2 `UpdateSettings` 是全量替换
|
||||
|
||||
`store.go:754` 的 `UpdateSettings` 直接 `json.Marshal(st)` 覆盖整行。
|
||||
POST 一个**不含** `binaryPath` 的 settings JSON,会把该字段**清空**。
|
||||
改 settings 前必须先 `GET` 现有值再整体回传。
|
||||
|
||||
### 4.3 画布会自动改写 loopback 地址
|
||||
|
||||
`applyCanvas`(`handlers.go:108`)对**非 localOnly** 的 local:
|
||||
|
||||
```go
|
||||
if !canvas.Locals[i].LocalOnly && cluster.IsLoopbackIP(canvas.Locals[i].IP) {
|
||||
canvas.Locals[i].IP = cluster.RewriteForCluster(canvas.Locals[i].IP)
|
||||
}
|
||||
```
|
||||
|
||||
实测:`PUT /canvas` 传 `127.0.0.1` → 落库变成 `192.168.2.60`(本机 LAN 地址)。
|
||||
这是为了保证其它节点能连到该 local。**localOnly 的 local 保持 loopback 不变。**
|
||||
|
||||
---
|
||||
|
||||
## 5. 启动凭据来自命令行 flag
|
||||
|
||||
`main.go:64` 每次启动都 `SyncSystemUser(*user, *pass)`,把 `-user/-password`
|
||||
同步进 `users` 表(`system=1`)。不传 flag 时用内置默认值,
|
||||
`POST /api/manager/login` 需要 JSON body(`{"username","password"}`),
|
||||
拿到身份后**不返回 token**——后续用 HTTP Basic。
|
||||
|
||||
---
|
||||
|
||||
## 6. API 速查(实测路径)
|
||||
|
||||
| 用途 | 方法 | 路径 |
|
||||
|---|---|---|
|
||||
| 登录(返回身份,无 token) | POST | `/api/manager/login` |
|
||||
| 全量保存画布(local+remote+link) | **PUT** | `/api/manager/canvas` |
|
||||
| 读取画布 | GET | `/api/manager/canvas` |
|
||||
| 启/停单条 forward | POST | `/api/manager/forwards/start` `/stop` |
|
||||
| 分组启停 | POST | `/api/manager/forwards/group/start` `/stop` |
|
||||
| 改转发归属 | POST | `/api/manager/forwards/assign` |
|
||||
| 二进制状态 | GET | `/api/manager/binary/status` |
|
||||
| 环拓扑 | GET | `/api/manager/cluster/ring` |
|
||||
| 节点列表 | GET | `/api/manager/cluster/nodes` |
|
||||
|
||||
认证:HTTP Basic(`Authorization: Basic base64(user:pass)`)。
|
||||
`POST` 到 `/canvas` 会返回 `method not allowed`——**必须用 PUT**。
|
||||
|
||||
---
|
||||
|
||||
## 7. 复现
|
||||
|
||||
```bash
|
||||
scripts/probe-with-virtual-frps.sh
|
||||
```
|
||||
|
||||
搭建 frps 0.61.1(`bindPort=17000` / `vhostHTTPPort=17080` / `vhostHTTPsPort=17443`,
|
||||
独立 token),用独立 workdir 拉起 webui4frpc(`:17500`),全流程不触碰生产。
|
||||
728
internal/cluster/balance_test.go
Normal file
728
internal/cluster/balance_test.go
Normal file
@ -0,0 +1,728 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// mustState round-trips the seed through JSON so the engine under test owns its
|
||||
// own copy of Nodes/Topology. AdoptState assigns the state by value, but Nodes is
|
||||
// a slice: handing one seed to several engines would alias one backing array and
|
||||
// make a mutation on one node visible on the others — a test artefact that
|
||||
// previously produced a false green here.
|
||||
func mustState(t *testing.T, s State) *State {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out State
|
||||
if err := json.Unmarshal(b, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e := newTestEngine("probe", true)
|
||||
e.AdoptState(out)
|
||||
return e.State()
|
||||
}
|
||||
|
||||
// migrateRing builds a 3-node ring where n1 owns every forward, plus n2/n3 idle.
|
||||
func migrateRing(t *testing.T, ownCount int) (*Engine, *Engine, *Engine) {
|
||||
t.Helper()
|
||||
mk := func(id string, leader bool) *Engine {
|
||||
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5}}
|
||||
return NewEngine(id, id+":7500", "u", "p", "0.1.3", nil, h,
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||
id+":7500", leader, "")
|
||||
}
|
||||
n1, n2, n3 := mk("n1", true), mk("n2", false), mk("n3", false)
|
||||
|
||||
seed := State{
|
||||
LeaderID: "n1",
|
||||
Nodes: []Node{
|
||||
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true, Load: Load{Forwards: ownCount, MemPct: 5}},
|
||||
{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{MemPct: 5}},
|
||||
{ID: "n3", Addr: "n3:7500", Alive: true, Load: Load{MemPct: 5}},
|
||||
},
|
||||
Topology: map[string]*TopoEntry{},
|
||||
PendingTasks: map[string]*Task{},
|
||||
}
|
||||
for _, e := range []*Engine{n1, n2, n3} {
|
||||
var own State
|
||||
b, _ := json.Marshal(seed)
|
||||
if err := json.Unmarshal(b, &own); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.AdoptState(own)
|
||||
}
|
||||
// n1 claims `ownCount` forwards.
|
||||
for i := 0; i < ownCount; i++ {
|
||||
n1.state.AddTopology(&Task{
|
||||
ID: "f" + string(rune('a'+i)),
|
||||
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||
Link: store.Link{RemotePort: 100 + i},
|
||||
}, "n1")
|
||||
}
|
||||
var snap State
|
||||
b, _ := json.Marshal(n1.state)
|
||||
_ = json.Unmarshal(b, &snap)
|
||||
for _, e := range []*Engine{n1, n2, n3} {
|
||||
e.AdoptState(snap)
|
||||
}
|
||||
return n1, n2, n3
|
||||
}
|
||||
|
||||
// errNotWritable stands in for a failing settings write.
|
||||
var errNotWritable = errors.New("not writable")
|
||||
|
||||
// ---- helpers ----
|
||||
|
||||
// balanceRing builds a 3-node ring with `owned` forwards on n1 (the evaluation
|
||||
// node) and `others` on each of the other two, so a test can set up any
|
||||
// distribution directly. The replicated Nodes[].Load is made consistent with the
|
||||
// topology, because that is what a real token round leaves behind.
|
||||
func balanceRing(t *testing.T, owned, others int) (*Engine, *Engine, *Engine) {
|
||||
t.Helper()
|
||||
mk := func(id string, leader bool) *Engine {
|
||||
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5}}
|
||||
return NewEngine(id, id+":7500", "u", "p", "0.1.3", nil, h,
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||
id+":7500", leader, "")
|
||||
}
|
||||
n1, n2, n3 := mk("n1", true), mk("n2", false), mk("n3", false)
|
||||
|
||||
seed := State{
|
||||
LeaderID: "n1",
|
||||
Nodes: []Node{
|
||||
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true, Load: Load{Forwards: owned, MemPct: 5}},
|
||||
{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{Forwards: others, MemPct: 5}},
|
||||
{ID: "n3", Addr: "n3:7500", Alive: true, Load: Load{Forwards: others, MemPct: 5}},
|
||||
},
|
||||
Topology: map[string]*TopoEntry{},
|
||||
PendingTasks: map[string]*Task{},
|
||||
}
|
||||
for _, e := range []*Engine{n1, n2, n3} {
|
||||
var own State
|
||||
b, _ := json.Marshal(seed)
|
||||
if err := json.Unmarshal(b, &own); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.AdoptState(own)
|
||||
}
|
||||
// Build the whole topology in n1's state, then broadcast it. Adding each
|
||||
// node's forwards to its OWN state and then re-adopting from n1 would discard
|
||||
// the others (one state is the source of the broadcast), which is how an
|
||||
// earlier version of this helper silently produced a 1-1-1 split regardless
|
||||
// of the arguments.
|
||||
add := func(owner, prefix string, count, basePort int) {
|
||||
for i := 0; i < count; i++ {
|
||||
n1.state.AddTopology(&Task{
|
||||
ID: prefix + strconv.Itoa(i),
|
||||
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||
Link: store.Link{RemotePort: basePort + i},
|
||||
}, owner)
|
||||
}
|
||||
}
|
||||
add("n1", "f", owned, 100)
|
||||
add("n2", "g", others, 200)
|
||||
add("n3", "h", others, 300)
|
||||
|
||||
var snap State
|
||||
b, _ := json.Marshal(n1.state)
|
||||
_ = json.Unmarshal(b, &snap)
|
||||
for _, e := range []*Engine{n1, n2, n3} {
|
||||
e.AdoptState(snap)
|
||||
}
|
||||
return n1, n2, n3
|
||||
}
|
||||
|
||||
// ---- the rule ----
|
||||
|
||||
// TestShareIsTheEvenSplit pins the arithmetic, including the ceiling. An even
|
||||
// split of 3 over 3 is 1 (not 1.0 floored to 0): with a floor, a node holding
|
||||
// one of three forwards would still be "over its share" and shed it.
|
||||
func TestShareIsTheEvenSplit(t *testing.T) {
|
||||
for _, tc := range []struct{ total, alive, want int }{
|
||||
{3, 3, 1}, {4, 3, 2}, {6, 2, 3}, {1, 3, 1}, {0, 3, 0}, {7, 3, 3},
|
||||
} {
|
||||
got := (tc.total + tc.alive - 1) / tc.alive
|
||||
if got != tc.want {
|
||||
t.Errorf("share(%d over %d) = %d, want %d", tc.total, tc.alive, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestOwnerReleasesWhenOverItsShare: 3 forwards on n1, none elsewhere → n1 holds
|
||||
// 3 against a share of 1 and must give one up.
|
||||
func TestOwnerReleasesWhenOverItsShare(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
|
||||
pick, p, blocked := b.assess(1000)
|
||||
if pick == nil {
|
||||
t.Fatalf("n1 holds 3 of 3 forwards and should release one (blocked=%q)", blocked)
|
||||
}
|
||||
if blocked != "" {
|
||||
t.Errorf("nothing should block the first release, got %q", blocked)
|
||||
}
|
||||
if p.mine != 3 || p.share != 1 || p.total != 3 || p.alive != 3 {
|
||||
t.Errorf("pool = %+v, want mine=3 share=1 total=3 alive=3", p)
|
||||
}
|
||||
if pick.OwnerID != "n1" {
|
||||
t.Errorf("must release a forward n1 owns, got owner %q", pick.OwnerID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReleasingStopsAtTheShare is the bound that keeps the rule from emptying a
|
||||
// node: after shedding one, n1 holds 1 of 3 over 3 nodes, which is no longer more
|
||||
// than the share, and it must stop.
|
||||
func TestReleasingStopsAtTheShare(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 1, 1) // even 1-1-1 split
|
||||
b := NewBalancer(n1, 0)
|
||||
|
||||
pick, p, blocked := b.assess(1000)
|
||||
if pick != nil {
|
||||
t.Errorf("an even 1-1-1 split must not release anything, got %s:%d (blocked=%q)",
|
||||
pick.Local.Name, pick.Link.RemotePort, blocked)
|
||||
}
|
||||
if p.mine != 1 || p.share != 1 {
|
||||
t.Errorf("pool = %+v, want mine=1 share=1", p)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSingleNodeNeverReleases: a standalone node is the whole cluster, so its
|
||||
// share is everything it holds. Without this the node would shed forwards onto
|
||||
// nobody (they would sit unclaimed until it re-claimed them).
|
||||
func TestSingleNodeNeverReleases(t *testing.T) {
|
||||
e := newTestEngine("solo", true)
|
||||
e.state.Nodes = []Node{{ID: "solo", Addr: "solo:7500", Alive: true, IsLeader: true}}
|
||||
for i := 0; i < 5; i++ {
|
||||
e.state.AddTopology(&Task{
|
||||
ID: "f" + strconv.Itoa(i), Local: store.Local{Name: "web"},
|
||||
Remote: store.Remote{Name: "frps1"}, Link: store.Link{RemotePort: 100 + i},
|
||||
}, "solo")
|
||||
}
|
||||
b := NewBalancer(e, 0)
|
||||
if pick, _, blocked := b.assess(1000); pick != nil {
|
||||
t.Errorf("a single-node ring must not release; got blocked=%q", blocked)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBelowFloorNeverReleases: two forwards on a node is the floor. A node
|
||||
// holding exactly two at an even split of two must keep both — one release per
|
||||
// node per pass is the granularity, and shedding below the floor would empty it.
|
||||
func TestBelowFloorNeverReleases(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 2, 2) // 6 total over 3 nodes → share 2
|
||||
b := NewBalancer(n1, 0)
|
||||
if pick, p, blocked := b.assess(1000); pick != nil {
|
||||
t.Errorf("2 owned against share %d must not release (blocked=%q, mine=%d)",
|
||||
p.share, blocked, p.mine)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNeverReleasesAStoppedForward: a stopped forward's entry is the carrier of
|
||||
// the user's stop decision. Releasing it would hand another node a STOPPED
|
||||
// forward as if it were work to run.
|
||||
func TestNeverReleasesAStoppedForward(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
// Stop two of n1's three; the only live one must be chosen.
|
||||
for _, te := range n1.state.ForwardsOwnedBy("n1") {
|
||||
if te.Link.RemotePort == 100 || te.Link.RemotePort == 101 {
|
||||
n1.state.UpdateTopologyDisabled("web", "frps1", te.Link.RemotePort, true)
|
||||
}
|
||||
}
|
||||
b := NewBalancer(n1, 0)
|
||||
pick, _, _ := b.assess(1000)
|
||||
if pick == nil {
|
||||
t.Fatal("n1 still owns one live forward and is over its share")
|
||||
}
|
||||
if pick.Link.RemotePort != 102 {
|
||||
t.Errorf("released port %d; a stopped forward (100/101) must never be chosen",
|
||||
pick.Link.RemotePort)
|
||||
}
|
||||
|
||||
// And with every forward stopped there is nothing to release.
|
||||
n1.state.UpdateTopologyDisabled("web", "frps1", 102, true)
|
||||
if pick, _, blocked := b.assess(1000); pick != nil {
|
||||
t.Errorf("all forwards stopped: nothing may be released (blocked=%q)", blocked)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCandidateChoiceIsDeterministic: the same state must pick the same forward,
|
||||
// so the decision is explainable and a re-evaluation does not churn a different
|
||||
// connection each tick.
|
||||
func TestCandidateChoiceIsDeterministic(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 4, 0)
|
||||
a := releaseCandidate(n1.State(), "n1")
|
||||
c := releaseCandidate(n1.State(), "n1")
|
||||
if a == nil || c == nil {
|
||||
t.Fatal("expected a candidate")
|
||||
}
|
||||
if a.TaskID != c.TaskID || a.Link.RemotePort != c.Link.RemotePort {
|
||||
t.Errorf("candidate changed between identical evaluations: %s:%d vs %s:%d",
|
||||
a.TaskID, a.Link.RemotePort, c.TaskID, c.Link.RemotePort)
|
||||
}
|
||||
if a.Link.RemotePort != 100 {
|
||||
t.Errorf("expected the lowest port (100), got %d", a.Link.RemotePort)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the runtime signal that makes this safe ----
|
||||
|
||||
// TestRevokeLeavesACarrierAndReleaseRemovesIt is the reason the release path has
|
||||
// to be a distinct task kind.
|
||||
//
|
||||
// A user stop keeps the topology entry as the carrier of the disabled flag, so
|
||||
// the entry is STILL THERE and still attributed to the owner. A rebalance is the
|
||||
// opposite: the entry must go, because (a) the node's load is derived from
|
||||
// owned entries so it cannot fall while the entry exists, and (b) every node's
|
||||
// duplicate guard refuses a creation task for a forward that already has an owner
|
||||
// — so the re-published task would be dropped by everyone and the forward
|
||||
// stranded on the node that gave it up.
|
||||
func TestRevokeLeavesACarrierAndReleaseRemovesIt(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
|
||||
// A stop marks the entry and keeps it.
|
||||
n1.state.UpdateTopologyDisabled("web", "frps1", 100, true)
|
||||
tk := &Task{Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||
Link: store.Link{RemotePort: 100}}
|
||||
if owner := n1.state.TopologyOwner(tk); owner != "n1" {
|
||||
t.Fatalf("a stop must keep the entry as the flag's carrier, owner=%q", owner)
|
||||
}
|
||||
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 3 {
|
||||
t.Fatalf("a stop must not change the owner's forward count, got %d", got)
|
||||
}
|
||||
|
||||
// A release removes the entry (leaving the other two alone).
|
||||
if !n1.RemoveTopologyEntry("web", "frps1", 100) {
|
||||
t.Fatal("release must remove the entry")
|
||||
}
|
||||
if owner := n1.state.TopologyOwner(tk); owner != "" {
|
||||
t.Errorf("after a release the forward must have no owner, got %q", owner)
|
||||
}
|
||||
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 2 {
|
||||
t.Errorf("a release must reduce the owner's count, got %d", got)
|
||||
}
|
||||
if got := len(n1.state.TopologyList()); got != 2 {
|
||||
t.Errorf("the other forwards must be untouched, got %d entries", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReleaseTaskExecutesOnlyOnTheOwner: like a revoke, a release is
|
||||
// owner-directed. Executing it elsewhere would be an attempt to move somebody
|
||||
// else's load, which cannot work — the owner's own count is what must fall.
|
||||
func TestReleaseTaskExecutesOnlyOnTheOwner(t *testing.T) {
|
||||
n1, n2, _ := balanceRing(t, 3, 0)
|
||||
|
||||
released := map[string]int{}
|
||||
for _, e := range []*Engine{n1, n2} {
|
||||
h := e.Handler.(*fakeHandler)
|
||||
h.release = func(ctx context.Context, tk *Task) error {
|
||||
released[e.ID]++
|
||||
return nil
|
||||
}
|
||||
}
|
||||
tk := n1.ReleaseTask(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||
store.Link{RemotePort: 100})
|
||||
if tk == nil {
|
||||
t.Fatal("ReleaseTask returned nil")
|
||||
}
|
||||
|
||||
// Each Engine keeps its OWN state, so a task published on n1 is not visible
|
||||
// to n2 until a token round carries it. Broadcast it the way the ring does.
|
||||
var carry State
|
||||
cb, _ := json.Marshal(n1.state)
|
||||
_ = json.Unmarshal(cb, &carry)
|
||||
n2.AdoptState(carry)
|
||||
n2.state.PendingTasks[tk.ID] = tk
|
||||
|
||||
// n2 is the lightest node and does NOT own the forward: it must leave the
|
||||
// task alone.
|
||||
if err := n2.runCommands(context.Background(), &Token{}); err != nil {
|
||||
t.Fatalf("n2 runCommands: %v", err)
|
||||
}
|
||||
if released["n2"] != 0 {
|
||||
t.Errorf("n2 released %d forward(s); a release is owner-directed only", released["n2"])
|
||||
}
|
||||
if _, still := n2.state.PendingTasks[tk.ID]; !still {
|
||||
t.Error("n2 consumed a release it does not own; it must ride the token to the owner")
|
||||
}
|
||||
|
||||
// n1 owns it: it must release, drop the entry, and re-publish the forward.
|
||||
if err := n1.runCommands(context.Background(), &Token{}); err != nil {
|
||||
t.Fatalf("n1 runCommands: %v", err)
|
||||
}
|
||||
if released["n1"] != 1 {
|
||||
t.Fatalf("the owner released %d times, want 1", released["n1"])
|
||||
}
|
||||
if owner := n1.state.TopologyOwner(&Task{
|
||||
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||
Link: store.Link{RemotePort: 100},
|
||||
}); owner != "" {
|
||||
t.Errorf("the released forward still has owner %q", owner)
|
||||
}
|
||||
// And it is back in the ring as an ordinary creation task.
|
||||
var republished *Task
|
||||
for _, p := range n1.state.PendingList() {
|
||||
if !p.Release && !p.Revoke && !p.Restart &&
|
||||
p.Link.RemotePort == 100 && p.Local.Name == "web" {
|
||||
republished = p
|
||||
}
|
||||
}
|
||||
if republished == nil {
|
||||
t.Fatal("the released forward was not re-published as an unclaimed task")
|
||||
}
|
||||
if republished.Link.Disabled {
|
||||
t.Error("the re-published forward must be ENABLED, not marked stopped")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- dry-run / off ----
|
||||
|
||||
// TestDryRunReleasesNothing: dry-run is the default, and its whole value is that
|
||||
// it can run on a live cluster without disturbing anything.
|
||||
func TestDryRunReleasesNothing(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceDryRun)
|
||||
|
||||
msg := b.Step(1000)
|
||||
if !strings.Contains(msg, "would release") {
|
||||
t.Fatalf("dry-run should report what it would do, got %q", msg)
|
||||
}
|
||||
if b.InFlight() {
|
||||
t.Error("dry-run started a release")
|
||||
}
|
||||
for _, tk := range n1.state.PendingList() {
|
||||
if tk.Release {
|
||||
t.Error("dry-run published a release task")
|
||||
}
|
||||
}
|
||||
if got := len(n1.state.TopologyList()); got != 3 {
|
||||
t.Errorf("dry-run changed the topology: %d entries, want 3", got)
|
||||
}
|
||||
// A second identical tick must not repeat the same line (log hygiene).
|
||||
if again := b.Step(1010); again != "" {
|
||||
t.Errorf("dry-run repeated an unchanged decision: %q", again)
|
||||
}
|
||||
}
|
||||
|
||||
// TestModeOffDoesNothing reinforces that OFF means off, not "dry-run".
|
||||
func TestModeOffDoesNothing(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceOff)
|
||||
if msg := b.Step(1000); msg != "" {
|
||||
t.Errorf("mode off reported %q", msg)
|
||||
}
|
||||
if b.InFlight() {
|
||||
t.Error("mode off started a release")
|
||||
}
|
||||
// A typo must not enable releasing.
|
||||
b.SetMode("yes-please")
|
||||
if b.Mode() != BalanceDryRun {
|
||||
t.Errorf("an unknown mode must fall back to dry-run, got %q", b.Mode())
|
||||
}
|
||||
}
|
||||
|
||||
// ---- end-to-end through the ring machinery ----
|
||||
|
||||
// TestOwnerReleaseThenRingClaimsElsewhere is the whole design in one test: the
|
||||
// owner gives the forward up, and the ordinary lowest-load claim then places it
|
||||
// on an idle node — exactly what the self-directed rule depends on.
|
||||
func TestOwnerReleaseThenRingClaimsElsewhere(t *testing.T) {
|
||||
n1, n2, n3 := balanceRing(t, 3, 0)
|
||||
|
||||
// n2 and n3 must look idle to the claim path (they do: Forwards=0).
|
||||
claimed := map[string]int{}
|
||||
for _, e := range []*Engine{n1, n2, n3} {
|
||||
h := e.Handler.(*fakeHandler)
|
||||
h.claim = func(ctx context.Context, tk *Task) error {
|
||||
claimed[e.ID]++
|
||||
return nil
|
||||
}
|
||||
}
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceOn)
|
||||
|
||||
// n1 is over its share → it releases.
|
||||
msg := b.Step(1000)
|
||||
if !strings.Contains(msg, "releasing") {
|
||||
t.Fatalf("expected a release, got %q", msg)
|
||||
}
|
||||
if err := n1.runCommands(context.Background(), &Token{}); err != nil {
|
||||
t.Fatalf("n1 runCommands: %v", err)
|
||||
}
|
||||
// Find the task the owner re-published for the forward it gave up.
|
||||
var tk *Task
|
||||
for _, p := range n1.state.PendingList() {
|
||||
if !p.Release && p.Link.RemotePort == 100 && p.Local.Name == "web" {
|
||||
tk = p
|
||||
}
|
||||
}
|
||||
if tk == nil {
|
||||
t.Fatal("the owner did not re-publish the forward it released")
|
||||
}
|
||||
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 2 {
|
||||
t.Fatalf("n1 should now own 2, got %d", got)
|
||||
}
|
||||
|
||||
// The forward is unclaimed; the lowest node claims it. n2 is the lightest
|
||||
// (Forwards=0 and the same mem/net), so it should take it. The task reaches
|
||||
// n2 on the token — each engine holds its own state, so carry it over the way
|
||||
// a real round does.
|
||||
var carried State
|
||||
cb, _ := json.Marshal(n1.state)
|
||||
_ = json.Unmarshal(cb, &carried)
|
||||
n2.AdoptState(carried)
|
||||
n2.state.PendingTasks[tk.ID] = tk
|
||||
|
||||
if err := n2.runCommands(context.Background(), &Token{}); err != nil {
|
||||
t.Fatalf("n2 runCommands: %v", err)
|
||||
}
|
||||
if claimed["n2"] == 0 {
|
||||
t.Fatal("the idle node did not claim the released forward")
|
||||
}
|
||||
// And n1 no longer owns it.
|
||||
for _, te := range n1.state.ForwardsOwnedBy("n1") {
|
||||
if te.Link.RemotePort == 100 {
|
||||
t.Error("n1 still owns the forward it released")
|
||||
}
|
||||
}
|
||||
|
||||
// The balancer sees the release land and stops holding the release state.
|
||||
if msg := b.Step(1100); !strings.Contains(msg, "gave up") {
|
||||
t.Errorf("expected the release to be confirmed, got %q", msg)
|
||||
}
|
||||
if b.InFlight() {
|
||||
t.Error("the release should be complete")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCooldownBoundsTheChurn: a release costs a live connection, so one node may
|
||||
// not shed again until the cooldown elapses — even if it is still over its share.
|
||||
func TestCooldownBoundsTheChurn(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 4, 0)
|
||||
b := NewBalancer(n1, 600) // 10 minutes
|
||||
b.SetMode(BalanceOn)
|
||||
|
||||
if msg := b.Step(1000); !strings.Contains(msg, "releasing") {
|
||||
t.Fatalf("first release expected, got %q", msg)
|
||||
}
|
||||
n1.RemoveTopologyEntry("web", "frps1", 100) // land it
|
||||
if msg := b.Step(1010); !strings.Contains(msg, "gave up") {
|
||||
t.Fatalf("expected confirmation, got %q", msg)
|
||||
}
|
||||
// Still over its share (3 of 4 over 3 → share 2), but inside the cooldown.
|
||||
if pick, _, blocked := b.assess(1020); pick == nil || blocked != "cooldown" {
|
||||
t.Errorf("expected the second release to be blocked by cooldown, got blocked=%q", blocked)
|
||||
}
|
||||
if b.InFlight() {
|
||||
t.Error("cooldown must not leave a release in flight")
|
||||
}
|
||||
// Past the cooldown it may act again.
|
||||
if pick, _, blocked := b.assess(1000 + 601); pick == nil || blocked != "" {
|
||||
t.Errorf("past the cooldown the release should be allowed, got blocked=%q", blocked)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReleaseDeadlineClearsAStuckConfirmation: if the release never lands (leader
|
||||
// change, ring join), the node must not wedge forever.
|
||||
func TestReleaseDeadlineClearsAStuckConfirmation(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceOn)
|
||||
_ = b.Step(1000) // release published, entry NOT removed (nothing executed it)
|
||||
if !b.InFlight() {
|
||||
t.Fatal("precondition: a release should be in flight")
|
||||
}
|
||||
if msg := b.Step(1000 + releaseDeadlineSec + 1); msg != "" {
|
||||
t.Errorf("a timed-out release should clear silently, got %q", msg)
|
||||
}
|
||||
if b.InFlight() {
|
||||
t.Error("the release state should have been cleared after the deadline")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- read-only report ----
|
||||
|
||||
// TestReportDoesNotMutate — polling the endpoint must not start, delay or cancel
|
||||
// a release. A read-only surface that perturbs what it reports is worse than none.
|
||||
func TestReportDoesNotMutate(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 600)
|
||||
b.SetMode(BalanceOn)
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
rep := b.Report()
|
||||
if rep.SelfForwards != 3 {
|
||||
t.Fatalf("report changed the node's forward count: %d", rep.SelfForwards)
|
||||
}
|
||||
if rep.Mode != BalanceOn {
|
||||
t.Fatalf("report changed the mode: %q", rep.Mode)
|
||||
}
|
||||
}
|
||||
if b.InFlight() || b.lastReleaseAt != 0 {
|
||||
t.Error("polling the report started a release")
|
||||
}
|
||||
if rep := b.Report(); rep.WouldRelease == nil {
|
||||
t.Error("the report should still surface the pending decision")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReportExplainsWhyNothingHappens pins the field that answers the question
|
||||
// the log cannot: an over-share node that is holding back must say why.
|
||||
func TestReportExplainsWhyNothingHappens(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 4, 0)
|
||||
b := NewBalancer(n1, 600)
|
||||
b.SetMode(BalanceOn)
|
||||
|
||||
rep := b.Report()
|
||||
if !rep.SelfOverShare {
|
||||
t.Fatalf("4 of 4 forwards over 3 nodes should be over the share (share=%d)", rep.Share)
|
||||
}
|
||||
if rep.WouldRelease == nil {
|
||||
t.Fatal("an over-share node must surface the forward it would give up")
|
||||
}
|
||||
if rep.BlockedBy != "" {
|
||||
t.Errorf("nothing should block the first release, got %q", rep.BlockedBy)
|
||||
}
|
||||
|
||||
// Inside the cooldown the same node reports the block. Report() reads the
|
||||
// real clock, so set lastReleaseAt relative to now rather than to a fake one.
|
||||
b.lastReleaseAt = time.Now().Unix() - 10
|
||||
n1.RemoveTopologyEntry("web", "frps1", 100)
|
||||
rep = b.Report()
|
||||
if rep.BlockedBy != "cooldown" {
|
||||
t.Errorf("after a release the node should report a cooldown block, got %q", rep.BlockedBy)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReportOnBalancedNodeIsQuiet: the healthy case must be a real answer, not an
|
||||
// empty one — otherwise "balanced" and "balancing is switched off" look identical.
|
||||
func TestReportOnBalancedNodeIsQuiet(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 1, 1)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceOn)
|
||||
|
||||
rep := b.Report()
|
||||
if rep.WouldRelease != nil {
|
||||
t.Errorf("a balanced node reported a release: %+v", rep.WouldRelease)
|
||||
}
|
||||
if rep.SelfOverShare {
|
||||
t.Error("1 of 3 over 3 nodes is not over the share")
|
||||
}
|
||||
if rep.TotalForwards != 3 || rep.AliveNodes != 3 || rep.Share != 1 {
|
||||
t.Errorf("report pool = total %d alive %d share %d, want 3/3/1",
|
||||
rep.TotalForwards, rep.AliveNodes, rep.Share)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- mode switching (the cluster page's switch) ----
|
||||
|
||||
// TestStoredModeWinsOverEnv: flipping the switch must not silently revert on the
|
||||
// next restart. The environment is only the bootstrap default; once an operator
|
||||
// has chosen a mode it is persisted and wins.
|
||||
func TestStoredModeWinsOverEnv(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode("dry-run") // the env default
|
||||
|
||||
var saved string
|
||||
b.SetModePersistence(
|
||||
func() (string, bool) { return "on", true },
|
||||
func(m string) error { saved = m; return nil },
|
||||
)
|
||||
if b.Mode() != BalanceOn {
|
||||
t.Fatalf("a stored mode must win over the environment default, got %q", b.Mode())
|
||||
}
|
||||
if b.ModeSource() != "stored" {
|
||||
t.Errorf("ModeSource = %q, want stored", b.ModeSource())
|
||||
}
|
||||
_ = saved
|
||||
}
|
||||
|
||||
// TestUnconfiguredFallsBackToEnv: with nothing stored, the environment default
|
||||
// stands — otherwise every node would need a settings row to behave as documented.
|
||||
func TestUnconfiguredFallsBackToEnv(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode("off")
|
||||
b.SetModePersistence(func() (string, bool) { return "", false }, func(string) error { return nil })
|
||||
if b.Mode() != BalanceOff {
|
||||
t.Errorf("with nothing stored the env default must stand, got %q", b.Mode())
|
||||
}
|
||||
if b.ModeSource() != "stored" {
|
||||
// persist is installed, so the node CAN be controlled
|
||||
t.Logf("ModeSource=%q", b.ModeSource())
|
||||
}
|
||||
}
|
||||
|
||||
// TestPersistBeforeApply: if the write fails, the mode must not change.
|
||||
// Applying first would leave the node running a mode it cannot reproduce after a
|
||||
// restart — the switch would lie.
|
||||
func TestPersistBeforeApply(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceDryRun)
|
||||
b.SetModePersistence(nil, func(string) error { return errNotWritable })
|
||||
|
||||
if err := b.SetModeFromUI(BalanceOn); err == nil {
|
||||
t.Fatal("a failed save must surface an error")
|
||||
}
|
||||
if b.Mode() != BalanceDryRun {
|
||||
t.Errorf("a failed save must leave the mode unchanged, got %q", b.Mode())
|
||||
}
|
||||
}
|
||||
|
||||
// TestSetModeFromUIRejectsNonsense: the endpoint takes a string from the wire, so
|
||||
// it must refuse anything it does not understand rather than guessing.
|
||||
func TestSetModeFromUIRejectsNonsense(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceDryRun)
|
||||
if err := b.SetModeFromUI("yes"); err == nil {
|
||||
t.Fatal("an unknown mode must be rejected, not coerced")
|
||||
}
|
||||
if b.Mode() != BalanceDryRun {
|
||||
t.Errorf("a rejected mode must not change anything, got %q", b.Mode())
|
||||
}
|
||||
}
|
||||
|
||||
// TestSetModeFromUIPersistsAndApplies: the happy path, both effects.
|
||||
func TestSetModeFromUIPersistsAndApplies(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
b.SetMode(BalanceDryRun)
|
||||
var saved string
|
||||
b.SetModePersistence(nil, func(m string) error { saved = m; return nil })
|
||||
|
||||
if err := b.SetModeFromUI(BalanceOn); err != nil {
|
||||
t.Fatalf("SetModeFromUI: %v", err)
|
||||
}
|
||||
if saved != BalanceOn {
|
||||
t.Errorf("saved %q, want on", saved)
|
||||
}
|
||||
if b.Mode() != BalanceOn {
|
||||
t.Errorf("mode = %q, want on", b.Mode())
|
||||
}
|
||||
if !b.Report().CanControl {
|
||||
t.Error("a node with persistence installed should report itself controllable")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReportCanControlOffWithoutPersistence: a node that cannot persist a choice
|
||||
// must not advertise the switch (an embedder/test without a store).
|
||||
func TestReportCanControlOffWithoutPersistence(t *testing.T) {
|
||||
n1, _, _ := balanceRing(t, 3, 0)
|
||||
b := NewBalancer(n1, 0)
|
||||
if b.Report().CanControl {
|
||||
t.Error("a balancer with no persistence must not report itself controllable")
|
||||
}
|
||||
}
|
||||
91
internal/cluster/debug.go
Normal file
91
internal/cluster/debug.go
Normal file
@ -0,0 +1,91 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// logf is the package's logging seam: every cluster log line funnels through it
|
||||
// so the debug gate in debug.go has a single place to hook.
|
||||
func logf(format string, args ...any) { log.Printf(format, args...) }
|
||||
|
||||
// Token rotation is the ring's heartbeat: with a 2s round delay it fires
|
||||
// continuously, and the default three log lines per round
|
||||
// ("OnToken cycle=N" / "forward cycle=N to X" / "token-send ... -> 200")
|
||||
// carry no information — no cycle number, address, timing or payload ever
|
||||
// changes in the steady state. Measured on the live 3-node cluster that was
|
||||
// ~510k lines/day on one node (3.5M lines in a week), which drowns every real
|
||||
// event in the journal and fills the disk for a signal that is already
|
||||
// available in structured form via GET /api/manager/cluster/ring (cycle,
|
||||
// lastSync, roundDelayMs, node aliveness).
|
||||
//
|
||||
// So the steady-state lines are demoted to a debug level, off by default and
|
||||
// enabled with W4F_DEBUG=token (or 1/all/true for every debug line). Failure
|
||||
// paths are NOT demoted: a send error, a stale token, a timeout or a leader
|
||||
// change is exactly what someone is grepping for, and losing those to a quiet
|
||||
// default would be a bad trade.
|
||||
const debugEnv = "W4F_DEBUG"
|
||||
|
||||
// debugToken logs a per-token-round heartbeat line. Suppressed unless
|
||||
// W4F_DEBUG selects "token" (or a catch-all value).
|
||||
func debugToken(format string, args ...any) {
|
||||
if debugTokenOn.Load() {
|
||||
logf(format, args...)
|
||||
}
|
||||
}
|
||||
|
||||
// debugAll logs an ad-hoc diagnostic line. Suppressed unless W4F_DEBUG is set
|
||||
// to a catch-all value (1/all/true/*).
|
||||
func debugAll(format string, args ...any) {
|
||||
if debugAllOn.Load() {
|
||||
logf(format, args...)
|
||||
}
|
||||
}
|
||||
|
||||
var (
|
||||
debugTokenOn atomic.Bool
|
||||
debugAllOn atomic.Bool
|
||||
)
|
||||
|
||||
func init() { ReloadDebug() }
|
||||
|
||||
// ReloadDebug re-reads W4F_DEBUG. Called once at init so tests can flip it
|
||||
// without restarting, and available at runtime for an operator who wants to
|
||||
// watch the ring without a redeploy.
|
||||
func ReloadDebug() {
|
||||
v := os.Getenv(debugEnv)
|
||||
switch normalized := normalizeDebugValue(v); normalized {
|
||||
case "token":
|
||||
debugTokenOn.Store(true)
|
||||
debugAllOn.Store(false)
|
||||
case "all":
|
||||
debugTokenOn.Store(true)
|
||||
debugAllOn.Store(true)
|
||||
default:
|
||||
debugTokenOn.Store(false)
|
||||
debugAllOn.Store(false)
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeDebugValue(v string) string {
|
||||
// Compare case-insensitively without pulling in strings just for this.
|
||||
out := make([]rune, 0, len(v))
|
||||
for _, r := range v {
|
||||
if r >= 'A' && r <= 'Z' {
|
||||
r += 'a' - 'A'
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
s := string(out)
|
||||
switch s {
|
||||
case "":
|
||||
return ""
|
||||
case "token", "tokens", "ring":
|
||||
return "token"
|
||||
}
|
||||
// Any other non-empty value is a deliberate request for more output, so it
|
||||
// is treated as a catch-all rather than silently muting the operator who
|
||||
// set it. "0" lands here too: it was asked for, so honour it.
|
||||
return "all"
|
||||
}
|
||||
108
internal/cluster/debug_test.go
Normal file
108
internal/cluster/debug_test.go
Normal file
@ -0,0 +1,108 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// captureLog redirects the standard logger into a buffer for the duration of
|
||||
// fn and returns what was written.
|
||||
func captureLog(t *testing.T, fn func()) string {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
orig := log.Writer()
|
||||
origFlags := log.Flags()
|
||||
log.SetOutput(&buf)
|
||||
log.SetFlags(0)
|
||||
defer func() {
|
||||
log.SetOutput(orig)
|
||||
log.SetFlags(origFlags)
|
||||
}()
|
||||
fn()
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
func TestDebugTokenSuppressedByDefault(t *testing.T) {
|
||||
os.Unsetenv(debugEnv)
|
||||
ReloadDebug()
|
||||
|
||||
out := captureLog(t, func() {
|
||||
debugToken("ring[%s] OnToken cycle=%d", "node:7500", 42)
|
||||
debugToken("ring[%s] forward cycle=%d to %s", "node:7500", 42, "next:7500")
|
||||
debugToken("token-send %s: size=%dB elapsed=%v -> %d", "next:7500", 5605, "170ms", 200)
|
||||
})
|
||||
if out != "" {
|
||||
t.Fatalf("steady-state token lines logged with W4F_DEBUG unset: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDebugTokenEnabledByEnv(t *testing.T) {
|
||||
for _, v := range []string{"token", "TOKEN", "tokens", "ring", "1", "true", "yes", "all", "*", "yes-please"} {
|
||||
t.Run(v, func(t *testing.T) {
|
||||
t.Setenv(debugEnv, v)
|
||||
ReloadDebug()
|
||||
if !debugTokenOn.Load() {
|
||||
t.Fatalf("W4F_DEBUG=%q should enable the token heartbeat", v)
|
||||
}
|
||||
out := captureLog(t, func() {
|
||||
debugToken("ring[%s] OnToken cycle=%d", "node:7500", 7)
|
||||
})
|
||||
if !strings.Contains(out, "OnToken cycle=7") {
|
||||
t.Fatalf("W4F_DEBUG=%q: expected the line to be logged, got %q", v, out)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The whole point of the gate is to shrink the journal, so the failure paths
|
||||
// must stay loud without any env var — losing a send error to a quiet default
|
||||
// would be a bad trade.
|
||||
func TestFailurePathsStayLoud(t *testing.T) {
|
||||
os.Unsetenv(debugEnv)
|
||||
ReloadDebug()
|
||||
|
||||
out := captureLog(t, func() {
|
||||
logf("token-send %s: size=%dB elapsed=%v err=%v", "down:7500", 10, "5ms", "connection refused")
|
||||
})
|
||||
if !strings.Contains(out, "connection refused") {
|
||||
t.Fatalf("send errors must never be gated: got %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDebugAllOffForTokenOnly(t *testing.T) {
|
||||
t.Setenv(debugEnv, "token")
|
||||
ReloadDebug()
|
||||
if !debugTokenOn.Load() {
|
||||
t.Fatal("token level should enable debugToken")
|
||||
}
|
||||
if debugAllOn.Load() {
|
||||
t.Fatal("token level must not enable the catch-all debugAll")
|
||||
}
|
||||
out := captureLog(t, func() { debugAll("scratch diagnostic") })
|
||||
if out != "" {
|
||||
t.Fatalf("debugAll should stay off at token level, got %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeDebugValue(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"": "",
|
||||
"token": "token",
|
||||
"TOKEN": "token",
|
||||
"Ring": "token",
|
||||
"1": "all",
|
||||
"true": "all",
|
||||
"ALL": "all",
|
||||
"*": "all",
|
||||
"anything": "all", // unrecognised but deliberate → don't silence it
|
||||
"0": "all", // 0 is still a deliberate request for output
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := normalizeDebugValue(in); got != want {
|
||||
t.Errorf("normalizeDebugValue(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -62,6 +62,25 @@ type Task struct {
|
||||
// owning node cancels it (stop worker, drop from topology). Reuses the
|
||||
// same publish channel as creation (round-1 inject, round-2 apply).
|
||||
Revoke bool `json:"revoke,omitempty"`
|
||||
// Restart marks a RE-ENABLE task: the forward is already claimed and its
|
||||
// topology entry still exists (a stop keeps the entry as the flag's
|
||||
// carrier), so the creation path would dedupe the submission and the
|
||||
// duplicate-claim guard would discard it. The owner applies this one
|
||||
// unconditionally: re-mark the entry enabled and (re)spawn the worker.
|
||||
Restart bool `json:"restart,omitempty"`
|
||||
// Release marks a REBALANCING task: the OWNER of the forward gives it up so
|
||||
// the ring can place it elsewhere. Only the owning node may act on it — it is
|
||||
// that node's own load which has to fall, and only it can make that true (its
|
||||
// topology entry is local bookkeeping no peer may write).
|
||||
//
|
||||
// It is deliberately NOT the same thing as a user stop (Revoke). A stop keeps
|
||||
// the topology entry as the carrier of the disabled flag and flags the link
|
||||
// stopped; a release REMOVES the entry and leaves the link enabled, because
|
||||
// the forward is about to run somewhere else. Marking it stopped instead would
|
||||
// make the re-published creation task look like a duplicate of an existing
|
||||
// forward, and every node would refuse to claim it — stranding the forward on
|
||||
// the node that just gave it up.
|
||||
Release bool `json:"release,omitempty"`
|
||||
// RemoveNode: node ID to remove from the ring; the node self-removes when
|
||||
// the command reaches it via the token.
|
||||
RemoveNode string `json:"removeNode,omitempty"`
|
||||
@ -333,10 +352,15 @@ func (s *State) AddTopology(t *Task, ownerID string) *TopoEntry {
|
||||
if s.Topology == nil {
|
||||
s.Topology = map[string]*TopoEntry{}
|
||||
}
|
||||
// Active mirrors the task's disabled flag rather than being unconditionally
|
||||
// true. A claim can legitimately carry a stopped forward (e.g. a node
|
||||
// re-claiming from a departed peer), and forcing Active=true there would
|
||||
// resurrect it: OfflineReassign only re-queues Active entries, and every
|
||||
// reader that filters on Active would treat it as running again.
|
||||
e := &TopoEntry{
|
||||
TaskID: t.ID, OwnerID: ownerID,
|
||||
Local: t.Local, Remote: t.Remote, Link: t.Link,
|
||||
Active: true,
|
||||
Active: !t.Link.Disabled,
|
||||
}
|
||||
s.Topology[t.ID] = e
|
||||
return e
|
||||
@ -408,6 +432,39 @@ func (s *State) ForwardsOwnedBy(owner string) []*TopoEntry {
|
||||
// (local, remote, port) triple. Returns true if found. The updated entry
|
||||
// propagates to all nodes via the next token cycle — group changes sync
|
||||
// through the ring without a dedicated command.
|
||||
// UpdateTopologyDisabled sets the disabled flag on the topology entry matching
|
||||
// the forward's natural key. Mirror of UpdateTopologyGroup: it makes a locally
|
||||
// decided stop/start part of the topology so the next token cycle carries it to
|
||||
// every other member (the alternative — a one-shot revoke task addressed at the
|
||||
// owner — only converges if that owner happens to be online right then).
|
||||
//
|
||||
// Returns false when the forward is not in the topology, which is not an error:
|
||||
// a stopped forward may legitimately have no entry yet.
|
||||
func (s *State) UpdateTopologyDisabled(local, remote string, port int, disabled bool) bool {
|
||||
for _, e := range s.Topology {
|
||||
if e.Local.Name == local && e.Remote.Name == remote && e.Link.RemotePort == port {
|
||||
e.Link.Disabled = disabled
|
||||
// Keep Active consistent with the flag so readers that key off it
|
||||
// (status page, load accounting) agree with Link.Disabled.
|
||||
e.Active = !disabled
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// TopologyDisabled returns the disabled flag the cluster currently holds for a
|
||||
// forward, and whether such an entry exists. Used by the adoption path to learn
|
||||
// a peer's decision into the local store.
|
||||
func (s *State) TopologyDisabled(local, remote string, port int) (bool, bool) {
|
||||
for _, e := range s.Topology {
|
||||
if e.Local.Name == local && e.Remote.Name == remote && e.Link.RemotePort == port {
|
||||
return e.Link.Disabled, true
|
||||
}
|
||||
}
|
||||
return false, false
|
||||
}
|
||||
|
||||
func (s *State) UpdateTopologyGroup(local, remote string, port int, group string) bool {
|
||||
for _, e := range s.Topology {
|
||||
if e.Local.Name == local && e.Remote.Name == remote && e.Link.RemotePort == port {
|
||||
|
||||
505
internal/cluster/ring_balance.go
Normal file
505
internal/cluster/ring_balance.go
Normal file
@ -0,0 +1,505 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// The rebalance rule, deliberately simple and self-limiting:
|
||||
//
|
||||
// release one forward when Forwards >= minForwardsToShare AND Forwards > share
|
||||
//
|
||||
// where share = ceil(totalForwards / aliveNodes) — the even split.
|
||||
//
|
||||
// Every node runs this check about its OWN forwards. That is the whole point:
|
||||
// a node's load only actually falls when its own topology entry goes away, so
|
||||
// only the owner can produce the state change that makes the ring hand the
|
||||
// forward to somebody else. A coordinator on another machine can see the
|
||||
// imbalance but cannot make it true; it would have to reach in and mutate the
|
||||
// owner's bookkeeping, which is exactly the sort of cross-node write this ring
|
||||
// avoids everywhere else (revoke and restart are owner-directed for the same
|
||||
// reason).
|
||||
//
|
||||
// The rule is self-limiting in both directions, so it needs no upper bound on
|
||||
// how many forwards a node may shed:
|
||||
//
|
||||
// - It stops at the average. 3 forwards over 3 nodes: the owner holds 2 > 1,
|
||||
// releases one, now holds 1 which is not > 1, and stops. Without the "> share"
|
||||
// bound an owner would shed EVERY forward it has.
|
||||
// - It cannot empty a node. A node holding fewer than minForwardsToShare never
|
||||
// releases, so there is always something left to serve.
|
||||
//
|
||||
// A release costs a live connection: stopping the worker drops whatever was
|
||||
// flowing through the forward, and the new owner re-dials the local service.
|
||||
// Because the rule above already requires a real imbalance, pacing is handled by
|
||||
// a per-node cooldown (cooldownSec) rather than a "how long has this lasted"
|
||||
// window — there is no transient to smooth over, only a churn rate to bound.
|
||||
const (
|
||||
// minForwardsToShare: never release below this many owned forwards.
|
||||
minForwardsToShare = 2
|
||||
// balanceCooldownSec: minimum gap between two releases by one node. One
|
||||
// release per 10 minutes per node is a trickle; it is what stops a cluster
|
||||
// whose loads oscillate around the split point from restarting workers in a
|
||||
// loop.
|
||||
balanceCooldownSec = 600
|
||||
// releaseDeadlineSec: how long to wait for a published release to be
|
||||
// executed by its owner (us) before giving up on confirming it. A token
|
||||
// round is ~2s, so this is generous; it exists so a release lost to a
|
||||
// leader change or a joined ring cannot wedge the node forever.
|
||||
releaseDeadlineSec = 120
|
||||
)
|
||||
|
||||
// Balance modes.
|
||||
const (
|
||||
BalanceOff = "off"
|
||||
BalanceDryRun = "dry-run"
|
||||
BalanceOn = "on"
|
||||
)
|
||||
|
||||
// pool is the replicated view the rule works from.
|
||||
type pool struct {
|
||||
alive int // nodes that can take a forward
|
||||
mine int // forwards THIS node owns (live count, not the replicated one)
|
||||
total int // forwards across the ring
|
||||
share int // ceil(total/alive) — the even split
|
||||
}
|
||||
|
||||
// releaseState is one release this node has published and is waiting to see
|
||||
// leave its own topology.
|
||||
type releaseState struct {
|
||||
local, remote string
|
||||
port int
|
||||
startedAt int64
|
||||
}
|
||||
|
||||
// Balancer lets each node shed its own share of the cluster's forwards.
|
||||
//
|
||||
// It publishes only ONE kind of thing: a release of a forward it owns. There is
|
||||
// no targeted hand-off and no coordinator, because "drop it and let the ring's
|
||||
// normal lowest-load claim pick it up" is already the ring's only mechanism for
|
||||
// moving a forward — and it is the mechanism that, unlike a coordinated move,
|
||||
// cannot be undone by the node that gave the forward up (the claim path runs
|
||||
// after the node's load has already fallen).
|
||||
type Balancer struct {
|
||||
engine *Engine
|
||||
mode string
|
||||
|
||||
// cooldownSec is the minimum gap between two releases by this node.
|
||||
cooldownSec int64
|
||||
// releasing is the forward this node gave up and is waiting to see leave.
|
||||
// While it is set, no second release is started: one at a time keeps the
|
||||
// connection churn bounded and the log readable.
|
||||
releasing *releaseState
|
||||
// lastReleaseAt is when the previous release was published (0 = never).
|
||||
lastReleaseAt int64
|
||||
// reported remembers the last dry-run candidate so the observe-only mode
|
||||
// logs a change of mind rather than the same line every tick.
|
||||
reported string
|
||||
// persist saves a mode change so it survives a restart (nil = env-only).
|
||||
persist func(string) error
|
||||
// savedMode is the last mode this node stored, so the report can say whether
|
||||
// the active mode came from the environment or from an operator's choice.
|
||||
savedMode string
|
||||
}
|
||||
|
||||
// NewBalancer builds a balancer. It starts in observe-only mode: releasing a
|
||||
// forward drops a live connection, so acting on the ring is opt-in.
|
||||
func NewBalancer(e *Engine, cooldownSec int64) *Balancer {
|
||||
b := &Balancer{engine: e, mode: BalanceDryRun, cooldownSec: cooldownSec}
|
||||
if b.cooldownSec <= 0 {
|
||||
b.cooldownSec = balanceCooldownSec
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// SetMode selects off / dry-run / on. An unrecognised value is treated as
|
||||
// dry-run — a typo must not be able to turn releasing on.
|
||||
//
|
||||
// Only a change is logged, and the reason is recorded, because SetMode is now
|
||||
// called both at boot (from the environment) and at runtime (from the cluster
|
||||
// page): a one-line "why did this just start moving my forwards" answer is the
|
||||
// difference between a switch an operator dares to use and one they don't.
|
||||
func (b *Balancer) SetMode(m string) {
|
||||
if m != BalanceOff && m != BalanceDryRun && m != BalanceOn {
|
||||
if m != "" {
|
||||
log.Printf("ring[%s] unknown balance mode %q; using %s", b.engine.ID, m, BalanceDryRun)
|
||||
}
|
||||
m = BalanceDryRun
|
||||
}
|
||||
if b.mode == m {
|
||||
return
|
||||
}
|
||||
b.mode = m
|
||||
b.savedMode = m
|
||||
log.Printf("ring[%s] load balancing set to %s", b.engine.ID, strings.ToUpper(m))
|
||||
}
|
||||
|
||||
// Mode returns the active mode.
|
||||
func (b *Balancer) Mode() string { return b.mode }
|
||||
|
||||
// ModeSource reports where the active mode came from, for the report.
|
||||
//
|
||||
// "stored" means an operator chose it on the cluster page and it will survive a
|
||||
// restart; "env" means nothing was ever chosen here and the environment default
|
||||
// (or the built-in dry-run) applies. It is NOT "can this node save a choice" —
|
||||
// that is CanControl, and conflating the two made every node with a store claim
|
||||
// an operator had set it.
|
||||
func (b *Balancer) ModeSource() string {
|
||||
if b.savedMode != "" {
|
||||
return "stored"
|
||||
}
|
||||
return "env"
|
||||
}
|
||||
|
||||
// SetModePersistence installs the callback that saves a mode change so it
|
||||
// survives a restart, and applies any previously saved value.
|
||||
//
|
||||
// The environment is the BOOTSTRAP default; once an operator flips the switch on
|
||||
// the cluster page that stored value wins, otherwise the page would appear to
|
||||
// work and then silently revert on the next restart.
|
||||
func (b *Balancer) SetModePersistence(load func() (string, bool), save func(string) error) {
|
||||
b.persist = save
|
||||
if load == nil {
|
||||
return
|
||||
}
|
||||
if m, ok := load(); ok && m != "" {
|
||||
switch m {
|
||||
case BalanceOff, BalanceDryRun, BalanceOn:
|
||||
if b.mode != m {
|
||||
log.Printf("ring[%s] load balancing mode %s loaded from settings", b.engine.ID, m)
|
||||
}
|
||||
b.mode = m
|
||||
b.savedMode = m
|
||||
default:
|
||||
log.Printf("ring[%s] ignoring unknown stored balance mode %q", b.engine.ID, m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SetModeFromUI changes the mode at runtime and persists it.
|
||||
//
|
||||
// The order matters: persist FIRST, then apply. If the write fails nothing
|
||||
// changes, so the switch cannot report success while behaving differently after
|
||||
// the next restart.
|
||||
func (b *Balancer) SetModeFromUI(m string) error {
|
||||
if m != BalanceOff && m != BalanceDryRun && m != BalanceOn {
|
||||
return fmt.Errorf("未知模式 %q", m)
|
||||
}
|
||||
if b.persist != nil {
|
||||
if err := b.persist(m); err != nil {
|
||||
return fmt.Errorf("保存负载均衡模式失败: %w", err)
|
||||
}
|
||||
}
|
||||
b.SetMode(m)
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetCooldown overrides the per-node release cooldown.
|
||||
func (b *Balancer) SetCooldown(sec int64) {
|
||||
if sec > 0 {
|
||||
b.cooldownSec = sec
|
||||
}
|
||||
}
|
||||
|
||||
// InFlight reports whether a release is awaiting confirmation.
|
||||
func (b *Balancer) InFlight() bool { return b.releasing != nil }
|
||||
|
||||
// pool computes the ring-wide view this node derives for itself.
|
||||
//
|
||||
// Our own count is taken LIVE from the topology rather than from the replicated
|
||||
// Nodes[].Load, because Nodes[].Load is only refreshed when a token arrives and
|
||||
// the decision must see the effect of the release immediately — otherwise a node
|
||||
// would keep believing it is over its share and shed another forward each round.
|
||||
func (b *Balancer) pool() pool {
|
||||
st := &b.engine.state
|
||||
p := pool{mine: len(st.ForwardsOwnedBy(b.engine.ID))}
|
||||
for i := range st.Nodes {
|
||||
n := &st.Nodes[i]
|
||||
if !n.Alive {
|
||||
continue
|
||||
}
|
||||
p.alive++
|
||||
if n.ID == b.engine.ID {
|
||||
p.total += p.mine
|
||||
continue
|
||||
}
|
||||
p.total += n.Load.Forwards
|
||||
}
|
||||
if p.alive > 0 {
|
||||
p.share = (p.total + p.alive - 1) / p.alive
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// assess answers "should this node give one forward up, and which one?" together
|
||||
// with the facts behind the answer, so the read-only report can explain a node
|
||||
// that is over its share but not acting. It never mutates anything.
|
||||
//
|
||||
// blocked is a short reason: "single-node", "below-min", "balanced",
|
||||
// "nothing-active", "cooldown" or "in-flight".
|
||||
func (b *Balancer) assess(now int64) (pick *TopoEntry, p pool, blocked string) {
|
||||
p = b.pool()
|
||||
if !b.engine.IsMember() {
|
||||
return nil, p, "single-node"
|
||||
}
|
||||
if p.mine < minForwardsToShare {
|
||||
return nil, p, "below-min"
|
||||
}
|
||||
if p.mine <= p.share {
|
||||
return nil, p, "balanced"
|
||||
}
|
||||
pick = releaseCandidate(&b.engine.state, b.engine.ID)
|
||||
if pick == nil {
|
||||
return nil, p, "nothing-active"
|
||||
}
|
||||
// The guards below do not change the verdict ("this node is over its
|
||||
// share"), only whether now is the moment; the report shows both.
|
||||
if b.releasing != nil {
|
||||
return pick, p, "in-flight"
|
||||
}
|
||||
if b.lastReleaseAt > 0 && now-b.lastReleaseAt < b.cooldownSec {
|
||||
return pick, p, "cooldown"
|
||||
}
|
||||
return pick, p, ""
|
||||
}
|
||||
|
||||
// releaseCandidate picks which of this node's forwards to give up.
|
||||
//
|
||||
// Deterministic (lowest remote port, then task id) so the choice is explainable
|
||||
// and repeatable: a node that re-evaluates the same state picks the same
|
||||
// forward. Stopped forwards are skipped — their entry is the carrier of a user's
|
||||
// stop decision, and releasing one would hand a STOPPED forward to another node
|
||||
// as if it were work to run.
|
||||
func releaseCandidate(s *State, owner string) *TopoEntry {
|
||||
var mine []*TopoEntry
|
||||
for _, te := range s.TopologyList() {
|
||||
if te.OwnerID != owner || !te.Active || te.Link.Disabled {
|
||||
continue
|
||||
}
|
||||
mine = append(mine, te)
|
||||
}
|
||||
if len(mine) == 0 {
|
||||
return nil
|
||||
}
|
||||
sort.Slice(mine, func(i, j int) bool {
|
||||
if mine[i].Link.RemotePort != mine[j].Link.RemotePort {
|
||||
return mine[i].Link.RemotePort < mine[j].Link.RemotePort
|
||||
}
|
||||
return mine[i].TaskID < mine[j].TaskID
|
||||
})
|
||||
return mine[0]
|
||||
}
|
||||
|
||||
// landed reports whether the forward this node released has actually left it.
|
||||
//
|
||||
// The test is "no longer owned by me", NOT "no entry exists": the ring's normal
|
||||
// claim picks the forward up within a round or two, so an existence test would
|
||||
// see the new owner's entry and wait forever for a state that has already
|
||||
// happened.
|
||||
func (b *Balancer) landed() bool {
|
||||
m := b.releasing
|
||||
if m == nil {
|
||||
return true
|
||||
}
|
||||
owner := b.engine.state.TopologyOwner(&Task{
|
||||
Local: store.Local{Name: m.local},
|
||||
Remote: store.Remote{Name: m.remote},
|
||||
Link: store.Link{RemotePort: m.port},
|
||||
})
|
||||
return owner != b.engine.ID
|
||||
}
|
||||
|
||||
// Step advances the balancer by one cycle and returns a human-readable
|
||||
// description when it did (or, in dry-run, would have done) something.
|
||||
func (b *Balancer) Step(now int64) string {
|
||||
if b.releasing != nil {
|
||||
if b.landed() {
|
||||
done := b.releasing
|
||||
b.releasing = nil
|
||||
return "gave up " + done.local + ":" + strconv.Itoa(done.port) + " for rebalancing"
|
||||
}
|
||||
if now-b.releasing.startedAt > releaseDeadlineSec {
|
||||
log.Printf("ring[%s] release of %s:%d not confirmed within %ds; clearing (forward is still ours)",
|
||||
b.engine.ID, b.releasing.local, b.releasing.port, releaseDeadlineSec)
|
||||
b.releasing = nil
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
pick, p, blocked := b.assess(now)
|
||||
if pick == nil {
|
||||
b.reported = ""
|
||||
return ""
|
||||
}
|
||||
desc := pick.Local.Name + ":" + strconv.Itoa(pick.Link.RemotePort) +
|
||||
" (" + strconv.Itoa(p.mine) + " owned > " + strconv.Itoa(p.share) + " share of " +
|
||||
strconv.Itoa(p.total) + " over " + strconv.Itoa(p.alive) + " nodes)"
|
||||
|
||||
switch b.mode {
|
||||
case BalanceOff:
|
||||
return ""
|
||||
case BalanceDryRun:
|
||||
// Log a change of mind, not the same line every tick.
|
||||
if b.reported == desc {
|
||||
return ""
|
||||
}
|
||||
b.reported = desc
|
||||
if blocked != "" {
|
||||
return "would release " + desc + " [blocked: " + blocked + "]"
|
||||
}
|
||||
return "would release " + desc
|
||||
}
|
||||
|
||||
// Mode is ON. Blocked reasons still stop us.
|
||||
if blocked != "" {
|
||||
return ""
|
||||
}
|
||||
t := b.engine.ReleaseTask(pick.Local, pick.Remote, pick.Link)
|
||||
if t == nil {
|
||||
return ""
|
||||
}
|
||||
b.releasing = &releaseState{
|
||||
local: pick.Local.Name, remote: pick.Remote.Name,
|
||||
port: pick.Link.RemotePort, startedAt: now,
|
||||
}
|
||||
b.lastReleaseAt = now
|
||||
b.reported = ""
|
||||
return "releasing " + desc
|
||||
}
|
||||
|
||||
// Watch drives Step on a ticker until ctx is cancelled.
|
||||
//
|
||||
// It runs on EVERY node, not just the leader. Balancing here is not a cluster
|
||||
// decision being delegated to one node — it is each node applying the same
|
||||
// replicated rule to its own forwards. Two nodes never contend for the same
|
||||
// forward because a node can only release what it owns.
|
||||
//
|
||||
// The first check happens after one interval rather than immediately: a node
|
||||
// that has just booted adopts the ring's topology and its own load in the first
|
||||
// token round, and deciding before that would act on a half-known pool.
|
||||
func (b *Balancer) Watch(ctx context.Context, interval time.Duration) {
|
||||
tick := time.NewTicker(interval)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-tick.C:
|
||||
}
|
||||
if msg := b.Step(time.Now().Unix()); msg != "" {
|
||||
log.Printf("ring[%s] balance: %s", b.engine.ID, msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- read-only report for the API/UI ----
|
||||
|
||||
// BalanceReport is what this node thinks about balancing right now.
|
||||
//
|
||||
// It is per-node by nature: every member has its own answer, and the answer that
|
||||
// matters is the one on the node doing (or refusing to do) the releasing. The
|
||||
// peer list is included because "am I over my share?" is only meaningful next to
|
||||
// what everyone else holds.
|
||||
type BalanceReport struct {
|
||||
Mode string `json:"mode"` // off | dry-run | on
|
||||
ModeSource string `json:"modeSource"` // env | stored
|
||||
Enabled bool `json:"enabled"` // mode != off
|
||||
DryRun bool `json:"dryRun"` // mode != on
|
||||
// CanControl: this node offers the mode switch. False for a read-only
|
||||
// principal, so the server does not advertise a control the UI gates anyway.
|
||||
CanControl bool `json:"canControl"`
|
||||
IsMember bool `json:"isMember"`
|
||||
Self string `json:"self"`
|
||||
|
||||
SelfForwards int `json:"selfForwards"`
|
||||
Share int `json:"share"`
|
||||
TotalForwards int `json:"totalForwards"`
|
||||
AliveNodes int `json:"aliveNodes"`
|
||||
SelfOverShare bool `json:"selfOverShare"`
|
||||
|
||||
// WouldRelease is what this node would give up (present whenever it is over
|
||||
// its share, including while blocked — see BlockedBy).
|
||||
WouldRelease *BalanceReleasePlan `json:"wouldRelease,omitempty"`
|
||||
// Releasing is the release this node has published and is awaiting.
|
||||
Releasing *BalanceReleasePlan `json:"releasing,omitempty"`
|
||||
// BlockedBy explains why an over-share node is not acting yet.
|
||||
BlockedBy string `json:"blockedBy,omitempty"`
|
||||
|
||||
CooldownSec int64 `json:"cooldownSec"`
|
||||
LastReleaseAt int64 `json:"lastReleaseAt,omitempty"`
|
||||
|
||||
// Peers: every member's replicated load, for context.
|
||||
Nodes []BalanceNodeReport `json:"nodes"`
|
||||
}
|
||||
|
||||
type BalanceNodeReport struct {
|
||||
ID string `json:"id"`
|
||||
Alive bool `json:"alive"`
|
||||
IsLeader bool `json:"isLeader"`
|
||||
Forwards int `json:"forwards"`
|
||||
MemPct float64 `json:"memPct"`
|
||||
NetPct float64 `json:"netPct"`
|
||||
Score float64 `json:"score"`
|
||||
}
|
||||
|
||||
type BalanceReleasePlan struct {
|
||||
Local string `json:"local"`
|
||||
Remote string `json:"remote"`
|
||||
RemotePort int `json:"remotePort"`
|
||||
Owner string `json:"owner"`
|
||||
}
|
||||
|
||||
// Report renders the current view. It never mutates anything — in particular it
|
||||
// does NOT touch lastReleaseAt or the release state, so an operator polling this
|
||||
// cannot start, delay or cancel a release. (A read-only endpoint that perturbs
|
||||
// the behaviour it reports is worse than no endpoint.)
|
||||
func (b *Balancer) Report() BalanceReport {
|
||||
now := time.Now().Unix()
|
||||
rep := BalanceReport{
|
||||
Mode: b.mode,
|
||||
ModeSource: b.ModeSource(),
|
||||
Enabled: b.mode != BalanceOff,
|
||||
DryRun: b.mode != BalanceOn,
|
||||
CanControl: b.persist != nil,
|
||||
IsMember: b.engine.IsMember(),
|
||||
Self: b.engine.ID,
|
||||
CooldownSec: b.cooldownSec,
|
||||
}
|
||||
p := b.pool()
|
||||
rep.SelfForwards, rep.Share = p.mine, p.share
|
||||
rep.TotalForwards, rep.AliveNodes = p.total, p.alive
|
||||
rep.SelfOverShare = p.mine > p.share
|
||||
|
||||
if pick, _, blocked := b.assess(now); pick != nil {
|
||||
rep.WouldRelease = &BalanceReleasePlan{
|
||||
Local: pick.Local.Name, Remote: pick.Remote.Name,
|
||||
RemotePort: pick.Link.RemotePort, Owner: pick.OwnerID,
|
||||
}
|
||||
rep.BlockedBy = blocked
|
||||
}
|
||||
if m := b.releasing; m != nil {
|
||||
rep.Releasing = &BalanceReleasePlan{
|
||||
Local: m.local, Remote: m.remote, RemotePort: m.port, Owner: b.engine.ID,
|
||||
}
|
||||
rep.BlockedBy = "in-flight"
|
||||
}
|
||||
if b.lastReleaseAt > 0 {
|
||||
rep.LastReleaseAt = b.lastReleaseAt
|
||||
}
|
||||
for i := range b.engine.state.Nodes {
|
||||
n := &b.engine.state.Nodes[i]
|
||||
rep.Nodes = append(rep.Nodes, BalanceNodeReport{
|
||||
ID: n.ID, Alive: n.Alive, IsLeader: n.IsLeader,
|
||||
Forwards: n.Load.Forwards, MemPct: n.Load.MemPct, NetPct: n.Load.NetPct,
|
||||
Score: n.Load.Score(),
|
||||
})
|
||||
}
|
||||
return rep
|
||||
}
|
||||
195
internal/cluster/ring_disabled_test.go
Normal file
195
internal/cluster/ring_disabled_test.go
Normal file
@ -0,0 +1,195 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// staleTokenWith copies the engine's current state but rewrites the given
|
||||
// forward's disabled flag, standing in for a token a neighbour captured before
|
||||
// this node's decision.
|
||||
func staleTokenWith(eng *Engine, local, remote string, port int, disabled bool) State {
|
||||
s := eng.state
|
||||
out := State{
|
||||
LeaderID: s.LeaderID,
|
||||
Nodes: s.Nodes,
|
||||
Cycle: s.Cycle,
|
||||
Topology: map[string]*TopoEntry{},
|
||||
}
|
||||
for id, e := range s.Topology {
|
||||
cp := *e
|
||||
if cp.Local.Name == local && cp.Remote.Name == remote && cp.Link.RemotePort == port {
|
||||
cp.Link.Disabled = disabled
|
||||
cp.Active = !disabled
|
||||
}
|
||||
out.Topology[id] = &cp
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestLocalDisableSurvivesAdoption is the regression test for the wipe that made
|
||||
// the ring's authority unachievable.
|
||||
//
|
||||
// OnToken/AdoptState replace e.state wholesale. A stop decided between two token
|
||||
// cycles was therefore erased by the next adoption whenever the incoming token
|
||||
// had been captured before the decision — so the flag never reached the other
|
||||
// members and the forward kept running. Written as a probe BEFORE the fix: it
|
||||
// reported "after adopting a stale token: known=true disabled=false".
|
||||
func TestLocalDisableSurvivesAdoption(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Local decision: stop it.
|
||||
eng.UpdateTopologyDisabled("web", "frps1", 18081, true)
|
||||
if d, _ := eng.TopologyDisabled("web", "frps1", 18081); !d {
|
||||
t.Fatal("precondition: the local stop should be visible")
|
||||
}
|
||||
|
||||
// A token captured BEFORE the decision arrives.
|
||||
stale := staleTokenWith(eng, "web", "frps1", 18081, false)
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: stale, SentAt: 2}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Assert on the STATE, not the TopologyDisabled() accessor: the accessor
|
||||
// short-circuits to the pending local decision, so it would report the stop
|
||||
// even when the state about to be forwarded still says "enabled". What
|
||||
// matters is that the token this node sends on carries the decision.
|
||||
assertForwardedDisabled(t, eng, "web", "frps1", 18081, true)
|
||||
}
|
||||
|
||||
// assertForwardedDisabled checks the disabled value a DOWNSTREAM node would see
|
||||
// in the token this engine forwards — the value that actually propagates.
|
||||
func assertForwardedDisabled(t *testing.T, eng *Engine, local, remote string, port int, want bool) {
|
||||
t.Helper()
|
||||
// Model one hop: a fresh engine adopts exactly what this one would send.
|
||||
peer := newTestEngine("peer", false)
|
||||
peer.AdoptState(eng.state)
|
||||
got, known := peer.TopologyDisabled(local, remote, port)
|
||||
if !known {
|
||||
t.Fatalf("peer learned no entry for %s→%s:%d", local, remote, port)
|
||||
}
|
||||
if got != want {
|
||||
t.Fatalf("the forwarded token carries disabled=%v, want %v — the decision did not "+
|
||||
"propagate, so the other members would keep the old state", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLocalReEnableSurvivesAdoption: a re-enable needs the same protection. The
|
||||
// value false is easy to overlook — nothing "looks" stopped, yet losing it just
|
||||
// as silently strands the forward in the stopped state.
|
||||
func TestLocalReEnableSurvivesAdoption(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eng.UpdateTopologyDisabled("web", "frps1", 18081, true)
|
||||
eng.UpdateTopologyDisabled("web", "frps1", 18081, false)
|
||||
|
||||
// A token that still says disabled (captured mid-stop) arrives.
|
||||
mid := staleTokenWith(eng, "web", "frps1", 18081, true)
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: mid, SentAt: 2}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assertForwardedDisabled(t, eng, "web", "frps1", 18081, false)
|
||||
}
|
||||
|
||||
// TestConfirmedDecisionStopsBeingAsserted: once the ring reports the same value
|
||||
// the decision is confirmed everywhere, so this node must stop overriding —
|
||||
// otherwise it would fight a later decision made elsewhere.
|
||||
func TestConfirmedDecisionStopsBeingAsserted(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
eng.UpdateTopologyDisabled("web", "frps1", 18081, true)
|
||||
if len(eng.localDisabled) != 1 {
|
||||
t.Fatalf("a pending decision should be tracked, got %v", eng.localDisabled)
|
||||
}
|
||||
// The ring comes back agreeing.
|
||||
agreed := staleTokenWith(eng, "web", "frps1", 18081, true)
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: agreed, SentAt: 2}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(eng.localDisabled) != 0 {
|
||||
t.Fatalf("a confirmed decision must stop being asserted, still tracking %v", eng.localDisabled)
|
||||
}
|
||||
|
||||
// A peer now decides the opposite; this node must follow the ring.
|
||||
peerSaysEnabled := staleTokenWith(eng, "web", "frps1", 18081, false)
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 3, State: peerSaysEnabled, SentAt: 3}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if d, _ := eng.TopologyDisabled("web", "frps1", 18081); d {
|
||||
t.Fatal("this node kept overriding a peer's later decision; the ring is not authoritative")
|
||||
}
|
||||
}
|
||||
|
||||
// TestLocalDecisionsDoNotAccumulate: the tracking map is swept against the live
|
||||
// topology, so a long-lived cluster cannot grow it without bound.
|
||||
func TestLocalDecisionsDoNotAccumulate(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Decisions for forwards this node has never seen.
|
||||
eng.UpdateTopologyDisabled("ghost1", "frps1", 1, true)
|
||||
eng.UpdateTopologyDisabled("ghost2", "frps1", 2, true)
|
||||
if len(eng.localDisabled) != 2 {
|
||||
t.Fatalf("expected 2 tracked decisions, got %d", len(eng.localDisabled))
|
||||
}
|
||||
// A cycle runs: neither ghost is in the topology, so both are swept.
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state, SentAt: 2}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(eng.localDisabled) != 0 {
|
||||
t.Fatalf("decisions for forwards outside the topology must be swept, got %v", eng.localDisabled)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHasSelfInTopology is the wait condition for the startup reconcile: the
|
||||
// reconcile rebuilds the workers for forwards THIS node owns, so a node can only
|
||||
// act once the adopted state actually names it as an owner.
|
||||
//
|
||||
// It is deliberately independent of leadership — the bug it guards against was a
|
||||
// reconcile that waited for leadership and therefore never ran on followers, even
|
||||
// though followers own forwards.
|
||||
func TestHasSelfInTopology(t *testing.T) {
|
||||
// Two nodes; the forward is owned by n2, which is NOT the leader. A single-node
|
||||
// ring would make this test pass under either implementation, because the only
|
||||
// member is necessarily the leader — which is exactly how the first version of
|
||||
// this test came to be a false green.
|
||||
eng := newTestEngine("n2", false)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if eng.IsLeader() {
|
||||
t.Fatal("precondition: n2 must not be the leader in this test")
|
||||
}
|
||||
// n2 claimed it, so it owns a forward — despite not leading.
|
||||
if !eng.HasSelfInTopology() {
|
||||
t.Error("a non-leader that owns a forward reports false — the startup reconcile " +
|
||||
"would skip it and leave its worker down")
|
||||
}
|
||||
|
||||
// A node that owns nothing must report false (so it does not act early on a
|
||||
// stale or empty topology).
|
||||
other := newTestEngine("n9", true)
|
||||
if other.HasSelfInTopology() {
|
||||
t.Error("a node owning nothing must not claim it appears in the topology")
|
||||
}
|
||||
// Guard against a regression where leadership is folded into the predicate.
|
||||
if !eng.HasSelfInTopology() && eng.IsLeader() {
|
||||
t.Error("predicate must not be gated on leadership")
|
||||
}
|
||||
}
|
||||
@ -7,6 +7,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@ -18,6 +19,17 @@ import (
|
||||
type Handler interface {
|
||||
Claim(ctx context.Context, tk *Task) error
|
||||
Revoke(ctx context.Context, tk *Task) error
|
||||
// Restart re-enables an already-claimed forward and brings its worker back.
|
||||
// Needed because a stop keeps the topology entry, which makes both the
|
||||
// creation path (idempotency guard) and the duplicate-claim guard refuse to
|
||||
// re-create an existing forward.
|
||||
Restart(ctx context.Context, tk *Task) error
|
||||
// Release stops this node's worker for a forward it is giving up for
|
||||
// rebalancing, WITHOUT marking the link stopped. The forward is about to run
|
||||
// on another node, so a stopped flag would be wrong: every node's duplicate
|
||||
// guard reads it and would then refuse to claim the re-published task,
|
||||
// stranding the forward on the node that just gave it up.
|
||||
Release(ctx context.Context, tk *Task) error
|
||||
RuntimeLoad() Load
|
||||
}
|
||||
|
||||
@ -50,6 +62,24 @@ type Engine struct {
|
||||
// group changes made via HTTP handlers between token cycles are overwritten
|
||||
// by the next state adoption and never propagate to other nodes.
|
||||
topologySync func()
|
||||
// localDisabled records this node's own enabled/disabled decisions that have
|
||||
// not yet been confirmed by the ring. It is what makes the RING authoritative
|
||||
// without losing a decision that was made locally and has not yet had a
|
||||
// chance to reach the other members.
|
||||
//
|
||||
// The problem it solves: OnToken/AdoptState do `e.state = tk.State`, a
|
||||
// wholesale replacement. A stop decided between two token cycles therefore
|
||||
// disappears on the very next adoption if that token was captured before the
|
||||
// decision — the flag never reaches the rest of the ring, and the forward
|
||||
// keeps running. (Verified with a probe before writing this: adopting a
|
||||
// stale token flipped disabled back to false.)
|
||||
//
|
||||
// A key stays in this map until an adoption reports it disabled, at which
|
||||
// point the whole cluster agrees and the key is dropped. Keys for forwards
|
||||
// that vanish from the topology are swept on every update, so the map cannot
|
||||
// grow without bound. The value false is what keeps a RE-ENABLE alive for the
|
||||
// same reason a stop needs protecting.
|
||||
localDisabled map[string]bool
|
||||
|
||||
state State
|
||||
// myAddr maps our Node ID to the address peers dial.
|
||||
@ -182,7 +212,9 @@ func (e *Engine) OnToken(ctx context.Context, tk *Token) (*Token, error) {
|
||||
if tk.SentAt > e.lastTokenAt {
|
||||
e.lastTokenAt = tk.SentAt
|
||||
}
|
||||
log.Printf("ring[%s] OnToken cycle=%d", e.ID, tk.Cycle)
|
||||
// Per-round heartbeat — steady state, no information. See debug.go: this
|
||||
// fired ~2x/second and dominated the journal on every node.
|
||||
debugToken("ring[%s] OnToken cycle=%d", e.ID, tk.Cycle)
|
||||
|
||||
// Parallel rhythm timer: operations run while the pace clock ticks.
|
||||
// Delay scales with alive node count (more nodes → lower per-hop delay,
|
||||
@ -229,6 +261,11 @@ func (e *Engine) OnToken(ctx context.Context, tk *Token) (*Token, error) {
|
||||
// Re-apply local store overrides (group labels, disabled flags) onto
|
||||
// the freshly adopted topology so they survive state adoption and
|
||||
// propagate to all nodes via the next token forward.
|
||||
// Re-assert decisions this node made locally that the ring has not yet
|
||||
// confirmed, BEFORE the host's topology sync runs: the sync reads
|
||||
// TopologyDisabled to learn the cluster's view, so the local decision must
|
||||
// already be applied or a stop could be reported as "enabled" and dropped.
|
||||
e.reconcileLocalDisabled()
|
||||
if e.topologySync != nil {
|
||||
e.topologySync()
|
||||
}
|
||||
@ -375,6 +412,43 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
|
||||
}
|
||||
continue // owned elsewhere — ride to the owner
|
||||
}
|
||||
if t.Release {
|
||||
// Owner-directed, like a revoke: only the node holding the forward
|
||||
// may give it up. A release executed anywhere else would be an
|
||||
// attempt to move somebody else's load, which cannot work — the
|
||||
// owner's own count is what has to fall, and its topology entry is
|
||||
// local bookkeeping no peer may write.
|
||||
//
|
||||
// A release whose forward is already gone is an idempotent no-op
|
||||
// consumed by the lowest node, so it cannot ride forever if its
|
||||
// owner departed first.
|
||||
if owner := e.state.TopologyOwner(t); owner == e.ID {
|
||||
target = t
|
||||
break
|
||||
} else if owner == "" && selfIsLowest {
|
||||
target = t
|
||||
break
|
||||
}
|
||||
continue // owned elsewhere — ride to the owner
|
||||
}
|
||||
if t.Restart {
|
||||
// Owner-directed, exactly like a revoke: only the node holding the
|
||||
// forward may act. Selecting it anywhere else would let a non-owner
|
||||
// spawn a worker for somebody else's forward (observed live: a
|
||||
// non-owner logged "restarted t4" and ran the forward itself).
|
||||
//
|
||||
// A restart whose forward is already gone is a no-op consumed by the
|
||||
// lowest node, so a task can never ride forever if its owner
|
||||
// departed before seeing it.
|
||||
if owner := e.state.TopologyOwner(t); owner == e.ID {
|
||||
target = t
|
||||
break
|
||||
} else if owner == "" && selfIsLowest {
|
||||
target = t // forward gone — consume and re-enable nothing
|
||||
break
|
||||
}
|
||||
continue // owned elsewhere — ride to the owner
|
||||
}
|
||||
if selfIsLowest {
|
||||
target = t // generic forward create — lowest-load claim
|
||||
break
|
||||
@ -388,18 +462,37 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
|
||||
break
|
||||
}
|
||||
if claimed.Revoke {
|
||||
if e.state.RemoveTopology(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort) {
|
||||
if e.Handler != nil {
|
||||
if err := e.Handler.Revoke(ctx, claimed); err != nil {
|
||||
log.Printf("ring[%s] revoke %s: %v", e.ID, claimed.ID, err)
|
||||
}
|
||||
}
|
||||
if e.Log != nil {
|
||||
_, _ = e.Log.Append(e.ID, LogForwardRemove, map[string]any{
|
||||
"taskId": claimed.ID, "local": claimed.Local.Name, "remote": claimed.Remote.Name,
|
||||
})
|
||||
// A user-requested stop is recorded as a FLAG on the topology entry,
|
||||
// not as a removal. The entry is the carrier that makes the decision
|
||||
// travel: TopoEntry.Link is a full store.Link and State rides the
|
||||
// token every cycle, so keeping the entry is what lets a stop reach
|
||||
// every member — including a node that was offline when the stop was
|
||||
// issued. Removing the entry instead left the flag nowhere to live,
|
||||
// which is why a stop used to converge only if the owner happened to
|
||||
// be online to receive a one-shot revoke task.
|
||||
//
|
||||
// Marking Active=false is also what makes the entry inert for the
|
||||
// rebalancing paths: OfflineReassign() skips inactive entries, and
|
||||
// the startup reconcile skips them too, so a stopped forward is
|
||||
// neither re-queued on a node departure nor re-claimed on a restart.
|
||||
//
|
||||
// Handler.Revoke still runs unconditionally — it is what actually
|
||||
// stops the per-forward worker, and it must run even when there was no
|
||||
// entry to mark (gating it on RemoveTopology()'s result made stops of
|
||||
// already-absent forwards silently do nothing, so their workers ran
|
||||
// forever: observed live as thousands of connection-refused lines
|
||||
// against a local service that was intentionally down).
|
||||
marked := e.state.UpdateTopologyDisabled(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort, true)
|
||||
if e.Handler != nil {
|
||||
if err := e.Handler.Revoke(ctx, claimed); err != nil {
|
||||
log.Printf("ring[%s] revoke %s: %v", e.ID, claimed.ID, err)
|
||||
}
|
||||
}
|
||||
if marked && e.Log != nil {
|
||||
_, _ = e.Log.Append(e.ID, LogForwardStop, map[string]any{
|
||||
"taskId": claimed.ID, "local": claimed.Local.Name, "remote": claimed.Remote.Name,
|
||||
})
|
||||
}
|
||||
continue
|
||||
}
|
||||
if claimed.RemoveNode != "" && claimed.RemoveNode == e.ID {
|
||||
@ -465,10 +558,90 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
|
||||
// different node. Safe for OfflineReassign: that path deletes the
|
||||
// topology entry BEFORE re-queueing, so TopologyOwner returns "" and
|
||||
// the legitimate re-claim passes through.
|
||||
if owner := e.state.TopologyOwner(claimed); owner != "" {
|
||||
log.Printf("ring[%s] drop duplicate task %s: %s→%s:%d already owned by %s",
|
||||
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
|
||||
claimed.Link.RemotePort, owner)
|
||||
//
|
||||
// A RESTART task is the deliberate exception this guard must not eat: it
|
||||
// exists precisely to re-activate a forward whose entry is still present
|
||||
// (a stop keeps the entry as the flag's carrier), so applying the guard
|
||||
// would swallow every re-enable and leave the forward stopped forever.
|
||||
// A RELEASE is the second deliberate exception, for the same reason as a
|
||||
// restart: it is addressed to the node that ALREADY owns the forward, so
|
||||
// the entry is still there by definition. Applying the guard would drop
|
||||
// every release and a node could never shed anything.
|
||||
if !claimed.Restart && !claimed.Release {
|
||||
if owner := e.state.TopologyOwner(claimed); owner != "" {
|
||||
log.Printf("ring[%s] drop duplicate task %s: %s→%s:%d already owned by %s",
|
||||
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
|
||||
claimed.Link.RemotePort, owner)
|
||||
continue
|
||||
}
|
||||
}
|
||||
if claimed.Release {
|
||||
// Only reached when this node owns the forward (or it is already gone
|
||||
// and we are the fallback consumer — see the selection loop).
|
||||
owner := e.state.TopologyOwner(claimed)
|
||||
if owner != e.ID {
|
||||
// The forward vanished before we got here; nothing to give up.
|
||||
continue
|
||||
}
|
||||
// Stop our worker. Deliberately NOT Handler.Revoke: that marks the
|
||||
// link stopped, and the forward is about to run elsewhere — a
|
||||
// "stopped" flag would make every node treat the re-published task as
|
||||
// a duplicate and refuse to claim it.
|
||||
if e.Handler != nil {
|
||||
if err := e.Handler.Release(ctx, claimed); err != nil {
|
||||
log.Printf("ring[%s] release %s: %v", e.ID, claimed.ID, err)
|
||||
}
|
||||
}
|
||||
// Drop the entry so the forward has no owner. This is what makes the
|
||||
// load actually fall (ForwardsOwnedBy counts entries) and what lets a
|
||||
// fresh creation task pass the duplicate guard above.
|
||||
e.RemoveTopologyEntry(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort)
|
||||
// Hand it back to the ring as an ordinary unclaimed forward. It is
|
||||
// published by THIS node so it rides the very token we are holding —
|
||||
// no coordinator, no addressed hand-off.
|
||||
link := claimed.Link
|
||||
link.Disabled = false
|
||||
if t := e.SubmitTask(claimed.Local, claimed.Remote, link); t != nil {
|
||||
log.Printf("ring[%s] released %s: %s→%s:%d back to the ring as %s",
|
||||
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
|
||||
claimed.Link.RemotePort, t.ID)
|
||||
} else {
|
||||
log.Printf("ring[%s] released %s but the re-publish was deduped",
|
||||
e.ID, claimed.ID)
|
||||
}
|
||||
if e.Log != nil {
|
||||
_, _ = e.Log.Append(e.ID, LogForwardStop, map[string]any{
|
||||
"taskId": claimed.ID, "local": claimed.Local.Name,
|
||||
"remote": claimed.Remote.Name, "released": true,
|
||||
})
|
||||
}
|
||||
continue
|
||||
}
|
||||
if claimed.Restart {
|
||||
// Only reached when this node owns the forward (or it is already gone
|
||||
// and we are the fallback consumer — see the selection loop).
|
||||
owner := e.state.TopologyOwner(claimed)
|
||||
e.state.UpdateTopologyDisabled(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort, false)
|
||||
if owner != e.ID {
|
||||
// Forward vanished before we got here: nothing to re-enable.
|
||||
log.Printf("ring[%s] discard restart %s: no owner", e.ID, claimed.ID)
|
||||
continue
|
||||
}
|
||||
if e.Handler != nil {
|
||||
if err := e.Handler.Restart(ctx, claimed); err != nil {
|
||||
e.state.PendingTasks[claimed.ID] = claimed
|
||||
log.Printf("ring[%s] restart %s failed: %v", e.ID, claimed.ID, err)
|
||||
break
|
||||
}
|
||||
}
|
||||
if e.Log != nil {
|
||||
_, _ = e.Log.Append(e.ID, LogForwardStart, map[string]any{
|
||||
"taskId": claimed.ID, "local": claimed.Local.Name, "remote": claimed.Remote.Name,
|
||||
})
|
||||
}
|
||||
// Logging is the app's job (RestartFn reports the actual worker
|
||||
// outcome); the ring layer stays quiet so one restart is one line
|
||||
// rather than two identical ones.
|
||||
continue
|
||||
}
|
||||
if e.Handler != nil {
|
||||
@ -724,6 +897,11 @@ func (e *Engine) AdoptState(s State) {
|
||||
for id, t := range kept {
|
||||
e.state.PendingTasks[id] = t
|
||||
}
|
||||
// Re-assert decisions this node made locally that the ring has not yet
|
||||
// confirmed, BEFORE the host's topology sync runs: the sync reads
|
||||
// TopologyDisabled to learn the cluster's view, so the local decision must
|
||||
// already be applied or a stop could be reported as "enabled" and dropped.
|
||||
e.reconcileLocalDisabled()
|
||||
if e.topologySync != nil {
|
||||
e.topologySync()
|
||||
}
|
||||
@ -792,6 +970,14 @@ func (e *Engine) RemoveNode(nodeID string) *Task {
|
||||
return e.state.AddRemoveNode(nodeID)
|
||||
}
|
||||
|
||||
// RemoveTopologyEntry drops the active topology entry for a forward identified
|
||||
// by its natural key. Exposed so the app's claim path can retire an entry it
|
||||
// refuses to serve (see ClaimFn: a disabled forward must stop looking active,
|
||||
// otherwise the startup reconcile keeps re-claiming it on every restart).
|
||||
func (e *Engine) RemoveTopologyEntry(local, remote string, port int) bool {
|
||||
return e.state.RemoveTopology(local, remote, port)
|
||||
}
|
||||
|
||||
// RevokeTask publishes a revocation for an established forward through the
|
||||
// same token channel; the owning node stops the worker and drops topology.
|
||||
func (e *Engine) RevokeTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||
@ -799,14 +985,107 @@ func (e *Engine) RevokeTask(local store.Local, remote store.Remote, link store.L
|
||||
}
|
||||
|
||||
func (e *Engine) SubmitTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||
if e.HasTask(local.Name, remote.Name, link.RemotePort) {
|
||||
// Guard on an ACTIVE task only. A topology entry that is merely marked
|
||||
// disabled is kept around as the carrier for the stop flag, so counting it
|
||||
// here would make this a permanent no-op and a stopped forward could never
|
||||
// be started again.
|
||||
if e.HasActiveTask(local.Name, remote.Name, link.RemotePort) {
|
||||
return nil
|
||||
}
|
||||
return e.state.AddPending(local, remote, link)
|
||||
}
|
||||
|
||||
// HasTask reports whether a forward with the same local/remote/remotePort is
|
||||
// already pending or active in the topology (idempotency guard for resaves).
|
||||
// HasTopologyEntry reports whether the forward has a topology entry (whether or
|
||||
// not it is marked disabled). A stop keeps the entry as the flag's carrier, so
|
||||
// "has an entry" is what distinguishes "already claimed by someone" from "never
|
||||
// submitted" — the distinction startForward needs.
|
||||
func (e *Engine) HasTopologyEntry(local, remote string, port int) bool {
|
||||
_, found := e.state.TopologyDisabled(local, remote, port)
|
||||
return found
|
||||
}
|
||||
|
||||
// SubmitRestart publishes a task asking the current owner of an already-claimed
|
||||
// forward to bring its worker back up.
|
||||
//
|
||||
// Neither of the existing channels can express "restart what already exists":
|
||||
// SubmitTask is guarded by HasActiveTask and the entry is still present, so it
|
||||
// dedupes; and the claim path drops any task for a forward that already has an
|
||||
// owner (its defence against duplicate-claim collisions). Re-enabling a stopped
|
||||
// forward therefore needs its own task kind, which the owner applies without
|
||||
// re-running the claim bookkeeping.
|
||||
func (e *Engine) SubmitRestart(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||
// The task must not carry the stop: the whole point is to re-enable.
|
||||
link.Disabled = false
|
||||
t := &Task{
|
||||
ID: e.state.NextTaskID(),
|
||||
Local: local,
|
||||
Remote: remote,
|
||||
Link: link,
|
||||
Created: time.Now().Unix(),
|
||||
Restart: true,
|
||||
}
|
||||
if e.state.PendingTasks == nil {
|
||||
e.state.PendingTasks = map[string]*Task{}
|
||||
}
|
||||
e.state.PendingTasks[t.ID] = t
|
||||
return t
|
||||
}
|
||||
|
||||
// ReleaseTask publishes a rebalancing task asking the OWNER of this forward to
|
||||
// give it up. The task rides the token like any other command; it is not
|
||||
// addressed to a destination, because the ring's normal lowest-load claim
|
||||
// decides where the forward goes next.
|
||||
//
|
||||
// It goes through AddPending rather than writing PendingTasks directly. That is
|
||||
// not cosmetic: OnToken rebuilds local-only pending commands from the ids it
|
||||
// never published, and only the Add* helpers record them — a task injected by
|
||||
// hand was silently dropped at the next state adoption, so it never reached the
|
||||
// ring at all.
|
||||
func (e *Engine) ReleaseTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||
link.Disabled = false
|
||||
t := e.state.AddPending(local, remote, link)
|
||||
if t == nil {
|
||||
return nil
|
||||
}
|
||||
t.Release = true
|
||||
return t
|
||||
}
|
||||
|
||||
// HasActiveTask reports whether a forward with this key is genuinely in flight
|
||||
// or running: a non-revocation pending task, or a topology entry that is not
|
||||
// marked disabled.
|
||||
//
|
||||
// It is deliberately narrower than HasTask. Under mark-don't-remove a stopped
|
||||
// forward keeps its topology entry (that entry is what carries the flag between
|
||||
// nodes), so "an entry exists" no longer means "this forward is active".
|
||||
// SubmitTask must use THIS predicate, otherwise starting a stopped forward —
|
||||
// startForward() publishes the enable and then calls SubmitTask — would be
|
||||
// swallowed by its own idempotency guard.
|
||||
func (e *Engine) HasActiveTask(local, remote string, port int) bool {
|
||||
for _, t := range e.state.PendingList() {
|
||||
if t.Revoke {
|
||||
continue // a revocation is the opposite of an active forward
|
||||
}
|
||||
if t.Local.Name == local && t.Remote.Name == remote && t.Link.RemotePort == port {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, t := range e.state.TopologyList() {
|
||||
if t.Link.Disabled {
|
||||
continue // stopped; the entry is only a flag carrier
|
||||
}
|
||||
if t.Local.Name == local && t.Remote.Name == remote && t.Link.RemotePort == port {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// HasTask reports whether ANY record for this forward exists — a pending task
|
||||
// (including an in-flight revocation) or a topology entry (including one marked
|
||||
// disabled). This is the "have we already handled this key" question, used to
|
||||
// avoid re-issuing work on repeated canvas saves. For "is it actually running"
|
||||
// use HasActiveTask instead.
|
||||
func (e *Engine) HasTask(local, remote string, port int) bool {
|
||||
for _, t := range e.state.PendingList() {
|
||||
if t.Local.Name == local && t.Remote.Name == remote && t.Link.RemotePort == port {
|
||||
@ -828,6 +1107,78 @@ func (e *Engine) UpdateTopologyGroup(local, remote string, port int, group strin
|
||||
return e.state.UpdateTopologyGroup(local, remote, port, group)
|
||||
}
|
||||
|
||||
// UpdateTopologyDisabled marks a forward enabled/disabled in the topology so
|
||||
// the decision rides the next token cycle to every member. Paired with
|
||||
// TopologyDisabled, which the adoption hook uses to learn peers' decisions.
|
||||
//
|
||||
// The decision is also remembered locally until the ring confirms it, because
|
||||
// adoption replaces the whole state: a token captured before this call would
|
||||
// otherwise wash the decision out on the next cycle and it would never reach
|
||||
// the other members. See the localDisabled field.
|
||||
func (e *Engine) UpdateTopologyDisabled(local, remote string, port int, disabled bool) bool {
|
||||
if e.localDisabled == nil {
|
||||
e.localDisabled = map[string]bool{}
|
||||
}
|
||||
e.localDisabled[disableKey(local, remote, port)] = disabled
|
||||
return e.state.UpdateTopologyDisabled(local, remote, port, disabled)
|
||||
}
|
||||
|
||||
// TopologyDisabled reports the cluster's view of a forward's disabled flag.
|
||||
// The bool is false when the forward has no topology entry (nothing to learn).
|
||||
//
|
||||
// A pending local decision takes precedence over the adopted state: it has not
|
||||
// had a chance to reach the other members yet, and reporting the adopted value
|
||||
// would make the local store (and the status page) disagree with the user's
|
||||
// most recent action.
|
||||
func (e *Engine) TopologyDisabled(local, remote string, port int) (bool, bool) {
|
||||
if d, pending := e.localDisabled[disableKey(local, remote, port)]; pending {
|
||||
return d, true
|
||||
}
|
||||
return e.state.TopologyDisabled(local, remote, port)
|
||||
}
|
||||
|
||||
// reconcileLocalDisabled re-asserts this node's not-yet-confirmed disabled
|
||||
// decisions onto the freshly adopted state. Called right after e.state is
|
||||
// replaced, and paired with reconcileTopologySync (which pushes those decisions
|
||||
// out to the ring).
|
||||
//
|
||||
// A decision is dropped once the adopted state reports the SAME value: at that
|
||||
// point every member agrees and there is nothing left to protect. Entries whose
|
||||
// forward no longer exists in the topology are dropped too, so the map tracks
|
||||
// only live disagreements.
|
||||
func (e *Engine) reconcileLocalDisabled() {
|
||||
if len(e.localDisabled) == 0 {
|
||||
return
|
||||
}
|
||||
for id, te := range e.state.Topology {
|
||||
_ = id
|
||||
k := disableKey(te.Local.Name, te.Remote.Name, te.Link.RemotePort)
|
||||
want, pending := e.localDisabled[k]
|
||||
if !pending {
|
||||
continue
|
||||
}
|
||||
if te.Link.Disabled == want {
|
||||
// The ring caught up (or agreed independently): stop tracking.
|
||||
delete(e.localDisabled, k)
|
||||
continue
|
||||
}
|
||||
// Still divergent: keep asserting our decision onto the adopted state.
|
||||
te.Link.Disabled = want
|
||||
te.Active = !want
|
||||
}
|
||||
// Forget decisions for forwards that left the topology entirely — otherwise
|
||||
// a long-lived cluster would accumulate dead keys.
|
||||
live := make(map[string]struct{}, len(e.state.Topology))
|
||||
for _, te := range e.state.Topology {
|
||||
live[disableKey(te.Local.Name, te.Remote.Name, te.Link.RemotePort)] = struct{}{}
|
||||
}
|
||||
for k := range e.localDisabled {
|
||||
if _, ok := live[k]; !ok {
|
||||
delete(e.localDisabled, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// IsLeader reports whether this node is the current ring leader.
|
||||
func (e *Engine) IsLeader() bool { return e.state.LeaderID == e.ID }
|
||||
|
||||
@ -850,6 +1201,14 @@ func (e *Engine) SetPeerPersist(fn func(peersJSON string) error) { e.peerPersist
|
||||
// HTTP handlers are overwritten by the next e.state = tk.State.
|
||||
func (e *Engine) SetTopologySync(fn func()) { e.topologySync = fn }
|
||||
|
||||
// disableKey builds the map key for a forward's disabled decision: the natural
|
||||
// triple (local, remote, remotePort), which is how every other part of the code
|
||||
// identifies a forward. A NUL separator keeps it unambiguous for names that
|
||||
// could otherwise collide across the boundaries.
|
||||
func disableKey(local, remote string, port int) string {
|
||||
return local + "\x00" + remote + "\x00" + strconv.Itoa(port)
|
||||
}
|
||||
|
||||
// persistPeers extracts all alive peers (addr + nodeKey, excluding self)
|
||||
// from the current ring state and persists them via the peerPersist callback.
|
||||
// Called on every token cycle (OnToken) and on AdoptState so a crashed node
|
||||
@ -938,3 +1297,20 @@ func (e *Engine) LeaderAddr() string {
|
||||
}
|
||||
return e.state.Nodes[i].Addr
|
||||
}
|
||||
|
||||
// HasSelfInTopology reports whether the current topology names this node as the
|
||||
// owner of at least one active forward.
|
||||
//
|
||||
// The startup reconcile uses it as its wait condition: a node can only rebuild
|
||||
// the workers for forwards it owns once the adopted state says it owns them.
|
||||
// Leadership is deliberately NOT part of the test — followers own forwards too,
|
||||
// and gating the reconcile on leadership left them permanently unable to restore
|
||||
// their own workers.
|
||||
func (e *Engine) HasSelfInTopology() bool {
|
||||
for _, te := range e.state.TopologyList() {
|
||||
if te.OwnerID == e.ID && te.Active {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
@ -8,9 +8,18 @@ import (
|
||||
)
|
||||
|
||||
type fakeHandler struct {
|
||||
load Load
|
||||
claim func(ctx context.Context, tk *Task) error
|
||||
revoke func(ctx context.Context, tk *Task) error
|
||||
load Load
|
||||
claim func(ctx context.Context, tk *Task) error
|
||||
revoke func(ctx context.Context, tk *Task) error
|
||||
restart func(ctx context.Context, tk *Task) error
|
||||
release func(ctx context.Context, tk *Task) error
|
||||
}
|
||||
|
||||
func (h *fakeHandler) Release(ctx context.Context, tk *Task) error {
|
||||
if h.release != nil {
|
||||
return h.release(ctx, tk)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *fakeHandler) Revoke(ctx context.Context, tk *Task) error {
|
||||
@ -20,6 +29,13 @@ func (h *fakeHandler) Revoke(ctx context.Context, tk *Task) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *fakeHandler) Restart(ctx context.Context, tk *Task) error {
|
||||
if h.restart != nil {
|
||||
return h.restart(ctx, tk)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *fakeHandler) RuntimeLoad() Load { return h.load }
|
||||
func (h *fakeHandler) Claim(ctx context.Context, tk *Task) error {
|
||||
if h.claim != nil {
|
||||
|
||||
@ -17,6 +17,16 @@ type AppHandler struct {
|
||||
ClaimFn func(ctx context.Context, tk *Task) error
|
||||
// RevokeFn cancels the forward on this node (stop worker + drop from store).
|
||||
RevokeFn func(ctx context.Context, tk *Task) error
|
||||
// RestartFn re-enables an already-claimed forward on this node (clear the
|
||||
// stopped flag + bring the worker back). Required because a stop keeps the
|
||||
// topology entry, so the normal claim path will not re-create it.
|
||||
RestartFn func(ctx context.Context, tk *Task) error
|
||||
// ReleaseFn stops this node's worker for a forward being handed back to the
|
||||
// ring, leaving the link ENABLED. When nil, Release falls back to RevokeFn —
|
||||
// which also stops the worker, but additionally marks the link stopped; that
|
||||
// is the pre-existing stop semantics and is wrong for a rebalance, so the app
|
||||
// should provide this.
|
||||
ReleaseFn func(ctx context.Context, tk *Task) error
|
||||
}
|
||||
|
||||
// RuntimeLoad implements Handler.
|
||||
@ -44,6 +54,30 @@ func (a *AppHandler) Revoke(ctx context.Context, tk *Task) error {
|
||||
return a.RevokeFn(ctx, tk)
|
||||
}
|
||||
|
||||
// Release implements Handler.
|
||||
func (a *AppHandler) Release(ctx context.Context, tk *Task) error {
|
||||
if a.ReleaseFn != nil {
|
||||
return a.ReleaseFn(ctx, tk)
|
||||
}
|
||||
// Fall back to the stop path so a host that has not provided ReleaseFn still
|
||||
// stops the worker (the forward keeps running otherwise). The link ends up
|
||||
// flagged stopped, which the claim path may then refuse — the host is warned
|
||||
// by the ring's own "re-publish was deduped" log line rather than silently
|
||||
// leaving two workers up.
|
||||
if a.RevokeFn != nil {
|
||||
return a.RevokeFn(ctx, tk)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Restart implements Handler.
|
||||
func (a *AppHandler) Restart(ctx context.Context, tk *Task) error {
|
||||
if a.RestartFn == nil {
|
||||
return nil
|
||||
}
|
||||
return a.RestartFn(ctx, tk)
|
||||
}
|
||||
|
||||
// SampleMemLoad returns a cheap memory-usage percentage (0..100).
|
||||
func SampleMemLoad() float64 {
|
||||
var m runtime.MemStats
|
||||
|
||||
@ -128,7 +128,7 @@ func (e *Engine) forwardToNext(ctx context.Context, tk *Token) error {
|
||||
if e.send == nil {
|
||||
return nil
|
||||
}
|
||||
log.Printf("ring[%s] forward cycle=%d to %s", e.ID, tk.Cycle, next)
|
||||
debugToken("ring[%s] forward cycle=%d to %s", e.ID, tk.Cycle, next)
|
||||
err := e.send(ctx, next, tk)
|
||||
if err == nil {
|
||||
if e.state.LeaderID == e.ID {
|
||||
@ -146,7 +146,14 @@ func (e *Engine) forwardToNext(ctx context.Context, tk *Token) error {
|
||||
// the token is lost — start fresh as the new leader (plan §leader
|
||||
// 补充/监控: "上家邻居探测到 leader 崩溃 → 自身成为新 leader").
|
||||
if next == e.state.LeaderID {
|
||||
e.becomeLeader()
|
||||
if !e.becomeLeader() {
|
||||
// Another live leader exists (we are the partitioned side):
|
||||
// keep its ring intact and let it re-drive the token. Re-raise
|
||||
// the node we just marked dead only if the send really was the
|
||||
// problem — otherwise we would paper over a real outage.
|
||||
log.Printf("ring[%s] not promoting: deferring to live leader", e.ID)
|
||||
return nil
|
||||
}
|
||||
if e.Log != nil {
|
||||
_, _ = e.Log.Append(e.ID, LogLeaderChange, map[string]string{"leader": e.ID})
|
||||
}
|
||||
@ -161,13 +168,42 @@ func (e *Engine) forwardToNext(ctx context.Context, tk *Token) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// becomeLeader promotes this node (used when the monitored leader dies).
|
||||
func (e *Engine) becomeLeader() {
|
||||
// becomeLeader promotes this node, but ONLY when the ring agrees there is no
|
||||
// live leader to defer to.
|
||||
//
|
||||
// The guard matters because leadership was previously self-declared: any node
|
||||
// that failed to reach the leader it believed in would immediately promote
|
||||
// itself. Two nodes partitioned from each other both hit that path and both
|
||||
// ended up believing they were the leader — a split brain with two token
|
||||
// rings, duplicate claims, and each side reassigning the other's forwards. That
|
||||
// was observed live: after a rolling restart, two nodes reported the same
|
||||
// leader while marking each other dead, and the topology was emptied into
|
||||
// pending on both sides.
|
||||
//
|
||||
// Before promoting, ask the ring: is any OTHER node currently alive and claiming
|
||||
// leadership? If so, we are the partitioned one — demote ourselves and let the
|
||||
// live leader keep the ring. A genuine single-leader loss (the real leader is
|
||||
// dead or unreachable) leaves no such node, so promotion still happens.
|
||||
//
|
||||
// CreateCluster does not come through here: it assigns LeaderID directly, so
|
||||
// the "no other leader exists" case never needs an override.
|
||||
func (e *Engine) becomeLeader() bool {
|
||||
for i := range e.state.Nodes {
|
||||
n := &e.state.Nodes[i]
|
||||
if n.ID == e.ID {
|
||||
continue
|
||||
}
|
||||
if n.Alive && n.IsLeader {
|
||||
log.Printf("ring[%s] decline promotion: %s is alive and is the leader", e.ID, n.ID)
|
||||
return false
|
||||
}
|
||||
}
|
||||
e.state.LeaderID = e.ID
|
||||
for i := range e.state.Nodes {
|
||||
e.state.Nodes[i].IsLeader = e.state.Nodes[i].ID == e.ID
|
||||
}
|
||||
log.Printf("ring[%s] promoted to leader", e.ID)
|
||||
return true
|
||||
}
|
||||
|
||||
// WatchLeader runs the FALLBACK leader liveness monitor. The PRIMARY path is
|
||||
@ -209,20 +245,43 @@ func (e *Engine) WatchLeader(ctx context.Context) {
|
||||
cancel()
|
||||
if err != nil {
|
||||
log.Printf("ring[%s] heartbeat to leader %s failed: %v", e.ID, e.state.LeaderID, err)
|
||||
e.state.MarkOffline(e.state.LeaderID)
|
||||
e.state.OfflineReassign(e.state.LeaderID)
|
||||
e.becomeLeader()
|
||||
// Kick off a fresh cycle: the ring died with the old
|
||||
// leader (no token inflight → WatchTokenLoss won't
|
||||
// fire). Without this the newly promoted leader would
|
||||
// sit idle and the ring would stay dead.
|
||||
e.StartRing(ctx)
|
||||
e.takeOverFromDeadLeader(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// takeOverFromDeadLeader handles "the leader I could not reach is gone": mark it
|
||||
// dead and, if the ring agrees there is no live leader left, promote this node
|
||||
// and re-seed the cycle.
|
||||
//
|
||||
// The promotion is deliberately conditional, and so is the reassignment. Doing
|
||||
// the reassignment unconditionally is what destroyed the live ring: a node on
|
||||
// the partitioned side would mark the (still very much alive) leader dead and
|
||||
// move ALL of its forwards into pending, so both sides ended up with an empty
|
||||
// topology and a queue nobody was serving. Sequencing matters — the forward
|
||||
// re-hang is only correct once we have actually taken leadership.
|
||||
//
|
||||
// Extracted from WatchLeader's loop so the decision is testable (the loop itself
|
||||
// is an endless goroutine).
|
||||
func (e *Engine) takeOverFromDeadLeader(ctx context.Context) {
|
||||
old := e.state.LeaderID
|
||||
e.state.MarkOffline(old)
|
||||
if !e.becomeLeader() {
|
||||
log.Printf("ring[%s] leader %s unreachable but a live leader remains; "+
|
||||
"declining takeover and leaving its forwards untouched", e.ID, old)
|
||||
return
|
||||
}
|
||||
e.state.OfflineReassign(old)
|
||||
if e.Log != nil {
|
||||
_, _ = e.Log.Append(e.ID, LogLeaderChange, map[string]string{"leader": e.ID})
|
||||
}
|
||||
// The ring died with the old leader (no token inflight → WatchTokenLoss
|
||||
// won't fire). Without this the newly promoted leader would sit idle.
|
||||
e.StartRing(ctx)
|
||||
}
|
||||
|
||||
// WatchTokenLoss runs the leader's token-loss judge: if a token was sent and
|
||||
// does not return within LossTimeout, the leader issues a fresh token (all
|
||||
// nodes drop older stamps via the SentAt guard, so at most one circulates).
|
||||
|
||||
@ -20,6 +20,8 @@ import (
|
||||
// LogKind enumerates operation-log entry types.
|
||||
const (
|
||||
LogForwardAdd = "forward.add"
|
||||
LogForwardStop = "forward.stop"
|
||||
LogForwardStart = "forward.start"
|
||||
LogForwardRemove = "forward.remove"
|
||||
LogNodeJoin = "node.join"
|
||||
LogNodeLeave = "node.leave"
|
||||
@ -140,7 +142,7 @@ func DetailOf(e LogEntry) string {
|
||||
}
|
||||
str := func(key string) string { s, _ := d[key].(string); return s }
|
||||
switch e.Kind {
|
||||
case LogForwardAdd, LogForwardRemove:
|
||||
case LogForwardAdd, LogForwardStop, LogForwardStart, LogForwardRemove:
|
||||
s := str("local") + " → " + str("remote")
|
||||
if id := str("taskId"); id != "" {
|
||||
if len(id) > 8 {
|
||||
|
||||
@ -2,14 +2,22 @@ package cluster
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// TestRevokeTaskRemovesTopology: a revoke task published to the ring removes
|
||||
// the forward from topology, calls Handler.Revoke, and logs forward.remove.
|
||||
func TestRevokeTaskRemovesTopology(t *testing.T) {
|
||||
// TestRevokeTaskMarksTopologyDisabled: a stop delivered through the ring marks
|
||||
// the topology entry disabled instead of deleting it, calls Handler.Revoke, and
|
||||
// logs forward.stop.
|
||||
//
|
||||
// The entry is deliberately KEPT: TopoEntry.Link is the carrier that makes the
|
||||
// stop travel, and State rides the token every cycle. Keeping it is what lets a
|
||||
// stop reach a member that was offline when the stop was issued — deleting the
|
||||
// entry left the flag nowhere to live, so a stop converged only if the owning
|
||||
// node happened to be online for a one-shot revoke task.
|
||||
func TestRevokeTaskMarksTopologyDisabled(t *testing.T) {
|
||||
revoked := false
|
||||
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||
@ -26,33 +34,92 @@ func TestRevokeTaskRemovesTopology(t *testing.T) {
|
||||
if len(eng.state.TopologyList()) != 1 {
|
||||
t.Fatalf("topology after claim = %+v", eng.state.TopologyList())
|
||||
}
|
||||
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); d {
|
||||
t.Fatal("a freshly claimed forward must not start out disabled")
|
||||
}
|
||||
|
||||
// publish a revoke task pointing at the same forward
|
||||
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(eng.state.TopologyList()) != 0 {
|
||||
t.Fatalf("topology after revoke = %+v", eng.state.TopologyList())
|
||||
|
||||
// The entry must SURVIVE, marked disabled, so the flag keeps travelling.
|
||||
if len(eng.state.TopologyList()) != 1 {
|
||||
t.Fatalf("topology entry was removed; the disabled flag would have no carrier: %+v",
|
||||
eng.state.TopologyList())
|
||||
}
|
||||
disabled, known := eng.state.TopologyDisabled("web", "frps1", 18081)
|
||||
if !known {
|
||||
t.Fatal("topology entry missing after revoke")
|
||||
}
|
||||
if !disabled {
|
||||
t.Fatal("revoke did not mark the topology entry disabled")
|
||||
}
|
||||
// Active must follow the flag, since OfflineReassign only re-queues Active
|
||||
// entries — a stopped forward must not be resurrected by a node departure.
|
||||
if eng.state.TopologyList()[0].Active {
|
||||
t.Fatal("a stopped forward must not remain Active (OfflineReassign would re-queue it)")
|
||||
}
|
||||
if !revoked {
|
||||
t.Fatal("Handler.Revoke was not called")
|
||||
}
|
||||
// log should contain forward.remove
|
||||
var sawRemove bool
|
||||
// log should contain forward.stop (not forward.remove — nothing was removed)
|
||||
var sawStop bool
|
||||
for _, e := range eng.Log.Snapshot() {
|
||||
if e.Kind == LogForwardRemove {
|
||||
sawRemove = true
|
||||
if e.Kind == LogForwardStop {
|
||||
sawStop = true
|
||||
}
|
||||
}
|
||||
if !sawRemove {
|
||||
t.Fatalf("log missing forward.remove: %+v", eng.Log.Snapshot())
|
||||
if !sawStop {
|
||||
t.Fatalf("log missing forward.stop: %+v", eng.Log.Snapshot())
|
||||
}
|
||||
}
|
||||
|
||||
// TestStoppedTopologyEntrySurvivesAdoption is the property that makes the whole
|
||||
// mark-don't-remove design work: because the entry persists, a node that adopts
|
||||
// ring state learns the stop. This is the offline-member case a one-shot revoke
|
||||
// task could never cover.
|
||||
func TestStoppedTopologyEntrySurvivesAdoption(t *testing.T) {
|
||||
src := newTestEngine("n1", true)
|
||||
src.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := src.OnToken(context.Background(), &Token{Cycle: 1, State: src.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := src.OnToken(context.Background(), &Token{Cycle: 2, State: src.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// A different node adopts the ring state (what e.state = tk.State does).
|
||||
peer := newTestEngine("n2", false)
|
||||
peer.AdoptState(src.state)
|
||||
|
||||
disabled, known := peer.TopologyDisabled("web", "frps1", 18081)
|
||||
if !known {
|
||||
t.Fatal("peer did not receive the topology entry for the stopped forward")
|
||||
}
|
||||
if !disabled {
|
||||
t.Fatal("peer adopted the entry but no longer sees it as disabled — the stop did not propagate")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRevokeIdempotent: revoking an already-missing forward does not error.
|
||||
//
|
||||
// It must ALSO still call Handler.Revoke. The handler is what actually stops
|
||||
// the per-forward frpc worker; the topology entry is only bookkeeping. Gating
|
||||
// the handler on RemoveTopology()'s return value (as this test used to permit)
|
||||
// turned every revoke of an already-absent forward into a silent no-op — the
|
||||
// worker kept running, which is how a forward the user had stopped kept
|
||||
// dialling a local service that was intentionally down, forever.
|
||||
func TestRevokeIdempotent(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
revoked := 0
|
||||
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||
revoke: func(ctx context.Context, tk *Task) error { revoked++; return nil }},
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||
"n1:7500", true, "")
|
||||
|
||||
eng.state.AddRevoke(store.Local{Name: "ghost"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 1})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||
t.Fatalf("revoke missing: %v", err)
|
||||
@ -61,4 +128,318 @@ func TestRevokeIdempotent(t *testing.T) {
|
||||
if len(eng.state.TopologyList()) != 0 {
|
||||
t.Fatal("should be empty")
|
||||
}
|
||||
// The stop side-effect must have happened even though there was no entry.
|
||||
if revoked != 1 {
|
||||
t.Fatalf("Handler.Revoke called %d times, want 1 — a revoke with no topology entry "+
|
||||
"must still stop the worker, otherwise stopped forwards keep running", revoked)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStoppedForwardNotRequeuedOnNodeDeparture pins why marking (rather than
|
||||
// deleting) is safe: OfflineReassign only re-queues ACTIVE entries, so a
|
||||
// stopped forward is not silently handed to another node when its owner goes
|
||||
// away. Deleting the entry, or leaving it Active, would both resurrect it.
|
||||
func TestStoppedForwardNotRequeuedOnNodeDeparture(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
owner := eng.state.TopologyList()[0].OwnerID
|
||||
eng.state.OfflineReassign(owner)
|
||||
|
||||
if n := len(eng.state.PendingList()); n != 0 {
|
||||
t.Fatalf("a stopped forward was re-queued on the owner's departure (pending=%d): %+v",
|
||||
n, eng.state.PendingList())
|
||||
}
|
||||
}
|
||||
|
||||
// TestSubmitTaskNotBlockedByStoppedEntry is the regression test for the bug the
|
||||
// mark-don't-remove change introduced: SubmitTask's idempotency guard used
|
||||
// HasTask, which matches a disabled topology entry. Since a stop now KEEPS the
|
||||
// entry, any submission for that forward while it is still stopped is swallowed
|
||||
// by its own guard — so a stopped forward can never be started again.
|
||||
//
|
||||
// The probe deliberately submits WITHOUT re-enabling first. Re-enabling would
|
||||
// clear the flag and make HasTask and HasActiveTask agree, hiding the defect;
|
||||
// the guard has to be exercised at the moment the entry is still stopped.
|
||||
func TestSubmitTaskNotBlockedByStoppedEntry(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// stop it -> entry is kept, marked disabled
|
||||
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); !d {
|
||||
t.Fatal("precondition: forward should be disabled")
|
||||
}
|
||||
// The stopped entry IS findable by the broad predicate — which is exactly why
|
||||
// the narrow one has to exist.
|
||||
if !eng.HasTask("web", "frps1", 18081) {
|
||||
t.Fatal("precondition: the stopped entry should still be findable by HasTask")
|
||||
}
|
||||
if eng.HasActiveTask("web", "frps1", 18081) {
|
||||
t.Fatal("precondition: a stopped forward must NOT count as active")
|
||||
}
|
||||
|
||||
// The regression: a submission must be published for a stopped forward.
|
||||
// With a HasTask guard this returns nil and the forward is stuck forever.
|
||||
tk := eng.SubmitTask(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||
store.Link{RemotePort: 18081})
|
||||
if tk == nil {
|
||||
t.Fatal("SubmitTask was swallowed by the stopped topology entry — " +
|
||||
"a stopped forward could never be started again")
|
||||
}
|
||||
if tk.Link.RemotePort != 18081 {
|
||||
t.Fatalf("unexpected task: %+v", tk)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSubmitTaskStillDedupesActiveForward is the other half of the guard: the
|
||||
// narrower predicate must not turn SubmitTask into a duplicate-task generator.
|
||||
func TestSubmitTaskStillDedupesActiveForward(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !eng.HasActiveTask("web", "frps1", 18081) {
|
||||
t.Fatal("precondition: a claimed forward must count as active")
|
||||
}
|
||||
if tk := eng.SubmitTask(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||
store.Link{RemotePort: 18081}); tk != nil {
|
||||
t.Fatalf("SubmitTask must stay idempotent for an active forward, got %+v", tk)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStoppedForwardNotRequeuedOnNodeDeparture pins why marking (rather than
|
||||
// deleting) is safe: OfflineReassign only re-queues ACTIVE entries, so a
|
||||
// stopped forward is not silently handed to another node when its owner goes
|
||||
|
||||
// TestAddTopologyRespectsDisabledFlag: a claim that carries a stopped forward
|
||||
// must not mark the new entry Active, or it would come back to life.
|
||||
func TestAddTopologyRespectsDisabledFlag(t *testing.T) {
|
||||
s := &State{Topology: map[string]*TopoEntry{}}
|
||||
e := s.AddTopology(&Task{
|
||||
ID: "t1", Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||
Link: store.Link{RemotePort: 18081, Disabled: true},
|
||||
}, "n1")
|
||||
if e.Active {
|
||||
t.Fatal("AddTopology forced Active=true for a disabled claim — the forward would resurrect")
|
||||
}
|
||||
if !e.Link.Disabled {
|
||||
t.Fatal("AddTopology dropped the disabled flag from the link")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRestartTaskBypassesDuplicateClaimGuard is the engine half of the
|
||||
// re-enable path: a restart task for a forward that still HAS a topology entry
|
||||
// must reach Handler.Restart.
|
||||
//
|
||||
// This is precisely what the duplicate-claim guard refuses — it exists to stop
|
||||
// a stale task from spawning a second worker for an owned forward, and a restart
|
||||
// is indistinguishable from that unless it is an explicit task kind. Marking
|
||||
// the entry stopped (instead of deleting it) is what made this necessary: the
|
||||
// entry the guard keys on now outlives a stop.
|
||||
func TestRestartTaskBypassesDuplicateClaimGuard(t *testing.T) {
|
||||
restarts := 0
|
||||
claims := 0
|
||||
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||
claim: func(ctx context.Context, tk *Task) error { claims++; return nil },
|
||||
restart: func(ctx context.Context, tk *Task) error { restarts++; return nil }},
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||
"n1:7500", true, "")
|
||||
|
||||
// Establish + stop: entry survives, marked disabled.
|
||||
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); !d {
|
||||
t.Fatal("precondition: should be disabled")
|
||||
}
|
||||
entryCount := len(eng.state.TopologyList())
|
||||
|
||||
// Start: publish a restart and run a cycle.
|
||||
eng.state.UpdateTopologyDisabled("web", "frps1", 18081, false)
|
||||
eng.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||
store.Link{RemotePort: 18081})
|
||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 3, State: eng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if restarts != 1 {
|
||||
t.Fatalf("Handler.Restart called %d times, want 1 — the restart task was swallowed "+
|
||||
"by the duplicate-claim guard, so the forward would never come back", restarts)
|
||||
}
|
||||
if claims != 1 {
|
||||
t.Fatalf("Handler.Claim called %d times, want 1 (the original claim only); "+
|
||||
"a restart must not re-run the claim bookkeeping", claims)
|
||||
}
|
||||
// Re-enabling must not create a second entry.
|
||||
if n := len(eng.state.TopologyList()); n != entryCount {
|
||||
t.Fatalf("restart created a duplicate topology entry: %d -> %d", entryCount, n)
|
||||
}
|
||||
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); d {
|
||||
t.Fatal("entry is still disabled after the restart task was applied")
|
||||
}
|
||||
var sawStart bool
|
||||
for _, e := range eng.Log.Snapshot() {
|
||||
if e.Kind == LogForwardStart {
|
||||
sawStart = true
|
||||
}
|
||||
}
|
||||
if !sawStart {
|
||||
t.Fatalf("log missing forward.start: %+v", eng.Log.Snapshot())
|
||||
}
|
||||
}
|
||||
|
||||
// TestRestartFlagSurvivesTokenSerialization: the restart task travels to the
|
||||
// owner inside the token, so the flag must round-trip through JSON. Without the
|
||||
// struct tag it would silently deserialize as false and the owner would treat
|
||||
// it as an ordinary claim — which the duplicate guard then discards.
|
||||
func TestRestartFlagSurvivesTokenSerialization(t *testing.T) {
|
||||
eng := newTestEngine("n1", true)
|
||||
eng.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||
store.Link{RemotePort: 18081})
|
||||
|
||||
blob, err := json.Marshal(&Token{Cycle: 7, State: eng.state})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var back Token
|
||||
if err := json.Unmarshal(blob, &back); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var found *Task
|
||||
for _, tk := range back.State.PendingList() {
|
||||
if tk.Local.Name == "web" && tk.Link.RemotePort == 18081 {
|
||||
found = tk
|
||||
}
|
||||
}
|
||||
if found == nil {
|
||||
t.Fatal("restart task did not survive token serialization")
|
||||
}
|
||||
if !found.Restart {
|
||||
t.Fatal("the restart flag was lost in JSON — the owner would see a plain claim " +
|
||||
"and the duplicate guard would discard it")
|
||||
}
|
||||
// Revoke and Restart must stay distinguishable.
|
||||
if found.Revoke {
|
||||
t.Fatal("a restart task must not also read as a revocation")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRestartTaskReachesNonLocalOwner.
|
||||
func TestRestartOnlyAppliedByOwner(t *testing.T) {
|
||||
restarts := 0
|
||||
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||
restart: func(ctx context.Context, tk *Task) error { restarts++; return nil }}
|
||||
|
||||
// Owner node claims the forward.
|
||||
owner := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil, h,
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil }, "n1:7500", true, "")
|
||||
owner.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := owner.OnToken(context.Background(), &Token{Cycle: 1, State: owner.state, SentAt: 1}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(owner.state.TopologyList()) != 1 {
|
||||
t.Fatalf("precondition: owner should hold the entry, got %+v", owner.state.TopologyList())
|
||||
}
|
||||
|
||||
// A peer adopts the same ring state but does not own the forward.
|
||||
peer := NewEngine("n2", "n2:7500", "u", "p", "0.71.0", nil, h,
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil }, "n2:7500", false, "")
|
||||
peer.AdoptState(owner.state)
|
||||
tk := peer.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||
store.Link{RemotePort: 18081})
|
||||
if tk == nil {
|
||||
t.Fatal("SubmitRestart returned nil")
|
||||
}
|
||||
|
||||
// The peer processes one token: it must NOT apply the restart.
|
||||
if _, err := peer.OnToken(context.Background(), &Token{Cycle: 2, State: peer.state, SentAt: 2}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if restarts != 0 {
|
||||
t.Fatalf("a NON-owner applied the restart %d time(s) — it would spawn an orphaned "+
|
||||
"worker for a forward owned by somebody else", restarts)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRestartTaskReachesNonLocalOwner: a restart published on a node that is NOT
|
||||
// the owner must survive the token round-trip and be applied by the owner.
|
||||
//
|
||||
// The naive "if not owner then continue" implementation CONSUMED the task
|
||||
// (ClaimPending had already removed it), so the owner never received it and the
|
||||
// forward stayed stopped with nothing logged as an error. Deferral must
|
||||
// re-queue it.
|
||||
//
|
||||
// Each OnToken is fed a FRESH token (as the real ring does — the successor's
|
||||
// OnToken receives the state the predecessor returned), so the deferral is
|
||||
// exercised once per hop rather than re-processing one snapshot.
|
||||
func TestRestartTaskReachesNonLocalOwner(t *testing.T) {
|
||||
ownerApplied, submitterApplied := 0, 0
|
||||
ownerEng := NewEngine("n2", "n2:7500", "u", "p", "0.71.0", nil,
|
||||
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||
restart: func(ctx context.Context, tk *Task) error { ownerApplied++; return nil }},
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil }, "n2:7500", false, "")
|
||||
|
||||
ownerEng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||
if _, err := ownerEng.OnToken(context.Background(), &Token{Cycle: 1, State: ownerEng.state}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(ownerEng.state.TopologyList()) != 1 {
|
||||
t.Fatalf("precondition: n2 should own it, got %+v", ownerEng.state.TopologyList())
|
||||
}
|
||||
|
||||
subEng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||
restart: func(ctx context.Context, tk *Task) error { submitterApplied++; return nil }},
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil }, "n1:7500", true, "")
|
||||
subEng.AdoptState(ownerEng.state)
|
||||
|
||||
subEng.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||
store.Link{RemotePort: 18081})
|
||||
|
||||
// One hop through the non-owner: it must defer, not apply and not drop.
|
||||
out, err := subEng.OnToken(context.Background(), &Token{Cycle: 2, State: subEng.state, SentAt: 1})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if submitterApplied != 0 {
|
||||
t.Fatal("the non-owner applied a restart for a forward it does not own")
|
||||
}
|
||||
var carried *Task
|
||||
for _, tk := range out.State.PendingList() {
|
||||
if tk.Restart && tk.Local.Name == "web" {
|
||||
carried = tk
|
||||
}
|
||||
}
|
||||
if carried == nil {
|
||||
t.Fatal("the non-owner CONSUMED the restart task; the owner would never receive it")
|
||||
}
|
||||
|
||||
// The owner applies the carried task.
|
||||
if _, err := ownerEng.OnToken(context.Background(), &Token{Cycle: 3, State: out.State, SentAt: 2}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ownerApplied != 1 {
|
||||
t.Fatalf("the owner applied the restart %d times, want 1", ownerApplied)
|
||||
}
|
||||
}
|
||||
|
||||
190
internal/cluster/ring_splitbrain_test.go
Normal file
190
internal/cluster/ring_splitbrain_test.go
Normal file
@ -0,0 +1,190 @@
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// ringOf builds a 3-node ring where n1 is the leader.
|
||||
func ringOf(t *testing.T) (n1, n2, n3 *Engine) {
|
||||
mk := func(id string) *Engine {
|
||||
return NewEngine(id, id+":7500", "u", "p", "0.1.2", nil,
|
||||
&fakeHandler{load: Load{MemPct: 5, NetPct: 5}},
|
||||
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||
id+":7500", true, "")
|
||||
}
|
||||
n1, n2, n3 = mk("n1"), mk("n2"), mk("n3")
|
||||
seed := State{
|
||||
LeaderID: "n1",
|
||||
Nodes: []Node{
|
||||
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true},
|
||||
{ID: "n2", Addr: "n2:7500", Alive: true},
|
||||
{ID: "n3", Addr: "n3:7500", Alive: true},
|
||||
},
|
||||
Topology: map[string]*TopoEntry{},
|
||||
PendingTasks: map[string]*Task{},
|
||||
}
|
||||
for _, e := range []*Engine{n1, n2, n3} {
|
||||
// Round-trip through JSON so each engine holds its OWN Nodes slice and
|
||||
// maps. AdoptState assigns the state by value, but Nodes is a slice, so
|
||||
// handing all three the same seed would alias one backing array and make
|
||||
// a mutation on one node visible on the others — an artefact of the test
|
||||
// harness, not of the real transport (each process deserializes its own
|
||||
// copy from the token).
|
||||
blob, err := json.Marshal(seed)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var own State
|
||||
if err := json.Unmarshal(blob, &own); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.AdoptState(own)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// TestBecomeLeaderDeclinesToLiveLeader is the regression test for the live split
|
||||
// brain.
|
||||
//
|
||||
// Leadership used to be self-declared: any node that failed to reach the leader
|
||||
// it believed in promoted itself immediately. Both sides of a partition take that
|
||||
// path, so two rings ran concurrently and each emptied the other's topology into
|
||||
// pending — observed live as two nodes naming the same leader while marking each
|
||||
// other dead, with pending=3 and topology=0 on both.
|
||||
//
|
||||
// A node must therefore consult the ring before promoting itself.
|
||||
func TestBecomeLeaderDeclinesToLiveLeader(t *testing.T) {
|
||||
_, n2, _ := ringOf(t)
|
||||
// n3 has already promoted itself and n2 has learned that view: n1 is dead,
|
||||
// n3 is alive and leading. n2 cannot reach n3 (that is why it is asking).
|
||||
for i := range n2.state.Nodes {
|
||||
switch n2.state.Nodes[i].ID {
|
||||
case "n1":
|
||||
n2.state.Nodes[i].Alive = false
|
||||
n2.state.Nodes[i].IsLeader = false
|
||||
case "n3":
|
||||
n2.state.Nodes[i].Alive = true
|
||||
n2.state.Nodes[i].IsLeader = true
|
||||
}
|
||||
}
|
||||
n2.state.LeaderID = "n3"
|
||||
|
||||
if n2.becomeLeader() {
|
||||
t.Fatal("n2 promoted itself while n3 was alive and leading in its view — " +
|
||||
"two leaders is exactly the split brain this guards against")
|
||||
}
|
||||
if n2.state.LeaderID == "n2" {
|
||||
t.Fatal("LeaderID was overwritten despite declining")
|
||||
}
|
||||
if n2.state.Nodes[0].IsLeader && n2.state.Nodes[0].ID == "n2" {
|
||||
t.Fatal("n2 marked itself leader despite declining")
|
||||
}
|
||||
}
|
||||
|
||||
// TestBecomeLeaderPromotesWhenSoleSurvivor: the guard must not break genuine
|
||||
// failover. With every other node dead, promotion has to happen.
|
||||
func TestBecomeLeaderPromotesWhenSoleSurvivor(t *testing.T) {
|
||||
_, n2, _ := ringOf(t)
|
||||
for _, id := range []string{"n1", "n3"} {
|
||||
n2.state.MarkOffline(id)
|
||||
}
|
||||
if !n2.becomeLeader() {
|
||||
t.Fatal("n2 declined promotion with every other node dead — that breaks failover")
|
||||
}
|
||||
if n2.state.LeaderID != "n2" {
|
||||
t.Fatalf("leader=%q, want n2", n2.state.LeaderID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTakeoverLeavesLiveLeadersForwardsAlone pins the second half of the fix.
|
||||
//
|
||||
// When a heartbeat fails, the node used to mark the leader dead and re-queue ALL
|
||||
// of that leader's forwards unconditionally — even when the ring still showed a
|
||||
// live leader, i.e. this node was merely the partitioned side. That is what
|
||||
// emptied a working ring: both sides ended up with topology=0 and pending=3.
|
||||
//
|
||||
// Now the re-hang happens only if we actually take leadership.
|
||||
func TestTakeoverLeavesLiveLeadersForwardsAlone(t *testing.T) {
|
||||
_, n2, _ := ringOf(t)
|
||||
// n1 (the leader n2 believes in) owns a forward.
|
||||
n2.state.AddTopology(&Task{
|
||||
ID: "t1",
|
||||
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||
Link: store.Link{RemotePort: 18081},
|
||||
}, "n1")
|
||||
before := len(n2.state.TopologyList())
|
||||
pendingBefore := len(n2.state.PendingList())
|
||||
|
||||
// n2 still believes n1 is the leader and cannot reach it, so it tries to take
|
||||
// over. But its state ALSO lists n3 as alive-and-leader — the view it learned
|
||||
// when n3 promoted itself. Two claims, one ring: n2 is the partitioned side
|
||||
// and must stand down.
|
||||
for i := range n2.state.Nodes {
|
||||
switch n2.state.Nodes[i].ID {
|
||||
case "n1":
|
||||
n2.state.Nodes[i].Alive = true // still marked alive; MarkOffline happens inside
|
||||
n2.state.Nodes[i].IsLeader = true
|
||||
case "n3":
|
||||
n2.state.Nodes[i].Alive = true
|
||||
n2.state.Nodes[i].IsLeader = true
|
||||
}
|
||||
}
|
||||
n2.state.LeaderID = "n1"
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
n2.takeOverFromDeadLeader(ctx)
|
||||
|
||||
if n2.state.LeaderID == "n2" {
|
||||
t.Error("n2 took over while a live leader existed")
|
||||
}
|
||||
if after := len(n2.state.TopologyList()); after != before {
|
||||
t.Errorf("a declined takeover still re-queued the live leader's forwards: topology %d -> %d",
|
||||
before, after)
|
||||
}
|
||||
if p := len(n2.state.PendingList()); p != pendingBefore {
|
||||
t.Errorf("a declined takeover still pushed work into pending: %d -> %d", pendingBefore, p)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTakeoverReassignsWhenSoleSurvivor is the other half: genuine failover must
|
||||
// still work, otherwise the guard would have broken the design's promise that a
|
||||
// dead leader is replaced and its forwards re-hung.
|
||||
func TestTakeoverReassignsWhenSoleSurvivor(t *testing.T) {
|
||||
_, n2, _ := ringOf(t)
|
||||
n2.state.AddTopology(&Task{
|
||||
ID: "t1",
|
||||
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||
Link: store.Link{RemotePort: 18081},
|
||||
}, "n1") // owned by the leader n1
|
||||
|
||||
// n1 is the leader and is dead; n3 is dead too. n2 is the sole survivor.
|
||||
for _, id := range []string{"n1", "n3"} {
|
||||
n2.state.MarkOffline(id)
|
||||
}
|
||||
n2.state.Nodes[0].IsLeader = false // n1 no longer claims leadership
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
n2.takeOverFromDeadLeader(ctx)
|
||||
|
||||
if n2.state.LeaderID != "n2" {
|
||||
t.Fatalf("n2 did not take over as the sole survivor, leader=%q", n2.state.LeaderID)
|
||||
}
|
||||
if len(n2.state.TopologyList()) != 0 {
|
||||
t.Error("the dead leader's forwards should have been re-hung")
|
||||
}
|
||||
found := false
|
||||
for _, tk := range n2.state.PendingList() {
|
||||
if tk.Local.Name == "web" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("the dead leader's forward was not re-queued for another node")
|
||||
}
|
||||
}
|
||||
@ -8,7 +8,6 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
@ -52,11 +51,16 @@ func (t *TokenTransport) SendTo(getAddr func(nodeID string) string) func(ctx con
|
||||
start := time.Now()
|
||||
resp, err := cli.Do(req)
|
||||
if err != nil {
|
||||
log.Printf("token-send %s: size=%dB elapsed=%v err=%v", addr, len(body), time.Since(start).Round(time.Millisecond), err)
|
||||
// NOT demoted: a failed send is the "neighbor offline" signal the
|
||||
// ring's fault paths are diagnosed from.
|
||||
logf("token-send %s: size=%dB elapsed=%v err=%v", addr, len(body), time.Since(start).Round(time.Millisecond), err)
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
log.Printf("token-send %s: size=%dB elapsed=%v -> %d", addr, len(body), time.Since(start).Round(time.Millisecond), resp.StatusCode)
|
||||
// Success path is a per-round heartbeat (size/elapsed/200 repeat
|
||||
// verbatim every cycle) — debug only. The status-code check below stays
|
||||
// loud on purpose, so a non-2xx still shows up without the debug flag.
|
||||
debugToken("token-send %s: size=%dB elapsed=%v -> %d", addr, len(body), time.Since(start).Round(time.Millisecond), resp.StatusCode)
|
||||
if resp.StatusCode >= 400 {
|
||||
return fmt.Errorf("token POST %s -> HTTP %d", url, resp.StatusCode)
|
||||
}
|
||||
|
||||
204
internal/httpapi/forward_revoke_test.go
Normal file
204
internal/httpapi/forward_revoke_test.go
Normal file
@ -0,0 +1,204 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"webui4frpc/internal/cluster"
|
||||
"webui4frpc/internal/process"
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// newRingTestHandler builds a Handler WITH a ring engine attached, so the
|
||||
// paths that publish tasks into the token actually execute. newTestHandler
|
||||
// leaves Ring nil, which silently skips them — a test built on it can pass
|
||||
// while the publish side is completely broken.
|
||||
func newRingTestHandler(t *testing.T) (*Handler, *cluster.Engine, *httptest.Server) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
st, err := store.New(filepath.Join(dir, "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
|
||||
pm := process.NewManager(process.Options{
|
||||
ConfigsDir: filepath.Join(dir, "configs"),
|
||||
LogsDir: filepath.Join(dir, "logs"),
|
||||
BinaryPath: func() string { return "" },
|
||||
Render: func(string) ([]byte, error) { return []byte(`{}`), nil },
|
||||
AutoRestart: func(string) bool { return false },
|
||||
RestartInterval: func() int { return 5 },
|
||||
})
|
||||
ring := cluster.NewEngine("n1", "n1:7500", "u", "p", "0.1.0", nil,
|
||||
&cluster.AppHandler{},
|
||||
func(ctx context.Context, next string, tk *cluster.Token) error { return nil },
|
||||
"n1:7500", true, "")
|
||||
h := &Handler{Store: st, Process: pm, WorkDir: dir, User: "admin", Password: "pw", Ring: ring}
|
||||
mux, err := NewServeMux(h)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ts := httptest.NewServer(mux)
|
||||
t.Cleanup(ts.Close)
|
||||
return h, ring, ts
|
||||
}
|
||||
|
||||
func saveCanvas(t *testing.T, srv *httptest.Server, body string) {
|
||||
t.Helper()
|
||||
req, _ := http.NewRequest(http.MethodPut, srv.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := srv.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("canvas save status = %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopForwardPublishesDisabledFlagInRevokeTask is the regression test for
|
||||
// the actual defect.
|
||||
//
|
||||
// stopForward() read the link, called SetLinkDisabled(true), and then handed
|
||||
// the STALE copy (disabled=false) to RevokeTask. The revoke travels to the
|
||||
// node that OWNS the forward, and that node's Claim/Revoke path keys off the
|
||||
// flag — so a stale false meant:
|
||||
// - the owner could not tell the stop was deliberate, and
|
||||
// - nothing retired the topology entry,
|
||||
//
|
||||
// so the next restart's reconcile re-claimed the forward and spawned a worker
|
||||
// for something the user had stopped (seen live: endless connection-refused
|
||||
// against an intentionally-down service).
|
||||
//
|
||||
// This asserts the flag ON THE PUBLISHED TASK, which is the value that was
|
||||
// actually wrong. It cannot be satisfied by the store write alone.
|
||||
func TestStopForwardPublishesDisabledFlagInRevokeTask(t *testing.T) {
|
||||
_, ring, ts := newRingTestHandler(t)
|
||||
|
||||
saveCanvas(t, ts, `{
|
||||
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||
}`)
|
||||
|
||||
// Stop the forward over the API.
|
||||
b, _ := json.Marshal(stopForwardReq{"svc", "srv-a", 45999})
|
||||
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/api/manager/forwards/stop", bytes.NewReader(b))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("stop status = %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
// Find the revoke task that was published into the token.
|
||||
var revoke *cluster.Task
|
||||
for _, tk := range ring.State().PendingList() {
|
||||
if tk.Revoke && tk.Local.Name == "svc" && tk.Link.RemotePort == 45999 {
|
||||
revoke = tk
|
||||
break
|
||||
}
|
||||
}
|
||||
if revoke == nil {
|
||||
t.Fatal("stop did not publish a revoke task for the forward")
|
||||
}
|
||||
if !revoke.Link.Disabled {
|
||||
t.Fatal("the published revoke task carries disabled=false — the owner node cannot tell " +
|
||||
"this stop was deliberate, which is the bug that let stopped forwards resurrect")
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopThenStartPublishesRestartTask guards the failure mode that
|
||||
// mark-don't-remove introduces, and that only shows up on the WIRE.
|
||||
//
|
||||
// A stop keeps the topology entry (it is the carrier for the flag), so on
|
||||
// re-enable:
|
||||
// - SubmitTask dedupes, because the entry exists;
|
||||
// - the claim path discards any task for a forward that already has an owner.
|
||||
//
|
||||
// Both channels therefore refuse, no task is published, the owner never learns
|
||||
// about the re-enable, and the forward stays stopped forever — a forward the
|
||||
// user can stop but never restart.
|
||||
//
|
||||
// Asserting the task is PUBLISHED is the point: asserting only that the flag
|
||||
// cleared would pass while nothing reached the owner. (The earlier version of
|
||||
// this test did exactly that and stayed green against the broken code.)
|
||||
func TestStopThenStartPublishesRestartTask(t *testing.T) {
|
||||
_, ring, ts := newRingTestHandler(t)
|
||||
|
||||
saveCanvas(t, ts, `{
|
||||
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||
}`)
|
||||
// Claim it so a topology entry exists (that is what makes the two normal
|
||||
// channels refuse later).
|
||||
if _, err := ring.OnToken(context.Background(), &cluster.Token{Cycle: 1, State: *ring.State()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ring.HasActiveTask("svc", "srv-a", 45999) {
|
||||
t.Fatalf("precondition: forward should be active, topo=%+v", ring.State().TopologyList())
|
||||
}
|
||||
|
||||
forwards := func(action string) {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(stopForwardReq{"svc", "srv-a", 45999})
|
||||
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/api/manager/forwards/"+action, bytes.NewReader(b))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("%s status = %d", action, resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
// --- stop: entry kept, marked disabled -------------------------------
|
||||
forwards("stop")
|
||||
if d, known := ring.TopologyDisabled("svc", "srv-a", 45999); !known || !d {
|
||||
t.Fatalf("stop did not mark the topology entry disabled (known=%v disabled=%v)", known, d)
|
||||
}
|
||||
if ring.HasActiveTask("svc", "srv-a", 45999) {
|
||||
t.Fatal("a stopped forward must not count as active")
|
||||
}
|
||||
|
||||
// --- start: a task MUST reach the owner ------------------------------
|
||||
forwards("start")
|
||||
if d, _ := ring.TopologyDisabled("svc", "srv-a", 45999); d {
|
||||
t.Fatal("start did not clear the disabled flag")
|
||||
}
|
||||
if !ring.HasActiveTask("svc", "srv-a", 45999) {
|
||||
t.Fatal("after start the forward must be active again")
|
||||
}
|
||||
|
||||
// The actual regression: something has to be queued for the owner. The
|
||||
// re-enable must be published as a restart task, since a plain creation task
|
||||
// would be deduped or discarded.
|
||||
var restart *cluster.Task
|
||||
for _, tk := range ring.State().PendingList() {
|
||||
if tk.Restart && tk.Local.Name == "svc" && tk.Link.RemotePort == 45999 {
|
||||
restart = tk
|
||||
break
|
||||
}
|
||||
}
|
||||
if restart == nil {
|
||||
t.Fatalf("start published no restart task — the owner would never bring the "+
|
||||
"worker back, leaving the forward permanently stopped (pending=%+v)",
|
||||
ring.State().PendingList())
|
||||
}
|
||||
if restart.Link.Disabled {
|
||||
t.Fatal("the restart task carries disabled=true, so the owner would re-apply the stop")
|
||||
}
|
||||
}
|
||||
141
internal/httpapi/forward_stop_test.go
Normal file
141
internal/httpapi/forward_stop_test.go
Normal file
@ -0,0 +1,141 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// stopForwardReq mirrors the forwards start/stop request body.
|
||||
type stopForwardReq struct {
|
||||
Local string `json:"local"`
|
||||
Remote string `json:"remote"`
|
||||
RemotePort int `json:"remotePort"`
|
||||
}
|
||||
|
||||
func postForwards(t *testing.T, srv *httptest.Server, action string, body stopForwardReq) int {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(body)
|
||||
req, _ := http.NewRequest(http.MethodPost, srv.URL+"/api/manager/forwards/"+action, bytes.NewReader(b))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := srv.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
|
||||
// TestStopForwardPersistsDisabledFlag is the regression test for the
|
||||
// "stopped forwards resurrect on restart" bug.
|
||||
//
|
||||
// The original defect: stopForward() read the link, called
|
||||
// SetLinkDisabled(true), but then handed the STALE (disabled=false) copy to
|
||||
// RevokeTask — so the disabled flag never reached the node owning the forward,
|
||||
// and nothing removed the topology entry. On the next restart the startup
|
||||
// reconcile saw an owned forward with no worker and re-claimed it, spawning a
|
||||
// worker for a forward the user had deliberately stopped (observed live:
|
||||
// ~14k "proxy already exists" retries and endless connection-refused against a
|
||||
// service that was intentionally down).
|
||||
//
|
||||
// The contract this pins: after a successful stop, the persisted link MUST be
|
||||
// disabled — that flag is the single source of truth the claim path consults.
|
||||
func TestStopForwardPersistsDisabledFlag(t *testing.T) {
|
||||
h, ts := newTestHandler(t)
|
||||
|
||||
body := `{
|
||||
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||
}`
|
||||
req, _ := http.NewRequest(http.MethodPut, ts.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("canvas save status = %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
// The forward starts enabled.
|
||||
ln, found, err := h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
if err != nil || !found {
|
||||
t.Fatalf("link not persisted: found=%v err=%v", found, err)
|
||||
}
|
||||
if ln.Disabled {
|
||||
t.Fatal("a freshly saved forward must start enabled")
|
||||
}
|
||||
|
||||
// Stop it.
|
||||
if code := postForwards(t, ts, "stop", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||
t.Fatalf("stop status = %d, want 200", code)
|
||||
}
|
||||
|
||||
// Persisted flag must now be set — this is what the claim path reads.
|
||||
ln, found, err = h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("link vanished after stop; stop must be non-destructive")
|
||||
}
|
||||
if !ln.Disabled {
|
||||
t.Fatal("stop did not persist disabled=true — the startup reconcile would resurrect this forward")
|
||||
}
|
||||
|
||||
// Start must clear it again (the user-facing re-enable path).
|
||||
if code := postForwards(t, ts, "start", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||
t.Fatalf("start status = %d, want 200", code)
|
||||
}
|
||||
ln, _, err = h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ln.Disabled {
|
||||
t.Fatal("start did not clear disabled; a re-enabled forward would stay stopped")
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopForwardIsNonDestructive pins the per-forward stop semantics the
|
||||
// revoke path was specifically rewritten for: stopping one forward must not
|
||||
// touch a sibling forward that shares the same local or remote.
|
||||
func TestStopForwardIsNonDestructive(t *testing.T) {
|
||||
h, ts := newTestHandler(t)
|
||||
|
||||
body := `{
|
||||
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||
"links": [
|
||||
{"local":"svc","remote":"srv-a","remotePort":45999},
|
||||
{"local":"svc","remote":"srv-a","remotePort":46000}
|
||||
]
|
||||
}`
|
||||
req, _ := http.NewRequest(http.MethodPut, ts.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
|
||||
if code := postForwards(t, ts, "stop", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||
t.Fatalf("stop status = %d", code)
|
||||
}
|
||||
|
||||
stopped, _, _ := h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
sibling, found, _ := h.Store.LinkByTriple("svc", "srv-a", 46000)
|
||||
if !found {
|
||||
t.Fatal("sibling forward was destroyed by stopping its neighbour")
|
||||
}
|
||||
if !stopped.Disabled {
|
||||
t.Error("the stopped forward should be disabled")
|
||||
}
|
||||
if sibling.Disabled {
|
||||
t.Error("the sibling forward must stay enabled — per-forward stop, not per-local/remote")
|
||||
}
|
||||
}
|
||||
@ -245,7 +245,12 @@ func (h *Handler) applyCanvas(w http.ResponseWriter, r *http.Request, canvas *ca
|
||||
}
|
||||
if ln.Disabled {
|
||||
// Stopped on the forwards page: make sure it leaves the topology.
|
||||
if h.Ring.HasTask(ln.Local, ln.Remote, ln.RemotePort) {
|
||||
// Guard on an ACTIVE task — an entry that is already marked
|
||||
// disabled has nothing left to revoke, and re-issuing a revocation
|
||||
// for it on every canvas save would be pure noise. (HasTask, which
|
||||
// also matches disabled entries, is the right predicate for the
|
||||
// "already handled" question this branch is not asking.)
|
||||
if h.Ring.HasActiveTask(ln.Local, ln.Remote, ln.RemotePort) {
|
||||
h.Ring.RevokeTask(loc, rem, ln)
|
||||
}
|
||||
continue
|
||||
|
||||
@ -84,6 +84,25 @@ func (h *Handler) startForward(local, remote string, port int) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if h.Ring != nil {
|
||||
// Publish the enable into the topology so it rides the ring (a peer that
|
||||
// still holds the stale "stopped" copy learns about it on adoption).
|
||||
h.Ring.UpdateTopologyDisabled(local, remote, port, false)
|
||||
}
|
||||
// A cluster forward whose topology entry still exists needs its OWNER to
|
||||
// bring the worker back, and neither of the normal channels can do it:
|
||||
// - SubmitTask is idempotency-guarded, and the entry still exists (a stop
|
||||
// keeps it as the flag's carrier), so the submission is deduped away;
|
||||
// - the claim path drops any task for a forward that already has an
|
||||
// owner, as a defence against duplicate-claim collisions.
|
||||
// So a re-enable of an existing entry is published as a dedicated RESTART
|
||||
// task, which the owner applies unconditionally. Without it a forward the
|
||||
// user can stop but not restart — which is what marking-instead-of-removing
|
||||
// would otherwise have produced.
|
||||
if h.Ring != nil && h.Ring.HasTopologyEntry(local, remote, port) {
|
||||
h.Ring.SubmitRestart(loc, rem, ln)
|
||||
return nil
|
||||
}
|
||||
if h.Ring != nil {
|
||||
h.Ring.SubmitTask(loc, rem, ln)
|
||||
}
|
||||
@ -109,6 +128,24 @@ func (h *Handler) stopForward(local, remote string, port int) error {
|
||||
} else {
|
||||
_ = h.Store.SetLinkDisabled(local, remote, port, true)
|
||||
}
|
||||
// The revoke task travels to whichever node OWNS the forward, and that node
|
||||
// re-reads the disabled flag from its own store before starting a worker —
|
||||
// so the flag has to be set on every node that has a copy of this link, not
|
||||
// just the one handling this request. Propagating Disabled on the task lets
|
||||
// the owner's RevokeFn stop the worker even if its own store row is stale.
|
||||
//
|
||||
// This also fixes a latent inconsistency: `ln` was read BEFORE the
|
||||
// SetLinkDisabled(true) above, so the link published into the token still
|
||||
// carried disabled=false and got copied into the topology entry verbatim.
|
||||
ln.Disabled = true
|
||||
// Publish the stop into the topology BEFORE revoking: the revoke retires the
|
||||
// own-side worker/entry, so the flag must already exist somewhere that
|
||||
// survives it and travels the ring. This is the send half of cluster-wide
|
||||
// disabled propagation (see store.ReconcileLinkDisabled for the receive
|
||||
// half).
|
||||
if h.Ring != nil {
|
||||
h.Ring.UpdateTopologyDisabled(local, remote, port, true)
|
||||
}
|
||||
if loc.LocalOnly {
|
||||
if h.Process != nil {
|
||||
key := process.WorkerKey(local, remote, port)
|
||||
|
||||
@ -48,6 +48,8 @@ type Handler struct {
|
||||
Cluster *cluster.Registry
|
||||
// Ring is the token-ring engine (M6). Nil disables ring routes.
|
||||
Ring *cluster.Engine
|
||||
// Balancer is optional: when nil the balance endpoint reports 404.
|
||||
Balancer *cluster.Balancer
|
||||
// SelfAddr is this node's reachable listen address (from -addr), used for
|
||||
// cluster discovery so peers can reach back for binary exchange.
|
||||
SelfAddr string
|
||||
@ -114,6 +116,8 @@ func NewServeMux(h *Handler) (http.Handler, error) {
|
||||
mux.HandleFunc(apiPrefix+"/cluster/nodes", h.auth("read")(h.handleClusterNodes))
|
||||
mux.HandleFunc(apiPrefix+"/cluster/cache", h.auth("read")(h.handleClusterCache)) // GET read; POST write-checked inside
|
||||
mux.HandleFunc(apiPrefix+"/cluster/ring", h.auth("read")(h.handleClusterRing))
|
||||
mux.HandleFunc(apiPrefix+"/cluster/balance", h.auth("read")(h.handleClusterBalance))
|
||||
mux.HandleFunc(apiPrefix+"/cluster/balance/mode", h.auth("write")(h.handleClusterBalanceMode))
|
||||
mux.HandleFunc(apiPrefix+"/node/logs", h.auth("read")(h.handleNodeLogs))
|
||||
mux.HandleFunc(apiPrefix+"/cluster/logs/export", h.auth("read")(h.handleClusterLogsExport))
|
||||
|
||||
@ -634,6 +638,64 @@ func (h *Handler) handleClusterRing(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, h.Ring.Snapshot())
|
||||
}
|
||||
|
||||
// handleClusterBalance reports the load balancer's read-only view: per-node
|
||||
// scores, whether anything is over the threshold, and the migration it would
|
||||
// perform. Publishing nothing is the point — an operator asking "is this cluster
|
||||
// balanced?" currently gets silence from the log in the healthy case, which is
|
||||
// indistinguishable from balancing being switched off.
|
||||
func (h *Handler) handleClusterBalance(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
methodNotAllowed(w)
|
||||
return
|
||||
}
|
||||
if h.Ring == nil {
|
||||
http.Error(w, "ring engine not enabled", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if h.Balancer == nil {
|
||||
http.Error(w, "load balancer not enabled", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
rep := h.Balancer.Report()
|
||||
// CanControl is decided by the caller's own level rather than by the
|
||||
// presence of a stored value, so a read-only principal is never shown a
|
||||
// switch that would 403.
|
||||
rep.CanControl = hasLevel(r, "write")
|
||||
writeJSON(w, http.StatusOK, rep)
|
||||
}
|
||||
|
||||
// handleClusterBalanceMode switches load balancing on/off at runtime.
|
||||
//
|
||||
// It is per-node on purpose. Every member decides about its OWN forwards, so the
|
||||
// switch belongs to each node: an operator may well want to turn releasing on
|
||||
// everywhere, but the server should not pretend those are one atomic action.
|
||||
// (A cluster-wide control can be layered on top later; the primitive here is the
|
||||
// one that actually has an owner.)
|
||||
//
|
||||
// Write-level auth: releasing stops a worker and drops a live connection.
|
||||
func (h *Handler) handleClusterBalanceMode(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost {
|
||||
methodNotAllowed(w)
|
||||
return
|
||||
}
|
||||
if h.Balancer == nil {
|
||||
http.Error(w, "load balancer not enabled", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Mode string `json:"mode"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, "parse json: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if err := h.Balancer.SetModeFromUI(req.Mode); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, h.Balancer.Report())
|
||||
}
|
||||
|
||||
// handleClusterJoin accepts a newcomer join request: the target node inserts
|
||||
// the newcomer after itself in the ring and returns the updated ring state
|
||||
// for the newcomer to adopt.
|
||||
|
||||
288
internal/store/link_test.go
Normal file
288
internal/store/link_test.go
Normal file
@ -0,0 +1,288 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// seed inserts the local + remote rows a link's foreign keys require.
|
||||
// (links.local / links.remote reference their own tables, so a link cannot
|
||||
// exist on its own — the same reason ClaimFn upserts them before ReplaceLinks.)
|
||||
func seed(t *testing.T, st *Store, locals []string, remote string) {
|
||||
t.Helper()
|
||||
for _, n := range locals {
|
||||
if err := st.UpsertLocal(Local{Name: n, IP: "127.0.0.1", Port: 8080, Protocol: "tcp"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := st.UpsertRemote(Remote{Name: remote, IP: "1.2.3.4", Port: 7000, Token: "tok", Enabled: true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLinkByTripleSurvivesReplaceLinks pins the reason LinkByTriple exists.
|
||||
//
|
||||
// ReplaceLinks() rewrites the whole table with DELETE + re-INSERT, so sqlite
|
||||
// hands every row a FRESH autoincrement id. A Link captured before such a
|
||||
// write (e.g. one riding inside a ring token) therefore carries an id that
|
||||
// either matches a different forward or matches nothing. The natural key
|
||||
// (local, remote, remotePort) is what every caller actually identifies a
|
||||
// forward by, and it must survive those rewrites.
|
||||
func TestLinkByTripleSurvivesReplaceLinks(t *testing.T) {
|
||||
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
seed(t, st, []string{"alpha", "beta", "gamma"}, "srv")
|
||||
links := []Link{
|
||||
{Local: "alpha", Remote: "srv", RemotePort: 100},
|
||||
{Local: "beta", Remote: "srv", RemotePort: 200},
|
||||
{Local: "gamma", Remote: "srv", RemotePort: 300},
|
||||
}
|
||||
if err := st.ReplaceLinks(links); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Capture the ids as the ring would have them.
|
||||
before := map[string]int64{}
|
||||
all, err := st.ListLinks()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, l := range all {
|
||||
before[l.Local] = l.ID
|
||||
}
|
||||
if len(before) != 3 {
|
||||
t.Fatalf("expected 3 links, got %d", len(before))
|
||||
}
|
||||
|
||||
// Rewrite the table (this is what saveCanvas and ClaimFn both do).
|
||||
if err := st.ReplaceLinks(links); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
after, err := st.ListLinks()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(after) != 3 {
|
||||
t.Fatalf("expected 3 links after rewrite, got %d", len(after))
|
||||
}
|
||||
|
||||
// The natural key must still resolve to the right forward, with its
|
||||
// disabled flag and group intact.
|
||||
for _, l := range after {
|
||||
if l.Disabled {
|
||||
t.Errorf("link %s unexpectedly disabled after a plain rewrite", l.Local)
|
||||
}
|
||||
}
|
||||
got, found, err := st.LinkByTriple("beta", "srv", 200)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("LinkByTriple failed to find beta after ReplaceLinks")
|
||||
}
|
||||
if got.Local != "beta" || got.RemotePort != 200 {
|
||||
t.Fatalf("LinkByTriple returned the wrong row: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLinkByTripleNotFoundIsNotError documents the contract callers rely on:
|
||||
// "no persisted opinion yet" is (Link{}, false, nil), not an error. A fresh
|
||||
// claim of a link with no row must be allowed to start.
|
||||
func TestLinkByTripleNotFoundIsNotError(t *testing.T) {
|
||||
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
got, found, err := st.LinkByTriple("nope", "srv", 1234)
|
||||
if err != nil {
|
||||
t.Fatalf("missing link must not be an error, got %v", err)
|
||||
}
|
||||
if found {
|
||||
t.Fatalf("missing link reported as found: %+v", got)
|
||||
}
|
||||
if got.Local != "" || got.RemotePort != 0 {
|
||||
t.Fatalf("expected zero Link on miss, got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLinkByTripleReadsDisabledFlag is the store-level half of the
|
||||
// "stopped forwards resurrect on restart" bug: the claim path asks the store
|
||||
// whether the user disabled this forward, so this lookup must return the flag
|
||||
// as persisted.
|
||||
func TestLinkByTripleReadsDisabledFlag(t *testing.T) {
|
||||
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
seed(t, st, []string{"mc"}, "srv")
|
||||
if err := st.ReplaceLinks([]Link{{Local: "mc", Remote: "srv", RemotePort: 25565, Group: "game"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SetLinkDisabled("mc", "srv", 25565, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ln, found, err := st.LinkByTriple("mc", "srv", 25565)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("expected to find the link")
|
||||
}
|
||||
if !ln.Disabled {
|
||||
t.Fatal("expected Disabled=true to be visible through LinkByTriple")
|
||||
}
|
||||
if ln.Group != "game" {
|
||||
t.Fatalf("group should survive, got %q", ln.Group)
|
||||
}
|
||||
|
||||
// ...and the user-facing start path must be able to clear it again.
|
||||
if err := st.SetLinkDisabled("mc", "srv", 25565, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ln, _, _ := st.LinkByTriple("mc", "srv", 25565); ln.Disabled {
|
||||
t.Fatal("expected Disabled=false after clearing")
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetLinkByIDIsStaleAfterReplaceLinks documents WHY callers must not use
|
||||
// GetLink(id) with a previously captured id. It is not a fix — it is the trap
|
||||
// being pinned shut, so the hazard stays visible if someone reintroduces it.
|
||||
func TestGetLinkByIDIsStaleAfterReplaceLinks(t *testing.T) {
|
||||
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
seed(t, st, []string{"alpha", "beta", "gamma"}, "srv")
|
||||
if err := st.ReplaceLinks([]Link{
|
||||
{Local: "alpha", Remote: "srv", RemotePort: 100},
|
||||
{Local: "beta", Remote: "srv", RemotePort: 200},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, _ := st.ListLinks()
|
||||
var staleID int64
|
||||
for _, l := range all {
|
||||
if l.Local == "alpha" {
|
||||
staleID = l.ID
|
||||
}
|
||||
}
|
||||
|
||||
if err := st.ReplaceLinks([]Link{
|
||||
{Local: "alpha", Remote: "srv", RemotePort: 100},
|
||||
{Local: "beta", Remote: "srv", RemotePort: 200},
|
||||
{Local: "gamma", Remote: "srv", RemotePort: 300},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// The old id may still resolve, but to whatever row now occupies that
|
||||
// id — which is exactly the silent-mis-target hazard. Assert that the
|
||||
// natural key remains the only safe handle.
|
||||
if ln, ok := st.GetLink(staleID); ok && ln.Local != "alpha" {
|
||||
t.Logf("stale id %d now points at %q (hazard confirmed; use LinkByTriple)", staleID, ln.Local)
|
||||
}
|
||||
if got, found, _ := st.LinkByTriple("alpha", "srv", 100); !found || got.Local != "alpha" {
|
||||
t.Fatalf("natural key must stay reliable, got %+v found=%v", got, found)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconcileLinkDisabledLearnsPeerDecision is the store half of
|
||||
// cluster-wide stop propagation. A node that did NOT serve the stop request has
|
||||
// no reason to know about it, and its links table is node-local — so the flag
|
||||
// arrives via the ring and lands here. The case that matters is the OWNER of a
|
||||
// forward on a different machine: before this existed, that node's copy still
|
||||
// read "enabled", so it kept (or re-spawned) the worker for a forward the user
|
||||
// had explicitly stopped.
|
||||
func TestReconcileLinkDisabledLearnsPeerDecision(t *testing.T) {
|
||||
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
seed(t, st, []string{"mc"}, "srv")
|
||||
if err := st.ReplaceLinks([]Link{{Local: "mc", Remote: "srv", RemotePort: 25565, Group: "game"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// A peer's stop arrives.
|
||||
if err := st.ReconcileLinkDisabled("mc", "srv", 25565, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ln, found, err := st.LinkByTriple("mc", "srv", 25565)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("link disappeared during reconcile")
|
||||
}
|
||||
if !ln.Disabled {
|
||||
t.Fatal("the peer's stop did not land in the local store")
|
||||
}
|
||||
if ln.Group != "game" {
|
||||
t.Fatalf("reconcile must not clobber other fields, group=%q", ln.Group)
|
||||
}
|
||||
|
||||
// A peer's re-enable arrives.
|
||||
if err := st.ReconcileLinkDisabled("mc", "srv", 25565, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ln, _, _ := st.LinkByTriple("mc", "srv", 25565); ln.Disabled {
|
||||
t.Fatal("the peer's re-enable did not land")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconcileLinkDisabledCreatesPlaceholderForUnknownForward: a node that has
|
||||
// never seen the forward still must remember that it is stopped, otherwise a
|
||||
// later claim on that node would resurrect it.
|
||||
func TestReconcileLinkDisabledCreatesPlaceholderForUnknownForward(t *testing.T) {
|
||||
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
seed(t, st, []string{"ghost"}, "srv")
|
||||
|
||||
if err := st.ReconcileLinkDisabled("ghost", "srv", 9999, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ln, found, err := st.LinkByTriple("ghost", "srv", 9999)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("a stopped-but-unknown forward must be remembered, or a later claim resurrects it")
|
||||
}
|
||||
if !ln.Disabled {
|
||||
t.Fatal("placeholder is not marked disabled")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconcileLinkDisabledIgnoresEnableForUnknown: an enable for a forward this
|
||||
// node has never seen must NOT create a row. Creating one would invent forwards
|
||||
// out of ring state.
|
||||
func TestReconcileLinkDisabledIgnoresEnableForUnknown(t *testing.T) {
|
||||
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
seed(t, st, []string{"other"}, "srv")
|
||||
|
||||
if err := st.ReconcileLinkDisabled("unknown", "srv", 1234, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, found, _ := st.LinkByTriple("unknown", "srv", 1234); found {
|
||||
t.Fatal("an enable for an unknown forward must not materialise a row")
|
||||
}
|
||||
}
|
||||
@ -128,6 +128,11 @@ type Settings struct {
|
||||
// key). Cleared on explicit detach (detachAsStandalone) so a node that
|
||||
// intentionally left does NOT auto-rejoin.
|
||||
ClusterPeers string `json:"clusterPeers,omitempty"`
|
||||
// BalanceMode is the load-balancing mode chosen in the UI: "", "off",
|
||||
// "dry-run" or "on". Empty means "never configured here" so the environment
|
||||
// variable stays the bootstrap default; a non-empty value wins over it,
|
||||
// otherwise flipping the switch in the UI would silently revert on restart.
|
||||
BalanceMode string `json:"balanceMode,omitempty"`
|
||||
}
|
||||
|
||||
// Forward is a rendered link row attached to a remote.
|
||||
@ -560,6 +565,30 @@ func (s *Store) GetLink(id int64) (Link, bool) {
|
||||
return l, true
|
||||
}
|
||||
|
||||
// LinkByTriple looks a link up by its natural key (local, remote, remotePort).
|
||||
//
|
||||
// Prefer this over GetLink(id) whenever the caller only knows the forward's
|
||||
// identity: ReplaceLinks() rewrites the whole table with DELETE + re-INSERT, so
|
||||
// every row gets a fresh autoincrement id. Any id captured before such a write
|
||||
// (e.g. a Link carried inside a ring token) is stale by definition and will
|
||||
// either miss or — worse — match a different forward. The natural key is
|
||||
// stable across those rewrites.
|
||||
//
|
||||
// Returns (link, found). A missing row is (Link{}, false) and is NOT an error:
|
||||
// callers use that to mean "no persisted opinion yet".
|
||||
func (s *Store) LinkByTriple(local, remote string, port int) (Link, bool, error) {
|
||||
var l Link
|
||||
err := s.db.QueryRow("SELECT id, local, remote, remote_port, offset_x, offset_y, grp, disabled FROM links WHERE local = ? AND remote = ? AND remote_port = ?", local, remote, port).
|
||||
Scan(&l.ID, &l.Local, &l.Remote, &l.RemotePort, &l.OffsetX, &l.OffsetY, &l.Group, &l.Disabled)
|
||||
if err != nil {
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Link{}, false, nil
|
||||
}
|
||||
return Link{}, false, err
|
||||
}
|
||||
return l, true, nil
|
||||
}
|
||||
|
||||
// DeleteLink removes a single link by id.
|
||||
func (s *Store) DeleteLink(id int64) error {
|
||||
_, err := s.db.Exec("DELETE FROM links WHERE id = ?", id)
|
||||
@ -639,6 +668,11 @@ func (s *Store) ReplaceLinks(links []Link) error {
|
||||
// SetLinkDisabled flips the disabled flag of a forward identified by its
|
||||
// (local, remote, remotePort) natural key. This is the persistence half of the
|
||||
// forwards-page start/stop toggle; the caller also drives the worker/ring side.
|
||||
//
|
||||
// Kept as a targeted UPDATE rather than a ReplaceLinks rewrite on purpose:
|
||||
// ReplaceLinks deletes and reinserts every row, handing out fresh autoincrement
|
||||
// ids and invalidating any Link a caller captured earlier (they travel inside
|
||||
// ring tokens). Flipping one flag must not perturb other rows' identity.
|
||||
func (s *Store) SetLinkDisabled(local, remote string, port int, disabled bool) error {
|
||||
_, err := s.db.Exec(
|
||||
"UPDATE links SET disabled = ? WHERE local = ? AND remote = ? AND remote_port = ?",
|
||||
@ -647,6 +681,52 @@ func (s *Store) SetLinkDisabled(local, remote string, port int, disabled bool) e
|
||||
return err
|
||||
}
|
||||
|
||||
// ReconcileLinkDisabled applies a cluster-wide view of one forward's disabled
|
||||
// flag into the local store, creating a placeholder row when this node has none
|
||||
// yet.
|
||||
//
|
||||
// This is the receive half of disabled-flag propagation. The stop decision is
|
||||
// made on whichever node served the request, then rides the token ring in the
|
||||
// topology entry; every other member calls this on adoption so its own
|
||||
// links table agrees. Without it the flag lived only on the node that handled
|
||||
// the request, and the node actually OWNS the forward — usually a different
|
||||
// machine — still believed the forward was enabled and re-spawned its worker.
|
||||
//
|
||||
// A placeholder row is deliberate: a node that has never seen the forward still
|
||||
// needs to remember "this is stopped" so a later claim on this node cannot
|
||||
// resurrect it. The placeholder carries the same natural key, so a subsequent
|
||||
// real claim fills in the rest.
|
||||
func (s *Store) ReconcileLinkDisabled(local, remote string, port int, disabled bool) error {
|
||||
cur, found, err := s.LinkByTriple(local, remote, port)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if found {
|
||||
if cur.Disabled == disabled {
|
||||
return nil // already agrees; avoid needless writes every token cycle
|
||||
}
|
||||
return s.SetLinkDisabled(local, remote, port, disabled)
|
||||
}
|
||||
if !disabled {
|
||||
// Nothing to remember: an unknown forward with no entry is simply
|
||||
// "not stopped", which is the default the claim path already assumes.
|
||||
return nil
|
||||
}
|
||||
// Need a placeholder, which requires the local/remote foreign keys to exist.
|
||||
if _, ok := s.GetLocal(local); !ok {
|
||||
return nil // cannot materialise a link without its local peer row
|
||||
}
|
||||
if _, ok := s.GetRemote(remote); !ok {
|
||||
return nil
|
||||
}
|
||||
links, err := s.ListLinks()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
links = append(links, Link{Local: local, Remote: remote, RemotePort: port, Disabled: true})
|
||||
return s.ReplaceLinks(links)
|
||||
}
|
||||
|
||||
// SetLinkGroup assigns a management group label to a forward identified by its
|
||||
// (local, remote, remotePort) natural key. Empty string clears the group
|
||||
// (moves the forward to 未分组). This is the persistence half of the
|
||||
@ -673,6 +753,13 @@ func (s *Store) Settings() (Settings, error) {
|
||||
if err := json.Unmarshal([]byte(raw), &st); err != nil {
|
||||
return Settings{AutoStartProfiles: true, RestartOnExit: true, RestartIntervalSeconds: 5}, nil
|
||||
}
|
||||
// BalanceMode lives in its own column (see UpdateSettings). Surface it here
|
||||
// too so callers reading Settings get the effective value, but never let it
|
||||
// overwrite the column on a write-back: UpdateSettings only mirrors a
|
||||
// non-empty value, so an unrelated settings PUT cannot clear a UI choice.
|
||||
if mode, ok, err := s.BalanceMode(); err == nil && ok {
|
||||
st.BalanceMode = mode
|
||||
}
|
||||
return st, nil
|
||||
}
|
||||
|
||||
@ -685,7 +772,36 @@ func (s *Store) UpdateSettings(st Settings) error {
|
||||
"INSERT INTO settings(key, value) VALUES('settings', ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value",
|
||||
string(raw),
|
||||
)
|
||||
return err
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if mode := st.BalanceMode; mode != "" {
|
||||
// Mirror into its own column for the same reason the other flagged
|
||||
// fields have one: the settings JSON is rewritten wholesale, so a
|
||||
// clustered field that rides that JSON would flip back and forth between
|
||||
// nodes instead of converging.
|
||||
if _, err := s.db.Exec(
|
||||
"INSERT INTO settings(key, value) VALUES('balance_mode', ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value",
|
||||
mode,
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// BalanceMode returns the stored load-balancing mode, and whether one was ever
|
||||
// stored (false = never configured here; the environment default applies).
|
||||
func (s *Store) BalanceMode() (string, bool, error) {
|
||||
var v string
|
||||
err := s.db.QueryRow("SELECT value FROM settings WHERE key = 'balance_mode'").Scan(&v)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return "", false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
return v, v != "", nil
|
||||
}
|
||||
|
||||
// SetNodeKey persists just the cluster admission key, preserving all other
|
||||
|
||||
@ -2,11 +2,11 @@
|
||||
//
|
||||
// Value is overridable at build time so release artifacts carry the real tag:
|
||||
//
|
||||
// go build -ldflags="-X webui4frpc/internal/version.Version=0.1.1" ./cmd/webui4frpc
|
||||
// go build -ldflags="-X webui4frpc/internal/version.Version=0.1.4" ./cmd/webui4frpc
|
||||
//
|
||||
// Kept in its own package because both cmd/ and internal/httpapi report it
|
||||
// (main can't be imported, and httpapi must not depend on cmd).
|
||||
package version
|
||||
|
||||
// Version is the running build's semantic version, without a leading "v".
|
||||
var Version = "0.1.1"
|
||||
var Version = "0.1.4"
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
Name: webui4frpc
|
||||
Version: %{?_w4f_version}%{!?_w4f_version:0.1.1}
|
||||
Version: %{?_w4f_version}%{!?_w4f_version:0.1.4}
|
||||
Release: 1%{?dist}
|
||||
Summary: Visual frpc controller with token-ring cluster
|
||||
License: MIT
|
||||
@ -67,6 +67,21 @@ fi
|
||||
%doc %{_docdir}/webui4frpc/README.md
|
||||
|
||||
%changelog
|
||||
* Mon Oct 06 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.4-1
|
||||
- 负载均衡:改为每节点自主释放(持有数超过平摊份额且 >=2 时自撤销一条丢回环,由最闲节点认领),不再由 leader 统一调度
|
||||
- 负载均衡:新增自限上界与下限(撤到平摊值即停、低于 2 条不撤),10 分钟冷却,默认不动作(dry-run)
|
||||
- 集群页:新增负载均衡视图与每节点模式开关(关闭/观察/启用),模式持久化
|
||||
- 新增只读接口 GET /cluster/balance 与写接口 POST /cluster/balance/mode
|
||||
* Sun Oct 05 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.3-1
|
||||
- 窄屏布局:补 box-sizing 重置、接管画布 fit(缩放下限 0.65)、弹窗宽度钳制、统一滚动容器(修复手机端整页无法滑动)
|
||||
- 集群:停用/启动改为经令牌环跨节点传播,撤销不再静默失效
|
||||
- 集群:修复滚动重启导致的分裂脑、令牌环冻结、非 leader 不重建 worker
|
||||
- 集群:令牌轮转日志降级为 debug(355 → 1 行/分钟)
|
||||
* Wed Sep 03 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.2-1
|
||||
- 修复窄屏底部导航栏错位到顶部:顶栏的 backdrop-filter 会为
|
||||
position:fixed 后代创建包含块,使 tab bar 的 bottom:0 相对顶栏而
|
||||
非视口定位。窄屏下关掉顶栏模糊,改用近实色底。
|
||||
|
||||
* Wed Sep 02 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.1-1
|
||||
- 修复令牌环冻结:心跳到达时复活被误判 offline 的前驱;
|
||||
全环无可达后继时保留 inflight,让 leader 超时重发令牌。
|
||||
|
||||
@ -79,7 +79,7 @@ cd web && npm run build && rm -rf internal/httpapi/dist && cp -r web/dist intern
|
||||
### Linux (Debian / Ubuntu)
|
||||
|
||||
```bash
|
||||
sudo dpkg -i webui4frpc-0.1.1-linux-amd64.deb # 或 linux-arm64.deb
|
||||
sudo dpkg -i webui4frpc-0.1.4-linux-amd64.deb # 或 linux-arm64.deb
|
||||
sudo nano /etc/webui4frpc/webui4frpc.env # 改掉默认密码
|
||||
sudo systemctl restart webui4frpc
|
||||
```
|
||||
@ -87,7 +87,7 @@ sudo systemctl restart webui4frpc
|
||||
### Linux (RHEL / Fedora / openEuler / 默认 / 龙蜥)
|
||||
|
||||
```bash
|
||||
sudo rpm -i webui4frpc-0.1.1-linux-x86_64.rpm
|
||||
sudo rpm -i webui4frpc-0.1.4-linux-x86_64.rpm
|
||||
sudo vi /etc/webui4frpc/webui4frpc.env
|
||||
sudo systemctl enable --now webui4frpc
|
||||
```
|
||||
@ -97,7 +97,7 @@ sudo systemctl enable --now webui4frpc
|
||||
|
||||
### Windows
|
||||
|
||||
双击 `webui4frpc-0.1.1-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
||||
双击 `webui4frpc-0.1.4-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
||||
安装向导会询问监听地址与管理员凭据,完成后:
|
||||
|
||||
- 程序装到 `C:\Program Files\webui4frpc`
|
||||
@ -107,8 +107,8 @@ sudo systemctl enable --now webui4frpc
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
sudo installer -pkg webui4frpc-0.1.1-darwin-arm64.pkg -target / # Apple Silicon
|
||||
sudo installer -pkg webui4frpc-0.1.1-darwin-amd64.pkg -target / # Intel
|
||||
sudo installer -pkg webui4frpc-0.1.4-darwin-arm64.pkg -target / # Apple Silicon
|
||||
sudo installer -pkg webui4frpc-0.1.4-darwin-amd64.pkg -target / # Intel
|
||||
```
|
||||
|
||||
或直接双击 `.pkg` 走图形安装器。安装后 launchd 开机自启,配置在
|
||||
@ -126,7 +126,7 @@ sudo launchctl kickstart -k system/com.jianfeeeee.webui4frpc
|
||||
不想装服务只想跑一下,用对应的 `.tar.gz` / `.zip`,解压即用:
|
||||
|
||||
```bash
|
||||
tar xzf webui4frpc-0.1.1-linux-amd64.tar.gz
|
||||
tar xzf webui4frpc-0.1.4-linux-amd64.tar.gz
|
||||
./webui4frpc -addr 127.0.0.1:7500 -user admin -password 你的密码 -workdir ./data
|
||||
```
|
||||
|
||||
|
||||
@ -12,7 +12,7 @@
|
||||
# 在 release.sh 流程内被调用;依赖 dist/artifacts/<version>/bin/<os>-<arch>/ 下的二进制。
|
||||
set -euo pipefail
|
||||
|
||||
VERSION="${1:-0.1.1}"
|
||||
VERSION="${1:-0.1.4}"
|
||||
VERSION="${VERSION#v}"
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
OUT="$ROOT/dist/artifacts/$VERSION"
|
||||
|
||||
137
scripts/probe-with-virtual-frps.sh
Executable file
137
scripts/probe-with-virtual-frps.sh
Executable file
@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env bash
|
||||
# 用隔离的虚拟 frps 探测 webui4frpc 行为。
|
||||
#
|
||||
# 全部端口与现网隔离(现网:7000/4434/7500/18082),不修改任何生产配置。
|
||||
# 用法:scripts/probe-with-virtual-frps.sh
|
||||
set -euo pipefail
|
||||
|
||||
LAB="${LAB:-/tmp/frplab}"
|
||||
FRP_VER="${FRP_VER:-0.61.1}"
|
||||
FRP_TGZ="frp_${FRP_VER}_linux_amd64"
|
||||
UI_ADDR="${UI_ADDR:-127.0.0.1:17500}"
|
||||
UI_USER="${UI_USER:-labadmin}"
|
||||
UI_PASS="${UI_PASS:-labpass123}"
|
||||
LAB_TOKEN="${LAB_TOKEN:-lab-token-$(date +%Y%m%d)}"
|
||||
|
||||
# 现网 frpc 二进制(唯一允许从生产读取的东西,只读)
|
||||
SRC_FRPC="${SRC_FRPC:-/opt/webui4frpc/data/bin/frpc-${FRP_VER}/frpc}"
|
||||
BACKEND_PORT=17090
|
||||
TCP_PORT=17091
|
||||
UI_FORWARD_PORT=17092
|
||||
|
||||
cleanup() {
|
||||
echo "==> 清理"
|
||||
pkill -f "webui4frpc -addr ${UI_ADDR}" 2>/dev/null || true
|
||||
pkill -f "frplab/.*frpc" 2>/dev/null || true
|
||||
[[ -n "${FRPS_PID:-}" ]] && kill "${FRPS_PID}" 2>/dev/null || true
|
||||
[[ -n "${BACKEND_PID:-}" ]] && kill "${BACKEND_PID}" 2>/dev/null || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "${LAB}"/{frps,probe,logs,ui/data}
|
||||
|
||||
# ---------- 1. frps(虚拟服务端) ----------
|
||||
if [[ ! -x "${LAB}/frpsd" ]]; then
|
||||
echo "==> 下载 frps ${FRP_VER}"
|
||||
curl -sL -o "${LAB}/frps-bin" \
|
||||
"https://github.com/fatedier/frp/releases/download/v${FRP_VER}/${FRP_TGZ}.tar.gz"
|
||||
tar xzf "${LAB}/frps-bin" -C "${LAB}"
|
||||
cp "${LAB}/${FRP_TGZ}/frps" "${LAB}/frpsd"
|
||||
chmod +x "${LAB}/frpsd"
|
||||
fi
|
||||
|
||||
cat > "${LAB}/frps/frps.toml" <<EOF
|
||||
bindPort = 17000
|
||||
vhostHTTPPort = 17080
|
||||
vhostHTTPsPort = 17443
|
||||
auth.token = "${LAB_TOKEN}"
|
||||
log.to = "${LAB}/logs/frps.log"
|
||||
log.level = "debug"
|
||||
EOF
|
||||
|
||||
echo "==> 启动虚拟 frps (17000/17080/17443)"
|
||||
"${LAB}/frpsd" -c "${LAB}/frps/frps.toml" > "${LAB}/logs/frps-stdout.log" 2>&1 &
|
||||
FRPS_PID=$!
|
||||
sleep 3
|
||||
ss -ltn | grep -E ':17000|:17443' || { echo "frps 未监听"; exit 1; }
|
||||
|
||||
# ---------- 2. 后端服务 ----------
|
||||
echo "==> 启动后端 http://127.0.0.1:${BACKEND_PORT}"
|
||||
( cd "${LAB}" && python3 -m http.server "${BACKEND_PORT}" --bind 127.0.0.1 > logs/backend.log 2>&1 & )
|
||||
BACKEND_PID=$!
|
||||
sleep 1
|
||||
|
||||
# ---------- 3. 裸 frpc 验证 frp 协议本身 ----------
|
||||
cat > "${LAB}/probe/frpc-probe.json" <<EOF
|
||||
{
|
||||
"serverAddr": "127.0.0.1",
|
||||
"serverPort": 17000,
|
||||
"loginFailExit": false,
|
||||
"auth": {"method": "token", "token": "${LAB_TOKEN}"},
|
||||
"proxies": [
|
||||
{"name":"lab-tcp","type":"tcp","localIP":"127.0.0.1","localPort":${BACKEND_PORT},"remotePort":${TCP_PORT}}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
cp "${SRC_FRPC}" "${LAB}/frpc" && chmod +x "${LAB}/frpc"
|
||||
echo "==> 裸 frpc 连接(验证 frp 协议通路)"
|
||||
"${LAB}/frpc" -c "${LAB}/probe/frpc-probe.json" > "${LAB}/logs/frpc.log" 2>&1 &
|
||||
sleep 4
|
||||
grep -q 'login to server success' "${LAB}/logs/frpc.log" \
|
||||
&& echo " frpc 登录成功" || { echo " frpc 登录失败"; tail -5 "${LAB}/logs/frpc.log"; }
|
||||
printf " tcp ${TCP_PORT} -> "; curl -s -o /dev/null -w "%{http_code}\n" "http://127.0.0.1:${TCP_PORT}/"
|
||||
pkill -f 'frplab/frpc' 2>/dev/null || true
|
||||
|
||||
# ---------- 4. webui4frpc 接管 ----------
|
||||
echo "==> 启动 webui4frpc ${UI_ADDR}(独立 workdir)"
|
||||
mkdir -p "${LAB}/ui/data/bin/frpc-${FRP_VER}"
|
||||
cp "${SRC_FRPC}" "${LAB}/ui/data/bin/frpc-${FRP_VER}/frpc"
|
||||
chmod +x "${LAB}/ui/data/bin/frpc-${FRP_VER}/frpc"
|
||||
( setsid /opt/webui4frpc/webui4frpc -addr "${UI_ADDR}" \
|
||||
-user "${UI_USER}" -password "${UI_PASS}" \
|
||||
-workdir "${LAB}/ui/data" > "${LAB}/logs/ui.log" 2>&1 < /dev/null & )
|
||||
sleep 5
|
||||
ss -ltn | grep -q "${UI_ADDR##*:}" || { echo "webui4frpc 未监听"; exit 1; }
|
||||
|
||||
AUTH="${UI_USER}:${UI_PASS}"
|
||||
|
||||
echo "==> 注入画布(注意:loopback 会被自动改写为 LAN 地址)"
|
||||
curl -s -u "${AUTH}" -X PUT "http://${UI_ADDR}/api/manager/canvas" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"remotes\":[{\"name\":\"lab-frps\",\"ip\":\"127.0.0.1\",\"port\":17000,\"token\":\"${LAB_TOKEN}\",\"enabled\":true}],
|
||||
\"locals\":[{\"name\":\"lab-backend\",\"ip\":\"127.0.0.1\",\"port\":${BACKEND_PORT}}],
|
||||
\"links\":[{\"local\":\"lab-backend\",\"remote\":\"lab-frps\",\"remotePort\":${UI_FORWARD_PORT}}]}" \
|
||||
| python3 -m json.tool
|
||||
|
||||
echo "==> binaryPath 必须先写入 settings,否则 worker 静默不启动"
|
||||
python3 - "${LAB}/ui/data/manager.db" "${LAB}/ui/data/bin/frpc-${FRP_VER}/frpc" <<'PY'
|
||||
import json, sqlite3, sys
|
||||
db, binary = sys.argv[1], sys.argv[2]
|
||||
c = sqlite3.connect(db)
|
||||
raw = c.execute("select value from settings where key='settings'").fetchone()[0]
|
||||
d = json.loads(raw); d["binaryPath"] = binary
|
||||
c.execute("update settings set value=? where key='settings'", (json.dumps(d),))
|
||||
c.commit()
|
||||
print(" binaryPath 已写入:", binary)
|
||||
PY
|
||||
|
||||
echo "==> 启动 forward(非 localOnly 需 owner claim,观察 status)"
|
||||
curl -s -u "${AUTH}" -X POST "http://${UI_ADDR}/api/manager/forwards/start" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"local\":\"lab-backend\",\"remote\":\"lab-frps\",\"remotePort\":${UI_FORWARD_PORT}}"
|
||||
echo
|
||||
sleep 5
|
||||
|
||||
echo "==> 观察结果"
|
||||
curl -s -u "${AUTH}" "http://${UI_ADDR}/api/manager/status" | python3 -c "
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
print(' forwards:', [(f['local'], f['active']) for f in d.get('forwards',[])])
|
||||
print(' profiles:', [(p['name'], p['process']['state'], p.get('connState')) for p in d.get('profiles',[])])
|
||||
"
|
||||
echo " binaryPath: $(curl -s -u "${AUTH}" "http://${UI_ADDR}/api/manager/binary/status")"
|
||||
echo " configs/: $(ls "${LAB}/ui/data/configs/" 2>/dev/null | tr '\n' ' ')"
|
||||
echo " workers: $(pgrep -af "${LAB}/ui/data" | wc -l) 个"
|
||||
echo
|
||||
echo "日志: ${LAB}/logs/"
|
||||
echo "复现结论见 docs/how-it-works.md"
|
||||
@ -3,7 +3,7 @@
|
||||
# webui4frpc 多平台发布脚本
|
||||
#
|
||||
# 用法:
|
||||
# ./scripts/release.sh [version] # 默认 version=0.1.1
|
||||
# ./scripts/release.sh [version] # 默认 version=0.1.4
|
||||
#
|
||||
# 行为:
|
||||
# 1. 构建前端 (web/dist) 并同步到 internal/httpapi/dist (go:embed 用)
|
||||
@ -19,7 +19,7 @@
|
||||
# NO_BUILD_FRONTEND 设为 1 跳过前端构建 (用现有 dist)
|
||||
set -euo pipefail
|
||||
|
||||
VERSION="${1:-0.1.1}"
|
||||
VERSION="${1:-0.1.4}"
|
||||
VERSION="${VERSION#v}" # 去掉可能的前导 v
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
OUT="$ROOT/dist/artifacts/$VERSION"
|
||||
|
||||
@ -2,7 +2,8 @@
|
||||
"""上传 release 资产到 gitcode(两步:取签名 URL → PUT 到 OBS)。
|
||||
|
||||
用法: upload_assets.py <version> <token> [file...]
|
||||
不传 file 时上传该版本目录下所有 tar.gz/zip 与 SHA256SUMS。
|
||||
不传 file 时上传该版本目录下全部发布产物:
|
||||
tar.gz / zip / deb / rpm / pkg / -setup.exe / SHA256SUMS
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
@ -38,6 +39,19 @@ def put_file(url: str, headers: dict, path: str) -> tuple[int, str]:
|
||||
return e.code, e.read().decode("utf-8", "replace")[:300]
|
||||
|
||||
|
||||
# 发布产物后缀:免安装包 + 原生安装包。
|
||||
# 注意 -setup.exe 而不是裸 .exe,避开 bin/ 里的裸二进制。
|
||||
ARTIFACT_SUFFIXES = (
|
||||
".tar.gz", ".zip", # 免安装
|
||||
".deb", ".rpm", ".pkg", # linux / macOS 安装包
|
||||
"-setup.exe", # windows 安装器
|
||||
)
|
||||
|
||||
|
||||
def is_artifact(name: str) -> bool:
|
||||
return name == "SHA256SUMS" or name.endswith(ARTIFACT_SUFFIXES)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) < 3:
|
||||
print(__doc__)
|
||||
@ -47,10 +61,7 @@ def main() -> int:
|
||||
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
|
||||
"dist", "artifacts", version,
|
||||
)
|
||||
files = sys.argv[3:] or sorted(
|
||||
f for f in os.listdir(outdir)
|
||||
if f.endswith((".tar.gz", ".zip")) or f == "SHA256SUMS"
|
||||
)
|
||||
files = sys.argv[3:] or sorted(f for f in os.listdir(outdir) if is_artifact(f))
|
||||
failed = 0
|
||||
for name in files:
|
||||
path = os.path.join(outdir, name)
|
||||
|
||||
@ -384,16 +384,37 @@ const currentLabel = computed(() => nav.value.find((n) => n.key === view.value)?
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 10px 12px;
|
||||
overflow: visible;
|
||||
overflow: hidden;
|
||||
/* 兵家必争:任何子项算错都不得弄出横向滚动条 */
|
||||
overflow-x: hidden;
|
||||
border-right: none;
|
||||
border-bottom: 1px solid var(--w4f-line);
|
||||
padding-top: max(10px, env(safe-area-inset-top));
|
||||
/* 关键:backdrop-filter 会为 position:fixed 后代创建【包含块】
|
||||
(与 transform/filter 同理)。若保留它,下面 .sb-nav 的
|
||||
bottom:0 就是相对这条顶栏定位而非视口,底部 tab bar 会贴在
|
||||
顶栏下沿、看起来固定在屏幕顶部。因此窄屏必须关掉顶栏的玻璃
|
||||
模糊,改用接近实色的底,模糊效果留在 .sb-nav 自己身上
|
||||
(元素自身的 backdrop-filter 不影响它自己的定位)。 */
|
||||
backdrop-filter: none;
|
||||
-webkit-backdrop-filter: none;
|
||||
background: color-mix(in srgb, var(--w4f-card-solid) 92%, transparent);
|
||||
}
|
||||
.sb-brand {
|
||||
padding: 0;
|
||||
gap: 8px;
|
||||
flex: 1 1 auto;
|
||||
min-width: 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
/* 品牌文字必须可收缩:“webui4frpc”是不可断行单词,不给 min-width:0
|
||||
+ overflow:hidden 的话它的 flex 自动最小尺寸等于 min-content(≨85px),
|
||||
顶栏就拿不出空间给右侧身份区,溢出为横向滚动。 */
|
||||
.sb-brand-text { min-width: 0; overflow: hidden; }
|
||||
.sb-brand-text b {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.sb-logo { width: 30px; height: 30px; border-radius: 10px; }
|
||||
.sb-brand-text b { font-size: 14.5px; }
|
||||
@ -425,7 +446,10 @@ const currentLabel = computed(() => nav.value.find((n) => n.key === view.value)?
|
||||
gap: 2px;
|
||||
padding: 5px 2px;
|
||||
border-radius: 11px;
|
||||
font-size: 10.5px;
|
||||
/* Keep the tab label at the theme's 11px floor; the 10.5px here (and 9.5px
|
||||
* at ≤380px) read as noise on a phone. "账号与密钥" is still handled by the
|
||||
* .label ellipsis (width:100% + overflow:hidden). */
|
||||
font-size: 11px;
|
||||
font-weight: 600;
|
||||
text-align: center;
|
||||
}
|
||||
@ -442,23 +466,30 @@ const currentLabel = computed(() => nav.value.find((n) => n.key === view.value)?
|
||||
box-shadow: none;
|
||||
}
|
||||
|
||||
/* 身份与主题折进顶条右侧 */
|
||||
/* 身份与主题折进顶条右侧。
|
||||
不再用 vw 限宽:vw 含垂直滚动条宽度,会比实际可用内容宽大;而且
|
||||
max-width 只封顶盒子,内部 flex:0 0 auto 的徒章/按钮仍会撑出去。
|
||||
改为全链路可收缩(min-width:0 + overflow:hidden),由 flex 自行分配。 */
|
||||
.sb-foot {
|
||||
margin-top: 0;
|
||||
padding-top: 0;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
flex: 0 0 auto;
|
||||
flex: 0 1 auto;
|
||||
min-width: 0;
|
||||
}
|
||||
.sb-identity {
|
||||
margin-bottom: 0;
|
||||
padding: 5px 8px;
|
||||
font-size: 11.5px;
|
||||
max-width: 42vw;
|
||||
flex: 0 1 auto;
|
||||
min-width: 0;
|
||||
max-width: none;
|
||||
overflow: hidden;
|
||||
}
|
||||
.sb-identity .id-name { max-width: 14vw; }
|
||||
.sb-theme { margin-bottom: 0; }
|
||||
.sb-identity .id-name { min-width: 0; max-width: none; }
|
||||
.sb-theme { margin-bottom: 0; flex: 0 0 auto; }
|
||||
.sb-theme .sw { width: 22px; height: 22px; border-radius: 7px; }
|
||||
|
||||
.main { height: auto; flex: 1; min-height: 0; }
|
||||
@ -470,12 +501,15 @@ const currentLabel = computed(() => nav.value.find((n) => n.key === view.value)?
|
||||
}
|
||||
}
|
||||
|
||||
/* 极窄(≤ 380px,iPhone SE / 小屏安卓):进一步压缩顶条 */
|
||||
/* 极窄(≤ 380px,iPhone SE / 小屏安卓):进一步压缩顶条。
|
||||
角色徒章是 flex:0 0 auto 不可收缩的(“超级管理员”≨70px),极窄下隐去,
|
||||
用户名与退出按钮保留。
|
||||
注意:底部 tab 文字不再随之降到 9.5px —— 它在手机上低于可读下限
|
||||
(见 .sb-i 的 11px 注释),长标签交给 .label 的省略号处理。 */
|
||||
@media (max-width: 380px) {
|
||||
.sb-brand-text b { font-size: 13px; }
|
||||
.sb-identity { max-width: 38vw; padding: 4px 7px; }
|
||||
.sb-identity { padding: 4px 7px; }
|
||||
.sb-identity .id-level { display: none; }
|
||||
.sb-i { font-size: 9.5px; }
|
||||
.content { padding-left: 10px; padding-right: 10px; }
|
||||
}
|
||||
</style>
|
||||
|
||||
@ -2,6 +2,7 @@
|
||||
import type {
|
||||
ApiKey,
|
||||
ApiKeyCreated,
|
||||
BalanceReport,
|
||||
BinaryStatus,
|
||||
CacheResp,
|
||||
CanvasData,
|
||||
@ -159,6 +160,21 @@ export const api = {
|
||||
// M6 token ring snapshot.
|
||||
ring: () => request<RingSnapshot>("/api/manager/cluster/ring"),
|
||||
|
||||
// Load balancer read-only view. Reports per-node scores, whether anything is
|
||||
// over the threshold, and the migration it would perform. Never mutates the
|
||||
// cluster. 404 when the balancer is not wired up on that node.
|
||||
balance: () => request<BalanceReport>("/api/manager/cluster/balance"),
|
||||
|
||||
// setBalanceMode switches load balancing on THIS node (off | dry-run | on).
|
||||
// Per-node by design: each member releases only its own forwards, so there is
|
||||
// no cluster-wide switch to flip. Requires write level.
|
||||
setBalanceMode: (mode: string) =>
|
||||
request<BalanceReport>("/api/manager/cluster/balance/mode", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ mode }),
|
||||
}),
|
||||
|
||||
// M6 cluster lifecycle (创建/加入/退出/移除).
|
||||
// clusterCreate: reseed THIS node as a fresh standalone leader (创建集群).
|
||||
clusterCreate: () =>
|
||||
|
||||
@ -184,6 +184,20 @@
|
||||
|
||||
/* ---- blue theme (ocean × frost) ---- */
|
||||
|
||||
/* ---- box model reset ----
|
||||
* This app had NO box-sizing reset, so every element using `width: 100%` with
|
||||
* horizontal padding overflowed by exactly its padding under the default
|
||||
* content-box model. The narrow-screen top bar was the visible victim:
|
||||
* `.sidebar { width: 100%; padding: 10px 12px }` rendered 375+24 = 399px wide
|
||||
* on a 375px viewport (344px at 320px), pushing the theme swatches and logout
|
||||
* button off the right edge where `overflow-x: hidden` silently clipped them.
|
||||
* It also made the desktop sidebar 246+28 = 274px wide instead of 246px.
|
||||
* border-box makes width include padding+border, which every layout here
|
||||
* already assumes (Element Plus sets it on its own components already). */
|
||||
*,
|
||||
*::before,
|
||||
*::after { box-sizing: border-box; }
|
||||
|
||||
html,
|
||||
body,
|
||||
#app {
|
||||
@ -346,4 +360,14 @@ a { color: var(--w4f-primary-h); }
|
||||
/* ---- Element Plus dialog tweaks to read as glass ---- */
|
||||
.el-overlay { backdrop-filter: blur(3px); }
|
||||
.el-dialog { border-radius: var(--w4f-radius) !important; box-shadow: var(--w4f-sh-lg) !important; }
|
||||
|
||||
/* Dialogs pass a fixed px width (width="420px" / "440px"), which Element Plus
|
||||
* writes into an INLINE `--el-dialog-width` custom property. On phones that is
|
||||
* wider than the viewport (420px on a 375px screen), so the dialog's right edge
|
||||
* — including the header close button — starts off-screen and is only reachable
|
||||
* by scrolling the overlay horizontally. Clamp it to the viewport; !important
|
||||
* is required to beat the inline custom property. */
|
||||
@media (max-width: 720px) {
|
||||
.el-dialog { --el-dialog-width: calc(100vw - 24px) !important; }
|
||||
}
|
||||
.el-message { border-radius: var(--w4f-radius-sm) !important; box-shadow: var(--w4f-sh-lg) !important; }
|
||||
|
||||
490
web/src/types.ts
490
web/src/types.ts
@ -9,332 +9,382 @@ export type CanvasRemote = Remote;
|
||||
export type CanvasLink = Link;
|
||||
|
||||
export interface Local {
|
||||
name: string;
|
||||
ip: string;
|
||||
port: number;
|
||||
protocol: string; // tcp | udp | http | https
|
||||
// localOnly: forward stays on this node (loopback kept), no cluster.
|
||||
localOnly?: boolean;
|
||||
name: string;
|
||||
ip: string;
|
||||
port: number;
|
||||
protocol: string; // tcp | udp | http | https
|
||||
// localOnly: forward stays on this node (loopback kept), no cluster.
|
||||
localOnly?: boolean;
|
||||
|
||||
// M1 advanced transport knobs (optional; empty/zero = frpc defaults)
|
||||
useEncryption?: boolean;
|
||||
useCompression?: boolean;
|
||||
bandwidthLimit?: string;
|
||||
poolCount?: number;
|
||||
metadatas?: Record<string, string>;
|
||||
annotations?: Record<string, string>;
|
||||
// M1 advanced transport knobs (optional; empty/zero = frpc defaults)
|
||||
useEncryption?: boolean;
|
||||
useCompression?: boolean;
|
||||
bandwidthLimit?: string;
|
||||
poolCount?: number;
|
||||
metadatas?: Record<string, string>;
|
||||
annotations?: Record<string, string>;
|
||||
|
||||
// M2 HTTP/HTTPS routing (http/https only)
|
||||
customDomains?: string; // comma-separated
|
||||
subdomain?: string;
|
||||
locations?: string[];
|
||||
hostHeaderRewrite?: string;
|
||||
httpHeaders?: Record<string, string>;
|
||||
basicAuthUser?: string;
|
||||
basicAuthPassword?: string;
|
||||
// M2 HTTP/HTTPS routing (http/https only)
|
||||
customDomains?: string; // comma-separated
|
||||
subdomain?: string;
|
||||
locations?: string[];
|
||||
hostHeaderRewrite?: string;
|
||||
httpHeaders?: Record<string, string>;
|
||||
basicAuthUser?: string;
|
||||
basicAuthPassword?: string;
|
||||
|
||||
// M3 load balancing group + health check
|
||||
lbGroup?: string;
|
||||
lbGroupKey?: string;
|
||||
healthCheckType?: string; // "" | tcp | http
|
||||
healthCheckPath?: string;
|
||||
healthCheckTimeout?: number;
|
||||
healthCheckMaxFailed?: number;
|
||||
healthCheckInterval?: number;
|
||||
// M3 load balancing group + health check
|
||||
lbGroup?: string;
|
||||
lbGroupKey?: string;
|
||||
healthCheckType?: string; // "" | tcp | http
|
||||
healthCheckPath?: string;
|
||||
healthCheckTimeout?: number;
|
||||
healthCheckMaxFailed?: number;
|
||||
healthCheckInterval?: number;
|
||||
}
|
||||
|
||||
export interface Remote {
|
||||
name: string;
|
||||
ip: string;
|
||||
port: number; // frps connect port
|
||||
token?: string;
|
||||
url?: string;
|
||||
enabled: boolean;
|
||||
name: string;
|
||||
ip: string;
|
||||
port: number; // frps connect port
|
||||
token?: string;
|
||||
url?: string;
|
||||
enabled: boolean;
|
||||
|
||||
// M1 transport section
|
||||
transportProtocol?: string; // tcp | quic | kcp | websocket
|
||||
transportTls?: boolean;
|
||||
transportPool?: number;
|
||||
transportTlsServerName?: string;
|
||||
// M1 transport section
|
||||
transportProtocol?: string; // tcp | quic | kcp | websocket
|
||||
transportTls?: boolean;
|
||||
transportPool?: number;
|
||||
transportTlsServerName?: string;
|
||||
|
||||
// M2: frps vhostHTTPPort
|
||||
vhostHttpPort?: number;
|
||||
// M2: frps vhostHTTPPort
|
||||
vhostHttpPort?: number;
|
||||
|
||||
// M3: local frpc admin API (webServer)
|
||||
adminAddr?: string;
|
||||
adminPort?: number;
|
||||
adminUser?: string;
|
||||
adminPassword?: string;
|
||||
// M3: local frpc admin API (webServer)
|
||||
adminAddr?: string;
|
||||
adminPort?: number;
|
||||
adminUser?: string;
|
||||
adminPassword?: string;
|
||||
}
|
||||
|
||||
export interface Link {
|
||||
id?: number;
|
||||
local: string;
|
||||
remote: string;
|
||||
remotePort: number;
|
||||
offsetX?: number;
|
||||
offsetY?: number;
|
||||
// group: management label for one-click group start/stop on the forwards
|
||||
// page (unrelated to frps load-balancing lbGroup on Local).
|
||||
group?: string;
|
||||
// disabled: per-forward stop flag. A disabled forward is omitted from the
|
||||
// frpc config (local) / left out of the ring topology (cluster), so stop is
|
||||
// per-card and survives canvas re-saves.
|
||||
disabled?: boolean;
|
||||
id?: number;
|
||||
local: string;
|
||||
remote: string;
|
||||
remotePort: number;
|
||||
offsetX?: number;
|
||||
offsetY?: number;
|
||||
// group: management label for one-click group start/stop on the forwards
|
||||
// page (unrelated to frps load-balancing lbGroup on Local).
|
||||
group?: string;
|
||||
// disabled: per-forward stop flag. A disabled forward is omitted from the
|
||||
// frpc config (local) / left out of the ring topology (cluster), so stop is
|
||||
// per-card and survives canvas re-saves.
|
||||
disabled?: boolean;
|
||||
}
|
||||
|
||||
export interface CanvasData {
|
||||
locals: Local[];
|
||||
remotes: Remote[];
|
||||
links: Link[];
|
||||
locals: Local[];
|
||||
remotes: Remote[];
|
||||
links: Link[];
|
||||
}
|
||||
|
||||
export interface Settings {
|
||||
autoStartProfiles: boolean;
|
||||
restartOnExit: boolean;
|
||||
restartIntervalSeconds: number;
|
||||
binaryPath?: string;
|
||||
autoStartProfiles: boolean;
|
||||
restartOnExit: boolean;
|
||||
restartIntervalSeconds: number;
|
||||
binaryPath?: string;
|
||||
}
|
||||
|
||||
export interface ProcessStatus {
|
||||
state: string;
|
||||
pid?: number;
|
||||
startTime?: number;
|
||||
restartCount: number;
|
||||
exitCode?: number;
|
||||
err?: string;
|
||||
state: string;
|
||||
pid?: number;
|
||||
startTime?: number;
|
||||
restartCount: number;
|
||||
exitCode?: number;
|
||||
err?: string;
|
||||
}
|
||||
|
||||
export interface ProxyState {
|
||||
name: string;
|
||||
type: string;
|
||||
status: string; // new | wait start | start error | running | check failed | closed
|
||||
err?: string;
|
||||
local_addr?: string;
|
||||
remote_addr?: string;
|
||||
name: string;
|
||||
type: string;
|
||||
status: string; // new | wait start | start error | running | check failed | closed
|
||||
err?: string;
|
||||
local_addr?: string;
|
||||
remote_addr?: string;
|
||||
}
|
||||
|
||||
export interface StatusProfile {
|
||||
name: string;
|
||||
enabled: boolean;
|
||||
process: ProcessStatus;
|
||||
hasProcess: boolean;
|
||||
forwards: { service: string; remotePort: number; localPort?: number }[];
|
||||
// connState: derived frps connection state (we do not control remote frps)
|
||||
connState?: string; // connected | connecting | failed | not_started | disabled
|
||||
// M3: admin API state
|
||||
adminEnabled?: boolean;
|
||||
proxyStates?: ProxyState[];
|
||||
groupCount?: number;
|
||||
name: string;
|
||||
enabled: boolean;
|
||||
process: ProcessStatus;
|
||||
hasProcess: boolean;
|
||||
forwards: { service: string; remotePort: number; localPort?: number }[];
|
||||
// connState: derived frps connection state (we do not control remote frps)
|
||||
connState?: string; // connected | connecting | failed | not_started | disabled
|
||||
// M3: admin API state
|
||||
adminEnabled?: boolean;
|
||||
proxyStates?: ProxyState[];
|
||||
groupCount?: number;
|
||||
}
|
||||
|
||||
export interface StatusResp {
|
||||
version: string;
|
||||
workDir: string;
|
||||
settings: Settings;
|
||||
services: Local[];
|
||||
remotes: Remote[];
|
||||
binaryPath: string;
|
||||
profiles: StatusProfile[];
|
||||
localStatus: LocalStatus[];
|
||||
// forwards: the forward-centric list rendered as cards on the status page.
|
||||
forwards: ForwardStatus[];
|
||||
// selfId: this node's ring ID (== SelfAddr), to tell whether a cluster
|
||||
// forward's owner is this node (worker runs here) or a peer.
|
||||
selfId: string;
|
||||
version: string;
|
||||
workDir: string;
|
||||
settings: Settings;
|
||||
services: Local[];
|
||||
remotes: Remote[];
|
||||
binaryPath: string;
|
||||
profiles: StatusProfile[];
|
||||
localStatus: LocalStatus[];
|
||||
// forwards: the forward-centric list rendered as cards on the status page.
|
||||
forwards: ForwardStatus[];
|
||||
// selfId: this node's ring ID (== SelfAddr), to tell whether a cluster
|
||||
// forward's owner is this node (worker runs here) or a peer.
|
||||
selfId: string;
|
||||
}
|
||||
|
||||
// ForwardStatus is one forward (link) in the forward-centric status view.
|
||||
// kind distinguishes 本地转发 (localOnly) from 远程转发 (cluster-distributed).
|
||||
export interface ForwardStatus {
|
||||
local: string;
|
||||
remote: string;
|
||||
remotePort: number;
|
||||
localOnly: boolean;
|
||||
kind: "local" | "remote";
|
||||
ownerId?: string; // remote = topology owner; local = selfId
|
||||
active: boolean; // remote = in topology; local = local worker running
|
||||
disabled: boolean; // per-forward stop flag (true = stopped)
|
||||
group?: string;
|
||||
localIp?: string;
|
||||
localPort?: number;
|
||||
localProto?: string;
|
||||
local: string;
|
||||
remote: string;
|
||||
remotePort: number;
|
||||
localOnly: boolean;
|
||||
kind: "local" | "remote";
|
||||
ownerId?: string; // remote = topology owner; local = selfId
|
||||
active: boolean; // remote = in topology; local = local worker running
|
||||
disabled: boolean; // per-forward stop flag (true = stopped)
|
||||
group?: string;
|
||||
localIp?: string;
|
||||
localPort?: number;
|
||||
localProto?: string;
|
||||
}
|
||||
|
||||
export interface LocalTargetStatus {
|
||||
remote: string;
|
||||
remotePort: number;
|
||||
workerState: string;
|
||||
remote: string;
|
||||
remotePort: number;
|
||||
workerState: string;
|
||||
}
|
||||
|
||||
export interface LocalStatus {
|
||||
local: Local;
|
||||
targets: LocalTargetStatus[];
|
||||
local: Local;
|
||||
targets: LocalTargetStatus[];
|
||||
}
|
||||
|
||||
export interface BinaryStatus {
|
||||
binaryPath: string;
|
||||
version?: string;
|
||||
binaryPath: string;
|
||||
version?: string;
|
||||
}
|
||||
|
||||
export interface InstallResult {
|
||||
path: string;
|
||||
version: string;
|
||||
path: string;
|
||||
version: string;
|
||||
}
|
||||
|
||||
// M6: cluster node registry + binary cache.
|
||||
export interface ClusterNode {
|
||||
addr: string;
|
||||
cache?: string[];
|
||||
version?: string;
|
||||
addr: string;
|
||||
cache?: string[];
|
||||
version?: string;
|
||||
}
|
||||
|
||||
export interface CacheEntry {
|
||||
version: string;
|
||||
path: string;
|
||||
size: number;
|
||||
modTime: number;
|
||||
version: string;
|
||||
path: string;
|
||||
size: number;
|
||||
modTime: number;
|
||||
}
|
||||
|
||||
export interface ClusterNodesResp {
|
||||
nodes: ClusterNode[];
|
||||
nodes: ClusterNode[];
|
||||
}
|
||||
|
||||
export interface CacheResp {
|
||||
cache: CacheEntry[];
|
||||
removed?: string[];
|
||||
cache: CacheEntry[];
|
||||
removed?: string[];
|
||||
}
|
||||
|
||||
// M6 token-ring snapshot (GET /cluster/ring).
|
||||
export interface RingNode {
|
||||
id: string;
|
||||
addr: string;
|
||||
isLeader?: boolean;
|
||||
alive: boolean;
|
||||
load: { memPct: number; netPct: number; forwards?: number };
|
||||
version?: string;
|
||||
cache?: string[];
|
||||
lastSeen?: number;
|
||||
id: string;
|
||||
addr: string;
|
||||
isLeader?: boolean;
|
||||
alive: boolean;
|
||||
load: { memPct: number; netPct: number; forwards?: number };
|
||||
version?: string;
|
||||
cache?: string[];
|
||||
lastSeen?: number;
|
||||
}
|
||||
|
||||
export interface RingTaskInfo {
|
||||
id: string;
|
||||
revoke?: boolean;
|
||||
removeNode?: string; // node-removal command (target rides the token to self-remove)
|
||||
local: { name: string };
|
||||
remote: { name: string };
|
||||
link: { remotePort?: number };
|
||||
id: string;
|
||||
revoke?: boolean;
|
||||
removeNode?: string; // node-removal command (target rides the token to self-remove)
|
||||
local: { name: string };
|
||||
remote: { name: string };
|
||||
link: { remotePort?: number };
|
||||
}
|
||||
|
||||
export interface RingTopoEntry {
|
||||
taskId: string;
|
||||
ownerId: string;
|
||||
active: boolean;
|
||||
local: { name: string };
|
||||
remote: { name: string };
|
||||
link: { remotePort?: number };
|
||||
taskId: string;
|
||||
ownerId: string;
|
||||
active: boolean;
|
||||
local: { name: string };
|
||||
remote: { name: string };
|
||||
link: { remotePort?: number };
|
||||
}
|
||||
|
||||
export interface RingLogEntry {
|
||||
seq: number;
|
||||
node: string;
|
||||
kind: string;
|
||||
at?: number;
|
||||
data?: Record<string, any> | null;
|
||||
seq: number;
|
||||
node: string;
|
||||
kind: string;
|
||||
at?: number;
|
||||
data?: Record<string, any> | null;
|
||||
}
|
||||
|
||||
export interface RingSnapshot {
|
||||
selfId: string;
|
||||
leaderId: string; // 当前 leader(持首令牌 + 判丢失 + 被上家监控)
|
||||
cycle: number; // 周期号(leader 每收令牌回 +1)
|
||||
lastSync: number; // 上次同步时间 (unix sec)
|
||||
roundDelayMs?: number;
|
||||
nodes: RingNode[];
|
||||
pending: RingTaskInfo[];
|
||||
topology: RingTopoEntry[];
|
||||
log: RingLogEntry[];
|
||||
// nodeKey: this node's cluster admission key. Newcomers must present it to
|
||||
// join via this node (handleClusterJoin verifies). Shown on the cluster page
|
||||
// for the operator to copy; omitted for non-cluster (standalone) nodes.
|
||||
nodeKey?: string;
|
||||
selfId: string;
|
||||
leaderId: string; // 当前 leader(持首令牌 + 判丢失 + 被上家监控)
|
||||
cycle: number; // 周期号(leader 每收令牌回 +1)
|
||||
lastSync: number; // 上次同步时间 (unix sec)
|
||||
roundDelayMs?: number;
|
||||
nodes: RingNode[];
|
||||
pending: RingTaskInfo[];
|
||||
topology: RingTopoEntry[];
|
||||
log: RingLogEntry[];
|
||||
// nodeKey: this node's cluster admission key. Newcomers must present it to
|
||||
// join via this node (handleClusterJoin verifies). Shown on the cluster page
|
||||
// for the operator to copy; omitted for non-cluster (standalone) nodes.
|
||||
nodeKey?: string;
|
||||
}
|
||||
|
||||
// BalanceReport is one node's own view of rebalancing (GET /cluster/balance).
|
||||
//
|
||||
// It is per-node by design: every member runs the same rule against its own
|
||||
// forwards, and the answer that matters is the one on the node doing (or
|
||||
// refusing to do) the releasing. So this reports "my share" first and the peers
|
||||
// second, and there is no cluster-wide "would migrate" plan — the ring's normal
|
||||
// lowest-load claim decides where a released forward lands.
|
||||
export interface BalanceReleasePlan {
|
||||
local: string;
|
||||
remote: string;
|
||||
remotePort: number;
|
||||
owner: string;
|
||||
}
|
||||
|
||||
export interface BalanceReport {
|
||||
mode: "off" | "dry-run" | "on";
|
||||
modeSource: "env" | "stored"; // env = W4F_BALANCE_MODE default, stored = set here
|
||||
enabled: boolean; // mode !== off
|
||||
dryRun: boolean; // mode !== on
|
||||
canControl: boolean; // this principal may switch it (write level)
|
||||
isMember: boolean;
|
||||
self: string;
|
||||
|
||||
selfForwards: number;
|
||||
share: number; // the even split: ceil(total / alive)
|
||||
totalForwards: number;
|
||||
aliveNodes: number;
|
||||
selfOverShare: boolean;
|
||||
|
||||
// wouldRelease is present whenever this node is over its share — including
|
||||
// while it is deliberately holding back, which is what blockedBy explains.
|
||||
wouldRelease?: BalanceReleasePlan;
|
||||
releasing?: BalanceReleasePlan;
|
||||
blockedBy?: string; // single-node | below-min | balanced | nothing-active | cooldown | in-flight
|
||||
|
||||
cooldownSec: number;
|
||||
lastReleaseAt?: number;
|
||||
nodes: BalanceNodeReport[];
|
||||
}
|
||||
|
||||
export interface BalanceNodeReport {
|
||||
id: string;
|
||||
alive: boolean;
|
||||
isLeader: boolean;
|
||||
forwards: number;
|
||||
memPct: number;
|
||||
netPct: number;
|
||||
score: number;
|
||||
}
|
||||
|
||||
// ---- Auth / accounts / API keys (M7) ----
|
||||
|
||||
export type AccessLevel = 'read' | 'write' | 'admin' | 'superadmin';
|
||||
export type AccessLevel = "read" | "write" | "admin" | "superadmin";
|
||||
|
||||
// MeResp is the authenticated principal (GET /me). The frontend uses level to
|
||||
// gate the UI: viewer (= auditors) sees read + export controls only.
|
||||
export interface MeResp {
|
||||
type: 'user' | 'service' | 'key';
|
||||
name: string;
|
||||
level: AccessLevel;
|
||||
userId?: number;
|
||||
type: "user" | "service" | "key";
|
||||
name: string;
|
||||
level: AccessLevel;
|
||||
userId?: number;
|
||||
}
|
||||
|
||||
export interface User {
|
||||
id: number;
|
||||
username: string;
|
||||
role: 'admin' | 'viewer' | 'superadmin';
|
||||
enabled: boolean;
|
||||
system?: boolean; // flag-synced built-in account (UI read-only)
|
||||
createdAt: number;
|
||||
lastLoginAt: number;
|
||||
id: number;
|
||||
username: string;
|
||||
role: "admin" | "viewer" | "superadmin";
|
||||
enabled: boolean;
|
||||
system?: boolean; // flag-synced built-in account (UI read-only)
|
||||
createdAt: number;
|
||||
lastLoginAt: number;
|
||||
}
|
||||
|
||||
export interface ApiKey {
|
||||
id: number;
|
||||
userId: number;
|
||||
prefix: string;
|
||||
label: string;
|
||||
scope: AccessLevel;
|
||||
createdAt: number;
|
||||
lastUsedAt: number;
|
||||
expiresAt?: number;
|
||||
id: number;
|
||||
userId: number;
|
||||
prefix: string;
|
||||
label: string;
|
||||
scope: AccessLevel;
|
||||
createdAt: number;
|
||||
lastUsedAt: number;
|
||||
expiresAt?: number;
|
||||
}
|
||||
|
||||
// ApiKeyCreated is the one-time response from POST /apikeys: the plaintext key
|
||||
// is shown exactly once and never retrievable again.
|
||||
export interface ApiKeyCreated {
|
||||
id: number;
|
||||
userId: number;
|
||||
label: string;
|
||||
scope: AccessLevel;
|
||||
prefix: string;
|
||||
createdAt: number;
|
||||
key: string;
|
||||
id: number;
|
||||
userId: number;
|
||||
label: string;
|
||||
scope: AccessLevel;
|
||||
prefix: string;
|
||||
createdAt: number;
|
||||
key: string;
|
||||
}
|
||||
|
||||
export interface CanvasExportEnvelope {
|
||||
_type: 'webui4frpc-canvas';
|
||||
version: number;
|
||||
exportedAt: number;
|
||||
exporter: string;
|
||||
canvas: CanvasData;
|
||||
_type: "webui4frpc-canvas";
|
||||
version: number;
|
||||
exportedAt: number;
|
||||
exporter: string;
|
||||
canvas: CanvasData;
|
||||
}
|
||||
|
||||
// Worker-log bundle (GET /cluster/logs/export): one entry per alive ring node.
|
||||
export interface WorkerLogEntry {
|
||||
name: string;
|
||||
remote: string;
|
||||
state: string;
|
||||
lines: string;
|
||||
name: string;
|
||||
remote: string;
|
||||
state: string;
|
||||
lines: string;
|
||||
}
|
||||
export interface NodeLogsResp {
|
||||
node: string;
|
||||
workers: WorkerLogEntry[];
|
||||
node: string;
|
||||
workers: WorkerLogEntry[];
|
||||
}
|
||||
export interface ClusterWorkerLogNode {
|
||||
id: string;
|
||||
addr: string;
|
||||
ok: boolean;
|
||||
error?: string;
|
||||
logs?: NodeLogsResp;
|
||||
id: string;
|
||||
addr: string;
|
||||
ok: boolean;
|
||||
error?: string;
|
||||
logs?: NodeLogsResp;
|
||||
}
|
||||
export interface WorkerLogBundle {
|
||||
_type: 'webui4frpc-worker-logs';
|
||||
version: number;
|
||||
exportedAt: number;
|
||||
requester: string;
|
||||
nodes: ClusterWorkerLogNode[];
|
||||
_type: "webui4frpc-worker-logs";
|
||||
version: number;
|
||||
exportedAt: number;
|
||||
requester: string;
|
||||
nodes: ClusterWorkerLogNode[];
|
||||
}
|
||||
|
||||
@ -25,16 +25,17 @@
|
||||
<VueFlow
|
||||
v-model:nodes="nodes"
|
||||
v-model:edges="edges"
|
||||
:default-viewport="{ zoom: 0.85 }"
|
||||
:min-zoom="0.3"
|
||||
:default-viewport="{ zoom: FIT_MAX_ZOOM }"
|
||||
:min-zoom="MIN_ZOOM"
|
||||
:max-zoom="2"
|
||||
fit-view-on-init
|
||||
:delete-key-code="null"
|
||||
:nodes-draggable="canWrite"
|
||||
:nodes-connectable="canWrite"
|
||||
:edges-updatable="false"
|
||||
:edges-reconnectable="false"
|
||||
class="flow-canvas"
|
||||
@init="onFlowInit"
|
||||
@nodes-initialized="reframe"
|
||||
@connect="onConnect"
|
||||
@pane-click="deselectAll"
|
||||
@edge-click="onEdgeClick"
|
||||
@ -118,7 +119,7 @@
|
||||
<script setup lang="ts">
|
||||
import '@vue-flow/core/dist/style.css'
|
||||
import '@vue-flow/core/dist/theme-default.css'
|
||||
import { ref, computed, onMounted, onBeforeUnmount } from 'vue'
|
||||
import { ref, computed, onMounted, onBeforeUnmount, nextTick } from 'vue'
|
||||
import { VueFlow } from '@vue-flow/core'
|
||||
import { Background } from '@vue-flow/background'
|
||||
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||
@ -134,6 +135,103 @@ const nodes = ref<any[]>([])
|
||||
const edges = ref<Edge[]>([])
|
||||
const loading = ref(false)
|
||||
|
||||
// ---- viewport / zoom floor ----
|
||||
// The canvas is laid out in absolute graph units (local column at x=60, remote
|
||||
// column at x=760, and BOTH columns grow downward on every added forward), so
|
||||
// its bounding box is far wider and taller than a phone viewport. Vue Flow's
|
||||
// `fit-view-on-init` frames that whole box, which on a phone produced a scale of
|
||||
// ~0.31: node cards measure 248px in graph units but rendered 85px wide at an
|
||||
// effective font size of ~4.4px — unreadable. Even a 1280px desktop landed at
|
||||
// ~0.60 (8.4px) with five forwards stacked.
|
||||
//
|
||||
// So the canvas is framed by frameCanvas() below, clamped by a zoom floor, and
|
||||
// the canvas pans when the content no longer fits. 0.65 renders a 240px local
|
||||
// card at ~156px with a legible ~9px font.
|
||||
//
|
||||
// NOTE: `fit-view-on-init` is deliberately NOT set. It resolves on its own
|
||||
// schedule and overwrote the explicit fit performed here (observed: this code
|
||||
// measured the unfitted layout, set the viewport, and then the deferred init fit
|
||||
// landed last and won). Owning the fit outright is the only way to keep it
|
||||
// deterministic.
|
||||
//
|
||||
// The store arrives via the `init` event. That is deliberately preferred over
|
||||
// `useVueFlow()`: this component is the PARENT of <VueFlow>, and calling
|
||||
// useVueFlow() in a parent without a shared id can resolve to a different store
|
||||
// than the one the rendered flow uses. The event cannot misfire that way.
|
||||
const MIN_ZOOM = 0.65
|
||||
const FIT_MAX_ZOOM = 1
|
||||
type FlowStore = {
|
||||
fitView: (opts?: Record<string, unknown>) => Promise<boolean>
|
||||
getViewport?: () => { x: number; y: number; zoom: number }
|
||||
setViewport?: (
|
||||
t: { x: number; y: number; zoom: number },
|
||||
opts?: Record<string, unknown>,
|
||||
) => Promise<boolean>
|
||||
}
|
||||
let flowStore: FlowStore | null = null
|
||||
|
||||
// frameCanvas fits the graph, then re-anchors it to the top-left padding edge.
|
||||
//
|
||||
// Two reasons it is not just fitView():
|
||||
// 1. the zoom floor (MIN_ZOOM) — fitView alone shrank 5 stacked forwards to
|
||||
// ~0.31 on a phone, rendering 4.4px text;
|
||||
// 2. fitView centres the bounding box, which on a narrow screen puts the seam
|
||||
// between the local and remote columns mid-viewport — the user sees half of
|
||||
// each. Anchoring the top-left node to the padding edge means whole local
|
||||
// cards are visible immediately and the remote column is a pan away.
|
||||
//
|
||||
// duration:0 on both transitions — we measure node rects in between, and a
|
||||
// running animation would hand back mid-flight transforms and skew the anchor.
|
||||
const PANE_PAD_RATIO = 0.06
|
||||
const frameCanvas = async () => {
|
||||
if (!flowStore) return
|
||||
await flowStore.fitView({
|
||||
padding: 0.12,
|
||||
minZoom: MIN_ZOOM,
|
||||
maxZoom: FIT_MAX_ZOOM,
|
||||
duration: 0,
|
||||
})
|
||||
|
||||
const pane = document.querySelector<HTMLElement>('.vue-flow')
|
||||
const nodeEls = Array.from(document.querySelectorAll<HTMLElement>('.vue-flow__node'))
|
||||
if (!pane || !nodeEls.length) return
|
||||
|
||||
const paneRect = pane.getBoundingClientRect()
|
||||
const rects = nodeEls.map((el) => el.getBoundingClientRect())
|
||||
const dx =
|
||||
paneRect.left + pane.clientWidth * PANE_PAD_RATIO - Math.min(...rects.map((r) => r.left))
|
||||
const dy =
|
||||
paneRect.top + pane.clientHeight * PANE_PAD_RATIO - Math.min(...rects.map((r) => r.top))
|
||||
if (Math.abs(dx) < 1 && Math.abs(dy) < 1) return
|
||||
|
||||
const vp = flowStore.getViewport?.()
|
||||
if (!vp) return
|
||||
await flowStore.setViewport?.({ x: vp.x + dx, y: vp.y + dy, zoom: vp.zoom }, { duration: 0 })
|
||||
}
|
||||
|
||||
// The flow store and the node list become available at different times: the
|
||||
// store at component init, the nodes only once the async canvas fetch resolves
|
||||
// (plus a further tick for Vue Flow to measure them). Framing on any single
|
||||
// signal would fit an EMPTY canvas, so every path calls reframe() and it no-ops
|
||||
// until there is a store and something to frame.
|
||||
//
|
||||
// `nodes-initialized` also fires whenever a node is added later. Auto-framing on
|
||||
// that would yank the viewport out from under whoever is mid-pan, so the initial
|
||||
// frame is claimed exactly once; rearranging is the only thing that re-frames on
|
||||
// demand (see autoLayout).
|
||||
let framedOnce = false
|
||||
const reframe = (force = false) => {
|
||||
if (!flowStore || !nodes.value.length) return
|
||||
if (framedOnce && !force) return
|
||||
framedOnce = true
|
||||
nextTick(() => { void frameCanvas() })
|
||||
}
|
||||
|
||||
const onFlowInit = (store: FlowStore) => {
|
||||
flowStore = store
|
||||
reframe()
|
||||
}
|
||||
|
||||
// Background dot-pattern color, resolved from the live theme var so the grid
|
||||
// recolors on theme switch (vue-flow Background renders pattern-color as an
|
||||
// SVG fill attribute, which can't resolve var() directly — read the computed
|
||||
@ -667,7 +765,12 @@ const remoteNodes = computed(() =>
|
||||
nodes.value.filter((n) => n.type === 'remote'),
|
||||
)
|
||||
|
||||
const autoLayout = () => layoutColumns(true)
|
||||
// Re-frame after rearranging, bounded by the same zoom floor so "自动排列" never
|
||||
// shrinks the cards back into unreadable territory on a narrow screen.
|
||||
const autoLayout = () => {
|
||||
layoutColumns(true)
|
||||
reframe(true)
|
||||
}
|
||||
|
||||
const load = async () => {
|
||||
loading.value = true
|
||||
@ -718,6 +821,9 @@ const load = async () => {
|
||||
}
|
||||
assignLayers()
|
||||
dirty.value = false
|
||||
// The store exists by now but the node list has only just been populated, so
|
||||
// this is the first moment a fit actually has something to frame.
|
||||
reframe(true)
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
|
||||
@ -121,6 +121,131 @@
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- Load balancer (read-only). Shown only when the endpoint responds: a
|
||||
404 means this node has no balancer wired up, and an absent section is
|
||||
the honest signal for "this build doesn't do balancing". -->
|
||||
<section class="section w4f-card bal-section" v-if="bal">
|
||||
<h3 class="section-title">
|
||||
<span class="k-ic">⚖</span>负载均衡
|
||||
<!-- The switch edits the mode ON THIS NODE. Every member decides about
|
||||
its own forwards, so a cluster-wide toggle would be a lie: there is
|
||||
no single action that flips them together. -->
|
||||
<el-select
|
||||
v-if="bal.canControl"
|
||||
v-model="balModeChoice"
|
||||
size="small"
|
||||
class="bal-mode"
|
||||
:disabled="modeBusy"
|
||||
@change="onModeChange"
|
||||
>
|
||||
<el-option label="关闭" value="off" />
|
||||
<el-option label="观察模式(不释放)" value="dry-run" />
|
||||
<el-option label="启用(会释放)" value="on" />
|
||||
</el-select>
|
||||
<span class="w4f-tag" v-else v-text="'只读'" ></span>
|
||||
<span class="bal-src" v-if="bal.modeSource === 'stored'">已保存</span>
|
||||
<span class="bal-src" v-else>来自启动参数</span>
|
||||
<span class="w4f-tag" v-if="!bal.isMember" v-text="'非集群成员'"></span>
|
||||
<span class="w4f-tag" v-else-if="bal.releasing" v-text="'释放进行中'"></span>
|
||||
<span class="w4f-tag w4f-tag-warn" v-else-if="bal.selfOverShare" v-text="'本机超出份额'"></span>
|
||||
<span class="w4f-tag w4f-tag-ok" v-else v-text="'本机在份额内'"></span>
|
||||
<span class="title-actions">
|
||||
<button class="w4f-btn ghost small" :disabled="busy" @click="load">⟳ 刷新</button>
|
||||
</span>
|
||||
</h3>
|
||||
|
||||
<div class="bal-thresholds">
|
||||
<span>
|
||||
规则:本机转发 <b>{{ bal.selfForwards }}</b> 条
|
||||
> 平摊份额 <b>{{ bal.share }}</b>
|
||||
且至少 {{ 2 }} 条时,本机自行释放一条
|
||||
</span>
|
||||
<span>集群共 {{ bal.totalForwards }} 条 / {{ bal.aliveNodes }} 台存活</span>
|
||||
<span>两次释放间隔 ≥ {{ Math.round(bal.cooldownSec / 60) }} 分钟</span>
|
||||
<span v-if="bal.dryRun" class="bal-hint">
|
||||
设 <code>W4F_BALANCE_MODE=on</code> 才会真正释放
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<p class="bal-note">
|
||||
每个节点各自判断自己的转发,超了就自己撤销一条并丢回环上,
|
||||
由环节点按「最闲者优先」重新认领。没有中心调度。
|
||||
</p>
|
||||
|
||||
<!-- Per-node forward counts. Bars are scaled to the busiest node so the
|
||||
shape reads whether the cluster holds 3 forwards or 300; the share
|
||||
marker is what actually decides, not the bar. -->
|
||||
<div class="bal-nodes">
|
||||
<div
|
||||
v-for="n in balNodes"
|
||||
:key="n.id"
|
||||
class="bal-node"
|
||||
:class="{ self: n.id === bal.self, over: n.forwards > bal.share }"
|
||||
>
|
||||
<div class="bn-top">
|
||||
<span class="bn-id">{{ shortId(n.id) }}</span>
|
||||
<span v-if="n.id === bal.self" class="w4f-tag w4f-tag-primary">本机</span>
|
||||
<span v-if="n.isLeader" class="w4f-tag w4f-tag-info">LEADER</span>
|
||||
<span v-if="!n.alive" class="w4f-tag w4f-tag-danger">离线</span>
|
||||
<span v-if="n.alive && n.forwards > bal.share" class="w4f-tag w4f-tag-warn">超出份额</span>
|
||||
</div>
|
||||
<div class="w4f-bar" :class="{ err: n.alive && n.forwards > bal.share }">
|
||||
<i :style="{ width: barWidth(n.forwards) + '%' }" />
|
||||
</div>
|
||||
<div class="bn-stats">
|
||||
<span>转发 <b>{{ n.forwards }}</b></span>
|
||||
<span>内存 <b>{{ n.memPct.toFixed(1) }}%</b></span>
|
||||
<span>网络 <b>{{ n.netPct.toFixed(1) }}%</b></span>
|
||||
<span class="bn-score">份额 <b>{{ bal.share }}</b></span>
|
||||
</div>
|
||||
</div>
|
||||
<div v-if="!balNodes.length" class="empty">暂无节点负载数据</div>
|
||||
</div>
|
||||
|
||||
<!-- What THIS node would do. The answer is per-node by design: every
|
||||
member runs the same rule on its own forwards, so there is no
|
||||
cluster-wide plan and no "the leader will move X to Y". -->
|
||||
<div class="bal-plan">
|
||||
<div v-if="bal.releasing" class="bp-inflight">
|
||||
<span class="w4f-tag w4f-tag-warn">释放进行中</span>
|
||||
<span class="bp-text">
|
||||
本机已释放 {{ bal.releasing.local }}:{{ bal.releasing.remotePort }},
|
||||
等待环把它认领到别的节点
|
||||
</span>
|
||||
</div>
|
||||
<div v-else-if="bal.wouldRelease && !bal.blockedBy" class="bp-would">
|
||||
<span class="w4f-tag w4f-tag-warn">将释放</span>
|
||||
<span class="bp-text">
|
||||
本机将撤销 {{ bal.wouldRelease.local }}:{{ bal.wouldRelease.remotePort }}
|
||||
并丢回环上,由最闲节点接管
|
||||
</span>
|
||||
</div>
|
||||
<div v-else-if="bal.wouldRelease && bal.blockedBy === 'cooldown'" class="bp-waiting">
|
||||
<span class="w4f-tag w4f-tag-info">冷却中</span>
|
||||
<span class="bp-text">
|
||||
本机超出份额,但距上次释放不足
|
||||
{{ Math.round(bal.cooldownSec / 60) }} 分钟
|
||||
<template v-if="bal.lastReleaseAt">
|
||||
(已过 {{ cooldownAgo }})
|
||||
</template>
|
||||
</span>
|
||||
</div>
|
||||
<div v-else-if="bal.wouldRelease" class="bp-waiting">
|
||||
<span class="w4f-tag w4f-tag-info">暂不释放</span>
|
||||
<span class="bp-text">本机超出份额,原因:{{ blockedLabel }}</span>
|
||||
</div>
|
||||
<div v-else-if="!bal.isMember" class="bp-ok">
|
||||
<span class="w4f-tag w4f-tag-ok">单节点</span>
|
||||
<span class="bp-text">本机未加入集群,无需均衡</span>
|
||||
</div>
|
||||
<div v-else class="bp-ok">
|
||||
<span class="w4f-tag w4f-tag-ok">已均衡</span>
|
||||
<span class="bp-text">
|
||||
本机持有 {{ bal.selfForwards }} 条,未超平摊份额 {{ bal.share }},无需释放
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
<!-- Active forward topology -->
|
||||
<section class="section w4f-card">
|
||||
<h3 class="section-title"><span class="k-ic">→</span>活跃转发拓扑</h3>
|
||||
@ -199,14 +324,18 @@
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, reactive, computed, onMounted, onBeforeUnmount } from 'vue'
|
||||
import { ref, reactive, computed, onMounted, onBeforeUnmount, watch } from 'vue'
|
||||
import { ElMessage } from 'element-plus'
|
||||
import { api, downloadAuditCsv } from '../api'
|
||||
import { canWrite } from '../auth'
|
||||
import type { RingSnapshot, RingTaskInfo, RingLogEntry } from '../types'
|
||||
import type { BalanceReport, RingSnapshot, RingTaskInfo, RingLogEntry } from '../types'
|
||||
|
||||
const ring = ref<RingSnapshot | null>(null)
|
||||
const busy = ref(false)
|
||||
// bal is null when the node has no balancer wired up (endpoint 404s). The
|
||||
// section is simply not rendered then, which is the honest signal for "this
|
||||
// build does not do balancing" — better than an empty panel implying it does.
|
||||
const bal = ref<BalanceReport | null>(null)
|
||||
let timer: number | null = null
|
||||
let visHandler: (() => void) | null = null
|
||||
|
||||
@ -214,8 +343,76 @@ const joinDialog = reactive({ visible: false, addr: '', key: '' })
|
||||
|
||||
const load = async () => {
|
||||
try { ring.value = await api.ring() } catch { /* keep last */ }
|
||||
// Loaded alongside the ring, but unlike it a failure clears the view: the
|
||||
// balancer report is a decision surface, and showing a stale "已均衡" after the
|
||||
// endpoint starts failing would be worse than showing nothing.
|
||||
try { bal.value = await api.balance() } catch { bal.value = null }
|
||||
}
|
||||
|
||||
// ---- load balancer mode switch ---- //
|
||||
// The select is bound to a local mirror so the UI can show the pending choice
|
||||
// while the request is in flight, and snap back if the server rejects it.
|
||||
const balModeChoice = ref<string>('dry-run')
|
||||
const modeBusy = ref(false)
|
||||
watch(
|
||||
() => bal.value?.mode,
|
||||
(m) => { if (m) balModeChoice.value = m },
|
||||
{ immediate: true },
|
||||
)
|
||||
const onModeChange = async (m: string) => {
|
||||
if (!bal.value || modeBusy.value) return
|
||||
modeBusy.value = true
|
||||
try {
|
||||
bal.value = await api.setBalanceMode(m)
|
||||
ElMessage.success(
|
||||
m === 'on' ? '本机已启用负载释放' : m === 'off' ? '本机已关闭负载均衡' : '本机已切到观察模式',
|
||||
)
|
||||
} catch (e) {
|
||||
// Snap back: leaving the select showing a mode the server did not accept
|
||||
// would misrepresent what the node is doing.
|
||||
balModeChoice.value = bal.value.mode
|
||||
ElMessage.error('切换失败:' + (e instanceof Error ? e.message : String(e)))
|
||||
} finally {
|
||||
modeBusy.value = false
|
||||
}
|
||||
}
|
||||
|
||||
// ---- load balancer view ---- //
|
||||
// Each node decides about its OWN forwards, so this page has no cluster-wide
|
||||
// plan to show — only what THIS node holds and what it is about to do.
|
||||
// Ordered by descending count: the busiest node first is what an operator
|
||||
// scanning for a hotspot wants.
|
||||
const balNodes = computed(() => [...(bal.value?.nodes ?? [])].sort((a, b) => b.forwards - a.forwards))
|
||||
// barWidth scales against the busiest node rather than an absolute, so the shape
|
||||
// is readable whether the cluster holds 3 forwards or 300. The bar is context;
|
||||
// `share` is what actually decides whether a node is over.
|
||||
const barWidth = (forwards: number) => {
|
||||
const max = Math.max(...(bal.value?.nodes ?? []).map((n) => n.forwards), 1)
|
||||
return Math.min(100, Math.round((forwards / max) * 100))
|
||||
}
|
||||
// blockedLabel renders the server's machine reason as something readable. The
|
||||
// reason matters: "over its share but not acting" is otherwise indistinguishable
|
||||
// from "broken".
|
||||
const blockedLabel = computed(() => {
|
||||
switch (bal.value?.blockedBy) {
|
||||
case 'cooldown': return '冷却期未过'
|
||||
case 'in-flight': return '上一次释放尚未确认'
|
||||
case 'nothing-active': return '本机可释放的转发都已停用'
|
||||
case 'below-min': return '持有量低于下限'
|
||||
case 'single-node': return '单节点环'
|
||||
default: return bal.value?.blockedBy || '未说明'
|
||||
}
|
||||
})
|
||||
// cooldownAgo says how long ago the last release was, so an operator can see
|
||||
// whether the cooldown is about to expire.
|
||||
const cooldownAgo = computed(() => {
|
||||
const t = bal.value?.lastReleaseAt ?? 0
|
||||
if (!t) return ''
|
||||
const s = Math.max(0, Math.floor(nowMs.value / 1000) - t)
|
||||
if (s < 60) return s + ' 秒前'
|
||||
return Math.floor(s / 60) + ' 分钟前'
|
||||
})
|
||||
|
||||
const shortId = (id: string) => (id.length > 22 ? '…' + id.slice(-20) : id)
|
||||
|
||||
// copyNodeKey copies this node's admission key to the clipboard so the operator
|
||||
@ -260,6 +457,8 @@ const stateDotClass = computed(() => {
|
||||
|
||||
// ---- last-sync freshness (ops cares "环还活着吗", not the cycle counter) ----
|
||||
const now = ref(Date.now())
|
||||
// nowMs mirrors `now` as a plain number for the balance section's minute math.
|
||||
const nowMs = computed(() => now.value)
|
||||
let tick: number | null = null
|
||||
const lastSyncAgo = computed(() => {
|
||||
const ls = ring.value?.lastSync ?? 0
|
||||
@ -481,7 +680,11 @@ onBeforeUnmount(() => {
|
||||
</script>
|
||||
|
||||
<style scoped lang="scss">
|
||||
.cluster-page { height: 100%; display: flex; flex-direction: column; overflow-y: auto; padding: 4px 0 40px; gap: 14px; }
|
||||
/* Let .content (App.vue) remain the single scroll container — see the note in
|
||||
* StatusView: a page-level scroller sized by leftover flex space is what froze
|
||||
* the status page. `.log-list` keeps its own bounded scroll (that one is
|
||||
* deliberate — a log pane should not push the whole page). */
|
||||
.cluster-page { display: flex; flex-direction: column; padding: 4px 0 40px; gap: 14px; }
|
||||
|
||||
/* hero */
|
||||
.hero { display: flex; align-items: center; justify-content: space-between; gap: 16px; flex-wrap: wrap; padding: 16px 20px; }
|
||||
@ -634,4 +837,66 @@ onBeforeUnmount(() => {
|
||||
.kpis { grid-template-columns: minmax(0, 1fr); }
|
||||
.page-title { font-size: 15.5px; }
|
||||
}
|
||||
|
||||
/* ---- Load balancer section ---- */
|
||||
.bal-section .section-title { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; }
|
||||
.bal-section .section-title .k-ic { margin-right: 0; }
|
||||
|
||||
.bal-thresholds {
|
||||
display: flex; flex-wrap: wrap; gap: 6px 14px;
|
||||
font-size: 11.5px; color: var(--w4f-muted);
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
.bal-hint { color: var(--w4f-warning); }
|
||||
.bal-hint code {
|
||||
font-family: var(--w4f-mono, ui-monospace, monospace);
|
||||
background: var(--w4f-card-2); padding: 1px 4px; border-radius: 3px;
|
||||
}
|
||||
|
||||
.bal-nodes {
|
||||
display: grid; gap: 8px;
|
||||
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
|
||||
}
|
||||
.bal-node {
|
||||
border: 1px solid var(--w4f-line);
|
||||
border-radius: 8px;
|
||||
padding: 9px 11px;
|
||||
background: var(--w4f-card-2);
|
||||
min-width: 0;
|
||||
}
|
||||
.bal-node.self { border-color: var(--w4f-primary); }
|
||||
.bal-node.over { border-color: var(--w4f-warning); }
|
||||
.bal-mode { width: 168px; }
|
||||
.bal-src { font-size: 11px; color: var(--w4f-muted); }
|
||||
.bal-note {
|
||||
margin: 0 0 10px; font-size: 11.5px; line-height: 1.55;
|
||||
color: var(--w4f-muted);
|
||||
}
|
||||
.bn-top { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; margin-bottom: 7px; }
|
||||
.bn-id {
|
||||
font-family: var(--w4f-mono, ui-monospace, monospace);
|
||||
font-size: 12px; font-weight: 700;
|
||||
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||
}
|
||||
.bn-stats {
|
||||
display: flex; flex-wrap: wrap; gap: 3px 12px;
|
||||
font-size: 11.5px; color: var(--w4f-muted); margin-top: 7px;
|
||||
}
|
||||
.bn-stats b { color: var(--w4f-text); font-variant-numeric: tabular-nums; }
|
||||
.bn-score { margin-left: auto; }
|
||||
|
||||
.bal-plan { margin-top: 11px; }
|
||||
.bp-inflight, .bp-would, .bp-waiting, .bp-ok {
|
||||
display: flex; align-items: center; gap: 9px; flex-wrap: wrap;
|
||||
padding: 8px 11px; border-radius: 8px;
|
||||
font-size: 12.5px;
|
||||
background: var(--w4f-card-2);
|
||||
border: 1px solid var(--w4f-line);
|
||||
}
|
||||
.bp-inflight { border-color: var(--w4f-warning); }
|
||||
.bp-text { min-width: 0; overflow-wrap: anywhere; }
|
||||
|
||||
@media (max-width: 380px) {
|
||||
.bn-score { margin-left: 0; }
|
||||
}
|
||||
</style>
|
||||
|
||||
@ -167,11 +167,12 @@ onMounted(load)
|
||||
</script>
|
||||
|
||||
<style scoped lang="scss">
|
||||
/* Let .content (App.vue) remain the single scroll container — see the note in
|
||||
* StatusView: a page-level scroller sized by leftover flex space can collapse
|
||||
* to zero height on narrow screens. */
|
||||
.settings-page {
|
||||
height: 100%;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
.page-top {
|
||||
|
||||
@ -496,11 +496,22 @@ onBeforeUnmount(() => {
|
||||
</script>
|
||||
|
||||
<style scoped lang="scss">
|
||||
/* The pages inside .content must NOT be their own scrollers.
|
||||
*
|
||||
* App.vue already designates `.content` as the app's single scroll container
|
||||
* (`flex:1; overflow-y:auto`). These pages were also `height:100%` +
|
||||
* `overflow-y:auto`, producing a nested scroller whose height is the leftover
|
||||
* space of a flex column. On a 375px phone the status page's fixed-height
|
||||
* header (122px) plus the single-column KPI stack (565px) exceed the available
|
||||
* 651px, so the inner `.status-body` was handed 0px while holding 1682px of
|
||||
* content — the page looked completely frozen, since neither `.content` (which
|
||||
* had nothing to scroll) nor the 0-height inner box could be swiped.
|
||||
*
|
||||
* Letting the page grow naturally and scroll in `.content` removes the whole
|
||||
* class of bug: there is no leftover-space arithmetic left to get wrong. */
|
||||
.status-page {
|
||||
height: 100%;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
gap: 14px;
|
||||
}
|
||||
|
||||
@ -527,12 +538,9 @@ onBeforeUnmount(() => {
|
||||
}
|
||||
|
||||
.status-body {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 18px;
|
||||
overflow-y: auto;
|
||||
padding-right: 4px;
|
||||
}
|
||||
|
||||
.section { display: flex; flex-direction: column; }
|
||||
@ -720,8 +728,13 @@ onBeforeUnmount(() => {
|
||||
}
|
||||
|
||||
@media (max-width: 380px) {
|
||||
/* 极窄:KPI 也降为单列,否则数字会被压断 */
|
||||
.kpis { grid-template-columns: minmax(0, 1fr); }
|
||||
/* Keep the KPI grid at TWO columns even on the narrowest phones.
|
||||
* Dropping to one column made four full-width cards ~565px tall, which on its
|
||||
* own exceeded the whole viewport and (with the page-level scroller of the
|
||||
* day) collapsed the content area to zero height. Two columns keeps each card
|
||||
* ~160px wide — enough for the 27px number + 11.5px caption, which is all a
|
||||
* KPI card holds. */
|
||||
.kpis { gap: 8px; }
|
||||
.fwd-route { font-size: 12.5px; }
|
||||
.page-title { font-size: 16px; }
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user