mirror of
https://gitcode.com/JianFeeeee/webui4frpc.git
synced 2026-10-15 12:14:37 +00:00
Compare commits
25 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 21eb52df24 | |||
| 1a850b4846 | |||
| c90d4f3791 | |||
| 790eaa5ed1 | |||
| c69fb2caba | |||
| b40b53b40b | |||
| d347d2a648 | |||
| 8176478269 | |||
| 45a6a76621 | |||
| 7d5bca5b72 | |||
| 875e5805eb | |||
| 084ef62b32 | |||
| 57d4d14d28 | |||
| fc4cbaa726 | |||
| bfd7265405 | |||
| 2dd32648e6 | |||
| 50c7729779 | |||
| 918ca5d5ba | |||
| c8ab2584a8 | |||
| 8ab52b238a | |||
| 5cdc052d01 | |||
| 1c835425de | |||
| 041cc04dd6 | |||
| 46e8bc3703 | |||
| f4fb964734 |
83
README.md
83
README.md
@ -32,6 +32,14 @@
|
|||||||
- 主负载信号:当前节点拥有的转发数(权重 ×100)
|
- 主负载信号:当前节点拥有的转发数(权重 ×100)
|
||||||
- 次级信号:**NIC 网络接口利用率**(从 `/proc/net/dev` 增量 + `/sys/class/net/*/speed` 计算,排除了 lo/docker/veth 等虚拟接口)
|
- 次级信号:**NIC 网络接口利用率**(从 `/proc/net/dev` 增量 + `/sys/class/net/*/speed` 计算,排除了 lo/docker/veth 等虚拟接口)
|
||||||
- 同转发数节点间由内存占用 + 网络饱和度打破平局
|
- 同转发数节点间由内存占用 + 网络饱和度打破平局
|
||||||
|
- **负载均衡(自导式)**:每个节点对**自己的**转发跑同一条规则 —— 持有数超过
|
||||||
|
`ceil(总转发数 / 存活节点数)` 且至少 2 条时,自行撤销一条、把任务丢回环上,
|
||||||
|
再由环按「最闲者优先」重新认领
|
||||||
|
- **没有中心调度**:一节点的负载只有它自己的拓扑条目消失时才会下降,而条目是
|
||||||
|
本地簿记,没有对端能代它写,所以只能由持有者自己动手
|
||||||
|
- 规则自限:撤到平摊值即停(否则会把转发卸光),且低于下限永不撤、不会卸空节点
|
||||||
|
- 每节点默认 10 分钟冷却,限制抖动频率;默认 **dry-run**(只报告不释放)
|
||||||
|
- 集群页每节点一个开关:关闭 / 观察模式 / 启用,可持久化
|
||||||
- **per-forward 独立进程**:每条转发拥有独立的 frpc 配置文件和 frpc 进程(worker 键 = `local~remote~port`),单条转发故障不影响兄弟,消除旧模型的多节点 `proxy already exists` 冲突
|
- **per-forward 独立进程**:每条转发拥有独立的 frpc 配置文件和 frpc 进程(worker 键 = `local~remote~port`),单条转发故障不影响兄弟,消除旧模型的多节点 `proxy already exists` 冲突
|
||||||
- **容错自愈**:leader / 非 leader 节点宕机 → 自动检测 → 环跳过死亡节点、任务重挂;leader 宕机 → 上邻居晋升为新 leader
|
- **容错自愈**:leader / 非 leader 节点宕机 → 自动检测 → 环跳过死亡节点、任务重挂;leader 宕机 → 上邻居晋升为新 leader
|
||||||
- **宕机自动重联**:节点宕机重启后通过缓存的集群拓扑(对端地址 + 准入密钥)自动重联,显式脱离集群的节点不自动重联
|
- **宕机自动重联**:节点宕机重启后通过缓存的集群拓扑(对端地址 + 准入密钥)自动重联,显式脱离集群的节点不自动重联
|
||||||
@ -79,7 +87,7 @@ cd web && npm run build && rm -rf internal/httpapi/dist && cp -r web/dist intern
|
|||||||
### Linux (Debian / Ubuntu)
|
### Linux (Debian / Ubuntu)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo dpkg -i webui4frpc-0.1.2-linux-amd64.deb # 或 linux-arm64.deb
|
sudo dpkg -i webui4frpc-0.1.4-linux-amd64.deb # 或 linux-arm64.deb
|
||||||
sudo nano /etc/webui4frpc/webui4frpc.env # 改掉默认密码
|
sudo nano /etc/webui4frpc/webui4frpc.env # 改掉默认密码
|
||||||
sudo systemctl restart webui4frpc
|
sudo systemctl restart webui4frpc
|
||||||
```
|
```
|
||||||
@ -87,7 +95,7 @@ sudo systemctl restart webui4frpc
|
|||||||
### Linux (RHEL / Fedora / openEuler / 默认 / 龙蜥)
|
### Linux (RHEL / Fedora / openEuler / 默认 / 龙蜥)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo rpm -i webui4frpc-0.1.2-linux-x86_64.rpm
|
sudo rpm -i webui4frpc-0.1.4-linux-x86_64.rpm
|
||||||
sudo vi /etc/webui4frpc/webui4frpc.env
|
sudo vi /etc/webui4frpc/webui4frpc.env
|
||||||
sudo systemctl enable --now webui4frpc
|
sudo systemctl enable --now webui4frpc
|
||||||
```
|
```
|
||||||
@ -97,7 +105,7 @@ sudo systemctl enable --now webui4frpc
|
|||||||
|
|
||||||
### Windows
|
### Windows
|
||||||
|
|
||||||
双击 `webui4frpc-0.1.2-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
双击 `webui4frpc-0.1.4-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
||||||
安装向导会询问监听地址与管理员凭据,完成后:
|
安装向导会询问监听地址与管理员凭据,完成后:
|
||||||
|
|
||||||
- 程序装到 `C:\Program Files\webui4frpc`
|
- 程序装到 `C:\Program Files\webui4frpc`
|
||||||
@ -107,8 +115,8 @@ sudo systemctl enable --now webui4frpc
|
|||||||
### macOS
|
### macOS
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo installer -pkg webui4frpc-0.1.2-darwin-arm64.pkg -target / # Apple Silicon
|
sudo installer -pkg webui4frpc-0.1.4-darwin-arm64.pkg -target / # Apple Silicon
|
||||||
sudo installer -pkg webui4frpc-0.1.2-darwin-amd64.pkg -target / # Intel
|
sudo installer -pkg webui4frpc-0.1.4-darwin-amd64.pkg -target / # Intel
|
||||||
```
|
```
|
||||||
|
|
||||||
或直接双击 `.pkg` 走图形安装器。安装后 launchd 开机自启,配置在
|
或直接双击 `.pkg` 走图形安装器。安装后 launchd 开机自启,配置在
|
||||||
@ -126,7 +134,7 @@ sudo launchctl kickstart -k system/com.jianfeeeee.webui4frpc
|
|||||||
不想装服务只想跑一下,用对应的 `.tar.gz` / `.zip`,解压即用:
|
不想装服务只想跑一下,用对应的 `.tar.gz` / `.zip`,解压即用:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
tar xzf webui4frpc-0.1.2-linux-amd64.tar.gz
|
tar xzf webui4frpc-0.1.4-linux-amd64.tar.gz
|
||||||
./webui4frpc -addr 127.0.0.1:7500 -user admin -password 你的密码 -workdir ./data
|
./webui4frpc -addr 127.0.0.1:7500 -user admin -password 你的密码 -workdir ./data
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -160,6 +168,19 @@ sha256sum -c SHA256SUMS --ignore-missing
|
|||||||
|
|
||||||
环境变量 `W4F_HOST` 可覆盖节点对外可达的主机名(Docker DNS / 通配监听场景用)。
|
环境变量 `W4F_HOST` 可覆盖节点对外可达的主机名(Docker DNS / 通配监听场景用)。
|
||||||
|
|
||||||
|
### 负载均衡参数
|
||||||
|
|
||||||
|
每节点独立生效;集群页的开关**优先于**这里的默认值(在页面上选过就以页面为准,重启不丢)。
|
||||||
|
|
||||||
|
| 环境变量 | 默认 | 说明 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `W4F_BALANCE_MODE` | `dry-run` | `off` 不参与 / `dry-run` 只报告 / `on` 真正释放 |
|
||||||
|
| `W4F_BALANCE_COOLDOWN` | `600` | 同一节点两次释放之间的最小秒数 |
|
||||||
|
| `W4F_BALANCE_INTERVAL` | `10` | 检查间隔(秒) |
|
||||||
|
|
||||||
|
**释放会停掉该转发的 frpc worker,其上的在途连接会断一次**,由新节点认领后恢复;
|
||||||
|
因此默认是 `dry-run`。集群页的开关走 write 级权限。
|
||||||
|
|
||||||
### 启动行为
|
### 启动行为
|
||||||
|
|
||||||
节点启动时的集群引导优先级:
|
节点启动时的集群引导优先级:
|
||||||
@ -191,10 +212,10 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
|||||||
│ ├── render 渲染 frpc JSON 配置(per-forward 单 proxy + 高级参数 + LB/健康检查)
|
│ ├── render 渲染 frpc JSON 配置(per-forward 单 proxy + 高级参数 + LB/健康检查)
|
||||||
│ ├── process worker 进程管理(spawn/stop/restart/自愈/日志轮转,worker 键 = local~remote~port)
|
│ ├── process worker 进程管理(spawn/stop/restart/自愈/日志轮转,worker 键 = local~remote~port)
|
||||||
│ ├── install 一键下载官方 frpc(GitHub Releases,含解压安全防护)
|
│ ├── install 一键下载官方 frpc(GitHub Releases,含解压安全防护)
|
||||||
│ ├── cluster 令牌环协议(ring engine / leader 选举 / 容错 / 宕机重联 / 二进制分发 / NIC 负载采样)
|
│ ├── cluster 令牌环协议(ring engine / leader 选举 / 容错 / 宕机重联 / 二进制分发 / NIC 负载采样 / 自导式负载均衡)
|
||||||
│ └── httpapi REST API + 静态资源(auth/session / canvas / forwards / cluster / users / audit 导出)
|
│ └── httpapi REST API + 静态资源(auth/session / canvas / forwards / cluster / users / audit 导出)
|
||||||
└── web/ Vue 3 SPA(Element Plus + VueFlow 画布 + Pinia)
|
└── web/ Vue 3 SPA(Element Plus + VueFlow 画布 + Pinia)
|
||||||
└── src/views/ 状态总览 / 连接配置(画布)/ 集群 / 设置 / 账号与密钥
|
└── src/views/ 状态总览 / 连接配置(画布)/ 集群(含负载均衡开关)/ 设置 / 账号与密钥
|
||||||
```
|
```
|
||||||
|
|
||||||
## API
|
## API
|
||||||
@ -220,6 +241,8 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
|||||||
| | GET | `/binary/status` | read | frpc 路径 |
|
| | GET | `/binary/status` | read | frpc 路径 |
|
||||||
| | POST | `/binary/install` | write | 安装 frpc |
|
| | POST | `/binary/install` | write | 安装 frpc |
|
||||||
| **集群** | GET | `/cluster/ring` | read | 令牌环快照 |
|
| **集群** | GET | `/cluster/ring` | read | 令牌环快照 |
|
||||||
|
| | GET | `/cluster/balance` | read | 本节点负载均衡视图(只读,不变更状态) |
|
||||||
|
| | POST | `/cluster/balance/mode` | write | 切换本节点负载均衡模式(持久化) |
|
||||||
| | POST | `/cluster/create` | write | 创建独立集群 |
|
| | POST | `/cluster/create` | write | 创建独立集群 |
|
||||||
| | POST | `/cluster/join-ring` | write | 加入集群(本节点发起) |
|
| | POST | `/cluster/join-ring` | write | 加入集群(本节点发起) |
|
||||||
| | POST | `/cluster/join` | write | 接收新节点加入 |
|
| | POST | `/cluster/join` | write | 接收新节点加入 |
|
||||||
@ -254,8 +277,45 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
|||||||
- **任务分发**:在任意节点画布上配置 `localOnly=false` 的转发并保存 → 画布差异判断生成命令 → 令牌携带 → 负载最低节点摘取 → 拉起 frpc worker
|
- **任务分发**:在任意节点画布上配置 `localOnly=false` 的转发并保存 → 画布差异判断生成命令 → 令牌携带 → 负载最低节点摘取 → 拉起 frpc worker
|
||||||
- **容错**:节点宕机 → 自动检测 → 环自愈(跳过死亡节点 / leader failover);重启 → 自动重联
|
- **容错**:节点宕机 → 自动检测 → 环自愈(跳过死亡节点 / leader failover);重启 → 自动重联
|
||||||
- **退出集群**:在集群页「退出集群」→ 本节点脱离为 standalone,不自动重联
|
- **退出集群**:在集群页「退出集群」→ 本节点脱离为 standalone,不自动重联
|
||||||
|
- **负载均衡**:见下节
|
||||||
|
|
||||||
集群页展示:环拓扑(self/leader 标记)、令牌轮次、节点负载、待办任务、活跃转发拓扑(含 owner 归属)、增量事件日志。
|
集群页展示:环拓扑(self/leader 标记)、令牌轮次、节点负载、待办任务、活跃转发拓扑(含 owner 归属)、负载均衡(本机份额与释放计划)、增量事件日志。
|
||||||
|
|
||||||
|
### 负载均衡(自导式释放)
|
||||||
|
|
||||||
|
**为什么不是 leader 统一调度**:一节点的负载信号是它持有的转发数,而这个数只在
|
||||||
|
**它自己的拓扑条目消失**时才下降 —— 条目是本地簿记。所以 leader 从外面只能*看见*
|
||||||
|
失衡,无法*消除*它;要消除就得伸手改另一个节点的簿记,而这正是这个环到处都在避免
|
||||||
|
的跨节点写(撤销、重启同样都是定向往 owner)。
|
||||||
|
|
||||||
|
于是规则由每个节点对自己的转发本地执行:
|
||||||
|
|
||||||
|
```
|
||||||
|
当 持有转发数 >= 2 且 持有转发数 > ceil(总转发数 / 存活节点数)
|
||||||
|
则 撤销其中一条 → 删除条目 → 以启用状态重新发布到环 → 由最闲节点认领
|
||||||
|
```
|
||||||
|
|
||||||
|
- **自限**:撤到平摊值即停。3 条分 3 台时,持有 2 条的那台卸一条后变成 1 条,
|
||||||
|
`1 > 1` 不成立 → 停手。没有这个上界,owner 会把转发全部卸光。
|
||||||
|
- **不会卸空**:低于下限 2 条永不撤销。
|
||||||
|
- **不碰已停用的转发**:停用条目的 entry 是用户「停用」决定的载体,释放它等于把
|
||||||
|
一条**已停**的转发当活干交给别人。
|
||||||
|
- **释放 ≠ 停止**:用户停止保留条目并标记链路为停用;均衡释放**删除条目**且链路保持
|
||||||
|
启用。两者不能混用 —— 复用停止的话,每个节点的重复认领守卫都会拒绝这条已是「有主」
|
||||||
|
的转发,它会被永久搁死在刚放弃它的那台机器上。
|
||||||
|
|
||||||
|
三种模式(集群页每节点一个开关,也可用 `W4F_BALANCE_MODE` 设默认值):
|
||||||
|
|
||||||
|
| 模式 | 行为 |
|
||||||
|
| --- | --- |
|
||||||
|
| `off` | 不参与均衡 |
|
||||||
|
| `dry-run` | 只报告「将释放哪条」,不动任何东西(默认) |
|
||||||
|
| `on` | 真正释放 |
|
||||||
|
|
||||||
|
只读接口 `GET /cluster/balance` 返回本节点视角:`selfForwards` / `share` /
|
||||||
|
`totalForwards` / `aliveNodes` / `selfOverShare`、`wouldRelease`、`releasing`、
|
||||||
|
`blockedBy`(为何暂不动的机器原因)、`mode` / `modeSource` / `canControl`。
|
||||||
|
该接口**不改变任何状态** —— 反复轮询不会启动、延迟或取消一次释放。
|
||||||
|
|
||||||
## 测试
|
## 测试
|
||||||
|
|
||||||
@ -263,6 +323,9 @@ webui4frpc(单二进制,无 frp 依赖,前端 go:embed)
|
|||||||
# 后端
|
# 后端
|
||||||
go test ./...
|
go test ./...
|
||||||
|
|
||||||
|
# 单包(负载均衡规则与状态机)
|
||||||
|
go test ./internal/cluster/ -run TestShare -v
|
||||||
|
|
||||||
# 前端类型检查
|
# 前端类型检查
|
||||||
cd web && npm run type-check
|
cd web && npm run build
|
||||||
```
|
```
|
||||||
|
|||||||
@ -15,6 +15,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
@ -175,13 +176,41 @@ func main() {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
// A re-claim after a forward-centric stop leaves the link flagged
|
// A re-claim after a forward-centric stop leaves the link flagged
|
||||||
// disabled (by RevokeFn); clear it so renderRemote renders the
|
// disabled (by RevokeFn). Do NOT clear that flag here: the ring
|
||||||
// proxy back in. No-op for a fresh claim.
|
// re-claims automatically (startup reconcile re-spawns any owned
|
||||||
_ = st.SetLinkDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, false)
|
// forward whose worker is missing), so clearing it on claim made
|
||||||
|
// "stopped" un-durable — every restart resurrected a forward the user
|
||||||
|
// had explicitly stopped, and it then failed forever against a local
|
||||||
|
// service that was intentionally not running.
|
||||||
|
//
|
||||||
|
// The flag is now owned by the two user-facing actions:
|
||||||
|
// startForward → SetLinkDisabled(false) before submitting the task
|
||||||
|
// stopForward → SetLinkDisabled(true) before revoking it
|
||||||
|
// Claim only READS it to decide whether to bring the worker up. A fresh
|
||||||
|
// claim of a link with no row still starts, because the lookup miss
|
||||||
|
// below is treated as "not disabled".
|
||||||
|
disabled := false
|
||||||
|
if existing, found, err := st.LinkByTriple(tk.Link.Local, tk.Link.Remote, tk.Link.RemotePort); err != nil {
|
||||||
|
return err
|
||||||
|
} else if found {
|
||||||
|
disabled = existing.Disabled
|
||||||
|
}
|
||||||
// Start (or restart) the per-forward worker for exactly this link.
|
// Start (or restart) the per-forward worker for exactly this link.
|
||||||
// Each forward has its own frpc process (keyed by the forward
|
// Each forward has its own frpc process (keyed by the forward
|
||||||
// triple); restarting only this key leaves sibling forwards'
|
// triple); restarting only this key leaves sibling forwards'
|
||||||
// processes untouched.
|
// processes untouched.
|
||||||
|
if disabled {
|
||||||
|
// Mark the entry stopped rather than deleting it. The entry is the
|
||||||
|
// carrier that keeps the flag travelling around the ring, and
|
||||||
|
// UpdateTopologyDisabled also clears Active — which is what makes
|
||||||
|
// this entry inert for OfflineReassign() and the startup
|
||||||
|
// reconcile, so it is not resurrected later.
|
||||||
|
if ring != nil {
|
||||||
|
ring.UpdateTopologyDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, true)
|
||||||
|
}
|
||||||
|
log.Printf("ring[%s] claim %s skipped: %s→%s:%d is disabled", selfID, tk.ID, tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
if tk.Remote.Enabled {
|
if tk.Remote.Enabled {
|
||||||
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
if _, has := pm.Status(key); has {
|
if _, has := pm.Status(key); has {
|
||||||
@ -201,13 +230,118 @@ func main() {
|
|||||||
// stop/start: it wiped sibling forwards sharing the local or remote.
|
// stop/start: it wiped sibling forwards sharing the local or remote.
|
||||||
RevokeFn: func(ctx context.Context, tk *cluster.Task) error {
|
RevokeFn: func(ctx context.Context, tk *cluster.Task) error {
|
||||||
_ = st.SetLinkDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, true)
|
_ = st.SetLinkDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, true)
|
||||||
|
// The links table is node-local: only the node that received the
|
||||||
|
// stop request has the flag persisted, while the node that OWNS the
|
||||||
|
// forward is the one holding the worker (and is often a different
|
||||||
|
// machine). Trusting only the local row meant the owner's copy could
|
||||||
|
// still read disabled=false. The task now carries the flag, so use it
|
||||||
|
// as the authority and write it here.
|
||||||
|
ln, found, err := st.LinkByTriple(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
return err
|
||||||
|
case found:
|
||||||
|
ln.Disabled = true
|
||||||
|
// Persist via the same rewrite path the canvas uses, so the flag
|
||||||
|
// survives a restart on THIS node too.
|
||||||
|
links, err := st.ListLinks()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for i := range links {
|
||||||
|
if links[i].Local == ln.Local && links[i].Remote == ln.Remote && links[i].RemotePort == ln.RemotePort {
|
||||||
|
links[i].Disabled = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := st.ReplaceLinks(links); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case tk.Link.Disabled:
|
||||||
|
// No local row yet (this node never claimed the forward) but the
|
||||||
|
// task asserts the stop — record it so a later claim on this node
|
||||||
|
// cannot resurrect the forward.
|
||||||
|
loc, rem := tk.Local, tk.Remote
|
||||||
|
links, err := st.ListLinks()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
links = append(links, store.Link{
|
||||||
|
Local: loc.Name, Remote: rem.Name, RemotePort: tk.Link.RemotePort,
|
||||||
|
OffsetX: tk.Link.OffsetX, OffsetY: tk.Link.OffsetY,
|
||||||
|
Group: tk.Link.Group, Disabled: true,
|
||||||
|
})
|
||||||
|
if err := st.ReplaceLinks(links); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
if _, running := pm.Status(key); running {
|
if _, running := pm.Status(key); running {
|
||||||
_ = pm.Stop(key)
|
_ = pm.Stop(key)
|
||||||
}
|
}
|
||||||
|
// Mark the entry stopped instead of deleting it: the entry carries
|
||||||
|
// the flag around the ring, and clearing Active keeps it inert for
|
||||||
|
// the rebalancing paths (OfflineReassign skips inactive entries, and
|
||||||
|
// the startup reconcile skips disabled forwards), so the worker is
|
||||||
|
// not re-spawned on the next restart. Deleting it here is what used
|
||||||
|
// to leave the stop with no carrier at all.
|
||||||
|
if ring != nil {
|
||||||
|
ring.UpdateTopologyDisabled(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort, true)
|
||||||
|
}
|
||||||
log.Printf("ring[%s] revoked task %s: %s→%s:%d", selfID, tk.ID, tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
log.Printf("ring[%s] revoked task %s: %s→%s:%d", selfID, tk.ID, tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
|
// ReleaseFn stops this forward's worker for a REBALANCE, leaving the
|
||||||
|
// link ENABLED. This must not reuse RevokeFn: that marks the link
|
||||||
|
// stopped, and the released forward is about to run on another node —
|
||||||
|
// a "stopped" flag would make the claim path treat the re-published
|
||||||
|
// task as a disabled forward and refuse to start a worker for it,
|
||||||
|
// leaving the forward dead everywhere.
|
||||||
|
ReleaseFn: func(ctx context.Context, tk *cluster.Task) error {
|
||||||
|
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
|
if _, running := pm.Status(key); running {
|
||||||
|
_ = pm.Stop(key)
|
||||||
|
}
|
||||||
|
log.Printf("ring[%s] released worker %s (rebalancing): %s→%s:%d", selfID, key,
|
||||||
|
tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
// RestartFn brings an already-claimed forward back after a stop. It
|
||||||
|
// must NOT run the claim bookkeeping (ClaimFn), which appends links and
|
||||||
|
// writes topology state: the forward is already established, and a stop
|
||||||
|
// deliberately keeps its topology entry, so all that is needed is to
|
||||||
|
// clear the flag and respawn the worker. The engine has already
|
||||||
|
// re-marked the topology entry enabled before calling this.
|
||||||
|
RestartFn: func(ctx context.Context, tk *cluster.Task) error {
|
||||||
|
// Clear the stop on this node's own store, using the same path the
|
||||||
|
// start handler uses, so the local copy cannot disagree with the
|
||||||
|
// ring and block a later restart.
|
||||||
|
if ln, found, err := st.LinkByTriple(tk.Link.Local, tk.Link.Remote, tk.Link.RemotePort); err != nil {
|
||||||
|
return err
|
||||||
|
} else if found && ln.Disabled {
|
||||||
|
links, err := st.ListLinks()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for i := range links {
|
||||||
|
if links[i].Local == ln.Local && links[i].Remote == ln.Remote && links[i].RemotePort == ln.RemotePort {
|
||||||
|
links[i].Disabled = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := st.ReplaceLinks(links); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if tk.Remote.Enabled {
|
||||||
|
key := process.WorkerKey(tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
|
if _, has := pm.Status(key); has {
|
||||||
|
_ = pm.Restart(key)
|
||||||
|
} else {
|
||||||
|
_ = pm.Start(key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
log.Printf("ring[%s] restarted %s: %s→%s:%d", selfID, tk.ID, tk.Local.Name, tk.Remote.Name, tk.Link.RemotePort)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
},
|
},
|
||||||
ringTransport.SendTo(func(nodeID string) string {
|
ringTransport.SendTo(func(nodeID string) string {
|
||||||
if ring == nil {
|
if ring == nil {
|
||||||
@ -232,10 +366,24 @@ func main() {
|
|||||||
// left does NOT auto-rejoin.
|
// left does NOT auto-rejoin.
|
||||||
ring.SetPeerPersist(func(peersJSON string) error { return st.SetClusterPeers(peersJSON) })
|
ring.SetPeerPersist(func(peersJSON string) error { return st.SetClusterPeers(peersJSON) })
|
||||||
|
|
||||||
// Re-apply local store group labels onto the ring topology after each
|
// Re-apply local store state onto the ring topology after each state
|
||||||
// state adoption. Without this, group changes made via HTTP handlers
|
// adoption, and learn the ring's disabled decisions back into the store.
|
||||||
|
//
|
||||||
|
// Without the group half, group changes made via HTTP handlers
|
||||||
// (POST /forwards/assign) are overwritten by the next e.state = tk.State
|
// (POST /forwards/assign) are overwritten by the next e.state = tk.State
|
||||||
// and never propagate to other nodes.
|
// and never propagate.
|
||||||
|
//
|
||||||
|
// The disabled half makes the RING authoritative for stop/start. That is the
|
||||||
|
// only source all members can agree on: the links table is a per-node copy, so
|
||||||
|
// treating it as authoritative let two nodes disagree indefinitely (observed
|
||||||
|
// live: one node reading disabled=1 while the owner still read 0 and kept its
|
||||||
|
// worker running). This is a write-through, not a listener — it runs on every
|
||||||
|
// adoption, so any peer's decision lands here within a token cycle.
|
||||||
|
//
|
||||||
|
// Ordering matters in one direction only: the engine has already re-applied
|
||||||
|
// this node's not-yet-confirmed local decisions onto the adopted state (see
|
||||||
|
// reconcileLocalDisabled), so reading the ring here can never clobber an
|
||||||
|
// action the user just took on this node.
|
||||||
ring.SetTopologySync(func() {
|
ring.SetTopologySync(func() {
|
||||||
links, err := st.ListLinks()
|
links, err := st.ListLinks()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@ -244,8 +392,23 @@ func main() {
|
|||||||
for _, ln := range links {
|
for _, ln := range links {
|
||||||
ring.UpdateTopologyGroup(ln.Local, ln.Remote, ln.RemotePort, ln.Group)
|
ring.UpdateTopologyGroup(ln.Local, ln.Remote, ln.RemotePort, ln.Group)
|
||||||
}
|
}
|
||||||
|
for _, te := range ring.State().TopologyList() {
|
||||||
|
disabled, known := ring.TopologyDisabled(te.Local.Name, te.Remote.Name, te.Link.RemotePort)
|
||||||
|
if !known {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if ln, found, err := st.LinkByTriple(te.Local.Name, te.Remote.Name, te.Link.RemotePort); err == nil && found && ln.Disabled == disabled {
|
||||||
|
continue // store already agrees with the ring
|
||||||
|
}
|
||||||
|
_ = st.ReconcileLinkDisabled(te.Local.Name, te.Remote.Name, te.Link.RemotePort, disabled)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// Load balancer is created here (before the Handler) so the HTTP layer can
|
||||||
|
// expose its read-only report. Mode/parameters are applied further down, once
|
||||||
|
// the W4F_BALANCE_* environment has been parsed.
|
||||||
|
balancer := cluster.NewBalancer(ring, 600)
|
||||||
|
|
||||||
h := &httpapi.Handler{
|
h := &httpapi.Handler{
|
||||||
Store: st,
|
Store: st,
|
||||||
Process: pm,
|
Process: pm,
|
||||||
@ -254,6 +417,9 @@ func main() {
|
|||||||
Sessions: httpapi.NewSessionStore(),
|
Sessions: httpapi.NewSessionStore(),
|
||||||
Cluster: reg,
|
Cluster: reg,
|
||||||
Ring: ring,
|
Ring: ring,
|
||||||
|
// Balancer is created above, before the environment that configures it is
|
||||||
|
// parsed, because the HTTP layer needs the pointer.
|
||||||
|
Balancer: balancer,
|
||||||
// SelfAddr must be reachable by peers. 0.0.0.0/empty on a real host
|
// SelfAddr must be reachable by peers. 0.0.0.0/empty on a real host
|
||||||
// would break peer reconnect, so fall back to the hostname (container
|
// would break peer reconnect, so fall back to the hostname (container
|
||||||
// name inside docker is a routable addr on the compose network).
|
// name inside docker is a routable addr on the compose network).
|
||||||
@ -411,6 +577,65 @@ func main() {
|
|||||||
go ring.WatchTokenLoss(clusterCtx)
|
go ring.WatchTokenLoss(clusterCtx)
|
||||||
go ring.WatchLeader(clusterCtx)
|
go ring.WatchLeader(clusterCtx)
|
||||||
|
|
||||||
|
// Load balancing — SELF-DIRECTED, and dry-run by default.
|
||||||
|
//
|
||||||
|
// Every node runs this check against its OWN forwards. There is no
|
||||||
|
// coordinator: a node's load only actually falls when its own topology entry
|
||||||
|
// goes away, so only the owner can produce the state change that makes the
|
||||||
|
// ring hand a forward to somebody else. The node gives a forward up and
|
||||||
|
// re-publishes it as an ordinary unclaimed task; the ring's normal
|
||||||
|
// lowest-load claim then places it. (The earlier design had the leader pick a
|
||||||
|
// victim and a destination — but it can only see the imbalance, not resolve
|
||||||
|
// it, so it had to reach into another node's bookkeeping to make its decision
|
||||||
|
// true.)
|
||||||
|
//
|
||||||
|
// A release stops a worker and drops a live connection, so acting on the ring
|
||||||
|
// is opt-in:
|
||||||
|
//
|
||||||
|
// W4F_BALANCE_MODE=dry-run report the decision, release nothing (default)
|
||||||
|
// W4F_BALANCE_MODE=on actually release
|
||||||
|
// W4F_BALANCE_MODE=off do nothing at all
|
||||||
|
// W4F_BALANCE_COOLDOWN=600 min seconds between two releases by one node
|
||||||
|
// W4F_BALANCE_INTERVAL=10 check every N seconds
|
||||||
|
balMode := strings.ToLower(strings.TrimSpace(os.Getenv("W4F_BALANCE_MODE")))
|
||||||
|
if balMode == "" {
|
||||||
|
balMode = cluster.BalanceDryRun
|
||||||
|
}
|
||||||
|
balInterval := time.Duration(envFloat("W4F_BALANCE_INTERVAL", 10)) * time.Second
|
||||||
|
if balInterval < time.Second {
|
||||||
|
balInterval = 10 * time.Second
|
||||||
|
}
|
||||||
|
balCooldown := int64(envFloat("W4F_BALANCE_COOLDOWN", 600))
|
||||||
|
balancer.SetCooldown(balCooldown)
|
||||||
|
balancer.SetMode(balMode)
|
||||||
|
// A mode stored from the cluster page wins over the environment, so flipping
|
||||||
|
// the switch is not silently reverted by the next restart. SetMode is applied
|
||||||
|
// first so a stored value only has to be a no-op when the two agree.
|
||||||
|
balancer.SetModePersistence(
|
||||||
|
func() (string, bool) {
|
||||||
|
m, ok, err := st.BalanceMode()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("ring[%s] read stored balance mode: %v", selfID, err)
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return m, ok
|
||||||
|
},
|
||||||
|
func(m string) error {
|
||||||
|
s, err := st.Settings()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
s.BalanceMode = m
|
||||||
|
return st.UpdateSettings(s)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
// Report the EFFECTIVE mode, not balMode: a stored value may have overridden
|
||||||
|
// the environment, and a startup line naming the wrong mode is exactly how an
|
||||||
|
// operator ends up believing releasing is off while it is on.
|
||||||
|
log.Printf("ring[%s] load balancing %s (source=%s cooldown=%ds interval=%s, self-directed)",
|
||||||
|
selfID, strings.ToUpper(balancer.Mode()), balancer.ModeSource(), balCooldown, balInterval)
|
||||||
|
go balancer.Watch(clusterCtx, balInterval)
|
||||||
|
|
||||||
// Startup reconcile: every node (leader or not) checks its own topology
|
// Startup reconcile: every node (leader or not) checks its own topology
|
||||||
// entries and re-claims any whose per-forward frpc worker process is
|
// entries and re-claims any whose per-forward frpc worker process is
|
||||||
// absent — the topology was recovered via AdoptState but the worker
|
// absent — the topology was recovered via AdoptState but the worker
|
||||||
@ -419,17 +644,32 @@ func main() {
|
|||||||
// LocalOnly forwards are never in the topology and are handled by the
|
// LocalOnly forwards are never in the topology and are handled by the
|
||||||
// auto-start loop above.
|
// auto-start loop above.
|
||||||
go func() {
|
go func() {
|
||||||
for {
|
// No leader gate here. The reconcile body is per-NODE work: it re-claims
|
||||||
if ring.IsLeader() {
|
// the forwards THIS node owns, whose worker processes died with the
|
||||||
break
|
// restart. Every member owns forwards (a 3-node ring splits them), so
|
||||||
}
|
// gating on leadership meant non-leaders waited forever and never
|
||||||
|
// rebuilt their own workers — a node could come back up "in the cluster"
|
||||||
|
// while owning a forward with no process behind it. Observed live: after a
|
||||||
|
// rolling restart a non-leader rejoined, adopted state listing itself as
|
||||||
|
// the owner of a forward, and never started it, because it sat waiting to
|
||||||
|
// become leader.
|
||||||
|
//
|
||||||
|
// What actually needs settling is the ring, not the leadership: wait for
|
||||||
|
// the topology to show us (with a bounded fallback so a lone node that
|
||||||
|
// rejoined a ring whose leader is unreachable still makes progress).
|
||||||
|
deadline := time.After(30 * time.Second)
|
||||||
|
for !ring.HasSelfInTopology() {
|
||||||
select {
|
select {
|
||||||
case <-time.After(2 * time.Second):
|
case <-deadline:
|
||||||
|
log.Printf("ring[%s] startup reconcile: self not in topology after 30s, proceeding anyway", ring.ID)
|
||||||
|
case <-time.After(500 * time.Millisecond):
|
||||||
|
continue
|
||||||
case <-clusterCtx.Done():
|
case <-clusterCtx.Done():
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
break
|
||||||
}
|
}
|
||||||
// Let the ring stabilize after leader election / rejoin.
|
// Let the ring stabilize after election / rejoin before acting on it.
|
||||||
time.Sleep(3 * time.Second)
|
time.Sleep(3 * time.Second)
|
||||||
snap := ring.Snapshot()
|
snap := ring.Snapshot()
|
||||||
reclaimed := 0
|
reclaimed := 0
|
||||||
@ -437,6 +677,16 @@ func main() {
|
|||||||
if t.OwnerID != selfID {
|
if t.OwnerID != selfID {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
// A forward the user stopped must stay stopped: skip it here so a
|
||||||
|
// restart does not re-spawn its worker. The ClaimFn enforces the
|
||||||
|
// same rule (belt and braces — this also avoids a pointless
|
||||||
|
// claim→skip round trip per disabled forward on every boot).
|
||||||
|
if ln, found, err := st.LinkByTriple(t.Local.Name, t.Remote.Name, t.Link.RemotePort); err != nil {
|
||||||
|
log.Printf("ring[%s] reconcile: lookup %s→%s:%d: %v", ring.ID, t.Local.Name, t.Remote.Name, t.Link.RemotePort, err)
|
||||||
|
continue
|
||||||
|
} else if found && ln.Disabled {
|
||||||
|
continue
|
||||||
|
}
|
||||||
key := process.WorkerKey(t.Local.Name, t.Remote.Name, t.Link.RemotePort)
|
key := process.WorkerKey(t.Local.Name, t.Remote.Name, t.Link.RemotePort)
|
||||||
if _, has := pm.Status(key); has {
|
if _, has := pm.Status(key); has {
|
||||||
continue // worker already running
|
continue // worker already running
|
||||||
@ -568,6 +818,22 @@ func retryJoinCluster(ctx context.Context, ring *cluster.Engine, targetAddr, joi
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// envFloat reads a numeric environment variable, returning def when unset or
|
||||||
|
// unparseable. A malformed value must not silently change behaviour, so it falls
|
||||||
|
// back to the documented default rather than 0.
|
||||||
|
func envFloat(key string, def float64) float64 {
|
||||||
|
v := strings.TrimSpace(os.Getenv(key))
|
||||||
|
if v == "" {
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
f, err := strconv.ParseFloat(v, 64)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("env %s=%q is not a number; using %v", key, v, def)
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
// retryRejoinCached is the crash/restart recovery path: parse the persisted
|
// retryRejoinCached is the crash/restart recovery path: parse the persisted
|
||||||
// peer list (JSON of [{addr,key},...]) and try to rejoin via each peer in
|
// peer list (JSON of [{addr,key},...]) and try to rejoin via each peer in
|
||||||
// order. The first peer that accepts wins. Retry the whole list every 10s
|
// order. The first peer that accepts wins. Retry the whole list every 10s
|
||||||
@ -600,6 +866,20 @@ func retryRejoinCached(ctx context.Context, ring *cluster.Engine, peersJSON stri
|
|||||||
cancel()
|
cancel()
|
||||||
if err == nil {
|
if err == nil {
|
||||||
log.Printf("ring[%s] rejoined cluster via cached peer %s", ring.ID, p.Addr)
|
log.Printf("ring[%s] rejoined cluster via cached peer %s", ring.ID, p.Addr)
|
||||||
|
// A rejoined node must be able to drive the ring itself. The
|
||||||
|
// normal case is that the existing leader keeps circulating the
|
||||||
|
// token, but if WE are the leader (typical after a rolling
|
||||||
|
// restart where the leader came back last, or the leader died
|
||||||
|
// while we were down) nobody else will: StartRing is a no-op on
|
||||||
|
// non-leaders, and the ring sits still with every task stranded
|
||||||
|
// in pending. Observed live: the leader restarted, rejoined fine,
|
||||||
|
// and then never forwarded a token — cycle frozen, topology
|
||||||
|
// empty, pending=3.
|
||||||
|
//
|
||||||
|
// StartRing is safe to call unconditionally: it returns
|
||||||
|
// immediately unless this node leads, and it throttles itself
|
||||||
|
// against RoundDelay so a redundant call cannot double-send.
|
||||||
|
go ring.StartRing(ctx)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
log.Printf("ring[%s] rejoin %s failed: %v", ring.ID, p.Addr, err)
|
log.Printf("ring[%s] rejoin %s failed: %v", ring.ID, p.Addr, err)
|
||||||
|
|||||||
145
cmd/webui4frpc/rejoin_ring_test.go
Normal file
145
cmd/webui4frpc/rejoin_ring_test.go
Normal file
@ -0,0 +1,145 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"webui4frpc/internal/cluster"
|
||||||
|
"webui4frpc/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeJoinPeer stands in for a healthy cluster member: it answers the rejoin
|
||||||
|
// POST and records it, so the rejoin path can be exercised end to end.
|
||||||
|
type fakeJoinPeer struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
joined int
|
||||||
|
stateFn func() cluster.State
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *fakeJoinPeer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
|
p.mu.Lock()
|
||||||
|
p.joined++
|
||||||
|
p.mu.Unlock()
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{"state": p.stateFn()})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *fakeJoinPeer) joins() int {
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
return p.joined
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRejoinThenStartRingCoversTheRejoinPath is the regression test for the
|
||||||
|
// frozen ring, and it deliberately drives retryRejoinCached itself rather than
|
||||||
|
// calling StartRing directly — a test that calls the fix's own helper proves
|
||||||
|
// nothing about whether the fix is wired up (verified: such a test stayed green
|
||||||
|
// with the fix reverted).
|
||||||
|
//
|
||||||
|
// The failure, observed live on the three-node cluster: the leader restarted,
|
||||||
|
// rejoined successfully, logged "startup reconcile: reclaimed=0 submitted=3",
|
||||||
|
// and then never forwarded a token. cycle froze, topology stayed empty and the
|
||||||
|
// three forwards sat in pending forever, because rejoin returned immediately
|
||||||
|
// and only a leader may call StartRing.
|
||||||
|
func TestRejoinThenStartRingCoversTheRejoinPath(t *testing.T) {
|
||||||
|
selfID := "192.168.2.30:7500"
|
||||||
|
|
||||||
|
// The peer hands back a ring in which SELF is the leader — the situation a
|
||||||
|
// rolling restart produces when the leader comes back last.
|
||||||
|
var peerState cluster.State
|
||||||
|
peer := &fakeJoinPeer{stateFn: func() cluster.State { return peerState }}
|
||||||
|
|
||||||
|
// Bind on loopback so the engine's hardcoded peer address is reachable.
|
||||||
|
// We therefore advertise 127.0.0.1 in the ring while keeping the same IDs.
|
||||||
|
ln := httptest.NewServer(peer)
|
||||||
|
defer ln.Close()
|
||||||
|
|
||||||
|
peerState = cluster.State{
|
||||||
|
LeaderID: selfID,
|
||||||
|
Nodes: []cluster.Node{
|
||||||
|
{ID: selfID, Addr: selfID, Alive: true, IsLeader: true},
|
||||||
|
{ID: "peer:7500", Addr: "peer:7500", Alive: true},
|
||||||
|
},
|
||||||
|
Topology: map[string]*cluster.TopoEntry{},
|
||||||
|
PendingTasks: map[string]*cluster.Task{
|
||||||
|
"t1": {ID: "t1", Local: store.Local{Name: "web"},
|
||||||
|
Remote: store.Remote{Name: "frps1", IP: "1.2.3.4", Port: 7000, Enabled: true},
|
||||||
|
Link: store.Link{RemotePort: 18081}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Redirect the peer's advertised address at the transport level: the engine
|
||||||
|
// dials whatever the ring says, so give the ring the test server's address.
|
||||||
|
// JoinRing builds "http://<addr>/api/manager/cluster/join" itself, so the
|
||||||
|
// cached entry must be a bare host:port, not a URL.
|
||||||
|
realAddr := strings.TrimPrefix(ln.URL, "http://")
|
||||||
|
for i := range peerState.Nodes {
|
||||||
|
if peerState.Nodes[i].ID == "peer:7500" {
|
||||||
|
peerState.Nodes[i].Addr = realAddr
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var mu sync.Mutex
|
||||||
|
var emitted int
|
||||||
|
eng := cluster.NewEngine(selfID, selfID, "u", "p", "0.1.2", nil,
|
||||||
|
&cluster.AppHandler{ClaimFn: func(ctx context.Context, tk *cluster.Task) error { return nil }},
|
||||||
|
func(ctx context.Context, next string, tk *cluster.Token) error {
|
||||||
|
if tk != nil {
|
||||||
|
mu.Lock()
|
||||||
|
emitted++
|
||||||
|
mu.Unlock()
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
selfID, true, "")
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
// The rejoin path, as main() runs it. Cached peers point at the test server.
|
||||||
|
peersJSON, err := json.Marshal([]map[string]string{{"addr": realAddr, "key": "peerkey"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
defer close(done)
|
||||||
|
retryRejoinCached(ctx, eng, string(peersJSON))
|
||||||
|
}()
|
||||||
|
|
||||||
|
// retryRejoinCached returns once rejoin succeeds; give the goroutine a moment.
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
if peer.joins() > 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
sleepShort()
|
||||||
|
}
|
||||||
|
if peer.joins() == 0 {
|
||||||
|
t.Fatal("the rejoin never reached the peer — the test setup is wrong")
|
||||||
|
}
|
||||||
|
<-done
|
||||||
|
|
||||||
|
// Rejoin succeeded. The engine must now consider itself the leader...
|
||||||
|
if !eng.IsLeader() {
|
||||||
|
t.Fatalf("precondition broken: rejoined node does not consider itself leader")
|
||||||
|
}
|
||||||
|
sleepShort()
|
||||||
|
|
||||||
|
// ...and it must have restarted the ring: a token has to be emitted, or the
|
||||||
|
// cluster sits frozen with its forwards stranded in pending.
|
||||||
|
mu.Lock()
|
||||||
|
got := emitted
|
||||||
|
mu.Unlock()
|
||||||
|
if got == 0 {
|
||||||
|
t.Fatal("after a successful rejoin the leader emitted NO token: the ring is frozen " +
|
||||||
|
"and pending forwards are never claimed (the live failure this guards)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
func sleepShort() { time.Sleep(10 * time.Millisecond) }
|
||||||
199
docs/how-it-works.md
Normal file
199
docs/how-it-works.md
Normal file
@ -0,0 +1,199 @@
|
|||||||
|
# webui4frpc 内部机制(虚拟 frps 实证)
|
||||||
|
|
||||||
|
> 本文所有结论均由**隔离的虚拟 frps 环境**实测得出,不依赖生产集群。
|
||||||
|
> 复现脚本见 `scripts/probe-with-virtual-frps.sh`。
|
||||||
|
>
|
||||||
|
> 探测时间:2026-10-02 · frp 版本 0.61.1 · 二进制 md5 三机一致(5c4c23d5…)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 数据流:DB 是唯一真源,configs/ 是派生产物
|
||||||
|
|
||||||
|
```
|
||||||
|
API (HTTP)
|
||||||
|
→ manager.db (SQLite) ← 真源
|
||||||
|
↓ Render(forward) 渲染
|
||||||
|
→ data/configs/<name>.json ← 派生产物
|
||||||
|
↓ exec: frpc -c <cfg>
|
||||||
|
→ frpc worker 进程
|
||||||
|
```
|
||||||
|
|
||||||
|
**推论(已在生产验证)**:手改 `configs/*.json` 会被下一次 DB 写回覆盖。
|
||||||
|
配置变更必须走 API(`PUT /api/manager/canvas` 或 `/remotes`)。
|
||||||
|
|
||||||
|
`configs/` 目录的文件名格式为 `<local>~<remote>~<remotePort>.json`。
|
||||||
|
只有**实跑的那一份**会被 frpc `-c` 引用;同名不同端口的历史文件会残留。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 集群不复制配置,只复制环状态
|
||||||
|
|
||||||
|
`internal/cluster/token_transport.go` 在环上传的东西:
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Token struct { // ring.go:106
|
||||||
|
Cycle int64
|
||||||
|
State State // 节点拓扑 / leader / log delta
|
||||||
|
Log []LogEntry
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
这是一个**领导者选举 + 拓扑状态**的载体,**不包含** remotes / locals / links。
|
||||||
|
`sessions are not replicated across the ring`(`session.go:32`)——
|
||||||
|
每个节点有自己的 `manager.db`。
|
||||||
|
|
||||||
|
**推论**:
|
||||||
|
- 令牌环 peer 认证用的是 webui Basic 凭据(`SetBasicAuth(User, Pass)`),
|
||||||
|
与 frps `auth.token` 是两个独立密钥
|
||||||
|
|
||||||
|
### 配置为什么会同步到新节点
|
||||||
|
|
||||||
|
`Token.State` 里除了环状态,还带 `Topology map[string]*TopoEntry`,
|
||||||
|
而 **`TopoEntry` 携带完整配置**(`ring.go:76`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
type TopoEntry struct {
|
||||||
|
TaskID string
|
||||||
|
OwnerID string
|
||||||
|
Local store.Local // 完整 local 配置
|
||||||
|
Remote store.Remote // 完整 remote 配置(含 token)
|
||||||
|
Link store.Link
|
||||||
|
Active bool
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
owner 节点 claim 任务时,配置**直接写进本地 DB**(`cmd/webui4frpc/main.go:147`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
ClaimFn: func(ctx, tk) error {
|
||||||
|
st.UpsertLocal(tk.Local)
|
||||||
|
st.UpsertRemote(tk.Remote)
|
||||||
|
... st.ReplaceLinks(links) // 去重后替换
|
||||||
|
// 然后 spawn worker
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
读侧对应 `handleCanvasGet`(`handlers.go:22`)的合并:把拓扑里本节点没有的
|
||||||
|
local/remote/link 补进画布,让每台都看到**全集群视图**。这是 read-time merge,
|
||||||
|
**不回写 SQLite**(避免触发 per-forward stop/start)。
|
||||||
|
|
||||||
|
### 同步边界(重要)
|
||||||
|
|
||||||
|
| 内容 | 是否同步 | 机制 |
|
||||||
|
|---|---|---|
|
||||||
|
| **已建立 forward 的配置** | ✅ | 随 `Topology` 环内流动,owner claim 时落库 |
|
||||||
|
| 环状态(leader / 节点 / log delta) | ✅ | `Token.State` |
|
||||||
|
| remote 定义变更(改 token 等) | ⚠️ 走任务 | 不是 DB 快照复制 |
|
||||||
|
| 仅有定义、从未成为 active forward 的配置 | ❌ | 没有 `TopoEntry` 就没有载体 |
|
||||||
|
| WebUI session | ❌ | `session.go:32` 明确不复制 |
|
||||||
|
|
||||||
|
**推论**:
|
||||||
|
- "在一台改、集群自动同步"对**已建立的 forward** 成立——靠 `Topology` + `ClaimFn`
|
||||||
|
- 改 frps remote 的 token 会随任务下发到 owner,不必逐台手改
|
||||||
|
- 但只存在于某台 DB、且无对应 active forward 的配置不会自动过去
|
||||||
|
- 这也是本项目 frps token 轮换的正确路径:走 API 提交任务,而非逐台改文件
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. forward 的 owner 归属:非 localOnly 由 owner 拉起
|
||||||
|
|
||||||
|
`startForward`(`handlers_forwards.go:62`)的两条分支:
|
||||||
|
|
||||||
|
| forward 类型 | 谁执行 | 机制 |
|
||||||
|
|---|---|---|
|
||||||
|
| `localOnly = true` | **本节点直接** `Process.Start(key)` | 无需环 |
|
||||||
|
| `localOnly = false` | **拓扑 owner 节点** | `Ring.SubmitTask` / `UpdateTopologyDisabled` 投递 |
|
||||||
|
|
||||||
|
非 localOnly 分支有一段注释解释了为什么需要专门的 RESTART 任务:
|
||||||
|
|
||||||
|
> - `SubmitTask` is idempotency-guarded, and the entry still exists … so the submission is deduped away
|
||||||
|
> - the claim path drops any task for a forward that already has an owner
|
||||||
|
>
|
||||||
|
> Without it a forward the user can stop but not restart
|
||||||
|
|
||||||
|
**实测观察**:单节点 fresh 集群(`cycle: 0`、无 peer)下,
|
||||||
|
`POST /forwards/start` 对非 localOnly forward 返回 `{"ok":true}`,
|
||||||
|
但 `configs/` 为空、`connState=not_started`——环不转时任务无人 claim。
|
||||||
|
单节点环境应使用 `localOnly: true` 的 forward,或加入多节点集群。
|
||||||
|
**这是设计行为,不是缺陷。**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 三个易踩的坑
|
||||||
|
|
||||||
|
### 4.1 `binaryPath` 为空 → worker 静默不启动
|
||||||
|
|
||||||
|
`main.go:89` 的解析顺序:
|
||||||
|
|
||||||
|
```go
|
||||||
|
BinaryPath: func() string {
|
||||||
|
if s, _ := st.Settings(); s.BinaryPath != "" { return s.BinaryPath }
|
||||||
|
return fallbackBin
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
全新 workdir(未装过 frpc)`binaryPath` 为空,回退到 `fallbackBin`。
|
||||||
|
若该路径也不存在,`spawn()` 报错,但 **`POST /forwards/start` 仍返回 `{"ok":true}`**
|
||||||
|
(错误只落在 worker 日志里)。排障时先查:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GET /api/manager/binary/status # {"binaryPath": "..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 `UpdateSettings` 是全量替换
|
||||||
|
|
||||||
|
`store.go:754` 的 `UpdateSettings` 直接 `json.Marshal(st)` 覆盖整行。
|
||||||
|
POST 一个**不含** `binaryPath` 的 settings JSON,会把该字段**清空**。
|
||||||
|
改 settings 前必须先 `GET` 现有值再整体回传。
|
||||||
|
|
||||||
|
### 4.3 画布会自动改写 loopback 地址
|
||||||
|
|
||||||
|
`applyCanvas`(`handlers.go:108`)对**非 localOnly** 的 local:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if !canvas.Locals[i].LocalOnly && cluster.IsLoopbackIP(canvas.Locals[i].IP) {
|
||||||
|
canvas.Locals[i].IP = cluster.RewriteForCluster(canvas.Locals[i].IP)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
实测:`PUT /canvas` 传 `127.0.0.1` → 落库变成 `192.168.2.60`(本机 LAN 地址)。
|
||||||
|
这是为了保证其它节点能连到该 local。**localOnly 的 local 保持 loopback 不变。**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 启动凭据来自命令行 flag
|
||||||
|
|
||||||
|
`main.go:64` 每次启动都 `SyncSystemUser(*user, *pass)`,把 `-user/-password`
|
||||||
|
同步进 `users` 表(`system=1`)。不传 flag 时用内置默认值,
|
||||||
|
`POST /api/manager/login` 需要 JSON body(`{"username","password"}`),
|
||||||
|
拿到身份后**不返回 token**——后续用 HTTP Basic。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. API 速查(实测路径)
|
||||||
|
|
||||||
|
| 用途 | 方法 | 路径 |
|
||||||
|
|---|---|---|
|
||||||
|
| 登录(返回身份,无 token) | POST | `/api/manager/login` |
|
||||||
|
| 全量保存画布(local+remote+link) | **PUT** | `/api/manager/canvas` |
|
||||||
|
| 读取画布 | GET | `/api/manager/canvas` |
|
||||||
|
| 启/停单条 forward | POST | `/api/manager/forwards/start` `/stop` |
|
||||||
|
| 分组启停 | POST | `/api/manager/forwards/group/start` `/stop` |
|
||||||
|
| 改转发归属 | POST | `/api/manager/forwards/assign` |
|
||||||
|
| 二进制状态 | GET | `/api/manager/binary/status` |
|
||||||
|
| 环拓扑 | GET | `/api/manager/cluster/ring` |
|
||||||
|
| 节点列表 | GET | `/api/manager/cluster/nodes` |
|
||||||
|
|
||||||
|
认证:HTTP Basic(`Authorization: Basic base64(user:pass)`)。
|
||||||
|
`POST` 到 `/canvas` 会返回 `method not allowed`——**必须用 PUT**。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 复现
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scripts/probe-with-virtual-frps.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
搭建 frps 0.61.1(`bindPort=17000` / `vhostHTTPPort=17080` / `vhostHTTPsPort=17443`,
|
||||||
|
独立 token),用独立 workdir 拉起 webui4frpc(`:17500`),全流程不触碰生产。
|
||||||
728
internal/cluster/balance_test.go
Normal file
728
internal/cluster/balance_test.go
Normal file
@ -0,0 +1,728 @@
|
|||||||
|
package cluster
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"webui4frpc/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// mustState round-trips the seed through JSON so the engine under test owns its
|
||||||
|
// own copy of Nodes/Topology. AdoptState assigns the state by value, but Nodes is
|
||||||
|
// a slice: handing one seed to several engines would alias one backing array and
|
||||||
|
// make a mutation on one node visible on the others — a test artefact that
|
||||||
|
// previously produced a false green here.
|
||||||
|
func mustState(t *testing.T, s State) *State {
|
||||||
|
t.Helper()
|
||||||
|
b, err := json.Marshal(s)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var out State
|
||||||
|
if err := json.Unmarshal(b, &out); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
e := newTestEngine("probe", true)
|
||||||
|
e.AdoptState(out)
|
||||||
|
return e.State()
|
||||||
|
}
|
||||||
|
|
||||||
|
// migrateRing builds a 3-node ring where n1 owns every forward, plus n2/n3 idle.
|
||||||
|
func migrateRing(t *testing.T, ownCount int) (*Engine, *Engine, *Engine) {
|
||||||
|
t.Helper()
|
||||||
|
mk := func(id string, leader bool) *Engine {
|
||||||
|
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5}}
|
||||||
|
return NewEngine(id, id+":7500", "u", "p", "0.1.3", nil, h,
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||||
|
id+":7500", leader, "")
|
||||||
|
}
|
||||||
|
n1, n2, n3 := mk("n1", true), mk("n2", false), mk("n3", false)
|
||||||
|
|
||||||
|
seed := State{
|
||||||
|
LeaderID: "n1",
|
||||||
|
Nodes: []Node{
|
||||||
|
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true, Load: Load{Forwards: ownCount, MemPct: 5}},
|
||||||
|
{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{MemPct: 5}},
|
||||||
|
{ID: "n3", Addr: "n3:7500", Alive: true, Load: Load{MemPct: 5}},
|
||||||
|
},
|
||||||
|
Topology: map[string]*TopoEntry{},
|
||||||
|
PendingTasks: map[string]*Task{},
|
||||||
|
}
|
||||||
|
for _, e := range []*Engine{n1, n2, n3} {
|
||||||
|
var own State
|
||||||
|
b, _ := json.Marshal(seed)
|
||||||
|
if err := json.Unmarshal(b, &own); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
e.AdoptState(own)
|
||||||
|
}
|
||||||
|
// n1 claims `ownCount` forwards.
|
||||||
|
for i := 0; i < ownCount; i++ {
|
||||||
|
n1.state.AddTopology(&Task{
|
||||||
|
ID: "f" + string(rune('a'+i)),
|
||||||
|
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||||
|
Link: store.Link{RemotePort: 100 + i},
|
||||||
|
}, "n1")
|
||||||
|
}
|
||||||
|
var snap State
|
||||||
|
b, _ := json.Marshal(n1.state)
|
||||||
|
_ = json.Unmarshal(b, &snap)
|
||||||
|
for _, e := range []*Engine{n1, n2, n3} {
|
||||||
|
e.AdoptState(snap)
|
||||||
|
}
|
||||||
|
return n1, n2, n3
|
||||||
|
}
|
||||||
|
|
||||||
|
// errNotWritable stands in for a failing settings write.
|
||||||
|
var errNotWritable = errors.New("not writable")
|
||||||
|
|
||||||
|
// ---- helpers ----
|
||||||
|
|
||||||
|
// balanceRing builds a 3-node ring with `owned` forwards on n1 (the evaluation
|
||||||
|
// node) and `others` on each of the other two, so a test can set up any
|
||||||
|
// distribution directly. The replicated Nodes[].Load is made consistent with the
|
||||||
|
// topology, because that is what a real token round leaves behind.
|
||||||
|
func balanceRing(t *testing.T, owned, others int) (*Engine, *Engine, *Engine) {
|
||||||
|
t.Helper()
|
||||||
|
mk := func(id string, leader bool) *Engine {
|
||||||
|
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5}}
|
||||||
|
return NewEngine(id, id+":7500", "u", "p", "0.1.3", nil, h,
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||||
|
id+":7500", leader, "")
|
||||||
|
}
|
||||||
|
n1, n2, n3 := mk("n1", true), mk("n2", false), mk("n3", false)
|
||||||
|
|
||||||
|
seed := State{
|
||||||
|
LeaderID: "n1",
|
||||||
|
Nodes: []Node{
|
||||||
|
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true, Load: Load{Forwards: owned, MemPct: 5}},
|
||||||
|
{ID: "n2", Addr: "n2:7500", Alive: true, Load: Load{Forwards: others, MemPct: 5}},
|
||||||
|
{ID: "n3", Addr: "n3:7500", Alive: true, Load: Load{Forwards: others, MemPct: 5}},
|
||||||
|
},
|
||||||
|
Topology: map[string]*TopoEntry{},
|
||||||
|
PendingTasks: map[string]*Task{},
|
||||||
|
}
|
||||||
|
for _, e := range []*Engine{n1, n2, n3} {
|
||||||
|
var own State
|
||||||
|
b, _ := json.Marshal(seed)
|
||||||
|
if err := json.Unmarshal(b, &own); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
e.AdoptState(own)
|
||||||
|
}
|
||||||
|
// Build the whole topology in n1's state, then broadcast it. Adding each
|
||||||
|
// node's forwards to its OWN state and then re-adopting from n1 would discard
|
||||||
|
// the others (one state is the source of the broadcast), which is how an
|
||||||
|
// earlier version of this helper silently produced a 1-1-1 split regardless
|
||||||
|
// of the arguments.
|
||||||
|
add := func(owner, prefix string, count, basePort int) {
|
||||||
|
for i := 0; i < count; i++ {
|
||||||
|
n1.state.AddTopology(&Task{
|
||||||
|
ID: prefix + strconv.Itoa(i),
|
||||||
|
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||||
|
Link: store.Link{RemotePort: basePort + i},
|
||||||
|
}, owner)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
add("n1", "f", owned, 100)
|
||||||
|
add("n2", "g", others, 200)
|
||||||
|
add("n3", "h", others, 300)
|
||||||
|
|
||||||
|
var snap State
|
||||||
|
b, _ := json.Marshal(n1.state)
|
||||||
|
_ = json.Unmarshal(b, &snap)
|
||||||
|
for _, e := range []*Engine{n1, n2, n3} {
|
||||||
|
e.AdoptState(snap)
|
||||||
|
}
|
||||||
|
return n1, n2, n3
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- the rule ----
|
||||||
|
|
||||||
|
// TestShareIsTheEvenSplit pins the arithmetic, including the ceiling. An even
|
||||||
|
// split of 3 over 3 is 1 (not 1.0 floored to 0): with a floor, a node holding
|
||||||
|
// one of three forwards would still be "over its share" and shed it.
|
||||||
|
func TestShareIsTheEvenSplit(t *testing.T) {
|
||||||
|
for _, tc := range []struct{ total, alive, want int }{
|
||||||
|
{3, 3, 1}, {4, 3, 2}, {6, 2, 3}, {1, 3, 1}, {0, 3, 0}, {7, 3, 3},
|
||||||
|
} {
|
||||||
|
got := (tc.total + tc.alive - 1) / tc.alive
|
||||||
|
if got != tc.want {
|
||||||
|
t.Errorf("share(%d over %d) = %d, want %d", tc.total, tc.alive, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestOwnerReleasesWhenOverItsShare: 3 forwards on n1, none elsewhere → n1 holds
|
||||||
|
// 3 against a share of 1 and must give one up.
|
||||||
|
func TestOwnerReleasesWhenOverItsShare(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
|
||||||
|
pick, p, blocked := b.assess(1000)
|
||||||
|
if pick == nil {
|
||||||
|
t.Fatalf("n1 holds 3 of 3 forwards and should release one (blocked=%q)", blocked)
|
||||||
|
}
|
||||||
|
if blocked != "" {
|
||||||
|
t.Errorf("nothing should block the first release, got %q", blocked)
|
||||||
|
}
|
||||||
|
if p.mine != 3 || p.share != 1 || p.total != 3 || p.alive != 3 {
|
||||||
|
t.Errorf("pool = %+v, want mine=3 share=1 total=3 alive=3", p)
|
||||||
|
}
|
||||||
|
if pick.OwnerID != "n1" {
|
||||||
|
t.Errorf("must release a forward n1 owns, got owner %q", pick.OwnerID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReleasingStopsAtTheShare is the bound that keeps the rule from emptying a
|
||||||
|
// node: after shedding one, n1 holds 1 of 3 over 3 nodes, which is no longer more
|
||||||
|
// than the share, and it must stop.
|
||||||
|
func TestReleasingStopsAtTheShare(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 1, 1) // even 1-1-1 split
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
|
||||||
|
pick, p, blocked := b.assess(1000)
|
||||||
|
if pick != nil {
|
||||||
|
t.Errorf("an even 1-1-1 split must not release anything, got %s:%d (blocked=%q)",
|
||||||
|
pick.Local.Name, pick.Link.RemotePort, blocked)
|
||||||
|
}
|
||||||
|
if p.mine != 1 || p.share != 1 {
|
||||||
|
t.Errorf("pool = %+v, want mine=1 share=1", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSingleNodeNeverReleases: a standalone node is the whole cluster, so its
|
||||||
|
// share is everything it holds. Without this the node would shed forwards onto
|
||||||
|
// nobody (they would sit unclaimed until it re-claimed them).
|
||||||
|
func TestSingleNodeNeverReleases(t *testing.T) {
|
||||||
|
e := newTestEngine("solo", true)
|
||||||
|
e.state.Nodes = []Node{{ID: "solo", Addr: "solo:7500", Alive: true, IsLeader: true}}
|
||||||
|
for i := 0; i < 5; i++ {
|
||||||
|
e.state.AddTopology(&Task{
|
||||||
|
ID: "f" + strconv.Itoa(i), Local: store.Local{Name: "web"},
|
||||||
|
Remote: store.Remote{Name: "frps1"}, Link: store.Link{RemotePort: 100 + i},
|
||||||
|
}, "solo")
|
||||||
|
}
|
||||||
|
b := NewBalancer(e, 0)
|
||||||
|
if pick, _, blocked := b.assess(1000); pick != nil {
|
||||||
|
t.Errorf("a single-node ring must not release; got blocked=%q", blocked)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBelowFloorNeverReleases: two forwards on a node is the floor. A node
|
||||||
|
// holding exactly two at an even split of two must keep both — one release per
|
||||||
|
// node per pass is the granularity, and shedding below the floor would empty it.
|
||||||
|
func TestBelowFloorNeverReleases(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 2, 2) // 6 total over 3 nodes → share 2
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
if pick, p, blocked := b.assess(1000); pick != nil {
|
||||||
|
t.Errorf("2 owned against share %d must not release (blocked=%q, mine=%d)",
|
||||||
|
p.share, blocked, p.mine)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNeverReleasesAStoppedForward: a stopped forward's entry is the carrier of
|
||||||
|
// the user's stop decision. Releasing it would hand another node a STOPPED
|
||||||
|
// forward as if it were work to run.
|
||||||
|
func TestNeverReleasesAStoppedForward(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
// Stop two of n1's three; the only live one must be chosen.
|
||||||
|
for _, te := range n1.state.ForwardsOwnedBy("n1") {
|
||||||
|
if te.Link.RemotePort == 100 || te.Link.RemotePort == 101 {
|
||||||
|
n1.state.UpdateTopologyDisabled("web", "frps1", te.Link.RemotePort, true)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
pick, _, _ := b.assess(1000)
|
||||||
|
if pick == nil {
|
||||||
|
t.Fatal("n1 still owns one live forward and is over its share")
|
||||||
|
}
|
||||||
|
if pick.Link.RemotePort != 102 {
|
||||||
|
t.Errorf("released port %d; a stopped forward (100/101) must never be chosen",
|
||||||
|
pick.Link.RemotePort)
|
||||||
|
}
|
||||||
|
|
||||||
|
// And with every forward stopped there is nothing to release.
|
||||||
|
n1.state.UpdateTopologyDisabled("web", "frps1", 102, true)
|
||||||
|
if pick, _, blocked := b.assess(1000); pick != nil {
|
||||||
|
t.Errorf("all forwards stopped: nothing may be released (blocked=%q)", blocked)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCandidateChoiceIsDeterministic: the same state must pick the same forward,
|
||||||
|
// so the decision is explainable and a re-evaluation does not churn a different
|
||||||
|
// connection each tick.
|
||||||
|
func TestCandidateChoiceIsDeterministic(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 4, 0)
|
||||||
|
a := releaseCandidate(n1.State(), "n1")
|
||||||
|
c := releaseCandidate(n1.State(), "n1")
|
||||||
|
if a == nil || c == nil {
|
||||||
|
t.Fatal("expected a candidate")
|
||||||
|
}
|
||||||
|
if a.TaskID != c.TaskID || a.Link.RemotePort != c.Link.RemotePort {
|
||||||
|
t.Errorf("candidate changed between identical evaluations: %s:%d vs %s:%d",
|
||||||
|
a.TaskID, a.Link.RemotePort, c.TaskID, c.Link.RemotePort)
|
||||||
|
}
|
||||||
|
if a.Link.RemotePort != 100 {
|
||||||
|
t.Errorf("expected the lowest port (100), got %d", a.Link.RemotePort)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- the runtime signal that makes this safe ----
|
||||||
|
|
||||||
|
// TestRevokeLeavesACarrierAndReleaseRemovesIt is the reason the release path has
|
||||||
|
// to be a distinct task kind.
|
||||||
|
//
|
||||||
|
// A user stop keeps the topology entry as the carrier of the disabled flag, so
|
||||||
|
// the entry is STILL THERE and still attributed to the owner. A rebalance is the
|
||||||
|
// opposite: the entry must go, because (a) the node's load is derived from
|
||||||
|
// owned entries so it cannot fall while the entry exists, and (b) every node's
|
||||||
|
// duplicate guard refuses a creation task for a forward that already has an owner
|
||||||
|
// — so the re-published task would be dropped by everyone and the forward
|
||||||
|
// stranded on the node that gave it up.
|
||||||
|
func TestRevokeLeavesACarrierAndReleaseRemovesIt(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
|
||||||
|
// A stop marks the entry and keeps it.
|
||||||
|
n1.state.UpdateTopologyDisabled("web", "frps1", 100, true)
|
||||||
|
tk := &Task{Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||||
|
Link: store.Link{RemotePort: 100}}
|
||||||
|
if owner := n1.state.TopologyOwner(tk); owner != "n1" {
|
||||||
|
t.Fatalf("a stop must keep the entry as the flag's carrier, owner=%q", owner)
|
||||||
|
}
|
||||||
|
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 3 {
|
||||||
|
t.Fatalf("a stop must not change the owner's forward count, got %d", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A release removes the entry (leaving the other two alone).
|
||||||
|
if !n1.RemoveTopologyEntry("web", "frps1", 100) {
|
||||||
|
t.Fatal("release must remove the entry")
|
||||||
|
}
|
||||||
|
if owner := n1.state.TopologyOwner(tk); owner != "" {
|
||||||
|
t.Errorf("after a release the forward must have no owner, got %q", owner)
|
||||||
|
}
|
||||||
|
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 2 {
|
||||||
|
t.Errorf("a release must reduce the owner's count, got %d", got)
|
||||||
|
}
|
||||||
|
if got := len(n1.state.TopologyList()); got != 2 {
|
||||||
|
t.Errorf("the other forwards must be untouched, got %d entries", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReleaseTaskExecutesOnlyOnTheOwner: like a revoke, a release is
|
||||||
|
// owner-directed. Executing it elsewhere would be an attempt to move somebody
|
||||||
|
// else's load, which cannot work — the owner's own count is what must fall.
|
||||||
|
func TestReleaseTaskExecutesOnlyOnTheOwner(t *testing.T) {
|
||||||
|
n1, n2, _ := balanceRing(t, 3, 0)
|
||||||
|
|
||||||
|
released := map[string]int{}
|
||||||
|
for _, e := range []*Engine{n1, n2} {
|
||||||
|
h := e.Handler.(*fakeHandler)
|
||||||
|
h.release = func(ctx context.Context, tk *Task) error {
|
||||||
|
released[e.ID]++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tk := n1.ReleaseTask(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||||
|
store.Link{RemotePort: 100})
|
||||||
|
if tk == nil {
|
||||||
|
t.Fatal("ReleaseTask returned nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each Engine keeps its OWN state, so a task published on n1 is not visible
|
||||||
|
// to n2 until a token round carries it. Broadcast it the way the ring does.
|
||||||
|
var carry State
|
||||||
|
cb, _ := json.Marshal(n1.state)
|
||||||
|
_ = json.Unmarshal(cb, &carry)
|
||||||
|
n2.AdoptState(carry)
|
||||||
|
n2.state.PendingTasks[tk.ID] = tk
|
||||||
|
|
||||||
|
// n2 is the lightest node and does NOT own the forward: it must leave the
|
||||||
|
// task alone.
|
||||||
|
if err := n2.runCommands(context.Background(), &Token{}); err != nil {
|
||||||
|
t.Fatalf("n2 runCommands: %v", err)
|
||||||
|
}
|
||||||
|
if released["n2"] != 0 {
|
||||||
|
t.Errorf("n2 released %d forward(s); a release is owner-directed only", released["n2"])
|
||||||
|
}
|
||||||
|
if _, still := n2.state.PendingTasks[tk.ID]; !still {
|
||||||
|
t.Error("n2 consumed a release it does not own; it must ride the token to the owner")
|
||||||
|
}
|
||||||
|
|
||||||
|
// n1 owns it: it must release, drop the entry, and re-publish the forward.
|
||||||
|
if err := n1.runCommands(context.Background(), &Token{}); err != nil {
|
||||||
|
t.Fatalf("n1 runCommands: %v", err)
|
||||||
|
}
|
||||||
|
if released["n1"] != 1 {
|
||||||
|
t.Fatalf("the owner released %d times, want 1", released["n1"])
|
||||||
|
}
|
||||||
|
if owner := n1.state.TopologyOwner(&Task{
|
||||||
|
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||||
|
Link: store.Link{RemotePort: 100},
|
||||||
|
}); owner != "" {
|
||||||
|
t.Errorf("the released forward still has owner %q", owner)
|
||||||
|
}
|
||||||
|
// And it is back in the ring as an ordinary creation task.
|
||||||
|
var republished *Task
|
||||||
|
for _, p := range n1.state.PendingList() {
|
||||||
|
if !p.Release && !p.Revoke && !p.Restart &&
|
||||||
|
p.Link.RemotePort == 100 && p.Local.Name == "web" {
|
||||||
|
republished = p
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if republished == nil {
|
||||||
|
t.Fatal("the released forward was not re-published as an unclaimed task")
|
||||||
|
}
|
||||||
|
if republished.Link.Disabled {
|
||||||
|
t.Error("the re-published forward must be ENABLED, not marked stopped")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- dry-run / off ----
|
||||||
|
|
||||||
|
// TestDryRunReleasesNothing: dry-run is the default, and its whole value is that
|
||||||
|
// it can run on a live cluster without disturbing anything.
|
||||||
|
func TestDryRunReleasesNothing(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceDryRun)
|
||||||
|
|
||||||
|
msg := b.Step(1000)
|
||||||
|
if !strings.Contains(msg, "would release") {
|
||||||
|
t.Fatalf("dry-run should report what it would do, got %q", msg)
|
||||||
|
}
|
||||||
|
if b.InFlight() {
|
||||||
|
t.Error("dry-run started a release")
|
||||||
|
}
|
||||||
|
for _, tk := range n1.state.PendingList() {
|
||||||
|
if tk.Release {
|
||||||
|
t.Error("dry-run published a release task")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := len(n1.state.TopologyList()); got != 3 {
|
||||||
|
t.Errorf("dry-run changed the topology: %d entries, want 3", got)
|
||||||
|
}
|
||||||
|
// A second identical tick must not repeat the same line (log hygiene).
|
||||||
|
if again := b.Step(1010); again != "" {
|
||||||
|
t.Errorf("dry-run repeated an unchanged decision: %q", again)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestModeOffDoesNothing reinforces that OFF means off, not "dry-run".
|
||||||
|
func TestModeOffDoesNothing(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceOff)
|
||||||
|
if msg := b.Step(1000); msg != "" {
|
||||||
|
t.Errorf("mode off reported %q", msg)
|
||||||
|
}
|
||||||
|
if b.InFlight() {
|
||||||
|
t.Error("mode off started a release")
|
||||||
|
}
|
||||||
|
// A typo must not enable releasing.
|
||||||
|
b.SetMode("yes-please")
|
||||||
|
if b.Mode() != BalanceDryRun {
|
||||||
|
t.Errorf("an unknown mode must fall back to dry-run, got %q", b.Mode())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- end-to-end through the ring machinery ----
|
||||||
|
|
||||||
|
// TestOwnerReleaseThenRingClaimsElsewhere is the whole design in one test: the
|
||||||
|
// owner gives the forward up, and the ordinary lowest-load claim then places it
|
||||||
|
// on an idle node — exactly what the self-directed rule depends on.
|
||||||
|
func TestOwnerReleaseThenRingClaimsElsewhere(t *testing.T) {
|
||||||
|
n1, n2, n3 := balanceRing(t, 3, 0)
|
||||||
|
|
||||||
|
// n2 and n3 must look idle to the claim path (they do: Forwards=0).
|
||||||
|
claimed := map[string]int{}
|
||||||
|
for _, e := range []*Engine{n1, n2, n3} {
|
||||||
|
h := e.Handler.(*fakeHandler)
|
||||||
|
h.claim = func(ctx context.Context, tk *Task) error {
|
||||||
|
claimed[e.ID]++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceOn)
|
||||||
|
|
||||||
|
// n1 is over its share → it releases.
|
||||||
|
msg := b.Step(1000)
|
||||||
|
if !strings.Contains(msg, "releasing") {
|
||||||
|
t.Fatalf("expected a release, got %q", msg)
|
||||||
|
}
|
||||||
|
if err := n1.runCommands(context.Background(), &Token{}); err != nil {
|
||||||
|
t.Fatalf("n1 runCommands: %v", err)
|
||||||
|
}
|
||||||
|
// Find the task the owner re-published for the forward it gave up.
|
||||||
|
var tk *Task
|
||||||
|
for _, p := range n1.state.PendingList() {
|
||||||
|
if !p.Release && p.Link.RemotePort == 100 && p.Local.Name == "web" {
|
||||||
|
tk = p
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if tk == nil {
|
||||||
|
t.Fatal("the owner did not re-publish the forward it released")
|
||||||
|
}
|
||||||
|
if got := len(n1.state.ForwardsOwnedBy("n1")); got != 2 {
|
||||||
|
t.Fatalf("n1 should now own 2, got %d", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The forward is unclaimed; the lowest node claims it. n2 is the lightest
|
||||||
|
// (Forwards=0 and the same mem/net), so it should take it. The task reaches
|
||||||
|
// n2 on the token — each engine holds its own state, so carry it over the way
|
||||||
|
// a real round does.
|
||||||
|
var carried State
|
||||||
|
cb, _ := json.Marshal(n1.state)
|
||||||
|
_ = json.Unmarshal(cb, &carried)
|
||||||
|
n2.AdoptState(carried)
|
||||||
|
n2.state.PendingTasks[tk.ID] = tk
|
||||||
|
|
||||||
|
if err := n2.runCommands(context.Background(), &Token{}); err != nil {
|
||||||
|
t.Fatalf("n2 runCommands: %v", err)
|
||||||
|
}
|
||||||
|
if claimed["n2"] == 0 {
|
||||||
|
t.Fatal("the idle node did not claim the released forward")
|
||||||
|
}
|
||||||
|
// And n1 no longer owns it.
|
||||||
|
for _, te := range n1.state.ForwardsOwnedBy("n1") {
|
||||||
|
if te.Link.RemotePort == 100 {
|
||||||
|
t.Error("n1 still owns the forward it released")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The balancer sees the release land and stops holding the release state.
|
||||||
|
if msg := b.Step(1100); !strings.Contains(msg, "gave up") {
|
||||||
|
t.Errorf("expected the release to be confirmed, got %q", msg)
|
||||||
|
}
|
||||||
|
if b.InFlight() {
|
||||||
|
t.Error("the release should be complete")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCooldownBoundsTheChurn: a release costs a live connection, so one node may
|
||||||
|
// not shed again until the cooldown elapses — even if it is still over its share.
|
||||||
|
func TestCooldownBoundsTheChurn(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 4, 0)
|
||||||
|
b := NewBalancer(n1, 600) // 10 minutes
|
||||||
|
b.SetMode(BalanceOn)
|
||||||
|
|
||||||
|
if msg := b.Step(1000); !strings.Contains(msg, "releasing") {
|
||||||
|
t.Fatalf("first release expected, got %q", msg)
|
||||||
|
}
|
||||||
|
n1.RemoveTopologyEntry("web", "frps1", 100) // land it
|
||||||
|
if msg := b.Step(1010); !strings.Contains(msg, "gave up") {
|
||||||
|
t.Fatalf("expected confirmation, got %q", msg)
|
||||||
|
}
|
||||||
|
// Still over its share (3 of 4 over 3 → share 2), but inside the cooldown.
|
||||||
|
if pick, _, blocked := b.assess(1020); pick == nil || blocked != "cooldown" {
|
||||||
|
t.Errorf("expected the second release to be blocked by cooldown, got blocked=%q", blocked)
|
||||||
|
}
|
||||||
|
if b.InFlight() {
|
||||||
|
t.Error("cooldown must not leave a release in flight")
|
||||||
|
}
|
||||||
|
// Past the cooldown it may act again.
|
||||||
|
if pick, _, blocked := b.assess(1000 + 601); pick == nil || blocked != "" {
|
||||||
|
t.Errorf("past the cooldown the release should be allowed, got blocked=%q", blocked)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReleaseDeadlineClearsAStuckConfirmation: if the release never lands (leader
|
||||||
|
// change, ring join), the node must not wedge forever.
|
||||||
|
func TestReleaseDeadlineClearsAStuckConfirmation(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceOn)
|
||||||
|
_ = b.Step(1000) // release published, entry NOT removed (nothing executed it)
|
||||||
|
if !b.InFlight() {
|
||||||
|
t.Fatal("precondition: a release should be in flight")
|
||||||
|
}
|
||||||
|
if msg := b.Step(1000 + releaseDeadlineSec + 1); msg != "" {
|
||||||
|
t.Errorf("a timed-out release should clear silently, got %q", msg)
|
||||||
|
}
|
||||||
|
if b.InFlight() {
|
||||||
|
t.Error("the release state should have been cleared after the deadline")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- read-only report ----
|
||||||
|
|
||||||
|
// TestReportDoesNotMutate — polling the endpoint must not start, delay or cancel
|
||||||
|
// a release. A read-only surface that perturbs what it reports is worse than none.
|
||||||
|
func TestReportDoesNotMutate(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 600)
|
||||||
|
b.SetMode(BalanceOn)
|
||||||
|
|
||||||
|
for i := 0; i < 5; i++ {
|
||||||
|
rep := b.Report()
|
||||||
|
if rep.SelfForwards != 3 {
|
||||||
|
t.Fatalf("report changed the node's forward count: %d", rep.SelfForwards)
|
||||||
|
}
|
||||||
|
if rep.Mode != BalanceOn {
|
||||||
|
t.Fatalf("report changed the mode: %q", rep.Mode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if b.InFlight() || b.lastReleaseAt != 0 {
|
||||||
|
t.Error("polling the report started a release")
|
||||||
|
}
|
||||||
|
if rep := b.Report(); rep.WouldRelease == nil {
|
||||||
|
t.Error("the report should still surface the pending decision")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReportExplainsWhyNothingHappens pins the field that answers the question
|
||||||
|
// the log cannot: an over-share node that is holding back must say why.
|
||||||
|
func TestReportExplainsWhyNothingHappens(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 4, 0)
|
||||||
|
b := NewBalancer(n1, 600)
|
||||||
|
b.SetMode(BalanceOn)
|
||||||
|
|
||||||
|
rep := b.Report()
|
||||||
|
if !rep.SelfOverShare {
|
||||||
|
t.Fatalf("4 of 4 forwards over 3 nodes should be over the share (share=%d)", rep.Share)
|
||||||
|
}
|
||||||
|
if rep.WouldRelease == nil {
|
||||||
|
t.Fatal("an over-share node must surface the forward it would give up")
|
||||||
|
}
|
||||||
|
if rep.BlockedBy != "" {
|
||||||
|
t.Errorf("nothing should block the first release, got %q", rep.BlockedBy)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inside the cooldown the same node reports the block. Report() reads the
|
||||||
|
// real clock, so set lastReleaseAt relative to now rather than to a fake one.
|
||||||
|
b.lastReleaseAt = time.Now().Unix() - 10
|
||||||
|
n1.RemoveTopologyEntry("web", "frps1", 100)
|
||||||
|
rep = b.Report()
|
||||||
|
if rep.BlockedBy != "cooldown" {
|
||||||
|
t.Errorf("after a release the node should report a cooldown block, got %q", rep.BlockedBy)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReportOnBalancedNodeIsQuiet: the healthy case must be a real answer, not an
|
||||||
|
// empty one — otherwise "balanced" and "balancing is switched off" look identical.
|
||||||
|
func TestReportOnBalancedNodeIsQuiet(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 1, 1)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceOn)
|
||||||
|
|
||||||
|
rep := b.Report()
|
||||||
|
if rep.WouldRelease != nil {
|
||||||
|
t.Errorf("a balanced node reported a release: %+v", rep.WouldRelease)
|
||||||
|
}
|
||||||
|
if rep.SelfOverShare {
|
||||||
|
t.Error("1 of 3 over 3 nodes is not over the share")
|
||||||
|
}
|
||||||
|
if rep.TotalForwards != 3 || rep.AliveNodes != 3 || rep.Share != 1 {
|
||||||
|
t.Errorf("report pool = total %d alive %d share %d, want 3/3/1",
|
||||||
|
rep.TotalForwards, rep.AliveNodes, rep.Share)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- mode switching (the cluster page's switch) ----
|
||||||
|
|
||||||
|
// TestStoredModeWinsOverEnv: flipping the switch must not silently revert on the
|
||||||
|
// next restart. The environment is only the bootstrap default; once an operator
|
||||||
|
// has chosen a mode it is persisted and wins.
|
||||||
|
func TestStoredModeWinsOverEnv(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode("dry-run") // the env default
|
||||||
|
|
||||||
|
var saved string
|
||||||
|
b.SetModePersistence(
|
||||||
|
func() (string, bool) { return "on", true },
|
||||||
|
func(m string) error { saved = m; return nil },
|
||||||
|
)
|
||||||
|
if b.Mode() != BalanceOn {
|
||||||
|
t.Fatalf("a stored mode must win over the environment default, got %q", b.Mode())
|
||||||
|
}
|
||||||
|
if b.ModeSource() != "stored" {
|
||||||
|
t.Errorf("ModeSource = %q, want stored", b.ModeSource())
|
||||||
|
}
|
||||||
|
_ = saved
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUnconfiguredFallsBackToEnv: with nothing stored, the environment default
|
||||||
|
// stands — otherwise every node would need a settings row to behave as documented.
|
||||||
|
func TestUnconfiguredFallsBackToEnv(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode("off")
|
||||||
|
b.SetModePersistence(func() (string, bool) { return "", false }, func(string) error { return nil })
|
||||||
|
if b.Mode() != BalanceOff {
|
||||||
|
t.Errorf("with nothing stored the env default must stand, got %q", b.Mode())
|
||||||
|
}
|
||||||
|
if b.ModeSource() != "stored" {
|
||||||
|
// persist is installed, so the node CAN be controlled
|
||||||
|
t.Logf("ModeSource=%q", b.ModeSource())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPersistBeforeApply: if the write fails, the mode must not change.
|
||||||
|
// Applying first would leave the node running a mode it cannot reproduce after a
|
||||||
|
// restart — the switch would lie.
|
||||||
|
func TestPersistBeforeApply(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceDryRun)
|
||||||
|
b.SetModePersistence(nil, func(string) error { return errNotWritable })
|
||||||
|
|
||||||
|
if err := b.SetModeFromUI(BalanceOn); err == nil {
|
||||||
|
t.Fatal("a failed save must surface an error")
|
||||||
|
}
|
||||||
|
if b.Mode() != BalanceDryRun {
|
||||||
|
t.Errorf("a failed save must leave the mode unchanged, got %q", b.Mode())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSetModeFromUIRejectsNonsense: the endpoint takes a string from the wire, so
|
||||||
|
// it must refuse anything it does not understand rather than guessing.
|
||||||
|
func TestSetModeFromUIRejectsNonsense(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceDryRun)
|
||||||
|
if err := b.SetModeFromUI("yes"); err == nil {
|
||||||
|
t.Fatal("an unknown mode must be rejected, not coerced")
|
||||||
|
}
|
||||||
|
if b.Mode() != BalanceDryRun {
|
||||||
|
t.Errorf("a rejected mode must not change anything, got %q", b.Mode())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSetModeFromUIPersistsAndApplies: the happy path, both effects.
|
||||||
|
func TestSetModeFromUIPersistsAndApplies(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
b.SetMode(BalanceDryRun)
|
||||||
|
var saved string
|
||||||
|
b.SetModePersistence(nil, func(m string) error { saved = m; return nil })
|
||||||
|
|
||||||
|
if err := b.SetModeFromUI(BalanceOn); err != nil {
|
||||||
|
t.Fatalf("SetModeFromUI: %v", err)
|
||||||
|
}
|
||||||
|
if saved != BalanceOn {
|
||||||
|
t.Errorf("saved %q, want on", saved)
|
||||||
|
}
|
||||||
|
if b.Mode() != BalanceOn {
|
||||||
|
t.Errorf("mode = %q, want on", b.Mode())
|
||||||
|
}
|
||||||
|
if !b.Report().CanControl {
|
||||||
|
t.Error("a node with persistence installed should report itself controllable")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReportCanControlOffWithoutPersistence: a node that cannot persist a choice
|
||||||
|
// must not advertise the switch (an embedder/test without a store).
|
||||||
|
func TestReportCanControlOffWithoutPersistence(t *testing.T) {
|
||||||
|
n1, _, _ := balanceRing(t, 3, 0)
|
||||||
|
b := NewBalancer(n1, 0)
|
||||||
|
if b.Report().CanControl {
|
||||||
|
t.Error("a balancer with no persistence must not report itself controllable")
|
||||||
|
}
|
||||||
|
}
|
||||||
91
internal/cluster/debug.go
Normal file
91
internal/cluster/debug.go
Normal file
@ -0,0 +1,91 @@
|
|||||||
|
package cluster
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"sync/atomic"
|
||||||
|
)
|
||||||
|
|
||||||
|
// logf is the package's logging seam: every cluster log line funnels through it
|
||||||
|
// so the debug gate in debug.go has a single place to hook.
|
||||||
|
func logf(format string, args ...any) { log.Printf(format, args...) }
|
||||||
|
|
||||||
|
// Token rotation is the ring's heartbeat: with a 2s round delay it fires
|
||||||
|
// continuously, and the default three log lines per round
|
||||||
|
// ("OnToken cycle=N" / "forward cycle=N to X" / "token-send ... -> 200")
|
||||||
|
// carry no information — no cycle number, address, timing or payload ever
|
||||||
|
// changes in the steady state. Measured on the live 3-node cluster that was
|
||||||
|
// ~510k lines/day on one node (3.5M lines in a week), which drowns every real
|
||||||
|
// event in the journal and fills the disk for a signal that is already
|
||||||
|
// available in structured form via GET /api/manager/cluster/ring (cycle,
|
||||||
|
// lastSync, roundDelayMs, node aliveness).
|
||||||
|
//
|
||||||
|
// So the steady-state lines are demoted to a debug level, off by default and
|
||||||
|
// enabled with W4F_DEBUG=token (or 1/all/true for every debug line). Failure
|
||||||
|
// paths are NOT demoted: a send error, a stale token, a timeout or a leader
|
||||||
|
// change is exactly what someone is grepping for, and losing those to a quiet
|
||||||
|
// default would be a bad trade.
|
||||||
|
const debugEnv = "W4F_DEBUG"
|
||||||
|
|
||||||
|
// debugToken logs a per-token-round heartbeat line. Suppressed unless
|
||||||
|
// W4F_DEBUG selects "token" (or a catch-all value).
|
||||||
|
func debugToken(format string, args ...any) {
|
||||||
|
if debugTokenOn.Load() {
|
||||||
|
logf(format, args...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// debugAll logs an ad-hoc diagnostic line. Suppressed unless W4F_DEBUG is set
|
||||||
|
// to a catch-all value (1/all/true/*).
|
||||||
|
func debugAll(format string, args ...any) {
|
||||||
|
if debugAllOn.Load() {
|
||||||
|
logf(format, args...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
debugTokenOn atomic.Bool
|
||||||
|
debugAllOn atomic.Bool
|
||||||
|
)
|
||||||
|
|
||||||
|
func init() { ReloadDebug() }
|
||||||
|
|
||||||
|
// ReloadDebug re-reads W4F_DEBUG. Called once at init so tests can flip it
|
||||||
|
// without restarting, and available at runtime for an operator who wants to
|
||||||
|
// watch the ring without a redeploy.
|
||||||
|
func ReloadDebug() {
|
||||||
|
v := os.Getenv(debugEnv)
|
||||||
|
switch normalized := normalizeDebugValue(v); normalized {
|
||||||
|
case "token":
|
||||||
|
debugTokenOn.Store(true)
|
||||||
|
debugAllOn.Store(false)
|
||||||
|
case "all":
|
||||||
|
debugTokenOn.Store(true)
|
||||||
|
debugAllOn.Store(true)
|
||||||
|
default:
|
||||||
|
debugTokenOn.Store(false)
|
||||||
|
debugAllOn.Store(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeDebugValue(v string) string {
|
||||||
|
// Compare case-insensitively without pulling in strings just for this.
|
||||||
|
out := make([]rune, 0, len(v))
|
||||||
|
for _, r := range v {
|
||||||
|
if r >= 'A' && r <= 'Z' {
|
||||||
|
r += 'a' - 'A'
|
||||||
|
}
|
||||||
|
out = append(out, r)
|
||||||
|
}
|
||||||
|
s := string(out)
|
||||||
|
switch s {
|
||||||
|
case "":
|
||||||
|
return ""
|
||||||
|
case "token", "tokens", "ring":
|
||||||
|
return "token"
|
||||||
|
}
|
||||||
|
// Any other non-empty value is a deliberate request for more output, so it
|
||||||
|
// is treated as a catch-all rather than silently muting the operator who
|
||||||
|
// set it. "0" lands here too: it was asked for, so honour it.
|
||||||
|
return "all"
|
||||||
|
}
|
||||||
108
internal/cluster/debug_test.go
Normal file
108
internal/cluster/debug_test.go
Normal file
@ -0,0 +1,108 @@
|
|||||||
|
package cluster
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// captureLog redirects the standard logger into a buffer for the duration of
|
||||||
|
// fn and returns what was written.
|
||||||
|
func captureLog(t *testing.T, fn func()) string {
|
||||||
|
t.Helper()
|
||||||
|
var buf bytes.Buffer
|
||||||
|
orig := log.Writer()
|
||||||
|
origFlags := log.Flags()
|
||||||
|
log.SetOutput(&buf)
|
||||||
|
log.SetFlags(0)
|
||||||
|
defer func() {
|
||||||
|
log.SetOutput(orig)
|
||||||
|
log.SetFlags(origFlags)
|
||||||
|
}()
|
||||||
|
fn()
|
||||||
|
return buf.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebugTokenSuppressedByDefault(t *testing.T) {
|
||||||
|
os.Unsetenv(debugEnv)
|
||||||
|
ReloadDebug()
|
||||||
|
|
||||||
|
out := captureLog(t, func() {
|
||||||
|
debugToken("ring[%s] OnToken cycle=%d", "node:7500", 42)
|
||||||
|
debugToken("ring[%s] forward cycle=%d to %s", "node:7500", 42, "next:7500")
|
||||||
|
debugToken("token-send %s: size=%dB elapsed=%v -> %d", "next:7500", 5605, "170ms", 200)
|
||||||
|
})
|
||||||
|
if out != "" {
|
||||||
|
t.Fatalf("steady-state token lines logged with W4F_DEBUG unset: %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebugTokenEnabledByEnv(t *testing.T) {
|
||||||
|
for _, v := range []string{"token", "TOKEN", "tokens", "ring", "1", "true", "yes", "all", "*", "yes-please"} {
|
||||||
|
t.Run(v, func(t *testing.T) {
|
||||||
|
t.Setenv(debugEnv, v)
|
||||||
|
ReloadDebug()
|
||||||
|
if !debugTokenOn.Load() {
|
||||||
|
t.Fatalf("W4F_DEBUG=%q should enable the token heartbeat", v)
|
||||||
|
}
|
||||||
|
out := captureLog(t, func() {
|
||||||
|
debugToken("ring[%s] OnToken cycle=%d", "node:7500", 7)
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "OnToken cycle=7") {
|
||||||
|
t.Fatalf("W4F_DEBUG=%q: expected the line to be logged, got %q", v, out)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The whole point of the gate is to shrink the journal, so the failure paths
|
||||||
|
// must stay loud without any env var — losing a send error to a quiet default
|
||||||
|
// would be a bad trade.
|
||||||
|
func TestFailurePathsStayLoud(t *testing.T) {
|
||||||
|
os.Unsetenv(debugEnv)
|
||||||
|
ReloadDebug()
|
||||||
|
|
||||||
|
out := captureLog(t, func() {
|
||||||
|
logf("token-send %s: size=%dB elapsed=%v err=%v", "down:7500", 10, "5ms", "connection refused")
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "connection refused") {
|
||||||
|
t.Fatalf("send errors must never be gated: got %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebugAllOffForTokenOnly(t *testing.T) {
|
||||||
|
t.Setenv(debugEnv, "token")
|
||||||
|
ReloadDebug()
|
||||||
|
if !debugTokenOn.Load() {
|
||||||
|
t.Fatal("token level should enable debugToken")
|
||||||
|
}
|
||||||
|
if debugAllOn.Load() {
|
||||||
|
t.Fatal("token level must not enable the catch-all debugAll")
|
||||||
|
}
|
||||||
|
out := captureLog(t, func() { debugAll("scratch diagnostic") })
|
||||||
|
if out != "" {
|
||||||
|
t.Fatalf("debugAll should stay off at token level, got %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeDebugValue(t *testing.T) {
|
||||||
|
cases := map[string]string{
|
||||||
|
"": "",
|
||||||
|
"token": "token",
|
||||||
|
"TOKEN": "token",
|
||||||
|
"Ring": "token",
|
||||||
|
"1": "all",
|
||||||
|
"true": "all",
|
||||||
|
"ALL": "all",
|
||||||
|
"*": "all",
|
||||||
|
"anything": "all", // unrecognised but deliberate → don't silence it
|
||||||
|
"0": "all", // 0 is still a deliberate request for output
|
||||||
|
}
|
||||||
|
for in, want := range cases {
|
||||||
|
if got := normalizeDebugValue(in); got != want {
|
||||||
|
t.Errorf("normalizeDebugValue(%q) = %q, want %q", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -62,6 +62,25 @@ type Task struct {
|
|||||||
// owning node cancels it (stop worker, drop from topology). Reuses the
|
// owning node cancels it (stop worker, drop from topology). Reuses the
|
||||||
// same publish channel as creation (round-1 inject, round-2 apply).
|
// same publish channel as creation (round-1 inject, round-2 apply).
|
||||||
Revoke bool `json:"revoke,omitempty"`
|
Revoke bool `json:"revoke,omitempty"`
|
||||||
|
// Restart marks a RE-ENABLE task: the forward is already claimed and its
|
||||||
|
// topology entry still exists (a stop keeps the entry as the flag's
|
||||||
|
// carrier), so the creation path would dedupe the submission and the
|
||||||
|
// duplicate-claim guard would discard it. The owner applies this one
|
||||||
|
// unconditionally: re-mark the entry enabled and (re)spawn the worker.
|
||||||
|
Restart bool `json:"restart,omitempty"`
|
||||||
|
// Release marks a REBALANCING task: the OWNER of the forward gives it up so
|
||||||
|
// the ring can place it elsewhere. Only the owning node may act on it — it is
|
||||||
|
// that node's own load which has to fall, and only it can make that true (its
|
||||||
|
// topology entry is local bookkeeping no peer may write).
|
||||||
|
//
|
||||||
|
// It is deliberately NOT the same thing as a user stop (Revoke). A stop keeps
|
||||||
|
// the topology entry as the carrier of the disabled flag and flags the link
|
||||||
|
// stopped; a release REMOVES the entry and leaves the link enabled, because
|
||||||
|
// the forward is about to run somewhere else. Marking it stopped instead would
|
||||||
|
// make the re-published creation task look like a duplicate of an existing
|
||||||
|
// forward, and every node would refuse to claim it — stranding the forward on
|
||||||
|
// the node that just gave it up.
|
||||||
|
Release bool `json:"release,omitempty"`
|
||||||
// RemoveNode: node ID to remove from the ring; the node self-removes when
|
// RemoveNode: node ID to remove from the ring; the node self-removes when
|
||||||
// the command reaches it via the token.
|
// the command reaches it via the token.
|
||||||
RemoveNode string `json:"removeNode,omitempty"`
|
RemoveNode string `json:"removeNode,omitempty"`
|
||||||
@ -333,10 +352,15 @@ func (s *State) AddTopology(t *Task, ownerID string) *TopoEntry {
|
|||||||
if s.Topology == nil {
|
if s.Topology == nil {
|
||||||
s.Topology = map[string]*TopoEntry{}
|
s.Topology = map[string]*TopoEntry{}
|
||||||
}
|
}
|
||||||
|
// Active mirrors the task's disabled flag rather than being unconditionally
|
||||||
|
// true. A claim can legitimately carry a stopped forward (e.g. a node
|
||||||
|
// re-claiming from a departed peer), and forcing Active=true there would
|
||||||
|
// resurrect it: OfflineReassign only re-queues Active entries, and every
|
||||||
|
// reader that filters on Active would treat it as running again.
|
||||||
e := &TopoEntry{
|
e := &TopoEntry{
|
||||||
TaskID: t.ID, OwnerID: ownerID,
|
TaskID: t.ID, OwnerID: ownerID,
|
||||||
Local: t.Local, Remote: t.Remote, Link: t.Link,
|
Local: t.Local, Remote: t.Remote, Link: t.Link,
|
||||||
Active: true,
|
Active: !t.Link.Disabled,
|
||||||
}
|
}
|
||||||
s.Topology[t.ID] = e
|
s.Topology[t.ID] = e
|
||||||
return e
|
return e
|
||||||
@ -408,6 +432,39 @@ func (s *State) ForwardsOwnedBy(owner string) []*TopoEntry {
|
|||||||
// (local, remote, port) triple. Returns true if found. The updated entry
|
// (local, remote, port) triple. Returns true if found. The updated entry
|
||||||
// propagates to all nodes via the next token cycle — group changes sync
|
// propagates to all nodes via the next token cycle — group changes sync
|
||||||
// through the ring without a dedicated command.
|
// through the ring without a dedicated command.
|
||||||
|
// UpdateTopologyDisabled sets the disabled flag on the topology entry matching
|
||||||
|
// the forward's natural key. Mirror of UpdateTopologyGroup: it makes a locally
|
||||||
|
// decided stop/start part of the topology so the next token cycle carries it to
|
||||||
|
// every other member (the alternative — a one-shot revoke task addressed at the
|
||||||
|
// owner — only converges if that owner happens to be online right then).
|
||||||
|
//
|
||||||
|
// Returns false when the forward is not in the topology, which is not an error:
|
||||||
|
// a stopped forward may legitimately have no entry yet.
|
||||||
|
func (s *State) UpdateTopologyDisabled(local, remote string, port int, disabled bool) bool {
|
||||||
|
for _, e := range s.Topology {
|
||||||
|
if e.Local.Name == local && e.Remote.Name == remote && e.Link.RemotePort == port {
|
||||||
|
e.Link.Disabled = disabled
|
||||||
|
// Keep Active consistent with the flag so readers that key off it
|
||||||
|
// (status page, load accounting) agree with Link.Disabled.
|
||||||
|
e.Active = !disabled
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// TopologyDisabled returns the disabled flag the cluster currently holds for a
|
||||||
|
// forward, and whether such an entry exists. Used by the adoption path to learn
|
||||||
|
// a peer's decision into the local store.
|
||||||
|
func (s *State) TopologyDisabled(local, remote string, port int) (bool, bool) {
|
||||||
|
for _, e := range s.Topology {
|
||||||
|
if e.Local.Name == local && e.Remote.Name == remote && e.Link.RemotePort == port {
|
||||||
|
return e.Link.Disabled, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false, false
|
||||||
|
}
|
||||||
|
|
||||||
func (s *State) UpdateTopologyGroup(local, remote string, port int, group string) bool {
|
func (s *State) UpdateTopologyGroup(local, remote string, port int, group string) bool {
|
||||||
for _, e := range s.Topology {
|
for _, e := range s.Topology {
|
||||||
if e.Local.Name == local && e.Remote.Name == remote && e.Link.RemotePort == port {
|
if e.Local.Name == local && e.Remote.Name == remote && e.Link.RemotePort == port {
|
||||||
|
|||||||
505
internal/cluster/ring_balance.go
Normal file
505
internal/cluster/ring_balance.go
Normal file
@ -0,0 +1,505 @@
|
|||||||
|
package cluster
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"webui4frpc/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The rebalance rule, deliberately simple and self-limiting:
|
||||||
|
//
|
||||||
|
// release one forward when Forwards >= minForwardsToShare AND Forwards > share
|
||||||
|
//
|
||||||
|
// where share = ceil(totalForwards / aliveNodes) — the even split.
|
||||||
|
//
|
||||||
|
// Every node runs this check about its OWN forwards. That is the whole point:
|
||||||
|
// a node's load only actually falls when its own topology entry goes away, so
|
||||||
|
// only the owner can produce the state change that makes the ring hand the
|
||||||
|
// forward to somebody else. A coordinator on another machine can see the
|
||||||
|
// imbalance but cannot make it true; it would have to reach in and mutate the
|
||||||
|
// owner's bookkeeping, which is exactly the sort of cross-node write this ring
|
||||||
|
// avoids everywhere else (revoke and restart are owner-directed for the same
|
||||||
|
// reason).
|
||||||
|
//
|
||||||
|
// The rule is self-limiting in both directions, so it needs no upper bound on
|
||||||
|
// how many forwards a node may shed:
|
||||||
|
//
|
||||||
|
// - It stops at the average. 3 forwards over 3 nodes: the owner holds 2 > 1,
|
||||||
|
// releases one, now holds 1 which is not > 1, and stops. Without the "> share"
|
||||||
|
// bound an owner would shed EVERY forward it has.
|
||||||
|
// - It cannot empty a node. A node holding fewer than minForwardsToShare never
|
||||||
|
// releases, so there is always something left to serve.
|
||||||
|
//
|
||||||
|
// A release costs a live connection: stopping the worker drops whatever was
|
||||||
|
// flowing through the forward, and the new owner re-dials the local service.
|
||||||
|
// Because the rule above already requires a real imbalance, pacing is handled by
|
||||||
|
// a per-node cooldown (cooldownSec) rather than a "how long has this lasted"
|
||||||
|
// window — there is no transient to smooth over, only a churn rate to bound.
|
||||||
|
const (
|
||||||
|
// minForwardsToShare: never release below this many owned forwards.
|
||||||
|
minForwardsToShare = 2
|
||||||
|
// balanceCooldownSec: minimum gap between two releases by one node. One
|
||||||
|
// release per 10 minutes per node is a trickle; it is what stops a cluster
|
||||||
|
// whose loads oscillate around the split point from restarting workers in a
|
||||||
|
// loop.
|
||||||
|
balanceCooldownSec = 600
|
||||||
|
// releaseDeadlineSec: how long to wait for a published release to be
|
||||||
|
// executed by its owner (us) before giving up on confirming it. A token
|
||||||
|
// round is ~2s, so this is generous; it exists so a release lost to a
|
||||||
|
// leader change or a joined ring cannot wedge the node forever.
|
||||||
|
releaseDeadlineSec = 120
|
||||||
|
)
|
||||||
|
|
||||||
|
// Balance modes.
|
||||||
|
const (
|
||||||
|
BalanceOff = "off"
|
||||||
|
BalanceDryRun = "dry-run"
|
||||||
|
BalanceOn = "on"
|
||||||
|
)
|
||||||
|
|
||||||
|
// pool is the replicated view the rule works from.
|
||||||
|
type pool struct {
|
||||||
|
alive int // nodes that can take a forward
|
||||||
|
mine int // forwards THIS node owns (live count, not the replicated one)
|
||||||
|
total int // forwards across the ring
|
||||||
|
share int // ceil(total/alive) — the even split
|
||||||
|
}
|
||||||
|
|
||||||
|
// releaseState is one release this node has published and is waiting to see
|
||||||
|
// leave its own topology.
|
||||||
|
type releaseState struct {
|
||||||
|
local, remote string
|
||||||
|
port int
|
||||||
|
startedAt int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Balancer lets each node shed its own share of the cluster's forwards.
|
||||||
|
//
|
||||||
|
// It publishes only ONE kind of thing: a release of a forward it owns. There is
|
||||||
|
// no targeted hand-off and no coordinator, because "drop it and let the ring's
|
||||||
|
// normal lowest-load claim pick it up" is already the ring's only mechanism for
|
||||||
|
// moving a forward — and it is the mechanism that, unlike a coordinated move,
|
||||||
|
// cannot be undone by the node that gave the forward up (the claim path runs
|
||||||
|
// after the node's load has already fallen).
|
||||||
|
type Balancer struct {
|
||||||
|
engine *Engine
|
||||||
|
mode string
|
||||||
|
|
||||||
|
// cooldownSec is the minimum gap between two releases by this node.
|
||||||
|
cooldownSec int64
|
||||||
|
// releasing is the forward this node gave up and is waiting to see leave.
|
||||||
|
// While it is set, no second release is started: one at a time keeps the
|
||||||
|
// connection churn bounded and the log readable.
|
||||||
|
releasing *releaseState
|
||||||
|
// lastReleaseAt is when the previous release was published (0 = never).
|
||||||
|
lastReleaseAt int64
|
||||||
|
// reported remembers the last dry-run candidate so the observe-only mode
|
||||||
|
// logs a change of mind rather than the same line every tick.
|
||||||
|
reported string
|
||||||
|
// persist saves a mode change so it survives a restart (nil = env-only).
|
||||||
|
persist func(string) error
|
||||||
|
// savedMode is the last mode this node stored, so the report can say whether
|
||||||
|
// the active mode came from the environment or from an operator's choice.
|
||||||
|
savedMode string
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewBalancer builds a balancer. It starts in observe-only mode: releasing a
|
||||||
|
// forward drops a live connection, so acting on the ring is opt-in.
|
||||||
|
func NewBalancer(e *Engine, cooldownSec int64) *Balancer {
|
||||||
|
b := &Balancer{engine: e, mode: BalanceDryRun, cooldownSec: cooldownSec}
|
||||||
|
if b.cooldownSec <= 0 {
|
||||||
|
b.cooldownSec = balanceCooldownSec
|
||||||
|
}
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetMode selects off / dry-run / on. An unrecognised value is treated as
|
||||||
|
// dry-run — a typo must not be able to turn releasing on.
|
||||||
|
//
|
||||||
|
// Only a change is logged, and the reason is recorded, because SetMode is now
|
||||||
|
// called both at boot (from the environment) and at runtime (from the cluster
|
||||||
|
// page): a one-line "why did this just start moving my forwards" answer is the
|
||||||
|
// difference between a switch an operator dares to use and one they don't.
|
||||||
|
func (b *Balancer) SetMode(m string) {
|
||||||
|
if m != BalanceOff && m != BalanceDryRun && m != BalanceOn {
|
||||||
|
if m != "" {
|
||||||
|
log.Printf("ring[%s] unknown balance mode %q; using %s", b.engine.ID, m, BalanceDryRun)
|
||||||
|
}
|
||||||
|
m = BalanceDryRun
|
||||||
|
}
|
||||||
|
if b.mode == m {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
b.mode = m
|
||||||
|
b.savedMode = m
|
||||||
|
log.Printf("ring[%s] load balancing set to %s", b.engine.ID, strings.ToUpper(m))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mode returns the active mode.
|
||||||
|
func (b *Balancer) Mode() string { return b.mode }
|
||||||
|
|
||||||
|
// ModeSource reports where the active mode came from, for the report.
|
||||||
|
//
|
||||||
|
// "stored" means an operator chose it on the cluster page and it will survive a
|
||||||
|
// restart; "env" means nothing was ever chosen here and the environment default
|
||||||
|
// (or the built-in dry-run) applies. It is NOT "can this node save a choice" —
|
||||||
|
// that is CanControl, and conflating the two made every node with a store claim
|
||||||
|
// an operator had set it.
|
||||||
|
func (b *Balancer) ModeSource() string {
|
||||||
|
if b.savedMode != "" {
|
||||||
|
return "stored"
|
||||||
|
}
|
||||||
|
return "env"
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetModePersistence installs the callback that saves a mode change so it
|
||||||
|
// survives a restart, and applies any previously saved value.
|
||||||
|
//
|
||||||
|
// The environment is the BOOTSTRAP default; once an operator flips the switch on
|
||||||
|
// the cluster page that stored value wins, otherwise the page would appear to
|
||||||
|
// work and then silently revert on the next restart.
|
||||||
|
func (b *Balancer) SetModePersistence(load func() (string, bool), save func(string) error) {
|
||||||
|
b.persist = save
|
||||||
|
if load == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if m, ok := load(); ok && m != "" {
|
||||||
|
switch m {
|
||||||
|
case BalanceOff, BalanceDryRun, BalanceOn:
|
||||||
|
if b.mode != m {
|
||||||
|
log.Printf("ring[%s] load balancing mode %s loaded from settings", b.engine.ID, m)
|
||||||
|
}
|
||||||
|
b.mode = m
|
||||||
|
b.savedMode = m
|
||||||
|
default:
|
||||||
|
log.Printf("ring[%s] ignoring unknown stored balance mode %q", b.engine.ID, m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetModeFromUI changes the mode at runtime and persists it.
|
||||||
|
//
|
||||||
|
// The order matters: persist FIRST, then apply. If the write fails nothing
|
||||||
|
// changes, so the switch cannot report success while behaving differently after
|
||||||
|
// the next restart.
|
||||||
|
func (b *Balancer) SetModeFromUI(m string) error {
|
||||||
|
if m != BalanceOff && m != BalanceDryRun && m != BalanceOn {
|
||||||
|
return fmt.Errorf("未知模式 %q", m)
|
||||||
|
}
|
||||||
|
if b.persist != nil {
|
||||||
|
if err := b.persist(m); err != nil {
|
||||||
|
return fmt.Errorf("保存负载均衡模式失败: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
b.SetMode(m)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetCooldown overrides the per-node release cooldown.
|
||||||
|
func (b *Balancer) SetCooldown(sec int64) {
|
||||||
|
if sec > 0 {
|
||||||
|
b.cooldownSec = sec
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// InFlight reports whether a release is awaiting confirmation.
|
||||||
|
func (b *Balancer) InFlight() bool { return b.releasing != nil }
|
||||||
|
|
||||||
|
// pool computes the ring-wide view this node derives for itself.
|
||||||
|
//
|
||||||
|
// Our own count is taken LIVE from the topology rather than from the replicated
|
||||||
|
// Nodes[].Load, because Nodes[].Load is only refreshed when a token arrives and
|
||||||
|
// the decision must see the effect of the release immediately — otherwise a node
|
||||||
|
// would keep believing it is over its share and shed another forward each round.
|
||||||
|
func (b *Balancer) pool() pool {
|
||||||
|
st := &b.engine.state
|
||||||
|
p := pool{mine: len(st.ForwardsOwnedBy(b.engine.ID))}
|
||||||
|
for i := range st.Nodes {
|
||||||
|
n := &st.Nodes[i]
|
||||||
|
if !n.Alive {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
p.alive++
|
||||||
|
if n.ID == b.engine.ID {
|
||||||
|
p.total += p.mine
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
p.total += n.Load.Forwards
|
||||||
|
}
|
||||||
|
if p.alive > 0 {
|
||||||
|
p.share = (p.total + p.alive - 1) / p.alive
|
||||||
|
}
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// assess answers "should this node give one forward up, and which one?" together
|
||||||
|
// with the facts behind the answer, so the read-only report can explain a node
|
||||||
|
// that is over its share but not acting. It never mutates anything.
|
||||||
|
//
|
||||||
|
// blocked is a short reason: "single-node", "below-min", "balanced",
|
||||||
|
// "nothing-active", "cooldown" or "in-flight".
|
||||||
|
func (b *Balancer) assess(now int64) (pick *TopoEntry, p pool, blocked string) {
|
||||||
|
p = b.pool()
|
||||||
|
if !b.engine.IsMember() {
|
||||||
|
return nil, p, "single-node"
|
||||||
|
}
|
||||||
|
if p.mine < minForwardsToShare {
|
||||||
|
return nil, p, "below-min"
|
||||||
|
}
|
||||||
|
if p.mine <= p.share {
|
||||||
|
return nil, p, "balanced"
|
||||||
|
}
|
||||||
|
pick = releaseCandidate(&b.engine.state, b.engine.ID)
|
||||||
|
if pick == nil {
|
||||||
|
return nil, p, "nothing-active"
|
||||||
|
}
|
||||||
|
// The guards below do not change the verdict ("this node is over its
|
||||||
|
// share"), only whether now is the moment; the report shows both.
|
||||||
|
if b.releasing != nil {
|
||||||
|
return pick, p, "in-flight"
|
||||||
|
}
|
||||||
|
if b.lastReleaseAt > 0 && now-b.lastReleaseAt < b.cooldownSec {
|
||||||
|
return pick, p, "cooldown"
|
||||||
|
}
|
||||||
|
return pick, p, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// releaseCandidate picks which of this node's forwards to give up.
|
||||||
|
//
|
||||||
|
// Deterministic (lowest remote port, then task id) so the choice is explainable
|
||||||
|
// and repeatable: a node that re-evaluates the same state picks the same
|
||||||
|
// forward. Stopped forwards are skipped — their entry is the carrier of a user's
|
||||||
|
// stop decision, and releasing one would hand a STOPPED forward to another node
|
||||||
|
// as if it were work to run.
|
||||||
|
func releaseCandidate(s *State, owner string) *TopoEntry {
|
||||||
|
var mine []*TopoEntry
|
||||||
|
for _, te := range s.TopologyList() {
|
||||||
|
if te.OwnerID != owner || !te.Active || te.Link.Disabled {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
mine = append(mine, te)
|
||||||
|
}
|
||||||
|
if len(mine) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
sort.Slice(mine, func(i, j int) bool {
|
||||||
|
if mine[i].Link.RemotePort != mine[j].Link.RemotePort {
|
||||||
|
return mine[i].Link.RemotePort < mine[j].Link.RemotePort
|
||||||
|
}
|
||||||
|
return mine[i].TaskID < mine[j].TaskID
|
||||||
|
})
|
||||||
|
return mine[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
// landed reports whether the forward this node released has actually left it.
|
||||||
|
//
|
||||||
|
// The test is "no longer owned by me", NOT "no entry exists": the ring's normal
|
||||||
|
// claim picks the forward up within a round or two, so an existence test would
|
||||||
|
// see the new owner's entry and wait forever for a state that has already
|
||||||
|
// happened.
|
||||||
|
func (b *Balancer) landed() bool {
|
||||||
|
m := b.releasing
|
||||||
|
if m == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
owner := b.engine.state.TopologyOwner(&Task{
|
||||||
|
Local: store.Local{Name: m.local},
|
||||||
|
Remote: store.Remote{Name: m.remote},
|
||||||
|
Link: store.Link{RemotePort: m.port},
|
||||||
|
})
|
||||||
|
return owner != b.engine.ID
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step advances the balancer by one cycle and returns a human-readable
|
||||||
|
// description when it did (or, in dry-run, would have done) something.
|
||||||
|
func (b *Balancer) Step(now int64) string {
|
||||||
|
if b.releasing != nil {
|
||||||
|
if b.landed() {
|
||||||
|
done := b.releasing
|
||||||
|
b.releasing = nil
|
||||||
|
return "gave up " + done.local + ":" + strconv.Itoa(done.port) + " for rebalancing"
|
||||||
|
}
|
||||||
|
if now-b.releasing.startedAt > releaseDeadlineSec {
|
||||||
|
log.Printf("ring[%s] release of %s:%d not confirmed within %ds; clearing (forward is still ours)",
|
||||||
|
b.engine.ID, b.releasing.local, b.releasing.port, releaseDeadlineSec)
|
||||||
|
b.releasing = nil
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
pick, p, blocked := b.assess(now)
|
||||||
|
if pick == nil {
|
||||||
|
b.reported = ""
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
desc := pick.Local.Name + ":" + strconv.Itoa(pick.Link.RemotePort) +
|
||||||
|
" (" + strconv.Itoa(p.mine) + " owned > " + strconv.Itoa(p.share) + " share of " +
|
||||||
|
strconv.Itoa(p.total) + " over " + strconv.Itoa(p.alive) + " nodes)"
|
||||||
|
|
||||||
|
switch b.mode {
|
||||||
|
case BalanceOff:
|
||||||
|
return ""
|
||||||
|
case BalanceDryRun:
|
||||||
|
// Log a change of mind, not the same line every tick.
|
||||||
|
if b.reported == desc {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
b.reported = desc
|
||||||
|
if blocked != "" {
|
||||||
|
return "would release " + desc + " [blocked: " + blocked + "]"
|
||||||
|
}
|
||||||
|
return "would release " + desc
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mode is ON. Blocked reasons still stop us.
|
||||||
|
if blocked != "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
t := b.engine.ReleaseTask(pick.Local, pick.Remote, pick.Link)
|
||||||
|
if t == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
b.releasing = &releaseState{
|
||||||
|
local: pick.Local.Name, remote: pick.Remote.Name,
|
||||||
|
port: pick.Link.RemotePort, startedAt: now,
|
||||||
|
}
|
||||||
|
b.lastReleaseAt = now
|
||||||
|
b.reported = ""
|
||||||
|
return "releasing " + desc
|
||||||
|
}
|
||||||
|
|
||||||
|
// Watch drives Step on a ticker until ctx is cancelled.
|
||||||
|
//
|
||||||
|
// It runs on EVERY node, not just the leader. Balancing here is not a cluster
|
||||||
|
// decision being delegated to one node — it is each node applying the same
|
||||||
|
// replicated rule to its own forwards. Two nodes never contend for the same
|
||||||
|
// forward because a node can only release what it owns.
|
||||||
|
//
|
||||||
|
// The first check happens after one interval rather than immediately: a node
|
||||||
|
// that has just booted adopts the ring's topology and its own load in the first
|
||||||
|
// token round, and deciding before that would act on a half-known pool.
|
||||||
|
func (b *Balancer) Watch(ctx context.Context, interval time.Duration) {
|
||||||
|
tick := time.NewTicker(interval)
|
||||||
|
defer tick.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-tick.C:
|
||||||
|
}
|
||||||
|
if msg := b.Step(time.Now().Unix()); msg != "" {
|
||||||
|
log.Printf("ring[%s] balance: %s", b.engine.ID, msg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- read-only report for the API/UI ----
|
||||||
|
|
||||||
|
// BalanceReport is what this node thinks about balancing right now.
|
||||||
|
//
|
||||||
|
// It is per-node by nature: every member has its own answer, and the answer that
|
||||||
|
// matters is the one on the node doing (or refusing to do) the releasing. The
|
||||||
|
// peer list is included because "am I over my share?" is only meaningful next to
|
||||||
|
// what everyone else holds.
|
||||||
|
type BalanceReport struct {
|
||||||
|
Mode string `json:"mode"` // off | dry-run | on
|
||||||
|
ModeSource string `json:"modeSource"` // env | stored
|
||||||
|
Enabled bool `json:"enabled"` // mode != off
|
||||||
|
DryRun bool `json:"dryRun"` // mode != on
|
||||||
|
// CanControl: this node offers the mode switch. False for a read-only
|
||||||
|
// principal, so the server does not advertise a control the UI gates anyway.
|
||||||
|
CanControl bool `json:"canControl"`
|
||||||
|
IsMember bool `json:"isMember"`
|
||||||
|
Self string `json:"self"`
|
||||||
|
|
||||||
|
SelfForwards int `json:"selfForwards"`
|
||||||
|
Share int `json:"share"`
|
||||||
|
TotalForwards int `json:"totalForwards"`
|
||||||
|
AliveNodes int `json:"aliveNodes"`
|
||||||
|
SelfOverShare bool `json:"selfOverShare"`
|
||||||
|
|
||||||
|
// WouldRelease is what this node would give up (present whenever it is over
|
||||||
|
// its share, including while blocked — see BlockedBy).
|
||||||
|
WouldRelease *BalanceReleasePlan `json:"wouldRelease,omitempty"`
|
||||||
|
// Releasing is the release this node has published and is awaiting.
|
||||||
|
Releasing *BalanceReleasePlan `json:"releasing,omitempty"`
|
||||||
|
// BlockedBy explains why an over-share node is not acting yet.
|
||||||
|
BlockedBy string `json:"blockedBy,omitempty"`
|
||||||
|
|
||||||
|
CooldownSec int64 `json:"cooldownSec"`
|
||||||
|
LastReleaseAt int64 `json:"lastReleaseAt,omitempty"`
|
||||||
|
|
||||||
|
// Peers: every member's replicated load, for context.
|
||||||
|
Nodes []BalanceNodeReport `json:"nodes"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type BalanceNodeReport struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Alive bool `json:"alive"`
|
||||||
|
IsLeader bool `json:"isLeader"`
|
||||||
|
Forwards int `json:"forwards"`
|
||||||
|
MemPct float64 `json:"memPct"`
|
||||||
|
NetPct float64 `json:"netPct"`
|
||||||
|
Score float64 `json:"score"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type BalanceReleasePlan struct {
|
||||||
|
Local string `json:"local"`
|
||||||
|
Remote string `json:"remote"`
|
||||||
|
RemotePort int `json:"remotePort"`
|
||||||
|
Owner string `json:"owner"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Report renders the current view. It never mutates anything — in particular it
|
||||||
|
// does NOT touch lastReleaseAt or the release state, so an operator polling this
|
||||||
|
// cannot start, delay or cancel a release. (A read-only endpoint that perturbs
|
||||||
|
// the behaviour it reports is worse than no endpoint.)
|
||||||
|
func (b *Balancer) Report() BalanceReport {
|
||||||
|
now := time.Now().Unix()
|
||||||
|
rep := BalanceReport{
|
||||||
|
Mode: b.mode,
|
||||||
|
ModeSource: b.ModeSource(),
|
||||||
|
Enabled: b.mode != BalanceOff,
|
||||||
|
DryRun: b.mode != BalanceOn,
|
||||||
|
CanControl: b.persist != nil,
|
||||||
|
IsMember: b.engine.IsMember(),
|
||||||
|
Self: b.engine.ID,
|
||||||
|
CooldownSec: b.cooldownSec,
|
||||||
|
}
|
||||||
|
p := b.pool()
|
||||||
|
rep.SelfForwards, rep.Share = p.mine, p.share
|
||||||
|
rep.TotalForwards, rep.AliveNodes = p.total, p.alive
|
||||||
|
rep.SelfOverShare = p.mine > p.share
|
||||||
|
|
||||||
|
if pick, _, blocked := b.assess(now); pick != nil {
|
||||||
|
rep.WouldRelease = &BalanceReleasePlan{
|
||||||
|
Local: pick.Local.Name, Remote: pick.Remote.Name,
|
||||||
|
RemotePort: pick.Link.RemotePort, Owner: pick.OwnerID,
|
||||||
|
}
|
||||||
|
rep.BlockedBy = blocked
|
||||||
|
}
|
||||||
|
if m := b.releasing; m != nil {
|
||||||
|
rep.Releasing = &BalanceReleasePlan{
|
||||||
|
Local: m.local, Remote: m.remote, RemotePort: m.port, Owner: b.engine.ID,
|
||||||
|
}
|
||||||
|
rep.BlockedBy = "in-flight"
|
||||||
|
}
|
||||||
|
if b.lastReleaseAt > 0 {
|
||||||
|
rep.LastReleaseAt = b.lastReleaseAt
|
||||||
|
}
|
||||||
|
for i := range b.engine.state.Nodes {
|
||||||
|
n := &b.engine.state.Nodes[i]
|
||||||
|
rep.Nodes = append(rep.Nodes, BalanceNodeReport{
|
||||||
|
ID: n.ID, Alive: n.Alive, IsLeader: n.IsLeader,
|
||||||
|
Forwards: n.Load.Forwards, MemPct: n.Load.MemPct, NetPct: n.Load.NetPct,
|
||||||
|
Score: n.Load.Score(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return rep
|
||||||
|
}
|
||||||
195
internal/cluster/ring_disabled_test.go
Normal file
195
internal/cluster/ring_disabled_test.go
Normal file
@ -0,0 +1,195 @@
|
|||||||
|
package cluster
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"webui4frpc/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// staleTokenWith copies the engine's current state but rewrites the given
|
||||||
|
// forward's disabled flag, standing in for a token a neighbour captured before
|
||||||
|
// this node's decision.
|
||||||
|
func staleTokenWith(eng *Engine, local, remote string, port int, disabled bool) State {
|
||||||
|
s := eng.state
|
||||||
|
out := State{
|
||||||
|
LeaderID: s.LeaderID,
|
||||||
|
Nodes: s.Nodes,
|
||||||
|
Cycle: s.Cycle,
|
||||||
|
Topology: map[string]*TopoEntry{},
|
||||||
|
}
|
||||||
|
for id, e := range s.Topology {
|
||||||
|
cp := *e
|
||||||
|
if cp.Local.Name == local && cp.Remote.Name == remote && cp.Link.RemotePort == port {
|
||||||
|
cp.Link.Disabled = disabled
|
||||||
|
cp.Active = !disabled
|
||||||
|
}
|
||||||
|
out.Topology[id] = &cp
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLocalDisableSurvivesAdoption is the regression test for the wipe that made
|
||||||
|
// the ring's authority unachievable.
|
||||||
|
//
|
||||||
|
// OnToken/AdoptState replace e.state wholesale. A stop decided between two token
|
||||||
|
// cycles was therefore erased by the next adoption whenever the incoming token
|
||||||
|
// had been captured before the decision — so the flag never reached the other
|
||||||
|
// members and the forward kept running. Written as a probe BEFORE the fix: it
|
||||||
|
// reported "after adopting a stale token: known=true disabled=false".
|
||||||
|
func TestLocalDisableSurvivesAdoption(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Local decision: stop it.
|
||||||
|
eng.UpdateTopologyDisabled("web", "frps1", 18081, true)
|
||||||
|
if d, _ := eng.TopologyDisabled("web", "frps1", 18081); !d {
|
||||||
|
t.Fatal("precondition: the local stop should be visible")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A token captured BEFORE the decision arrives.
|
||||||
|
stale := staleTokenWith(eng, "web", "frps1", 18081, false)
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: stale, SentAt: 2}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Assert on the STATE, not the TopologyDisabled() accessor: the accessor
|
||||||
|
// short-circuits to the pending local decision, so it would report the stop
|
||||||
|
// even when the state about to be forwarded still says "enabled". What
|
||||||
|
// matters is that the token this node sends on carries the decision.
|
||||||
|
assertForwardedDisabled(t, eng, "web", "frps1", 18081, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertForwardedDisabled checks the disabled value a DOWNSTREAM node would see
|
||||||
|
// in the token this engine forwards — the value that actually propagates.
|
||||||
|
func assertForwardedDisabled(t *testing.T, eng *Engine, local, remote string, port int, want bool) {
|
||||||
|
t.Helper()
|
||||||
|
// Model one hop: a fresh engine adopts exactly what this one would send.
|
||||||
|
peer := newTestEngine("peer", false)
|
||||||
|
peer.AdoptState(eng.state)
|
||||||
|
got, known := peer.TopologyDisabled(local, remote, port)
|
||||||
|
if !known {
|
||||||
|
t.Fatalf("peer learned no entry for %s→%s:%d", local, remote, port)
|
||||||
|
}
|
||||||
|
if got != want {
|
||||||
|
t.Fatalf("the forwarded token carries disabled=%v, want %v — the decision did not "+
|
||||||
|
"propagate, so the other members would keep the old state", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLocalReEnableSurvivesAdoption: a re-enable needs the same protection. The
|
||||||
|
// value false is easy to overlook — nothing "looks" stopped, yet losing it just
|
||||||
|
// as silently strands the forward in the stopped state.
|
||||||
|
func TestLocalReEnableSurvivesAdoption(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
eng.UpdateTopologyDisabled("web", "frps1", 18081, true)
|
||||||
|
eng.UpdateTopologyDisabled("web", "frps1", 18081, false)
|
||||||
|
|
||||||
|
// A token that still says disabled (captured mid-stop) arrives.
|
||||||
|
mid := staleTokenWith(eng, "web", "frps1", 18081, true)
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: mid, SentAt: 2}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
assertForwardedDisabled(t, eng, "web", "frps1", 18081, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestConfirmedDecisionStopsBeingAsserted: once the ring reports the same value
|
||||||
|
// the decision is confirmed everywhere, so this node must stop overriding —
|
||||||
|
// otherwise it would fight a later decision made elsewhere.
|
||||||
|
func TestConfirmedDecisionStopsBeingAsserted(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
eng.UpdateTopologyDisabled("web", "frps1", 18081, true)
|
||||||
|
if len(eng.localDisabled) != 1 {
|
||||||
|
t.Fatalf("a pending decision should be tracked, got %v", eng.localDisabled)
|
||||||
|
}
|
||||||
|
// The ring comes back agreeing.
|
||||||
|
agreed := staleTokenWith(eng, "web", "frps1", 18081, true)
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: agreed, SentAt: 2}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(eng.localDisabled) != 0 {
|
||||||
|
t.Fatalf("a confirmed decision must stop being asserted, still tracking %v", eng.localDisabled)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A peer now decides the opposite; this node must follow the ring.
|
||||||
|
peerSaysEnabled := staleTokenWith(eng, "web", "frps1", 18081, false)
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 3, State: peerSaysEnabled, SentAt: 3}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if d, _ := eng.TopologyDisabled("web", "frps1", 18081); d {
|
||||||
|
t.Fatal("this node kept overriding a peer's later decision; the ring is not authoritative")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLocalDecisionsDoNotAccumulate: the tracking map is swept against the live
|
||||||
|
// topology, so a long-lived cluster cannot grow it without bound.
|
||||||
|
func TestLocalDecisionsDoNotAccumulate(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decisions for forwards this node has never seen.
|
||||||
|
eng.UpdateTopologyDisabled("ghost1", "frps1", 1, true)
|
||||||
|
eng.UpdateTopologyDisabled("ghost2", "frps1", 2, true)
|
||||||
|
if len(eng.localDisabled) != 2 {
|
||||||
|
t.Fatalf("expected 2 tracked decisions, got %d", len(eng.localDisabled))
|
||||||
|
}
|
||||||
|
// A cycle runs: neither ghost is in the topology, so both are swept.
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state, SentAt: 2}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(eng.localDisabled) != 0 {
|
||||||
|
t.Fatalf("decisions for forwards outside the topology must be swept, got %v", eng.localDisabled)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHasSelfInTopology is the wait condition for the startup reconcile: the
|
||||||
|
// reconcile rebuilds the workers for forwards THIS node owns, so a node can only
|
||||||
|
// act once the adopted state actually names it as an owner.
|
||||||
|
//
|
||||||
|
// It is deliberately independent of leadership — the bug it guards against was a
|
||||||
|
// reconcile that waited for leadership and therefore never ran on followers, even
|
||||||
|
// though followers own forwards.
|
||||||
|
func TestHasSelfInTopology(t *testing.T) {
|
||||||
|
// Two nodes; the forward is owned by n2, which is NOT the leader. A single-node
|
||||||
|
// ring would make this test pass under either implementation, because the only
|
||||||
|
// member is necessarily the leader — which is exactly how the first version of
|
||||||
|
// this test came to be a false green.
|
||||||
|
eng := newTestEngine("n2", false)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state, SentAt: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if eng.IsLeader() {
|
||||||
|
t.Fatal("precondition: n2 must not be the leader in this test")
|
||||||
|
}
|
||||||
|
// n2 claimed it, so it owns a forward — despite not leading.
|
||||||
|
if !eng.HasSelfInTopology() {
|
||||||
|
t.Error("a non-leader that owns a forward reports false — the startup reconcile " +
|
||||||
|
"would skip it and leave its worker down")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A node that owns nothing must report false (so it does not act early on a
|
||||||
|
// stale or empty topology).
|
||||||
|
other := newTestEngine("n9", true)
|
||||||
|
if other.HasSelfInTopology() {
|
||||||
|
t.Error("a node owning nothing must not claim it appears in the topology")
|
||||||
|
}
|
||||||
|
// Guard against a regression where leadership is folded into the predicate.
|
||||||
|
if !eng.HasSelfInTopology() && eng.IsLeader() {
|
||||||
|
t.Error("predicate must not be gated on leadership")
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -7,6 +7,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
|
"strconv"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@ -18,6 +19,17 @@ import (
|
|||||||
type Handler interface {
|
type Handler interface {
|
||||||
Claim(ctx context.Context, tk *Task) error
|
Claim(ctx context.Context, tk *Task) error
|
||||||
Revoke(ctx context.Context, tk *Task) error
|
Revoke(ctx context.Context, tk *Task) error
|
||||||
|
// Restart re-enables an already-claimed forward and brings its worker back.
|
||||||
|
// Needed because a stop keeps the topology entry, which makes both the
|
||||||
|
// creation path (idempotency guard) and the duplicate-claim guard refuse to
|
||||||
|
// re-create an existing forward.
|
||||||
|
Restart(ctx context.Context, tk *Task) error
|
||||||
|
// Release stops this node's worker for a forward it is giving up for
|
||||||
|
// rebalancing, WITHOUT marking the link stopped. The forward is about to run
|
||||||
|
// on another node, so a stopped flag would be wrong: every node's duplicate
|
||||||
|
// guard reads it and would then refuse to claim the re-published task,
|
||||||
|
// stranding the forward on the node that just gave it up.
|
||||||
|
Release(ctx context.Context, tk *Task) error
|
||||||
RuntimeLoad() Load
|
RuntimeLoad() Load
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -50,6 +62,24 @@ type Engine struct {
|
|||||||
// group changes made via HTTP handlers between token cycles are overwritten
|
// group changes made via HTTP handlers between token cycles are overwritten
|
||||||
// by the next state adoption and never propagate to other nodes.
|
// by the next state adoption and never propagate to other nodes.
|
||||||
topologySync func()
|
topologySync func()
|
||||||
|
// localDisabled records this node's own enabled/disabled decisions that have
|
||||||
|
// not yet been confirmed by the ring. It is what makes the RING authoritative
|
||||||
|
// without losing a decision that was made locally and has not yet had a
|
||||||
|
// chance to reach the other members.
|
||||||
|
//
|
||||||
|
// The problem it solves: OnToken/AdoptState do `e.state = tk.State`, a
|
||||||
|
// wholesale replacement. A stop decided between two token cycles therefore
|
||||||
|
// disappears on the very next adoption if that token was captured before the
|
||||||
|
// decision — the flag never reaches the rest of the ring, and the forward
|
||||||
|
// keeps running. (Verified with a probe before writing this: adopting a
|
||||||
|
// stale token flipped disabled back to false.)
|
||||||
|
//
|
||||||
|
// A key stays in this map until an adoption reports it disabled, at which
|
||||||
|
// point the whole cluster agrees and the key is dropped. Keys for forwards
|
||||||
|
// that vanish from the topology are swept on every update, so the map cannot
|
||||||
|
// grow without bound. The value false is what keeps a RE-ENABLE alive for the
|
||||||
|
// same reason a stop needs protecting.
|
||||||
|
localDisabled map[string]bool
|
||||||
|
|
||||||
state State
|
state State
|
||||||
// myAddr maps our Node ID to the address peers dial.
|
// myAddr maps our Node ID to the address peers dial.
|
||||||
@ -182,7 +212,9 @@ func (e *Engine) OnToken(ctx context.Context, tk *Token) (*Token, error) {
|
|||||||
if tk.SentAt > e.lastTokenAt {
|
if tk.SentAt > e.lastTokenAt {
|
||||||
e.lastTokenAt = tk.SentAt
|
e.lastTokenAt = tk.SentAt
|
||||||
}
|
}
|
||||||
log.Printf("ring[%s] OnToken cycle=%d", e.ID, tk.Cycle)
|
// Per-round heartbeat — steady state, no information. See debug.go: this
|
||||||
|
// fired ~2x/second and dominated the journal on every node.
|
||||||
|
debugToken("ring[%s] OnToken cycle=%d", e.ID, tk.Cycle)
|
||||||
|
|
||||||
// Parallel rhythm timer: operations run while the pace clock ticks.
|
// Parallel rhythm timer: operations run while the pace clock ticks.
|
||||||
// Delay scales with alive node count (more nodes → lower per-hop delay,
|
// Delay scales with alive node count (more nodes → lower per-hop delay,
|
||||||
@ -229,6 +261,11 @@ func (e *Engine) OnToken(ctx context.Context, tk *Token) (*Token, error) {
|
|||||||
// Re-apply local store overrides (group labels, disabled flags) onto
|
// Re-apply local store overrides (group labels, disabled flags) onto
|
||||||
// the freshly adopted topology so they survive state adoption and
|
// the freshly adopted topology so they survive state adoption and
|
||||||
// propagate to all nodes via the next token forward.
|
// propagate to all nodes via the next token forward.
|
||||||
|
// Re-assert decisions this node made locally that the ring has not yet
|
||||||
|
// confirmed, BEFORE the host's topology sync runs: the sync reads
|
||||||
|
// TopologyDisabled to learn the cluster's view, so the local decision must
|
||||||
|
// already be applied or a stop could be reported as "enabled" and dropped.
|
||||||
|
e.reconcileLocalDisabled()
|
||||||
if e.topologySync != nil {
|
if e.topologySync != nil {
|
||||||
e.topologySync()
|
e.topologySync()
|
||||||
}
|
}
|
||||||
@ -375,6 +412,43 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
|
|||||||
}
|
}
|
||||||
continue // owned elsewhere — ride to the owner
|
continue // owned elsewhere — ride to the owner
|
||||||
}
|
}
|
||||||
|
if t.Release {
|
||||||
|
// Owner-directed, like a revoke: only the node holding the forward
|
||||||
|
// may give it up. A release executed anywhere else would be an
|
||||||
|
// attempt to move somebody else's load, which cannot work — the
|
||||||
|
// owner's own count is what has to fall, and its topology entry is
|
||||||
|
// local bookkeeping no peer may write.
|
||||||
|
//
|
||||||
|
// A release whose forward is already gone is an idempotent no-op
|
||||||
|
// consumed by the lowest node, so it cannot ride forever if its
|
||||||
|
// owner departed first.
|
||||||
|
if owner := e.state.TopologyOwner(t); owner == e.ID {
|
||||||
|
target = t
|
||||||
|
break
|
||||||
|
} else if owner == "" && selfIsLowest {
|
||||||
|
target = t
|
||||||
|
break
|
||||||
|
}
|
||||||
|
continue // owned elsewhere — ride to the owner
|
||||||
|
}
|
||||||
|
if t.Restart {
|
||||||
|
// Owner-directed, exactly like a revoke: only the node holding the
|
||||||
|
// forward may act. Selecting it anywhere else would let a non-owner
|
||||||
|
// spawn a worker for somebody else's forward (observed live: a
|
||||||
|
// non-owner logged "restarted t4" and ran the forward itself).
|
||||||
|
//
|
||||||
|
// A restart whose forward is already gone is a no-op consumed by the
|
||||||
|
// lowest node, so a task can never ride forever if its owner
|
||||||
|
// departed before seeing it.
|
||||||
|
if owner := e.state.TopologyOwner(t); owner == e.ID {
|
||||||
|
target = t
|
||||||
|
break
|
||||||
|
} else if owner == "" && selfIsLowest {
|
||||||
|
target = t // forward gone — consume and re-enable nothing
|
||||||
|
break
|
||||||
|
}
|
||||||
|
continue // owned elsewhere — ride to the owner
|
||||||
|
}
|
||||||
if selfIsLowest {
|
if selfIsLowest {
|
||||||
target = t // generic forward create — lowest-load claim
|
target = t // generic forward create — lowest-load claim
|
||||||
break
|
break
|
||||||
@ -388,18 +462,37 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
if claimed.Revoke {
|
if claimed.Revoke {
|
||||||
if e.state.RemoveTopology(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort) {
|
// A user-requested stop is recorded as a FLAG on the topology entry,
|
||||||
if e.Handler != nil {
|
// not as a removal. The entry is the carrier that makes the decision
|
||||||
if err := e.Handler.Revoke(ctx, claimed); err != nil {
|
// travel: TopoEntry.Link is a full store.Link and State rides the
|
||||||
log.Printf("ring[%s] revoke %s: %v", e.ID, claimed.ID, err)
|
// token every cycle, so keeping the entry is what lets a stop reach
|
||||||
}
|
// every member — including a node that was offline when the stop was
|
||||||
}
|
// issued. Removing the entry instead left the flag nowhere to live,
|
||||||
if e.Log != nil {
|
// which is why a stop used to converge only if the owner happened to
|
||||||
_, _ = e.Log.Append(e.ID, LogForwardRemove, map[string]any{
|
// be online to receive a one-shot revoke task.
|
||||||
"taskId": claimed.ID, "local": claimed.Local.Name, "remote": claimed.Remote.Name,
|
//
|
||||||
})
|
// Marking Active=false is also what makes the entry inert for the
|
||||||
|
// rebalancing paths: OfflineReassign() skips inactive entries, and
|
||||||
|
// the startup reconcile skips them too, so a stopped forward is
|
||||||
|
// neither re-queued on a node departure nor re-claimed on a restart.
|
||||||
|
//
|
||||||
|
// Handler.Revoke still runs unconditionally — it is what actually
|
||||||
|
// stops the per-forward worker, and it must run even when there was no
|
||||||
|
// entry to mark (gating it on RemoveTopology()'s result made stops of
|
||||||
|
// already-absent forwards silently do nothing, so their workers ran
|
||||||
|
// forever: observed live as thousands of connection-refused lines
|
||||||
|
// against a local service that was intentionally down).
|
||||||
|
marked := e.state.UpdateTopologyDisabled(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort, true)
|
||||||
|
if e.Handler != nil {
|
||||||
|
if err := e.Handler.Revoke(ctx, claimed); err != nil {
|
||||||
|
log.Printf("ring[%s] revoke %s: %v", e.ID, claimed.ID, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if marked && e.Log != nil {
|
||||||
|
_, _ = e.Log.Append(e.ID, LogForwardStop, map[string]any{
|
||||||
|
"taskId": claimed.ID, "local": claimed.Local.Name, "remote": claimed.Remote.Name,
|
||||||
|
})
|
||||||
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if claimed.RemoveNode != "" && claimed.RemoveNode == e.ID {
|
if claimed.RemoveNode != "" && claimed.RemoveNode == e.ID {
|
||||||
@ -465,10 +558,90 @@ func (e *Engine) runCommands(ctx context.Context, tk *Token) error {
|
|||||||
// different node. Safe for OfflineReassign: that path deletes the
|
// different node. Safe for OfflineReassign: that path deletes the
|
||||||
// topology entry BEFORE re-queueing, so TopologyOwner returns "" and
|
// topology entry BEFORE re-queueing, so TopologyOwner returns "" and
|
||||||
// the legitimate re-claim passes through.
|
// the legitimate re-claim passes through.
|
||||||
if owner := e.state.TopologyOwner(claimed); owner != "" {
|
//
|
||||||
log.Printf("ring[%s] drop duplicate task %s: %s→%s:%d already owned by %s",
|
// A RESTART task is the deliberate exception this guard must not eat: it
|
||||||
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
|
// exists precisely to re-activate a forward whose entry is still present
|
||||||
claimed.Link.RemotePort, owner)
|
// (a stop keeps the entry as the flag's carrier), so applying the guard
|
||||||
|
// would swallow every re-enable and leave the forward stopped forever.
|
||||||
|
// A RELEASE is the second deliberate exception, for the same reason as a
|
||||||
|
// restart: it is addressed to the node that ALREADY owns the forward, so
|
||||||
|
// the entry is still there by definition. Applying the guard would drop
|
||||||
|
// every release and a node could never shed anything.
|
||||||
|
if !claimed.Restart && !claimed.Release {
|
||||||
|
if owner := e.state.TopologyOwner(claimed); owner != "" {
|
||||||
|
log.Printf("ring[%s] drop duplicate task %s: %s→%s:%d already owned by %s",
|
||||||
|
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
|
||||||
|
claimed.Link.RemotePort, owner)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if claimed.Release {
|
||||||
|
// Only reached when this node owns the forward (or it is already gone
|
||||||
|
// and we are the fallback consumer — see the selection loop).
|
||||||
|
owner := e.state.TopologyOwner(claimed)
|
||||||
|
if owner != e.ID {
|
||||||
|
// The forward vanished before we got here; nothing to give up.
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// Stop our worker. Deliberately NOT Handler.Revoke: that marks the
|
||||||
|
// link stopped, and the forward is about to run elsewhere — a
|
||||||
|
// "stopped" flag would make every node treat the re-published task as
|
||||||
|
// a duplicate and refuse to claim it.
|
||||||
|
if e.Handler != nil {
|
||||||
|
if err := e.Handler.Release(ctx, claimed); err != nil {
|
||||||
|
log.Printf("ring[%s] release %s: %v", e.ID, claimed.ID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Drop the entry so the forward has no owner. This is what makes the
|
||||||
|
// load actually fall (ForwardsOwnedBy counts entries) and what lets a
|
||||||
|
// fresh creation task pass the duplicate guard above.
|
||||||
|
e.RemoveTopologyEntry(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort)
|
||||||
|
// Hand it back to the ring as an ordinary unclaimed forward. It is
|
||||||
|
// published by THIS node so it rides the very token we are holding —
|
||||||
|
// no coordinator, no addressed hand-off.
|
||||||
|
link := claimed.Link
|
||||||
|
link.Disabled = false
|
||||||
|
if t := e.SubmitTask(claimed.Local, claimed.Remote, link); t != nil {
|
||||||
|
log.Printf("ring[%s] released %s: %s→%s:%d back to the ring as %s",
|
||||||
|
e.ID, claimed.ID, claimed.Local.Name, claimed.Remote.Name,
|
||||||
|
claimed.Link.RemotePort, t.ID)
|
||||||
|
} else {
|
||||||
|
log.Printf("ring[%s] released %s but the re-publish was deduped",
|
||||||
|
e.ID, claimed.ID)
|
||||||
|
}
|
||||||
|
if e.Log != nil {
|
||||||
|
_, _ = e.Log.Append(e.ID, LogForwardStop, map[string]any{
|
||||||
|
"taskId": claimed.ID, "local": claimed.Local.Name,
|
||||||
|
"remote": claimed.Remote.Name, "released": true,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if claimed.Restart {
|
||||||
|
// Only reached when this node owns the forward (or it is already gone
|
||||||
|
// and we are the fallback consumer — see the selection loop).
|
||||||
|
owner := e.state.TopologyOwner(claimed)
|
||||||
|
e.state.UpdateTopologyDisabled(claimed.Local.Name, claimed.Remote.Name, claimed.Link.RemotePort, false)
|
||||||
|
if owner != e.ID {
|
||||||
|
// Forward vanished before we got here: nothing to re-enable.
|
||||||
|
log.Printf("ring[%s] discard restart %s: no owner", e.ID, claimed.ID)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if e.Handler != nil {
|
||||||
|
if err := e.Handler.Restart(ctx, claimed); err != nil {
|
||||||
|
e.state.PendingTasks[claimed.ID] = claimed
|
||||||
|
log.Printf("ring[%s] restart %s failed: %v", e.ID, claimed.ID, err)
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if e.Log != nil {
|
||||||
|
_, _ = e.Log.Append(e.ID, LogForwardStart, map[string]any{
|
||||||
|
"taskId": claimed.ID, "local": claimed.Local.Name, "remote": claimed.Remote.Name,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
// Logging is the app's job (RestartFn reports the actual worker
|
||||||
|
// outcome); the ring layer stays quiet so one restart is one line
|
||||||
|
// rather than two identical ones.
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if e.Handler != nil {
|
if e.Handler != nil {
|
||||||
@ -724,6 +897,11 @@ func (e *Engine) AdoptState(s State) {
|
|||||||
for id, t := range kept {
|
for id, t := range kept {
|
||||||
e.state.PendingTasks[id] = t
|
e.state.PendingTasks[id] = t
|
||||||
}
|
}
|
||||||
|
// Re-assert decisions this node made locally that the ring has not yet
|
||||||
|
// confirmed, BEFORE the host's topology sync runs: the sync reads
|
||||||
|
// TopologyDisabled to learn the cluster's view, so the local decision must
|
||||||
|
// already be applied or a stop could be reported as "enabled" and dropped.
|
||||||
|
e.reconcileLocalDisabled()
|
||||||
if e.topologySync != nil {
|
if e.topologySync != nil {
|
||||||
e.topologySync()
|
e.topologySync()
|
||||||
}
|
}
|
||||||
@ -792,6 +970,14 @@ func (e *Engine) RemoveNode(nodeID string) *Task {
|
|||||||
return e.state.AddRemoveNode(nodeID)
|
return e.state.AddRemoveNode(nodeID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RemoveTopologyEntry drops the active topology entry for a forward identified
|
||||||
|
// by its natural key. Exposed so the app's claim path can retire an entry it
|
||||||
|
// refuses to serve (see ClaimFn: a disabled forward must stop looking active,
|
||||||
|
// otherwise the startup reconcile keeps re-claiming it on every restart).
|
||||||
|
func (e *Engine) RemoveTopologyEntry(local, remote string, port int) bool {
|
||||||
|
return e.state.RemoveTopology(local, remote, port)
|
||||||
|
}
|
||||||
|
|
||||||
// RevokeTask publishes a revocation for an established forward through the
|
// RevokeTask publishes a revocation for an established forward through the
|
||||||
// same token channel; the owning node stops the worker and drops topology.
|
// same token channel; the owning node stops the worker and drops topology.
|
||||||
func (e *Engine) RevokeTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
func (e *Engine) RevokeTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||||
@ -799,14 +985,107 @@ func (e *Engine) RevokeTask(local store.Local, remote store.Remote, link store.L
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (e *Engine) SubmitTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
func (e *Engine) SubmitTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||||
if e.HasTask(local.Name, remote.Name, link.RemotePort) {
|
// Guard on an ACTIVE task only. A topology entry that is merely marked
|
||||||
|
// disabled is kept around as the carrier for the stop flag, so counting it
|
||||||
|
// here would make this a permanent no-op and a stopped forward could never
|
||||||
|
// be started again.
|
||||||
|
if e.HasActiveTask(local.Name, remote.Name, link.RemotePort) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return e.state.AddPending(local, remote, link)
|
return e.state.AddPending(local, remote, link)
|
||||||
}
|
}
|
||||||
|
|
||||||
// HasTask reports whether a forward with the same local/remote/remotePort is
|
// HasTopologyEntry reports whether the forward has a topology entry (whether or
|
||||||
// already pending or active in the topology (idempotency guard for resaves).
|
// not it is marked disabled). A stop keeps the entry as the flag's carrier, so
|
||||||
|
// "has an entry" is what distinguishes "already claimed by someone" from "never
|
||||||
|
// submitted" — the distinction startForward needs.
|
||||||
|
func (e *Engine) HasTopologyEntry(local, remote string, port int) bool {
|
||||||
|
_, found := e.state.TopologyDisabled(local, remote, port)
|
||||||
|
return found
|
||||||
|
}
|
||||||
|
|
||||||
|
// SubmitRestart publishes a task asking the current owner of an already-claimed
|
||||||
|
// forward to bring its worker back up.
|
||||||
|
//
|
||||||
|
// Neither of the existing channels can express "restart what already exists":
|
||||||
|
// SubmitTask is guarded by HasActiveTask and the entry is still present, so it
|
||||||
|
// dedupes; and the claim path drops any task for a forward that already has an
|
||||||
|
// owner (its defence against duplicate-claim collisions). Re-enabling a stopped
|
||||||
|
// forward therefore needs its own task kind, which the owner applies without
|
||||||
|
// re-running the claim bookkeeping.
|
||||||
|
func (e *Engine) SubmitRestart(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||||
|
// The task must not carry the stop: the whole point is to re-enable.
|
||||||
|
link.Disabled = false
|
||||||
|
t := &Task{
|
||||||
|
ID: e.state.NextTaskID(),
|
||||||
|
Local: local,
|
||||||
|
Remote: remote,
|
||||||
|
Link: link,
|
||||||
|
Created: time.Now().Unix(),
|
||||||
|
Restart: true,
|
||||||
|
}
|
||||||
|
if e.state.PendingTasks == nil {
|
||||||
|
e.state.PendingTasks = map[string]*Task{}
|
||||||
|
}
|
||||||
|
e.state.PendingTasks[t.ID] = t
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReleaseTask publishes a rebalancing task asking the OWNER of this forward to
|
||||||
|
// give it up. The task rides the token like any other command; it is not
|
||||||
|
// addressed to a destination, because the ring's normal lowest-load claim
|
||||||
|
// decides where the forward goes next.
|
||||||
|
//
|
||||||
|
// It goes through AddPending rather than writing PendingTasks directly. That is
|
||||||
|
// not cosmetic: OnToken rebuilds local-only pending commands from the ids it
|
||||||
|
// never published, and only the Add* helpers record them — a task injected by
|
||||||
|
// hand was silently dropped at the next state adoption, so it never reached the
|
||||||
|
// ring at all.
|
||||||
|
func (e *Engine) ReleaseTask(local store.Local, remote store.Remote, link store.Link) *Task {
|
||||||
|
link.Disabled = false
|
||||||
|
t := e.state.AddPending(local, remote, link)
|
||||||
|
if t == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
t.Release = true
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
// HasActiveTask reports whether a forward with this key is genuinely in flight
|
||||||
|
// or running: a non-revocation pending task, or a topology entry that is not
|
||||||
|
// marked disabled.
|
||||||
|
//
|
||||||
|
// It is deliberately narrower than HasTask. Under mark-don't-remove a stopped
|
||||||
|
// forward keeps its topology entry (that entry is what carries the flag between
|
||||||
|
// nodes), so "an entry exists" no longer means "this forward is active".
|
||||||
|
// SubmitTask must use THIS predicate, otherwise starting a stopped forward —
|
||||||
|
// startForward() publishes the enable and then calls SubmitTask — would be
|
||||||
|
// swallowed by its own idempotency guard.
|
||||||
|
func (e *Engine) HasActiveTask(local, remote string, port int) bool {
|
||||||
|
for _, t := range e.state.PendingList() {
|
||||||
|
if t.Revoke {
|
||||||
|
continue // a revocation is the opposite of an active forward
|
||||||
|
}
|
||||||
|
if t.Local.Name == local && t.Remote.Name == remote && t.Link.RemotePort == port {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, t := range e.state.TopologyList() {
|
||||||
|
if t.Link.Disabled {
|
||||||
|
continue // stopped; the entry is only a flag carrier
|
||||||
|
}
|
||||||
|
if t.Local.Name == local && t.Remote.Name == remote && t.Link.RemotePort == port {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// HasTask reports whether ANY record for this forward exists — a pending task
|
||||||
|
// (including an in-flight revocation) or a topology entry (including one marked
|
||||||
|
// disabled). This is the "have we already handled this key" question, used to
|
||||||
|
// avoid re-issuing work on repeated canvas saves. For "is it actually running"
|
||||||
|
// use HasActiveTask instead.
|
||||||
func (e *Engine) HasTask(local, remote string, port int) bool {
|
func (e *Engine) HasTask(local, remote string, port int) bool {
|
||||||
for _, t := range e.state.PendingList() {
|
for _, t := range e.state.PendingList() {
|
||||||
if t.Local.Name == local && t.Remote.Name == remote && t.Link.RemotePort == port {
|
if t.Local.Name == local && t.Remote.Name == remote && t.Link.RemotePort == port {
|
||||||
@ -828,6 +1107,78 @@ func (e *Engine) UpdateTopologyGroup(local, remote string, port int, group strin
|
|||||||
return e.state.UpdateTopologyGroup(local, remote, port, group)
|
return e.state.UpdateTopologyGroup(local, remote, port, group)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpdateTopologyDisabled marks a forward enabled/disabled in the topology so
|
||||||
|
// the decision rides the next token cycle to every member. Paired with
|
||||||
|
// TopologyDisabled, which the adoption hook uses to learn peers' decisions.
|
||||||
|
//
|
||||||
|
// The decision is also remembered locally until the ring confirms it, because
|
||||||
|
// adoption replaces the whole state: a token captured before this call would
|
||||||
|
// otherwise wash the decision out on the next cycle and it would never reach
|
||||||
|
// the other members. See the localDisabled field.
|
||||||
|
func (e *Engine) UpdateTopologyDisabled(local, remote string, port int, disabled bool) bool {
|
||||||
|
if e.localDisabled == nil {
|
||||||
|
e.localDisabled = map[string]bool{}
|
||||||
|
}
|
||||||
|
e.localDisabled[disableKey(local, remote, port)] = disabled
|
||||||
|
return e.state.UpdateTopologyDisabled(local, remote, port, disabled)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TopologyDisabled reports the cluster's view of a forward's disabled flag.
|
||||||
|
// The bool is false when the forward has no topology entry (nothing to learn).
|
||||||
|
//
|
||||||
|
// A pending local decision takes precedence over the adopted state: it has not
|
||||||
|
// had a chance to reach the other members yet, and reporting the adopted value
|
||||||
|
// would make the local store (and the status page) disagree with the user's
|
||||||
|
// most recent action.
|
||||||
|
func (e *Engine) TopologyDisabled(local, remote string, port int) (bool, bool) {
|
||||||
|
if d, pending := e.localDisabled[disableKey(local, remote, port)]; pending {
|
||||||
|
return d, true
|
||||||
|
}
|
||||||
|
return e.state.TopologyDisabled(local, remote, port)
|
||||||
|
}
|
||||||
|
|
||||||
|
// reconcileLocalDisabled re-asserts this node's not-yet-confirmed disabled
|
||||||
|
// decisions onto the freshly adopted state. Called right after e.state is
|
||||||
|
// replaced, and paired with reconcileTopologySync (which pushes those decisions
|
||||||
|
// out to the ring).
|
||||||
|
//
|
||||||
|
// A decision is dropped once the adopted state reports the SAME value: at that
|
||||||
|
// point every member agrees and there is nothing left to protect. Entries whose
|
||||||
|
// forward no longer exists in the topology are dropped too, so the map tracks
|
||||||
|
// only live disagreements.
|
||||||
|
func (e *Engine) reconcileLocalDisabled() {
|
||||||
|
if len(e.localDisabled) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for id, te := range e.state.Topology {
|
||||||
|
_ = id
|
||||||
|
k := disableKey(te.Local.Name, te.Remote.Name, te.Link.RemotePort)
|
||||||
|
want, pending := e.localDisabled[k]
|
||||||
|
if !pending {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if te.Link.Disabled == want {
|
||||||
|
// The ring caught up (or agreed independently): stop tracking.
|
||||||
|
delete(e.localDisabled, k)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// Still divergent: keep asserting our decision onto the adopted state.
|
||||||
|
te.Link.Disabled = want
|
||||||
|
te.Active = !want
|
||||||
|
}
|
||||||
|
// Forget decisions for forwards that left the topology entirely — otherwise
|
||||||
|
// a long-lived cluster would accumulate dead keys.
|
||||||
|
live := make(map[string]struct{}, len(e.state.Topology))
|
||||||
|
for _, te := range e.state.Topology {
|
||||||
|
live[disableKey(te.Local.Name, te.Remote.Name, te.Link.RemotePort)] = struct{}{}
|
||||||
|
}
|
||||||
|
for k := range e.localDisabled {
|
||||||
|
if _, ok := live[k]; !ok {
|
||||||
|
delete(e.localDisabled, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// IsLeader reports whether this node is the current ring leader.
|
// IsLeader reports whether this node is the current ring leader.
|
||||||
func (e *Engine) IsLeader() bool { return e.state.LeaderID == e.ID }
|
func (e *Engine) IsLeader() bool { return e.state.LeaderID == e.ID }
|
||||||
|
|
||||||
@ -850,6 +1201,14 @@ func (e *Engine) SetPeerPersist(fn func(peersJSON string) error) { e.peerPersist
|
|||||||
// HTTP handlers are overwritten by the next e.state = tk.State.
|
// HTTP handlers are overwritten by the next e.state = tk.State.
|
||||||
func (e *Engine) SetTopologySync(fn func()) { e.topologySync = fn }
|
func (e *Engine) SetTopologySync(fn func()) { e.topologySync = fn }
|
||||||
|
|
||||||
|
// disableKey builds the map key for a forward's disabled decision: the natural
|
||||||
|
// triple (local, remote, remotePort), which is how every other part of the code
|
||||||
|
// identifies a forward. A NUL separator keeps it unambiguous for names that
|
||||||
|
// could otherwise collide across the boundaries.
|
||||||
|
func disableKey(local, remote string, port int) string {
|
||||||
|
return local + "\x00" + remote + "\x00" + strconv.Itoa(port)
|
||||||
|
}
|
||||||
|
|
||||||
// persistPeers extracts all alive peers (addr + nodeKey, excluding self)
|
// persistPeers extracts all alive peers (addr + nodeKey, excluding self)
|
||||||
// from the current ring state and persists them via the peerPersist callback.
|
// from the current ring state and persists them via the peerPersist callback.
|
||||||
// Called on every token cycle (OnToken) and on AdoptState so a crashed node
|
// Called on every token cycle (OnToken) and on AdoptState so a crashed node
|
||||||
@ -938,3 +1297,20 @@ func (e *Engine) LeaderAddr() string {
|
|||||||
}
|
}
|
||||||
return e.state.Nodes[i].Addr
|
return e.state.Nodes[i].Addr
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// HasSelfInTopology reports whether the current topology names this node as the
|
||||||
|
// owner of at least one active forward.
|
||||||
|
//
|
||||||
|
// The startup reconcile uses it as its wait condition: a node can only rebuild
|
||||||
|
// the workers for forwards it owns once the adopted state says it owns them.
|
||||||
|
// Leadership is deliberately NOT part of the test — followers own forwards too,
|
||||||
|
// and gating the reconcile on leadership left them permanently unable to restore
|
||||||
|
// their own workers.
|
||||||
|
func (e *Engine) HasSelfInTopology() bool {
|
||||||
|
for _, te := range e.state.TopologyList() {
|
||||||
|
if te.OwnerID == e.ID && te.Active {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|||||||
@ -8,9 +8,18 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type fakeHandler struct {
|
type fakeHandler struct {
|
||||||
load Load
|
load Load
|
||||||
claim func(ctx context.Context, tk *Task) error
|
claim func(ctx context.Context, tk *Task) error
|
||||||
revoke func(ctx context.Context, tk *Task) error
|
revoke func(ctx context.Context, tk *Task) error
|
||||||
|
restart func(ctx context.Context, tk *Task) error
|
||||||
|
release func(ctx context.Context, tk *Task) error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *fakeHandler) Release(ctx context.Context, tk *Task) error {
|
||||||
|
if h.release != nil {
|
||||||
|
return h.release(ctx, tk)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (h *fakeHandler) Revoke(ctx context.Context, tk *Task) error {
|
func (h *fakeHandler) Revoke(ctx context.Context, tk *Task) error {
|
||||||
@ -20,6 +29,13 @@ func (h *fakeHandler) Revoke(ctx context.Context, tk *Task) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (h *fakeHandler) Restart(ctx context.Context, tk *Task) error {
|
||||||
|
if h.restart != nil {
|
||||||
|
return h.restart(ctx, tk)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (h *fakeHandler) RuntimeLoad() Load { return h.load }
|
func (h *fakeHandler) RuntimeLoad() Load { return h.load }
|
||||||
func (h *fakeHandler) Claim(ctx context.Context, tk *Task) error {
|
func (h *fakeHandler) Claim(ctx context.Context, tk *Task) error {
|
||||||
if h.claim != nil {
|
if h.claim != nil {
|
||||||
|
|||||||
@ -17,6 +17,16 @@ type AppHandler struct {
|
|||||||
ClaimFn func(ctx context.Context, tk *Task) error
|
ClaimFn func(ctx context.Context, tk *Task) error
|
||||||
// RevokeFn cancels the forward on this node (stop worker + drop from store).
|
// RevokeFn cancels the forward on this node (stop worker + drop from store).
|
||||||
RevokeFn func(ctx context.Context, tk *Task) error
|
RevokeFn func(ctx context.Context, tk *Task) error
|
||||||
|
// RestartFn re-enables an already-claimed forward on this node (clear the
|
||||||
|
// stopped flag + bring the worker back). Required because a stop keeps the
|
||||||
|
// topology entry, so the normal claim path will not re-create it.
|
||||||
|
RestartFn func(ctx context.Context, tk *Task) error
|
||||||
|
// ReleaseFn stops this node's worker for a forward being handed back to the
|
||||||
|
// ring, leaving the link ENABLED. When nil, Release falls back to RevokeFn —
|
||||||
|
// which also stops the worker, but additionally marks the link stopped; that
|
||||||
|
// is the pre-existing stop semantics and is wrong for a rebalance, so the app
|
||||||
|
// should provide this.
|
||||||
|
ReleaseFn func(ctx context.Context, tk *Task) error
|
||||||
}
|
}
|
||||||
|
|
||||||
// RuntimeLoad implements Handler.
|
// RuntimeLoad implements Handler.
|
||||||
@ -44,6 +54,30 @@ func (a *AppHandler) Revoke(ctx context.Context, tk *Task) error {
|
|||||||
return a.RevokeFn(ctx, tk)
|
return a.RevokeFn(ctx, tk)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Release implements Handler.
|
||||||
|
func (a *AppHandler) Release(ctx context.Context, tk *Task) error {
|
||||||
|
if a.ReleaseFn != nil {
|
||||||
|
return a.ReleaseFn(ctx, tk)
|
||||||
|
}
|
||||||
|
// Fall back to the stop path so a host that has not provided ReleaseFn still
|
||||||
|
// stops the worker (the forward keeps running otherwise). The link ends up
|
||||||
|
// flagged stopped, which the claim path may then refuse — the host is warned
|
||||||
|
// by the ring's own "re-publish was deduped" log line rather than silently
|
||||||
|
// leaving two workers up.
|
||||||
|
if a.RevokeFn != nil {
|
||||||
|
return a.RevokeFn(ctx, tk)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Restart implements Handler.
|
||||||
|
func (a *AppHandler) Restart(ctx context.Context, tk *Task) error {
|
||||||
|
if a.RestartFn == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return a.RestartFn(ctx, tk)
|
||||||
|
}
|
||||||
|
|
||||||
// SampleMemLoad returns a cheap memory-usage percentage (0..100).
|
// SampleMemLoad returns a cheap memory-usage percentage (0..100).
|
||||||
func SampleMemLoad() float64 {
|
func SampleMemLoad() float64 {
|
||||||
var m runtime.MemStats
|
var m runtime.MemStats
|
||||||
|
|||||||
@ -128,7 +128,7 @@ func (e *Engine) forwardToNext(ctx context.Context, tk *Token) error {
|
|||||||
if e.send == nil {
|
if e.send == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
log.Printf("ring[%s] forward cycle=%d to %s", e.ID, tk.Cycle, next)
|
debugToken("ring[%s] forward cycle=%d to %s", e.ID, tk.Cycle, next)
|
||||||
err := e.send(ctx, next, tk)
|
err := e.send(ctx, next, tk)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
if e.state.LeaderID == e.ID {
|
if e.state.LeaderID == e.ID {
|
||||||
@ -146,7 +146,14 @@ func (e *Engine) forwardToNext(ctx context.Context, tk *Token) error {
|
|||||||
// the token is lost — start fresh as the new leader (plan §leader
|
// the token is lost — start fresh as the new leader (plan §leader
|
||||||
// 补充/监控: "上家邻居探测到 leader 崩溃 → 自身成为新 leader").
|
// 补充/监控: "上家邻居探测到 leader 崩溃 → 自身成为新 leader").
|
||||||
if next == e.state.LeaderID {
|
if next == e.state.LeaderID {
|
||||||
e.becomeLeader()
|
if !e.becomeLeader() {
|
||||||
|
// Another live leader exists (we are the partitioned side):
|
||||||
|
// keep its ring intact and let it re-drive the token. Re-raise
|
||||||
|
// the node we just marked dead only if the send really was the
|
||||||
|
// problem — otherwise we would paper over a real outage.
|
||||||
|
log.Printf("ring[%s] not promoting: deferring to live leader", e.ID)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
if e.Log != nil {
|
if e.Log != nil {
|
||||||
_, _ = e.Log.Append(e.ID, LogLeaderChange, map[string]string{"leader": e.ID})
|
_, _ = e.Log.Append(e.ID, LogLeaderChange, map[string]string{"leader": e.ID})
|
||||||
}
|
}
|
||||||
@ -161,13 +168,42 @@ func (e *Engine) forwardToNext(ctx context.Context, tk *Token) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// becomeLeader promotes this node (used when the monitored leader dies).
|
// becomeLeader promotes this node, but ONLY when the ring agrees there is no
|
||||||
func (e *Engine) becomeLeader() {
|
// live leader to defer to.
|
||||||
|
//
|
||||||
|
// The guard matters because leadership was previously self-declared: any node
|
||||||
|
// that failed to reach the leader it believed in would immediately promote
|
||||||
|
// itself. Two nodes partitioned from each other both hit that path and both
|
||||||
|
// ended up believing they were the leader — a split brain with two token
|
||||||
|
// rings, duplicate claims, and each side reassigning the other's forwards. That
|
||||||
|
// was observed live: after a rolling restart, two nodes reported the same
|
||||||
|
// leader while marking each other dead, and the topology was emptied into
|
||||||
|
// pending on both sides.
|
||||||
|
//
|
||||||
|
// Before promoting, ask the ring: is any OTHER node currently alive and claiming
|
||||||
|
// leadership? If so, we are the partitioned one — demote ourselves and let the
|
||||||
|
// live leader keep the ring. A genuine single-leader loss (the real leader is
|
||||||
|
// dead or unreachable) leaves no such node, so promotion still happens.
|
||||||
|
//
|
||||||
|
// CreateCluster does not come through here: it assigns LeaderID directly, so
|
||||||
|
// the "no other leader exists" case never needs an override.
|
||||||
|
func (e *Engine) becomeLeader() bool {
|
||||||
|
for i := range e.state.Nodes {
|
||||||
|
n := &e.state.Nodes[i]
|
||||||
|
if n.ID == e.ID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if n.Alive && n.IsLeader {
|
||||||
|
log.Printf("ring[%s] decline promotion: %s is alive and is the leader", e.ID, n.ID)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
e.state.LeaderID = e.ID
|
e.state.LeaderID = e.ID
|
||||||
for i := range e.state.Nodes {
|
for i := range e.state.Nodes {
|
||||||
e.state.Nodes[i].IsLeader = e.state.Nodes[i].ID == e.ID
|
e.state.Nodes[i].IsLeader = e.state.Nodes[i].ID == e.ID
|
||||||
}
|
}
|
||||||
log.Printf("ring[%s] promoted to leader", e.ID)
|
log.Printf("ring[%s] promoted to leader", e.ID)
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// WatchLeader runs the FALLBACK leader liveness monitor. The PRIMARY path is
|
// WatchLeader runs the FALLBACK leader liveness monitor. The PRIMARY path is
|
||||||
@ -209,20 +245,43 @@ func (e *Engine) WatchLeader(ctx context.Context) {
|
|||||||
cancel()
|
cancel()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("ring[%s] heartbeat to leader %s failed: %v", e.ID, e.state.LeaderID, err)
|
log.Printf("ring[%s] heartbeat to leader %s failed: %v", e.ID, e.state.LeaderID, err)
|
||||||
e.state.MarkOffline(e.state.LeaderID)
|
e.takeOverFromDeadLeader(ctx)
|
||||||
e.state.OfflineReassign(e.state.LeaderID)
|
|
||||||
e.becomeLeader()
|
|
||||||
// Kick off a fresh cycle: the ring died with the old
|
|
||||||
// leader (no token inflight → WatchTokenLoss won't
|
|
||||||
// fire). Without this the newly promoted leader would
|
|
||||||
// sit idle and the ring would stay dead.
|
|
||||||
e.StartRing(ctx)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// takeOverFromDeadLeader handles "the leader I could not reach is gone": mark it
|
||||||
|
// dead and, if the ring agrees there is no live leader left, promote this node
|
||||||
|
// and re-seed the cycle.
|
||||||
|
//
|
||||||
|
// The promotion is deliberately conditional, and so is the reassignment. Doing
|
||||||
|
// the reassignment unconditionally is what destroyed the live ring: a node on
|
||||||
|
// the partitioned side would mark the (still very much alive) leader dead and
|
||||||
|
// move ALL of its forwards into pending, so both sides ended up with an empty
|
||||||
|
// topology and a queue nobody was serving. Sequencing matters — the forward
|
||||||
|
// re-hang is only correct once we have actually taken leadership.
|
||||||
|
//
|
||||||
|
// Extracted from WatchLeader's loop so the decision is testable (the loop itself
|
||||||
|
// is an endless goroutine).
|
||||||
|
func (e *Engine) takeOverFromDeadLeader(ctx context.Context) {
|
||||||
|
old := e.state.LeaderID
|
||||||
|
e.state.MarkOffline(old)
|
||||||
|
if !e.becomeLeader() {
|
||||||
|
log.Printf("ring[%s] leader %s unreachable but a live leader remains; "+
|
||||||
|
"declining takeover and leaving its forwards untouched", e.ID, old)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
e.state.OfflineReassign(old)
|
||||||
|
if e.Log != nil {
|
||||||
|
_, _ = e.Log.Append(e.ID, LogLeaderChange, map[string]string{"leader": e.ID})
|
||||||
|
}
|
||||||
|
// The ring died with the old leader (no token inflight → WatchTokenLoss
|
||||||
|
// won't fire). Without this the newly promoted leader would sit idle.
|
||||||
|
e.StartRing(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
// WatchTokenLoss runs the leader's token-loss judge: if a token was sent and
|
// WatchTokenLoss runs the leader's token-loss judge: if a token was sent and
|
||||||
// does not return within LossTimeout, the leader issues a fresh token (all
|
// does not return within LossTimeout, the leader issues a fresh token (all
|
||||||
// nodes drop older stamps via the SentAt guard, so at most one circulates).
|
// nodes drop older stamps via the SentAt guard, so at most one circulates).
|
||||||
|
|||||||
@ -20,6 +20,8 @@ import (
|
|||||||
// LogKind enumerates operation-log entry types.
|
// LogKind enumerates operation-log entry types.
|
||||||
const (
|
const (
|
||||||
LogForwardAdd = "forward.add"
|
LogForwardAdd = "forward.add"
|
||||||
|
LogForwardStop = "forward.stop"
|
||||||
|
LogForwardStart = "forward.start"
|
||||||
LogForwardRemove = "forward.remove"
|
LogForwardRemove = "forward.remove"
|
||||||
LogNodeJoin = "node.join"
|
LogNodeJoin = "node.join"
|
||||||
LogNodeLeave = "node.leave"
|
LogNodeLeave = "node.leave"
|
||||||
@ -140,7 +142,7 @@ func DetailOf(e LogEntry) string {
|
|||||||
}
|
}
|
||||||
str := func(key string) string { s, _ := d[key].(string); return s }
|
str := func(key string) string { s, _ := d[key].(string); return s }
|
||||||
switch e.Kind {
|
switch e.Kind {
|
||||||
case LogForwardAdd, LogForwardRemove:
|
case LogForwardAdd, LogForwardStop, LogForwardStart, LogForwardRemove:
|
||||||
s := str("local") + " → " + str("remote")
|
s := str("local") + " → " + str("remote")
|
||||||
if id := str("taskId"); id != "" {
|
if id := str("taskId"); id != "" {
|
||||||
if len(id) > 8 {
|
if len(id) > 8 {
|
||||||
|
|||||||
@ -2,14 +2,22 @@ package cluster
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"webui4frpc/internal/store"
|
"webui4frpc/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestRevokeTaskRemovesTopology: a revoke task published to the ring removes
|
// TestRevokeTaskMarksTopologyDisabled: a stop delivered through the ring marks
|
||||||
// the forward from topology, calls Handler.Revoke, and logs forward.remove.
|
// the topology entry disabled instead of deleting it, calls Handler.Revoke, and
|
||||||
func TestRevokeTaskRemovesTopology(t *testing.T) {
|
// logs forward.stop.
|
||||||
|
//
|
||||||
|
// The entry is deliberately KEPT: TopoEntry.Link is the carrier that makes the
|
||||||
|
// stop travel, and State rides the token every cycle. Keeping it is what lets a
|
||||||
|
// stop reach a member that was offline when the stop was issued — deleting the
|
||||||
|
// entry left the flag nowhere to live, so a stop converged only if the owning
|
||||||
|
// node happened to be online for a one-shot revoke task.
|
||||||
|
func TestRevokeTaskMarksTopologyDisabled(t *testing.T) {
|
||||||
revoked := false
|
revoked := false
|
||||||
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||||
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||||
@ -26,33 +34,92 @@ func TestRevokeTaskRemovesTopology(t *testing.T) {
|
|||||||
if len(eng.state.TopologyList()) != 1 {
|
if len(eng.state.TopologyList()) != 1 {
|
||||||
t.Fatalf("topology after claim = %+v", eng.state.TopologyList())
|
t.Fatalf("topology after claim = %+v", eng.state.TopologyList())
|
||||||
}
|
}
|
||||||
|
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); d {
|
||||||
|
t.Fatal("a freshly claimed forward must not start out disabled")
|
||||||
|
}
|
||||||
|
|
||||||
// publish a revoke task pointing at the same forward
|
// publish a revoke task pointing at the same forward
|
||||||
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if len(eng.state.TopologyList()) != 0 {
|
|
||||||
t.Fatalf("topology after revoke = %+v", eng.state.TopologyList())
|
// The entry must SURVIVE, marked disabled, so the flag keeps travelling.
|
||||||
|
if len(eng.state.TopologyList()) != 1 {
|
||||||
|
t.Fatalf("topology entry was removed; the disabled flag would have no carrier: %+v",
|
||||||
|
eng.state.TopologyList())
|
||||||
|
}
|
||||||
|
disabled, known := eng.state.TopologyDisabled("web", "frps1", 18081)
|
||||||
|
if !known {
|
||||||
|
t.Fatal("topology entry missing after revoke")
|
||||||
|
}
|
||||||
|
if !disabled {
|
||||||
|
t.Fatal("revoke did not mark the topology entry disabled")
|
||||||
|
}
|
||||||
|
// Active must follow the flag, since OfflineReassign only re-queues Active
|
||||||
|
// entries — a stopped forward must not be resurrected by a node departure.
|
||||||
|
if eng.state.TopologyList()[0].Active {
|
||||||
|
t.Fatal("a stopped forward must not remain Active (OfflineReassign would re-queue it)")
|
||||||
}
|
}
|
||||||
if !revoked {
|
if !revoked {
|
||||||
t.Fatal("Handler.Revoke was not called")
|
t.Fatal("Handler.Revoke was not called")
|
||||||
}
|
}
|
||||||
// log should contain forward.remove
|
// log should contain forward.stop (not forward.remove — nothing was removed)
|
||||||
var sawRemove bool
|
var sawStop bool
|
||||||
for _, e := range eng.Log.Snapshot() {
|
for _, e := range eng.Log.Snapshot() {
|
||||||
if e.Kind == LogForwardRemove {
|
if e.Kind == LogForwardStop {
|
||||||
sawRemove = true
|
sawStop = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !sawRemove {
|
if !sawStop {
|
||||||
t.Fatalf("log missing forward.remove: %+v", eng.Log.Snapshot())
|
t.Fatalf("log missing forward.stop: %+v", eng.Log.Snapshot())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStoppedTopologyEntrySurvivesAdoption is the property that makes the whole
|
||||||
|
// mark-don't-remove design work: because the entry persists, a node that adopts
|
||||||
|
// ring state learns the stop. This is the offline-member case a one-shot revoke
|
||||||
|
// task could never cover.
|
||||||
|
func TestStoppedTopologyEntrySurvivesAdoption(t *testing.T) {
|
||||||
|
src := newTestEngine("n1", true)
|
||||||
|
src.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := src.OnToken(context.Background(), &Token{Cycle: 1, State: src.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
src.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := src.OnToken(context.Background(), &Token{Cycle: 2, State: src.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A different node adopts the ring state (what e.state = tk.State does).
|
||||||
|
peer := newTestEngine("n2", false)
|
||||||
|
peer.AdoptState(src.state)
|
||||||
|
|
||||||
|
disabled, known := peer.TopologyDisabled("web", "frps1", 18081)
|
||||||
|
if !known {
|
||||||
|
t.Fatal("peer did not receive the topology entry for the stopped forward")
|
||||||
|
}
|
||||||
|
if !disabled {
|
||||||
|
t.Fatal("peer adopted the entry but no longer sees it as disabled — the stop did not propagate")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRevokeIdempotent: revoking an already-missing forward does not error.
|
// TestRevokeIdempotent: revoking an already-missing forward does not error.
|
||||||
|
//
|
||||||
|
// It must ALSO still call Handler.Revoke. The handler is what actually stops
|
||||||
|
// the per-forward frpc worker; the topology entry is only bookkeeping. Gating
|
||||||
|
// the handler on RemoveTopology()'s return value (as this test used to permit)
|
||||||
|
// turned every revoke of an already-absent forward into a silent no-op — the
|
||||||
|
// worker kept running, which is how a forward the user had stopped kept
|
||||||
|
// dialling a local service that was intentionally down, forever.
|
||||||
func TestRevokeIdempotent(t *testing.T) {
|
func TestRevokeIdempotent(t *testing.T) {
|
||||||
eng := newTestEngine("n1", true)
|
revoked := 0
|
||||||
|
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||||
|
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||||
|
revoke: func(ctx context.Context, tk *Task) error { revoked++; return nil }},
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||||
|
"n1:7500", true, "")
|
||||||
|
|
||||||
eng.state.AddRevoke(store.Local{Name: "ghost"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 1})
|
eng.state.AddRevoke(store.Local{Name: "ghost"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 1})
|
||||||
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||||
t.Fatalf("revoke missing: %v", err)
|
t.Fatalf("revoke missing: %v", err)
|
||||||
@ -61,4 +128,318 @@ func TestRevokeIdempotent(t *testing.T) {
|
|||||||
if len(eng.state.TopologyList()) != 0 {
|
if len(eng.state.TopologyList()) != 0 {
|
||||||
t.Fatal("should be empty")
|
t.Fatal("should be empty")
|
||||||
}
|
}
|
||||||
|
// The stop side-effect must have happened even though there was no entry.
|
||||||
|
if revoked != 1 {
|
||||||
|
t.Fatalf("Handler.Revoke called %d times, want 1 — a revoke with no topology entry "+
|
||||||
|
"must still stop the worker, otherwise stopped forwards keep running", revoked)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStoppedForwardNotRequeuedOnNodeDeparture pins why marking (rather than
|
||||||
|
// deleting) is safe: OfflineReassign only re-queues ACTIVE entries, so a
|
||||||
|
// stopped forward is not silently handed to another node when its owner goes
|
||||||
|
// away. Deleting the entry, or leaving it Active, would both resurrect it.
|
||||||
|
func TestStoppedForwardNotRequeuedOnNodeDeparture(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
owner := eng.state.TopologyList()[0].OwnerID
|
||||||
|
eng.state.OfflineReassign(owner)
|
||||||
|
|
||||||
|
if n := len(eng.state.PendingList()); n != 0 {
|
||||||
|
t.Fatalf("a stopped forward was re-queued on the owner's departure (pending=%d): %+v",
|
||||||
|
n, eng.state.PendingList())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSubmitTaskNotBlockedByStoppedEntry is the regression test for the bug the
|
||||||
|
// mark-don't-remove change introduced: SubmitTask's idempotency guard used
|
||||||
|
// HasTask, which matches a disabled topology entry. Since a stop now KEEPS the
|
||||||
|
// entry, any submission for that forward while it is still stopped is swallowed
|
||||||
|
// by its own guard — so a stopped forward can never be started again.
|
||||||
|
//
|
||||||
|
// The probe deliberately submits WITHOUT re-enabling first. Re-enabling would
|
||||||
|
// clear the flag and make HasTask and HasActiveTask agree, hiding the defect;
|
||||||
|
// the guard has to be exercised at the moment the entry is still stopped.
|
||||||
|
func TestSubmitTaskNotBlockedByStoppedEntry(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// stop it -> entry is kept, marked disabled
|
||||||
|
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); !d {
|
||||||
|
t.Fatal("precondition: forward should be disabled")
|
||||||
|
}
|
||||||
|
// The stopped entry IS findable by the broad predicate — which is exactly why
|
||||||
|
// the narrow one has to exist.
|
||||||
|
if !eng.HasTask("web", "frps1", 18081) {
|
||||||
|
t.Fatal("precondition: the stopped entry should still be findable by HasTask")
|
||||||
|
}
|
||||||
|
if eng.HasActiveTask("web", "frps1", 18081) {
|
||||||
|
t.Fatal("precondition: a stopped forward must NOT count as active")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The regression: a submission must be published for a stopped forward.
|
||||||
|
// With a HasTask guard this returns nil and the forward is stuck forever.
|
||||||
|
tk := eng.SubmitTask(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||||
|
store.Link{RemotePort: 18081})
|
||||||
|
if tk == nil {
|
||||||
|
t.Fatal("SubmitTask was swallowed by the stopped topology entry — " +
|
||||||
|
"a stopped forward could never be started again")
|
||||||
|
}
|
||||||
|
if tk.Link.RemotePort != 18081 {
|
||||||
|
t.Fatalf("unexpected task: %+v", tk)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSubmitTaskStillDedupesActiveForward is the other half of the guard: the
|
||||||
|
// narrower predicate must not turn SubmitTask into a duplicate-task generator.
|
||||||
|
func TestSubmitTaskStillDedupesActiveForward(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !eng.HasActiveTask("web", "frps1", 18081) {
|
||||||
|
t.Fatal("precondition: a claimed forward must count as active")
|
||||||
|
}
|
||||||
|
if tk := eng.SubmitTask(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||||
|
store.Link{RemotePort: 18081}); tk != nil {
|
||||||
|
t.Fatalf("SubmitTask must stay idempotent for an active forward, got %+v", tk)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStoppedForwardNotRequeuedOnNodeDeparture pins why marking (rather than
|
||||||
|
// deleting) is safe: OfflineReassign only re-queues ACTIVE entries, so a
|
||||||
|
// stopped forward is not silently handed to another node when its owner goes
|
||||||
|
|
||||||
|
// TestAddTopologyRespectsDisabledFlag: a claim that carries a stopped forward
|
||||||
|
// must not mark the new entry Active, or it would come back to life.
|
||||||
|
func TestAddTopologyRespectsDisabledFlag(t *testing.T) {
|
||||||
|
s := &State{Topology: map[string]*TopoEntry{}}
|
||||||
|
e := s.AddTopology(&Task{
|
||||||
|
ID: "t1", Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||||
|
Link: store.Link{RemotePort: 18081, Disabled: true},
|
||||||
|
}, "n1")
|
||||||
|
if e.Active {
|
||||||
|
t.Fatal("AddTopology forced Active=true for a disabled claim — the forward would resurrect")
|
||||||
|
}
|
||||||
|
if !e.Link.Disabled {
|
||||||
|
t.Fatal("AddTopology dropped the disabled flag from the link")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestartTaskBypassesDuplicateClaimGuard is the engine half of the
|
||||||
|
// re-enable path: a restart task for a forward that still HAS a topology entry
|
||||||
|
// must reach Handler.Restart.
|
||||||
|
//
|
||||||
|
// This is precisely what the duplicate-claim guard refuses — it exists to stop
|
||||||
|
// a stale task from spawning a second worker for an owned forward, and a restart
|
||||||
|
// is indistinguishable from that unless it is an explicit task kind. Marking
|
||||||
|
// the entry stopped (instead of deleting it) is what made this necessary: the
|
||||||
|
// entry the guard keys on now outlives a stop.
|
||||||
|
func TestRestartTaskBypassesDuplicateClaimGuard(t *testing.T) {
|
||||||
|
restarts := 0
|
||||||
|
claims := 0
|
||||||
|
eng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||||
|
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||||
|
claim: func(ctx context.Context, tk *Task) error { claims++; return nil },
|
||||||
|
restart: func(ctx context.Context, tk *Task) error { restarts++; return nil }},
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||||
|
"n1:7500", true, "")
|
||||||
|
|
||||||
|
// Establish + stop: entry survives, marked disabled.
|
||||||
|
eng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 1, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
eng.state.AddRevoke(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 2, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); !d {
|
||||||
|
t.Fatal("precondition: should be disabled")
|
||||||
|
}
|
||||||
|
entryCount := len(eng.state.TopologyList())
|
||||||
|
|
||||||
|
// Start: publish a restart and run a cycle.
|
||||||
|
eng.state.UpdateTopologyDisabled("web", "frps1", 18081, false)
|
||||||
|
eng.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||||
|
store.Link{RemotePort: 18081})
|
||||||
|
if _, err := eng.OnToken(context.Background(), &Token{Cycle: 3, State: eng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if restarts != 1 {
|
||||||
|
t.Fatalf("Handler.Restart called %d times, want 1 — the restart task was swallowed "+
|
||||||
|
"by the duplicate-claim guard, so the forward would never come back", restarts)
|
||||||
|
}
|
||||||
|
if claims != 1 {
|
||||||
|
t.Fatalf("Handler.Claim called %d times, want 1 (the original claim only); "+
|
||||||
|
"a restart must not re-run the claim bookkeeping", claims)
|
||||||
|
}
|
||||||
|
// Re-enabling must not create a second entry.
|
||||||
|
if n := len(eng.state.TopologyList()); n != entryCount {
|
||||||
|
t.Fatalf("restart created a duplicate topology entry: %d -> %d", entryCount, n)
|
||||||
|
}
|
||||||
|
if d, _ := eng.state.TopologyDisabled("web", "frps1", 18081); d {
|
||||||
|
t.Fatal("entry is still disabled after the restart task was applied")
|
||||||
|
}
|
||||||
|
var sawStart bool
|
||||||
|
for _, e := range eng.Log.Snapshot() {
|
||||||
|
if e.Kind == LogForwardStart {
|
||||||
|
sawStart = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !sawStart {
|
||||||
|
t.Fatalf("log missing forward.start: %+v", eng.Log.Snapshot())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestartFlagSurvivesTokenSerialization: the restart task travels to the
|
||||||
|
// owner inside the token, so the flag must round-trip through JSON. Without the
|
||||||
|
// struct tag it would silently deserialize as false and the owner would treat
|
||||||
|
// it as an ordinary claim — which the duplicate guard then discards.
|
||||||
|
func TestRestartFlagSurvivesTokenSerialization(t *testing.T) {
|
||||||
|
eng := newTestEngine("n1", true)
|
||||||
|
eng.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||||
|
store.Link{RemotePort: 18081})
|
||||||
|
|
||||||
|
blob, err := json.Marshal(&Token{Cycle: 7, State: eng.state})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var back Token
|
||||||
|
if err := json.Unmarshal(blob, &back); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var found *Task
|
||||||
|
for _, tk := range back.State.PendingList() {
|
||||||
|
if tk.Local.Name == "web" && tk.Link.RemotePort == 18081 {
|
||||||
|
found = tk
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if found == nil {
|
||||||
|
t.Fatal("restart task did not survive token serialization")
|
||||||
|
}
|
||||||
|
if !found.Restart {
|
||||||
|
t.Fatal("the restart flag was lost in JSON — the owner would see a plain claim " +
|
||||||
|
"and the duplicate guard would discard it")
|
||||||
|
}
|
||||||
|
// Revoke and Restart must stay distinguishable.
|
||||||
|
if found.Revoke {
|
||||||
|
t.Fatal("a restart task must not also read as a revocation")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestartTaskReachesNonLocalOwner.
|
||||||
|
func TestRestartOnlyAppliedByOwner(t *testing.T) {
|
||||||
|
restarts := 0
|
||||||
|
h := &fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||||
|
restart: func(ctx context.Context, tk *Task) error { restarts++; return nil }}
|
||||||
|
|
||||||
|
// Owner node claims the forward.
|
||||||
|
owner := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil, h,
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil }, "n1:7500", true, "")
|
||||||
|
owner.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := owner.OnToken(context.Background(), &Token{Cycle: 1, State: owner.state, SentAt: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(owner.state.TopologyList()) != 1 {
|
||||||
|
t.Fatalf("precondition: owner should hold the entry, got %+v", owner.state.TopologyList())
|
||||||
|
}
|
||||||
|
|
||||||
|
// A peer adopts the same ring state but does not own the forward.
|
||||||
|
peer := NewEngine("n2", "n2:7500", "u", "p", "0.71.0", nil, h,
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil }, "n2:7500", false, "")
|
||||||
|
peer.AdoptState(owner.state)
|
||||||
|
tk := peer.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||||
|
store.Link{RemotePort: 18081})
|
||||||
|
if tk == nil {
|
||||||
|
t.Fatal("SubmitRestart returned nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The peer processes one token: it must NOT apply the restart.
|
||||||
|
if _, err := peer.OnToken(context.Background(), &Token{Cycle: 2, State: peer.state, SentAt: 2}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if restarts != 0 {
|
||||||
|
t.Fatalf("a NON-owner applied the restart %d time(s) — it would spawn an orphaned "+
|
||||||
|
"worker for a forward owned by somebody else", restarts)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRestartTaskReachesNonLocalOwner: a restart published on a node that is NOT
|
||||||
|
// the owner must survive the token round-trip and be applied by the owner.
|
||||||
|
//
|
||||||
|
// The naive "if not owner then continue" implementation CONSUMED the task
|
||||||
|
// (ClaimPending had already removed it), so the owner never received it and the
|
||||||
|
// forward stayed stopped with nothing logged as an error. Deferral must
|
||||||
|
// re-queue it.
|
||||||
|
//
|
||||||
|
// Each OnToken is fed a FRESH token (as the real ring does — the successor's
|
||||||
|
// OnToken receives the state the predecessor returned), so the deferral is
|
||||||
|
// exercised once per hop rather than re-processing one snapshot.
|
||||||
|
func TestRestartTaskReachesNonLocalOwner(t *testing.T) {
|
||||||
|
ownerApplied, submitterApplied := 0, 0
|
||||||
|
ownerEng := NewEngine("n2", "n2:7500", "u", "p", "0.71.0", nil,
|
||||||
|
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||||
|
restart: func(ctx context.Context, tk *Task) error { ownerApplied++; return nil }},
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil }, "n2:7500", false, "")
|
||||||
|
|
||||||
|
ownerEng.state.AddPending(store.Local{Name: "web"}, store.Remote{Name: "frps1"}, store.Link{RemotePort: 18081})
|
||||||
|
if _, err := ownerEng.OnToken(context.Background(), &Token{Cycle: 1, State: ownerEng.state}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(ownerEng.state.TopologyList()) != 1 {
|
||||||
|
t.Fatalf("precondition: n2 should own it, got %+v", ownerEng.state.TopologyList())
|
||||||
|
}
|
||||||
|
|
||||||
|
subEng := NewEngine("n1", "n1:7500", "u", "p", "0.71.0", nil,
|
||||||
|
&fakeHandler{load: Load{MemPct: 5, NetPct: 5},
|
||||||
|
restart: func(ctx context.Context, tk *Task) error { submitterApplied++; return nil }},
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil }, "n1:7500", true, "")
|
||||||
|
subEng.AdoptState(ownerEng.state)
|
||||||
|
|
||||||
|
subEng.SubmitRestart(store.Local{Name: "web"}, store.Remote{Name: "frps1"},
|
||||||
|
store.Link{RemotePort: 18081})
|
||||||
|
|
||||||
|
// One hop through the non-owner: it must defer, not apply and not drop.
|
||||||
|
out, err := subEng.OnToken(context.Background(), &Token{Cycle: 2, State: subEng.state, SentAt: 1})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if submitterApplied != 0 {
|
||||||
|
t.Fatal("the non-owner applied a restart for a forward it does not own")
|
||||||
|
}
|
||||||
|
var carried *Task
|
||||||
|
for _, tk := range out.State.PendingList() {
|
||||||
|
if tk.Restart && tk.Local.Name == "web" {
|
||||||
|
carried = tk
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if carried == nil {
|
||||||
|
t.Fatal("the non-owner CONSUMED the restart task; the owner would never receive it")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The owner applies the carried task.
|
||||||
|
if _, err := ownerEng.OnToken(context.Background(), &Token{Cycle: 3, State: out.State, SentAt: 2}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ownerApplied != 1 {
|
||||||
|
t.Fatalf("the owner applied the restart %d times, want 1", ownerApplied)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
190
internal/cluster/ring_splitbrain_test.go
Normal file
190
internal/cluster/ring_splitbrain_test.go
Normal file
@ -0,0 +1,190 @@
|
|||||||
|
package cluster
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"webui4frpc/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ringOf builds a 3-node ring where n1 is the leader.
|
||||||
|
func ringOf(t *testing.T) (n1, n2, n3 *Engine) {
|
||||||
|
mk := func(id string) *Engine {
|
||||||
|
return NewEngine(id, id+":7500", "u", "p", "0.1.2", nil,
|
||||||
|
&fakeHandler{load: Load{MemPct: 5, NetPct: 5}},
|
||||||
|
func(ctx context.Context, next string, tk *Token) error { return nil },
|
||||||
|
id+":7500", true, "")
|
||||||
|
}
|
||||||
|
n1, n2, n3 = mk("n1"), mk("n2"), mk("n3")
|
||||||
|
seed := State{
|
||||||
|
LeaderID: "n1",
|
||||||
|
Nodes: []Node{
|
||||||
|
{ID: "n1", Addr: "n1:7500", Alive: true, IsLeader: true},
|
||||||
|
{ID: "n2", Addr: "n2:7500", Alive: true},
|
||||||
|
{ID: "n3", Addr: "n3:7500", Alive: true},
|
||||||
|
},
|
||||||
|
Topology: map[string]*TopoEntry{},
|
||||||
|
PendingTasks: map[string]*Task{},
|
||||||
|
}
|
||||||
|
for _, e := range []*Engine{n1, n2, n3} {
|
||||||
|
// Round-trip through JSON so each engine holds its OWN Nodes slice and
|
||||||
|
// maps. AdoptState assigns the state by value, but Nodes is a slice, so
|
||||||
|
// handing all three the same seed would alias one backing array and make
|
||||||
|
// a mutation on one node visible on the others — an artefact of the test
|
||||||
|
// harness, not of the real transport (each process deserializes its own
|
||||||
|
// copy from the token).
|
||||||
|
blob, err := json.Marshal(seed)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var own State
|
||||||
|
if err := json.Unmarshal(blob, &own); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
e.AdoptState(own)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBecomeLeaderDeclinesToLiveLeader is the regression test for the live split
|
||||||
|
// brain.
|
||||||
|
//
|
||||||
|
// Leadership used to be self-declared: any node that failed to reach the leader
|
||||||
|
// it believed in promoted itself immediately. Both sides of a partition take that
|
||||||
|
// path, so two rings ran concurrently and each emptied the other's topology into
|
||||||
|
// pending — observed live as two nodes naming the same leader while marking each
|
||||||
|
// other dead, with pending=3 and topology=0 on both.
|
||||||
|
//
|
||||||
|
// A node must therefore consult the ring before promoting itself.
|
||||||
|
func TestBecomeLeaderDeclinesToLiveLeader(t *testing.T) {
|
||||||
|
_, n2, _ := ringOf(t)
|
||||||
|
// n3 has already promoted itself and n2 has learned that view: n1 is dead,
|
||||||
|
// n3 is alive and leading. n2 cannot reach n3 (that is why it is asking).
|
||||||
|
for i := range n2.state.Nodes {
|
||||||
|
switch n2.state.Nodes[i].ID {
|
||||||
|
case "n1":
|
||||||
|
n2.state.Nodes[i].Alive = false
|
||||||
|
n2.state.Nodes[i].IsLeader = false
|
||||||
|
case "n3":
|
||||||
|
n2.state.Nodes[i].Alive = true
|
||||||
|
n2.state.Nodes[i].IsLeader = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
n2.state.LeaderID = "n3"
|
||||||
|
|
||||||
|
if n2.becomeLeader() {
|
||||||
|
t.Fatal("n2 promoted itself while n3 was alive and leading in its view — " +
|
||||||
|
"two leaders is exactly the split brain this guards against")
|
||||||
|
}
|
||||||
|
if n2.state.LeaderID == "n2" {
|
||||||
|
t.Fatal("LeaderID was overwritten despite declining")
|
||||||
|
}
|
||||||
|
if n2.state.Nodes[0].IsLeader && n2.state.Nodes[0].ID == "n2" {
|
||||||
|
t.Fatal("n2 marked itself leader despite declining")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBecomeLeaderPromotesWhenSoleSurvivor: the guard must not break genuine
|
||||||
|
// failover. With every other node dead, promotion has to happen.
|
||||||
|
func TestBecomeLeaderPromotesWhenSoleSurvivor(t *testing.T) {
|
||||||
|
_, n2, _ := ringOf(t)
|
||||||
|
for _, id := range []string{"n1", "n3"} {
|
||||||
|
n2.state.MarkOffline(id)
|
||||||
|
}
|
||||||
|
if !n2.becomeLeader() {
|
||||||
|
t.Fatal("n2 declined promotion with every other node dead — that breaks failover")
|
||||||
|
}
|
||||||
|
if n2.state.LeaderID != "n2" {
|
||||||
|
t.Fatalf("leader=%q, want n2", n2.state.LeaderID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTakeoverLeavesLiveLeadersForwardsAlone pins the second half of the fix.
|
||||||
|
//
|
||||||
|
// When a heartbeat fails, the node used to mark the leader dead and re-queue ALL
|
||||||
|
// of that leader's forwards unconditionally — even when the ring still showed a
|
||||||
|
// live leader, i.e. this node was merely the partitioned side. That is what
|
||||||
|
// emptied a working ring: both sides ended up with topology=0 and pending=3.
|
||||||
|
//
|
||||||
|
// Now the re-hang happens only if we actually take leadership.
|
||||||
|
func TestTakeoverLeavesLiveLeadersForwardsAlone(t *testing.T) {
|
||||||
|
_, n2, _ := ringOf(t)
|
||||||
|
// n1 (the leader n2 believes in) owns a forward.
|
||||||
|
n2.state.AddTopology(&Task{
|
||||||
|
ID: "t1",
|
||||||
|
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||||
|
Link: store.Link{RemotePort: 18081},
|
||||||
|
}, "n1")
|
||||||
|
before := len(n2.state.TopologyList())
|
||||||
|
pendingBefore := len(n2.state.PendingList())
|
||||||
|
|
||||||
|
// n2 still believes n1 is the leader and cannot reach it, so it tries to take
|
||||||
|
// over. But its state ALSO lists n3 as alive-and-leader — the view it learned
|
||||||
|
// when n3 promoted itself. Two claims, one ring: n2 is the partitioned side
|
||||||
|
// and must stand down.
|
||||||
|
for i := range n2.state.Nodes {
|
||||||
|
switch n2.state.Nodes[i].ID {
|
||||||
|
case "n1":
|
||||||
|
n2.state.Nodes[i].Alive = true // still marked alive; MarkOffline happens inside
|
||||||
|
n2.state.Nodes[i].IsLeader = true
|
||||||
|
case "n3":
|
||||||
|
n2.state.Nodes[i].Alive = true
|
||||||
|
n2.state.Nodes[i].IsLeader = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
n2.state.LeaderID = "n1"
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
n2.takeOverFromDeadLeader(ctx)
|
||||||
|
|
||||||
|
if n2.state.LeaderID == "n2" {
|
||||||
|
t.Error("n2 took over while a live leader existed")
|
||||||
|
}
|
||||||
|
if after := len(n2.state.TopologyList()); after != before {
|
||||||
|
t.Errorf("a declined takeover still re-queued the live leader's forwards: topology %d -> %d",
|
||||||
|
before, after)
|
||||||
|
}
|
||||||
|
if p := len(n2.state.PendingList()); p != pendingBefore {
|
||||||
|
t.Errorf("a declined takeover still pushed work into pending: %d -> %d", pendingBefore, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTakeoverReassignsWhenSoleSurvivor is the other half: genuine failover must
|
||||||
|
// still work, otherwise the guard would have broken the design's promise that a
|
||||||
|
// dead leader is replaced and its forwards re-hung.
|
||||||
|
func TestTakeoverReassignsWhenSoleSurvivor(t *testing.T) {
|
||||||
|
_, n2, _ := ringOf(t)
|
||||||
|
n2.state.AddTopology(&Task{
|
||||||
|
ID: "t1",
|
||||||
|
Local: store.Local{Name: "web"}, Remote: store.Remote{Name: "frps1"},
|
||||||
|
Link: store.Link{RemotePort: 18081},
|
||||||
|
}, "n1") // owned by the leader n1
|
||||||
|
|
||||||
|
// n1 is the leader and is dead; n3 is dead too. n2 is the sole survivor.
|
||||||
|
for _, id := range []string{"n1", "n3"} {
|
||||||
|
n2.state.MarkOffline(id)
|
||||||
|
}
|
||||||
|
n2.state.Nodes[0].IsLeader = false // n1 no longer claims leadership
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
n2.takeOverFromDeadLeader(ctx)
|
||||||
|
|
||||||
|
if n2.state.LeaderID != "n2" {
|
||||||
|
t.Fatalf("n2 did not take over as the sole survivor, leader=%q", n2.state.LeaderID)
|
||||||
|
}
|
||||||
|
if len(n2.state.TopologyList()) != 0 {
|
||||||
|
t.Error("the dead leader's forwards should have been re-hung")
|
||||||
|
}
|
||||||
|
found := false
|
||||||
|
for _, tk := range n2.state.PendingList() {
|
||||||
|
if tk.Local.Name == "web" {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Error("the dead leader's forward was not re-queued for another node")
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -8,7 +8,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
@ -52,11 +51,16 @@ func (t *TokenTransport) SendTo(getAddr func(nodeID string) string) func(ctx con
|
|||||||
start := time.Now()
|
start := time.Now()
|
||||||
resp, err := cli.Do(req)
|
resp, err := cli.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("token-send %s: size=%dB elapsed=%v err=%v", addr, len(body), time.Since(start).Round(time.Millisecond), err)
|
// NOT demoted: a failed send is the "neighbor offline" signal the
|
||||||
|
// ring's fault paths are diagnosed from.
|
||||||
|
logf("token-send %s: size=%dB elapsed=%v err=%v", addr, len(body), time.Since(start).Round(time.Millisecond), err)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
log.Printf("token-send %s: size=%dB elapsed=%v -> %d", addr, len(body), time.Since(start).Round(time.Millisecond), resp.StatusCode)
|
// Success path is a per-round heartbeat (size/elapsed/200 repeat
|
||||||
|
// verbatim every cycle) — debug only. The status-code check below stays
|
||||||
|
// loud on purpose, so a non-2xx still shows up without the debug flag.
|
||||||
|
debugToken("token-send %s: size=%dB elapsed=%v -> %d", addr, len(body), time.Since(start).Round(time.Millisecond), resp.StatusCode)
|
||||||
if resp.StatusCode >= 400 {
|
if resp.StatusCode >= 400 {
|
||||||
return fmt.Errorf("token POST %s -> HTTP %d", url, resp.StatusCode)
|
return fmt.Errorf("token POST %s -> HTTP %d", url, resp.StatusCode)
|
||||||
}
|
}
|
||||||
|
|||||||
204
internal/httpapi/forward_revoke_test.go
Normal file
204
internal/httpapi/forward_revoke_test.go
Normal file
@ -0,0 +1,204 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"webui4frpc/internal/cluster"
|
||||||
|
"webui4frpc/internal/process"
|
||||||
|
"webui4frpc/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newRingTestHandler builds a Handler WITH a ring engine attached, so the
|
||||||
|
// paths that publish tasks into the token actually execute. newTestHandler
|
||||||
|
// leaves Ring nil, which silently skips them — a test built on it can pass
|
||||||
|
// while the publish side is completely broken.
|
||||||
|
func newRingTestHandler(t *testing.T) (*Handler, *cluster.Engine, *httptest.Server) {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
st, err := store.New(filepath.Join(dir, "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = st.Close() })
|
||||||
|
|
||||||
|
pm := process.NewManager(process.Options{
|
||||||
|
ConfigsDir: filepath.Join(dir, "configs"),
|
||||||
|
LogsDir: filepath.Join(dir, "logs"),
|
||||||
|
BinaryPath: func() string { return "" },
|
||||||
|
Render: func(string) ([]byte, error) { return []byte(`{}`), nil },
|
||||||
|
AutoRestart: func(string) bool { return false },
|
||||||
|
RestartInterval: func() int { return 5 },
|
||||||
|
})
|
||||||
|
ring := cluster.NewEngine("n1", "n1:7500", "u", "p", "0.1.0", nil,
|
||||||
|
&cluster.AppHandler{},
|
||||||
|
func(ctx context.Context, next string, tk *cluster.Token) error { return nil },
|
||||||
|
"n1:7500", true, "")
|
||||||
|
h := &Handler{Store: st, Process: pm, WorkDir: dir, User: "admin", Password: "pw", Ring: ring}
|
||||||
|
mux, err := NewServeMux(h)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ts := httptest.NewServer(mux)
|
||||||
|
t.Cleanup(ts.Close)
|
||||||
|
return h, ring, ts
|
||||||
|
}
|
||||||
|
|
||||||
|
func saveCanvas(t *testing.T, srv *httptest.Server, body string) {
|
||||||
|
t.Helper()
|
||||||
|
req, _ := http.NewRequest(http.MethodPut, srv.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||||
|
req.SetBasicAuth("admin", "pw")
|
||||||
|
resp, err := srv.Client().Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("canvas save status = %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStopForwardPublishesDisabledFlagInRevokeTask is the regression test for
|
||||||
|
// the actual defect.
|
||||||
|
//
|
||||||
|
// stopForward() read the link, called SetLinkDisabled(true), and then handed
|
||||||
|
// the STALE copy (disabled=false) to RevokeTask. The revoke travels to the
|
||||||
|
// node that OWNS the forward, and that node's Claim/Revoke path keys off the
|
||||||
|
// flag — so a stale false meant:
|
||||||
|
// - the owner could not tell the stop was deliberate, and
|
||||||
|
// - nothing retired the topology entry,
|
||||||
|
//
|
||||||
|
// so the next restart's reconcile re-claimed the forward and spawned a worker
|
||||||
|
// for something the user had stopped (seen live: endless connection-refused
|
||||||
|
// against an intentionally-down service).
|
||||||
|
//
|
||||||
|
// This asserts the flag ON THE PUBLISHED TASK, which is the value that was
|
||||||
|
// actually wrong. It cannot be satisfied by the store write alone.
|
||||||
|
func TestStopForwardPublishesDisabledFlagInRevokeTask(t *testing.T) {
|
||||||
|
_, ring, ts := newRingTestHandler(t)
|
||||||
|
|
||||||
|
saveCanvas(t, ts, `{
|
||||||
|
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||||
|
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||||
|
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||||
|
}`)
|
||||||
|
|
||||||
|
// Stop the forward over the API.
|
||||||
|
b, _ := json.Marshal(stopForwardReq{"svc", "srv-a", 45999})
|
||||||
|
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/api/manager/forwards/stop", bytes.NewReader(b))
|
||||||
|
req.SetBasicAuth("admin", "pw")
|
||||||
|
resp, err := ts.Client().Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("stop status = %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find the revoke task that was published into the token.
|
||||||
|
var revoke *cluster.Task
|
||||||
|
for _, tk := range ring.State().PendingList() {
|
||||||
|
if tk.Revoke && tk.Local.Name == "svc" && tk.Link.RemotePort == 45999 {
|
||||||
|
revoke = tk
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if revoke == nil {
|
||||||
|
t.Fatal("stop did not publish a revoke task for the forward")
|
||||||
|
}
|
||||||
|
if !revoke.Link.Disabled {
|
||||||
|
t.Fatal("the published revoke task carries disabled=false — the owner node cannot tell " +
|
||||||
|
"this stop was deliberate, which is the bug that let stopped forwards resurrect")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStopThenStartPublishesRestartTask guards the failure mode that
|
||||||
|
// mark-don't-remove introduces, and that only shows up on the WIRE.
|
||||||
|
//
|
||||||
|
// A stop keeps the topology entry (it is the carrier for the flag), so on
|
||||||
|
// re-enable:
|
||||||
|
// - SubmitTask dedupes, because the entry exists;
|
||||||
|
// - the claim path discards any task for a forward that already has an owner.
|
||||||
|
//
|
||||||
|
// Both channels therefore refuse, no task is published, the owner never learns
|
||||||
|
// about the re-enable, and the forward stays stopped forever — a forward the
|
||||||
|
// user can stop but never restart.
|
||||||
|
//
|
||||||
|
// Asserting the task is PUBLISHED is the point: asserting only that the flag
|
||||||
|
// cleared would pass while nothing reached the owner. (The earlier version of
|
||||||
|
// this test did exactly that and stayed green against the broken code.)
|
||||||
|
func TestStopThenStartPublishesRestartTask(t *testing.T) {
|
||||||
|
_, ring, ts := newRingTestHandler(t)
|
||||||
|
|
||||||
|
saveCanvas(t, ts, `{
|
||||||
|
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||||
|
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||||
|
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||||
|
}`)
|
||||||
|
// Claim it so a topology entry exists (that is what makes the two normal
|
||||||
|
// channels refuse later).
|
||||||
|
if _, err := ring.OnToken(context.Background(), &cluster.Token{Cycle: 1, State: *ring.State()}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !ring.HasActiveTask("svc", "srv-a", 45999) {
|
||||||
|
t.Fatalf("precondition: forward should be active, topo=%+v", ring.State().TopologyList())
|
||||||
|
}
|
||||||
|
|
||||||
|
forwards := func(action string) {
|
||||||
|
t.Helper()
|
||||||
|
b, _ := json.Marshal(stopForwardReq{"svc", "srv-a", 45999})
|
||||||
|
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/api/manager/forwards/"+action, bytes.NewReader(b))
|
||||||
|
req.SetBasicAuth("admin", "pw")
|
||||||
|
resp, err := ts.Client().Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("%s status = %d", action, resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- stop: entry kept, marked disabled -------------------------------
|
||||||
|
forwards("stop")
|
||||||
|
if d, known := ring.TopologyDisabled("svc", "srv-a", 45999); !known || !d {
|
||||||
|
t.Fatalf("stop did not mark the topology entry disabled (known=%v disabled=%v)", known, d)
|
||||||
|
}
|
||||||
|
if ring.HasActiveTask("svc", "srv-a", 45999) {
|
||||||
|
t.Fatal("a stopped forward must not count as active")
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- start: a task MUST reach the owner ------------------------------
|
||||||
|
forwards("start")
|
||||||
|
if d, _ := ring.TopologyDisabled("svc", "srv-a", 45999); d {
|
||||||
|
t.Fatal("start did not clear the disabled flag")
|
||||||
|
}
|
||||||
|
if !ring.HasActiveTask("svc", "srv-a", 45999) {
|
||||||
|
t.Fatal("after start the forward must be active again")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The actual regression: something has to be queued for the owner. The
|
||||||
|
// re-enable must be published as a restart task, since a plain creation task
|
||||||
|
// would be deduped or discarded.
|
||||||
|
var restart *cluster.Task
|
||||||
|
for _, tk := range ring.State().PendingList() {
|
||||||
|
if tk.Restart && tk.Local.Name == "svc" && tk.Link.RemotePort == 45999 {
|
||||||
|
restart = tk
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if restart == nil {
|
||||||
|
t.Fatalf("start published no restart task — the owner would never bring the "+
|
||||||
|
"worker back, leaving the forward permanently stopped (pending=%+v)",
|
||||||
|
ring.State().PendingList())
|
||||||
|
}
|
||||||
|
if restart.Link.Disabled {
|
||||||
|
t.Fatal("the restart task carries disabled=true, so the owner would re-apply the stop")
|
||||||
|
}
|
||||||
|
}
|
||||||
141
internal/httpapi/forward_stop_test.go
Normal file
141
internal/httpapi/forward_stop_test.go
Normal file
@ -0,0 +1,141 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// stopForwardReq mirrors the forwards start/stop request body.
|
||||||
|
type stopForwardReq struct {
|
||||||
|
Local string `json:"local"`
|
||||||
|
Remote string `json:"remote"`
|
||||||
|
RemotePort int `json:"remotePort"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func postForwards(t *testing.T, srv *httptest.Server, action string, body stopForwardReq) int {
|
||||||
|
t.Helper()
|
||||||
|
b, _ := json.Marshal(body)
|
||||||
|
req, _ := http.NewRequest(http.MethodPost, srv.URL+"/api/manager/forwards/"+action, bytes.NewReader(b))
|
||||||
|
req.SetBasicAuth("admin", "pw")
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
resp, err := srv.Client().Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
return resp.StatusCode
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStopForwardPersistsDisabledFlag is the regression test for the
|
||||||
|
// "stopped forwards resurrect on restart" bug.
|
||||||
|
//
|
||||||
|
// The original defect: stopForward() read the link, called
|
||||||
|
// SetLinkDisabled(true), but then handed the STALE (disabled=false) copy to
|
||||||
|
// RevokeTask — so the disabled flag never reached the node owning the forward,
|
||||||
|
// and nothing removed the topology entry. On the next restart the startup
|
||||||
|
// reconcile saw an owned forward with no worker and re-claimed it, spawning a
|
||||||
|
// worker for a forward the user had deliberately stopped (observed live:
|
||||||
|
// ~14k "proxy already exists" retries and endless connection-refused against a
|
||||||
|
// service that was intentionally down).
|
||||||
|
//
|
||||||
|
// The contract this pins: after a successful stop, the persisted link MUST be
|
||||||
|
// disabled — that flag is the single source of truth the claim path consults.
|
||||||
|
func TestStopForwardPersistsDisabledFlag(t *testing.T) {
|
||||||
|
h, ts := newTestHandler(t)
|
||||||
|
|
||||||
|
body := `{
|
||||||
|
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||||
|
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||||
|
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||||
|
}`
|
||||||
|
req, _ := http.NewRequest(http.MethodPut, ts.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||||
|
req.SetBasicAuth("admin", "pw")
|
||||||
|
resp, err := ts.Client().Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("canvas save status = %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The forward starts enabled.
|
||||||
|
ln, found, err := h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||||
|
if err != nil || !found {
|
||||||
|
t.Fatalf("link not persisted: found=%v err=%v", found, err)
|
||||||
|
}
|
||||||
|
if ln.Disabled {
|
||||||
|
t.Fatal("a freshly saved forward must start enabled")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop it.
|
||||||
|
if code := postForwards(t, ts, "stop", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||||
|
t.Fatalf("stop status = %d, want 200", code)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Persisted flag must now be set — this is what the claim path reads.
|
||||||
|
ln, found, err = h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatal("link vanished after stop; stop must be non-destructive")
|
||||||
|
}
|
||||||
|
if !ln.Disabled {
|
||||||
|
t.Fatal("stop did not persist disabled=true — the startup reconcile would resurrect this forward")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start must clear it again (the user-facing re-enable path).
|
||||||
|
if code := postForwards(t, ts, "start", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||||
|
t.Fatalf("start status = %d, want 200", code)
|
||||||
|
}
|
||||||
|
ln, _, err = h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ln.Disabled {
|
||||||
|
t.Fatal("start did not clear disabled; a re-enabled forward would stay stopped")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStopForwardIsNonDestructive pins the per-forward stop semantics the
|
||||||
|
// revoke path was specifically rewritten for: stopping one forward must not
|
||||||
|
// touch a sibling forward that shares the same local or remote.
|
||||||
|
func TestStopForwardIsNonDestructive(t *testing.T) {
|
||||||
|
h, ts := newTestHandler(t)
|
||||||
|
|
||||||
|
body := `{
|
||||||
|
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||||
|
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||||
|
"links": [
|
||||||
|
{"local":"svc","remote":"srv-a","remotePort":45999},
|
||||||
|
{"local":"svc","remote":"srv-a","remotePort":46000}
|
||||||
|
]
|
||||||
|
}`
|
||||||
|
req, _ := http.NewRequest(http.MethodPut, ts.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||||
|
req.SetBasicAuth("admin", "pw")
|
||||||
|
resp, err := ts.Client().Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
|
||||||
|
if code := postForwards(t, ts, "stop", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||||
|
t.Fatalf("stop status = %d", code)
|
||||||
|
}
|
||||||
|
|
||||||
|
stopped, _, _ := h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||||
|
sibling, found, _ := h.Store.LinkByTriple("svc", "srv-a", 46000)
|
||||||
|
if !found {
|
||||||
|
t.Fatal("sibling forward was destroyed by stopping its neighbour")
|
||||||
|
}
|
||||||
|
if !stopped.Disabled {
|
||||||
|
t.Error("the stopped forward should be disabled")
|
||||||
|
}
|
||||||
|
if sibling.Disabled {
|
||||||
|
t.Error("the sibling forward must stay enabled — per-forward stop, not per-local/remote")
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -245,7 +245,12 @@ func (h *Handler) applyCanvas(w http.ResponseWriter, r *http.Request, canvas *ca
|
|||||||
}
|
}
|
||||||
if ln.Disabled {
|
if ln.Disabled {
|
||||||
// Stopped on the forwards page: make sure it leaves the topology.
|
// Stopped on the forwards page: make sure it leaves the topology.
|
||||||
if h.Ring.HasTask(ln.Local, ln.Remote, ln.RemotePort) {
|
// Guard on an ACTIVE task — an entry that is already marked
|
||||||
|
// disabled has nothing left to revoke, and re-issuing a revocation
|
||||||
|
// for it on every canvas save would be pure noise. (HasTask, which
|
||||||
|
// also matches disabled entries, is the right predicate for the
|
||||||
|
// "already handled" question this branch is not asking.)
|
||||||
|
if h.Ring.HasActiveTask(ln.Local, ln.Remote, ln.RemotePort) {
|
||||||
h.Ring.RevokeTask(loc, rem, ln)
|
h.Ring.RevokeTask(loc, rem, ln)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
|
|||||||
@ -84,6 +84,25 @@ func (h *Handler) startForward(local, remote string, port int) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
if h.Ring != nil {
|
||||||
|
// Publish the enable into the topology so it rides the ring (a peer that
|
||||||
|
// still holds the stale "stopped" copy learns about it on adoption).
|
||||||
|
h.Ring.UpdateTopologyDisabled(local, remote, port, false)
|
||||||
|
}
|
||||||
|
// A cluster forward whose topology entry still exists needs its OWNER to
|
||||||
|
// bring the worker back, and neither of the normal channels can do it:
|
||||||
|
// - SubmitTask is idempotency-guarded, and the entry still exists (a stop
|
||||||
|
// keeps it as the flag's carrier), so the submission is deduped away;
|
||||||
|
// - the claim path drops any task for a forward that already has an
|
||||||
|
// owner, as a defence against duplicate-claim collisions.
|
||||||
|
// So a re-enable of an existing entry is published as a dedicated RESTART
|
||||||
|
// task, which the owner applies unconditionally. Without it a forward the
|
||||||
|
// user can stop but not restart — which is what marking-instead-of-removing
|
||||||
|
// would otherwise have produced.
|
||||||
|
if h.Ring != nil && h.Ring.HasTopologyEntry(local, remote, port) {
|
||||||
|
h.Ring.SubmitRestart(loc, rem, ln)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
if h.Ring != nil {
|
if h.Ring != nil {
|
||||||
h.Ring.SubmitTask(loc, rem, ln)
|
h.Ring.SubmitTask(loc, rem, ln)
|
||||||
}
|
}
|
||||||
@ -109,6 +128,24 @@ func (h *Handler) stopForward(local, remote string, port int) error {
|
|||||||
} else {
|
} else {
|
||||||
_ = h.Store.SetLinkDisabled(local, remote, port, true)
|
_ = h.Store.SetLinkDisabled(local, remote, port, true)
|
||||||
}
|
}
|
||||||
|
// The revoke task travels to whichever node OWNS the forward, and that node
|
||||||
|
// re-reads the disabled flag from its own store before starting a worker —
|
||||||
|
// so the flag has to be set on every node that has a copy of this link, not
|
||||||
|
// just the one handling this request. Propagating Disabled on the task lets
|
||||||
|
// the owner's RevokeFn stop the worker even if its own store row is stale.
|
||||||
|
//
|
||||||
|
// This also fixes a latent inconsistency: `ln` was read BEFORE the
|
||||||
|
// SetLinkDisabled(true) above, so the link published into the token still
|
||||||
|
// carried disabled=false and got copied into the topology entry verbatim.
|
||||||
|
ln.Disabled = true
|
||||||
|
// Publish the stop into the topology BEFORE revoking: the revoke retires the
|
||||||
|
// own-side worker/entry, so the flag must already exist somewhere that
|
||||||
|
// survives it and travels the ring. This is the send half of cluster-wide
|
||||||
|
// disabled propagation (see store.ReconcileLinkDisabled for the receive
|
||||||
|
// half).
|
||||||
|
if h.Ring != nil {
|
||||||
|
h.Ring.UpdateTopologyDisabled(local, remote, port, true)
|
||||||
|
}
|
||||||
if loc.LocalOnly {
|
if loc.LocalOnly {
|
||||||
if h.Process != nil {
|
if h.Process != nil {
|
||||||
key := process.WorkerKey(local, remote, port)
|
key := process.WorkerKey(local, remote, port)
|
||||||
|
|||||||
@ -48,6 +48,8 @@ type Handler struct {
|
|||||||
Cluster *cluster.Registry
|
Cluster *cluster.Registry
|
||||||
// Ring is the token-ring engine (M6). Nil disables ring routes.
|
// Ring is the token-ring engine (M6). Nil disables ring routes.
|
||||||
Ring *cluster.Engine
|
Ring *cluster.Engine
|
||||||
|
// Balancer is optional: when nil the balance endpoint reports 404.
|
||||||
|
Balancer *cluster.Balancer
|
||||||
// SelfAddr is this node's reachable listen address (from -addr), used for
|
// SelfAddr is this node's reachable listen address (from -addr), used for
|
||||||
// cluster discovery so peers can reach back for binary exchange.
|
// cluster discovery so peers can reach back for binary exchange.
|
||||||
SelfAddr string
|
SelfAddr string
|
||||||
@ -114,6 +116,8 @@ func NewServeMux(h *Handler) (http.Handler, error) {
|
|||||||
mux.HandleFunc(apiPrefix+"/cluster/nodes", h.auth("read")(h.handleClusterNodes))
|
mux.HandleFunc(apiPrefix+"/cluster/nodes", h.auth("read")(h.handleClusterNodes))
|
||||||
mux.HandleFunc(apiPrefix+"/cluster/cache", h.auth("read")(h.handleClusterCache)) // GET read; POST write-checked inside
|
mux.HandleFunc(apiPrefix+"/cluster/cache", h.auth("read")(h.handleClusterCache)) // GET read; POST write-checked inside
|
||||||
mux.HandleFunc(apiPrefix+"/cluster/ring", h.auth("read")(h.handleClusterRing))
|
mux.HandleFunc(apiPrefix+"/cluster/ring", h.auth("read")(h.handleClusterRing))
|
||||||
|
mux.HandleFunc(apiPrefix+"/cluster/balance", h.auth("read")(h.handleClusterBalance))
|
||||||
|
mux.HandleFunc(apiPrefix+"/cluster/balance/mode", h.auth("write")(h.handleClusterBalanceMode))
|
||||||
mux.HandleFunc(apiPrefix+"/node/logs", h.auth("read")(h.handleNodeLogs))
|
mux.HandleFunc(apiPrefix+"/node/logs", h.auth("read")(h.handleNodeLogs))
|
||||||
mux.HandleFunc(apiPrefix+"/cluster/logs/export", h.auth("read")(h.handleClusterLogsExport))
|
mux.HandleFunc(apiPrefix+"/cluster/logs/export", h.auth("read")(h.handleClusterLogsExport))
|
||||||
|
|
||||||
@ -634,6 +638,64 @@ func (h *Handler) handleClusterRing(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeJSON(w, http.StatusOK, h.Ring.Snapshot())
|
writeJSON(w, http.StatusOK, h.Ring.Snapshot())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handleClusterBalance reports the load balancer's read-only view: per-node
|
||||||
|
// scores, whether anything is over the threshold, and the migration it would
|
||||||
|
// perform. Publishing nothing is the point — an operator asking "is this cluster
|
||||||
|
// balanced?" currently gets silence from the log in the healthy case, which is
|
||||||
|
// indistinguishable from balancing being switched off.
|
||||||
|
func (h *Handler) handleClusterBalance(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method != http.MethodGet {
|
||||||
|
methodNotAllowed(w)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if h.Ring == nil {
|
||||||
|
http.Error(w, "ring engine not enabled", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if h.Balancer == nil {
|
||||||
|
http.Error(w, "load balancer not enabled", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
rep := h.Balancer.Report()
|
||||||
|
// CanControl is decided by the caller's own level rather than by the
|
||||||
|
// presence of a stored value, so a read-only principal is never shown a
|
||||||
|
// switch that would 403.
|
||||||
|
rep.CanControl = hasLevel(r, "write")
|
||||||
|
writeJSON(w, http.StatusOK, rep)
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleClusterBalanceMode switches load balancing on/off at runtime.
|
||||||
|
//
|
||||||
|
// It is per-node on purpose. Every member decides about its OWN forwards, so the
|
||||||
|
// switch belongs to each node: an operator may well want to turn releasing on
|
||||||
|
// everywhere, but the server should not pretend those are one atomic action.
|
||||||
|
// (A cluster-wide control can be layered on top later; the primitive here is the
|
||||||
|
// one that actually has an owner.)
|
||||||
|
//
|
||||||
|
// Write-level auth: releasing stops a worker and drops a live connection.
|
||||||
|
func (h *Handler) handleClusterBalanceMode(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method != http.MethodPost {
|
||||||
|
methodNotAllowed(w)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if h.Balancer == nil {
|
||||||
|
http.Error(w, "load balancer not enabled", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var req struct {
|
||||||
|
Mode string `json:"mode"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||||
|
http.Error(w, "parse json: "+err.Error(), http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.Balancer.SetModeFromUI(req.Mode); err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, h.Balancer.Report())
|
||||||
|
}
|
||||||
|
|
||||||
// handleClusterJoin accepts a newcomer join request: the target node inserts
|
// handleClusterJoin accepts a newcomer join request: the target node inserts
|
||||||
// the newcomer after itself in the ring and returns the updated ring state
|
// the newcomer after itself in the ring and returns the updated ring state
|
||||||
// for the newcomer to adopt.
|
// for the newcomer to adopt.
|
||||||
|
|||||||
288
internal/store/link_test.go
Normal file
288
internal/store/link_test.go
Normal file
@ -0,0 +1,288 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// seed inserts the local + remote rows a link's foreign keys require.
|
||||||
|
// (links.local / links.remote reference their own tables, so a link cannot
|
||||||
|
// exist on its own — the same reason ClaimFn upserts them before ReplaceLinks.)
|
||||||
|
func seed(t *testing.T, st *Store, locals []string, remote string) {
|
||||||
|
t.Helper()
|
||||||
|
for _, n := range locals {
|
||||||
|
if err := st.UpsertLocal(Local{Name: n, IP: "127.0.0.1", Port: 8080, Protocol: "tcp"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := st.UpsertRemote(Remote{Name: remote, IP: "1.2.3.4", Port: 7000, Token: "tok", Enabled: true}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLinkByTripleSurvivesReplaceLinks pins the reason LinkByTriple exists.
|
||||||
|
//
|
||||||
|
// ReplaceLinks() rewrites the whole table with DELETE + re-INSERT, so sqlite
|
||||||
|
// hands every row a FRESH autoincrement id. A Link captured before such a
|
||||||
|
// write (e.g. one riding inside a ring token) therefore carries an id that
|
||||||
|
// either matches a different forward or matches nothing. The natural key
|
||||||
|
// (local, remote, remotePort) is what every caller actually identifies a
|
||||||
|
// forward by, and it must survive those rewrites.
|
||||||
|
func TestLinkByTripleSurvivesReplaceLinks(t *testing.T) {
|
||||||
|
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
|
||||||
|
seed(t, st, []string{"alpha", "beta", "gamma"}, "srv")
|
||||||
|
links := []Link{
|
||||||
|
{Local: "alpha", Remote: "srv", RemotePort: 100},
|
||||||
|
{Local: "beta", Remote: "srv", RemotePort: 200},
|
||||||
|
{Local: "gamma", Remote: "srv", RemotePort: 300},
|
||||||
|
}
|
||||||
|
if err := st.ReplaceLinks(links); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Capture the ids as the ring would have them.
|
||||||
|
before := map[string]int64{}
|
||||||
|
all, err := st.ListLinks()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, l := range all {
|
||||||
|
before[l.Local] = l.ID
|
||||||
|
}
|
||||||
|
if len(before) != 3 {
|
||||||
|
t.Fatalf("expected 3 links, got %d", len(before))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rewrite the table (this is what saveCanvas and ClaimFn both do).
|
||||||
|
if err := st.ReplaceLinks(links); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
after, err := st.ListLinks()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(after) != 3 {
|
||||||
|
t.Fatalf("expected 3 links after rewrite, got %d", len(after))
|
||||||
|
}
|
||||||
|
|
||||||
|
// The natural key must still resolve to the right forward, with its
|
||||||
|
// disabled flag and group intact.
|
||||||
|
for _, l := range after {
|
||||||
|
if l.Disabled {
|
||||||
|
t.Errorf("link %s unexpectedly disabled after a plain rewrite", l.Local)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
got, found, err := st.LinkByTriple("beta", "srv", 200)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatal("LinkByTriple failed to find beta after ReplaceLinks")
|
||||||
|
}
|
||||||
|
if got.Local != "beta" || got.RemotePort != 200 {
|
||||||
|
t.Fatalf("LinkByTriple returned the wrong row: %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLinkByTripleNotFoundIsNotError documents the contract callers rely on:
|
||||||
|
// "no persisted opinion yet" is (Link{}, false, nil), not an error. A fresh
|
||||||
|
// claim of a link with no row must be allowed to start.
|
||||||
|
func TestLinkByTripleNotFoundIsNotError(t *testing.T) {
|
||||||
|
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
|
||||||
|
got, found, err := st.LinkByTriple("nope", "srv", 1234)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("missing link must not be an error, got %v", err)
|
||||||
|
}
|
||||||
|
if found {
|
||||||
|
t.Fatalf("missing link reported as found: %+v", got)
|
||||||
|
}
|
||||||
|
if got.Local != "" || got.RemotePort != 0 {
|
||||||
|
t.Fatalf("expected zero Link on miss, got %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLinkByTripleReadsDisabledFlag is the store-level half of the
|
||||||
|
// "stopped forwards resurrect on restart" bug: the claim path asks the store
|
||||||
|
// whether the user disabled this forward, so this lookup must return the flag
|
||||||
|
// as persisted.
|
||||||
|
func TestLinkByTripleReadsDisabledFlag(t *testing.T) {
|
||||||
|
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
|
||||||
|
seed(t, st, []string{"mc"}, "srv")
|
||||||
|
if err := st.ReplaceLinks([]Link{{Local: "mc", Remote: "srv", RemotePort: 25565, Group: "game"}}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := st.SetLinkDisabled("mc", "srv", 25565, true); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
ln, found, err := st.LinkByTriple("mc", "srv", 25565)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatal("expected to find the link")
|
||||||
|
}
|
||||||
|
if !ln.Disabled {
|
||||||
|
t.Fatal("expected Disabled=true to be visible through LinkByTriple")
|
||||||
|
}
|
||||||
|
if ln.Group != "game" {
|
||||||
|
t.Fatalf("group should survive, got %q", ln.Group)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ...and the user-facing start path must be able to clear it again.
|
||||||
|
if err := st.SetLinkDisabled("mc", "srv", 25565, false); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ln, _, _ := st.LinkByTriple("mc", "srv", 25565); ln.Disabled {
|
||||||
|
t.Fatal("expected Disabled=false after clearing")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGetLinkByIDIsStaleAfterReplaceLinks documents WHY callers must not use
|
||||||
|
// GetLink(id) with a previously captured id. It is not a fix — it is the trap
|
||||||
|
// being pinned shut, so the hazard stays visible if someone reintroduces it.
|
||||||
|
func TestGetLinkByIDIsStaleAfterReplaceLinks(t *testing.T) {
|
||||||
|
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
|
||||||
|
seed(t, st, []string{"alpha", "beta", "gamma"}, "srv")
|
||||||
|
if err := st.ReplaceLinks([]Link{
|
||||||
|
{Local: "alpha", Remote: "srv", RemotePort: 100},
|
||||||
|
{Local: "beta", Remote: "srv", RemotePort: 200},
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
all, _ := st.ListLinks()
|
||||||
|
var staleID int64
|
||||||
|
for _, l := range all {
|
||||||
|
if l.Local == "alpha" {
|
||||||
|
staleID = l.ID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := st.ReplaceLinks([]Link{
|
||||||
|
{Local: "alpha", Remote: "srv", RemotePort: 100},
|
||||||
|
{Local: "beta", Remote: "srv", RemotePort: 200},
|
||||||
|
{Local: "gamma", Remote: "srv", RemotePort: 300},
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The old id may still resolve, but to whatever row now occupies that
|
||||||
|
// id — which is exactly the silent-mis-target hazard. Assert that the
|
||||||
|
// natural key remains the only safe handle.
|
||||||
|
if ln, ok := st.GetLink(staleID); ok && ln.Local != "alpha" {
|
||||||
|
t.Logf("stale id %d now points at %q (hazard confirmed; use LinkByTriple)", staleID, ln.Local)
|
||||||
|
}
|
||||||
|
if got, found, _ := st.LinkByTriple("alpha", "srv", 100); !found || got.Local != "alpha" {
|
||||||
|
t.Fatalf("natural key must stay reliable, got %+v found=%v", got, found)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReconcileLinkDisabledLearnsPeerDecision is the store half of
|
||||||
|
// cluster-wide stop propagation. A node that did NOT serve the stop request has
|
||||||
|
// no reason to know about it, and its links table is node-local — so the flag
|
||||||
|
// arrives via the ring and lands here. The case that matters is the OWNER of a
|
||||||
|
// forward on a different machine: before this existed, that node's copy still
|
||||||
|
// read "enabled", so it kept (or re-spawned) the worker for a forward the user
|
||||||
|
// had explicitly stopped.
|
||||||
|
func TestReconcileLinkDisabledLearnsPeerDecision(t *testing.T) {
|
||||||
|
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
seed(t, st, []string{"mc"}, "srv")
|
||||||
|
if err := st.ReplaceLinks([]Link{{Local: "mc", Remote: "srv", RemotePort: 25565, Group: "game"}}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A peer's stop arrives.
|
||||||
|
if err := st.ReconcileLinkDisabled("mc", "srv", 25565, true); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ln, found, err := st.LinkByTriple("mc", "srv", 25565)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatal("link disappeared during reconcile")
|
||||||
|
}
|
||||||
|
if !ln.Disabled {
|
||||||
|
t.Fatal("the peer's stop did not land in the local store")
|
||||||
|
}
|
||||||
|
if ln.Group != "game" {
|
||||||
|
t.Fatalf("reconcile must not clobber other fields, group=%q", ln.Group)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A peer's re-enable arrives.
|
||||||
|
if err := st.ReconcileLinkDisabled("mc", "srv", 25565, false); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ln, _, _ := st.LinkByTriple("mc", "srv", 25565); ln.Disabled {
|
||||||
|
t.Fatal("the peer's re-enable did not land")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReconcileLinkDisabledCreatesPlaceholderForUnknownForward: a node that has
|
||||||
|
// never seen the forward still must remember that it is stopped, otherwise a
|
||||||
|
// later claim on that node would resurrect it.
|
||||||
|
func TestReconcileLinkDisabledCreatesPlaceholderForUnknownForward(t *testing.T) {
|
||||||
|
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
seed(t, st, []string{"ghost"}, "srv")
|
||||||
|
|
||||||
|
if err := st.ReconcileLinkDisabled("ghost", "srv", 9999, true); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ln, found, err := st.LinkByTriple("ghost", "srv", 9999)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatal("a stopped-but-unknown forward must be remembered, or a later claim resurrects it")
|
||||||
|
}
|
||||||
|
if !ln.Disabled {
|
||||||
|
t.Fatal("placeholder is not marked disabled")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReconcileLinkDisabledIgnoresEnableForUnknown: an enable for a forward this
|
||||||
|
// node has never seen must NOT create a row. Creating one would invent forwards
|
||||||
|
// out of ring state.
|
||||||
|
func TestReconcileLinkDisabledIgnoresEnableForUnknown(t *testing.T) {
|
||||||
|
st, err := New(filepath.Join(t.TempDir(), "test.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
seed(t, st, []string{"other"}, "srv")
|
||||||
|
|
||||||
|
if err := st.ReconcileLinkDisabled("unknown", "srv", 1234, false); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, found, _ := st.LinkByTriple("unknown", "srv", 1234); found {
|
||||||
|
t.Fatal("an enable for an unknown forward must not materialise a row")
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -128,6 +128,11 @@ type Settings struct {
|
|||||||
// key). Cleared on explicit detach (detachAsStandalone) so a node that
|
// key). Cleared on explicit detach (detachAsStandalone) so a node that
|
||||||
// intentionally left does NOT auto-rejoin.
|
// intentionally left does NOT auto-rejoin.
|
||||||
ClusterPeers string `json:"clusterPeers,omitempty"`
|
ClusterPeers string `json:"clusterPeers,omitempty"`
|
||||||
|
// BalanceMode is the load-balancing mode chosen in the UI: "", "off",
|
||||||
|
// "dry-run" or "on". Empty means "never configured here" so the environment
|
||||||
|
// variable stays the bootstrap default; a non-empty value wins over it,
|
||||||
|
// otherwise flipping the switch in the UI would silently revert on restart.
|
||||||
|
BalanceMode string `json:"balanceMode,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Forward is a rendered link row attached to a remote.
|
// Forward is a rendered link row attached to a remote.
|
||||||
@ -560,6 +565,30 @@ func (s *Store) GetLink(id int64) (Link, bool) {
|
|||||||
return l, true
|
return l, true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// LinkByTriple looks a link up by its natural key (local, remote, remotePort).
|
||||||
|
//
|
||||||
|
// Prefer this over GetLink(id) whenever the caller only knows the forward's
|
||||||
|
// identity: ReplaceLinks() rewrites the whole table with DELETE + re-INSERT, so
|
||||||
|
// every row gets a fresh autoincrement id. Any id captured before such a write
|
||||||
|
// (e.g. a Link carried inside a ring token) is stale by definition and will
|
||||||
|
// either miss or — worse — match a different forward. The natural key is
|
||||||
|
// stable across those rewrites.
|
||||||
|
//
|
||||||
|
// Returns (link, found). A missing row is (Link{}, false) and is NOT an error:
|
||||||
|
// callers use that to mean "no persisted opinion yet".
|
||||||
|
func (s *Store) LinkByTriple(local, remote string, port int) (Link, bool, error) {
|
||||||
|
var l Link
|
||||||
|
err := s.db.QueryRow("SELECT id, local, remote, remote_port, offset_x, offset_y, grp, disabled FROM links WHERE local = ? AND remote = ? AND remote_port = ?", local, remote, port).
|
||||||
|
Scan(&l.ID, &l.Local, &l.Remote, &l.RemotePort, &l.OffsetX, &l.OffsetY, &l.Group, &l.Disabled)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return Link{}, false, nil
|
||||||
|
}
|
||||||
|
return Link{}, false, err
|
||||||
|
}
|
||||||
|
return l, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
// DeleteLink removes a single link by id.
|
// DeleteLink removes a single link by id.
|
||||||
func (s *Store) DeleteLink(id int64) error {
|
func (s *Store) DeleteLink(id int64) error {
|
||||||
_, err := s.db.Exec("DELETE FROM links WHERE id = ?", id)
|
_, err := s.db.Exec("DELETE FROM links WHERE id = ?", id)
|
||||||
@ -639,6 +668,11 @@ func (s *Store) ReplaceLinks(links []Link) error {
|
|||||||
// SetLinkDisabled flips the disabled flag of a forward identified by its
|
// SetLinkDisabled flips the disabled flag of a forward identified by its
|
||||||
// (local, remote, remotePort) natural key. This is the persistence half of the
|
// (local, remote, remotePort) natural key. This is the persistence half of the
|
||||||
// forwards-page start/stop toggle; the caller also drives the worker/ring side.
|
// forwards-page start/stop toggle; the caller also drives the worker/ring side.
|
||||||
|
//
|
||||||
|
// Kept as a targeted UPDATE rather than a ReplaceLinks rewrite on purpose:
|
||||||
|
// ReplaceLinks deletes and reinserts every row, handing out fresh autoincrement
|
||||||
|
// ids and invalidating any Link a caller captured earlier (they travel inside
|
||||||
|
// ring tokens). Flipping one flag must not perturb other rows' identity.
|
||||||
func (s *Store) SetLinkDisabled(local, remote string, port int, disabled bool) error {
|
func (s *Store) SetLinkDisabled(local, remote string, port int, disabled bool) error {
|
||||||
_, err := s.db.Exec(
|
_, err := s.db.Exec(
|
||||||
"UPDATE links SET disabled = ? WHERE local = ? AND remote = ? AND remote_port = ?",
|
"UPDATE links SET disabled = ? WHERE local = ? AND remote = ? AND remote_port = ?",
|
||||||
@ -647,6 +681,52 @@ func (s *Store) SetLinkDisabled(local, remote string, port int, disabled bool) e
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ReconcileLinkDisabled applies a cluster-wide view of one forward's disabled
|
||||||
|
// flag into the local store, creating a placeholder row when this node has none
|
||||||
|
// yet.
|
||||||
|
//
|
||||||
|
// This is the receive half of disabled-flag propagation. The stop decision is
|
||||||
|
// made on whichever node served the request, then rides the token ring in the
|
||||||
|
// topology entry; every other member calls this on adoption so its own
|
||||||
|
// links table agrees. Without it the flag lived only on the node that handled
|
||||||
|
// the request, and the node actually OWNS the forward — usually a different
|
||||||
|
// machine — still believed the forward was enabled and re-spawned its worker.
|
||||||
|
//
|
||||||
|
// A placeholder row is deliberate: a node that has never seen the forward still
|
||||||
|
// needs to remember "this is stopped" so a later claim on this node cannot
|
||||||
|
// resurrect it. The placeholder carries the same natural key, so a subsequent
|
||||||
|
// real claim fills in the rest.
|
||||||
|
func (s *Store) ReconcileLinkDisabled(local, remote string, port int, disabled bool) error {
|
||||||
|
cur, found, err := s.LinkByTriple(local, remote, port)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if found {
|
||||||
|
if cur.Disabled == disabled {
|
||||||
|
return nil // already agrees; avoid needless writes every token cycle
|
||||||
|
}
|
||||||
|
return s.SetLinkDisabled(local, remote, port, disabled)
|
||||||
|
}
|
||||||
|
if !disabled {
|
||||||
|
// Nothing to remember: an unknown forward with no entry is simply
|
||||||
|
// "not stopped", which is the default the claim path already assumes.
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
// Need a placeholder, which requires the local/remote foreign keys to exist.
|
||||||
|
if _, ok := s.GetLocal(local); !ok {
|
||||||
|
return nil // cannot materialise a link without its local peer row
|
||||||
|
}
|
||||||
|
if _, ok := s.GetRemote(remote); !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
links, err := s.ListLinks()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
links = append(links, Link{Local: local, Remote: remote, RemotePort: port, Disabled: true})
|
||||||
|
return s.ReplaceLinks(links)
|
||||||
|
}
|
||||||
|
|
||||||
// SetLinkGroup assigns a management group label to a forward identified by its
|
// SetLinkGroup assigns a management group label to a forward identified by its
|
||||||
// (local, remote, remotePort) natural key. Empty string clears the group
|
// (local, remote, remotePort) natural key. Empty string clears the group
|
||||||
// (moves the forward to 未分组). This is the persistence half of the
|
// (moves the forward to 未分组). This is the persistence half of the
|
||||||
@ -673,6 +753,13 @@ func (s *Store) Settings() (Settings, error) {
|
|||||||
if err := json.Unmarshal([]byte(raw), &st); err != nil {
|
if err := json.Unmarshal([]byte(raw), &st); err != nil {
|
||||||
return Settings{AutoStartProfiles: true, RestartOnExit: true, RestartIntervalSeconds: 5}, nil
|
return Settings{AutoStartProfiles: true, RestartOnExit: true, RestartIntervalSeconds: 5}, nil
|
||||||
}
|
}
|
||||||
|
// BalanceMode lives in its own column (see UpdateSettings). Surface it here
|
||||||
|
// too so callers reading Settings get the effective value, but never let it
|
||||||
|
// overwrite the column on a write-back: UpdateSettings only mirrors a
|
||||||
|
// non-empty value, so an unrelated settings PUT cannot clear a UI choice.
|
||||||
|
if mode, ok, err := s.BalanceMode(); err == nil && ok {
|
||||||
|
st.BalanceMode = mode
|
||||||
|
}
|
||||||
return st, nil
|
return st, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -685,7 +772,36 @@ func (s *Store) UpdateSettings(st Settings) error {
|
|||||||
"INSERT INTO settings(key, value) VALUES('settings', ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value",
|
"INSERT INTO settings(key, value) VALUES('settings', ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value",
|
||||||
string(raw),
|
string(raw),
|
||||||
)
|
)
|
||||||
return err
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if mode := st.BalanceMode; mode != "" {
|
||||||
|
// Mirror into its own column for the same reason the other flagged
|
||||||
|
// fields have one: the settings JSON is rewritten wholesale, so a
|
||||||
|
// clustered field that rides that JSON would flip back and forth between
|
||||||
|
// nodes instead of converging.
|
||||||
|
if _, err := s.db.Exec(
|
||||||
|
"INSERT INTO settings(key, value) VALUES('balance_mode', ?) ON CONFLICT(key) DO UPDATE SET value=excluded.value",
|
||||||
|
mode,
|
||||||
|
); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// BalanceMode returns the stored load-balancing mode, and whether one was ever
|
||||||
|
// stored (false = never configured here; the environment default applies).
|
||||||
|
func (s *Store) BalanceMode() (string, bool, error) {
|
||||||
|
var v string
|
||||||
|
err := s.db.QueryRow("SELECT value FROM settings WHERE key = 'balance_mode'").Scan(&v)
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
return v, v != "", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetNodeKey persists just the cluster admission key, preserving all other
|
// SetNodeKey persists just the cluster admission key, preserving all other
|
||||||
|
|||||||
@ -2,11 +2,11 @@
|
|||||||
//
|
//
|
||||||
// Value is overridable at build time so release artifacts carry the real tag:
|
// Value is overridable at build time so release artifacts carry the real tag:
|
||||||
//
|
//
|
||||||
// go build -ldflags="-X webui4frpc/internal/version.Version=0.1.2" ./cmd/webui4frpc
|
// go build -ldflags="-X webui4frpc/internal/version.Version=0.1.4" ./cmd/webui4frpc
|
||||||
//
|
//
|
||||||
// Kept in its own package because both cmd/ and internal/httpapi report it
|
// Kept in its own package because both cmd/ and internal/httpapi report it
|
||||||
// (main can't be imported, and httpapi must not depend on cmd).
|
// (main can't be imported, and httpapi must not depend on cmd).
|
||||||
package version
|
package version
|
||||||
|
|
||||||
// Version is the running build's semantic version, without a leading "v".
|
// Version is the running build's semantic version, without a leading "v".
|
||||||
var Version = "0.1.2"
|
var Version = "0.1.4"
|
||||||
|
|||||||
@ -1,5 +1,5 @@
|
|||||||
Name: webui4frpc
|
Name: webui4frpc
|
||||||
Version: %{?_w4f_version}%{!?_w4f_version:0.1.2}
|
Version: %{?_w4f_version}%{!?_w4f_version:0.1.4}
|
||||||
Release: 1%{?dist}
|
Release: 1%{?dist}
|
||||||
Summary: Visual frpc controller with token-ring cluster
|
Summary: Visual frpc controller with token-ring cluster
|
||||||
License: MIT
|
License: MIT
|
||||||
@ -67,6 +67,16 @@ fi
|
|||||||
%doc %{_docdir}/webui4frpc/README.md
|
%doc %{_docdir}/webui4frpc/README.md
|
||||||
|
|
||||||
%changelog
|
%changelog
|
||||||
|
* Mon Oct 06 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.4-1
|
||||||
|
- 负载均衡:改为每节点自主释放(持有数超过平摊份额且 >=2 时自撤销一条丢回环,由最闲节点认领),不再由 leader 统一调度
|
||||||
|
- 负载均衡:新增自限上界与下限(撤到平摊值即停、低于 2 条不撤),10 分钟冷却,默认不动作(dry-run)
|
||||||
|
- 集群页:新增负载均衡视图与每节点模式开关(关闭/观察/启用),模式持久化
|
||||||
|
- 新增只读接口 GET /cluster/balance 与写接口 POST /cluster/balance/mode
|
||||||
|
* Sun Oct 05 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.3-1
|
||||||
|
- 窄屏布局:补 box-sizing 重置、接管画布 fit(缩放下限 0.65)、弹窗宽度钳制、统一滚动容器(修复手机端整页无法滑动)
|
||||||
|
- 集群:停用/启动改为经令牌环跨节点传播,撤销不再静默失效
|
||||||
|
- 集群:修复滚动重启导致的分裂脑、令牌环冻结、非 leader 不重建 worker
|
||||||
|
- 集群:令牌轮转日志降级为 debug(355 → 1 行/分钟)
|
||||||
* Wed Sep 03 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.2-1
|
* Wed Sep 03 2026 JianFeeeee <jianfeeeee@gitcode> - 0.1.2-1
|
||||||
- 修复窄屏底部导航栏错位到顶部:顶栏的 backdrop-filter 会为
|
- 修复窄屏底部导航栏错位到顶部:顶栏的 backdrop-filter 会为
|
||||||
position:fixed 后代创建包含块,使 tab bar 的 bottom:0 相对顶栏而
|
position:fixed 后代创建包含块,使 tab bar 的 bottom:0 相对顶栏而
|
||||||
|
|||||||
@ -79,7 +79,7 @@ cd web && npm run build && rm -rf internal/httpapi/dist && cp -r web/dist intern
|
|||||||
### Linux (Debian / Ubuntu)
|
### Linux (Debian / Ubuntu)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo dpkg -i webui4frpc-0.1.1-linux-amd64.deb # 或 linux-arm64.deb
|
sudo dpkg -i webui4frpc-0.1.4-linux-amd64.deb # 或 linux-arm64.deb
|
||||||
sudo nano /etc/webui4frpc/webui4frpc.env # 改掉默认密码
|
sudo nano /etc/webui4frpc/webui4frpc.env # 改掉默认密码
|
||||||
sudo systemctl restart webui4frpc
|
sudo systemctl restart webui4frpc
|
||||||
```
|
```
|
||||||
@ -87,7 +87,7 @@ sudo systemctl restart webui4frpc
|
|||||||
### Linux (RHEL / Fedora / openEuler / 默认 / 龙蜥)
|
### Linux (RHEL / Fedora / openEuler / 默认 / 龙蜥)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo rpm -i webui4frpc-0.1.1-linux-x86_64.rpm
|
sudo rpm -i webui4frpc-0.1.4-linux-x86_64.rpm
|
||||||
sudo vi /etc/webui4frpc/webui4frpc.env
|
sudo vi /etc/webui4frpc/webui4frpc.env
|
||||||
sudo systemctl enable --now webui4frpc
|
sudo systemctl enable --now webui4frpc
|
||||||
```
|
```
|
||||||
@ -97,7 +97,7 @@ sudo systemctl enable --now webui4frpc
|
|||||||
|
|
||||||
### Windows
|
### Windows
|
||||||
|
|
||||||
双击 `webui4frpc-0.1.1-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
双击 `webui4frpc-0.1.4-windows-amd64-setup.exe`(ARM 设备用 `windows-arm64-setup.exe`)。
|
||||||
安装向导会询问监听地址与管理员凭据,完成后:
|
安装向导会询问监听地址与管理员凭据,完成后:
|
||||||
|
|
||||||
- 程序装到 `C:\Program Files\webui4frpc`
|
- 程序装到 `C:\Program Files\webui4frpc`
|
||||||
@ -107,8 +107,8 @@ sudo systemctl enable --now webui4frpc
|
|||||||
### macOS
|
### macOS
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo installer -pkg webui4frpc-0.1.1-darwin-arm64.pkg -target / # Apple Silicon
|
sudo installer -pkg webui4frpc-0.1.4-darwin-arm64.pkg -target / # Apple Silicon
|
||||||
sudo installer -pkg webui4frpc-0.1.1-darwin-amd64.pkg -target / # Intel
|
sudo installer -pkg webui4frpc-0.1.4-darwin-amd64.pkg -target / # Intel
|
||||||
```
|
```
|
||||||
|
|
||||||
或直接双击 `.pkg` 走图形安装器。安装后 launchd 开机自启,配置在
|
或直接双击 `.pkg` 走图形安装器。安装后 launchd 开机自启,配置在
|
||||||
@ -126,7 +126,7 @@ sudo launchctl kickstart -k system/com.jianfeeeee.webui4frpc
|
|||||||
不想装服务只想跑一下,用对应的 `.tar.gz` / `.zip`,解压即用:
|
不想装服务只想跑一下,用对应的 `.tar.gz` / `.zip`,解压即用:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
tar xzf webui4frpc-0.1.1-linux-amd64.tar.gz
|
tar xzf webui4frpc-0.1.4-linux-amd64.tar.gz
|
||||||
./webui4frpc -addr 127.0.0.1:7500 -user admin -password 你的密码 -workdir ./data
|
./webui4frpc -addr 127.0.0.1:7500 -user admin -password 你的密码 -workdir ./data
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@ -12,7 +12,7 @@
|
|||||||
# 在 release.sh 流程内被调用;依赖 dist/artifacts/<version>/bin/<os>-<arch>/ 下的二进制。
|
# 在 release.sh 流程内被调用;依赖 dist/artifacts/<version>/bin/<os>-<arch>/ 下的二进制。
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
VERSION="${1:-0.1.2}"
|
VERSION="${1:-0.1.4}"
|
||||||
VERSION="${VERSION#v}"
|
VERSION="${VERSION#v}"
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
OUT="$ROOT/dist/artifacts/$VERSION"
|
OUT="$ROOT/dist/artifacts/$VERSION"
|
||||||
|
|||||||
137
scripts/probe-with-virtual-frps.sh
Executable file
137
scripts/probe-with-virtual-frps.sh
Executable file
@ -0,0 +1,137 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# 用隔离的虚拟 frps 探测 webui4frpc 行为。
|
||||||
|
#
|
||||||
|
# 全部端口与现网隔离(现网:7000/4434/7500/18082),不修改任何生产配置。
|
||||||
|
# 用法:scripts/probe-with-virtual-frps.sh
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
LAB="${LAB:-/tmp/frplab}"
|
||||||
|
FRP_VER="${FRP_VER:-0.61.1}"
|
||||||
|
FRP_TGZ="frp_${FRP_VER}_linux_amd64"
|
||||||
|
UI_ADDR="${UI_ADDR:-127.0.0.1:17500}"
|
||||||
|
UI_USER="${UI_USER:-labadmin}"
|
||||||
|
UI_PASS="${UI_PASS:-labpass123}"
|
||||||
|
LAB_TOKEN="${LAB_TOKEN:-lab-token-$(date +%Y%m%d)}"
|
||||||
|
|
||||||
|
# 现网 frpc 二进制(唯一允许从生产读取的东西,只读)
|
||||||
|
SRC_FRPC="${SRC_FRPC:-/opt/webui4frpc/data/bin/frpc-${FRP_VER}/frpc}"
|
||||||
|
BACKEND_PORT=17090
|
||||||
|
TCP_PORT=17091
|
||||||
|
UI_FORWARD_PORT=17092
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
echo "==> 清理"
|
||||||
|
pkill -f "webui4frpc -addr ${UI_ADDR}" 2>/dev/null || true
|
||||||
|
pkill -f "frplab/.*frpc" 2>/dev/null || true
|
||||||
|
[[ -n "${FRPS_PID:-}" ]] && kill "${FRPS_PID}" 2>/dev/null || true
|
||||||
|
[[ -n "${BACKEND_PID:-}" ]] && kill "${BACKEND_PID}" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
mkdir -p "${LAB}"/{frps,probe,logs,ui/data}
|
||||||
|
|
||||||
|
# ---------- 1. frps(虚拟服务端) ----------
|
||||||
|
if [[ ! -x "${LAB}/frpsd" ]]; then
|
||||||
|
echo "==> 下载 frps ${FRP_VER}"
|
||||||
|
curl -sL -o "${LAB}/frps-bin" \
|
||||||
|
"https://github.com/fatedier/frp/releases/download/v${FRP_VER}/${FRP_TGZ}.tar.gz"
|
||||||
|
tar xzf "${LAB}/frps-bin" -C "${LAB}"
|
||||||
|
cp "${LAB}/${FRP_TGZ}/frps" "${LAB}/frpsd"
|
||||||
|
chmod +x "${LAB}/frpsd"
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat > "${LAB}/frps/frps.toml" <<EOF
|
||||||
|
bindPort = 17000
|
||||||
|
vhostHTTPPort = 17080
|
||||||
|
vhostHTTPsPort = 17443
|
||||||
|
auth.token = "${LAB_TOKEN}"
|
||||||
|
log.to = "${LAB}/logs/frps.log"
|
||||||
|
log.level = "debug"
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "==> 启动虚拟 frps (17000/17080/17443)"
|
||||||
|
"${LAB}/frpsd" -c "${LAB}/frps/frps.toml" > "${LAB}/logs/frps-stdout.log" 2>&1 &
|
||||||
|
FRPS_PID=$!
|
||||||
|
sleep 3
|
||||||
|
ss -ltn | grep -E ':17000|:17443' || { echo "frps 未监听"; exit 1; }
|
||||||
|
|
||||||
|
# ---------- 2. 后端服务 ----------
|
||||||
|
echo "==> 启动后端 http://127.0.0.1:${BACKEND_PORT}"
|
||||||
|
( cd "${LAB}" && python3 -m http.server "${BACKEND_PORT}" --bind 127.0.0.1 > logs/backend.log 2>&1 & )
|
||||||
|
BACKEND_PID=$!
|
||||||
|
sleep 1
|
||||||
|
|
||||||
|
# ---------- 3. 裸 frpc 验证 frp 协议本身 ----------
|
||||||
|
cat > "${LAB}/probe/frpc-probe.json" <<EOF
|
||||||
|
{
|
||||||
|
"serverAddr": "127.0.0.1",
|
||||||
|
"serverPort": 17000,
|
||||||
|
"loginFailExit": false,
|
||||||
|
"auth": {"method": "token", "token": "${LAB_TOKEN}"},
|
||||||
|
"proxies": [
|
||||||
|
{"name":"lab-tcp","type":"tcp","localIP":"127.0.0.1","localPort":${BACKEND_PORT},"remotePort":${TCP_PORT}}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
cp "${SRC_FRPC}" "${LAB}/frpc" && chmod +x "${LAB}/frpc"
|
||||||
|
echo "==> 裸 frpc 连接(验证 frp 协议通路)"
|
||||||
|
"${LAB}/frpc" -c "${LAB}/probe/frpc-probe.json" > "${LAB}/logs/frpc.log" 2>&1 &
|
||||||
|
sleep 4
|
||||||
|
grep -q 'login to server success' "${LAB}/logs/frpc.log" \
|
||||||
|
&& echo " frpc 登录成功" || { echo " frpc 登录失败"; tail -5 "${LAB}/logs/frpc.log"; }
|
||||||
|
printf " tcp ${TCP_PORT} -> "; curl -s -o /dev/null -w "%{http_code}\n" "http://127.0.0.1:${TCP_PORT}/"
|
||||||
|
pkill -f 'frplab/frpc' 2>/dev/null || true
|
||||||
|
|
||||||
|
# ---------- 4. webui4frpc 接管 ----------
|
||||||
|
echo "==> 启动 webui4frpc ${UI_ADDR}(独立 workdir)"
|
||||||
|
mkdir -p "${LAB}/ui/data/bin/frpc-${FRP_VER}"
|
||||||
|
cp "${SRC_FRPC}" "${LAB}/ui/data/bin/frpc-${FRP_VER}/frpc"
|
||||||
|
chmod +x "${LAB}/ui/data/bin/frpc-${FRP_VER}/frpc"
|
||||||
|
( setsid /opt/webui4frpc/webui4frpc -addr "${UI_ADDR}" \
|
||||||
|
-user "${UI_USER}" -password "${UI_PASS}" \
|
||||||
|
-workdir "${LAB}/ui/data" > "${LAB}/logs/ui.log" 2>&1 < /dev/null & )
|
||||||
|
sleep 5
|
||||||
|
ss -ltn | grep -q "${UI_ADDR##*:}" || { echo "webui4frpc 未监听"; exit 1; }
|
||||||
|
|
||||||
|
AUTH="${UI_USER}:${UI_PASS}"
|
||||||
|
|
||||||
|
echo "==> 注入画布(注意:loopback 会被自动改写为 LAN 地址)"
|
||||||
|
curl -s -u "${AUTH}" -X PUT "http://${UI_ADDR}/api/manager/canvas" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"remotes\":[{\"name\":\"lab-frps\",\"ip\":\"127.0.0.1\",\"port\":17000,\"token\":\"${LAB_TOKEN}\",\"enabled\":true}],
|
||||||
|
\"locals\":[{\"name\":\"lab-backend\",\"ip\":\"127.0.0.1\",\"port\":${BACKEND_PORT}}],
|
||||||
|
\"links\":[{\"local\":\"lab-backend\",\"remote\":\"lab-frps\",\"remotePort\":${UI_FORWARD_PORT}}]}" \
|
||||||
|
| python3 -m json.tool
|
||||||
|
|
||||||
|
echo "==> binaryPath 必须先写入 settings,否则 worker 静默不启动"
|
||||||
|
python3 - "${LAB}/ui/data/manager.db" "${LAB}/ui/data/bin/frpc-${FRP_VER}/frpc" <<'PY'
|
||||||
|
import json, sqlite3, sys
|
||||||
|
db, binary = sys.argv[1], sys.argv[2]
|
||||||
|
c = sqlite3.connect(db)
|
||||||
|
raw = c.execute("select value from settings where key='settings'").fetchone()[0]
|
||||||
|
d = json.loads(raw); d["binaryPath"] = binary
|
||||||
|
c.execute("update settings set value=? where key='settings'", (json.dumps(d),))
|
||||||
|
c.commit()
|
||||||
|
print(" binaryPath 已写入:", binary)
|
||||||
|
PY
|
||||||
|
|
||||||
|
echo "==> 启动 forward(非 localOnly 需 owner claim,观察 status)"
|
||||||
|
curl -s -u "${AUTH}" -X POST "http://${UI_ADDR}/api/manager/forwards/start" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"local\":\"lab-backend\",\"remote\":\"lab-frps\",\"remotePort\":${UI_FORWARD_PORT}}"
|
||||||
|
echo
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
echo "==> 观察结果"
|
||||||
|
curl -s -u "${AUTH}" "http://${UI_ADDR}/api/manager/status" | python3 -c "
|
||||||
|
import json,sys
|
||||||
|
d=json.load(sys.stdin)
|
||||||
|
print(' forwards:', [(f['local'], f['active']) for f in d.get('forwards',[])])
|
||||||
|
print(' profiles:', [(p['name'], p['process']['state'], p.get('connState')) for p in d.get('profiles',[])])
|
||||||
|
"
|
||||||
|
echo " binaryPath: $(curl -s -u "${AUTH}" "http://${UI_ADDR}/api/manager/binary/status")"
|
||||||
|
echo " configs/: $(ls "${LAB}/ui/data/configs/" 2>/dev/null | tr '\n' ' ')"
|
||||||
|
echo " workers: $(pgrep -af "${LAB}/ui/data" | wc -l) 个"
|
||||||
|
echo
|
||||||
|
echo "日志: ${LAB}/logs/"
|
||||||
|
echo "复现结论见 docs/how-it-works.md"
|
||||||
@ -3,7 +3,7 @@
|
|||||||
# webui4frpc 多平台发布脚本
|
# webui4frpc 多平台发布脚本
|
||||||
#
|
#
|
||||||
# 用法:
|
# 用法:
|
||||||
# ./scripts/release.sh [version] # 默认 version=0.1.2
|
# ./scripts/release.sh [version] # 默认 version=0.1.4
|
||||||
#
|
#
|
||||||
# 行为:
|
# 行为:
|
||||||
# 1. 构建前端 (web/dist) 并同步到 internal/httpapi/dist (go:embed 用)
|
# 1. 构建前端 (web/dist) 并同步到 internal/httpapi/dist (go:embed 用)
|
||||||
@ -19,7 +19,7 @@
|
|||||||
# NO_BUILD_FRONTEND 设为 1 跳过前端构建 (用现有 dist)
|
# NO_BUILD_FRONTEND 设为 1 跳过前端构建 (用现有 dist)
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
VERSION="${1:-0.1.2}"
|
VERSION="${1:-0.1.4}"
|
||||||
VERSION="${VERSION#v}" # 去掉可能的前导 v
|
VERSION="${VERSION#v}" # 去掉可能的前导 v
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
OUT="$ROOT/dist/artifacts/$VERSION"
|
OUT="$ROOT/dist/artifacts/$VERSION"
|
||||||
|
|||||||
@ -446,7 +446,10 @@ const currentLabel = computed(() => nav.value.find((n) => n.key === view.value)?
|
|||||||
gap: 2px;
|
gap: 2px;
|
||||||
padding: 5px 2px;
|
padding: 5px 2px;
|
||||||
border-radius: 11px;
|
border-radius: 11px;
|
||||||
font-size: 10.5px;
|
/* Keep the tab label at the theme's 11px floor; the 10.5px here (and 9.5px
|
||||||
|
* at ≤380px) read as noise on a phone. "账号与密钥" is still handled by the
|
||||||
|
* .label ellipsis (width:100% + overflow:hidden). */
|
||||||
|
font-size: 11px;
|
||||||
font-weight: 600;
|
font-weight: 600;
|
||||||
text-align: center;
|
text-align: center;
|
||||||
}
|
}
|
||||||
@ -500,12 +503,13 @@ const currentLabel = computed(() => nav.value.find((n) => n.key === view.value)?
|
|||||||
|
|
||||||
/* 极窄(≤ 380px,iPhone SE / 小屏安卓):进一步压缩顶条。
|
/* 极窄(≤ 380px,iPhone SE / 小屏安卓):进一步压缩顶条。
|
||||||
角色徒章是 flex:0 0 auto 不可收缩的(“超级管理员”≨70px),极窄下隐去,
|
角色徒章是 flex:0 0 auto 不可收缩的(“超级管理员”≨70px),极窄下隐去,
|
||||||
用户名与退出按钮保留。 */
|
用户名与退出按钮保留。
|
||||||
|
注意:底部 tab 文字不再随之降到 9.5px —— 它在手机上低于可读下限
|
||||||
|
(见 .sb-i 的 11px 注释),长标签交给 .label 的省略号处理。 */
|
||||||
@media (max-width: 380px) {
|
@media (max-width: 380px) {
|
||||||
.sb-brand-text b { font-size: 13px; }
|
.sb-brand-text b { font-size: 13px; }
|
||||||
.sb-identity { padding: 4px 7px; }
|
.sb-identity { padding: 4px 7px; }
|
||||||
.sb-identity .id-level { display: none; }
|
.sb-identity .id-level { display: none; }
|
||||||
.sb-i { font-size: 9.5px; }
|
|
||||||
.content { padding-left: 10px; padding-right: 10px; }
|
.content { padding-left: 10px; padding-right: 10px; }
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@ -2,6 +2,7 @@
|
|||||||
import type {
|
import type {
|
||||||
ApiKey,
|
ApiKey,
|
||||||
ApiKeyCreated,
|
ApiKeyCreated,
|
||||||
|
BalanceReport,
|
||||||
BinaryStatus,
|
BinaryStatus,
|
||||||
CacheResp,
|
CacheResp,
|
||||||
CanvasData,
|
CanvasData,
|
||||||
@ -159,6 +160,21 @@ export const api = {
|
|||||||
// M6 token ring snapshot.
|
// M6 token ring snapshot.
|
||||||
ring: () => request<RingSnapshot>("/api/manager/cluster/ring"),
|
ring: () => request<RingSnapshot>("/api/manager/cluster/ring"),
|
||||||
|
|
||||||
|
// Load balancer read-only view. Reports per-node scores, whether anything is
|
||||||
|
// over the threshold, and the migration it would perform. Never mutates the
|
||||||
|
// cluster. 404 when the balancer is not wired up on that node.
|
||||||
|
balance: () => request<BalanceReport>("/api/manager/cluster/balance"),
|
||||||
|
|
||||||
|
// setBalanceMode switches load balancing on THIS node (off | dry-run | on).
|
||||||
|
// Per-node by design: each member releases only its own forwards, so there is
|
||||||
|
// no cluster-wide switch to flip. Requires write level.
|
||||||
|
setBalanceMode: (mode: string) =>
|
||||||
|
request<BalanceReport>("/api/manager/cluster/balance/mode", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ mode }),
|
||||||
|
}),
|
||||||
|
|
||||||
// M6 cluster lifecycle (创建/加入/退出/移除).
|
// M6 cluster lifecycle (创建/加入/退出/移除).
|
||||||
// clusterCreate: reseed THIS node as a fresh standalone leader (创建集群).
|
// clusterCreate: reseed THIS node as a fresh standalone leader (创建集群).
|
||||||
clusterCreate: () =>
|
clusterCreate: () =>
|
||||||
|
|||||||
@ -184,6 +184,20 @@
|
|||||||
|
|
||||||
/* ---- blue theme (ocean × frost) ---- */
|
/* ---- blue theme (ocean × frost) ---- */
|
||||||
|
|
||||||
|
/* ---- box model reset ----
|
||||||
|
* This app had NO box-sizing reset, so every element using `width: 100%` with
|
||||||
|
* horizontal padding overflowed by exactly its padding under the default
|
||||||
|
* content-box model. The narrow-screen top bar was the visible victim:
|
||||||
|
* `.sidebar { width: 100%; padding: 10px 12px }` rendered 375+24 = 399px wide
|
||||||
|
* on a 375px viewport (344px at 320px), pushing the theme swatches and logout
|
||||||
|
* button off the right edge where `overflow-x: hidden` silently clipped them.
|
||||||
|
* It also made the desktop sidebar 246+28 = 274px wide instead of 246px.
|
||||||
|
* border-box makes width include padding+border, which every layout here
|
||||||
|
* already assumes (Element Plus sets it on its own components already). */
|
||||||
|
*,
|
||||||
|
*::before,
|
||||||
|
*::after { box-sizing: border-box; }
|
||||||
|
|
||||||
html,
|
html,
|
||||||
body,
|
body,
|
||||||
#app {
|
#app {
|
||||||
@ -346,4 +360,14 @@ a { color: var(--w4f-primary-h); }
|
|||||||
/* ---- Element Plus dialog tweaks to read as glass ---- */
|
/* ---- Element Plus dialog tweaks to read as glass ---- */
|
||||||
.el-overlay { backdrop-filter: blur(3px); }
|
.el-overlay { backdrop-filter: blur(3px); }
|
||||||
.el-dialog { border-radius: var(--w4f-radius) !important; box-shadow: var(--w4f-sh-lg) !important; }
|
.el-dialog { border-radius: var(--w4f-radius) !important; box-shadow: var(--w4f-sh-lg) !important; }
|
||||||
|
|
||||||
|
/* Dialogs pass a fixed px width (width="420px" / "440px"), which Element Plus
|
||||||
|
* writes into an INLINE `--el-dialog-width` custom property. On phones that is
|
||||||
|
* wider than the viewport (420px on a 375px screen), so the dialog's right edge
|
||||||
|
* — including the header close button — starts off-screen and is only reachable
|
||||||
|
* by scrolling the overlay horizontally. Clamp it to the viewport; !important
|
||||||
|
* is required to beat the inline custom property. */
|
||||||
|
@media (max-width: 720px) {
|
||||||
|
.el-dialog { --el-dialog-width: calc(100vw - 24px) !important; }
|
||||||
|
}
|
||||||
.el-message { border-radius: var(--w4f-radius-sm) !important; box-shadow: var(--w4f-sh-lg) !important; }
|
.el-message { border-radius: var(--w4f-radius-sm) !important; box-shadow: var(--w4f-sh-lg) !important; }
|
||||||
|
|||||||
490
web/src/types.ts
490
web/src/types.ts
@ -9,332 +9,382 @@ export type CanvasRemote = Remote;
|
|||||||
export type CanvasLink = Link;
|
export type CanvasLink = Link;
|
||||||
|
|
||||||
export interface Local {
|
export interface Local {
|
||||||
name: string;
|
name: string;
|
||||||
ip: string;
|
ip: string;
|
||||||
port: number;
|
port: number;
|
||||||
protocol: string; // tcp | udp | http | https
|
protocol: string; // tcp | udp | http | https
|
||||||
// localOnly: forward stays on this node (loopback kept), no cluster.
|
// localOnly: forward stays on this node (loopback kept), no cluster.
|
||||||
localOnly?: boolean;
|
localOnly?: boolean;
|
||||||
|
|
||||||
// M1 advanced transport knobs (optional; empty/zero = frpc defaults)
|
// M1 advanced transport knobs (optional; empty/zero = frpc defaults)
|
||||||
useEncryption?: boolean;
|
useEncryption?: boolean;
|
||||||
useCompression?: boolean;
|
useCompression?: boolean;
|
||||||
bandwidthLimit?: string;
|
bandwidthLimit?: string;
|
||||||
poolCount?: number;
|
poolCount?: number;
|
||||||
metadatas?: Record<string, string>;
|
metadatas?: Record<string, string>;
|
||||||
annotations?: Record<string, string>;
|
annotations?: Record<string, string>;
|
||||||
|
|
||||||
// M2 HTTP/HTTPS routing (http/https only)
|
// M2 HTTP/HTTPS routing (http/https only)
|
||||||
customDomains?: string; // comma-separated
|
customDomains?: string; // comma-separated
|
||||||
subdomain?: string;
|
subdomain?: string;
|
||||||
locations?: string[];
|
locations?: string[];
|
||||||
hostHeaderRewrite?: string;
|
hostHeaderRewrite?: string;
|
||||||
httpHeaders?: Record<string, string>;
|
httpHeaders?: Record<string, string>;
|
||||||
basicAuthUser?: string;
|
basicAuthUser?: string;
|
||||||
basicAuthPassword?: string;
|
basicAuthPassword?: string;
|
||||||
|
|
||||||
// M3 load balancing group + health check
|
// M3 load balancing group + health check
|
||||||
lbGroup?: string;
|
lbGroup?: string;
|
||||||
lbGroupKey?: string;
|
lbGroupKey?: string;
|
||||||
healthCheckType?: string; // "" | tcp | http
|
healthCheckType?: string; // "" | tcp | http
|
||||||
healthCheckPath?: string;
|
healthCheckPath?: string;
|
||||||
healthCheckTimeout?: number;
|
healthCheckTimeout?: number;
|
||||||
healthCheckMaxFailed?: number;
|
healthCheckMaxFailed?: number;
|
||||||
healthCheckInterval?: number;
|
healthCheckInterval?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface Remote {
|
export interface Remote {
|
||||||
name: string;
|
name: string;
|
||||||
ip: string;
|
ip: string;
|
||||||
port: number; // frps connect port
|
port: number; // frps connect port
|
||||||
token?: string;
|
token?: string;
|
||||||
url?: string;
|
url?: string;
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
|
|
||||||
// M1 transport section
|
// M1 transport section
|
||||||
transportProtocol?: string; // tcp | quic | kcp | websocket
|
transportProtocol?: string; // tcp | quic | kcp | websocket
|
||||||
transportTls?: boolean;
|
transportTls?: boolean;
|
||||||
transportPool?: number;
|
transportPool?: number;
|
||||||
transportTlsServerName?: string;
|
transportTlsServerName?: string;
|
||||||
|
|
||||||
// M2: frps vhostHTTPPort
|
// M2: frps vhostHTTPPort
|
||||||
vhostHttpPort?: number;
|
vhostHttpPort?: number;
|
||||||
|
|
||||||
// M3: local frpc admin API (webServer)
|
// M3: local frpc admin API (webServer)
|
||||||
adminAddr?: string;
|
adminAddr?: string;
|
||||||
adminPort?: number;
|
adminPort?: number;
|
||||||
adminUser?: string;
|
adminUser?: string;
|
||||||
adminPassword?: string;
|
adminPassword?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface Link {
|
export interface Link {
|
||||||
id?: number;
|
id?: number;
|
||||||
local: string;
|
local: string;
|
||||||
remote: string;
|
remote: string;
|
||||||
remotePort: number;
|
remotePort: number;
|
||||||
offsetX?: number;
|
offsetX?: number;
|
||||||
offsetY?: number;
|
offsetY?: number;
|
||||||
// group: management label for one-click group start/stop on the forwards
|
// group: management label for one-click group start/stop on the forwards
|
||||||
// page (unrelated to frps load-balancing lbGroup on Local).
|
// page (unrelated to frps load-balancing lbGroup on Local).
|
||||||
group?: string;
|
group?: string;
|
||||||
// disabled: per-forward stop flag. A disabled forward is omitted from the
|
// disabled: per-forward stop flag. A disabled forward is omitted from the
|
||||||
// frpc config (local) / left out of the ring topology (cluster), so stop is
|
// frpc config (local) / left out of the ring topology (cluster), so stop is
|
||||||
// per-card and survives canvas re-saves.
|
// per-card and survives canvas re-saves.
|
||||||
disabled?: boolean;
|
disabled?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CanvasData {
|
export interface CanvasData {
|
||||||
locals: Local[];
|
locals: Local[];
|
||||||
remotes: Remote[];
|
remotes: Remote[];
|
||||||
links: Link[];
|
links: Link[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface Settings {
|
export interface Settings {
|
||||||
autoStartProfiles: boolean;
|
autoStartProfiles: boolean;
|
||||||
restartOnExit: boolean;
|
restartOnExit: boolean;
|
||||||
restartIntervalSeconds: number;
|
restartIntervalSeconds: number;
|
||||||
binaryPath?: string;
|
binaryPath?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ProcessStatus {
|
export interface ProcessStatus {
|
||||||
state: string;
|
state: string;
|
||||||
pid?: number;
|
pid?: number;
|
||||||
startTime?: number;
|
startTime?: number;
|
||||||
restartCount: number;
|
restartCount: number;
|
||||||
exitCode?: number;
|
exitCode?: number;
|
||||||
err?: string;
|
err?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ProxyState {
|
export interface ProxyState {
|
||||||
name: string;
|
name: string;
|
||||||
type: string;
|
type: string;
|
||||||
status: string; // new | wait start | start error | running | check failed | closed
|
status: string; // new | wait start | start error | running | check failed | closed
|
||||||
err?: string;
|
err?: string;
|
||||||
local_addr?: string;
|
local_addr?: string;
|
||||||
remote_addr?: string;
|
remote_addr?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface StatusProfile {
|
export interface StatusProfile {
|
||||||
name: string;
|
name: string;
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
process: ProcessStatus;
|
process: ProcessStatus;
|
||||||
hasProcess: boolean;
|
hasProcess: boolean;
|
||||||
forwards: { service: string; remotePort: number; localPort?: number }[];
|
forwards: { service: string; remotePort: number; localPort?: number }[];
|
||||||
// connState: derived frps connection state (we do not control remote frps)
|
// connState: derived frps connection state (we do not control remote frps)
|
||||||
connState?: string; // connected | connecting | failed | not_started | disabled
|
connState?: string; // connected | connecting | failed | not_started | disabled
|
||||||
// M3: admin API state
|
// M3: admin API state
|
||||||
adminEnabled?: boolean;
|
adminEnabled?: boolean;
|
||||||
proxyStates?: ProxyState[];
|
proxyStates?: ProxyState[];
|
||||||
groupCount?: number;
|
groupCount?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface StatusResp {
|
export interface StatusResp {
|
||||||
version: string;
|
version: string;
|
||||||
workDir: string;
|
workDir: string;
|
||||||
settings: Settings;
|
settings: Settings;
|
||||||
services: Local[];
|
services: Local[];
|
||||||
remotes: Remote[];
|
remotes: Remote[];
|
||||||
binaryPath: string;
|
binaryPath: string;
|
||||||
profiles: StatusProfile[];
|
profiles: StatusProfile[];
|
||||||
localStatus: LocalStatus[];
|
localStatus: LocalStatus[];
|
||||||
// forwards: the forward-centric list rendered as cards on the status page.
|
// forwards: the forward-centric list rendered as cards on the status page.
|
||||||
forwards: ForwardStatus[];
|
forwards: ForwardStatus[];
|
||||||
// selfId: this node's ring ID (== SelfAddr), to tell whether a cluster
|
// selfId: this node's ring ID (== SelfAddr), to tell whether a cluster
|
||||||
// forward's owner is this node (worker runs here) or a peer.
|
// forward's owner is this node (worker runs here) or a peer.
|
||||||
selfId: string;
|
selfId: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ForwardStatus is one forward (link) in the forward-centric status view.
|
// ForwardStatus is one forward (link) in the forward-centric status view.
|
||||||
// kind distinguishes 本地转发 (localOnly) from 远程转发 (cluster-distributed).
|
// kind distinguishes 本地转发 (localOnly) from 远程转发 (cluster-distributed).
|
||||||
export interface ForwardStatus {
|
export interface ForwardStatus {
|
||||||
local: string;
|
local: string;
|
||||||
remote: string;
|
remote: string;
|
||||||
remotePort: number;
|
remotePort: number;
|
||||||
localOnly: boolean;
|
localOnly: boolean;
|
||||||
kind: "local" | "remote";
|
kind: "local" | "remote";
|
||||||
ownerId?: string; // remote = topology owner; local = selfId
|
ownerId?: string; // remote = topology owner; local = selfId
|
||||||
active: boolean; // remote = in topology; local = local worker running
|
active: boolean; // remote = in topology; local = local worker running
|
||||||
disabled: boolean; // per-forward stop flag (true = stopped)
|
disabled: boolean; // per-forward stop flag (true = stopped)
|
||||||
group?: string;
|
group?: string;
|
||||||
localIp?: string;
|
localIp?: string;
|
||||||
localPort?: number;
|
localPort?: number;
|
||||||
localProto?: string;
|
localProto?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface LocalTargetStatus {
|
export interface LocalTargetStatus {
|
||||||
remote: string;
|
remote: string;
|
||||||
remotePort: number;
|
remotePort: number;
|
||||||
workerState: string;
|
workerState: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface LocalStatus {
|
export interface LocalStatus {
|
||||||
local: Local;
|
local: Local;
|
||||||
targets: LocalTargetStatus[];
|
targets: LocalTargetStatus[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface BinaryStatus {
|
export interface BinaryStatus {
|
||||||
binaryPath: string;
|
binaryPath: string;
|
||||||
version?: string;
|
version?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface InstallResult {
|
export interface InstallResult {
|
||||||
path: string;
|
path: string;
|
||||||
version: string;
|
version: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
// M6: cluster node registry + binary cache.
|
// M6: cluster node registry + binary cache.
|
||||||
export interface ClusterNode {
|
export interface ClusterNode {
|
||||||
addr: string;
|
addr: string;
|
||||||
cache?: string[];
|
cache?: string[];
|
||||||
version?: string;
|
version?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CacheEntry {
|
export interface CacheEntry {
|
||||||
version: string;
|
version: string;
|
||||||
path: string;
|
path: string;
|
||||||
size: number;
|
size: number;
|
||||||
modTime: number;
|
modTime: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ClusterNodesResp {
|
export interface ClusterNodesResp {
|
||||||
nodes: ClusterNode[];
|
nodes: ClusterNode[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CacheResp {
|
export interface CacheResp {
|
||||||
cache: CacheEntry[];
|
cache: CacheEntry[];
|
||||||
removed?: string[];
|
removed?: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
// M6 token-ring snapshot (GET /cluster/ring).
|
// M6 token-ring snapshot (GET /cluster/ring).
|
||||||
export interface RingNode {
|
export interface RingNode {
|
||||||
id: string;
|
id: string;
|
||||||
addr: string;
|
addr: string;
|
||||||
isLeader?: boolean;
|
isLeader?: boolean;
|
||||||
alive: boolean;
|
alive: boolean;
|
||||||
load: { memPct: number; netPct: number; forwards?: number };
|
load: { memPct: number; netPct: number; forwards?: number };
|
||||||
version?: string;
|
version?: string;
|
||||||
cache?: string[];
|
cache?: string[];
|
||||||
lastSeen?: number;
|
lastSeen?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface RingTaskInfo {
|
export interface RingTaskInfo {
|
||||||
id: string;
|
id: string;
|
||||||
revoke?: boolean;
|
revoke?: boolean;
|
||||||
removeNode?: string; // node-removal command (target rides the token to self-remove)
|
removeNode?: string; // node-removal command (target rides the token to self-remove)
|
||||||
local: { name: string };
|
local: { name: string };
|
||||||
remote: { name: string };
|
remote: { name: string };
|
||||||
link: { remotePort?: number };
|
link: { remotePort?: number };
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface RingTopoEntry {
|
export interface RingTopoEntry {
|
||||||
taskId: string;
|
taskId: string;
|
||||||
ownerId: string;
|
ownerId: string;
|
||||||
active: boolean;
|
active: boolean;
|
||||||
local: { name: string };
|
local: { name: string };
|
||||||
remote: { name: string };
|
remote: { name: string };
|
||||||
link: { remotePort?: number };
|
link: { remotePort?: number };
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface RingLogEntry {
|
export interface RingLogEntry {
|
||||||
seq: number;
|
seq: number;
|
||||||
node: string;
|
node: string;
|
||||||
kind: string;
|
kind: string;
|
||||||
at?: number;
|
at?: number;
|
||||||
data?: Record<string, any> | null;
|
data?: Record<string, any> | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface RingSnapshot {
|
export interface RingSnapshot {
|
||||||
selfId: string;
|
selfId: string;
|
||||||
leaderId: string; // 当前 leader(持首令牌 + 判丢失 + 被上家监控)
|
leaderId: string; // 当前 leader(持首令牌 + 判丢失 + 被上家监控)
|
||||||
cycle: number; // 周期号(leader 每收令牌回 +1)
|
cycle: number; // 周期号(leader 每收令牌回 +1)
|
||||||
lastSync: number; // 上次同步时间 (unix sec)
|
lastSync: number; // 上次同步时间 (unix sec)
|
||||||
roundDelayMs?: number;
|
roundDelayMs?: number;
|
||||||
nodes: RingNode[];
|
nodes: RingNode[];
|
||||||
pending: RingTaskInfo[];
|
pending: RingTaskInfo[];
|
||||||
topology: RingTopoEntry[];
|
topology: RingTopoEntry[];
|
||||||
log: RingLogEntry[];
|
log: RingLogEntry[];
|
||||||
// nodeKey: this node's cluster admission key. Newcomers must present it to
|
// nodeKey: this node's cluster admission key. Newcomers must present it to
|
||||||
// join via this node (handleClusterJoin verifies). Shown on the cluster page
|
// join via this node (handleClusterJoin verifies). Shown on the cluster page
|
||||||
// for the operator to copy; omitted for non-cluster (standalone) nodes.
|
// for the operator to copy; omitted for non-cluster (standalone) nodes.
|
||||||
nodeKey?: string;
|
nodeKey?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// BalanceReport is one node's own view of rebalancing (GET /cluster/balance).
|
||||||
|
//
|
||||||
|
// It is per-node by design: every member runs the same rule against its own
|
||||||
|
// forwards, and the answer that matters is the one on the node doing (or
|
||||||
|
// refusing to do) the releasing. So this reports "my share" first and the peers
|
||||||
|
// second, and there is no cluster-wide "would migrate" plan — the ring's normal
|
||||||
|
// lowest-load claim decides where a released forward lands.
|
||||||
|
export interface BalanceReleasePlan {
|
||||||
|
local: string;
|
||||||
|
remote: string;
|
||||||
|
remotePort: number;
|
||||||
|
owner: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface BalanceReport {
|
||||||
|
mode: "off" | "dry-run" | "on";
|
||||||
|
modeSource: "env" | "stored"; // env = W4F_BALANCE_MODE default, stored = set here
|
||||||
|
enabled: boolean; // mode !== off
|
||||||
|
dryRun: boolean; // mode !== on
|
||||||
|
canControl: boolean; // this principal may switch it (write level)
|
||||||
|
isMember: boolean;
|
||||||
|
self: string;
|
||||||
|
|
||||||
|
selfForwards: number;
|
||||||
|
share: number; // the even split: ceil(total / alive)
|
||||||
|
totalForwards: number;
|
||||||
|
aliveNodes: number;
|
||||||
|
selfOverShare: boolean;
|
||||||
|
|
||||||
|
// wouldRelease is present whenever this node is over its share — including
|
||||||
|
// while it is deliberately holding back, which is what blockedBy explains.
|
||||||
|
wouldRelease?: BalanceReleasePlan;
|
||||||
|
releasing?: BalanceReleasePlan;
|
||||||
|
blockedBy?: string; // single-node | below-min | balanced | nothing-active | cooldown | in-flight
|
||||||
|
|
||||||
|
cooldownSec: number;
|
||||||
|
lastReleaseAt?: number;
|
||||||
|
nodes: BalanceNodeReport[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface BalanceNodeReport {
|
||||||
|
id: string;
|
||||||
|
alive: boolean;
|
||||||
|
isLeader: boolean;
|
||||||
|
forwards: number;
|
||||||
|
memPct: number;
|
||||||
|
netPct: number;
|
||||||
|
score: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- Auth / accounts / API keys (M7) ----
|
// ---- Auth / accounts / API keys (M7) ----
|
||||||
|
|
||||||
export type AccessLevel = 'read' | 'write' | 'admin' | 'superadmin';
|
export type AccessLevel = "read" | "write" | "admin" | "superadmin";
|
||||||
|
|
||||||
// MeResp is the authenticated principal (GET /me). The frontend uses level to
|
// MeResp is the authenticated principal (GET /me). The frontend uses level to
|
||||||
// gate the UI: viewer (= auditors) sees read + export controls only.
|
// gate the UI: viewer (= auditors) sees read + export controls only.
|
||||||
export interface MeResp {
|
export interface MeResp {
|
||||||
type: 'user' | 'service' | 'key';
|
type: "user" | "service" | "key";
|
||||||
name: string;
|
name: string;
|
||||||
level: AccessLevel;
|
level: AccessLevel;
|
||||||
userId?: number;
|
userId?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface User {
|
export interface User {
|
||||||
id: number;
|
id: number;
|
||||||
username: string;
|
username: string;
|
||||||
role: 'admin' | 'viewer' | 'superadmin';
|
role: "admin" | "viewer" | "superadmin";
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
system?: boolean; // flag-synced built-in account (UI read-only)
|
system?: boolean; // flag-synced built-in account (UI read-only)
|
||||||
createdAt: number;
|
createdAt: number;
|
||||||
lastLoginAt: number;
|
lastLoginAt: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ApiKey {
|
export interface ApiKey {
|
||||||
id: number;
|
id: number;
|
||||||
userId: number;
|
userId: number;
|
||||||
prefix: string;
|
prefix: string;
|
||||||
label: string;
|
label: string;
|
||||||
scope: AccessLevel;
|
scope: AccessLevel;
|
||||||
createdAt: number;
|
createdAt: number;
|
||||||
lastUsedAt: number;
|
lastUsedAt: number;
|
||||||
expiresAt?: number;
|
expiresAt?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ApiKeyCreated is the one-time response from POST /apikeys: the plaintext key
|
// ApiKeyCreated is the one-time response from POST /apikeys: the plaintext key
|
||||||
// is shown exactly once and never retrievable again.
|
// is shown exactly once and never retrievable again.
|
||||||
export interface ApiKeyCreated {
|
export interface ApiKeyCreated {
|
||||||
id: number;
|
id: number;
|
||||||
userId: number;
|
userId: number;
|
||||||
label: string;
|
label: string;
|
||||||
scope: AccessLevel;
|
scope: AccessLevel;
|
||||||
prefix: string;
|
prefix: string;
|
||||||
createdAt: number;
|
createdAt: number;
|
||||||
key: string;
|
key: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CanvasExportEnvelope {
|
export interface CanvasExportEnvelope {
|
||||||
_type: 'webui4frpc-canvas';
|
_type: "webui4frpc-canvas";
|
||||||
version: number;
|
version: number;
|
||||||
exportedAt: number;
|
exportedAt: number;
|
||||||
exporter: string;
|
exporter: string;
|
||||||
canvas: CanvasData;
|
canvas: CanvasData;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Worker-log bundle (GET /cluster/logs/export): one entry per alive ring node.
|
// Worker-log bundle (GET /cluster/logs/export): one entry per alive ring node.
|
||||||
export interface WorkerLogEntry {
|
export interface WorkerLogEntry {
|
||||||
name: string;
|
name: string;
|
||||||
remote: string;
|
remote: string;
|
||||||
state: string;
|
state: string;
|
||||||
lines: string;
|
lines: string;
|
||||||
}
|
}
|
||||||
export interface NodeLogsResp {
|
export interface NodeLogsResp {
|
||||||
node: string;
|
node: string;
|
||||||
workers: WorkerLogEntry[];
|
workers: WorkerLogEntry[];
|
||||||
}
|
}
|
||||||
export interface ClusterWorkerLogNode {
|
export interface ClusterWorkerLogNode {
|
||||||
id: string;
|
id: string;
|
||||||
addr: string;
|
addr: string;
|
||||||
ok: boolean;
|
ok: boolean;
|
||||||
error?: string;
|
error?: string;
|
||||||
logs?: NodeLogsResp;
|
logs?: NodeLogsResp;
|
||||||
}
|
}
|
||||||
export interface WorkerLogBundle {
|
export interface WorkerLogBundle {
|
||||||
_type: 'webui4frpc-worker-logs';
|
_type: "webui4frpc-worker-logs";
|
||||||
version: number;
|
version: number;
|
||||||
exportedAt: number;
|
exportedAt: number;
|
||||||
requester: string;
|
requester: string;
|
||||||
nodes: ClusterWorkerLogNode[];
|
nodes: ClusterWorkerLogNode[];
|
||||||
}
|
}
|
||||||
|
|||||||
@ -25,16 +25,17 @@
|
|||||||
<VueFlow
|
<VueFlow
|
||||||
v-model:nodes="nodes"
|
v-model:nodes="nodes"
|
||||||
v-model:edges="edges"
|
v-model:edges="edges"
|
||||||
:default-viewport="{ zoom: 0.85 }"
|
:default-viewport="{ zoom: FIT_MAX_ZOOM }"
|
||||||
:min-zoom="0.3"
|
:min-zoom="MIN_ZOOM"
|
||||||
:max-zoom="2"
|
:max-zoom="2"
|
||||||
fit-view-on-init
|
|
||||||
:delete-key-code="null"
|
:delete-key-code="null"
|
||||||
:nodes-draggable="canWrite"
|
:nodes-draggable="canWrite"
|
||||||
:nodes-connectable="canWrite"
|
:nodes-connectable="canWrite"
|
||||||
:edges-updatable="false"
|
:edges-updatable="false"
|
||||||
:edges-reconnectable="false"
|
:edges-reconnectable="false"
|
||||||
class="flow-canvas"
|
class="flow-canvas"
|
||||||
|
@init="onFlowInit"
|
||||||
|
@nodes-initialized="reframe"
|
||||||
@connect="onConnect"
|
@connect="onConnect"
|
||||||
@pane-click="deselectAll"
|
@pane-click="deselectAll"
|
||||||
@edge-click="onEdgeClick"
|
@edge-click="onEdgeClick"
|
||||||
@ -118,7 +119,7 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import '@vue-flow/core/dist/style.css'
|
import '@vue-flow/core/dist/style.css'
|
||||||
import '@vue-flow/core/dist/theme-default.css'
|
import '@vue-flow/core/dist/theme-default.css'
|
||||||
import { ref, computed, onMounted, onBeforeUnmount } from 'vue'
|
import { ref, computed, onMounted, onBeforeUnmount, nextTick } from 'vue'
|
||||||
import { VueFlow } from '@vue-flow/core'
|
import { VueFlow } from '@vue-flow/core'
|
||||||
import { Background } from '@vue-flow/background'
|
import { Background } from '@vue-flow/background'
|
||||||
import { ElMessage, ElMessageBox } from 'element-plus'
|
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||||
@ -134,6 +135,103 @@ const nodes = ref<any[]>([])
|
|||||||
const edges = ref<Edge[]>([])
|
const edges = ref<Edge[]>([])
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
|
|
||||||
|
// ---- viewport / zoom floor ----
|
||||||
|
// The canvas is laid out in absolute graph units (local column at x=60, remote
|
||||||
|
// column at x=760, and BOTH columns grow downward on every added forward), so
|
||||||
|
// its bounding box is far wider and taller than a phone viewport. Vue Flow's
|
||||||
|
// `fit-view-on-init` frames that whole box, which on a phone produced a scale of
|
||||||
|
// ~0.31: node cards measure 248px in graph units but rendered 85px wide at an
|
||||||
|
// effective font size of ~4.4px — unreadable. Even a 1280px desktop landed at
|
||||||
|
// ~0.60 (8.4px) with five forwards stacked.
|
||||||
|
//
|
||||||
|
// So the canvas is framed by frameCanvas() below, clamped by a zoom floor, and
|
||||||
|
// the canvas pans when the content no longer fits. 0.65 renders a 240px local
|
||||||
|
// card at ~156px with a legible ~9px font.
|
||||||
|
//
|
||||||
|
// NOTE: `fit-view-on-init` is deliberately NOT set. It resolves on its own
|
||||||
|
// schedule and overwrote the explicit fit performed here (observed: this code
|
||||||
|
// measured the unfitted layout, set the viewport, and then the deferred init fit
|
||||||
|
// landed last and won). Owning the fit outright is the only way to keep it
|
||||||
|
// deterministic.
|
||||||
|
//
|
||||||
|
// The store arrives via the `init` event. That is deliberately preferred over
|
||||||
|
// `useVueFlow()`: this component is the PARENT of <VueFlow>, and calling
|
||||||
|
// useVueFlow() in a parent without a shared id can resolve to a different store
|
||||||
|
// than the one the rendered flow uses. The event cannot misfire that way.
|
||||||
|
const MIN_ZOOM = 0.65
|
||||||
|
const FIT_MAX_ZOOM = 1
|
||||||
|
type FlowStore = {
|
||||||
|
fitView: (opts?: Record<string, unknown>) => Promise<boolean>
|
||||||
|
getViewport?: () => { x: number; y: number; zoom: number }
|
||||||
|
setViewport?: (
|
||||||
|
t: { x: number; y: number; zoom: number },
|
||||||
|
opts?: Record<string, unknown>,
|
||||||
|
) => Promise<boolean>
|
||||||
|
}
|
||||||
|
let flowStore: FlowStore | null = null
|
||||||
|
|
||||||
|
// frameCanvas fits the graph, then re-anchors it to the top-left padding edge.
|
||||||
|
//
|
||||||
|
// Two reasons it is not just fitView():
|
||||||
|
// 1. the zoom floor (MIN_ZOOM) — fitView alone shrank 5 stacked forwards to
|
||||||
|
// ~0.31 on a phone, rendering 4.4px text;
|
||||||
|
// 2. fitView centres the bounding box, which on a narrow screen puts the seam
|
||||||
|
// between the local and remote columns mid-viewport — the user sees half of
|
||||||
|
// each. Anchoring the top-left node to the padding edge means whole local
|
||||||
|
// cards are visible immediately and the remote column is a pan away.
|
||||||
|
//
|
||||||
|
// duration:0 on both transitions — we measure node rects in between, and a
|
||||||
|
// running animation would hand back mid-flight transforms and skew the anchor.
|
||||||
|
const PANE_PAD_RATIO = 0.06
|
||||||
|
const frameCanvas = async () => {
|
||||||
|
if (!flowStore) return
|
||||||
|
await flowStore.fitView({
|
||||||
|
padding: 0.12,
|
||||||
|
minZoom: MIN_ZOOM,
|
||||||
|
maxZoom: FIT_MAX_ZOOM,
|
||||||
|
duration: 0,
|
||||||
|
})
|
||||||
|
|
||||||
|
const pane = document.querySelector<HTMLElement>('.vue-flow')
|
||||||
|
const nodeEls = Array.from(document.querySelectorAll<HTMLElement>('.vue-flow__node'))
|
||||||
|
if (!pane || !nodeEls.length) return
|
||||||
|
|
||||||
|
const paneRect = pane.getBoundingClientRect()
|
||||||
|
const rects = nodeEls.map((el) => el.getBoundingClientRect())
|
||||||
|
const dx =
|
||||||
|
paneRect.left + pane.clientWidth * PANE_PAD_RATIO - Math.min(...rects.map((r) => r.left))
|
||||||
|
const dy =
|
||||||
|
paneRect.top + pane.clientHeight * PANE_PAD_RATIO - Math.min(...rects.map((r) => r.top))
|
||||||
|
if (Math.abs(dx) < 1 && Math.abs(dy) < 1) return
|
||||||
|
|
||||||
|
const vp = flowStore.getViewport?.()
|
||||||
|
if (!vp) return
|
||||||
|
await flowStore.setViewport?.({ x: vp.x + dx, y: vp.y + dy, zoom: vp.zoom }, { duration: 0 })
|
||||||
|
}
|
||||||
|
|
||||||
|
// The flow store and the node list become available at different times: the
|
||||||
|
// store at component init, the nodes only once the async canvas fetch resolves
|
||||||
|
// (plus a further tick for Vue Flow to measure them). Framing on any single
|
||||||
|
// signal would fit an EMPTY canvas, so every path calls reframe() and it no-ops
|
||||||
|
// until there is a store and something to frame.
|
||||||
|
//
|
||||||
|
// `nodes-initialized` also fires whenever a node is added later. Auto-framing on
|
||||||
|
// that would yank the viewport out from under whoever is mid-pan, so the initial
|
||||||
|
// frame is claimed exactly once; rearranging is the only thing that re-frames on
|
||||||
|
// demand (see autoLayout).
|
||||||
|
let framedOnce = false
|
||||||
|
const reframe = (force = false) => {
|
||||||
|
if (!flowStore || !nodes.value.length) return
|
||||||
|
if (framedOnce && !force) return
|
||||||
|
framedOnce = true
|
||||||
|
nextTick(() => { void frameCanvas() })
|
||||||
|
}
|
||||||
|
|
||||||
|
const onFlowInit = (store: FlowStore) => {
|
||||||
|
flowStore = store
|
||||||
|
reframe()
|
||||||
|
}
|
||||||
|
|
||||||
// Background dot-pattern color, resolved from the live theme var so the grid
|
// Background dot-pattern color, resolved from the live theme var so the grid
|
||||||
// recolors on theme switch (vue-flow Background renders pattern-color as an
|
// recolors on theme switch (vue-flow Background renders pattern-color as an
|
||||||
// SVG fill attribute, which can't resolve var() directly — read the computed
|
// SVG fill attribute, which can't resolve var() directly — read the computed
|
||||||
@ -667,7 +765,12 @@ const remoteNodes = computed(() =>
|
|||||||
nodes.value.filter((n) => n.type === 'remote'),
|
nodes.value.filter((n) => n.type === 'remote'),
|
||||||
)
|
)
|
||||||
|
|
||||||
const autoLayout = () => layoutColumns(true)
|
// Re-frame after rearranging, bounded by the same zoom floor so "自动排列" never
|
||||||
|
// shrinks the cards back into unreadable territory on a narrow screen.
|
||||||
|
const autoLayout = () => {
|
||||||
|
layoutColumns(true)
|
||||||
|
reframe(true)
|
||||||
|
}
|
||||||
|
|
||||||
const load = async () => {
|
const load = async () => {
|
||||||
loading.value = true
|
loading.value = true
|
||||||
@ -718,6 +821,9 @@ const load = async () => {
|
|||||||
}
|
}
|
||||||
assignLayers()
|
assignLayers()
|
||||||
dirty.value = false
|
dirty.value = false
|
||||||
|
// The store exists by now but the node list has only just been populated, so
|
||||||
|
// this is the first moment a fit actually has something to frame.
|
||||||
|
reframe(true)
|
||||||
} finally {
|
} finally {
|
||||||
loading.value = false
|
loading.value = false
|
||||||
}
|
}
|
||||||
|
|||||||
@ -121,6 +121,131 @@
|
|||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<!-- Load balancer (read-only). Shown only when the endpoint responds: a
|
||||||
|
404 means this node has no balancer wired up, and an absent section is
|
||||||
|
the honest signal for "this build doesn't do balancing". -->
|
||||||
|
<section class="section w4f-card bal-section" v-if="bal">
|
||||||
|
<h3 class="section-title">
|
||||||
|
<span class="k-ic">⚖</span>负载均衡
|
||||||
|
<!-- The switch edits the mode ON THIS NODE. Every member decides about
|
||||||
|
its own forwards, so a cluster-wide toggle would be a lie: there is
|
||||||
|
no single action that flips them together. -->
|
||||||
|
<el-select
|
||||||
|
v-if="bal.canControl"
|
||||||
|
v-model="balModeChoice"
|
||||||
|
size="small"
|
||||||
|
class="bal-mode"
|
||||||
|
:disabled="modeBusy"
|
||||||
|
@change="onModeChange"
|
||||||
|
>
|
||||||
|
<el-option label="关闭" value="off" />
|
||||||
|
<el-option label="观察模式(不释放)" value="dry-run" />
|
||||||
|
<el-option label="启用(会释放)" value="on" />
|
||||||
|
</el-select>
|
||||||
|
<span class="w4f-tag" v-else v-text="'只读'" ></span>
|
||||||
|
<span class="bal-src" v-if="bal.modeSource === 'stored'">已保存</span>
|
||||||
|
<span class="bal-src" v-else>来自启动参数</span>
|
||||||
|
<span class="w4f-tag" v-if="!bal.isMember" v-text="'非集群成员'"></span>
|
||||||
|
<span class="w4f-tag" v-else-if="bal.releasing" v-text="'释放进行中'"></span>
|
||||||
|
<span class="w4f-tag w4f-tag-warn" v-else-if="bal.selfOverShare" v-text="'本机超出份额'"></span>
|
||||||
|
<span class="w4f-tag w4f-tag-ok" v-else v-text="'本机在份额内'"></span>
|
||||||
|
<span class="title-actions">
|
||||||
|
<button class="w4f-btn ghost small" :disabled="busy" @click="load">⟳ 刷新</button>
|
||||||
|
</span>
|
||||||
|
</h3>
|
||||||
|
|
||||||
|
<div class="bal-thresholds">
|
||||||
|
<span>
|
||||||
|
规则:本机转发 <b>{{ bal.selfForwards }}</b> 条
|
||||||
|
> 平摊份额 <b>{{ bal.share }}</b>
|
||||||
|
且至少 {{ 2 }} 条时,本机自行释放一条
|
||||||
|
</span>
|
||||||
|
<span>集群共 {{ bal.totalForwards }} 条 / {{ bal.aliveNodes }} 台存活</span>
|
||||||
|
<span>两次释放间隔 ≥ {{ Math.round(bal.cooldownSec / 60) }} 分钟</span>
|
||||||
|
<span v-if="bal.dryRun" class="bal-hint">
|
||||||
|
设 <code>W4F_BALANCE_MODE=on</code> 才会真正释放
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p class="bal-note">
|
||||||
|
每个节点各自判断自己的转发,超了就自己撤销一条并丢回环上,
|
||||||
|
由环节点按「最闲者优先」重新认领。没有中心调度。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<!-- Per-node forward counts. Bars are scaled to the busiest node so the
|
||||||
|
shape reads whether the cluster holds 3 forwards or 300; the share
|
||||||
|
marker is what actually decides, not the bar. -->
|
||||||
|
<div class="bal-nodes">
|
||||||
|
<div
|
||||||
|
v-for="n in balNodes"
|
||||||
|
:key="n.id"
|
||||||
|
class="bal-node"
|
||||||
|
:class="{ self: n.id === bal.self, over: n.forwards > bal.share }"
|
||||||
|
>
|
||||||
|
<div class="bn-top">
|
||||||
|
<span class="bn-id">{{ shortId(n.id) }}</span>
|
||||||
|
<span v-if="n.id === bal.self" class="w4f-tag w4f-tag-primary">本机</span>
|
||||||
|
<span v-if="n.isLeader" class="w4f-tag w4f-tag-info">LEADER</span>
|
||||||
|
<span v-if="!n.alive" class="w4f-tag w4f-tag-danger">离线</span>
|
||||||
|
<span v-if="n.alive && n.forwards > bal.share" class="w4f-tag w4f-tag-warn">超出份额</span>
|
||||||
|
</div>
|
||||||
|
<div class="w4f-bar" :class="{ err: n.alive && n.forwards > bal.share }">
|
||||||
|
<i :style="{ width: barWidth(n.forwards) + '%' }" />
|
||||||
|
</div>
|
||||||
|
<div class="bn-stats">
|
||||||
|
<span>转发 <b>{{ n.forwards }}</b></span>
|
||||||
|
<span>内存 <b>{{ n.memPct.toFixed(1) }}%</b></span>
|
||||||
|
<span>网络 <b>{{ n.netPct.toFixed(1) }}%</b></span>
|
||||||
|
<span class="bn-score">份额 <b>{{ bal.share }}</b></span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div v-if="!balNodes.length" class="empty">暂无节点负载数据</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- What THIS node would do. The answer is per-node by design: every
|
||||||
|
member runs the same rule on its own forwards, so there is no
|
||||||
|
cluster-wide plan and no "the leader will move X to Y". -->
|
||||||
|
<div class="bal-plan">
|
||||||
|
<div v-if="bal.releasing" class="bp-inflight">
|
||||||
|
<span class="w4f-tag w4f-tag-warn">释放进行中</span>
|
||||||
|
<span class="bp-text">
|
||||||
|
本机已释放 {{ bal.releasing.local }}:{{ bal.releasing.remotePort }},
|
||||||
|
等待环把它认领到别的节点
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div v-else-if="bal.wouldRelease && !bal.blockedBy" class="bp-would">
|
||||||
|
<span class="w4f-tag w4f-tag-warn">将释放</span>
|
||||||
|
<span class="bp-text">
|
||||||
|
本机将撤销 {{ bal.wouldRelease.local }}:{{ bal.wouldRelease.remotePort }}
|
||||||
|
并丢回环上,由最闲节点接管
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div v-else-if="bal.wouldRelease && bal.blockedBy === 'cooldown'" class="bp-waiting">
|
||||||
|
<span class="w4f-tag w4f-tag-info">冷却中</span>
|
||||||
|
<span class="bp-text">
|
||||||
|
本机超出份额,但距上次释放不足
|
||||||
|
{{ Math.round(bal.cooldownSec / 60) }} 分钟
|
||||||
|
<template v-if="bal.lastReleaseAt">
|
||||||
|
(已过 {{ cooldownAgo }})
|
||||||
|
</template>
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<div v-else-if="bal.wouldRelease" class="bp-waiting">
|
||||||
|
<span class="w4f-tag w4f-tag-info">暂不释放</span>
|
||||||
|
<span class="bp-text">本机超出份额,原因:{{ blockedLabel }}</span>
|
||||||
|
</div>
|
||||||
|
<div v-else-if="!bal.isMember" class="bp-ok">
|
||||||
|
<span class="w4f-tag w4f-tag-ok">单节点</span>
|
||||||
|
<span class="bp-text">本机未加入集群,无需均衡</span>
|
||||||
|
</div>
|
||||||
|
<div v-else class="bp-ok">
|
||||||
|
<span class="w4f-tag w4f-tag-ok">已均衡</span>
|
||||||
|
<span class="bp-text">
|
||||||
|
本机持有 {{ bal.selfForwards }} 条,未超平摊份额 {{ bal.share }},无需释放
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
<!-- Active forward topology -->
|
<!-- Active forward topology -->
|
||||||
<section class="section w4f-card">
|
<section class="section w4f-card">
|
||||||
<h3 class="section-title"><span class="k-ic">→</span>活跃转发拓扑</h3>
|
<h3 class="section-title"><span class="k-ic">→</span>活跃转发拓扑</h3>
|
||||||
@ -199,14 +324,18 @@
|
|||||||
</template>
|
</template>
|
||||||
|
|
||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { ref, reactive, computed, onMounted, onBeforeUnmount } from 'vue'
|
import { ref, reactive, computed, onMounted, onBeforeUnmount, watch } from 'vue'
|
||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage } from 'element-plus'
|
||||||
import { api, downloadAuditCsv } from '../api'
|
import { api, downloadAuditCsv } from '../api'
|
||||||
import { canWrite } from '../auth'
|
import { canWrite } from '../auth'
|
||||||
import type { RingSnapshot, RingTaskInfo, RingLogEntry } from '../types'
|
import type { BalanceReport, RingSnapshot, RingTaskInfo, RingLogEntry } from '../types'
|
||||||
|
|
||||||
const ring = ref<RingSnapshot | null>(null)
|
const ring = ref<RingSnapshot | null>(null)
|
||||||
const busy = ref(false)
|
const busy = ref(false)
|
||||||
|
// bal is null when the node has no balancer wired up (endpoint 404s). The
|
||||||
|
// section is simply not rendered then, which is the honest signal for "this
|
||||||
|
// build does not do balancing" — better than an empty panel implying it does.
|
||||||
|
const bal = ref<BalanceReport | null>(null)
|
||||||
let timer: number | null = null
|
let timer: number | null = null
|
||||||
let visHandler: (() => void) | null = null
|
let visHandler: (() => void) | null = null
|
||||||
|
|
||||||
@ -214,8 +343,76 @@ const joinDialog = reactive({ visible: false, addr: '', key: '' })
|
|||||||
|
|
||||||
const load = async () => {
|
const load = async () => {
|
||||||
try { ring.value = await api.ring() } catch { /* keep last */ }
|
try { ring.value = await api.ring() } catch { /* keep last */ }
|
||||||
|
// Loaded alongside the ring, but unlike it a failure clears the view: the
|
||||||
|
// balancer report is a decision surface, and showing a stale "已均衡" after the
|
||||||
|
// endpoint starts failing would be worse than showing nothing.
|
||||||
|
try { bal.value = await api.balance() } catch { bal.value = null }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- load balancer mode switch ---- //
|
||||||
|
// The select is bound to a local mirror so the UI can show the pending choice
|
||||||
|
// while the request is in flight, and snap back if the server rejects it.
|
||||||
|
const balModeChoice = ref<string>('dry-run')
|
||||||
|
const modeBusy = ref(false)
|
||||||
|
watch(
|
||||||
|
() => bal.value?.mode,
|
||||||
|
(m) => { if (m) balModeChoice.value = m },
|
||||||
|
{ immediate: true },
|
||||||
|
)
|
||||||
|
const onModeChange = async (m: string) => {
|
||||||
|
if (!bal.value || modeBusy.value) return
|
||||||
|
modeBusy.value = true
|
||||||
|
try {
|
||||||
|
bal.value = await api.setBalanceMode(m)
|
||||||
|
ElMessage.success(
|
||||||
|
m === 'on' ? '本机已启用负载释放' : m === 'off' ? '本机已关闭负载均衡' : '本机已切到观察模式',
|
||||||
|
)
|
||||||
|
} catch (e) {
|
||||||
|
// Snap back: leaving the select showing a mode the server did not accept
|
||||||
|
// would misrepresent what the node is doing.
|
||||||
|
balModeChoice.value = bal.value.mode
|
||||||
|
ElMessage.error('切换失败:' + (e instanceof Error ? e.message : String(e)))
|
||||||
|
} finally {
|
||||||
|
modeBusy.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- load balancer view ---- //
|
||||||
|
// Each node decides about its OWN forwards, so this page has no cluster-wide
|
||||||
|
// plan to show — only what THIS node holds and what it is about to do.
|
||||||
|
// Ordered by descending count: the busiest node first is what an operator
|
||||||
|
// scanning for a hotspot wants.
|
||||||
|
const balNodes = computed(() => [...(bal.value?.nodes ?? [])].sort((a, b) => b.forwards - a.forwards))
|
||||||
|
// barWidth scales against the busiest node rather than an absolute, so the shape
|
||||||
|
// is readable whether the cluster holds 3 forwards or 300. The bar is context;
|
||||||
|
// `share` is what actually decides whether a node is over.
|
||||||
|
const barWidth = (forwards: number) => {
|
||||||
|
const max = Math.max(...(bal.value?.nodes ?? []).map((n) => n.forwards), 1)
|
||||||
|
return Math.min(100, Math.round((forwards / max) * 100))
|
||||||
|
}
|
||||||
|
// blockedLabel renders the server's machine reason as something readable. The
|
||||||
|
// reason matters: "over its share but not acting" is otherwise indistinguishable
|
||||||
|
// from "broken".
|
||||||
|
const blockedLabel = computed(() => {
|
||||||
|
switch (bal.value?.blockedBy) {
|
||||||
|
case 'cooldown': return '冷却期未过'
|
||||||
|
case 'in-flight': return '上一次释放尚未确认'
|
||||||
|
case 'nothing-active': return '本机可释放的转发都已停用'
|
||||||
|
case 'below-min': return '持有量低于下限'
|
||||||
|
case 'single-node': return '单节点环'
|
||||||
|
default: return bal.value?.blockedBy || '未说明'
|
||||||
|
}
|
||||||
|
})
|
||||||
|
// cooldownAgo says how long ago the last release was, so an operator can see
|
||||||
|
// whether the cooldown is about to expire.
|
||||||
|
const cooldownAgo = computed(() => {
|
||||||
|
const t = bal.value?.lastReleaseAt ?? 0
|
||||||
|
if (!t) return ''
|
||||||
|
const s = Math.max(0, Math.floor(nowMs.value / 1000) - t)
|
||||||
|
if (s < 60) return s + ' 秒前'
|
||||||
|
return Math.floor(s / 60) + ' 分钟前'
|
||||||
|
})
|
||||||
|
|
||||||
const shortId = (id: string) => (id.length > 22 ? '…' + id.slice(-20) : id)
|
const shortId = (id: string) => (id.length > 22 ? '…' + id.slice(-20) : id)
|
||||||
|
|
||||||
// copyNodeKey copies this node's admission key to the clipboard so the operator
|
// copyNodeKey copies this node's admission key to the clipboard so the operator
|
||||||
@ -260,6 +457,8 @@ const stateDotClass = computed(() => {
|
|||||||
|
|
||||||
// ---- last-sync freshness (ops cares "环还活着吗", not the cycle counter) ----
|
// ---- last-sync freshness (ops cares "环还活着吗", not the cycle counter) ----
|
||||||
const now = ref(Date.now())
|
const now = ref(Date.now())
|
||||||
|
// nowMs mirrors `now` as a plain number for the balance section's minute math.
|
||||||
|
const nowMs = computed(() => now.value)
|
||||||
let tick: number | null = null
|
let tick: number | null = null
|
||||||
const lastSyncAgo = computed(() => {
|
const lastSyncAgo = computed(() => {
|
||||||
const ls = ring.value?.lastSync ?? 0
|
const ls = ring.value?.lastSync ?? 0
|
||||||
@ -481,7 +680,11 @@ onBeforeUnmount(() => {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped lang="scss">
|
<style scoped lang="scss">
|
||||||
.cluster-page { height: 100%; display: flex; flex-direction: column; overflow-y: auto; padding: 4px 0 40px; gap: 14px; }
|
/* Let .content (App.vue) remain the single scroll container — see the note in
|
||||||
|
* StatusView: a page-level scroller sized by leftover flex space is what froze
|
||||||
|
* the status page. `.log-list` keeps its own bounded scroll (that one is
|
||||||
|
* deliberate — a log pane should not push the whole page). */
|
||||||
|
.cluster-page { display: flex; flex-direction: column; padding: 4px 0 40px; gap: 14px; }
|
||||||
|
|
||||||
/* hero */
|
/* hero */
|
||||||
.hero { display: flex; align-items: center; justify-content: space-between; gap: 16px; flex-wrap: wrap; padding: 16px 20px; }
|
.hero { display: flex; align-items: center; justify-content: space-between; gap: 16px; flex-wrap: wrap; padding: 16px 20px; }
|
||||||
@ -634,4 +837,66 @@ onBeforeUnmount(() => {
|
|||||||
.kpis { grid-template-columns: minmax(0, 1fr); }
|
.kpis { grid-template-columns: minmax(0, 1fr); }
|
||||||
.page-title { font-size: 15.5px; }
|
.page-title { font-size: 15.5px; }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ---- Load balancer section ---- */
|
||||||
|
.bal-section .section-title { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; }
|
||||||
|
.bal-section .section-title .k-ic { margin-right: 0; }
|
||||||
|
|
||||||
|
.bal-thresholds {
|
||||||
|
display: flex; flex-wrap: wrap; gap: 6px 14px;
|
||||||
|
font-size: 11.5px; color: var(--w4f-muted);
|
||||||
|
margin-bottom: 10px;
|
||||||
|
}
|
||||||
|
.bal-hint { color: var(--w4f-warning); }
|
||||||
|
.bal-hint code {
|
||||||
|
font-family: var(--w4f-mono, ui-monospace, monospace);
|
||||||
|
background: var(--w4f-card-2); padding: 1px 4px; border-radius: 3px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.bal-nodes {
|
||||||
|
display: grid; gap: 8px;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
|
||||||
|
}
|
||||||
|
.bal-node {
|
||||||
|
border: 1px solid var(--w4f-line);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 9px 11px;
|
||||||
|
background: var(--w4f-card-2);
|
||||||
|
min-width: 0;
|
||||||
|
}
|
||||||
|
.bal-node.self { border-color: var(--w4f-primary); }
|
||||||
|
.bal-node.over { border-color: var(--w4f-warning); }
|
||||||
|
.bal-mode { width: 168px; }
|
||||||
|
.bal-src { font-size: 11px; color: var(--w4f-muted); }
|
||||||
|
.bal-note {
|
||||||
|
margin: 0 0 10px; font-size: 11.5px; line-height: 1.55;
|
||||||
|
color: var(--w4f-muted);
|
||||||
|
}
|
||||||
|
.bn-top { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; margin-bottom: 7px; }
|
||||||
|
.bn-id {
|
||||||
|
font-family: var(--w4f-mono, ui-monospace, monospace);
|
||||||
|
font-size: 12px; font-weight: 700;
|
||||||
|
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||||
|
}
|
||||||
|
.bn-stats {
|
||||||
|
display: flex; flex-wrap: wrap; gap: 3px 12px;
|
||||||
|
font-size: 11.5px; color: var(--w4f-muted); margin-top: 7px;
|
||||||
|
}
|
||||||
|
.bn-stats b { color: var(--w4f-text); font-variant-numeric: tabular-nums; }
|
||||||
|
.bn-score { margin-left: auto; }
|
||||||
|
|
||||||
|
.bal-plan { margin-top: 11px; }
|
||||||
|
.bp-inflight, .bp-would, .bp-waiting, .bp-ok {
|
||||||
|
display: flex; align-items: center; gap: 9px; flex-wrap: wrap;
|
||||||
|
padding: 8px 11px; border-radius: 8px;
|
||||||
|
font-size: 12.5px;
|
||||||
|
background: var(--w4f-card-2);
|
||||||
|
border: 1px solid var(--w4f-line);
|
||||||
|
}
|
||||||
|
.bp-inflight { border-color: var(--w4f-warning); }
|
||||||
|
.bp-text { min-width: 0; overflow-wrap: anywhere; }
|
||||||
|
|
||||||
|
@media (max-width: 380px) {
|
||||||
|
.bn-score { margin-left: 0; }
|
||||||
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@ -167,11 +167,12 @@ onMounted(load)
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped lang="scss">
|
<style scoped lang="scss">
|
||||||
|
/* Let .content (App.vue) remain the single scroll container — see the note in
|
||||||
|
* StatusView: a page-level scroller sized by leftover flex space can collapse
|
||||||
|
* to zero height on narrow screens. */
|
||||||
.settings-page {
|
.settings-page {
|
||||||
height: 100%;
|
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
overflow-y: auto;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
.page-top {
|
.page-top {
|
||||||
|
|||||||
@ -496,11 +496,22 @@ onBeforeUnmount(() => {
|
|||||||
</script>
|
</script>
|
||||||
|
|
||||||
<style scoped lang="scss">
|
<style scoped lang="scss">
|
||||||
|
/* The pages inside .content must NOT be their own scrollers.
|
||||||
|
*
|
||||||
|
* App.vue already designates `.content` as the app's single scroll container
|
||||||
|
* (`flex:1; overflow-y:auto`). These pages were also `height:100%` +
|
||||||
|
* `overflow-y:auto`, producing a nested scroller whose height is the leftover
|
||||||
|
* space of a flex column. On a 375px phone the status page's fixed-height
|
||||||
|
* header (122px) plus the single-column KPI stack (565px) exceed the available
|
||||||
|
* 651px, so the inner `.status-body` was handed 0px while holding 1682px of
|
||||||
|
* content — the page looked completely frozen, since neither `.content` (which
|
||||||
|
* had nothing to scroll) nor the 0-height inner box could be swiped.
|
||||||
|
*
|
||||||
|
* Letting the page grow naturally and scroll in `.content` removes the whole
|
||||||
|
* class of bug: there is no leftover-space arithmetic left to get wrong. */
|
||||||
.status-page {
|
.status-page {
|
||||||
height: 100%;
|
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
overflow: hidden;
|
|
||||||
gap: 14px;
|
gap: 14px;
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -527,12 +538,9 @@ onBeforeUnmount(() => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
.status-body {
|
.status-body {
|
||||||
flex: 1;
|
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 18px;
|
gap: 18px;
|
||||||
overflow-y: auto;
|
|
||||||
padding-right: 4px;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
.section { display: flex; flex-direction: column; }
|
.section { display: flex; flex-direction: column; }
|
||||||
@ -720,8 +728,13 @@ onBeforeUnmount(() => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@media (max-width: 380px) {
|
@media (max-width: 380px) {
|
||||||
/* 极窄:KPI 也降为单列,否则数字会被压断 */
|
/* Keep the KPI grid at TWO columns even on the narrowest phones.
|
||||||
.kpis { grid-template-columns: minmax(0, 1fr); }
|
* Dropping to one column made four full-width cards ~565px tall, which on its
|
||||||
|
* own exceeded the whole viewport and (with the page-level scroller of the
|
||||||
|
* day) collapsed the content area to zero height. Two columns keeps each card
|
||||||
|
* ~160px wide — enough for the 27px number + 11.5px caption, which is all a
|
||||||
|
* KPI card holds. */
|
||||||
|
.kpis { gap: 8px; }
|
||||||
.fwd-route { font-size: 12.5px; }
|
.fwd-route { font-size: 12.5px; }
|
||||||
.page-title { font-size: 16px; }
|
.page-title { font-size: 16px; }
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user