mirror of
https://gitcode.com/JianFeeeee/webui4frpc.git
synced 2026-10-03 07:34:00 +00:00
fix(cluster): 停用的转发会在重启后自己复活 + 令牌轮转日志刷屏
从线上三节点(192.168.2.{30,106,60})的日志里挖出四个问题,本轮修三个。
## 1. 停用的转发会复活(功能性缺陷,实测仍在发生)
线上现象:`minecraft` 在 store 里 disabled=1,worker 却仍在跑,今天
09:46 还在刷 `connect to local service [192.168.2.60:25565]: connection
refused` —— 对着一个用户刻意没启的本地服务死刷。同时三台 logs 里躺着
13956 / 3769 条 `proxy [x] already exists`,每 33 秒一轮。
四处叠加导致:
- stopForward 先读 link,再 SetLinkDisabled(true),然后把**改之前**的
副本交给 RevokeTask ⇒ published task 带 disabled=false(实测 id/flag
都对不上:topology 里 link.id=223,store 里同一行是 199)
- ClaimFn **无条件** SetLinkDisabled(...,false)。原意是"重新认领时清掉
停用标记",但启动 reconcile 只要 worker 不在就重新 Claim ⇒ 每次重启
都是"先清标记再起 worker"
- RevokeFn 停了 worker 却**没删 topology 条目**,条目活过 worker
- 于是下次重启 reconcile 看到"owned 但 worker 不在"→ 再次 Claim → 死循环
修法(把 disabled 的所有权交回两个用户动作):
- Claim **只读** disabled 决定要不要起 worker;为 true 时连 topology
条目一起摘掉,绝不复活
- 启动 reconcile 先按 store 跳过 disabled 的条目(省掉无谓的
claim→skip 往返)
- RevokeFn 除停 worker 外,同时 RemoveTopologyEntry —— 撤销必须是
完整退役,不能只是"停一下"
- stopForward 把 Disabled=true 随 task 发布出去,让持有该转发的节点
即使本地 store 行陈旧也能判断这次停用是用户主动的
## 2. 令牌轮转日志零信息量却占满磁盘
每轮固定 3 行(OnToken cycle=N / forward cycle=N / token-send -> 200),
2 轮/秒,实测本机 **355 行/分钟、7 天 357 万行**,把真事件全淹了。
同一份信息(cycle / lastSync / roundDelayMs / 成员存活)本来就能从
GET /api/manager/cluster/ring 结构化拿到。
加 W4F_DEBUG 开关(沿用项目既有 W4F_ 前缀约定):稳态三行降级为 debug、
默认关闭;**失败路径一律保留** —— 发送失败、陈旧令牌、非 2xx 正是别人
grep 的对象,静音它们是坏交易。实测同样 12 秒:36 行 → 4 行。
## 3. Link.ID 在 ReplaceLinks 之后必然失效
ReplaceLinks 是 DELETE + 重新 INSERT,sqlite 给每行**新的自增 id**。
任何在改写前捕获的 Link(典型:随 token 环跑的 Link)手里的 id 要么查无
此行,要么命中另一条转发 —— 实测捕获 alpha id=1,改写后新表是 3/4/5,
GetLink(1) 直接落空。
新增 LinkByTriple(local, remote, port) 按自然键查(业务代码本来就一律用
这个三元组标识转发),并把 claim/reconcile 切过去。查无行返回
(Link{}, false, nil) 而非 error:新建的转发没有行,应当照常启动。
## 4. homeagent_device 孤儿(已澄清,非独立缺陷)
它 disabled=1 且从不在 topology 里,是缺陷 1 的另一面(停用标记没进
token),随本次修复覆盖,无需单独处理。
## 测试
新增 4 个测试文件,重点是**验证测试本身抓得住 bug**:
- 临时回退 `ln.Disabled = true` 这行 → TestStopForwardPublishesDisabled-
FlagInRevokeTask 如期变红,还原后变绿
- ⚠️ 第一版回归测试只断言 store 层,是**假绿**:newTestHandler 的 Ring
为 nil,RevokeTask 那条(真正坏掉的)路根本没执行。补了带 ring 的
newRingTestHandler,直接断言**发布出去的 task 上的 flag**
- LinkByTriple 在 ReplaceLinks 前后保持稳定;GetLink(id) 的失效被固化成
一个可见的说明性测试
- 停用/start 往返、per-forward 停用不误伤兄弟转发
- 错误路径不静音、W4F_DEBUG 各种取值
go build / go vet / go test ./... 全绿,gofmt 干净。
This commit is contained in:
119
internal/httpapi/forward_revoke_test.go
Normal file
119
internal/httpapi/forward_revoke_test.go
Normal file
@ -0,0 +1,119 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"webui4frpc/internal/cluster"
|
||||
"webui4frpc/internal/process"
|
||||
"webui4frpc/internal/store"
|
||||
)
|
||||
|
||||
// newRingTestHandler builds a Handler WITH a ring engine attached, so the
|
||||
// paths that publish tasks into the token actually execute. newTestHandler
|
||||
// leaves Ring nil, which silently skips them — a test built on it can pass
|
||||
// while the publish side is completely broken.
|
||||
func newRingTestHandler(t *testing.T) (*Handler, *cluster.Engine, *httptest.Server) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
st, err := store.New(filepath.Join(dir, "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
|
||||
pm := process.NewManager(process.Options{
|
||||
ConfigsDir: filepath.Join(dir, "configs"),
|
||||
LogsDir: filepath.Join(dir, "logs"),
|
||||
BinaryPath: func() string { return "" },
|
||||
Render: func(string) ([]byte, error) { return []byte(`{}`), nil },
|
||||
AutoRestart: func(string) bool { return false },
|
||||
RestartInterval: func() int { return 5 },
|
||||
})
|
||||
ring := cluster.NewEngine("n1", "n1:7500", "u", "p", "0.1.0", nil,
|
||||
&cluster.AppHandler{},
|
||||
func(ctx context.Context, next string, tk *cluster.Token) error { return nil },
|
||||
"n1:7500", true, "")
|
||||
h := &Handler{Store: st, Process: pm, WorkDir: dir, User: "admin", Password: "pw", Ring: ring}
|
||||
mux, err := NewServeMux(h)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ts := httptest.NewServer(mux)
|
||||
t.Cleanup(ts.Close)
|
||||
return h, ring, ts
|
||||
}
|
||||
|
||||
func saveCanvas(t *testing.T, srv *httptest.Server, body string) {
|
||||
t.Helper()
|
||||
req, _ := http.NewRequest(http.MethodPut, srv.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := srv.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("canvas save status = %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopForwardPublishesDisabledFlagInRevokeTask is the regression test for
|
||||
// the actual defect.
|
||||
//
|
||||
// stopForward() read the link, called SetLinkDisabled(true), and then handed
|
||||
// the STALE copy (disabled=false) to RevokeTask. The revoke travels to the
|
||||
// node that OWNS the forward, and that node's Claim/Revoke path keys off the
|
||||
// flag — so a stale false meant:
|
||||
// - the owner could not tell the stop was deliberate, and
|
||||
// - nothing retired the topology entry,
|
||||
//
|
||||
// so the next restart's reconcile re-claimed the forward and spawned a worker
|
||||
// for something the user had stopped (seen live: endless connection-refused
|
||||
// against an intentionally-down service).
|
||||
//
|
||||
// This asserts the flag ON THE PUBLISHED TASK, which is the value that was
|
||||
// actually wrong. It cannot be satisfied by the store write alone.
|
||||
func TestStopForwardPublishesDisabledFlagInRevokeTask(t *testing.T) {
|
||||
_, ring, ts := newRingTestHandler(t)
|
||||
|
||||
saveCanvas(t, ts, `{
|
||||
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||
}`)
|
||||
|
||||
// Stop the forward over the API.
|
||||
b, _ := json.Marshal(stopForwardReq{"svc", "srv-a", 45999})
|
||||
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/api/manager/forwards/stop", bytes.NewReader(b))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("stop status = %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
// Find the revoke task that was published into the token.
|
||||
var revoke *cluster.Task
|
||||
for _, tk := range ring.State().PendingList() {
|
||||
if tk.Revoke && tk.Local.Name == "svc" && tk.Link.RemotePort == 45999 {
|
||||
revoke = tk
|
||||
break
|
||||
}
|
||||
}
|
||||
if revoke == nil {
|
||||
t.Fatal("stop did not publish a revoke task for the forward")
|
||||
}
|
||||
if !revoke.Link.Disabled {
|
||||
t.Fatal("the published revoke task carries disabled=false — the owner node cannot tell " +
|
||||
"this stop was deliberate, which is the bug that let stopped forwards resurrect")
|
||||
}
|
||||
}
|
||||
141
internal/httpapi/forward_stop_test.go
Normal file
141
internal/httpapi/forward_stop_test.go
Normal file
@ -0,0 +1,141 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// stopForwardReq mirrors the forwards start/stop request body.
|
||||
type stopForwardReq struct {
|
||||
Local string `json:"local"`
|
||||
Remote string `json:"remote"`
|
||||
RemotePort int `json:"remotePort"`
|
||||
}
|
||||
|
||||
func postForwards(t *testing.T, srv *httptest.Server, action string, body stopForwardReq) int {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(body)
|
||||
req, _ := http.NewRequest(http.MethodPost, srv.URL+"/api/manager/forwards/"+action, bytes.NewReader(b))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := srv.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
|
||||
// TestStopForwardPersistsDisabledFlag is the regression test for the
|
||||
// "stopped forwards resurrect on restart" bug.
|
||||
//
|
||||
// The original defect: stopForward() read the link, called
|
||||
// SetLinkDisabled(true), but then handed the STALE (disabled=false) copy to
|
||||
// RevokeTask — so the disabled flag never reached the node owning the forward,
|
||||
// and nothing removed the topology entry. On the next restart the startup
|
||||
// reconcile saw an owned forward with no worker and re-claimed it, spawning a
|
||||
// worker for a forward the user had deliberately stopped (observed live:
|
||||
// ~14k "proxy already exists" retries and endless connection-refused against a
|
||||
// service that was intentionally down).
|
||||
//
|
||||
// The contract this pins: after a successful stop, the persisted link MUST be
|
||||
// disabled — that flag is the single source of truth the claim path consults.
|
||||
func TestStopForwardPersistsDisabledFlag(t *testing.T) {
|
||||
h, ts := newTestHandler(t)
|
||||
|
||||
body := `{
|
||||
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||
"links": [{"local":"svc","remote":"srv-a","remotePort":45999}]
|
||||
}`
|
||||
req, _ := http.NewRequest(http.MethodPut, ts.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("canvas save status = %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
// The forward starts enabled.
|
||||
ln, found, err := h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
if err != nil || !found {
|
||||
t.Fatalf("link not persisted: found=%v err=%v", found, err)
|
||||
}
|
||||
if ln.Disabled {
|
||||
t.Fatal("a freshly saved forward must start enabled")
|
||||
}
|
||||
|
||||
// Stop it.
|
||||
if code := postForwards(t, ts, "stop", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||
t.Fatalf("stop status = %d, want 200", code)
|
||||
}
|
||||
|
||||
// Persisted flag must now be set — this is what the claim path reads.
|
||||
ln, found, err = h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("link vanished after stop; stop must be non-destructive")
|
||||
}
|
||||
if !ln.Disabled {
|
||||
t.Fatal("stop did not persist disabled=true — the startup reconcile would resurrect this forward")
|
||||
}
|
||||
|
||||
// Start must clear it again (the user-facing re-enable path).
|
||||
if code := postForwards(t, ts, "start", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||
t.Fatalf("start status = %d, want 200", code)
|
||||
}
|
||||
ln, _, err = h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ln.Disabled {
|
||||
t.Fatal("start did not clear disabled; a re-enabled forward would stay stopped")
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopForwardIsNonDestructive pins the per-forward stop semantics the
|
||||
// revoke path was specifically rewritten for: stopping one forward must not
|
||||
// touch a sibling forward that shares the same local or remote.
|
||||
func TestStopForwardIsNonDestructive(t *testing.T) {
|
||||
h, ts := newTestHandler(t)
|
||||
|
||||
body := `{
|
||||
"locals": [{"name":"svc","ip":"127.0.0.1","port":59999,"protocol":"tcp"}],
|
||||
"remotes": [{"name":"srv-a","ip":"1.2.3.4","port":7000,"enabled":true}],
|
||||
"links": [
|
||||
{"local":"svc","remote":"srv-a","remotePort":45999},
|
||||
{"local":"svc","remote":"srv-a","remotePort":46000}
|
||||
]
|
||||
}`
|
||||
req, _ := http.NewRequest(http.MethodPut, ts.URL+"/api/manager/canvas", bytes.NewBufferString(body))
|
||||
req.SetBasicAuth("admin", "pw")
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
|
||||
if code := postForwards(t, ts, "stop", stopForwardReq{"svc", "srv-a", 45999}); code != http.StatusOK {
|
||||
t.Fatalf("stop status = %d", code)
|
||||
}
|
||||
|
||||
stopped, _, _ := h.Store.LinkByTriple("svc", "srv-a", 45999)
|
||||
sibling, found, _ := h.Store.LinkByTriple("svc", "srv-a", 46000)
|
||||
if !found {
|
||||
t.Fatal("sibling forward was destroyed by stopping its neighbour")
|
||||
}
|
||||
if !stopped.Disabled {
|
||||
t.Error("the stopped forward should be disabled")
|
||||
}
|
||||
if sibling.Disabled {
|
||||
t.Error("the sibling forward must stay enabled — per-forward stop, not per-local/remote")
|
||||
}
|
||||
}
|
||||
@ -109,6 +109,16 @@ func (h *Handler) stopForward(local, remote string, port int) error {
|
||||
} else {
|
||||
_ = h.Store.SetLinkDisabled(local, remote, port, true)
|
||||
}
|
||||
// The revoke task travels to whichever node OWNS the forward, and that node
|
||||
// re-reads the disabled flag from its own store before starting a worker —
|
||||
// so the flag has to be set on every node that has a copy of this link, not
|
||||
// just the one handling this request. Propagating Disabled on the task lets
|
||||
// the owner's RevokeFn stop the worker even if its own store row is stale.
|
||||
//
|
||||
// This also fixes a latent inconsistency: `ln` was read BEFORE the
|
||||
// SetLinkDisabled(true) above, so the link published into the token still
|
||||
// carried disabled=false and got copied into the topology entry verbatim.
|
||||
ln.Disabled = true
|
||||
if loc.LocalOnly {
|
||||
if h.Process != nil {
|
||||
key := process.WorkerKey(local, remote, port)
|
||||
|
||||
Reference in New Issue
Block a user