fix: 集群操作日志同步修复 — 全量回填 + 重启节点 seq 水位抬升

两个根因:
1. 离线错过条目永久缺失: token delta 被下游 trim (keep=seq>wm),
   离线节点错过的 seq 再也收不到 → 'log delta gap: want N got N+1' 死循环。
   修复: OnToken 改为附带自己的全量日志 (Snapshot),接收方 ApplyDelta 按 seq 幂等去重,
   缺口节点下一轮自动补齐。日志量小(几十条),开销可忽略。

2. 重启节点重新从 seq=1 编号: NewClusterLog 从 Seq=0 起,重启后产生的新事件
   与环上历史 seq 冲突 → ApplyDelta 视为 already-have 静默丢弃 + 全量附带出现歧义 id。
   修复: OnToken 收到日志时把本地 Seq 抬到环高水位之上,新编号接在历史之后。

测试: TestFullLogBackfillAfterGap / TestApplyDeltaIdempotentOnFullResend

实测: 三台集群撤销 minecraft 转发 → 三台均记录 seq=10 forward.remove;
恢复后三台均记录 seq=11 forward.add
This commit is contained in:
JianFeeeee
2026-08-25 11:55:34 +08:00
parent 94b6396738
commit 1abf1bb447
5 changed files with 350 additions and 218 deletions

View File

@ -238,6 +238,16 @@ func (e *Engine) OnToken(ctx context.Context, tk *Token) (*Token, error) {
// (b) apply incremental log delta; trim consumed entries off the token.
if e.Log != nil && len(tk.Log) > 0 {
// Raise the local seq allocator above the ring's high-water mark so a
// restarted node (whose log was rebuilt from scratch at Seq=0) never
// re-issues sequence numbers that already exist in the shared history —
// duplicate seqs would make ApplyDelta silently drop the new entries as
// "already have" and pollute full-log attachments with ambiguous ids.
for _, en := range tk.Log {
if en.Seq > e.Log.Seq {
e.Log.Seq = en.Seq
}
}
wm, err := e.Log.ApplyDelta(tk.Log)
if err != nil {
log.Printf("ring[%s] log delta gap: %v (request full sync later)", e.ID, err)
@ -275,9 +285,15 @@ func (e *Engine) OnToken(ctx context.Context, tk *Token) (*Token, error) {
// flows to the newcomer (its successor) so it can participate.
e.injectPendingJoin()
// re-attach own fresh log entries so peers converge.
// re-attach OWN full log so peers converge even after gaps. A node that
// was offline when seq=N circulated never gets seq=N from the delta (it
// was trimmed off by peers whose watermark advanced past N). Attaching the
// FULL local log every cycle lets any peer missing entries backfill them
// next round — the cluster log is small (tens of entries) so the cost is
// negligible. ApplyDelta dedupes by seq so re-sent entries are a no-op
// for peers that already have them.
if !e.selfRemoved && e.Log != nil {
mine := e.Log.EntriesAfter(e.lastLogSent)
mine := e.Log.Snapshot()
if len(mine) > 0 {
tk.Log = append(tk.Log, mine...)
if last := mine[len(mine)-1]; last.Seq > e.lastLogSent {