test(harness): 变异测试拒绝在脏工作区上运行

两个变异 harness(Go 版 discover、Python 版 quota)都是「把变异写进源码
文件、跑测试、再还原」。这个形状有两个陷阱,上一轮都真踩到了:

一、脏工作区上的 baseline 是错的

变异被覆盖写入实现文件。如果文件本来就有未提交改动(上一轮被 background
task 超时杀掉、Ctrl-C 中断、编辑器里改了一半),那么:

  - 捕获的「原始内容」不是已提交的源码,baseline 跑的是别的东西
  - 留下的那个变异会被下一轮当成 baseline

两种情况下所有裁决都毫无意义,而 harness 仍然报出干净的分数 —— 正是
「判据失效」最坏的一种:它不报错。

改为直接拒绝运行(Go 版 t.Fatalf,Python 版退出码 2),并且不自动 checkout:
脚本分不清「残留变异」和「人正在改」,静默回退真实改动比停下来更糟。

二、注入必须原子,还原必须覆盖所有退出路径

Python 版原来用 open(path,"w"),就地截断。与它并发的构建(background
go test、编辑器 gofmt、编译器自己读文件)可能看到写了一半的文件,报
syntax error,然后被当成 COMPILE 静默豁免 —— 一个真实的变异就这样被放过。
改为 tempfile + os.replace。还原登记到 atexit 与 SIGINT/SIGTERM。

三、路径必须相对当前目录

Go 版最初用 cwd=包目录 + 仓根相对路径,git 匹配不到任何东西,守卫永远不
触发。两个 harness 现在都用 ./name,各自在自己目录里跑 git。

两个方向都验证过:干净时全杀(discover 11/11、quota 12/12),脏时拒绝运行。

(cherry picked from commit d01712a697)
This commit is contained in:
JianFeeeee
2026-10-07 03:58:51 +08:00
parent 0858a747ca
commit b6d0358079

View File

@ -15,6 +15,11 @@ package gateway
// the filter selects a non-trivial number of tests (see baseline below), and it
// treats "[no tests to run]" as a hard error rather than a green run.
//
// It also must not be run against a DIRTY tree: the mutant is written over
// discover.go, and a leftover mutant from a killed run would make the baseline
// green against broken code while every verdict came from the wrong source.
// See the guard at the top of TestRealMutants.
//
// Run: go test -tags luajit ./internal/gateway/ -run TestRealMutants -v
import (
@ -49,6 +54,23 @@ func TestRealMutants(t *testing.T) {
if err != nil {
t.Fatal(err)
}
// Refuse to run against a dirty tree.
//
// The mutant is written OVER discover.go, so on a dirty file the "original"
// captured above is not the committed source — the baseline then runs against
// whatever was there, and a killed run (a background-task timeout, a Ctrl-C)
// leaves a mutant behind that the NEXT run treats as the baseline. Both make
// every verdict meaningless while the harness still reports a clean score.
//
// Refusing rather than restoring: this cannot tell "leftover mutant" apart from
// "the developer is mid-edit", and silently reverting real work is worse than
// stopping.
if dirty, derr := mutationTargetIsDirty(wd, "discover.go"); derr == nil && dirty {
t.Fatalf("%s has uncommitted changes — commit or stash them first. A mutant "+
"left behind by a killed run would otherwise become the baseline, and "+
"every verdict below would be about code nobody wrote.", target)
}
defer func() {
if werr := os.WriteFile(target, orig, 0o644); werr != nil {
t.Fatalf("restore discover.go: %v", werr)
@ -234,6 +256,23 @@ func runSuite(wd, filter string) string {
return string(out)
}
// mutationTargetIsDirty reports whether a file under internal/gateway has
// uncommitted changes, so the harness can refuse to run instead of treating a
// leftover mutant as the baseline.
//
// git runs from the package directory with a PATH RELATIVE to it. An earlier
// version of the sibling script passed a repo-relative pathspec while running in
// the package directory; git matched nothing and the guard never fired.
func mutationTargetIsDirty(wd, name string) (bool, error) {
cmd := exec.Command("git", "diff", "--name-only", "--", "./"+name)
cmd.Dir = wd
out, err := cmd.Output()
if err != nil {
return false, err
}
return strings.TrimSpace(string(out)) != "", nil
}
func countTests(out string) int {
n := 0
for _, line := range strings.Split(out, "\n") {