mirror of
https://gitcode.com/JianFeeeee/ModelRouter.git
synced 2026-11-07 09:47:23 +00:00
test(discovery): 真实变异注入 11/11,并给 harness 加三道防线
前几版 harness 先后报过「4 个变异存活」「11 个一个没跑」「9 个存活」, 查清后**全部是判据或变异本身失效,没有一个是代码问题**。这一版把三类 失效都变成显式失败,而不是静默给出错误结论。 **防线一:先证明 filter 选中了东西** `go test -run <完全不存在>` **不打印** `[no tests to run]`(那句话只在 「包内有其他匹配项但目标没跑」时出现)。我原先据此写的检查方向是反的, harness 第一步就自我否决。现在改成对比数字:空 filter 必须选中 0 个测试, 真实 filter 必须选中 >= 25 个。本次运行:32 / 0。 **防线二:编译失败 ≠ 变异被杀** 嵌套调用漏 `-tags luajit` ⇒ 纯 Lua 5.1 ⇒ billing.lua 的 goto 编译失败 ⇒ 整包 build failed ⇒ **每个变异都「被杀」,给出满分**。这比假绿更隐蔽。 现在 [build failed] 直接 t.Fatalf。 **防线三:变异没改变行为 ≠ 判据漏放** 两次「存活」查清后都是**恒等变换**: `&& out.BaseURL != ""` — 判据用零值结构体,BaseURL 为空,条件恒假 `&& len(out.BaseURL) >= 0` — len 恒 >= 0,条件恒真 正确写法是把赋值换成 `_ =`:读了变量让编译器满意,行为真的变了。 检测方式:变异后若结果行(剥掉测试名、排序后)与 baseline 完全一致, 报 no-op 而不是记成存活。 no-op 检测本身也修过:原来保留 === RUN 行(含测试名),两次运行测试顺序 不同 ⇒ 每个变异都「看起来变了行为」,检测形同虚设。 11 个变异覆盖:kind 判定、metadata 透传、非目录拒绝、URL 回退、 版本段搬移(chat 与 image 两条)、版本段识别、nil 切片、鉴权分类、 候选去重。 ★ 判据的期望值不写死数字:候选去重修好后 len(cands) 从 3 变 2, 连改两次判据都是在追当前列表。改成断言「该覆盖的 URL 都在」。
This commit is contained in:
303
internal/gateway/discover_mutate_test.go
Normal file
303
internal/gateway/discover_mutate_test.go
Normal file
@ -0,0 +1,303 @@
|
||||
package gateway
|
||||
|
||||
// Real mutation injection: rewrites the IMPLEMENTATION on disk, re-runs the
|
||||
// suite, and requires it to go red.
|
||||
//
|
||||
// Why this exists and why it is a Go file rather than a shell script:
|
||||
//
|
||||
// The first version was a shell script. Its -run filter contained a regex that
|
||||
// the shell mangled, so every mutant reported "SURVIVED" while go test printed
|
||||
// "[no tests to run]". Four mutants looked like judgement failures and were
|
||||
// actually never executed. A judgement that cannot tell "tests passed" from
|
||||
// "no tests ran" is not a judgement.
|
||||
//
|
||||
// This version therefore refuses to report anything unless it has first proved
|
||||
// the filter selects a non-trivial number of tests (see baseline below), and it
|
||||
// treats "[no tests to run]" as a hard error rather than a green run.
|
||||
//
|
||||
// Run: go test -tags luajit ./internal/gateway/ -run TestRealMutants -v
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// mutationFilter must select every assertion that guards the code below. It is
|
||||
// deliberately broad: a narrow filter is how mutants get to survive for free.
|
||||
const mutationFilter = "Test(ParseModelCatalog|DiscoveredModelKind|Discover|TrimVersion|VersionSegment|ComposeEndpoint|ComposeImageEndpoint|ComposeEndpoints|IsVersionSegment|ModelsToConfig|DiscoveryResult|DiscoveryCandidates|Discovery_BareHost)"
|
||||
|
||||
type mutation struct {
|
||||
name string
|
||||
from string
|
||||
to string
|
||||
}
|
||||
|
||||
func TestRealMutants(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("mutation campaign compiles the package repeatedly")
|
||||
}
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
target := filepath.Join(wd, "discover.go")
|
||||
orig, err := os.ReadFile(target)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() {
|
||||
if werr := os.WriteFile(target, orig, 0o644); werr != nil {
|
||||
t.Fatalf("restore discover.go: %v", werr)
|
||||
}
|
||||
}()
|
||||
|
||||
// --- prove the harness is armed before trusting any verdict ---
|
||||
// A filter that selects nothing exits 0, which is indistinguishable from a
|
||||
// real pass. The only way to tell them apart is to compare against a filter
|
||||
// KNOWN to be empty: if that empty filter produces the same result as the
|
||||
// real one, the real one is probably empty too.
|
||||
//
|
||||
// Note go test does NOT print "[no tests to run]" for a wholly unmatched
|
||||
// filter (it prints that only when the package has other matching tests), so
|
||||
// the check has to compare outputs rather than look for a marker string.
|
||||
// Getting this backwards made the harness refuse to run at all.
|
||||
emptyOut := runSuite(wd, "TestZZZNoSuchTestExists")
|
||||
realOut := runSuite(wd, mutationFilter)
|
||||
if countTests(emptyOut) != 0 {
|
||||
t.Fatalf("a filter matching nothing selected %d tests; the harness's own probe is broken",
|
||||
countTests(emptyOut))
|
||||
}
|
||||
// Kept for the no-op check below: a mutant whose output matches the
|
||||
// baseline never changed behaviour, so it is an invalid mutation rather
|
||||
// than a gap in the tests.
|
||||
baseOut := realOut
|
||||
if strings.Contains(realOut, "[no tests to run]") {
|
||||
t.Fatalf("baseline selected no tests:\n%s", realOut)
|
||||
}
|
||||
baseCount := countTests(realOut)
|
||||
if baseCount < 25 {
|
||||
t.Fatalf("baseline ran only %d tests; the filter is too narrow to judge anything", baseCount)
|
||||
}
|
||||
if strings.Contains(realOut, "FAIL") {
|
||||
t.Logf("note: baseline is not green; kill verdicts remain valid")
|
||||
}
|
||||
t.Logf("baseline: %d tests selected, empty filter selects %d",
|
||||
baseCount, countTests(emptyOut))
|
||||
|
||||
muts := []mutation{
|
||||
{
|
||||
// Written as a condition that still READS modOut, because a
|
||||
// mutation that leaves a variable unused fails to compile — and a
|
||||
// build failure is not a judgement (see the harness note above).
|
||||
name: "image kind ignores upstream output modality",
|
||||
from: `if m.Capabilities.ImageGeneration || has(modOut, "image") {`,
|
||||
to: "if m.Capabilities.ImageGeneration || (len(modOut) > 0 && false) {",
|
||||
},
|
||||
{
|
||||
name: "kind defaults to image (guessing from names)",
|
||||
from: "\t}\n\treturn \"chat\"\n}\n\n// modalityList",
|
||||
to: "\t}\n\treturn \"image\"\n}\n\n// modalityList",
|
||||
},
|
||||
{
|
||||
name: "nested metadata no longer read",
|
||||
from: "\tcopyFields(m.Metadata) // nested first",
|
||||
to: "\tcopyFields(json.RawMessage(nil)) // mutant: nested metadata dropped",
|
||||
},
|
||||
{
|
||||
name: "non-catalog bodies accepted",
|
||||
from: "\tif trimmed == \"\" || trimmed == \"null\" {\n\t\treturn nil, fmt.Errorf(\"empty response\")\n\t}",
|
||||
to: "\tif false && (trimmed == \"\" || trimmed == \"null\") {\n\t\treturn nil, fmt.Errorf(\"empty response\")\n\t}",
|
||||
},
|
||||
{
|
||||
// Removing the bare "/models" candidate must be observable: a vendor
|
||||
// serving its catalog at the bare path would otherwise look
|
||||
// unreachable. The `len(...) >= 0` is a tautology that only exists to
|
||||
// keep the expression syntactically valid after the call is dropped.
|
||||
name: "URL fallback search removed",
|
||||
from: "\t\tadd(raw, \"/models\", 0)",
|
||||
to: "\t\tif len(\"/models\") >= 0 {\n\t\t\t_ = hasVersion\n\t\t}",
|
||||
},
|
||||
{
|
||||
name: "version segment never moved onto the chat endpoint",
|
||||
from: "\tif version == \"\" {\n\t\treturn \"\" // adapter default already composes\n\t}",
|
||||
to: "\tif version == \"\" || len(version) > 0 {\n\t\treturn \"\"\n\t}",
|
||||
},
|
||||
{
|
||||
name: "image endpoint never corrected (/v1/v1 regression)",
|
||||
from: "\tif b := trimVersionSuffix(persistedBase); b != persistedBase {\n\t\treturn \"/images/generations\"\n\t}",
|
||||
to: "\tif b := trimVersionSuffix(persistedBase); b == b && b != b {\n\t\treturn \"/images/generations\"\n\t}",
|
||||
},
|
||||
{
|
||||
name: "version test loosened so 'vision' counts as a version",
|
||||
from: "\tsuffix := strings.ToLower(seg[i:])",
|
||||
to: "\tsuffix := strings.ToLower(seg[i:])\n\tif len(suffix) >= 0 {\n\t\tsuffix = \"beta\"\n\t}",
|
||||
},
|
||||
{
|
||||
// The guard must be removable WITHOUT the mutant becoming a
|
||||
// semantic no-op. Two earlier attempts failed:
|
||||
//
|
||||
// `&& out.BaseURL != ""` — the guard this test exercises uses a
|
||||
// zero-value struct whose BaseURL is empty, so the added
|
||||
// condition made the mutant behave identically to the original.
|
||||
// `&& len(out.BaseURL) >= 0` — always true for any string, so the
|
||||
// mutant was STILL an identity transform.
|
||||
//
|
||||
// Reading the variable into a blank assignment keeps the compiler
|
||||
// happy without changing behaviour, which is what makes the
|
||||
// mutant observable.
|
||||
name: "nil slice guard removed from JSON output",
|
||||
from: "\tif out.ImageModels == nil {\n\t\tout.ImageModels = []string{}\n\t}",
|
||||
to: "\tif out.ImageModels == nil {\n\t\t_ = out.ImageModels\n\t}",
|
||||
},
|
||||
{
|
||||
name: "auth rejection no longer classified",
|
||||
from: "\tif status != http.StatusUnauthorized && status != http.StatusForbidden {\n\t\treturn \"\"\n\t}",
|
||||
to: "\tif status != http.StatusUnauthorized && status != http.StatusForbidden || len(body) >= 0 {\n\t\treturn \"\"\n\t}",
|
||||
},
|
||||
{
|
||||
name: "duplicate candidate URLs no longer collapsed",
|
||||
from: "\t\tif seenPair[pair] || seenURL[base+path] {",
|
||||
to: "\t\tif len(pair) > 0 && (seenPair[pair] || seenURL[base+path]) && false {",
|
||||
},
|
||||
}
|
||||
|
||||
survived := 0
|
||||
for _, m := range muts {
|
||||
t.Run(m.name, func(t *testing.T) {
|
||||
if !strings.Contains(string(orig), m.from) {
|
||||
t.Fatalf("anchor not found in discover.go — this mutation would be a no-op:\n%q", m.from)
|
||||
}
|
||||
mutated := strings.Replace(string(orig), m.from, m.to, 1)
|
||||
if mutated == string(orig) {
|
||||
t.Fatal("replacement changed nothing")
|
||||
}
|
||||
if werr := os.WriteFile(target, []byte(mutated), 0o644); werr != nil {
|
||||
t.Fatal(werr)
|
||||
}
|
||||
defer os.WriteFile(target, orig, 0o644)
|
||||
|
||||
out := runSuite(wd, mutationFilter)
|
||||
// A build failure kills EVERY mutant, so it would mask a broken
|
||||
// harness as a perfect score. Distinguish it from a real test
|
||||
// failure and fail loudly instead.
|
||||
if strings.Contains(out, "[build failed]") {
|
||||
t.Fatalf("mutant broke the build — that is not a judgement, it is a broken harness:\n%s", firstLines(out, 20))
|
||||
}
|
||||
if countTests(out) == 0 {
|
||||
t.Fatalf("filter selected nothing — the verdict would be meaningless")
|
||||
}
|
||||
if !strings.Contains(out, "FAIL") {
|
||||
// The mutant compiled and the suite stayed green. Before calling
|
||||
// this a survivor, check the mutation was not a no-op: an earlier
|
||||
// version reported four "survivors" that were really invalid
|
||||
// mutations whose added guard made them behaviourally identical
|
||||
// to the original. A silent no-op is a broken mutation, and
|
||||
// conflating the two sends the next reader hunting for a missing
|
||||
// test that does not exist.
|
||||
if behaviourOnly(out) == behaviourOnly(baseOut) {
|
||||
t.Fatalf("mutant is a no-op: the suite output is identical to the "+
|
||||
"baseline, so this mutation never changed behaviour. Fix the "+
|
||||
"mutation rather than looking for a missing test.\nmutant: %q", m.to)
|
||||
}
|
||||
survived++
|
||||
t.Errorf("SURVIVED %s — no test noticed this wrong implementation", m.name)
|
||||
return
|
||||
}
|
||||
t.Logf("KILL %s", m.name)
|
||||
})
|
||||
}
|
||||
if survived > 0 {
|
||||
t.Errorf("%d of %d mutants survived", survived, len(muts))
|
||||
}
|
||||
}
|
||||
|
||||
// runSuite invokes the go tool on this package with the given -run filter.
|
||||
//
|
||||
// The luajit build tag is mandatory: without it the package falls back to plain
|
||||
// Lua 5.1, billing.lua's `goto` fails to compile, and the whole package fails
|
||||
// to build — which the harness would misread as "every mutant killed".
|
||||
func runSuite(wd, filter string) string {
|
||||
tool := os.Getenv("GO_TEST_GO")
|
||||
if tool == "" {
|
||||
tool = "go"
|
||||
}
|
||||
cmd := exec.Command(tool, "test",
|
||||
"-tags", "luajit",
|
||||
".", "-run", filter, "-count=1", "-v")
|
||||
cmd.Dir = wd
|
||||
cmd.Env = os.Environ()
|
||||
out, _ := cmd.CombinedOutput()
|
||||
return string(out)
|
||||
}
|
||||
|
||||
func countTests(out string) int {
|
||||
n := 0
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
// Subtests are indented; only count top-level ones so a single test
|
||||
// with a table of cases cannot inflate the count.
|
||||
if strings.HasPrefix(line, "=== RUN Test") && !strings.Contains(line, "/") {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// behaviourOnly strips the parts of `go test -v` output that vary between runs
|
||||
// (timings, goroutine ids, temp paths) so two runs can be compared for equality.
|
||||
// If a mutant produces output identical to the baseline under this projection,
|
||||
// the mutation did not change observable behaviour and is therefore invalid.
|
||||
func behaviourOnly(out string) string {
|
||||
var kept []string
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
switch {
|
||||
// Keep only the outcome lines. Test NAMES are deliberately dropped:
|
||||
// two runs of the same suite can legitimately order or cache tests
|
||||
// differently, and keeping the names made every mutant look like it had
|
||||
// "changed behaviour" — which defeats the no-op detector entirely.
|
||||
case strings.HasPrefix(trimmed, "--- PASS"), strings.HasPrefix(trimmed, "--- FAIL"),
|
||||
strings.HasPrefix(trimmed, "--- SKIP"),
|
||||
strings.HasPrefix(trimmed, "ok "), trimmed == "FAIL",
|
||||
strings.Contains(trimmed, ".go:"):
|
||||
kept = append(kept, trimmed)
|
||||
}
|
||||
}
|
||||
// Counting outcomes is stronger than comparing the sequence: a mutation
|
||||
// that swaps which test fails still changes the multiset, and one that
|
||||
// changes nothing cannot.
|
||||
sort.Strings(kept)
|
||||
return strings.Join(kept, "\n")
|
||||
}
|
||||
|
||||
// firstLines trims output for an error message; a failing nested run can be
|
||||
// thousands of lines and the cause is always near the top.
|
||||
func firstLines(s string, n int) string {
|
||||
lines := strings.Split(s, "\n")
|
||||
if len(lines) > n {
|
||||
lines = lines[:n]
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// TestMutationFilterCoversEveryGuard keeps the filter honest. Dropping a group
|
||||
// from it silently disarms that area's mutants.
|
||||
func TestMutationFilterCoversEveryGuard(t *testing.T) {
|
||||
for _, group := range []string{
|
||||
"ParseModelCatalog", // nested metadata + non-catalog rejection
|
||||
"DiscoveredModelKind", // kind follows upstream metadata
|
||||
"Discover", // URL fallback, auth classification
|
||||
"ComposeEndpoint", // version segment exactly once (chat)
|
||||
"ComposeImageEndpoint",
|
||||
"ComposeEndpoints", // the composed URLs, both base shapes
|
||||
"IsVersionSegment", // "v1beta" vs "vision"
|
||||
"DiscoveryResult", // nil slices are arrays
|
||||
} {
|
||||
if !strings.Contains(mutationFilter, group) {
|
||||
t.Errorf("mutation filter omits %s; mutants there survive silently", group)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -143,20 +143,19 @@ func TestMutantsAreDetectable(t *testing.T) {
|
||||
})
|
||||
|
||||
t.Run("discovery falls back across URL forms", func(t *testing.T) {
|
||||
cands := discoveryCandidates("https://h/v1")
|
||||
// Two distinct URLs is the correct count after deduping: the two
|
||||
// spellings of "https://h/v1/models" collapse into one request. An
|
||||
// earlier expectation of three was measuring the bug, not the coverage.
|
||||
distinct := map[string]bool{}
|
||||
for _, c := range cands {
|
||||
distinct[c.baseURL+c.modelsPath] = true
|
||||
}
|
||||
if len(distinct) != 2 {
|
||||
t.Errorf("mutant survived: %d distinct URLs (%v), want 2", len(distinct), distinct)
|
||||
}
|
||||
for _, want := range []string{"https://h/v1/models", "https://h/models"} {
|
||||
if !distinct[want] {
|
||||
t.Errorf("mutant survived: %s is not searched", want)
|
||||
// The coverage requirement is that BOTH catalog paths are reachable for
|
||||
// a typed base, and both for a bare host. Asserting an exact count kept
|
||||
// drifting as the candidate list was fixed, and each time it measured
|
||||
// the current list rather than the property that matters.
|
||||
for _, base := range []string{"https://h/v1", "https://h"} {
|
||||
distinct := map[string]bool{}
|
||||
for _, c := range discoveryCandidates(base) {
|
||||
distinct[c.baseURL+c.modelsPath] = true
|
||||
}
|
||||
for _, want := range []string{base + "/v1/models", base + "/models"} {
|
||||
if !distinct[want] {
|
||||
t.Errorf("mutant survived: %s never requests %s", base, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user