test(memory): bound what P1's green does and does not establish (P1 follow-up)
Two discipline holes left after1309afe/09be4de, both closed here. 1. The boundary was unstated. A one-hot keyword-family fixture proves the PLUMBING — that the computed query vector reaches scoring — and says nothing about retrieval quality. Left implicit, a green P1 invites the reading "semantic recall works". The measured counter-evidence is now in the file: on the real chinese-clip space a true hit scored 0.8345 while an unrelated block scored 0.8598, so absolute scores are not calibrated and no threshold may be read off them. That is a known unresolved problem outside P1's scope. 2. `dsh` missing used to `t.skip`. A skip keeps `npm test` green while the only end-to-end evidence for P1 never runs — the same shape as a criterion that is committed but never speaks. It now fails, and says why. No behavioural change: 272/272 tests, 12/12 checkpoints, 0 skipped on a box where `dsh` is present. Verified the mutation discipline still bites — deleting `'hybrid'` from ORDINARY_DEGRADATIONS turns L10, L11, L10† and L11† red.
This commit is contained in:
@ -47,6 +47,15 @@
|
||||
*
|
||||
* Without P1a/P1b a passing test would only show that *something* answered.
|
||||
*
|
||||
* ★ What this does NOT establish.
|
||||
* A one-hot keyword-family fixture proves PLUMBING, not retrieval quality.
|
||||
* It cannot speak to how a real embedder ranks real Chinese sentences — that
|
||||
* boundary was measured separately and is a KNOWN UNRESOLVED problem outside
|
||||
* P1's scope: on the real chinese-clip space a true hit scored 0.8345 while an
|
||||
* unrelated block scored 0.8598, so absolute scores are not calibrated and no
|
||||
* threshold may be read off them. A green P1 means "the wiring carries the
|
||||
* vector", never "semantic recall works".
|
||||
*
|
||||
* ## The corpus, and why these exact strings
|
||||
*
|
||||
* The fake embedder is a deterministic keyword-family projection into 8 unit
|
||||
@ -62,21 +71,25 @@
|
||||
* extractor (`[A-Z]{2}[0-9]{2}-[0-9]{3}`) can find it and nothing else in the
|
||||
* prompt can collide with it.
|
||||
*
|
||||
* ## Skip conditions
|
||||
* ## Missing-harness conditions
|
||||
*
|
||||
* The harness needs `dsh` and the two `dsh-base` peers resolvable from this
|
||||
* package. When they are not installed there is no headless process to drive, and
|
||||
* the test skips LOUDLY rather than asserting against a half-mounted app — the
|
||||
* same rule `loop-system-e2e.test.js` follows.
|
||||
* The harness needs `dsh` resolvable from PATH. When it is not, there is no
|
||||
* headless process to drive, and the criterion FAILS rather than skipping.
|
||||
*
|
||||
* ★ Why `assert.fail` and not `t.skip`: a skip is green. `npm test` would report
|
||||
* "1 skipped, N passed" and the only end-to-end evidence for P1 would never run —
|
||||
* exactly the "criterion is in the repo but never speaks" shape this whole slice
|
||||
* exists to close. It is also worse than a plain hole, because a green suite reads
|
||||
* as proof. `loop-system-e2e.test.js` still skips in the same situation; that file
|
||||
* was not the subject of P1 and is not changed here.
|
||||
*
|
||||
* @module dsh-memory/test/phase-b-p1-vector-wiring
|
||||
*/
|
||||
|
||||
import assert from 'node:assert/strict'
|
||||
import { execFileSync, spawn } from 'node:child_process'
|
||||
import { createReadStream, existsSync, mkdtempSync, rmSync } from 'node:fs'
|
||||
import { existsSync, mkdtempSync, rmSync } from 'node:fs'
|
||||
import { createServer } from 'node:http'
|
||||
import { createServer as createNetServer } from 'node:net'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
@ -573,8 +586,11 @@ test('P1-0: the corpus is admissible — symbol path blind, vector path reaches
|
||||
|
||||
test('P1: a semantic-only question is answered from block memory via the vector route', async (t) => {
|
||||
if (!dshAvailable()) {
|
||||
t.skip('no `dsh` binary on PATH — the headless harness cannot be driven here')
|
||||
return
|
||||
assert.fail(
|
||||
'the P1 end-to-end criterion cannot be established: no `dsh` binary on PATH. ' +
|
||||
'This is a SKIP-shaped hole, not a pass — a silent skip would keep npm test green ' +
|
||||
'while the only end-to-end evidence for P1 never runs.',
|
||||
)
|
||||
}
|
||||
|
||||
const workspace = mkdtempSync(join(tmpdir(), 'p1-e2e-'))
|
||||
|
||||
Reference in New Issue
Block a user