jianf/HologramAgent-DSH-Memory-Port
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

test(memory): pin the C4 gates that were left unguarded (C4 follow-up, supervision)

Browse Source 
Three of C4's control surfaces had no criterion at all: deleting them left every
suite green. This commit adds the criteria, and the two defects they exposed.

## What was unguarded, measured

Each mutation below was applied to a scratch copy of lib/ and run against the
suite as it stood at 05e93b2.

1. The `isAnomalous` gate in `scoreVectorCandidates`
   `centroid?.isAnomalous === true &&` -> `centroid !== undefined &&`
   Result: c4 6/6 green, phase-b-h 137/137 green, phase-b-a 38/38 green.
   Why: every C4 fixture is anisotropic (two of them assert isAnomalous ===
   true), and C4c is "no centroid row", which is a different branch from "a
   centroid that was judged isotropic". The only isotropic assertion in the repo
   (phase-b-h:1133) checks `rebuildCentroid`'s verdict, not the scoring path.

2. `countVectored` on the recall hot path
   `store.countVectored(fingerprint)` inserted before the centroid read.
   Result: c4 6/6 green, phase-b-h + phase-b-a 175/175 green.
   Why: nothing scans lib/recall.js for the C3-k boundary. phase-b-a:67 does read
   recall.js, but it asserts the F2 fingerprint gate.

3. MIN_SAMPLE (this one is a defect, not just a missing criterion)
   A one-vector mean IS that vector, so its norm is exactly 1.0 and it clears
   0.6 by construction. Measured with a real clock and the real commit path on
   the C4 corpus: right after the 8th commit the centroid reported
   `vectorCount=1 meanLength=1.0000`, and a recall that returns 3 rows without a
   centroid returned 1 with it — the block that formed the "mean" was centred to
   the zero vector and dropped. The drift rebuild corrects it ~2s later, so the
   window is short but the state is wrong, and in a slowly growing space it lasts
   until the throttle expires.

## What this commit adds

- `BlockStore.MIN_SAMPLE` (= 2) and `vectors.length >= MIN_SAMPLE` in the
  anisotropy verdict. Two is the smallest number for which the mean is a property
  of the set rather than of one member.
- **C4h** pins both halves of that: the verdict (`isAnomalous === false` for one
  vector, with `meanLength === 1` asserted as the precondition that makes the
  0.6 test insufficient) and the consequence (the only block stays recallable).
- **C4g** pins the isotropic gate. Fixture numbers come from running the same
  probe against a gated and an ungated build, not from paper: 8 orthogonal
  one-hot vectors, `meanLength = 0.353553`, `isAnomalous === false`; a query
  along `unit(ones)` yields 8 hits of cosine 0.3536 with the gate, and 0 hits
  without it (every centred cosine turns negative and V5's structural gate drops
  the lot). A three-axis query additionally pins 0.8111 / 0.4867 / 0.3244.
  The identical-pair query is deliberately NOT the main assertion: with both
  sides centred its cosine is 1 either way, so it cannot discriminate.
- **C3-k** statically scans lib/recall.js for `countVectored(` / `centroidStatus(`
  after stripping comments, and carries a positive self-check: the same scan must
  find `centroidStatus(` in lib/index.js, so a broken regex cannot leave the two
  "must not contain" assertions vacuously true.
- **C4d** gained the assertion it was missing. It asserted that a centroid
  exists, is anomalous and has the right width — all true of a centroid built
  from a single block. It now also asserts `vectorCount === CORPUS.length`, and
  waits for the drift rebuild before asserting on contents. Without the count
  assertion C4d is green while the mean describes one block.

## Mutations verified (each red in a scratch copy, working tree untouched)

  delete isAnomalous gate        -> C4g red
  countVectored in recall.js     -> C3-k red
  drop MIN_SAMPLE                -> C4h red
  revert "refresh a drifted centroid" -> C4d + C4e red

## Documentation that had drifted from the code

- recall.js still said "No centroid centering here. C4 ... is a separate slice
  layered on top of this one" directly above the function that now implements
  centering. Replaced with a pointer to where it actually lives.
- The one-sided-centring cosine was recorded as 0.3787 in the comment and "near
  0.38" in C4b's message. Measured on C4b's own five-block fixture it is 0.1897
  (cosine is symmetric, so centring either single side gives the same value).
  Both now say 0.1897 and name the fixture.

## Measured costs (for the write path this slice touches)

Pre-populated space, 512 dims, 2000 vectored blocks, empty graph_centroid — the
state a production database is in when it upgrades:

    FIRST commit after upgrade   105.1 ms   (synchronous O(n*d) rebuild)
    STEADY commits               8.7-9.4 ms (loadCentroid SELECT + centroidStatus COUNT)

The first build is accepted as-is: it happens once per space lifetime, it is
bounded, and moving it to the async path would leave the first recalls after an
upgrade silently uncorrected — a bounded stall is easier to notice than a silent
degradation. The steady-state COUNT is new with C4; C3-k keeps it off the recall
path, where it would run every turn.

## Real path (dsh headless, overlay on the real profile)

Same corpus and query, only the centroid row differs:

    centroid present (count=8, meanLength=0.972934)
      [1] (text 0.63) 第七号仓库的备用门禁码是 QP47-812。
    centroid deleted
      [1] (text 0.69) 这段时间一切照旧，没有什么特别的事情。
      [2] (text 0.69) 第七号仓库的备用门禁码是 QP47-812。

top-1 moves from the hub to the target, which is the behaviour C4 exists for.

## What this does NOT establish

Centring changes angles; it does not calibrate absolute scores. DEFAULT_MIN_SCORE
= 0.5 was never derived from a centred distribution, and the uncalibrated-absolute-
score problem recorded in phase-b-p1-vector-wiring.test.js:51-58 remains open.
A green C4 means "centring is wired into scoring and changes the ranking in an
anisotropic space", never "semantic recall is good now".

C5 (distinguishing no_centroid / dim_mismatch / stale / isotropic reasons) is
untouched: `centroidRecallState` still returns `stale: null`.

Gates: npm test 281/281 (0 skipped), npm run check 12/12, npm pack 10 files.
Frozen faces untouched: docs/, README.md, scripts/, lib/loop*.
This commit is contained in:
jianf
2026-11-05 11:21:02 +08:00
parent 05e93b2f5c
commit febf9f6f3c
3 changed files with 229 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
lib/recall.js
View File

@ -780,9 +780,9 @@ export async function recallBlocksHybrid(store, embedder, query, options = {}) {
* the whole recall down (decision §0.4), and the migration path already records
* corrupt widths as a negative `vector_dim` so they are filterable.
*
* ★ No centroid centering here. C4 (subtract the space mean from BOTH sides) is a
* separate slice layered on top of this one; doing it inline now would mix two
* concerns in one diff and make C4's mutation unciteable.
* ★ Centering lives at the BOTTOM of this function (see the C4 block by the
* scoring loop): the query is centred once, and each block vector is centred by
* the same mean inside the loop. Nothing above this line knows about it.
*
* ★ Not L2-normalised on the way out. Cosine already divides by both norms, and
* `putBlock` does not promise unit vectors, so normalising would change nothing
@ -844,7 +844,8 @@ function scoreVectorCandidates(store, queryVector, { fingerprint, dimension } =
// ★ 两边都中心化。只处理一边 = 一边带公共分量、另一边不带，算出来的
// 余弦是两种量纲的混合（HA centroid_store.go:439-442 的原话）。
// 实测：只减 query 时同值对的余弦从 1.0 掉到 0.3787 —— 那就不是夹角了。
// 实测：只减 query 时，C4b 那个 5 块 fixture 上同值对的余弦从 1.0 掉到
// 0.1897（余弦对称，只减块侧同值）—— 那就不是在量夹角了。
const scoringVector = centering ? centerVector(vector, mean) : vector
if (scoringVector === null) continue

21
lib/store.js
View File

@ -541,6 +541,21 @@ export class BlockStore {
return 0.6
}
/** Minimum number of contribution vectors before a "mean direction" exists.
*
* A one-vector mean IS that vector, so its norm is exactly 1.0 and it clears
* {@link BlockStore.MEAN_THRESHOLD} by construction — a young space would be
* declared anisotropic on the strength of a single block. Subtracting that
* "mean" from its own block yields the zero vector, which the scorer drops, so
* centering would silently remove the only block the space contains. Measured
* on the C4 fixture: right after the first commit the centroid reported
* `vectorCount=1 meanLength=1.0000` and a recall that returns 3 rows without a
* centroid returned 1 with it. Two is the smallest number for which the mean
* is a property of the SET rather than of one member. */
static get MIN_SAMPLE() {
return 2
}
/** Relative change in the vectored-block count that marks a centroid stale.
* Ported from `centroidRebuildRatio` (internal/memory/centroid_store.go:55).
* The Go comment pins the rationale: below 10% the shift does not move the
@ -784,7 +799,11 @@ export class BlockStore {
let meanLength = 0
for (const value of mean) meanLength += value * value
meanLength = Math.sqrt(meanLength)
const isAnomalous = meanLength >= BlockStore.MEAN_THRESHOLD
// ★ Both conditions matter. `meanLength >= threshold` alone would call a
// single-block space anisotropic (norm of a unit vector is 1.0), and the
// resulting centering subtracts a block from itself. See MIN_SAMPLE.
const isAnomalous = vectors.length >= BlockStore.MIN_SAMPLE
&& meanLength >= BlockStore.MEAN_THRESHOLD
this.saveCentroid({
fingerprint,

205
test/phase-b-c4-centering.test.js
View File

@ -176,7 +176,8 @@ test('C4b: a block identical to the query always scores a cosine of 1', async (t
assert.ok(
Math.abs(twinHit.vectorHit - 1) < 1e-9,
'C4b: an identical vector pair must score exactly 1. ' +
'A value near 0.38 is the signature of centering ONE side only ' +
'A value near 0.1897 on this 5-block fixture is the signature of centering ONE '
+ 'side only ' +
'(M-C4) — subtracting the mean from the query but not the block mixes ' +
'two coordinate systems and the cosine stops being an angle. ' +
`got ${twinHit.vectorHit}`,
@ -253,6 +254,16 @@ test('C4d: the production commit path builds the centroid it later reads', async
assert.equal(written.embedded, true, `precondition: ${block.key} must actually be embedded`)
}
// ★ The first commit builds a centroid from the blocks it has seen SO FAR,
// so right after a burst of writes the centroid still describes only the
// first one — and with MIN_SAMPLE it is not even allowed to claim a direction
// yet. The write path schedules a drift rebuild; wait for it before asserting
// on the centroid's CONTENTS rather than merely on its existence.
//
// Measured without this wait: vectorCount=1, meanLength=1.0000 on the same
// fixture, and a 1-block "mean" is that block itself.
await new Promise((resolve) => setTimeout(resolve, 3000))
const centroid = mem.store.loadCentroid(SPACE)
assert.ok(
centroid !== undefined,
@ -263,6 +274,18 @@ test('C4d: the production commit path builds the centroid it later reads', async
)
assert.equal(centroid.isAnomalous, true, 'C4d: and it must be judged anisotropic for this fixture')
assert.equal(centroid.dim, DIM, 'C4d: its width must match the space')
// ★ The existence check above passes even when the centroid describes a
// single block — which is exactly the state right after the first commit.
// A 1-block mean is that block itself (norm 1.0, so it clears the 0.6
// threshold by construction), and centering by it drops that block as a zero
// vector. Pin the coverage, not just the presence.
assert.equal(
centroid.vectorCount,
CORPUS.length,
'C4d: the centroid must describe every block in the space, not only the ones '
+ 'present at the first commit. A smaller count means the drift rebuild '
+ 'never caught up and centering is subtracting the wrong mean.',
)
// 端到端：同一个库，召回必须已经用上校正
const hits = await recallBlocksHybrid(mem.store, makeEmbedder(), QUERY_TEXT, { topK: 8 })
@ -357,3 +380,183 @@ test('C4e: a drifted centroid is refreshed by the write path, not left stale for
rmSync(dir, { recursive: true, force: true })
}
})
// ── C4g: 各向同性空间必须完全不校正（isAnomalous 门控）────────────────────
//
// ★ 这条守的是「门控本身」，不是「校正算得对不对」。
// C4a/b/c/d/e 五条用的全是有各向异性的 fixture（C4-0 与 C4d 还专门断言
// isAnomalous === true），C4c 是「没有中心行」—— 三条都照不到「有中心、但
// 中心判定为各向同性」这个分支。实测把 lib/recall.js 里的
// `centroid?.isAnomalous === true &&` 删掉，上述五条加 phase-b-h、phase-b-a
// 全部全绿：门控是裸的。
//
// 数字来源：同一份探针分别打「门控在」与「门控被拆」两个 lib，实测对照，
// 不是纸上推的。各向同性空间（8 条正交 one-hot）：
// meanLength = 0.353553（= 1/sqrt(8)），isAnomalous = false
// query = unit(ones) 门控在 → 8 条命中，每条 cosine = 0.3536
// 门控拆 → 0 条命中（去均值后各块余弦转负，被 V5 的
// 结构性门 `vectorScore <= 0 && symbolHit === 0` 全丢）
// 为什么用这条 query：它的方向就是均值方向，去均值后与所有块的夹角都变成钝角，
// 落差是二值化的（8 → 0），不依赖阈值也不依赖浮点容差。
test('C4g: an isotropic space is left uncorrected — the isAnomalous gate', async (t) => {
const { Context } = await import('@deepseek-ai/cordis')
const { SystemPrompt } = await import('@deepseek-ai/dsh-system-prompt')
const { ToolRuntime } = await import('@deepseek-ai/dsh-tools')
const { apply: applyMemory } = await import('../lib/index.js')
const { mkdtempSync, rmSync } = await import('node:fs')
const { tmpdir } = await import('node:os')
const { join } = await import('node:path')
const ISO_DIM = 8
const ISO_SPACE = 'c4g:isotropic:test'
const ones = new Array(ISO_DIM).fill(1)
const queryA = unit(ones) // ≈ 均值方向
const queryB = unit([0.5, 0.3, 0.2, 0, 0, 0, 0, 0]) // 三个方向轴的混合
const texts = Array.from({ length: ISO_DIM }, (_, i) => `正交块 ${i}`)
const vecOf = new Map(texts.map((text, i) => [text, axis(i)]))
vecOf.set('Q-A', queryA)
vecOf.set('Q-B', queryB)
const isoEmbedder = {
async embedText(text) { return vecOf.get(text) ?? new Array(ISO_DIM).fill(0) },
info: () => ({ fingerprint: ISO_SPACE, dimension: ISO_DIM }),
}
const dir = mkdtempSync(join(tmpdir(), 'c4g-'))
const ctx = new Context()
new SystemPrompt(ctx, {})
new ToolRuntime(ctx, {})
try {
const mem = applyMemory(ctx, { path: join(dir, 'm.db'), embedder: isoEmbedder })
const keyOf = new Map()
for (const text of texts) {
const written = await mem.commit({ text })
keyOf.set(written.id, text)
}
// 让写入侧的漂移重建跑完，断言的是「生产路径建出的中心」而不是半成品
await new Promise((resolve) => setTimeout(resolve, 3000))
const centroid = mem.store.loadCentroid(mem.space().fingerprint)
assert.ok(centroid !== undefined, 'C4g precondition: the production path must still build a centroid')
assert.equal(centroid.isAnomalous, false, 'C4g precondition: 8 orthogonal axes are isotropic')
assert.ok(
Math.abs((centroid.stats?.meanLength ?? 0) - 1 / Math.sqrt(ISO_DIM)) < 1e-6,
`C4g precondition: meanLength must be 1/sqrt(8); got ${centroid.stats?.meanLength}`,
)
assert.equal(centroid.vectorCount, ISO_DIM, 'C4g precondition: every block contributed')
// ① 方向接近均值的 query：门控在 ⇒ 全部命中，且余弦就是 1/sqrt(8)
const hitsA = await recallBlocksHybrid(mem.store, isoEmbedder, 'Q-A', { topK: ISO_DIM, minScore: 0 })
t.diagnostic(`C4g mean-like query: n=${hitsA.length} ${hitsA.map((h) => `${Number(h.vectorHit ?? 0).toFixed(4)}`).join(' ')}`)
assert.equal(
hitsA.length,
ISO_DIM,
'C4g: an isotropic space must NOT be centred. Removing the isAnomalous gate turns '
+ 'every cosine negative here and V5 drops the lot — the count collapses from 8 to 0.',
)
for (const hit of hitsA) {
assert.ok(
Math.abs(Number(hit.vectorHit) - 1 / Math.sqrt(ISO_DIM)) < 1e-4,
`C4g: each block keeps its raw cosine 1/sqrt(8); got ${hit.vectorHit}`,
)
}
// ② 三轴混合的 query：钉死「完全没做去均值」的具体数值
const hitsB = await recallBlocksHybrid(mem.store, isoEmbedder, 'Q-B', { topK: 3, minScore: 0 })
const byKey = new Map(hitsB.map((h) => [keyOf.get(h.id), Number(h.vectorHit)]))
t.diagnostic(`C4g three-axis query: ${[...byKey].map(([k, v]) => `${k}=${v.toFixed(4)}`).join(' ')}`)
assert.ok(Math.abs((byKey.get('正交块 0') ?? 0) - 0.8111) < 1e-4, 'C4g: block 0 keeps cosine 0.8111')
assert.ok(Math.abs((byKey.get('正交块 1') ?? 0) - 0.4867) < 1e-4, 'C4g: block 1 keeps cosine 0.4867')
assert.ok(Math.abs((byKey.get('正交块 2') ?? 0) - 0.3244) < 1e-4, 'C4g: block 2 keeps cosine 0.3244')
} finally {
await ctx.fiber.dispose()
rmSync(dir, { recursive: true, force: true })
}
})
// ── C3-k: 召回热路径不得出现计数查询 ────────────────────────────────────────
//
// `centroidStatus` 的第一行是 `countVectored`（一条 COUNT），而它每轮 pre-step
// 都会跑 —— C3-k 明令该查询只能出现在写入侧。实测往 lib/recall.js 里塞一行
// `store.countVectored(fingerprint)`，c4 6/6、phase-b-h、phase-b-a 全绿：
// 没有判据在守这条边界。所以这里用静态扫描把它钉住。
//
// ★ 扫描必须自证有效：同一段逻辑对 lib/index.js 跑一遍，必须能找到
// `centroidStatus(` —— 否则正则写错时，上面两条「不包含」会永远为真。
test('C3-k: the recall hot path must not run a COUNT query', async () => {
const { readFileSync } = await import('node:fs')
const strip = (relative) => readFileSync(new URL(relative, import.meta.url), 'utf8')
.replace(/\/\*[\s\S]*?\*\//g, ' ')
.split('\n')
.map((line) => line.replace(/\/\/.*$/, ''))
.join('\n')
const recall = strip('../lib/recall.js')
assert.ok(
!/\bcountVectored\s*\(/.test(recall),
'C3-k: lib/recall.js must not call countVectored — that COUNT belongs to the write path',
)
assert.ok(
!/\bcentroidStatus\s*\(/.test(recall),
'C3-k: lib/recall.js must not call centroidStatus — it runs countVectored internally, '
+ 'and the recall path may only read the centroid (loadCentroid)',
)
// 反向自检：同样的扫描必须能在写入侧看见那次调用，证明扫描不是空转
const index = strip('../lib/index.js')
assert.ok(
/\bcentroidStatus\s*\(/.test(index),
'C3-k self-check: the scan must be able to SEE a centroidStatus call — lib/index.js '
+ 'calls it on the write path. If this fails the two assertions above prove nothing.',
)
assert.ok(
/\bloadCentroid\s*\(/.test(recall),
'C3-k: the recall path reads the centroid through loadCentroid — and only that',
)
})
// ── C4h: 单块空间不得被判为各向异性（MIN_SAMPLE）──────────────────────────
//
// 一条向量的「均值」就是它自己，模长恒为 1.0，必然越过 0.6 阈值 —— 于是
// 「只有一条块的空间」会被判成各向异性，然后拿这条块自己去均值。减完是零
// 向量，被 recall.js 的 `scoringVector === null` 跳过，唯一那条块就此从
// 召回里消失。实测（真时钟、真 commit）：第 8 次 commit 之前 centroidCount=1、
// meanLength=1.0000，同一库有中心时命中 1 条、无中心时命中 3 条。
//
// 判据落在两处：判定本身（isAnomalous 必须 false），以及后果（唯一那条块
// 必须仍然可召回）。
test('C4h: a single-block space is not anisotropic, and its only block stays recallable', async () => {
const store = new BlockStore({ path: ':memory:' }).open()
try {
const fp = 'c4h:single:test'
const only = axis(0)
const written = store.putBlock({ text: '唯一的一条块', vector: only, fingerprint: fp })
const built = store.rebuildCentroid(fp)
assert.equal(built.vectorCount, 1, 'C4h precondition: exactly one vector contributed')
assert.ok(Math.abs(built.meanLength - 1) < 1e-9, 'C4h precondition: a one-vector mean has norm 1')
assert.equal(
built.isAnomalous,
false,
'C4h: a one-vector mean IS that vector (norm 1.0), so the 0.6 test alone always says '
+ '"anisotropic". MIN_SAMPLE must refuse to call a single sample a direction — otherwise '
+ 'centering subtracts the block from itself and the zero vector is dropped from recall.',
)
const embedder = {
async embedText() { return only },
info: () => ({ fingerprint: fp, dimension: DIM }),
}
const hits = await recallBlocksHybrid(store, embedder, '与那条块方向相同的问题', { topK: 3, minScore: 0 })
assert.equal(
hits.length,
1,
'C4h: the only block in the space must still be recallable. A count of 0 means the '
+ 'single-block "centroid" was applied and its own block was projected onto the zero '
+ 'vector.',
)
assert.equal(hits[0].id, written.id, 'C4h: and it must be the block that was written')
} finally {
store.close()
}
})