fix(memory-b): phase A 存量库迁移 + 成功路径 degraded 标签断言
⚠️ 先说明顺序问题:H 组我已在 00a4a5f 提交并推送,在你这条指令之前。
你要求「迁移是 H 的前置」,我同意,但已经发出去了。
本提交补上迁移,00a4a5f..本提交 之间是一个真实的破窗状态
(存量库会被打成只读),已记在这里,不掩饰。
阻塞① 迁移(硬 bug,我复现了)
复现:用 phase-A 形库(无 vector/fingerprint 列,user_version=0)
opened OK ← 加载不报错
PUTBLOCK THREW: table memory_blocks has no column named vector
blockFingerprints THREW: no such column: fingerprint
根因:DDL 全是 CREATE TABLE IF NOT EXISTS,对已存在表静默 no-op。
实现(lib/store.js)
- 新增 SCHEMA_VERSION = 1 与 migrate(db)
- 探测而非猜:读 pragma_table_info('memory_blocks'),缺哪列 ALTER 哪列,
不靠 try/catch 吞异常(半个迁移被吞掉是最难查的形态)
- user_version 为 0 不等于缺列(phase B 崩溃前未盖戳的库会是这样),
所以判据是列存在性,不是版本号
- graph_centroid:phase A 的空表直接 DROP 重建;有数据则**显式拒绝并报错**,
不销毁数据
- 新增 userVersion() 供读取
- open() 在 DDL 之后调用 migrate():新库已具当前形状,只有老库有活干
判据
MIG1 phase-A 库:列补齐 → 老块 getBlock 读得到 → FTS 仍命中
→ putBlock 成功 → user_version==1 → 重开幂等
MIG2 新库直接是 version 1
MIG1† 不迁移时那条真实写入必须失败(并断言 phase-A 表确实无 vector 列)
变异打红原文(把 migrate(db) 注释掉)
not ok 21 - MIG1: a phase-A database gains the vector columns and stays writable
error: 'missing vector column; have id,modality,text_content,source,scene,semantic_type,created_at,updated_at'
not ok 22 - MIG2: a fresh database is created at version 1
恢复 → # pass 23 / # fail 0
★ 我第一版 MIG1† 写错了:用了一条不提到 vector 的 INSERT 去断言会失败,
而那种 INSERT 本来就会成功。已改成 phase B 实际执行的那种写(显式带
vector/fingerprint),并加断言确认 phase-A 表确实没有该列。
阻塞② 成功路径的 degraded 标签
你复现过:把 lib/recall.js 的 'vector_ready_unused' 改回 'hybrid',
76 条全绿 —— 成功路径的标签没有任何断言。已补:
A3c: 健康 embedder ⇒ degraded === 'vector_ready_unused',
且断言 != 'hybrid',并确认 embedText 真被调用了一次
A3c mutation: 交换常量必须抛 ERR_ASSERTION
真实代码变异(把标签改回 'hybrid',跑全量)
not ok 12 - A3c: a healthy embedder reports "vector_ready_unused", NOT "hybrid"
error: group A computes the vector but does not score with it — the label
must say that, because group V will need a different label once
vectors really participate
expected: 'vector_ready_unused'
# tests 81 # pass 80 # fail 1
恢复 → # tests 81 # pass 81 # fail 0
端到端复验(真实 phase-A 库过一遍插件)
loaded on phase-A DB, user_version = 1
commit: Committed block blk_5e3653f55544.
legacy block recallable: true -> 值班室分机号 4324
验证
- npm test 81/81(76 + MIG 三条 + A3c 两条)
- npm run check 9/9
- 提交前连跑 3 次 test + 3 次 check,6 次全绿
待办
- 票③ D7 fail-closed(中心表空但有向量行 ⇒ dim 报 null)跟 F 组
This commit is contained in:
28
README.md
28
README.md
@ -361,6 +361,32 @@ Failures are **hard at the adapter** and **soft at the recall layer**:
|
||||
A wrong dimension is never truncated or padded — a silently mis-sized vector
|
||||
produces a plausible-looking but meaningless score (`block_recall.go:135-138`).
|
||||
|
||||
### Schema migration
|
||||
|
||||
`PRAGMA user_version` gates the upgrade (currently `0 → 1`).
|
||||
|
||||
Phase A shipped only `CREATE TABLE IF NOT EXISTS`, which is a **silent no-op** on
|
||||
an existing table. A database created by the phase-A build therefore opened
|
||||
without error, recalled fine through FTS, and then threw on the **first commit**:
|
||||
|
||||
```
|
||||
PUTBLOCK THREW: table memory_blocks has no column named vector
|
||||
```
|
||||
|
||||
The memory store became read-only with no diagnostic at load time — a worse
|
||||
failure shape than a loud error, because nothing looks wrong until someone tries
|
||||
to write.
|
||||
|
||||
The migration probes `pragma_table_info` instead of guessing from the version
|
||||
number (a version-0 database created by a crashed phase-B build may already have
|
||||
the columns), then `ALTER TABLE ... ADD COLUMN` with `DEFAULT ''` — the same
|
||||
"not embedded yet" sentinel `putBlock` uses. `graph_centroid` from phase A is
|
||||
dropped and rebuilt, but only when it is empty; a populated one refuses loudly
|
||||
rather than destroying data.
|
||||
|
||||
Verified end to end: a real phase-A database opens, reports `user_version = 1`,
|
||||
accepts a commit, and its legacy block is still recallable.
|
||||
|
||||
---
|
||||
|
||||
## Schema
|
||||
@ -455,7 +481,7 @@ because they will bite anyone writing a second dsh service plugin.
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
npm test # 76 tests: store, recall, wiring, loader, isolation, regression, G6/G6b/G7, phase-B A + G + H
|
||||
npm test # 81 tests: store, recall, wiring, loader, isolation, regression, G6/G6b/G7, phase-B A + G + H
|
||||
npm run check # 8 checkpoints with human-readable evidence (V1–V5 + mutations)
|
||||
```
|
||||
|
||||
|
||||
85
lib/store.js
85
lib/store.js
@ -26,6 +26,83 @@ export const MODALITY_TEXT = 'text'
|
||||
|
||||
/** The three tables. Kept as one exec to keep schema creation atomic-ish and
|
||||
* readable next to its HologramAgent counterpart. */
|
||||
/**
|
||||
* Current schema version, stamped into `PRAGMA user_version`.
|
||||
*
|
||||
* 0 phase A: memory_blocks without vector/fingerprint
|
||||
* 1 phase B: + memory_blocks.vector, + memory_blocks.fingerprint,
|
||||
* graph_centroid aligned with HA + is_anomalous column
|
||||
*
|
||||
* ★ Why a version and not just ALTER-and-hope:
|
||||
* every phase-A statement is `CREATE TABLE IF NOT EXISTS`, which is a **silent
|
||||
* no-op** on an existing table. An old database therefore opens without error,
|
||||
* recalls fine through FTS, and throws on the FIRST commit
|
||||
* (`table memory_blocks has no column named vector`). The store becomes
|
||||
* read-only with no diagnostic at load time — the worst shape a failure can
|
||||
* take. The version makes the upgrade explicit and testable.
|
||||
*/
|
||||
export const SCHEMA_VERSION = 1
|
||||
|
||||
/**
|
||||
* Bring an existing database up to {@link SCHEMA_VERSION}.
|
||||
*
|
||||
* Probes the actual column list rather than guessing or try/catching: a
|
||||
* `user_version` of 0 is not proof that the columns are absent (a database
|
||||
* created by a phase-B build that crashed before stamping would look
|
||||
* version-0 while already having them), and relying on ALTER throwing is how
|
||||
* half-applied migrations become invisible.
|
||||
*
|
||||
* @param {import('node:sqlite').DatabaseSync} db an open handle
|
||||
* @returns {number} the version after migrating
|
||||
*/
|
||||
function migrate(db) {
|
||||
let version = db.prepare('PRAGMA user_version').get().user_version
|
||||
|
||||
if (version < 1) {
|
||||
const existing = new Set(
|
||||
db.prepare(`SELECT name FROM pragma_table_info('memory_blocks')`).all().map((row) => row.name),
|
||||
)
|
||||
if (existing.size === 0) {
|
||||
// No table at all: DDL below creates it at the current shape. Nothing to
|
||||
// migrate, and stamping the version is still correct.
|
||||
version = SCHEMA_VERSION
|
||||
} else {
|
||||
// SQLite requires a non-NULL default when adding a NOT NULL column to a
|
||||
// populated table; '' is the same "not embedded yet" sentinel putBlock uses.
|
||||
if (!existing.has('vector')) {
|
||||
db.exec("ALTER TABLE memory_blocks ADD COLUMN vector TEXT NOT NULL DEFAULT ''")
|
||||
}
|
||||
if (!existing.has('fingerprint')) {
|
||||
db.exec("ALTER TABLE memory_blocks ADD COLUMN fingerprint TEXT NOT NULL DEFAULT ''")
|
||||
}
|
||||
version = 1
|
||||
}
|
||||
}
|
||||
|
||||
// graph_centroid changed shape between "created empty in phase A" and phase B.
|
||||
// Phase A created it with (fingerprint, dim, count, sum_json, updated_at); an
|
||||
// empty one is simply dropped so DDL rebuilds it. A populated one never
|
||||
// happens in practice (phase A never wrote a centroid), so refuse loudly
|
||||
// instead of destroying data.
|
||||
const centroidColumns = db
|
||||
.prepare(`SELECT name FROM pragma_table_info('graph_centroid')`)
|
||||
.all()
|
||||
.map((row) => row.name)
|
||||
if (centroidColumns.length > 0 && centroidColumns.includes('sum_json')) {
|
||||
const rows = db.prepare('SELECT COUNT(*) AS n FROM graph_centroid').get().n
|
||||
if (rows > 0) {
|
||||
throw new Error(
|
||||
`dsh-memory: refusing to migrate graph_centroid: phase-A shape with ${rows} row(s). ` +
|
||||
'Phase A never wrote a centroid, so this is unexpected — export the data before upgrading.',
|
||||
)
|
||||
}
|
||||
db.exec('DROP TABLE graph_centroid')
|
||||
}
|
||||
|
||||
db.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`)
|
||||
return SCHEMA_VERSION
|
||||
}
|
||||
|
||||
const DDL = `
|
||||
CREATE TABLE IF NOT EXISTS memory_blocks (
|
||||
id TEXT PRIMARY KEY,
|
||||
@ -170,6 +247,9 @@ export class BlockStore {
|
||||
if (this.#wantsWal && this.#path !== ':memory:') db.exec('PRAGMA journal_mode = WAL')
|
||||
db.exec('PRAGMA foreign_keys = ON')
|
||||
db.exec(DDL)
|
||||
// After DDL: a brand-new database now has the current shape, so the
|
||||
// migration only has work to do for databases that predate phase B.
|
||||
migrate(db)
|
||||
this.#db = db
|
||||
return this
|
||||
}
|
||||
@ -397,6 +477,11 @@ export class BlockStore {
|
||||
return db.prepare(`SELECT name FROM pragma_table_info(?)`).all(table).map((row) => row.name)
|
||||
}
|
||||
|
||||
/** @returns {number} the schema version recorded in `PRAGMA user_version` */
|
||||
userVersion() {
|
||||
return this.#handle().prepare('PRAGMA user_version').get().user_version
|
||||
}
|
||||
|
||||
/** @returns {string} the journal mode actually in force */
|
||||
journalMode() {
|
||||
return this.#handle().prepare('PRAGMA journal_mode').get().journal_mode
|
||||
|
||||
@ -194,3 +194,53 @@ test('A3 mutation: a wrapper that adds any suffix to `why` must break A3', async
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
// ── the SUCCESS path's label (group A: vector computed, not yet used) ──────
|
||||
//
|
||||
// The earlier A3b only pinned the two failure labels. Changing the success label
|
||||
// back to 'hybrid' left all 76 tests green — a label nobody asserts is a label
|
||||
// that will drift. It matters because group V must distinguish "vector computed
|
||||
// but not in scoring" (group A) from "vector actually participating" (group V);
|
||||
// if both say 'hybrid', nobody can tell which build they are looking at.
|
||||
test('A3c: a healthy embedder reports "vector_ready_unused", NOT "hybrid"', async () => {
|
||||
const store = seededStore()
|
||||
try {
|
||||
let embedCalls = 0
|
||||
const healthy = {
|
||||
async embedText() {
|
||||
embedCalls += 1
|
||||
return [0.1, 0.2, 0.3]
|
||||
},
|
||||
info: () => ({ fingerprint: 'fp-healthy', dimension: 3 }),
|
||||
}
|
||||
const hits = await recallBlocksHybrid(store, healthy, '4324', { topK: 1 })
|
||||
|
||||
assert.equal(embedCalls, 1, 'the embedder must actually be called')
|
||||
assert.equal(
|
||||
hits.degraded,
|
||||
'vector_ready_unused',
|
||||
'group A computes the vector but does not score with it — the label must say that, ' +
|
||||
'because group V will need a different label once vectors really participate',
|
||||
)
|
||||
assert.notEqual(hits.degraded, 'hybrid', "calling it 'hybrid' would be a lie until group V lands")
|
||||
// And the result is still the pure-symbol ordering.
|
||||
assert.equal(hits[0].text.includes('4324'), true)
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('A3c mutation: labelling the success path "hybrid" must fail', () => {
|
||||
// Mirrors the exact one-line edit that used to slip through: with no assertion
|
||||
// on this value, reverting the rename was invisible.
|
||||
const LABELS = { groupA: 'vector_ready_unused', groupV: 'hybrid' }
|
||||
// The test above asserts hits.degraded === LABELS.groupA. Prove that swapping
|
||||
// the constant would break it:
|
||||
assert.notEqual(LABELS.groupA, LABELS.groupV)
|
||||
const simulatedDeployed = LABELS.groupV // what the mutated lib returns
|
||||
assert.throws(
|
||||
() => assert.equal(simulatedDeployed, LABELS.groupA),
|
||||
(err) => err.code === 'ERR_ASSERTION',
|
||||
'if lib returns "hybrid", the A3c assertion must throw',
|
||||
)
|
||||
})
|
||||
|
||||
@ -22,7 +22,11 @@
|
||||
*/
|
||||
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdtempSync, rmSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import test from 'node:test'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import { createHttpEmbedder, deriveFingerprint, normalizeEndpoint } from '../lib/embed/http.js'
|
||||
import { BlockStore } from '../lib/store.js'
|
||||
import { recallBlocksHybrid } from '../lib/recall.js'
|
||||
@ -345,3 +349,117 @@ test('ticket②: re-committing without a vector must not erase the stored one',
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
// ── migration: a phase-A database must keep working ───────────────────────
|
||||
//
|
||||
// Phase A shipped `CREATE TABLE IF NOT EXISTS` with no `vector`/`fingerprint`
|
||||
// columns. `open()` re-runs that DDL, which is a SILENT no-op on an existing
|
||||
// table. So a database created by the phase-A build opens fine, recalls fine via
|
||||
// FTS, and then throws on the FIRST commit:
|
||||
//
|
||||
// PUTBLOCK THREW: table memory_blocks has no column named vector
|
||||
//
|
||||
// The memory store becomes read-only with no error at load time. Worse than the
|
||||
// G5 bug: a false alarm gets noticed, this one does not.
|
||||
|
||||
/** Build a database with exactly the phase-A schema (no vector/fingerprint). */
|
||||
function buildPhaseADatabase(path) {
|
||||
const db = new DatabaseSync(path)
|
||||
db.exec(`CREATE TABLE IF NOT EXISTS memory_blocks (
|
||||
id TEXT PRIMARY KEY,
|
||||
modality TEXT NOT NULL,
|
||||
text_content TEXT NOT NULL DEFAULT '',
|
||||
source TEXT NOT NULL DEFAULT '',
|
||||
scene TEXT NOT NULL DEFAULT '',
|
||||
semantic_type TEXT NOT NULL DEFAULT '',
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL)`)
|
||||
db.exec(`CREATE TABLE IF NOT EXISTS memory_block_edges (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, source_kind TEXT NOT NULL, source_id TEXT NOT NULL,
|
||||
target_kind TEXT NOT NULL, target_id TEXT NOT NULL, type TEXT NOT NULL,
|
||||
confidence REAL NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT '', created_at INTEGER NOT NULL)`)
|
||||
db.exec(`CREATE VIRTUAL TABLE IF NOT EXISTS memory_blocks_fts
|
||||
USING fts5(text_content, block_id UNINDEXED, tokenize='unicode61 remove_diacritics 2')`)
|
||||
db.exec(`INSERT INTO memory_blocks (id, modality, text_content, source, created_at, updated_at)
|
||||
VALUES ('blk_legacy', 'text', '值班室分机号 4324', 'legacy', 1, 1)`)
|
||||
db.exec(`INSERT INTO memory_blocks_fts (text_content, block_id) VALUES ('值班室分机号 4324', 'blk_legacy')`)
|
||||
db.close()
|
||||
}
|
||||
|
||||
test('MIG1: a phase-A database gains the vector columns and stays writable', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'mig-'))
|
||||
const path = join(dir, 'phase-a.db')
|
||||
try {
|
||||
buildPhaseADatabase(path)
|
||||
const store = new BlockStore({ path }).open()
|
||||
|
||||
// 1. Columns exist after open.
|
||||
const columns = store.columns('memory_blocks')
|
||||
assert.ok(columns.includes('vector'), `missing vector column; have ${columns.join(',')}`)
|
||||
assert.ok(columns.includes('fingerprint'), `missing fingerprint column; have ${columns.join(',')}`)
|
||||
|
||||
// 2. The legacy block is still readable, and still findable by FTS.
|
||||
const legacy = store.getBlock('blk_legacy')
|
||||
assert.ok(legacy, 'the phase-A block must survive')
|
||||
assert.equal(legacy.text_content, '值班室分机号 4324')
|
||||
assert.equal(legacy.vector, '', 'a pre-existing block has no vector, not a fabricated one')
|
||||
const hits = store.searchFts('"4324"', 5)
|
||||
assert.ok(hits.length >= 1, 'FTS must still find the legacy block')
|
||||
|
||||
// 3. Writing works — this is what used to throw.
|
||||
const { created } = store.putBlock({ text: '迁移之后的新块 8499' })
|
||||
assert.equal(created, true, 'commit must work after migration')
|
||||
|
||||
// 4. The version is stamped.
|
||||
assert.equal(store.userVersion(), 1, 'schema version must be recorded')
|
||||
|
||||
// 5. Re-opening is idempotent (migration must not re-run destructively).
|
||||
store.close()
|
||||
const again = new BlockStore({ path }).open()
|
||||
assert.equal(again.userVersion(), 1)
|
||||
assert.equal(again.countBlocks(), 2)
|
||||
again.close()
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('MIG2: a fresh database is created at version 1', () => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
assert.equal(store.userVersion(), 1, 'a new database starts current, not at 0')
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('MIG1 mutation: skipping the ALTER leaves the store unwritable', () => {
|
||||
// Prove the migration is load-bearing: build the phase-A schema, do NOT
|
||||
// migrate, and show the write fails with the exact error users would hit.
|
||||
const dir = mkdtempSync(join(tmpdir(), 'mig-mut-'))
|
||||
const path = join(dir, 'phase-a.db')
|
||||
try {
|
||||
buildPhaseADatabase(path)
|
||||
const raw = new DatabaseSync(path)
|
||||
// The FAILING write is the one phase B actually performs — it always names
|
||||
// `vector` and `fingerprint`. That is why the failure is a *runtime* error
|
||||
// a long way from load, and why an unwritable store hides behind a clean
|
||||
// start. (A plain insert that names neither column succeeds, which is
|
||||
// exactly the trap: the store looks alive right up to the first real write.)
|
||||
assert.throws(
|
||||
() =>
|
||||
raw.exec(
|
||||
"INSERT INTO memory_blocks (id, modality, text_content, vector, fingerprint, created_at, updated_at) " +
|
||||
"VALUES ('x','text','y','','fp',1,1)",
|
||||
),
|
||||
(err) => /no column named|has no column/i.test(err.message),
|
||||
'skipping the migration must fail at write time',
|
||||
)
|
||||
// And prove the surviving column list is the phase-A one.
|
||||
const cols = raw.prepare(`SELECT name FROM pragma_table_info('memory_blocks')`).all().map((r) => r.name)
|
||||
assert.ok(!cols.includes('vector'), 'a phase-A table has no vector column')
|
||||
raw.close()
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user