jianf
16cfdb75d6
The C4 slice reported this method as dead code: `grep -rn centroidRecallState lib/` returned nothing but the definition, and the instruction was "wire it into recall or delete it — do not leave it as furniture". Neither, and the reason is worth writing down rather than re-deriving: - It cannot be wired into recall. It calls `centroidStatus`, whose first line is `countVectored` — a COUNT that C3-k forbids on the per-turn pre-step path. The scoring path therefore uses `loadCentroid` (one indexed SELECT). - It should not be deleted. `assertC3RecallUnusable` in test/phase-b-h.test.js runs the SAME predicate against every C3 mutant, so this method is what pins Go's `LoadCentroid` semantics (`centroid_store.go:179-185`: a zero-sample centre is unusable, and the recall side carries no stale bit). So it is a diagnostic seam, like `StableMem#saveSpaceCentroid`. The name "Recall-facing projection" was the actual defect — it invites exactly the wiring C3-k forbids. The name stays (renaming would churn a mutation-shared predicate for no behavioural gain); the doc now says what it is and what it is not. Comment only, no behaviour change. Gates: npm test 300/300, npm run check 12/12.
1199 lines
51 KiB
JavaScript
1199 lines
51 KiB
JavaScript
function safeParseJson(value) {
|
||
try { return JSON.parse(value ?? '{}') } catch { return null }
|
||
}
|
||
|
||
function defaultCentroidSchedule(fn, delayMs) {
|
||
setTimeout(fn, delayMs)
|
||
}
|
||
|
||
/**
|
||
* SQLite storage for block memory: the three tables required by the MVP slice.
|
||
*
|
||
* Mirrors HologramAgent's `memory_blocks` / `memory_block_edges` / `graph_centroid`
|
||
* (internal/memory/graph.go:2577, :320, centroid_store.go:58), restricted to the
|
||
* text modality — media blocks are deliberately out of scope for this slice
|
||
* (docs/feasibility.md §5.4: dsh attachment is CAS, which conflicts with
|
||
* HologramAgent's "bytes live inside the block" design).
|
||
*
|
||
* Why node:sqlite instead of a package: it is built in (Node >= 22.5), it already
|
||
* ships FTS5, and dsh itself uses it (dsh-session-query-sqlite/lib/types/schema.d.ts:2).
|
||
* `node --experimental-sqlite` is NOT required on Node 22.23 — the module is
|
||
* available behind an ExperimentalWarning only.
|
||
*
|
||
* @module dsh-memory/store
|
||
*/
|
||
|
||
import { createHash } from 'node:crypto'
|
||
import { mkdirSync } from 'node:fs'
|
||
import { dirname, resolve } from 'node:path'
|
||
import { DatabaseSync } from 'node:sqlite'
|
||
|
||
/** Modality union, mirroring `memory.BlockModality` (internal/memory/block.go:12).
|
||
* Only `text` is produced by this slice; the enum keeps the column honest. */
|
||
export const MODALITY_TEXT = 'text'
|
||
|
||
/** The three tables. Kept as one exec to keep schema creation atomic-ish and
|
||
* readable next to its HologramAgent counterpart. */
|
||
/**
|
||
* Current schema version, stamped into `PRAGMA user_version`.
|
||
*
|
||
* 0 phase A: memory_blocks without vector/fingerprint
|
||
* 1 phase B: + memory_blocks.vector, + memory_blocks.fingerprint,
|
||
* graph_centroid aligned with HA + is_anomalous column
|
||
*
|
||
* ★ Why a version and not just ALTER-and-hope:
|
||
* every phase-A statement is `CREATE TABLE IF NOT EXISTS`, which is a **silent
|
||
* no-op** on an existing table. An old database therefore opens without error,
|
||
* recalls fine through FTS, and throws on the FIRST commit
|
||
* (`table memory_blocks has no column named vector`). The store becomes
|
||
* read-only with no diagnostic at load time — the worst shape a failure can
|
||
* take. The version makes the upgrade explicit and testable.
|
||
*/
|
||
export const SCHEMA_VERSION = 1
|
||
|
||
/**
|
||
* Bring an existing database up to {@link SCHEMA_VERSION}.
|
||
*
|
||
* Probes the actual column list rather than guessing or try/catching: a
|
||
* `user_version` of 0 is not proof that the columns are absent (a database
|
||
* created by a phase-B build that crashed before stamping would look
|
||
* version-0 while already having them), and relying on ALTER throwing is how
|
||
* half-applied migrations become invisible.
|
||
*
|
||
* @param {import('node:sqlite').DatabaseSync} db an open handle
|
||
* @returns {number} the version after migrating
|
||
*/
|
||
function migrate(db) {
|
||
let version = db.prepare('PRAGMA user_version').get().user_version
|
||
|
||
if (version < 1) {
|
||
const existing = new Set(
|
||
db.prepare(`SELECT name FROM pragma_table_info('memory_blocks')`).all().map((row) => row.name),
|
||
)
|
||
// `existing` is read by the vector_dim block below as well, so it is computed
|
||
// once here rather than inside either branch.
|
||
if (existing.size === 0) {
|
||
// No table at all: DDL below creates it at the current shape. Nothing to
|
||
// migrate, and stamping the version is still correct. `existing` is empty,
|
||
// so the vector_dim guard below only runs against the DDL-created column.
|
||
version = SCHEMA_VERSION
|
||
} else {
|
||
// SQLite requires a non-NULL default when adding a NOT NULL column to a
|
||
// populated table; '' is the same "not embedded yet" sentinel putBlock uses.
|
||
if (!existing.has('vector')) {
|
||
db.exec("ALTER TABLE memory_blocks ADD COLUMN vector TEXT NOT NULL DEFAULT ''")
|
||
}
|
||
if (!existing.has('fingerprint')) {
|
||
db.exec("ALTER TABLE memory_blocks ADD COLUMN fingerprint TEXT NOT NULL DEFAULT ''")
|
||
}
|
||
}
|
||
if (!existing.has('vector_dim')) {
|
||
// Backfill from the payload once, at migration time — not on every read.
|
||
// Rows with no vector stay 0.
|
||
db.exec('ALTER TABLE memory_blocks ADD COLUMN vector_dim INTEGER NOT NULL DEFAULT 0')
|
||
for (const row of db.prepare(`SELECT id, vector FROM memory_blocks`).all()) {
|
||
const raw = String(row.vector ?? '')
|
||
if (raw === '') continue
|
||
let dim = 0
|
||
try {
|
||
dim = JSON.parse(raw).length
|
||
} catch {
|
||
dim = -1 // corrupt payload: a non-zero sentinel so it is never "active"
|
||
}
|
||
db.prepare('UPDATE memory_blocks SET vector_dim = ? WHERE id = ?').run(dim, row.id)
|
||
}
|
||
}
|
||
{
|
||
version = 1
|
||
}
|
||
}
|
||
|
||
// graph_centroid changed shape between phase A
|
||
// (fingerprint, dim, count, sum_json, updated_at) and phase B
|
||
// (fingerprint, dim, vector_count, vector, stats, is_anomalous, built_at).
|
||
//
|
||
// ★ The rebuild MUST happen here, not via the DDL above. DDL already ran, so on
|
||
// a pre-existing database its `CREATE TABLE IF NOT EXISTS` was a no-op; if the
|
||
// old table is dropped without rebuilding, the store opens cleanly, stamps
|
||
// user_version=1, accepts putBlock — and throws `no such table` on the first
|
||
// saveCentroid. That is the narrowest possible failure window, which is why
|
||
// MIG1 (memory_blocks only) could not see it. MIG3 pins it.
|
||
//
|
||
// ★ Why a crash between RENAME and the version stamp is NOT a problem — and
|
||
// the reason is NOT the DDL ordering (I initially wrote that, then checked it):
|
||
//
|
||
// `pragma_table_info` on a table that does not exist returns ZERO rows, so
|
||
// `centroidColumns.length === 0` and this whole branch is skipped on the retry.
|
||
// No second RENAME, therefore no `table already exists` — regardless of whether
|
||
// the DDL runs before or after migrate(). Verified by reversing the order: the
|
||
// suite stays green, which is exactly what told me the DDL explanation was
|
||
// wrong.
|
||
//
|
||
// The legacy name is second-resolution; two crashes inside one second still
|
||
// cannot collide because the second attempt never reaches the RENAME at all.
|
||
// MIG4 pins the recovery.
|
||
const centroidColumns = db
|
||
.prepare(`SELECT name FROM pragma_table_info('graph_centroid')`)
|
||
.all()
|
||
.map((row) => row.name)
|
||
if (centroidColumns.length > 0 && centroidColumns.includes('sum_json')) {
|
||
const rows = db.prepare('SELECT COUNT(*) AS n FROM graph_centroid').get().n
|
||
if (rows > 0) {
|
||
// Phase A never wrote a centroid, so this is a "should not happen" branch.
|
||
// Failing the whole open() would punish the user for a shape the plugin
|
||
// could not produce by normal use; dropping the rows would lose data.
|
||
// Renaming aside keeps both properties: startup continues, nothing is
|
||
// destroyed, and the old rows stay inspectable with sqlite3 if anyone cares.
|
||
const stamp = new Date().toISOString().replace(/[^0-9]/g, '').slice(0, 14)
|
||
const legacy = `graph_centroid_phase_a_${stamp}`
|
||
db.exec(`ALTER TABLE graph_centroid RENAME TO "${legacy}"`)
|
||
} else {
|
||
db.exec('DROP TABLE graph_centroid')
|
||
}
|
||
// Rebuild with the phase-B shape, right here (see the note above).
|
||
db.exec(DDL_CENTROID)
|
||
}
|
||
|
||
db.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`)
|
||
return SCHEMA_VERSION
|
||
}
|
||
|
||
/** Phase-B `graph_centroid` shape, reused by the migration to rebuild the table. */
|
||
const DDL_CENTROID = `CREATE TABLE IF NOT EXISTS graph_centroid (
|
||
fingerprint TEXT PRIMARY KEY,
|
||
dim INTEGER NOT NULL,
|
||
vector_count INTEGER NOT NULL,
|
||
vector TEXT NOT NULL,
|
||
stats TEXT NOT NULL DEFAULT '{}',
|
||
is_anomalous INTEGER NOT NULL DEFAULT 0,
|
||
built_at INTEGER NOT NULL
|
||
)`
|
||
|
||
const DDL = `
|
||
CREATE TABLE IF NOT EXISTS memory_blocks (
|
||
id TEXT PRIMARY KEY,
|
||
modality TEXT NOT NULL,
|
||
text_content TEXT NOT NULL DEFAULT '',
|
||
source TEXT NOT NULL DEFAULT '',
|
||
scene TEXT NOT NULL DEFAULT '',
|
||
-- semantic_type mirrors memory_blocks.semantic_type (graph.go:2577)
|
||
semantic_type TEXT NOT NULL DEFAULT '',
|
||
-- ── phase B: vectors ──
|
||
-- vector is a JSON array, matching HA's memory_blocks.vector (graph.go:2577).
|
||
-- Empty string (never 'null', never a zero vector) means "not embedded yet" —
|
||
-- an invented zero vector would participate in scoring and always rank last,
|
||
-- which is a silently wrong memory (distill/blocks.go:110-114).
|
||
vector TEXT NOT NULL DEFAULT '',
|
||
-- vector_dim is vector's LENGTH, stored as its own column so accounting can
|
||
-- bucket by dimension WITHOUT parsing the payload.
|
||
--
|
||
-- Why this exists: blockStates runs on the pre-step hot path (via
|
||
-- renderIndex), and parsing every vector cost 255 ms of a 354 ms call at
|
||
-- 2000 blocks x 2048 dims — a fixed per-turn cost that grows linearly. The
|
||
-- dimension is a property of the row, not of the payload's contents, so a
|
||
-- column is the honest place for it. 0 means "no vector".
|
||
vector_dim INTEGER NOT NULL DEFAULT 0,
|
||
-- fingerprint is the vector-space identity this row's vector belongs to.
|
||
--
|
||
-- ★ It MUST be the same string that graph_centroid.fingerprint holds for the
|
||
-- same space. HA looks the centroid up BY fingerprint
|
||
-- (block_recall.go:92 -> LoadCentroid(q.Fingerprint, ...)); two tables
|
||
-- storing two spellings makes that lookup miss silently — centering simply
|
||
-- never applies, with no error anywhere. Pinned by the two-table-fp test.
|
||
fingerprint TEXT NOT NULL DEFAULT '',
|
||
created_at INTEGER NOT NULL,
|
||
updated_at INTEGER NOT NULL
|
||
);
|
||
|
||
CREATE TABLE IF NOT EXISTS memory_block_edges (
|
||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
source_kind TEXT NOT NULL,
|
||
source_id TEXT NOT NULL,
|
||
target_kind TEXT NOT NULL,
|
||
target_id TEXT NOT NULL,
|
||
type TEXT NOT NULL,
|
||
confidence REAL NOT NULL DEFAULT 0,
|
||
status TEXT NOT NULL DEFAULT '',
|
||
created_at INTEGER NOT NULL
|
||
);
|
||
|
||
CREATE UNIQUE INDEX IF NOT EXISTS idx_edge_identity
|
||
ON memory_block_edges(source_kind, source_id, target_kind, target_id, type);
|
||
CREATE INDEX IF NOT EXISTS idx_edge_source
|
||
ON memory_block_edges(source_kind, source_id);
|
||
CREATE INDEX IF NOT EXISTS idx_edge_target
|
||
ON memory_block_edges(target_kind, target_id);
|
||
|
||
-- graph_centroid: one row per vector space.
|
||
--
|
||
-- Column names follow HA's ddlCentroid (centroid_store.go:58-66) so the two
|
||
-- implementations stay legible side by side: vector_count / vector / stats /
|
||
-- built_at.
|
||
--
|
||
-- is_anomalous is its OWN column rather than a field inside stats.
|
||
-- LoadCentroid decides whether to enable centering from meta.Anomalous
|
||
-- (centroid_store.go:205-209); putting that decision inside a JSON blob would
|
||
-- mean every reader has to parse the blob to learn the one fact that gates the
|
||
-- whole feature. HA keeps it in stats; this plugin lifts it out deliberately.
|
||
|
||
|
||
${DDL_CENTROID};
|
||
|
||
-- ★ vector_dim must never disagree with vector. normalise_vector_dim is invoked
|
||
-- by BOTH triggers below, so a row written by putBlock, by a migration, or by an
|
||
-- operator's raw UPDATE all end up consistent.
|
||
--
|
||
-- Why a trigger rather than "putBlock sets it": the column exists so accounting
|
||
-- can avoid parsing payloads, which means every writer is now responsible for
|
||
-- keeping them in step. One forgotten writer (a repair script, a test fixture, a
|
||
-- future migration) produces a row whose dimension is a lie — and the accounting
|
||
-- would report it as active while recall silently skips it. Enforcing it in the
|
||
-- schema makes that state unrepresentable instead of merely discouraged.
|
||
CREATE TRIGGER IF NOT EXISTS memory_blocks_vector_dim_insert
|
||
AFTER INSERT ON memory_blocks
|
||
FOR EACH ROW
|
||
WHEN NEW.vector_dim != (CASE WHEN NEW.vector IS NULL OR NEW.vector = '' THEN 0
|
||
ELSE json_array_length(NEW.vector) END)
|
||
BEGIN
|
||
UPDATE memory_blocks
|
||
SET vector_dim = CASE WHEN NEW.vector IS NULL OR NEW.vector = '' THEN 0
|
||
ELSE json_array_length(NEW.vector) END
|
||
WHERE id = NEW.id;
|
||
END;
|
||
|
||
CREATE TRIGGER IF NOT EXISTS memory_blocks_vector_dim_update
|
||
AFTER UPDATE OF vector ON memory_blocks
|
||
FOR EACH ROW
|
||
WHEN NEW.vector_dim != (CASE WHEN NEW.vector IS NULL OR NEW.vector = '' THEN 0
|
||
ELSE json_array_length(NEW.vector) END)
|
||
BEGIN
|
||
UPDATE memory_blocks
|
||
SET vector_dim = CASE WHEN NEW.vector IS NULL OR NEW.vector = '' THEN 0
|
||
ELSE json_array_length(NEW.vector) END
|
||
WHERE id = NEW.id;
|
||
END;
|
||
|
||
-- FTS5 index over block text. External-content-free (content stored twice) on
|
||
-- purpose: rows are small, and a plain FTS table keeps deletes trivial.
|
||
CREATE VIRTUAL TABLE IF NOT EXISTS memory_blocks_fts
|
||
USING fts5(
|
||
text_content,
|
||
block_id UNINDEXED,
|
||
-- unicode61 with remove_diacritics handles Latin; CJK needs the trigram
|
||
-- tokenizer to be useful at all (see store.js recall notes).
|
||
tokenize = 'unicode61 remove_diacritics 2'
|
||
);
|
||
`
|
||
|
||
/**
|
||
* Deterministic content-derived block id, ported from
|
||
* `distill.BlockID` (internal/memory/distill/blocks.go:99):
|
||
* `sha256(sentence \0 subject \0 dimension \0 value)[:12]` prefixed `blk_`.
|
||
*
|
||
* The MVP slice has no splitter, so the id derives from the block text alone —
|
||
* same idea (content-addressed ⇒ idempotent re-commit), fewer inputs.
|
||
*/
|
||
export function blockId(text) {
|
||
// createHash avoids a dependency; node:crypto is built in.
|
||
return 'blk_' + createHash('sha256').update(text, 'utf8').digest('hex').slice(0, 12)
|
||
}
|
||
|
||
/**
|
||
* A block memory store backed by one SQLite file.
|
||
*
|
||
* Lifecycle: construct → `open()` → use → `close()`. `close()` is idempotent so
|
||
* a plugin `dispose` and a test teardown can both call it (checkpoint V5).
|
||
*/
|
||
export class BlockStore {
|
||
/** @type {DatabaseSync | null} */
|
||
#db = null
|
||
/**
|
||
* When each space's rebuild was last SCHEDULED (throttle state).
|
||
*
|
||
* ★ In-memory on purpose, with the trade-off written down.
|
||
*
|
||
* The obvious "derive it from the DB" candidate is `graph_centroid.built_at`,
|
||
* and it is the wrong column: `built_at` records when a centroid was
|
||
* successfully BUILT, while a throttle needs to know when a rebuild was last
|
||
* SCHEDULED. The two diverge exactly in the case the throttle exists for —
|
||
* a rebuild that throws. With `built_at` as the throttle, a failing rebuild
|
||
* leaves it untouched, so every subsequent commit schedules again and fails
|
||
* again: unbounded write amplification on the failure path. Deriving a
|
||
* correct value would need a second timestamp column ("last scheduled") to be
|
||
* written on every attempt — a schema change for a per-process optimisation.
|
||
*
|
||
* The cost of keeping it here: a process restart empties the map, so the first
|
||
* drift check after a restart may schedule one extra rebuild. That costs one
|
||
* bounded, asynchronous O(n*d) pass (measured 85.5 ms at 2000 blocks x 512
|
||
* dims) and it happens at most once per space per process lifetime. Against
|
||
* that: a wrong-derived DB throttle would re-schedule on EVERY commit after a
|
||
* failed rebuild.
|
||
*
|
||
* @type {Map<string, number>}
|
||
*/
|
||
#centroidRebuiltAt = new Map()
|
||
/** @type {string} */
|
||
#path
|
||
|
||
/**
|
||
* @param {object} options
|
||
* @param {string} options.path absolute path, or ':memory:' for tests
|
||
* @param {boolean} [options.wal] enable WAL (default true; ignored for :memory:)
|
||
*/
|
||
constructor({ path, wal = true }) {
|
||
this.#path = path === ':memory:' ? ':memory:' : resolve(path)
|
||
this.#wantsWal = wal
|
||
}
|
||
|
||
#wantsWal
|
||
|
||
/**
|
||
* @returns {boolean} whether the handle is open.
|
||
*
|
||
* Named `isOpen`, not `open` — a getter and a method cannot share a name, and
|
||
* the method (which opens the handle) is the one callers reach for.
|
||
*/
|
||
get isOpen() {
|
||
return this.#db !== null
|
||
}
|
||
|
||
/** @returns {string} the resolved database path */
|
||
get path() {
|
||
return this.#path
|
||
}
|
||
|
||
/**
|
||
* Open the handle and create the schema.
|
||
* @returns {this}
|
||
*/
|
||
open() {
|
||
if (this.#db !== null) return this
|
||
if (this.#path !== ':memory:') mkdirSync(dirname(this.#path), { recursive: true })
|
||
const db = new DatabaseSync(this.#path)
|
||
// WAL lets a reader run while a writer holds the write lock. Kept off for
|
||
// :memory: where it is meaningless and would only add a journal file.
|
||
if (this.#wantsWal && this.#path !== ':memory:') db.exec('PRAGMA journal_mode = WAL')
|
||
db.exec('PRAGMA foreign_keys = ON')
|
||
db.exec(DDL)
|
||
// After DDL: a brand-new database now has the current shape, so the
|
||
// migration only has work to do for databases that predate phase B.
|
||
migrate(db)
|
||
this.#db = db
|
||
return this
|
||
}
|
||
|
||
/** @returns {DatabaseSync} the open handle @throws when closed */
|
||
#handle() {
|
||
if (this.#db === null) throw new Error('dsh-memory: store is not open')
|
||
return this.#db
|
||
}
|
||
|
||
/**
|
||
* Insert or update one text block and its FTS row, atomically.
|
||
*
|
||
* Ports the "empty value does not overwrite" rule from `PutMemoryBlocks`
|
||
* (internal/memory/block.go:203-300) in its MVP-relevant part: re-committing
|
||
* the same text must not blank `source` / `scene` / `semantic_type`.
|
||
*
|
||
* @param {object} block
|
||
* @param {string} block.text block body (the payload; `memory_blocks.text_content`)
|
||
* @param {string} [block.source] where it came from
|
||
* @param {string} [block.scene] scene key
|
||
* @param {string} [block.semanticType] semantic category
|
||
* @param {string} [block.id] explicit id; defaults to content-derived
|
||
* @returns {{ id: string, created: boolean }} the id and whether it was new
|
||
*/
|
||
putBlock({ text, source = '', scene = '', semanticType = '', id, vector, fingerprint,
|
||
expectedDimension } = {}) {
|
||
const db = this.#handle()
|
||
const trimmed = String(text ?? '').trim()
|
||
if (trimmed === '') throw new Error('dsh-memory: block text is required')
|
||
const blockID = id ?? blockId(trimmed)
|
||
const now = Date.now()
|
||
// `vector` may be omitted (no embedder) — stored as '' meaning "not embedded".
|
||
// A vector without a fingerprint is refused: an unlabelled vector cannot be
|
||
// compared to anything, and admitting one is exactly the silent cross-space
|
||
// mixing that §0.2 fail-closed exists to prevent.
|
||
const vectorJSON = Array.isArray(vector) && vector.length > 0 ? JSON.stringify(vector) : ''
|
||
if (vectorJSON !== '' && (typeof fingerprint !== 'string' || fingerprint === '')) {
|
||
throw new Error('dsh-memory: a block with a vector requires the vector-space fingerprint')
|
||
}
|
||
// Coerce so a guard bypass cannot surface as an unrelated SQLite bind error;
|
||
// the guard above is what refuses the write, not the type system.
|
||
const vectorFP = vectorJSON === '' ? '' : String(fingerprint ?? '')
|
||
|
||
// ── F8: a vector of the wrong WIDTH is a hard error, not a stored bad row ──
|
||
//
|
||
// HA's rule (`block_recall.go:135-138`, "不截断、不填充"): never coerce a
|
||
// mis-sized vector, because a silently resized vector produces a plausible but
|
||
// meaningless score. This project goes one step further and refuses the write
|
||
// when the caller declares the expected width — storing it and counting it
|
||
// later is "accounting", not "validation", and the criteria ask for validation.
|
||
//
|
||
// `expectedDimension` is opt-in: callers that legitimately do not know the
|
||
// active width (migrations, fixtures, the embed-less phase-A path) are not
|
||
// forced to invent one.
|
||
if (Number.isInteger(expectedDimension) && expectedDimension > 0 && vectorJSON !== '') {
|
||
const actual = JSON.parse(vectorJSON).length
|
||
if (actual !== expectedDimension) {
|
||
throw new Error(
|
||
`dsh-memory: block vector dimension ${actual} does not match the active space's ${expectedDimension}`,
|
||
)
|
||
}
|
||
}
|
||
|
||
db.exec('BEGIN IMMEDIATE')
|
||
try {
|
||
const existing = db.prepare('SELECT id FROM memory_blocks WHERE id = ?').get(blockID)
|
||
const created = existing === undefined
|
||
// ON CONFLICT: every optional column keeps its old value when the new one
|
||
// is empty — the same "空值不覆盖" policy as PutMemoryBlocks.
|
||
db.prepare(
|
||
`INSERT INTO memory_blocks
|
||
(id, modality, text_content, source, scene, semantic_type,
|
||
vector, vector_dim, fingerprint, created_at, updated_at)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||
ON CONFLICT(id) DO UPDATE SET
|
||
text_content = excluded.text_content,
|
||
source = CASE WHEN excluded.source != '' THEN excluded.source ELSE memory_blocks.source END,
|
||
scene = CASE WHEN excluded.scene != '' THEN excluded.scene ELSE memory_blocks.scene END,
|
||
semantic_type = CASE WHEN excluded.semantic_type != '' THEN excluded.semantic_type ELSE memory_blocks.semantic_type END,
|
||
-- Same "empty does not overwrite" rule as PutMemoryBlocks: a re-commit
|
||
-- that carries no vector must not erase a vector computed earlier.
|
||
vector = CASE WHEN excluded.vector != '' THEN excluded.vector ELSE memory_blocks.vector END,
|
||
-- Kept in step with vector by the same "empty does not overwrite" rule.
|
||
vector_dim = CASE WHEN excluded.vector != '' THEN excluded.vector_dim ELSE memory_blocks.vector_dim END,
|
||
fingerprint = CASE WHEN excluded.fingerprint != '' THEN excluded.fingerprint ELSE memory_blocks.fingerprint END,
|
||
updated_at = excluded.updated_at`,
|
||
).run(blockID, MODALITY_TEXT, trimmed, source, scene, semanticType,
|
||
vectorJSON, vectorJSON === '' ? 0 : JSON.parse(vectorJSON).length, vectorFP, now, now)
|
||
|
||
// FTS is not auto-maintained; delete-then-insert keeps it in step.
|
||
db.prepare('DELETE FROM memory_blocks_fts WHERE block_id = ?').run(blockID)
|
||
db.prepare('INSERT INTO memory_blocks_fts (text_content, block_id) VALUES (?, ?)').run(
|
||
trimmed,
|
||
blockID,
|
||
)
|
||
db.exec('COMMIT')
|
||
return { id: blockID, created }
|
||
} catch (error) {
|
||
db.exec('ROLLBACK')
|
||
throw error
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Add one structural/semantic edge. Endpoint existence is NOT validated here
|
||
* (HologramAgent validates it in `AddMemoryBlockEdge`, block.go:497); this
|
||
* slice deliberately allows edges to arbitrary `kind`s so scene/document
|
||
* endpoints can be added in the L2 phase without a schema change.
|
||
*
|
||
* @param {object} edge
|
||
* @returns {{ created: boolean }}
|
||
*/
|
||
addEdge({ sourceKind, sourceId, targetKind, targetId, type, confidence = 0, status = '' }) {
|
||
const db = this.#handle()
|
||
for (const [name, value] of Object.entries({ sourceKind, sourceId, targetKind, targetId, type })) {
|
||
if (typeof value !== 'string' || value === '') {
|
||
throw new Error(`dsh-memory: edge.${name} is required`)
|
||
}
|
||
}
|
||
const result = db
|
||
.prepare(
|
||
`INSERT INTO memory_block_edges
|
||
(source_kind, source_id, target_kind, target_id, type, confidence, status, created_at)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||
ON CONFLICT(source_kind, source_id, target_kind, target_id, type) DO NOTHING`,
|
||
)
|
||
.run(sourceKind, sourceId, targetKind, targetId, type, confidence, status, Date.now())
|
||
return { created: result.changes > 0 }
|
||
}
|
||
|
||
/** @returns {object | undefined} one block row by id */
|
||
getBlock(id) {
|
||
return this.#handle().prepare('SELECT * FROM memory_blocks WHERE id = ?').get(id)
|
||
}
|
||
|
||
/**
|
||
* Persist one vector-space centroid.
|
||
*
|
||
* ★ The fingerprint written here MUST be the exact same string that
|
||
* `putBlock` stores on each block of that space. HA looks the centroid up by
|
||
* fingerprint (`block_recall.go:92`), so two spellings make the lookup miss
|
||
* with no error at all — centering just never happens. The two-table
|
||
* fingerprint test pins this.
|
||
*
|
||
* @param {object} centroid
|
||
* @param {string} centroid.fingerprint
|
||
* @param {number} centroid.dim
|
||
* @param {number} centroid.vectorCount
|
||
* @param {number[]} centroid.mean the mean vector (already normalised)
|
||
* @param {boolean} centroid.isAnomalous whether centering is worth enabling
|
||
* @param {object} [centroid.stats] free-form diagnostics
|
||
* @returns {void}
|
||
*/
|
||
saveCentroid({ fingerprint, dim, vectorCount, mean, isAnomalous, stats = {} }) {
|
||
if (typeof fingerprint !== 'string' || fingerprint === '') {
|
||
throw new Error('dsh-memory: centroid requires a fingerprint')
|
||
}
|
||
if (!Array.isArray(mean) || mean.length !== dim) {
|
||
throw new Error(`dsh-memory: centroid mean must be ${dim}-dimensional`)
|
||
}
|
||
this.#handle()
|
||
.prepare(
|
||
`INSERT INTO graph_centroid
|
||
(fingerprint, dim, vector_count, vector, stats, is_anomalous, built_at)
|
||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||
ON CONFLICT(fingerprint) DO UPDATE SET
|
||
dim = excluded.dim,
|
||
vector_count = excluded.vector_count,
|
||
vector = excluded.vector,
|
||
stats = excluded.stats,
|
||
is_anomalous = excluded.is_anomalous,
|
||
built_at = excluded.built_at`,
|
||
)
|
||
.run(fingerprint, dim, vectorCount, JSON.stringify(mean), JSON.stringify(stats),
|
||
isAnomalous ? 1 : 0, Date.now())
|
||
}
|
||
|
||
/** Mean-vector L2 norm at or above which centering is worth enabling.
|
||
* Ported from `meanThreshold` (internal/memory/centroid.go:87). Measured
|
||
* rationale: at norm 0.9374 de-meaning lifts separation 7x; near 0 the space is
|
||
* already isotropic and centering only amplifies noise. */
|
||
static get MEAN_THRESHOLD() {
|
||
return 0.6
|
||
}
|
||
|
||
/** Minimum number of contribution vectors before a "mean direction" exists.
|
||
*
|
||
* A one-vector mean IS that vector, so its norm is exactly 1.0 and it clears
|
||
* {@link BlockStore.MEAN_THRESHOLD} by construction — a young space would be
|
||
* declared anisotropic on the strength of a single block. Subtracting that
|
||
* "mean" from its own block yields the zero vector, which the scorer drops, so
|
||
* centering would silently remove the only block the space contains. Measured
|
||
* on the C4 fixture: right after the first commit the centroid reported
|
||
* `vectorCount=1 meanLength=1.0000` and a recall that returns 3 rows without a
|
||
* centroid returned 1 with it. Two is the smallest number for which the mean
|
||
* is a property of the SET rather than of one member. */
|
||
static get MIN_SAMPLE() {
|
||
return 2
|
||
}
|
||
|
||
/** Relative change in the vectored-block count that marks a centroid stale.
|
||
* Ported from `centroidRebuildRatio` (internal/memory/centroid_store.go:55).
|
||
* The Go comment pins the rationale: below 10% the shift does not move the
|
||
* principal direction, and being more sensitive than 20% makes similarity
|
||
* drift with every write — a non-reproducible recall cannot serve regression. */
|
||
static get CENTROID_REBUILD_RATIO() {
|
||
return 0.2
|
||
}
|
||
|
||
/** Minimum interval between two scheduled rebuilds of the same space.
|
||
* Ported from `centroidRebuildWait` (internal/memory/centroid_store.go:324). */
|
||
static get CENTROID_REBUILD_WAIT_MS() {
|
||
return 10 * 60 * 1000
|
||
}
|
||
|
||
/** Delay before the scheduled rebuild runs, so the current read settles.
|
||
* Ported from `time.Sleep(2 * time.Second)` (internal/memory/centroid_store.go:338). */
|
||
static get CENTROID_REBUILD_DELAY_MS() {
|
||
return 2000
|
||
}
|
||
|
||
/**
|
||
* Number of blocks in `fingerprint`'s space that currently carry a vector.
|
||
*
|
||
* This is `countVectoredBlocksLocked` (internal/memory/centroid_store.go:178):
|
||
* the DENOMINATOR side of the staleness ratio. It is deliberately NOT called
|
||
* from `blockStates` (this file's pre-step hot path) — it only feeds
|
||
* {@link centroidStatus}.
|
||
*
|
||
* @param {string} fingerprint
|
||
* @returns {number}
|
||
*/
|
||
countVectored(fingerprint) {
|
||
return this.#handle()
|
||
.prepare(
|
||
`SELECT COUNT(*) AS n FROM memory_blocks
|
||
WHERE fingerprint = ? AND vector != '' AND vector != 'null'`,
|
||
)
|
||
.get(String(fingerprint ?? '')).n
|
||
}
|
||
|
||
/**
|
||
* Centroid staleness FACTS for one space (C3).
|
||
*
|
||
* Ports the two Go call sites that answer different questions:
|
||
*
|
||
* · recall path (internal/memory/centroid_store.go:179-204): a centroid
|
||
* built from zero samples is simply UNUSABLE — no staleness judgement.
|
||
* · status path (internal/memory/centroid_store.go:425-433): the same row
|
||
* IS reported stale when blocks now carry vectors.
|
||
*
|
||
* They are separate fields (`available` / `stale`) rather than one merged flag
|
||
* because merging them makes the report self-contradictory.
|
||
*
|
||
* ★ Why the rebuild is scheduled and never run inline (Go :196-204): Go holds
|
||
* `g.mu.RLock` for this whole function and `RebuildCentroid`/`SaveCentroid`
|
||
* need the write lock, so an inline rebuild self-deadlocks (their test hit a
|
||
* 180 s timeout). The JS equivalent is not a lock but a transaction: node:sqlite
|
||
* refuses a nested `BEGIN IMMEDIATE` ("cannot start a transaction within a
|
||
* transaction") and throws SQLITE_BUSY when a statement is still iterating. So
|
||
* this method only REGISTERS a rebuild and returns "no centering this round".
|
||
*
|
||
* ★ C5: every branch now names WHY centring was disabled, in `reason`.
|
||
* Without it the four causes were only inferable from field combinations —
|
||
* `no_centroid` and `dim_mismatch` both report `stale: null`, and `isotropic`
|
||
* is indistinguishable from a healthy space except by `available`. That is
|
||
* fine for a caller that only asks "may I centre?", and useless for an operator
|
||
* asking "why is semantic recall not correcting?".
|
||
*
|
||
* @param {string} fingerprint
|
||
* @param {{queryDim?: number, now?: () => number,
|
||
* schedule?: (fn: () => void, delayMs: number) => void}} [options]
|
||
* `now`/`schedule` are injectable so the throttle and the delay are testable
|
||
* against the real branch logic instead of a stubbed global timer.
|
||
* @returns {{available: boolean, stale: boolean|null, dim: number|null,
|
||
* vectorCount: number|null, currentBlocks: number, delta: number|null,
|
||
* rebuildScheduled: boolean, reason: CentroidReason}}
|
||
*
|
||
* `reason` is one of `ok` / `no_centroid` / `dim_mismatch` / `zero_sample` /
|
||
* `stale` / `isotropic`. `ok` is the only one that coincides with
|
||
* `available: true`; `zero_sample` is a fifth cause that exists in the Go
|
||
* branch (`centroid_store.go:179-185`) even though the plan only names four.
|
||
* @throws when the stored centroid metadata is corrupt (`dim <= 0`,
|
||
* internal/memory/centroid_store.go:165-167)
|
||
*/
|
||
centroidStatus(fingerprint, options = {}) {
|
||
const { queryDim, now = Date.now, schedule = defaultCentroidSchedule } = options
|
||
const currentBlocks = this.countVectored(fingerprint)
|
||
const row = this.#handle()
|
||
.prepare('SELECT dim, vector_count, stats FROM graph_centroid WHERE fingerprint = ?')
|
||
.get(String(fingerprint ?? ''))
|
||
|
||
if (row === undefined) {
|
||
// :153-155 — no row at all is a normal state (Saved=false), not an error.
|
||
return {
|
||
available: false, stale: null, dim: null, vectorCount: null,
|
||
currentBlocks, delta: null, rebuildScheduled: false,
|
||
reason: 'no_centroid',
|
||
}
|
||
}
|
||
|
||
const dim = Number(row.dim)
|
||
if (dim <= 0) {
|
||
// :165-167 — corrupt metadata is an ERROR, not a silent "no centering".
|
||
throw new Error(`dsh-memory: centroid dim is ${dim} (corrupted?)`)
|
||
}
|
||
const vectorCount = Number(row.vector_count)
|
||
|
||
if (Number.isInteger(queryDim) && queryDim > 0 && dim !== queryDim) {
|
||
// :168-170 — wrong width for this query: unusable, not stale.
|
||
return {
|
||
available: false, stale: null, dim, vectorCount, currentBlocks,
|
||
delta: null, rebuildScheduled: false,
|
||
reason: 'dim_mismatch',
|
||
}
|
||
}
|
||
|
||
// :186-189 — (current − built) / built, sign discarded.
|
||
const raw = vectorCount > 0 ? (currentBlocks - vectorCount) / vectorCount : null
|
||
const delta = raw === null ? null : Math.abs(raw)
|
||
|
||
if (vectorCount <= 0) {
|
||
// :179-185 (recall) vs :431-432 (status): NOT available either way, but the
|
||
// status side still calls it stale when blocks exist now.
|
||
return {
|
||
available: false,
|
||
stale: currentBlocks > 0,
|
||
dim, vectorCount, currentBlocks, delta, rebuildScheduled: false,
|
||
reason: 'zero_sample',
|
||
}
|
||
}
|
||
|
||
// :190 — STRICTLY greater; exactly 20% is not stale.
|
||
if (delta > BlockStore.CENTROID_REBUILD_RATIO) {
|
||
const rebuildScheduled = this.#scheduleCentroidRebuild(fingerprint, { now, schedule })
|
||
return {
|
||
available: false, stale: true, dim, vectorCount, currentBlocks, delta, rebuildScheduled,
|
||
reason: 'stale',
|
||
}
|
||
}
|
||
|
||
const stats = safeParseJson(row.stats)
|
||
if (stats?.anomalous === false || !this.loadCentroid(fingerprint)?.isAnomalous) {
|
||
// :205-208 — built as isotropic: centering is not enabled for this space.
|
||
return {
|
||
available: false, stale: false, dim, vectorCount, currentBlocks, delta, rebuildScheduled: false,
|
||
reason: 'isotropic',
|
||
}
|
||
}
|
||
|
||
return {
|
||
available: true, stale: false, dim, vectorCount, currentBlocks, delta, rebuildScheduled: false,
|
||
reason: 'ok',
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Projection of C3 facts for a caller that only asks "may I centre?".
|
||
*
|
||
* Go `LoadCentroid` returns `(nil,false,nil)` for every unusable center,
|
||
* including a zero-sample one (`internal/memory/centroid_store.go:179-185`), and
|
||
* does NOT expose the operator-only stale bit from `CentroidStatusOf:425-433`.
|
||
* This mirrors that: `available` answers the yes/no question, `stale` is forced
|
||
* to null, and `reason` still travels for anyone who needs to log why.
|
||
*
|
||
* ★ NOT the recall path's entry, despite the name. The scoring path uses
|
||
* `loadCentroid` (a single indexed SELECT) because this method calls
|
||
* `centroidStatus`, whose first line is `countVectored` — a COUNT that C3-k
|
||
* forbids on the per-turn pre-step path. Wiring this into recall would put a
|
||
* full-space count on every turn.
|
||
*
|
||
* So it is a diagnostic seam, in the same category as
|
||
* `StableMem#saveSpaceCentroid`: no production caller, kept because it pins the
|
||
* Go semantics (`assertC3RecallUnusable` in test/phase-b-h.test.js runs the same
|
||
* predicate against every C3 mutant). Recorded here because "defined, never
|
||
* called" looked identical to the three real dangling-wire bugs found in this
|
||
* branch — and the only difference worth writing down is this paragraph.
|
||
*
|
||
* @param {string} fingerprint
|
||
* @param {{queryDim?: number, now?: () => number,
|
||
* schedule?: (fn: () => void, delayMs: number) => void}} [options]
|
||
*/
|
||
centroidRecallState(fingerprint, options = {}) {
|
||
const status = this.centroidStatus(fingerprint, options)
|
||
return { ...status, available: status.available, stale: null }
|
||
}
|
||
/**
|
||
* Port of `scheduleCentroidRebuild` (internal/memory/centroid_store.go:327-347).
|
||
*
|
||
* @param {string} fingerprint
|
||
* @param {{now: () => number, schedule: (fn: () => void, delayMs: number) => void}} clock
|
||
* @returns {boolean} whether a rebuild was registered
|
||
*/
|
||
#scheduleCentroidRebuild(fingerprint, { now, schedule }) {
|
||
const last = this.#centroidRebuiltAt.get(fingerprint)
|
||
if (last !== undefined && now() - last < BlockStore.CENTROID_REBUILD_WAIT_MS) {
|
||
return false // :329-332 — inside the throttle window
|
||
}
|
||
this.#centroidRebuiltAt.set(fingerprint, now())
|
||
schedule(() => {
|
||
try {
|
||
this.rebuildCentroid(fingerprint)
|
||
} catch {
|
||
// :340-343 — a failed rebuild must not turn into a recall failure; the
|
||
// next status call sees the still-stale centroid and schedules again.
|
||
}
|
||
}, BlockStore.CENTROID_REBUILD_DELAY_MS)
|
||
return true
|
||
}
|
||
|
||
/**
|
||
* Rebuild the centroid for one vector space FROM ITS BLOCKS.
|
||
*
|
||
* This is the missing entry point: without it `graph_centroid` stays empty
|
||
* forever while blocks keep carrying vectors, `loadCentroid` always misses, and
|
||
* centering silently never applies — the票③ "dim unknown" state would become
|
||
* the permanent normal case rather than an edge case.
|
||
*
|
||
* Port of `RebuildCentroid` (internal/memory/centroid_store.go:246-320):
|
||
* · only rows that carry a vector in THIS space take part
|
||
* · a row whose dimension differs from the first is skipped, not coerced
|
||
* · the mean is accumulated over normalised vectors, then its norm decides
|
||
* whether the space is anomalous enough to centre at all
|
||
*
|
||
* @param {string} fingerprint the space to rebuild
|
||
* @returns {{ fingerprint: string, dim: number, vectorCount: number,
|
||
* meanLength: number, isAnomalous: boolean, saved: boolean }}
|
||
* @throws when the space has no embedded blocks to estimate from
|
||
*/
|
||
rebuildCentroid(fingerprint) {
|
||
if (typeof fingerprint !== 'string' || fingerprint === '') {
|
||
throw new Error('dsh-memory: rebuildCentroid requires a fingerprint')
|
||
}
|
||
const rows = this.#handle()
|
||
.prepare(
|
||
`SELECT vector FROM memory_blocks
|
||
WHERE fingerprint = ? AND vector != '' AND vector != 'null'`,
|
||
)
|
||
.all(fingerprint)
|
||
if (rows.length === 0) {
|
||
throw new Error(`dsh-memory: no embedded blocks for space ${fingerprint}`)
|
||
}
|
||
|
||
let dim = 0
|
||
const vectors = []
|
||
for (const row of rows) {
|
||
let vec
|
||
try {
|
||
vec = JSON.parse(row.vector)
|
||
} catch {
|
||
continue // a corrupt row must not poison the estimate
|
||
}
|
||
if (!Array.isArray(vec) || vec.length === 0) continue
|
||
if (dim === 0) dim = vec.length
|
||
if (vec.length !== dim) continue // mixed provider: skip, never coerce
|
||
vectors.push(vec)
|
||
}
|
||
if (vectors.length === 0) {
|
||
throw new Error(`dsh-memory: no usable vectors for space ${fingerprint}`)
|
||
}
|
||
|
||
// Mean over L2-normalised vectors, so a long vector cannot dominate.
|
||
const sum = new Array(dim).fill(0)
|
||
for (const vec of vectors) {
|
||
let norm = 0
|
||
for (const value of vec) norm += value * value
|
||
norm = Math.sqrt(norm)
|
||
if (norm === 0) continue
|
||
for (let i = 0; i < dim; i += 1) sum[i] += vec[i] / norm
|
||
}
|
||
const mean = sum.map((value) => value / vectors.length)
|
||
let meanLength = 0
|
||
for (const value of mean) meanLength += value * value
|
||
meanLength = Math.sqrt(meanLength)
|
||
// ★ Both conditions matter. `meanLength >= threshold` alone would call a
|
||
// single-block space anisotropic (norm of a unit vector is 1.0), and the
|
||
// resulting centering subtracts a block from itself. See MIN_SAMPLE.
|
||
const isAnomalous = vectors.length >= BlockStore.MIN_SAMPLE
|
||
&& meanLength >= BlockStore.MEAN_THRESHOLD
|
||
|
||
this.saveCentroid({
|
||
fingerprint,
|
||
dim,
|
||
vectorCount: vectors.length,
|
||
mean,
|
||
isAnomalous,
|
||
stats: {
|
||
source: 'rebuildCentroid',
|
||
meanLength,
|
||
avgPairCosine: meanLength * meanLength,
|
||
skippedRows: rows.length - vectors.length,
|
||
},
|
||
})
|
||
return { fingerprint, dim, vectorCount: vectors.length, meanLength, isAnomalous, saved: true }
|
||
}
|
||
|
||
/**
|
||
* Read one centroid, or undefined.
|
||
* @param {string} fingerprint
|
||
* @returns {{fingerprint: string, dim: number, vectorCount: number,
|
||
* mean: number[], isAnomalous: boolean, stats: object} | undefined}
|
||
*/
|
||
loadCentroid(fingerprint) {
|
||
const row = this.#handle()
|
||
.prepare('SELECT * FROM graph_centroid WHERE fingerprint = ?')
|
||
.get(fingerprint)
|
||
if (row === undefined) return undefined
|
||
return {
|
||
fingerprint: row.fingerprint,
|
||
dim: row.dim,
|
||
vectorCount: row.vector_count,
|
||
mean: JSON.parse(row.vector),
|
||
isAnomalous: row.is_anomalous === 1,
|
||
stats: JSON.parse(row.stats),
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Derive the space-accounting state of every block, relative to `activeFingerprint`.
|
||
*
|
||
* ★ Derived from the DB on every call — deliberately NOT an in-memory counter.
|
||
* An accumulator resets on restart and drifts between requests; the row state is
|
||
* the only durable fact. The F3 test mutates the database behind the plugin's
|
||
* back and asserts the counts follow.
|
||
*
|
||
* The four states and the remedy each implies:
|
||
*
|
||
* | state | condition | operator action |
|
||
* |---|---|---|
|
||
* | `noVector` | vector is empty | backfill (embed it) |
|
||
* | `unknownSpace` | has a vector, fingerprint empty | re-embed (it names no space) |
|
||
* | `spaceMismatch` | has a vector, fingerprint ≠ active | re-embed into the active space |
|
||
* | `dimensionMismatch` | has a vector of the wrong width | fix the provider config |
|
||
*
|
||
* `dimensionMismatch` is separate from `spaceMismatch` on purpose (F4): the
|
||
* remedies differ, so one merged number cannot be acted on.
|
||
*
|
||
* `embedFailure` is NOT here: it is a per-call event, not stored state.
|
||
*
|
||
* @param {string} [activeFingerprint] the space to compare against. When absent
|
||
* or empty, no block can be classified as mismatched — the function still
|
||
* reports `noVector` and `unknownSpace`, which are absolute.
|
||
* @param {number} [activeDimension] the active space width, for F4. Omit to skip
|
||
* the dimension comparison.
|
||
* @returns {{total: number, active: number, noVector: number,
|
||
* unknownSpace: number, spaceMismatch: number, dimensionMismatch: number}}
|
||
*/
|
||
blockStates(activeFingerprint, activeDimension) {
|
||
// ★ No vector payload is read here. `vector_dim` carries the length, so this
|
||
// stays O(n) row scans with zero JSON parsing — it runs on the pre-step hot
|
||
// path via renderIndex, where parsing every payload measured 255 ms of a
|
||
// 354 ms call at 2000 blocks x 2048 dims.
|
||
const rows = this.#handle()
|
||
.prepare(`SELECT vector_dim, fingerprint FROM memory_blocks`)
|
||
.all()
|
||
|
||
const active = typeof activeFingerprint === 'string' ? activeFingerprint : ''
|
||
const counts = {
|
||
total: rows.length,
|
||
active: 0,
|
||
noVector: 0,
|
||
unknownSpace: 0,
|
||
spaceMismatch: 0,
|
||
dimensionMismatch: 0,
|
||
}
|
||
|
||
for (const row of rows) {
|
||
const dim = Number(row.vector_dim ?? 0)
|
||
// 0 = no vector; negative = corrupt payload recorded at migration time.
|
||
const hasVector = dim !== 0
|
||
if (!hasVector) {
|
||
counts.noVector += 1
|
||
continue
|
||
}
|
||
const fp = String(row.fingerprint ?? '')
|
||
if (fp === '') {
|
||
// Fail-closed classification (§0.2): an unlabelled vector is not usable,
|
||
// so it is reported as its own state rather than counted as active.
|
||
counts.unknownSpace += 1
|
||
continue
|
||
}
|
||
if (active !== '' && fp !== active) {
|
||
counts.spaceMismatch += 1
|
||
continue
|
||
}
|
||
if (Number.isInteger(activeDimension) && activeDimension > 0 && dim !== activeDimension) {
|
||
// Covers the corrupt-payload sentinel too (dim = -1), which must never
|
||
// classify as active.
|
||
counts.dimensionMismatch += 1
|
||
continue
|
||
}
|
||
counts.active += 1
|
||
}
|
||
return counts
|
||
}
|
||
|
||
/**
|
||
* Distinct fingerprints actually present on blocks, with counts.
|
||
*
|
||
* Read-only and, importantly, does NOT parse vectors: it groups on the
|
||
* `fingerprint` column alone. Grouping must stay cheap because the stats tool
|
||
* calls it on every use.
|
||
*
|
||
* @returns {Array<{fingerprint: string, blocks: number, embedded: number}>}
|
||
*/
|
||
blockFingerprints() {
|
||
return this.#handle()
|
||
.prepare(
|
||
`SELECT fingerprint,
|
||
COUNT(*) AS blocks,
|
||
SUM(CASE WHEN vector != '' THEN 1 ELSE 0 END) AS embedded
|
||
FROM memory_blocks
|
||
WHERE fingerprint != ''
|
||
GROUP BY fingerprint
|
||
ORDER BY fingerprint`,
|
||
)
|
||
.all()
|
||
.map((row) => ({ fingerprint: row.fingerprint, blocks: row.blocks, embedded: row.embedded }))
|
||
}
|
||
|
||
/**
|
||
* F7 / D4 / D7: derive inventory from SQL, never decode block vectors.
|
||
* A missing centroid means unknown dimension (null), even if blocks have vectors.
|
||
* Only vector-bearing blocks name a vector space; unlabelled ones are separate.
|
||
*/
|
||
vectorStats(activeFingerprint = '') {
|
||
const db = this.#handle()
|
||
const totals = db.prepare(`SELECT COUNT(*) AS total,
|
||
COALESCE(SUM(CASE WHEN length(vector) > 0 THEN 1 ELSE 0 END), 0) AS with_vector,
|
||
COALESCE(SUM(CASE WHEN length(vector) > 0 AND fingerprint = '' THEN 1 ELSE 0 END), 0) AS no_fingerprint
|
||
FROM memory_blocks`).get()
|
||
const groups = db.prepare(`SELECT b.fingerprint, c.dim, COUNT(*) AS blocks
|
||
FROM memory_blocks b LEFT JOIN graph_centroid c ON c.fingerprint = b.fingerprint
|
||
WHERE length(b.vector) > 0 AND b.fingerprint != ''
|
||
GROUP BY b.fingerprint ORDER BY b.fingerprint`).all()
|
||
const active = db.prepare('SELECT dim FROM graph_centroid WHERE fingerprint = ?').get(activeFingerprint)
|
||
return {
|
||
blocks: { total: totals.total, with_vector: totals.with_vector, no_vector: totals.total - totals.with_vector },
|
||
spaces: {
|
||
active: { fingerprint: activeFingerprint, dim: active?.dim ?? null },
|
||
by_fingerprint: groups.map((row) => ({
|
||
fingerprint: row.fingerprint,
|
||
dim: row.dim,
|
||
blocks: row.blocks,
|
||
status: row.fingerprint === activeFingerprint ? 'active' : 'stale',
|
||
})),
|
||
no_fingerprint: totals.no_fingerprint,
|
||
},
|
||
}
|
||
}
|
||
|
||
/** @returns {number} block count */
|
||
countBlocks() {
|
||
return this.#handle().prepare('SELECT COUNT(*) AS n FROM memory_blocks').get().n
|
||
}
|
||
|
||
/** @returns {number} edge count */
|
||
countEdges() {
|
||
return this.#handle().prepare('SELECT COUNT(*) AS n FROM memory_block_edges').get().n
|
||
}
|
||
|
||
/**
|
||
* Column names of one table.
|
||
*
|
||
* Introduced for the group-G guard that asserts this slice never grows media
|
||
* columns — the check needs schema introspection, and exposing it here beats
|
||
* reaching into the private handle from a test.
|
||
*
|
||
* @param {string} table table name (asserted against sqlite_master first)
|
||
* @returns {string[]}
|
||
*/
|
||
columns(table) {
|
||
const db = this.#handle()
|
||
const exists = db
|
||
.prepare("SELECT COUNT(*) AS n FROM sqlite_master WHERE type='table' AND name=?")
|
||
.get(table).n
|
||
if (exists === 0) throw new Error(`dsh-memory: no such table: ${table}`)
|
||
return db.prepare(`SELECT name FROM pragma_table_info(?)`).all(table).map((row) => row.name)
|
||
}
|
||
|
||
/**
|
||
* @returns {string[]} user table names (excludes sqlite_ internals)
|
||
*/
|
||
tables() {
|
||
return this.#handle()
|
||
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name")
|
||
.all()
|
||
.map((row) => row.name)
|
||
}
|
||
|
||
/**
|
||
* The raw handle, for assertions the public surface deliberately does not
|
||
* expose (e.g. counting rows in a renamed legacy table). Tests only.
|
||
* @returns {import('node:sqlite').DatabaseSync}
|
||
*/
|
||
raw() {
|
||
return this.#handle()
|
||
}
|
||
|
||
/** @returns {number} the schema version recorded in `PRAGMA user_version` */
|
||
userVersion() {
|
||
return this.#handle().prepare('PRAGMA user_version').get().user_version
|
||
}
|
||
|
||
/** @returns {string} the journal mode actually in force */
|
||
journalMode() {
|
||
return this.#handle().prepare('PRAGMA journal_mode').get().journal_mode
|
||
}
|
||
|
||
/**
|
||
* FTS5 MATCH recall, optionally restricted to rows whose text also contains
|
||
* an exact symbol (see `recall.js` for why both signals are needed).
|
||
*
|
||
* @param {string} match an FTS5 query string (already quoted/sanitised)
|
||
* @param {number} limit
|
||
* @returns {Array<{id: string, text: string, source: string, score: number}>}
|
||
*/
|
||
searchFts(match, limit = 10) {
|
||
const db = this.#handle()
|
||
// bm25() returns a NEGATIVE score where more negative == better; flip it so
|
||
// callers sort descending. FTS5 `rank` is the same value; bm25 is explicit.
|
||
const rows = db
|
||
.prepare(
|
||
`SELECT b.id, b.text_content, b.source, b.fingerprint, b.vector_dim,
|
||
CASE WHEN b.vector != '' THEN 1 ELSE 0 END AS has_vector,
|
||
bm25(memory_blocks_fts) AS raw
|
||
FROM memory_blocks_fts f
|
||
JOIN memory_blocks b ON b.id = f.block_id
|
||
WHERE memory_blocks_fts MATCH ?
|
||
ORDER BY raw
|
||
LIMIT ?`,
|
||
)
|
||
.all(match, limit)
|
||
return rows.map((row) => ({
|
||
id: row.id,
|
||
text: row.text_content,
|
||
source: row.source,
|
||
// Space identity and vector presence travel with every hit, so the symbol
|
||
// path can apply the SAME fingerprint filter the vector path does. HA had
|
||
// to add this after the fact (`fusion.go:481-495`, whose comment records
|
||
// it as "一个真实的安全缺陷": text matching does not depend on the vector
|
||
// space, so blocks from an old space show up in symbol results even after
|
||
// the spaces diverge).
|
||
fingerprint: row.fingerprint,
|
||
hasVector: row.has_vector === 1,
|
||
dimension: row.vector_dim ?? 0,
|
||
// Normalise: bm25 is unbounded negative; map into (0,1) monotonically
|
||
// with 1/(1+|raw|) so exact/rare matches rank highest.
|
||
score: 1 / (1 + Math.abs(row.raw)),
|
||
}))
|
||
}
|
||
|
||
/**
|
||
* All blocks, newest first. Used as the LIKE fallback corpus and by tests.
|
||
* @param {number} [limit]
|
||
*/
|
||
allBlocks(limit = 1000) {
|
||
return this.#handle()
|
||
.prepare('SELECT * FROM memory_blocks ORDER BY created_at DESC, id LIMIT ?')
|
||
.all(limit)
|
||
}
|
||
|
||
/**
|
||
* Candidate pool for the vector route: every block comparable to one space.
|
||
*
|
||
* Three narrowings, each of which `recallBlocks` would otherwise have to repeat
|
||
* (and would eventually repeat inconsistently — the exact half-coverage bug F2
|
||
* exists to stop):
|
||
*
|
||
* · `vector != ''` a row with no vector cannot score; decoding it
|
||
* would only produce a zero.
|
||
* · `fingerprint = ?` same space. Comparing vectors across spaces is
|
||
* meaningless, not merely inaccurate (§0.1/§0.2).
|
||
* · `vector_dim = ?` same width, when the caller names one. A stale
|
||
* row of the wrong width is a decode hazard, and
|
||
* `vector_dim` is maintained by a trigger, so
|
||
* this filter is exact and index-friendly.
|
||
*
|
||
* ★ `SELECT id, vector` and not `SELECT *`: the scoring pass needs the vector
|
||
* and the id, nothing else. Measured on this box at 2000 blocks x 512 dims,
|
||
* the narrow read is 21 ms against 70 ms for `allBlocks(-1)` — and the gap is
|
||
* dominated by the text columns, which grow with note length while vectors do
|
||
* not.
|
||
*
|
||
* ★ Why this lives in the store and not in `recall.js`: every other query in
|
||
* this file is a prepared statement over one table; a recall module that
|
||
* reached for `raw()` to hand-build SQL would put storage knowledge back into
|
||
* the scoring layer.
|
||
*
|
||
* @param {string} fingerprint the vector space to score against
|
||
* @param {number} [dimension] required width; omitted means "any width"
|
||
* @returns {Array<{id: string, vector: string, vector_dim: number}>} raw JSON text
|
||
*/
|
||
vectorCandidates(fingerprint, dimension) {
|
||
const space = String(fingerprint ?? '')
|
||
if (space === '') return []
|
||
const sql = Number.isInteger(dimension) && dimension > 0
|
||
? `SELECT id, vector, vector_dim FROM memory_blocks
|
||
WHERE fingerprint = ? AND vector != '' AND vector != 'null' AND vector_dim = ?`
|
||
: `SELECT id, vector, vector_dim FROM memory_blocks
|
||
WHERE fingerprint = ? AND vector != '' AND vector != 'null'`
|
||
const statement = this.#handle().prepare(sql)
|
||
return Number.isInteger(dimension) && dimension > 0
|
||
? statement.all(space, dimension)
|
||
: statement.all(space)
|
||
}
|
||
|
||
/** Close the handle. Idempotent (checkpoint V5). */
|
||
close() {
|
||
if (this.#db === null) return
|
||
try {
|
||
this.#db.close()
|
||
} finally {
|
||
this.#db = null
|
||
}
|
||
}
|
||
}
|