/** * Block recall without embeddings: FTS5 lexical match + exact-symbol pinning. * * ## Why this exists at all * * HologramAgent's primary recall path is vector cosine over `memory_blocks.vector` * (`internal/memory/block_recall.go:78`). This slice deliberately has no embedding * provider (docs/feasibility.md §5.1, P0 blocker), so it ports only the *second* * half of the fusion: the symbol path. * * That is not a placeholder. `docs/zh/recall-capability-tiers.md` measures which * query shapes actually work, and the strongest one is "带精确串" (queries with an * exact token: port, IP, version, path) — which is exactly the symbol path. The * weakest is "纯泛化描述", which needs vectors. So this slice covers the tier that * empirically works best, and leaves the tier that needs phase B. * * ## The two signals, and why they are merged the way they are * * 1. **Exact symbol hit** — a numeric/Latin token from the query appears verbatim * in the block text. HologramAgent treats this as a BOOLEAN that pins the * candidate to the top, NOT as a weight: * `Score = 1.0 + VectorHit*0.01` (internal/memory/fusion.go:427). * The reason is measured: if you fold a 1.0 symbol hit into * `0.7*vector + 0.3*symbol`, it becomes 0.30 and *loses* to a noise candidate * scoring 0.534 (fusion.go:43-49 documents exactly that regression). * * 2. **FTS5 bm25 lexical rank** — everything else. * * ## Known limitation, deliberately inherited * * `unicode61` does not segment Chinese: a CJK run becomes one token, so * `MATCH '分机号'` will not find `值班室分机号 4324` (the block's run is * `值班室分机号`, a different single token). HologramAgent has the same shape of * problem and solves it with jieba (`internal/memory/fusion.go:145`); this slice * has no jieba (`docs/feasibility.md` §5.5). * * The CJK containment fallback below recovers the common case (the query is a * substring of the text) without a dictionary — the same *effect* as * HologramAgent's sliding-window path (`cjkWindows`, fusion.go:240) at coarser * granularity, and it only runs when FTS found nothing. What it cannot do is * segmentation- or synonym-aware matching; that is phase B/C's job. * * @module dsh-memory/recall */ // ported from go: internal/memory/block_recall.go:263. const largeTopK = 2000 /** * Default vector-route admission threshold for the fused recall path. * * Ported from Go `blockRecallMinScore` (internal/agent/core/toolcall.go:1405), * which is the value production HA actually passes. Its comment there records * why the threshold must NOT be 0: without it, recall on a query his gograph * has nothing on still returns high-scoring noise, and abstention (`AbstainCheck`) * becomes the only thing standing between the model and a confident wrong answer. * * ★ This is an admission gate for the PURE VECTOR route only. `recallBlocks` * applies it per path (see the V5 block), so an exact pin and a symbol hit are * unaffected — that asymmetry is the whole point of V5 and is pinned separately. */ export const DEFAULT_MIN_SCORE = 0.5 /** * Smallest centred-vector norm that still counts as a direction. * * A block equal to the space mean differs from it only by floating-point dust * (~1e-17). Renormalising that dust produces a unit vector pointing in an * arbitrary direction, which then scores against the query as if it were * signal — measured on a space of two identical blocks: cosine 0.2183 out of * dust where the raw cosine was 0.8608. Anything below this floor is "no * direction", which is also what the Go side sees (cosine of a zero vector * is 0, not a random angle). */ const CENTER_EPSILON = 1e-9 /** * Extract high-information symbols from a query. * * Port of `memory.QuerySymbols` (internal/memory/fusion.go:145) restricted to the * parts that need no dictionary: numeric runs and Latin words. The jieba half and * the CJK sliding window are intentionally absent (see the module note). * * @param {string} query * @returns {string[]} lowercased, de-duplicated symbols */ export function querySymbols(query) { const out = [] const seen = new Set() const add = (value) => { const symbol = value.trim() // Single-character symbols are not symbols. Ported from AbstainCheck's guard // (internal/memory/abstain.go:115-116, "单字不成符号"): a bare `0` would // otherwise "exactly" match every block containing a zero and pin noise to // the top. The V2 corpus caught precisely this via `billing0`. if ([...symbol].length < 2 || seen.has(symbol)) return seen.add(symbol) out.push(symbol) } const text = String(query ?? '') // Numeric runs. Regexes copied verbatim from HologramAgent // (internal/memory/fusion.go:76-79) so this slice extracts the SAME symbols: // symbolNumericRe = \d+(?:[./]\d+)*(?:\.\d+)?%?|[vV]\d+(?:\.\d+)+ // symbolLatinRe = [A-Za-z][A-Za-z0-9_-]{2,} // The `{2,}` on the Latin rule is load-bearing: without it `billing0` also // yields a bare `0`, which then "exactly" matches every block containing any // zero — the regression the V2 corpus caught. for (const match of text.matchAll(/\d+(?:[./]\d+)*(?:\.\d+)?%?|[vV]\d+(?:\.\d+)+/g)) add(match[0]) for (const match of text.matchAll(/[A-Za-z][A-Za-z0-9_-]{2,}/g)) add(match[0].toLowerCase()) return out } /** * True when a symbol carries a digit (or is a `v`-prefixed version). * * Port of `isNumericSymbol` (internal/memory/fusion.go:316). It matters because * HologramAgent only trusts "zero hits ⇒ abstain" when the query held an exact * symbol (`internal/memory/abstain.go:79-86`): a pure-CJK zero-hit cannot be told * apart from a tokenizer failure, so abstaining there would be wrong. * * @param {string} symbol * @returns {boolean} */ export function isNumericSymbol(symbol) { if (!symbol) return false if (/\d/.test(symbol)) return true // v2.31.5-style: leading v plus at least one digit. return /^v\d/i.test(symbol) } /** * Build an FTS5 MATCH expression from a plain query. * * Every term is double-quoted so user text cannot inject FTS5 operators * (`AND`, `NEAR`, `*`, `:`). Terms are OR-ed: recall should be generous, and the * fusion step below does the ranking. Returns null when nothing usable remains — * an empty MATCH throws in SQLite. * * @param {string} query * @returns {string | null} */ export function buildMatchExpression(query) { const terms = [] const seen = new Set() const add = (term) => { if (term === '' || seen.has(term)) return seen.add(term) terms.push(`"${term.replace(/"/g, '""')}"`) } const text = String(query ?? '') // 1. The symbols, verbatim. These are the terms that reliably retrieve, // because they are what FTS actually tokenised in the block text // (`billing0服务` is ONE unicode61 token; `billing0` alone is not a token, // so the *word* is what must be matched, not the raw query chunk). for (const symbol of querySymbols(text)) add(symbol.toLowerCase().replace(/[.]/g, ' ')) // 2. Digit groups from the query: `8499`, `2379`, `13010`. Length-gated for // the same reason querySymbols is — a lone `0` (from `billing0`) is not a // symbol, and admitting it turns the MATCH into an everything-query. for (const match of text.matchAll(/\d+(?:[./]\d+)*/g)) { if (match[0].length >= 2) add(match[0]) } // 3. CJK runs, emitted whole. unicode61 keeps a run as ONE token, so quoting // the run matches only identical runs — narrow but correct. Broader CJK // matching is the containment fallback's job (see recallBlocks). for (const match of text.matchAll(/[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]+/g)) add(match[0]) if (terms.length === 0) { // Nothing tokenisable: let the caller fall through to containment rather // than issuing an empty MATCH (which SQLite rejects). return null } return terms.join(' OR ') } /** * Sliding-window needles for a CJK query run. * * Port of `cjkWindows` (internal/memory/fusion.go:240) at the same 2..4 width. * It exists because this slice has no dictionary tokenizer, so "which substring * of the question names the subject" has to be approximated by trying every * window inside the plausible width range. * * Non-overlapping prefixes only would be cheaper but loses `分机号` inside * `值班室分机号` when the question is `值班室分机号是多少`; taking every window * costs a hash probe per block and is still bounded (n·3 needles). * * @param {string} text * @returns {string[]} unique windows, longest first */ export function cjkNeedles(text) { const longestFirst = [] const seen = new Set() for (const run of String(text ?? '').match(/[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]+/g) ?? []) { const chars = [...run] // 4..2 as well as the whole run: the whole run is the most specific needle. for (const width of [chars.length, 4, 3, 2]) { if (width < 2 || width > chars.length) continue for (let start = 0; start + width <= chars.length; start += 1) { const needle = chars.slice(start, start + width).join('') if (seen.has(needle)) continue seen.add(needle) longestFirst.push(needle) } } } return longestFirst } /** * Every start offset of `needle` in `haystack` (non-overlapping scan). * @param {string} haystack already lowercased * @param {string} needle already lowercased * @returns {number[]} */ function allIndexesOf(haystack, needle) { if (needle === '') return [] const out = [] let from = 0 for (;;) { const at = haystack.indexOf(needle, from) if (at === -1) return out out.push(at) from = at + 1 } } /** * Recall blocks for a query. * * Ordering contract (mirrors `RecallBlocksWithArbitration`, * internal/memory/block_recall.go:229 — full scoring first, truncation last): * * 1. FTS5 candidates over a generous pool * 2. exact-symbol boolean pin — pinned candidates first, ordered among * themselves by lexical score * 3. truncate to `topK` * * A LIKE scan runs only when FTS5 yields nothing, mirroring the "符号路兜底" * shape. It is not the primary path: LIKE `%x%` matches almost everything on * short queries and has no notion of relevance, which is exactly what the * A5 mutation test asserts by breaking it. * * @param {import('./store.js').BlockStore} store * @param {string} query * @param {object} [options] * @param {number} [options.topK] max hits returned (default 5) * @param {number} [options.poolSize] FTS candidate pool before truncation (default 50) * @param {Map} [options.vectorScores] optional vector scores for V1 exact-band tie adjustment * @param {number} [options.minScore] opt-in V5 fused route; 0 disables only the vector threshold, * while absence preserves the phase-A path * @returns {Array<{id: string, text: string, source: string, score: number, * exact: boolean, why: string}>} */ export function recallBlocks(store, query, { topK = 5, poolSize = 50, fingerprint, dimension, vectorScores, minScore, skippedUnscoreable = 0 } = {}) { const text = String(query ?? '').trim() if (text === '') return [] const fusedCandidates = Number.isFinite(minScore) const symbols = querySymbols(text) const match = buildMatchExpression(text) // Computed once: used both for fallback admission and for the CJK coverage // signal below, so a longer matching window outranks a shorter one. const cjkWindows = cjkNeedles(text) /** @type {Map} */ const pool = new Map() if (match !== null) { for (const hit of store.searchFts(match, poolSize)) { pool.set(hit.id, { ...hit, lexical: hit.score }) } } // Containment fallback, only when FTS found nothing. // // Three distinct jobs, all legitimate: // · a query with an exact token must still find its block even when FTS // tokenization missed it (a number inside a CJK run, `billing0` glued to // `billing0服务`); // · a CJK query must find text that literally contains it, which unicode61 // cannot answer because it keeps a whole run as ONE token; // · a CJK *question* (`值班室分机号是多少`) must find the note it is asking // about (`值班室分机号 4324`). The runs differ, so whole-run containment // misses — this is where HologramAgent uses jieba and we use sliding // windows (`cjkWindows`, internal/memory/fusion.go:240) instead: every // 2..4-char window of the query run becomes a needle. // // It is NOT the primary path: containment has no relevance notion and matches // almost everything on short queries — the A5 mutation asserts exactly that by // substituting it for FTS as the primary path. if (pool.size === 0) { const needles = symbols.map((symbol) => symbol.toLowerCase()) for (const run of cjkWindows) needles.push(run) if (needles.length > 0) { for (const block of store.allBlocks(poolSize)) { const haystack = String(block.text_content ?? '').toLowerCase() const matched = needles.filter((needle) => haystack.includes(needle)) if (matched.length === 0) continue // Longer needles are more informative; rank containment hits by their // best needle length so `分机号` beats a stray shared bigram. const best = matched.reduce((longest, needle) => Math.max(longest, [...needle].length), 0) pool.set(block.id, { id: block.id, text: block.text_content, source: block.source, // Carry the same identity fields the FTS path carries, or the space // filter below would silently see `hasVector === undefined` and let // every containment hit through — a filter that only covers one of // two intake paths is the bug this whole group is about. fingerprint: block.fingerprint ?? '', hasVector: String(block.vector ?? '') !== '', dimension: Number(block.vector_dim ?? 0), lexical: Math.min(0.02 + best * 0.02, 0.5), // stays under the 1.0 pin band }) } } } // ported from go: internal/memory/fusion.go:374-408, :473-498. // Opt-in fused candidate assembly; absent minScore preserves the frozen A2 // path and the V1/V2 scoring seam. Map membership is vector-route provenance, // not the existence of attribution fields on a returned record. // ported from go: internal/memory/fusion.go:466-468 and block_recall.go:263. // Bound the VECTOR route before union with all symbol matches (fusion.go:473). // This is not a cap on symbol candidates or on the eventual fused result. const vectorCandidates = fusedCandidates && vectorScores instanceof Map ? new Set([...vectorScores].sort((a, b) => b[1] - a[1]).slice(0, largeTopK).map(([id]) => id)) : new Set() if (fusedCandidates) { for (const block of store.allBlocks(-1)) { const symbolMatch = symbols.some((symbol) => String(block.text_content ?? '').toLowerCase().includes(symbol.toLowerCase())) const vectorSource = vectorCandidates.has(block.id) if (!symbolMatch && !vectorSource) continue if (pool.has(block.id)) continue pool.set(block.id, { id: block.id, text: block.text_content, source: block.source, fingerprint: block.fingerprint ?? '', hasVector: String(block.vector ?? '') !== '', dimension: Number(block.vector_dim ?? 0), lexical: 0, }) } } // ── F2: space filter, applied to the SYMBOL path too ──────────────────── // // Text matching does not depend on the vector space, so a block written under // an old space matches the query just as well as a current one. HA shipped // without this filter and recorded the result as "一个真实的安全缺陷" // (`internal/memory/fusion.go:481-495`): the vector path skipped mismatched // blocks while the symbol path returned them, so blocks from a dead space // polluted results after any provider change. // // ★ The same three-condition shape HA uses (`q.Fingerprint != "" && // b.Fingerprint != "" && b.Fingerprint != q.Fingerprint`) is deliberately // NOT reproduced. Per the §0.2 ruling this project is fail-CLOSED on an // unidentifiable block: a block that carries a vector but cannot name its // space is not comparable to anything, so it is excluded rather than passed // through. HA's fail-open arrives from the middle condition being false. // // Blocks with NO vector are unaffected: they never participate in the vector // path either, and their text is still legitimately searchable. // // ★ `skipped` is counted HERE, at the single choke point both intake paths // (FTS and containment) funnel through. Counting at the query site instead would // cover only whichever path the author happened to be looking at — the same // half-coverage that made HA's bug possible. let skipped = 0 if (typeof fingerprint === 'string' && fingerprint !== '') { for (const [id, candidate] of pool) { // A vector plus a mismatched/absent space identity ⇒ not comparable. if (candidate.hasVector === true) { if (candidate.fingerprint === '' || candidate.fingerprint !== fingerprint) { pool.delete(id) skipped += 1 } } } } // ── F4b: dimension filter, counted SEPARATELY from the space filter ────── // // A block can be in the right space and still be unusable: same fingerprint, // wrong width. Only this check catches it, and F4 requires the two counts kept // apart because the remedies differ (fix provider config vs re-embed). // // Order matters: the space check runs first, so a block that is BOTH stale and // narrow counts once, as a space skip — otherwise the two counters would // double-count one row and neither would add up to the number of blocks dropped. let skippedDim = 0 if (Number.isInteger(dimension) && dimension > 0) { for (const [id, candidate] of pool) { if (candidate.hasVector !== true) continue // 0 = no stored width (pre-migration row); treat as unknown, not as wrong. if (candidate.dimension === 0) continue if (candidate.dimension !== dimension) { pool.delete(id) skippedDim += 1 } } } const scored = [] // ── D2: every drop must be countable ───────────────────────────────── // // Three ways a candidate can disappear without becoming a hit, and until now // only the first two (space / width, both upstream) were reported: // // · gatedByScore failed the V5 MinScore gate on its own vector score // · gatedNoEvidence neither a usable vector score nor a symbol match // · truncated survived every gate, fell outside the caller's topK // // Without these, "6 candidates, 1 returned" is indistinguishable from "the // space only holds 1 block" — the operator sees skipped=0 and concludes // nothing was dropped. Measured before this change: 6 blocks, minScore 0.99 → // hits=0, skipped=0, skippedDim=0. let gatedByScore = 0 let gatedNoEvidence = 0 for (const candidate of pool.values()) { const haystack = candidate.text.toLowerCase() // Port of `memory.SymbolScore` (internal/memory/fusion.go:284-308): // exactHit counts symbols that are NUMERIC and appear verbatim; // wordHit counts the rest. // Only a numeric hit earns the boolean pin. A matched Latin word is a // coverage signal, not an exact one — without that distinction any shared // common word would outrank a real port number match. let exactHit = 0 let wordHit = 0 const matched = [] for (const symbol of symbols) { if (!haystack.includes(symbol.toLowerCase())) continue matched.push(symbol) if (isNumericSymbol(symbol)) exactHit += 1 else wordHit += 1 } const exact = exactHit > 0 // 1.0 is a BOOLEAN pin, not a weight (fusion.go:427 and the note above it). // Without any match, fall back to lexical rank, never above the pin band. const coverage = symbols.length === 0 ? 0 : wordHit / symbols.length // Specificity tie-break. When several candidates match the same word, prefer // the one where that word is NOT merely a prefix of a longer word: // query "billing 端口" must prefer `billing服务` over `billing2服务`. // HologramAgent has the same problem and solves it in the splitter with a // word-boundary check (`isCrossWordTailMatch`, distill/split.go:383); here // the cheap equivalent is "the matched symbol is followed by a non-alnum // character or a CJK run boundary". let boundaryBonus = 0 for (const symbol of matched) { for (const at of allIndexesOf(haystack, symbol.toLowerCase())) { const next = haystack[at + symbol.length] // End of text, a non-word char, or a CJK char (word ended cleanly). if (next === undefined || !/[a-z0-9]/.test(next)) { boundaryBonus += 1 break } } } // CJK coverage: the longest query window present in the block, normalised by // the longest possible window. This is what makes `值班室分机号 4324` rank // above a block that merely shares `分机号`, without a dictionary. let cjkCoverage = 0 if (cjkWindows.length > 0) { const longestPossible = Math.max(...cjkWindows.map((window) => [...window].length)) for (const window of cjkWindows) { if (!haystack.includes(window.toLowerCase())) continue const ratio = [...window].length / longestPossible if (ratio > cjkCoverage) cjkCoverage = ratio } } // Banding keeps the signals from swapping places across bands: // [1.0+) exact numeric pin — must always outrank everything else // [0..1) non-exact match — lexical / word / CJK coverage const nonExact = Math.max(candidate.lexical, coverage * 0.5, cjkCoverage * 0.75) // Ported from Go: internal/memory/fusion.go:425-428. // ExactHit is a boolean pin; an explicitly supplied vector score only makes // a same-band tie adjustment. Keep the old no-vector path byte-compatible // for A2: it has no VectorHit to apply. const exactScore = vectorScores instanceof Map ? 1.0 + (vectorScores.get(candidate.id) ?? 0) * 0.01 : 1.0 + candidate.lexical * 0.01 + boundaryBonus * 1e-4 // Ported from Go: internal/memory/fusion.go:61-66 and :430-434. // Non-exact candidates use vector*0.7 + symbol*0.3. A missing vector score // is the symbol-only fallback and must not invent a vector contribution. const hasVectorScore = vectorScores instanceof Map && vectorScores.has(candidate.id) && (!fusedCandidates || vectorCandidates.has(candidate.id)) const vectorScore = hasVectorScore ? vectorScores.get(candidate.id) : 0 const score = exact ? exactScore : fusedCandidates // ported from go: internal/memory/fusion.go:430-434. ? vectorScore * 0.7 + coverage * 0.3 : vectorScores instanceof Map && hasVectorScore ? vectorScore * 0.7 + nonExact * 0.3 : nonExact + boundaryBonus * 1e-3 // ── V3: attribution ────────────────────────────────────────────── // Ported from Go: internal/memory/fusion.go:34-46 (FusedHit's VectorHit / // SymbolHit / Why) and :394-398 (the symbol branch that fills them). // // ★ Every value comes from the branch that already ran — nothing is a // constant decided per shape. Go's own note (`:38-39`) is explicit that 0 // means "did not hit", so the two signals stay independently observable: // a block can carry a vector score and no symbol match, or the reverse. const vectorHit = vectorScore const symbolHit = exact ? 1.0 : coverage // ── V5: asymmetric MinScore gate ───────────────────────────────── // Ported from Go: internal/memory/fusion.go:541-554. // // The threshold applies PER PATH, never to the fused score: the fused score // is a mix of three different scales (exact pin ≥1.0, pure-symbol 0..0.3, // pure-vector 0..0.7), so one threshold over it would either drop every // symbol hit or let every vector hit through. Go records exactly that // mistake and its two failing criteria (:528-529). // // exact hit → never gated (the pin is decisive) // has vector → gated by MinScore against the VECTOR score // symbol only → not gated by MinScore; dropped only when it matched // nothing at all (go :549-553) // // ★ The branch reads `hasVectorScore`, the CANDIDATE-SOURCE flag, not the // presence of the `vectorHit` field. V3 now puts that field on every record, // so existence-based branching would route symbol-only hits into the vector // gate and the "asymmetric" property would silently disappear. if (fusedCandidates) { if (!exact) { if (hasVectorScore && vectorScore > 0) { if (minScore > 0 && vectorScore < minScore) { gatedByScore += 1 continue } } else if (!(symbolHit > 0)) { gatedNoEvidence += 1 continue } } } scored.push({ id: candidate.id, text: candidate.text, source: candidate.source, score, exact, vectorHit, symbolHit, why: vectorScores instanceof Map ? `vector:${vectorHit};symbol:${symbolHit};${exact ? 'exact' : matched.length > 0 ? `word:${matched.join(',')}` : 'lexical'}` : exact ? `exact:${matched.filter(isNumericSymbol).join(',')}` : matched.length > 0 ? `word:${matched.join(',')}` : 'lexical', }) } scored.sort((a, b) => (b.score === a.score ? a.id.localeCompare(b.id) : b.score - a.score)) // ported from go: internal/memory/fusion.go:557-559 — final caller topK, // after candidate union, scoring, filtering and the asymmetric gate. const drops = { gatedByScore, gatedNoEvidence, skippedUnscoreable, truncated: Math.max(0, scored.length - topK), } return withSkipped(scored.slice(0, topK), skipped, skippedDim, drops) } /** * Attach a non-enumerable `skipped` count to a recall result. * * Non-enumerable for the same reason as `degraded`: the phase-A golden fixture * compares result arrays, and an enumerable field would change every stored * expectation. Callers that want the number read `hits.skipped` explicitly, which * also makes it obvious at the call site that they are consuming an out-of-band * signal rather than a hit. * * @param {Array} hits * @param {number} skipped blocks excluded by the space filter * @param {number} [skippedDim] blocks excluded because their width did not match * @param {{gatedByScore?: number, gatedNoEvidence?: number, truncated?: number}} [drops] * the three downstream ways a candidate can disappear. Separate numbers on * purpose: "failed the threshold" and "had nothing to score" have opposite * remedies (lower the threshold vs. re-embed), and a merged count could not be * acted on. `truncated` is the caller's own topK, which is not a defect — it is * reported so that `hits + all counter fields` reconciles with the candidate * pool. * @returns {Array & {skipped: number}} */ function withSkipped(hits, skipped, skippedDim = 0, drops = {}) { Object.defineProperty(hits, 'skipped', { value: skipped, enumerable: false, configurable: true, }) // Reported separately (F4): a dimension skip and a space skip have different // remedies, so one merged number could not be acted on. Object.defineProperty(hits, 'skippedDim', { value: skippedDim, enumerable: false, configurable: true, }) // ── D2: the three downstream drop counts ───────────────────────────── // Also non-enumerable, for the same reason as `skipped`: the phase-A golden // fixture compares result arrays. for (const [key, label] of [ ['gatedByScore', 'skippedScore'], ['gatedNoEvidence', 'skippedNoEvidence'], ['truncated', 'truncated'], // D2's second hole: blocks dropped BEFORE the candidate pool existed. They // are not "gated" (no gate ever saw them) — they could not be scored at all. ['skippedUnscoreable', 'skippedUnscoreable'], ]) { Object.defineProperty(hits, label, { value: Number.isInteger(drops[key]) ? drops[key] : 0, enumerable: false, configurable: true, }) } return hits } /** * Attach a non-enumerable diagnosis to a result array. * * Used instead of widening the return type: `recallBlocks` must stay * byte-compatible with phase A (group A), and every existing caller does * `hits.length` / `hits[0]` on a plain array. A non-enumerable property rides * along without appearing in `JSON.stringify`, `Object.keys`, or deep equality * against phase A output. * * @param {Array} hits * @param {string|null} mode one of `no_embedder` / `embed_error` / `aborted` / * `no_vector_space` / `vector_ready_unused` / null * @param {Error} [error] * @returns {Array} the same array */ function withDiagnosis(hits, mode, error) { Object.defineProperty(hits, 'degraded', { value: mode, enumerable: false, configurable: true, }) if (error !== undefined) { Object.defineProperty(hits, 'degradedError', { value: error instanceof Error ? error.message : String(error), enumerable: false, configurable: true, }) } return hits } /** * The only vector capability this plugin depends on. * * Deliberately three members wide and provider-free: no endpoint, no model name, * no transport. Phase B decides *whether* vectors participate; it does not * decide *which* provider supplies them. Swapping the HTTP adapter for a local * ONNX one must not touch a single criterion. * * @typedef {object} Embedder * @property {(text: string, signal?: AbortSignal) => Promise} embedText * query-side encoding. `signal` fuses the caller's cancellation with the * adapter's own timeout; an implementation that ignores it silently makes a * cancelled turn block until its own bound (the W9 bug). * @property {() => { fingerprint: string, dimension: number }} info * synchronous: the fingerprint/dimension must be checkable *before* paying for * a network round-trip, so dimension mismatch can fail fast * @property {() => Promise} [close] */ /** * Vectors-assisted recall. * * With no embedder (or a failing one) this is exactly `recallBlocks`. With a * healthy one it scores the query vector against the space's stored vectors and * hands `recallBlocks` BOTH `vectorScores` and `minScore`, which together open * the fused candidate assembly, the V1 exact-band tie adjustment, the V2 weighted * blend and the V5 asymmetric gate. * * ★ Why one call and not two: the symbol and vector routes are fused into ONE * scoring pass so that a block the symbol path missed can still be admitted by * its vector score (V5) and so that the candidate pool is never truncated before * fusion (V7). Calling `recallBlocks` twice and merging afterwards cannot * reproduce either property. * * Degradation is reported on the returned array (`arr.degraded`), never by * throwing — the recall main path must survive a dead embedding service * (decision §0.4). * * @param {import('./store.js').BlockStore} store * @param {Embedder | null | undefined} embedder * @param {string} query * @param {object} [options] * @param {number} [options.topK] * @param {number} [options.poolSize] * @param {AbortSignal} [options.signal] * @returns {Promise & {degraded?: string|null}>} */ export async function recallBlocksHybrid(store, embedder, query, options = {}) { // ★ The active space is resolved ONCE and injected into `options` before any // branch can run. Every one of the six degradation paths below funnels into // `recallBlocks(store, query, options)`, so injecting it here means the space // filter applies on ALL of them — the symbol path included. Setting it at each // call site would leave exactly one path unfiltered the first time somebody adds // a seventh, which is how HA's symbol path ended up unfiltered // (`fusion.go:481-495`, recorded there as "一个真实的安全缺陷"). // // When the embedder cannot name its space, `fingerprint` stays undefined and the // filter is inert — recall degrades to its pre-F2 behaviour rather than // silently excluding every vectorised block. const space = embedder && typeof embedder.info === 'function' ? embedder.info() : undefined const scoped = { ...options, fingerprint: typeof space?.fingerprint === 'string' && space.fingerprint !== '' ? space.fingerprint : undefined, // The declared width travels with the space so the recall path can apply the // F4b dimension filter — without it, `skipped_dim` could only ever be 0. dimension: typeof space?.dimension === 'number' && space.dimension > 0 ? space.dimension : undefined, } if (embedder === null || embedder === undefined || typeof embedder.embedText !== 'function') { return withDiagnosis(recallBlocks(store, query, scoped), 'no_embedder') } const info = space if (info === undefined || typeof info.fingerprint !== 'string' || info.fingerprint === '') { // An embedder that cannot name its own space cannot produce comparable // vectors — treat it as absent rather than silently comparing across spaces. return withDiagnosis(recallBlocks(store, query, scoped), 'no_embedder') } let queryVector try { queryVector = await embedder.embedText(String(query ?? ''), options.signal) } catch (error) { // Decision §0.4: never let the vector side take the whole recall down. // // ★ Cancellation and failure are different facts and must not share a label. // `aborted` means "we stopped on purpose", so the model must not be told the // search was incomplete in a way that invites a retry; `embed_error` means // "the provider failed", where absence of evidence really is suspicious. // Collapsing them made the notice say "search FAILED" for a deliberate // cancellation — misleading in exactly the case where the user is watching. const aborted = options.signal?.aborted === true || (error instanceof Error && (error.name === 'AbortError' || /abort/i.test(error.message))) return withDiagnosis(recallBlocks(store, query, scoped), aborted ? 'aborted' : 'embed_error', error) } if (!Array.isArray(queryVector) || queryVector.length === 0) { return withDiagnosis(recallBlocks(store, query, scoped), 'embed_error', new Error(`embedder returned ${Array.isArray(queryVector) ? 'an empty vector' : 'a non-array'}`)) } if (Number.isInteger(info.dimension) && queryVector.length !== info.dimension) { return withDiagnosis(recallBlocks(store, query, scoped), 'embed_error', new Error(`embedder returned dim ${queryVector.length}, declared ${info.dimension}`)) } // ── the wiring ──────────────────────────────────────────────────────────── // // This is the line P1 exists for. Until now the query vector was computed, // dimension-checked, and then DISCARDED: the call below passed `scoped`, which // carries `topK` / `signal` / `fingerprint` / `dimension` and nothing else. So // every V1–V7 scoring branch in `recallBlocks` was reachable only from unit // tests that hand-built a `vectorScores` Map; production recall was pure // symbol/FTS. // // ★ BOTH options have to be passed, and the reason is structural rather than // stylistic: // · `vectorScores` alone reaches only the V1 exact-band tie adjustment // (`:438`), because the fused candidate assembly at `:233` is gated on // `Number.isFinite(minScore)`; without it the vector route can never // admit a block the symbol path missed, which is exactly what V5 needs. // · `minScore` alone does nothing at all, because `:299` additionally // requires `vectorScores` to BE a Map before it builds vector candidates. // The two are one seam split across three guards, so they travel together. // // ★ Scheme A (score the existing space) over scheme B (teach `recallBlocks` to // accept a query vector). B would change a signature that phase A's golden // fixture and nine call sites depend on being synchronous and vector-free, and // it would move cosine arithmetic into a function whose contract is "store + // query text". A leaves `recallBlocks` completely untouched: the seam it // already exposes is the seam we drive. // // Cost, measured on this box before choosing (2000 blocks x 512 dims, the // profile's default space width): narrow `SELECT id, vector` 21 ms + parse and // cosine 60 ms ≈ 81 ms per recall; at 8000 blocks ≈ 310 ms. That is affordable // because the scan is bounded by the space and width filters below and runs // once per TURN (pre-step), not per candidate. Two design choices keep it from // growing worse than linear: // · the space/width filters are pushed into SQL, so blocks from a retired // space or a wrong width are never decoded; // · a space holding no comparable vectors scores nothing at all, so a database // with no embedded blocks (every phase-A store) pays one indexed SELECT that // returns no rows and takes the symbol path untouched. // The remaining risk is a very large single space, which is a ceiling to // measure before raising `topK`/space size — not a reason to ship a vector path // that never runs. Note the fail-closed consequence (§0.2/F5): blocks whose // fingerprint is empty are NOT scored, by design. // // ★ This does not introduce a NEW order of cost. `recallBlocks` already walks // every block for symbol union when `minScore` is finite (the `fusedCandidates` // step, `allBlocks(-1)`), so the fused path was always O(total blocks); what is // added here is one narrow indexed read plus the cosine arithmetic, and the // decode is bounded to the active space and width instead of the whole table. // ── D2: blocks that could not be scored at all ────────────────────────── // // `scoreVectorCandidates` skips a row when it cannot produce a number: bad // JSON, wrong width, a non-finite component, a zero-norm vector, or — since C4 // — a vector that centring collapses to dust. Such a block never enters // `vectorScores`, so `recallBlocks` never puts it in the candidate pool, so // none of the downstream counters can see it. // // Measured: three identical blocks with centring active produced // `hits=0 skipped=0 skippedDim=0 skippedScore=0 skippedNoEvidence=0 truncated=0` // — the whole space vanished while every counter read zero. That is exactly the // silent branch D2 forbids, one stage EARLIER than where D2 first looked. // // The counter is threaded by reference so it can be reported on both the fused // path and the early `no_vector_space` return. const unscoreable = { count: 0 } const vectorScores = scoreVectorCandidates(store, queryVector, scoped, unscoreable) const vectorRoute = vectorScores.size > 0 if (!vectorRoute) { // An empty map would still open the fused candidate assembly (minScore is // finite) and then admit symbol matches with a vector weight of 0 — the // `0.7*0 + 0.3*coverage` regression V2 forbids. Reporting the honest reason // and taking the untouched symbol path is both correct and cheaper. return withDiagnosis( recallBlocks(store, query, { ...scoped, skippedUnscoreable: unscoreable.count }), 'no_vector_space', ) } // `minScore` is only defaulted when the caller did not decide: an explicit // `0` is a deliberate "no threshold" and must survive (it is how the V7 and // pool-truncation criteria disable the gate). const minScore = Number.isFinite(scoped.minScore) ? scoped.minScore : DEFAULT_MIN_SCORE return withDiagnosis( recallBlocks(store, query, { ...scoped, vectorScores, minScore, skippedUnscoreable: unscoreable.count }), // 'hybrid' from here on is a statement of fact, not an aspiration: the vector // route contributed at least one scoreable candidate to the same call that // produced these hits. Group A shipped `vector_ready_unused` precisely so this // rename could not happen before the wiring existed. 'hybrid', ) } /** * Cosine similarity of the query vector against every block in its space. * * Returns the `Map` that `recallBlocks` consumes. An empty Map means * "this space has nothing to score", which the caller must treat as the symbol * path rather than as a vector route with score 0 everywhere. * * Vectors are stored as JSON text (`memory_blocks.vector`, per the DDL comment: * "vector is a JSON array, matching HA's memory_blocks.vector"). A row that fails * to parse is SKIPPED rather than thrown on: one corrupt payload must not take * the whole recall down (decision §0.4), and the migration path already records * corrupt widths as a negative `vector_dim` so they are filterable. * * ★ Centering lives at the BOTTOM of this function (see the C4 block by the * scoring loop): the query is centred once, and each block vector is centred by * the same mean inside the loop. Nothing above this line knows about it. * * ★ Not L2-normalised on the way out. Cosine already divides by both norms, and * `putBlock` does not promise unit vectors, so normalising would change nothing * except to hide a zero vector behind a NaN. * * @param {import('./store.js').BlockStore} store * @param {number[]} queryVector already dimension-checked by the caller * @param {{fingerprint?: string, dimension?: number}} options * @returns {Map} */ function scoreVectorCandidates(store, queryVector, { fingerprint, dimension } = {}, unscoreable = { count: 0 }) { const scores = new Map() // Without a space identity there is nothing to compare against: the vectors in // the table may belong to several incompatible spaces (§0.1), and a cosine // across two of them is a number without a meaning. if (typeof fingerprint !== 'string' || fingerprint === '') return scores const rows = store.vectorCandidates(fingerprint, dimension) if (rows.length === 0) return scores // ── C4 · 中心化校正 ──────────────────────────────────────────────────── // // 一次索引查询取本空间的中心,仅当它被判定为各向异性、且宽度与查询一致 // 时才启用。维度不符时**不校正也不报错**:按错误维度截断或补齐造出来的 // 余弦,看起来像分数,实际没有意义。 // // 门控条件照 C2:`isAnomalous === false` 的空间不校正。这不是优化开关。 // 各向同性空间的均值范数本来就低,减掉它只会放大噪声 —— 实测 8 块正交 // (mean norm 0.3536)在无条件校正下命中数 8 → 1,且因为 V5 的结构性门 // (vectorScore <= 0 且 symbolHit === 0 直接丢弃)无法靠 minScore 调参救回。 // 所以这条 if 是正确性的一部分,不是性能的一部分。 // // ★ loadCentroid 是纯读的一次索引查询,不进写路径、不触发重建。 // 重建挂在写入侧(lib/index.js 的 commit 之后),因为它是 O(n·d), // 放进每轮 pre-step 就是灾难;C3-k 也明令计数查询不得进热路径。 const centroid = store.loadCentroid(fingerprint) const mean = centroid?.isAnomalous === true && centroid.dim === queryVector.length ? centroid.mean : null const centeredQuery = mean === null ? null : centerVector(queryVector, mean) // `mean !== null` 但 `centeredQuery === null` 表示减完是零向量(没有方向), // 同样退回未校正路径,而不是拿零向量去算余弦。 const centering = centeredQuery !== null const scoringQuery = centering ? centeredQuery : queryVector let queryNorm = 0 for (const value of scoringQuery) queryNorm += value * value queryNorm = Math.sqrt(queryNorm) if (!(queryNorm > 0)) return scores for (const row of rows) { let vector try { vector = JSON.parse(String(row.vector ?? '')) } catch { unscoreable.count += 1 continue } if (!Array.isArray(vector) || vector.length !== queryVector.length) { unscoreable.count += 1 continue } // ★ 两边都中心化。只处理一边 = 一边带公共分量、另一边不带,算出来的 // 余弦是两种量纲的混合(HA centroid_store.go:439-442 的原话)。 // 实测:只减 query 时,C4b 那个 5 块 fixture 上同值对的余弦从 1.0 掉到 // 0.1897(余弦对称,只减块侧同值)—— 那就不是在量夹角了。 const scoringVector = centering ? centerVector(vector, mean) : vector if (scoringVector === null) { // ★ C4's dust guard fired: the block sits on the space mean, so it has no // direction to compare against. Counted explicitly, because otherwise this // block disappears BEFORE the candidate pool is assembled and no downstream // counter can see it — measured as "the whole space vanished while every // counter read zero". unscoreable.count += 1 continue } let dot = 0 let norm = 0 for (let index = 0; index < scoringVector.length; index += 1) { const value = Number(scoringVector[index]) // A non-numeric element makes the whole score meaningless. Skip the row // instead of letting NaN propagate into the sort, where it would silently // randomise the order (NaN comparisons are always false). if (!Number.isFinite(value)) { dot = Number.NaN break } dot += value * scoringQuery[index] norm += value * value } if (!Number.isFinite(dot) || !(norm > 0)) { unscoreable.count += 1 continue } scores.set(row.id, dot / (queryNorm * Math.sqrt(norm))) } return scores } /** * 减均值再 L2 归一化(HA `Centroid.CenterVector`,`centroid.go:185-188`)。 * * ★ 归一化不能省。减去均值会改变模长,不归一化就是拿两把不同的尺子量。 * 实测:同一对相同向量,两侧都归一化时余弦恒为 1;只减不归一时它会变成 0.61 * —— 看起来仍是个「分数」,但已经不是在算夹角了。 * * ★ 零向量返回 `null`。减完均值恰好没有方向(向量就躺在中心上)时, * 余弦无定义。调用方必须跳过这一行,而不是拿零向量去算出一个 0 分 * 再让它参与排序 —— 0 分在 V5 的门里等于「未命中」,会把一次真实的 * 无法比较伪装成一次「不相关」。 * * @param {number[]} vector * @param {number[]} mean the space centroid mean, same length * @returns {number[]|null} centered unit vector, or null when it has no direction */ function centerVector(vector, mean) { const out = new Array(vector.length) let norm = 0 for (let index = 0; index < vector.length; index += 1) { const delta = Number(vector[index]) - mean[index] // A non-numeric component poisons the whole direction: bail out rather than // let NaN reach the sort (NaN comparisons are always false ⇒ silent shuffle). if (!Number.isFinite(delta)) return null out[index] = delta norm += delta * delta } norm = Math.sqrt(norm) // ★ Not `norm > 0`. // // A block that IS the mean differs from it only by floating-point dust // (~1e-17). `norm > 0` accepts that dust, and the renormalisation below turns // it into a unit vector pointing in an arbitrary direction — which then scores // against the query as if it were signal. Measured on a space of two identical // blocks: cosine 0.2183 out of dust, where the raw cosine was 0.8608. // // Treating dust as "no direction" also matches the Go side, where the cosine of // a zero vector is 0 rather than a random angle. if (!(norm > CENTER_EPSILON)) return null for (let index = 0; index < out.length; index += 1) out[index] /= norm return out } /** * Render recall hits as the model-facing text injected into the turn. * * Shape follows HologramAgent's `crossModalMarkdown` * (internal/agent/core/crossmodal.go:256): a header that says what the content * IS, then one line per hit. The header matters — the measured failure mode in * `docs/zh/memory-usability-plan.md` was that the model ignored injected memory * because nothing told it these were *historical* fragments rather than the * current conversation. * * ★ A degraded recall renders a notice line. `hits.degraded` is non-enumerable * (so it cannot disturb the phase-A golden comparison), which means it is * invisible to JSON.stringify, Object.keys, and every serialised log. Asserting * it in a test therefore proves nothing about what the MODEL sees — this text is * the only channel by which the model learns that recall was partial. * * @param {Array<{text: string, exact: boolean, score: number, degraded?: string}>} hits * @param {string} [sourceLabel] * @returns {string} rendered block, or '' when there is nothing to say */ export function renderRecall(hits, sourceLabel = 'stablemem') { if (!Array.isArray(hits)) return '' const lines = hits.map((hit, index) => { const tag = hit.exact ? 'exact' : 'text' const body = String(hit.text).replace(/\s+/g, ' ').trim().slice(0, 200) return `[${index + 1}] (${tag} ${hit.score.toFixed(2)}) ${body}` }) const notice = degradationNotice(hits) // ★ "no entries" and "nothing to say" are different questions. // // A degraded recall with zero lexical hits still owes the model an explanation: // otherwise silence means "memory has nothing on this", when the truth is // "memory could not be searched fully". Staying silent there is the false // negative the notice exists to prevent. // // The converse must hold too: a HEALTHY empty recall stays silent. Emitting the // header on every empty turn would burn tokens on the commonest case and train // the model to ignore the memory section entirely. if (lines.length === 0 && notice === '') return '' const header = `[Memory · ${sourceLabel}] Recalled from durable block memory — these are PAST notes, not the current conversation.` return [header, ...(notice === '' ? [] : [notice]), ...lines].join('\n') } /** * Human-readable text for a degraded recall, or '' when nothing was degraded. * * Wording matters: the model must be able to tell "memory has nothing on this" * apart from "memory could not be searched fully", because in the second case * absence of evidence is not evidence of absence. * * @param {Array & {degraded?: string|null}} hits * @returns {string} */ export function degradationNotice(hits) { const mode = hits?.degraded if (mode === undefined || mode === null) return '' // One question, asked once: are there entries below? Every branch shares it. // Wording contract (pinned by W11f, table-driven): nothing === true => no branch // may say "matches below"; nothing === false => the branch must point at them. const nothing = !Array.isArray(hits) || hits.length === 0 switch (mode) { case 'hybrid': // NOT a degradation: this is the healthy, fully-wired path (symbol + vector // fused in one scoring pass). It carries no notice for the same reason `ok` // would not — there is nothing the model needs to know. // // ★ Kept as an explicit case rather than relying on the `default:` branch. // `default:` exists to make an UNKNOWN mode fail loudly-safe ("unknown, not // absent"), so routing the healthiest mode through it would print a false // degradation notice on every single turn that had a vector hit. return '' case 'no_embedder': // Permanent configuration state, not a per-turn failure. The constant index // (systemPrompt.context -> renderIndex) already tells the model that block // memory exists and is searched lexically. Repeating it on every zero-hit // turn costs ~213 chars on the COMMONEST case — measured 10/10 ordinary // turns under the shipping default (no embedder) — and trains the model to // skip the memory section, which is exactly what W11 exists to prevent for // real degradations. W11e pins the silence. return '' case 'embed_error': { // Two wordings, because "matches below are …" is a lie when nothing was // found. The distinction matters: with zero entries the model must know the // search was incomplete AND empty, not merely incomplete. return nothing ? '[Memory] Semantic search FAILED for this turn and the lexical fallback found nothing. Absence of a memory here does not prove it is unrecorded.' : '[Memory] Semantic search FAILED for this turn; matches below are lexical/symbolic only. Absence of a memory here does not prove it is unrecorded.' } case 'aborted': // Deliberate stop, not a fault: say so plainly and do NOT imply the memory // is incomplete, which would invite a pointless retry. Same two-wording rule // as embed_error — a cancelled turn that ALSO found nothing is the commonest // shape of an abort (the user stops, and their query matches nothing), so // "matches below are …" is a lie in exactly the case that happens most. return nothing ? '[Memory] Semantic search was cancelled for this turn and the lexical fallback found nothing.' : '[Memory] Semantic search was cancelled for this turn; matches below are lexical/symbolic only.' case 'vector_ready_unused': // Group A state: the vector was computed but does not take part in scoring // yet. Not something the model needs to hear about — but keeping the case // explicit means a future mode cannot silently fall through to ''. return '' case 'no_vector_space': // A space IS configured and the embedder answered, but no block in that // space carries a usable vector (a fresh space, or every row pre-dates the // embedder). Unlike `no_embedder` this is a transitional DATA state, and it // is the one an operator most often mistakes for "semantic recall is on". // Still silent for the model: the fallback it received is the correct one, // and `stablemem_stats` is where the operator is told why. return '' default: // A mode nobody anticipated. "Not found" must not read as "not recorded": // this is the ONLY path a future mode can take, so it is the worst possible // place to start lying. Carries the same obligation as embed_error. return nothing ? `[Memory] Recall degraded (${mode}); nothing was found, so treat this as "unknown", not "absent".` : `[Memory] Recall degraded (${mode}); matches below may be incomplete.` } } /** Fixed text heuristic matching dsh-token-meter: 4 chars/token plus framing. */ export function estimateMemoryTokens(text) { return Math.ceil(String(text ?? "").length / 4) + 8 } /** Estimate rendered hit cost from its note body, not serialized JSON. */ export function estimateRecallHitTokens(hit) { const body = String(hit?.text ?? "").replace(/\s+/g, " ").trim().slice(0, 200) return estimateMemoryTokens(body) + 20 } /** Select a recall count under explicit, host-derived, or fallback budget. */ export function segmentRecallBudget(hits, options = {}) { const topK = Number.isFinite(options.topK) ? Math.max(1, Math.floor(options.topK)) : 3 const explicit = Number.isFinite(options.budgetTokens) && options.budgetTokens > 0 ? options.budgetTokens : undefined const window = Number.isFinite(options.contextWindow) && options.contextWindow > 0 ? options.contextWindow : undefined const source = explicit !== undefined ? "explicit" : window !== undefined ? "host" : "fallback" const budget = explicit ?? (window === undefined ? 4096 : Math.max(1, Math.floor(window * 0.8))) const rows = Array.isArray(hits) ? hits : [] if (rows.length === 0) return { hits: rows, budgetTokens: budget, source, singleHitOverBudget: false } const costs = rows.map(estimateRecallHitTokens) const average = costs.reduce((sum, cost) => sum + cost, 0) / costs.length const count = Math.max(1, Math.min(topK, Math.floor(budget / Math.max(1, average)))) const retained = rows.slice(0, count) // ★ `slice` copies indexed elements only. `degraded` / `degradedError` / // `skipped` are NON-enumerable by design (so the phase-A golden fixture and // every serialised log keep their shape), which means a plain slice silently // drops them — and the degradation notice would then never reach the model. // Re-attach every out-of-band channel the recall path is known to carry. for (const key of ['degraded', 'degradedError', 'skipped', 'skippedDim']) { if (Object.prototype.hasOwnProperty.call(rows, key)) { Object.defineProperty(retained, key, { value: rows[key], enumerable: false, configurable: true, }) } } const singleHitOverBudget = costs[0] > budget Object.defineProperty(retained, "budgetNotice", { value: singleHitOverBudget ? `[Memory] Top note estimate ${costs[0]} tokens exceeds memory budget ${budget}; retained without truncation.` : count < rows.length ? `[Memory] Recall limited to ${count} note(s) by memory budget ${budget}.` : "", enumerable: false }) return { hits: retained, budgetTokens: budget, source, singleHitOverBudget } }