jianf
7652184832
The C4 slice instruction said, verbatim: "判 stale 的那一轮仍返回结果（用未校正向量）".
It was not implemented. `grep stale lib/recall.js` returned one comment and no code,
and `loadCentroid` carries no staleness verdict — so recall centred with whatever
mean it found, however old.
## Why not simply ask on the recall side
The obvious fix — call `centroidStatus` in `recallBlocks` — was built and measured:
# tests 305 # pass 304 # fail 1
not ok 97 - C3-k: the recall hot path must not run a COUNT query
error: 'lib/recall.js must not call centroidStatus — it runs countVectored
internally, and the recall path may only read the centroid'
C3-k is a rule this project set deliberately. A fix that breaks it is not a fix,
and the criterion caught the attempt — which is what it is for.
## What this does instead
The WRITE path already pays for the judgement on every commit (`#ensureCentroid`
calls `centroidStatus`). So it persists the verdict:
· `BlockStore#markCentroidStale(fp)` — `json_set(stats, '$.stale', 1)` when the
space is judged stale
· `#ensureCentroid` marks on `status.stale === true`
· `recallBlocks` honours it: `!centroid.stats?.stale` before centring
· cleared implicitly — `rebuildCentroid` → `saveCentroid` upserts stats without
the key
Recall never runs a COUNT; C3-k stays green.
## The bug this shipped with for one revision — and why it matters
The guard was first written `centroid.stats?.stale !== true`. The flag is stored as
the JSON number `1`, and in JS **`1 !== true` is true** — a strict comparison across
types — so the guard was a permanent no-op. Stale centroids kept being used.
**The entire suite was 305/305 green with that bug in place.** Nothing covered the
path. It was only visible on a fixture where centring is observable (the C4
hub/target space):
fresh centroid top1=target 0.9000 / 0.2129 / 0.1814
marked stale top1=hub 0.9928 / 0.9859 / 0.9803 <- raw cosine, uncorrected
after rebuild top1=target 0.9000 / 0.2129 / 0.1814
Hence C4j (below) rather than a comment claiming it works.
## Criterion C4j
Three assertions on that fixture: fresh → target; marked stale → hub AND the score
is the raw cosine (0.9928); rebuild → target again. Mutation — deleting the guard:
not ok 11 - C4j: a centroid marked stale falls back to uncorrected vectors
error: ... If this still says "target", the flag is being ignored — the exact bug
that a strict `!== true` comparison produced while the whole suite stayed green.
'target' !== 'hub'
## What this does NOT establish
That falling back improves ranking quality. An attempt to measure it could not
separate the cases: with a near-duplicate late corpus, scores saturate near 0.99
whether the mean is stale (0.997) or absent (0.994); only a freshly rebuilt mean
spread them (0.978..0.970). So this is a safety choice — a mean built from a
fraction of the space describes a space that no longer exists, and applying it is
unjustifiable on principle. The measurable win belongs to keeping the centroid
FRESH (the throttle in C3), not to skipping it.
Gates: npm test 306/306 (0 skipped), npm run check 12/12, frozen faces untouched.
1196 lines
58 KiB
JavaScript
1196 lines
58 KiB
JavaScript
/**
|
||
* Block recall without embeddings: FTS5 lexical match + exact-symbol pinning.
|
||
*
|
||
* ## Why this exists at all
|
||
*
|
||
* HologramAgent's primary recall path is vector cosine over `memory_blocks.vector`
|
||
* (`internal/memory/block_recall.go:78`). This slice deliberately has no embedding
|
||
* provider (docs/feasibility.md §5.1, P0 blocker), so it ports only the *second*
|
||
* half of the fusion: the symbol path.
|
||
*
|
||
* That is not a placeholder. `docs/zh/recall-capability-tiers.md` measures which
|
||
* query shapes actually work, and the strongest one is "带精确串" (queries with an
|
||
* exact token: port, IP, version, path) — which is exactly the symbol path. The
|
||
* weakest is "纯泛化描述", which needs vectors. So this slice covers the tier that
|
||
* empirically works best, and leaves the tier that needs phase B.
|
||
*
|
||
* ## The two signals, and why they are merged the way they are
|
||
*
|
||
* 1. **Exact symbol hit** — a numeric/Latin token from the query appears verbatim
|
||
* in the block text. HologramAgent treats this as a BOOLEAN that pins the
|
||
* candidate to the top, NOT as a weight:
|
||
* `Score = 1.0 + VectorHit*0.01` (internal/memory/fusion.go:427).
|
||
* The reason is measured: if you fold a 1.0 symbol hit into
|
||
* `0.7*vector + 0.3*symbol`, it becomes 0.30 and *loses* to a noise candidate
|
||
* scoring 0.534 (fusion.go:43-49 documents exactly that regression).
|
||
*
|
||
* 2. **FTS5 bm25 lexical rank** — everything else.
|
||
*
|
||
* ## Known limitation, deliberately inherited
|
||
*
|
||
* `unicode61` does not segment Chinese: a CJK run becomes one token, so
|
||
* `MATCH '分机号'` will not find `值班室分机号 4324` (the block's run is
|
||
* `值班室分机号`, a different single token). HologramAgent has the same shape of
|
||
* problem and solves it with jieba (`internal/memory/fusion.go:145`); this slice
|
||
* has no jieba (`docs/feasibility.md` §5.5).
|
||
*
|
||
* The CJK containment fallback below recovers the common case (the query is a
|
||
* substring of the text) without a dictionary — the same *effect* as
|
||
* HologramAgent's sliding-window path (`cjkWindows`, fusion.go:240) at coarser
|
||
* granularity, and it only runs when FTS found nothing. What it cannot do is
|
||
* segmentation- or synonym-aware matching; that is phase B/C's job.
|
||
*
|
||
* @module dsh-memory/recall
|
||
*/
|
||
|
||
// ported from go: internal/memory/block_recall.go:263.
|
||
const largeTopK = 2000
|
||
|
||
/**
|
||
* Default vector-route admission threshold for the fused recall path.
|
||
*
|
||
* Ported from Go `blockRecallMinScore` (internal/agent/core/toolcall.go:1405),
|
||
* which is the value production HA actually passes. Its comment there records
|
||
* why the threshold must NOT be 0: without it, recall on a query his gograph
|
||
* has nothing on still returns high-scoring noise, and abstention (`AbstainCheck`)
|
||
* becomes the only thing standing between the model and a confident wrong answer.
|
||
*
|
||
* ★ This is an admission gate for the PURE VECTOR route only. `recallBlocks`
|
||
* applies it per path (see the V5 block), so an exact pin and a symbol hit are
|
||
* unaffected — that asymmetry is the whole point of V5 and is pinned separately.
|
||
*/
|
||
export const DEFAULT_MIN_SCORE = 0.5
|
||
|
||
/**
|
||
* Smallest centred-vector norm that still counts as a direction.
|
||
*
|
||
* A block equal to the space mean differs from it only by floating-point dust
|
||
* (~1e-17). Renormalising that dust produces a unit vector pointing in an
|
||
* arbitrary direction, which then scores against the query as if it were
|
||
* signal — measured on a space of two identical blocks: cosine 0.2183 out of
|
||
* dust where the raw cosine was 0.8608. Anything below this floor is "no
|
||
* direction", which is also what the Go side sees (cosine of a zero vector
|
||
* is 0, not a random angle).
|
||
*/
|
||
const CENTER_EPSILON = 1e-9
|
||
|
||
/**
|
||
* Extract high-information symbols from a query.
|
||
*
|
||
* Port of `memory.QuerySymbols` (internal/memory/fusion.go:145) restricted to the
|
||
* parts that need no dictionary: numeric runs and Latin words. The jieba half and
|
||
* the CJK sliding window are intentionally absent (see the module note).
|
||
*
|
||
* @param {string} query
|
||
* @returns {string[]} lowercased, de-duplicated symbols
|
||
*/
|
||
export function querySymbols(query) {
|
||
const out = []
|
||
const seen = new Set()
|
||
const add = (value) => {
|
||
const symbol = value.trim()
|
||
// Single-character symbols are not symbols. Ported from AbstainCheck's guard
|
||
// (internal/memory/abstain.go:115-116, "单字不成符号"): a bare `0` would
|
||
// otherwise "exactly" match every block containing a zero and pin noise to
|
||
// the top. The V2 corpus caught precisely this via `billing0`.
|
||
if ([...symbol].length < 2 || seen.has(symbol)) return
|
||
seen.add(symbol)
|
||
out.push(symbol)
|
||
}
|
||
const text = String(query ?? '')
|
||
// Numeric runs. Regexes copied verbatim from HologramAgent
|
||
// (internal/memory/fusion.go:76-79) so this slice extracts the SAME symbols:
|
||
// symbolNumericRe = \d+(?:[./]\d+)*(?:\.\d+)?%?|[vV]\d+(?:\.\d+)+
|
||
// symbolLatinRe = [A-Za-z][A-Za-z0-9_-]{2,}
|
||
// The `{2,}` on the Latin rule is load-bearing: without it `billing0` also
|
||
// yields a bare `0`, which then "exactly" matches every block containing any
|
||
// zero — the regression the V2 corpus caught.
|
||
for (const match of text.matchAll(/\d+(?:[./]\d+)*(?:\.\d+)?%?|[vV]\d+(?:\.\d+)+/g)) add(match[0])
|
||
for (const match of text.matchAll(/[A-Za-z][A-Za-z0-9_-]{2,}/g)) add(match[0].toLowerCase())
|
||
return out
|
||
}
|
||
|
||
/**
|
||
* True when a symbol carries a digit (or is a `v`-prefixed version).
|
||
*
|
||
* Port of `isNumericSymbol` (internal/memory/fusion.go:316). It matters because
|
||
* HologramAgent only trusts "zero hits ⇒ abstain" when the query held an exact
|
||
* symbol (`internal/memory/abstain.go:79-86`): a pure-CJK zero-hit cannot be told
|
||
* apart from a tokenizer failure, so abstaining there would be wrong.
|
||
*
|
||
* @param {string} symbol
|
||
* @returns {boolean}
|
||
*/
|
||
export function isNumericSymbol(symbol) {
|
||
if (!symbol) return false
|
||
if (/\d/.test(symbol)) return true
|
||
// v2.31.5-style: leading v plus at least one digit.
|
||
return /^v\d/i.test(symbol)
|
||
}
|
||
|
||
/**
|
||
* Build an FTS5 MATCH expression from a plain query.
|
||
*
|
||
* Every term is double-quoted so user text cannot inject FTS5 operators
|
||
* (`AND`, `NEAR`, `*`, `:`). Terms are OR-ed: recall should be generous, and the
|
||
* fusion step below does the ranking. Returns null when nothing usable remains —
|
||
* an empty MATCH throws in SQLite.
|
||
*
|
||
* @param {string} query
|
||
* @returns {string | null}
|
||
*/
|
||
export function buildMatchExpression(query) {
|
||
const terms = []
|
||
const seen = new Set()
|
||
const add = (term) => {
|
||
if (term === '' || seen.has(term)) return
|
||
seen.add(term)
|
||
terms.push(`"${term.replace(/"/g, '""')}"`)
|
||
}
|
||
const text = String(query ?? '')
|
||
|
||
// 1. The symbols, verbatim. These are the terms that reliably retrieve,
|
||
// because they are what FTS actually tokenised in the block text
|
||
// (`billing0服务` is ONE unicode61 token; `billing0` alone is not a token,
|
||
// so the *word* is what must be matched, not the raw query chunk).
|
||
for (const symbol of querySymbols(text)) add(symbol.toLowerCase().replace(/[.]/g, ' '))
|
||
|
||
// 2. Digit groups from the query: `8499`, `2379`, `13010`. Length-gated for
|
||
// the same reason querySymbols is — a lone `0` (from `billing0`) is not a
|
||
// symbol, and admitting it turns the MATCH into an everything-query.
|
||
for (const match of text.matchAll(/\d+(?:[./]\d+)*/g)) {
|
||
if (match[0].length >= 2) add(match[0])
|
||
}
|
||
|
||
// 3. CJK runs, emitted whole. unicode61 keeps a run as ONE token, so quoting
|
||
// the run matches only identical runs — narrow but correct. Broader CJK
|
||
// matching is the containment fallback's job (see recallBlocks).
|
||
for (const match of text.matchAll(/[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]+/g)) add(match[0])
|
||
|
||
if (terms.length === 0) {
|
||
// Nothing tokenisable: let the caller fall through to containment rather
|
||
// than issuing an empty MATCH (which SQLite rejects).
|
||
return null
|
||
}
|
||
return terms.join(' OR ')
|
||
}
|
||
|
||
/**
|
||
* Sliding-window needles for a CJK query run.
|
||
*
|
||
* Port of `cjkWindows` (internal/memory/fusion.go:240) at the same 2..4 width.
|
||
* It exists because this slice has no dictionary tokenizer, so "which substring
|
||
* of the question names the subject" has to be approximated by trying every
|
||
* window inside the plausible width range.
|
||
*
|
||
* Non-overlapping prefixes only would be cheaper but loses `分机号` inside
|
||
* `值班室分机号` when the question is `值班室分机号是多少`; taking every window
|
||
* costs a hash probe per block and is still bounded (n·3 needles).
|
||
*
|
||
* @param {string} text
|
||
* @returns {string[]} unique windows, longest first
|
||
*/
|
||
export function cjkNeedles(text) {
|
||
const longestFirst = []
|
||
const seen = new Set()
|
||
for (const run of String(text ?? '').match(/[\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff]+/g) ?? []) {
|
||
const chars = [...run]
|
||
// 4..2 as well as the whole run: the whole run is the most specific needle.
|
||
for (const width of [chars.length, 4, 3, 2]) {
|
||
if (width < 2 || width > chars.length) continue
|
||
for (let start = 0; start + width <= chars.length; start += 1) {
|
||
const needle = chars.slice(start, start + width).join('')
|
||
if (seen.has(needle)) continue
|
||
seen.add(needle)
|
||
longestFirst.push(needle)
|
||
}
|
||
}
|
||
}
|
||
return longestFirst
|
||
}
|
||
|
||
/**
|
||
* Every start offset of `needle` in `haystack` (non-overlapping scan).
|
||
* @param {string} haystack already lowercased
|
||
* @param {string} needle already lowercased
|
||
* @returns {number[]}
|
||
*/
|
||
function allIndexesOf(haystack, needle) {
|
||
if (needle === '') return []
|
||
const out = []
|
||
let from = 0
|
||
for (;;) {
|
||
const at = haystack.indexOf(needle, from)
|
||
if (at === -1) return out
|
||
out.push(at)
|
||
from = at + 1
|
||
}
|
||
}
|
||
|
||
/**
|
||
* Recall blocks for a query.
|
||
*
|
||
* Ordering contract (mirrors `RecallBlocksWithArbitration`,
|
||
* internal/memory/block_recall.go:229 — full scoring first, truncation last):
|
||
*
|
||
* 1. FTS5 candidates over a generous pool
|
||
* 2. exact-symbol boolean pin — pinned candidates first, ordered among
|
||
* themselves by lexical score
|
||
* 3. truncate to `topK`
|
||
*
|
||
* A LIKE scan runs only when FTS5 yields nothing, mirroring the "符号路兜底"
|
||
* shape. It is not the primary path: LIKE `%x%` matches almost everything on
|
||
* short queries and has no notion of relevance, which is exactly what the
|
||
* A5 mutation test asserts by breaking it.
|
||
*
|
||
* @param {import('./store.js').BlockStore} store
|
||
* @param {string} query
|
||
* @param {object} [options]
|
||
* @param {number} [options.topK] max hits returned (default 5)
|
||
* @param {number} [options.poolSize] FTS candidate pool before truncation (default 50)
|
||
* @param {Map<string, number>} [options.vectorScores] optional vector scores for V1 exact-band tie adjustment
|
||
* @param {number} [options.minScore] opt-in V5 fused route; 0 disables only the vector threshold,
|
||
* while absence preserves the phase-A path
|
||
* @returns {Array<{id: string, text: string, source: string, score: number,
|
||
* exact: boolean, why: string}>}
|
||
*/
|
||
export function recallBlocks(store, query, { topK = 5, poolSize = 50, fingerprint, dimension, vectorScores, minScore, skippedUnscoreable = 0 } = {}) {
|
||
const text = String(query ?? '').trim()
|
||
if (text === '') return []
|
||
|
||
const fusedCandidates = Number.isFinite(minScore)
|
||
const symbols = querySymbols(text)
|
||
const match = buildMatchExpression(text)
|
||
// Computed once: used both for fallback admission and for the CJK coverage
|
||
// signal below, so a longer matching window outranks a shorter one.
|
||
const cjkWindows = cjkNeedles(text)
|
||
|
||
/** @type {Map<string, {id: string, text: string, source: string, lexical: number}>} */
|
||
const pool = new Map()
|
||
if (match !== null) {
|
||
for (const hit of store.searchFts(match, poolSize)) {
|
||
pool.set(hit.id, { ...hit, lexical: hit.score })
|
||
}
|
||
}
|
||
// Containment fallback, only when FTS found nothing.
|
||
//
|
||
// Three distinct jobs, all legitimate:
|
||
// · a query with an exact token must still find its block even when FTS
|
||
// tokenization missed it (a number inside a CJK run, `billing0` glued to
|
||
// `billing0服务`);
|
||
// · a CJK query must find text that literally contains it, which unicode61
|
||
// cannot answer because it keeps a whole run as ONE token;
|
||
// · a CJK *question* (`值班室分机号是多少`) must find the note it is asking
|
||
// about (`值班室分机号 4324`). The runs differ, so whole-run containment
|
||
// misses — this is where HologramAgent uses jieba and we use sliding
|
||
// windows (`cjkWindows`, internal/memory/fusion.go:240) instead: every
|
||
// 2..4-char window of the query run becomes a needle.
|
||
//
|
||
// It is NOT the primary path: containment has no relevance notion and matches
|
||
// almost everything on short queries — the A5 mutation asserts exactly that by
|
||
// substituting it for FTS as the primary path.
|
||
if (pool.size === 0) {
|
||
const needles = symbols.map((symbol) => symbol.toLowerCase())
|
||
for (const run of cjkWindows) needles.push(run)
|
||
if (needles.length > 0) {
|
||
for (const block of store.allBlocks(poolSize)) {
|
||
const haystack = String(block.text_content ?? '').toLowerCase()
|
||
const matched = needles.filter((needle) => haystack.includes(needle))
|
||
if (matched.length === 0) continue
|
||
// Longer needles are more informative; rank containment hits by their
|
||
// best needle length so `分机号` beats a stray shared bigram.
|
||
const best = matched.reduce((longest, needle) => Math.max(longest, [...needle].length), 0)
|
||
pool.set(block.id, {
|
||
id: block.id,
|
||
text: block.text_content,
|
||
source: block.source,
|
||
// Carry the same identity fields the FTS path carries, or the space
|
||
// filter below would silently see `hasVector === undefined` and let
|
||
// every containment hit through — a filter that only covers one of
|
||
// two intake paths is the bug this whole group is about.
|
||
fingerprint: block.fingerprint ?? '',
|
||
hasVector: String(block.vector ?? '') !== '',
|
||
dimension: Number(block.vector_dim ?? 0),
|
||
lexical: Math.min(0.02 + best * 0.02, 0.5), // stays under the 1.0 pin band
|
||
})
|
||
}
|
||
}
|
||
}
|
||
|
||
// ported from go: internal/memory/fusion.go:374-408, :473-498.
|
||
// Opt-in fused candidate assembly; absent minScore preserves the frozen A2
|
||
// path and the V1/V2 scoring seam. Map membership is vector-route provenance,
|
||
// not the existence of attribution fields on a returned record.
|
||
// ported from go: internal/memory/fusion.go:466-468 and block_recall.go:263.
|
||
// Bound the VECTOR route before union with all symbol matches (fusion.go:473).
|
||
// This is not a cap on symbol candidates or on the eventual fused result.
|
||
const vectorCandidates = fusedCandidates && vectorScores instanceof Map
|
||
? new Set([...vectorScores].sort((a, b) => b[1] - a[1]).slice(0, largeTopK).map(([id]) => id))
|
||
: new Set()
|
||
if (fusedCandidates) {
|
||
for (const block of store.symbolCandidates()) {
|
||
const symbolMatch = symbols.some((symbol) =>
|
||
String(block.text_content ?? '').toLowerCase().includes(symbol.toLowerCase()))
|
||
const vectorSource = vectorCandidates.has(block.id)
|
||
if (!symbolMatch && !vectorSource) continue
|
||
if (pool.has(block.id)) continue
|
||
pool.set(block.id, {
|
||
id: block.id, text: block.text_content, source: block.source,
|
||
fingerprint: block.fingerprint ?? '', hasVector: block.has_vector === 1,
|
||
dimension: Number(block.vector_dim ?? 0), lexical: 0,
|
||
})
|
||
}
|
||
}
|
||
|
||
// ── F2: space filter, applied to the SYMBOL path too ────────────────────
|
||
//
|
||
// Text matching does not depend on the vector space, so a block written under
|
||
// an old space matches the query just as well as a current one. HA shipped
|
||
// without this filter and recorded the result as "一个真实的安全缺陷"
|
||
// (`internal/memory/fusion.go:481-495`): the vector path skipped mismatched
|
||
// blocks while the symbol path returned them, so blocks from a dead space
|
||
// polluted results after any provider change.
|
||
//
|
||
// ★ The same three-condition shape HA uses (`q.Fingerprint != "" &&
|
||
// b.Fingerprint != "" && b.Fingerprint != q.Fingerprint`) is deliberately
|
||
// NOT reproduced. Per the §0.2 ruling this project is fail-CLOSED on an
|
||
// unidentifiable block: a block that carries a vector but cannot name its
|
||
// space is not comparable to anything, so it is excluded rather than passed
|
||
// through. HA's fail-open arrives from the middle condition being false.
|
||
//
|
||
// Blocks with NO vector are unaffected: they never participate in the vector
|
||
// path either, and their text is still legitimately searchable.
|
||
//
|
||
// ★ `skipped` is counted HERE, at the single choke point both intake paths
|
||
// (FTS and containment) funnel through. Counting at the query site instead would
|
||
// cover only whichever path the author happened to be looking at — the same
|
||
// half-coverage that made HA's bug possible.
|
||
let skipped = 0
|
||
if (typeof fingerprint === 'string' && fingerprint !== '') {
|
||
for (const [id, candidate] of pool) {
|
||
// A vector plus a mismatched/absent space identity ⇒ not comparable.
|
||
if (candidate.hasVector === true) {
|
||
if (candidate.fingerprint === '' || candidate.fingerprint !== fingerprint) {
|
||
pool.delete(id)
|
||
skipped += 1
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
// ── F4b: dimension filter, counted SEPARATELY from the space filter ──────
|
||
//
|
||
// A block can be in the right space and still be unusable: same fingerprint,
|
||
// wrong width. Only this check catches it, and F4 requires the two counts kept
|
||
// apart because the remedies differ (fix provider config vs re-embed).
|
||
//
|
||
// Order matters: the space check runs first, so a block that is BOTH stale and
|
||
// narrow counts once, as a space skip — otherwise the two counters would
|
||
// double-count one row and neither would add up to the number of blocks dropped.
|
||
let skippedDim = 0
|
||
if (Number.isInteger(dimension) && dimension > 0) {
|
||
for (const [id, candidate] of pool) {
|
||
if (candidate.hasVector !== true) continue
|
||
// 0 = no stored width (pre-migration row); treat as unknown, not as wrong.
|
||
if (candidate.dimension === 0) continue
|
||
if (candidate.dimension !== dimension) {
|
||
pool.delete(id)
|
||
skippedDim += 1
|
||
}
|
||
}
|
||
}
|
||
|
||
const scored = []
|
||
// ── D2: every drop must be countable ─────────────────────────────────
|
||
//
|
||
// Three ways a candidate can disappear without becoming a hit, and until now
|
||
// only the first two (space / width, both upstream) were reported:
|
||
//
|
||
// · gatedByScore failed the V5 MinScore gate on its own vector score
|
||
// · gatedNoEvidence neither a usable vector score nor a symbol match
|
||
// · truncated survived every gate, fell outside the caller's topK
|
||
//
|
||
// Without these, "6 candidates, 1 returned" is indistinguishable from "the
|
||
// space only holds 1 block" — the operator sees skipped=0 and concludes
|
||
// nothing was dropped. Measured before this change: 6 blocks, minScore 0.99 →
|
||
// hits=0, skipped=0, skippedDim=0.
|
||
let gatedByScore = 0
|
||
let gatedNoEvidence = 0
|
||
for (const candidate of pool.values()) {
|
||
const haystack = candidate.text.toLowerCase()
|
||
// Port of `memory.SymbolScore` (internal/memory/fusion.go:284-308):
|
||
// exactHit counts symbols that are NUMERIC and appear verbatim;
|
||
// wordHit counts the rest.
|
||
// Only a numeric hit earns the boolean pin. A matched Latin word is a
|
||
// coverage signal, not an exact one — without that distinction any shared
|
||
// common word would outrank a real port number match.
|
||
let exactHit = 0
|
||
let wordHit = 0
|
||
const matched = []
|
||
for (const symbol of symbols) {
|
||
if (!haystack.includes(symbol.toLowerCase())) continue
|
||
matched.push(symbol)
|
||
if (isNumericSymbol(symbol)) exactHit += 1
|
||
else wordHit += 1
|
||
}
|
||
const exact = exactHit > 0
|
||
// 1.0 is a BOOLEAN pin, not a weight (fusion.go:427 and the note above it).
|
||
// Without any match, fall back to lexical rank, never above the pin band.
|
||
const coverage = symbols.length === 0 ? 0 : wordHit / symbols.length
|
||
|
||
// Specificity tie-break. When several candidates match the same word, prefer
|
||
// the one where that word is NOT merely a prefix of a longer word:
|
||
// query "billing 端口" must prefer `billing服务` over `billing2服务`.
|
||
// HologramAgent has the same problem and solves it in the splitter with a
|
||
// word-boundary check (`isCrossWordTailMatch`, distill/split.go:383); here
|
||
// the cheap equivalent is "the matched symbol is followed by a non-alnum
|
||
// character or a CJK run boundary".
|
||
let boundaryBonus = 0
|
||
for (const symbol of matched) {
|
||
for (const at of allIndexesOf(haystack, symbol.toLowerCase())) {
|
||
const next = haystack[at + symbol.length]
|
||
// End of text, a non-word char, or a CJK char (word ended cleanly).
|
||
if (next === undefined || !/[a-z0-9]/.test(next)) {
|
||
boundaryBonus += 1
|
||
break
|
||
}
|
||
}
|
||
}
|
||
|
||
// CJK coverage: the longest query window present in the block, normalised by
|
||
// the longest possible window. This is what makes `值班室分机号 4324` rank
|
||
// above a block that merely shares `分机号`, without a dictionary.
|
||
let cjkCoverage = 0
|
||
if (cjkWindows.length > 0) {
|
||
const longestPossible = Math.max(...cjkWindows.map((window) => [...window].length))
|
||
for (const window of cjkWindows) {
|
||
if (!haystack.includes(window.toLowerCase())) continue
|
||
const ratio = [...window].length / longestPossible
|
||
if (ratio > cjkCoverage) cjkCoverage = ratio
|
||
}
|
||
}
|
||
|
||
// Banding keeps the signals from swapping places across bands:
|
||
// [1.0+) exact numeric pin — must always outrank everything else
|
||
// [0..1) non-exact match — lexical / word / CJK coverage
|
||
const nonExact = Math.max(candidate.lexical, coverage * 0.5, cjkCoverage * 0.75)
|
||
// Ported from Go: internal/memory/fusion.go:425-428.
|
||
// ExactHit is a boolean pin; an explicitly supplied vector score only makes
|
||
// a same-band tie adjustment. Keep the old no-vector path byte-compatible
|
||
// for A2: it has no VectorHit to apply.
|
||
const exactScore = vectorScores instanceof Map
|
||
? 1.0 + (vectorScores.get(candidate.id) ?? 0) * 0.01
|
||
: 1.0 + candidate.lexical * 0.01 + boundaryBonus * 1e-4
|
||
// Ported from Go: internal/memory/fusion.go:61-66 and :430-434.
|
||
// Non-exact candidates use vector*0.7 + symbol*0.3. A missing vector score
|
||
// is the symbol-only fallback and must not invent a vector contribution.
|
||
const hasVectorScore = vectorScores instanceof Map && vectorScores.has(candidate.id)
|
||
&& (!fusedCandidates || vectorCandidates.has(candidate.id))
|
||
const vectorScore = hasVectorScore ? vectorScores.get(candidate.id) : 0
|
||
const score = exact
|
||
? exactScore
|
||
: fusedCandidates
|
||
// ported from go: internal/memory/fusion.go:430-434.
|
||
? vectorScore * 0.7 + coverage * 0.3
|
||
: vectorScores instanceof Map && hasVectorScore
|
||
? vectorScore * 0.7 + nonExact * 0.3
|
||
: nonExact + boundaryBonus * 1e-3
|
||
// ── V3: attribution ──────────────────────────────────────────────
|
||
// Ported from Go: internal/memory/fusion.go:34-46 (FusedHit's VectorHit /
|
||
// SymbolHit / Why) and :394-398 (the symbol branch that fills them).
|
||
//
|
||
// ★ Every value comes from the branch that already ran — nothing is a
|
||
// constant decided per shape. Go's own note (`:38-39`) is explicit that 0
|
||
// means "did not hit", so the two signals stay independently observable:
|
||
// a block can carry a vector score and no symbol match, or the reverse.
|
||
const vectorHit = vectorScore
|
||
const symbolHit = exact ? 1.0 : coverage
|
||
// ── V5: asymmetric MinScore gate ─────────────────────────────────
|
||
// Ported from Go: internal/memory/fusion.go:541-554.
|
||
//
|
||
// The threshold applies PER PATH, never to the fused score: the fused score
|
||
// is a mix of three different scales (exact pin ≥1.0, pure-symbol 0..0.3,
|
||
// pure-vector 0..0.7), so one threshold over it would either drop every
|
||
// symbol hit or let every vector hit through. Go records exactly that
|
||
// mistake and its two failing criteria (:528-529).
|
||
//
|
||
// exact hit → never gated (the pin is decisive)
|
||
// has vector → gated by MinScore against the VECTOR score
|
||
// symbol only → not gated by MinScore; dropped only when it matched
|
||
// nothing at all (go :549-553)
|
||
//
|
||
// ★ The branch reads `hasVectorScore`, the CANDIDATE-SOURCE flag, not the
|
||
// presence of the `vectorHit` field. V3 now puts that field on every record,
|
||
// so existence-based branching would route symbol-only hits into the vector
|
||
// gate and the "asymmetric" property would silently disappear.
|
||
if (fusedCandidates) {
|
||
if (!exact) {
|
||
if (hasVectorScore && vectorScore > 0) {
|
||
if (minScore > 0 && vectorScore < minScore) {
|
||
gatedByScore += 1
|
||
continue
|
||
}
|
||
} else if (!(symbolHit > 0)) {
|
||
gatedNoEvidence += 1
|
||
continue
|
||
}
|
||
}
|
||
}
|
||
|
||
scored.push({
|
||
id: candidate.id,
|
||
text: candidate.text,
|
||
source: candidate.source,
|
||
score,
|
||
exact,
|
||
vectorHit,
|
||
symbolHit,
|
||
why: vectorScores instanceof Map
|
||
? `vector:${vectorHit};symbol:${symbolHit};${exact ? 'exact' : matched.length > 0 ? `word:${matched.join(',')}` : 'lexical'}`
|
||
: exact ? `exact:${matched.filter(isNumericSymbol).join(',')}` : matched.length > 0 ? `word:${matched.join(',')}` : 'lexical',
|
||
})
|
||
}
|
||
|
||
scored.sort((a, b) => (b.score === a.score ? a.id.localeCompare(b.id) : b.score - a.score))
|
||
// ported from go: internal/memory/fusion.go:557-559 — final caller topK,
|
||
// after candidate union, scoring, filtering and the asymmetric gate.
|
||
const drops = {
|
||
gatedByScore, gatedNoEvidence, skippedUnscoreable,
|
||
truncated: Math.max(0, scored.length - topK),
|
||
}
|
||
return withSkipped(scored.slice(0, topK), skipped, skippedDim, drops)
|
||
}
|
||
|
||
/**
|
||
* Attach a non-enumerable `skipped` count to a recall result.
|
||
*
|
||
* Non-enumerable for the same reason as `degraded`: the phase-A golden fixture
|
||
* compares result arrays, and an enumerable field would change every stored
|
||
* expectation. Callers that want the number read `hits.skipped` explicitly, which
|
||
* also makes it obvious at the call site that they are consuming an out-of-band
|
||
* signal rather than a hit.
|
||
*
|
||
* @param {Array<object>} hits
|
||
* @param {number} skipped blocks excluded by the space filter
|
||
* @param {number} [skippedDim] blocks excluded because their width did not match
|
||
* @param {{gatedByScore?: number, gatedNoEvidence?: number, truncated?: number}} [drops]
|
||
* the three downstream ways a candidate can disappear. Separate numbers on
|
||
* purpose: "failed the threshold" and "had nothing to score" have opposite
|
||
* remedies (lower the threshold vs. re-embed), and a merged count could not be
|
||
* acted on. `truncated` is the caller's own topK, which is not a defect — it is
|
||
* reported so that `hits + all counter fields` reconciles with the candidate
|
||
* pool.
|
||
* @returns {Array<object> & {skipped: number}}
|
||
*/
|
||
function withSkipped(hits, skipped, skippedDim = 0, drops = {}) {
|
||
Object.defineProperty(hits, 'skipped', {
|
||
value: skipped,
|
||
enumerable: false,
|
||
configurable: true,
|
||
})
|
||
// Reported separately (F4): a dimension skip and a space skip have different
|
||
// remedies, so one merged number could not be acted on.
|
||
Object.defineProperty(hits, 'skippedDim', {
|
||
value: skippedDim,
|
||
enumerable: false,
|
||
configurable: true,
|
||
})
|
||
// ── D2: the three downstream drop counts ─────────────────────────────
|
||
// Also non-enumerable, for the same reason as `skipped`: the phase-A golden
|
||
// fixture compares result arrays.
|
||
for (const [key, label] of [
|
||
['gatedByScore', 'skippedScore'],
|
||
['gatedNoEvidence', 'skippedNoEvidence'],
|
||
['truncated', 'truncated'],
|
||
// D2's second hole: blocks dropped BEFORE the candidate pool existed. They
|
||
// are not "gated" (no gate ever saw them) — they could not be scored at all.
|
||
['skippedUnscoreable', 'skippedUnscoreable'],
|
||
]) {
|
||
Object.defineProperty(hits, label, {
|
||
value: Number.isInteger(drops[key]) ? drops[key] : 0,
|
||
enumerable: false,
|
||
configurable: true,
|
||
})
|
||
}
|
||
return hits
|
||
}
|
||
|
||
/**
|
||
* Attach a non-enumerable diagnosis to a result array.
|
||
*
|
||
* Used instead of widening the return type: `recallBlocks` must stay
|
||
* byte-compatible with phase A (group A), and every existing caller does
|
||
* `hits.length` / `hits[0]` on a plain array. A non-enumerable property rides
|
||
* along without appearing in `JSON.stringify`, `Object.keys`, or deep equality
|
||
* against phase A output.
|
||
*
|
||
* @param {Array<object>} hits
|
||
* @param {string|null} mode one of `no_embedder` / `embed_error` / `aborted` /
|
||
* `no_vector_space` / `vector_ready_unused` / null
|
||
* @param {Error} [error]
|
||
* @returns {Array<object>} the same array
|
||
*/
|
||
function withDiagnosis(hits, mode, error) {
|
||
Object.defineProperty(hits, 'degraded', {
|
||
value: mode,
|
||
enumerable: false,
|
||
configurable: true,
|
||
})
|
||
if (error !== undefined) {
|
||
Object.defineProperty(hits, 'degradedError', {
|
||
value: error instanceof Error ? error.message : String(error),
|
||
enumerable: false,
|
||
configurable: true,
|
||
})
|
||
}
|
||
return hits
|
||
}
|
||
|
||
/**
|
||
* The only vector capability this plugin depends on.
|
||
*
|
||
* Deliberately three members wide and provider-free: no endpoint, no model name,
|
||
* no transport. Phase B decides *whether* vectors participate; it does not
|
||
* decide *which* provider supplies them. Swapping the HTTP adapter for a local
|
||
* ONNX one must not touch a single criterion.
|
||
*
|
||
* @typedef {object} Embedder
|
||
* @property {(text: string, signal?: AbortSignal) => Promise<number[]>} embedText
|
||
* query-side encoding. `signal` fuses the caller's cancellation with the
|
||
* adapter's own timeout; an implementation that ignores it silently makes a
|
||
* cancelled turn block until its own bound (the W9 bug).
|
||
* @property {() => { fingerprint: string, dimension: number }} info
|
||
* synchronous: the fingerprint/dimension must be checkable *before* paying for
|
||
* a network round-trip, so dimension mismatch can fail fast
|
||
* @property {() => Promise<void>} [close]
|
||
*/
|
||
|
||
/**
|
||
* Vectors-assisted recall.
|
||
*
|
||
* With no embedder (or a failing one) this is exactly `recallBlocks`. With a
|
||
* healthy one it scores the query vector against the space's stored vectors and
|
||
* hands `recallBlocks` BOTH `vectorScores` and `minScore`, which together open
|
||
* the fused candidate assembly, the V1 exact-band tie adjustment, the V2 weighted
|
||
* blend and the V5 asymmetric gate.
|
||
*
|
||
* ★ Why one call and not two: the symbol and vector routes are fused into ONE
|
||
* scoring pass so that a block the symbol path missed can still be admitted by
|
||
* its vector score (V5) and so that the candidate pool is never truncated before
|
||
* fusion (V7). Calling `recallBlocks` twice and merging afterwards cannot
|
||
* reproduce either property.
|
||
*
|
||
* Degradation is reported on the returned array (`arr.degraded`), never by
|
||
* throwing — the recall main path must survive a dead embedding service
|
||
* (decision §0.4).
|
||
*
|
||
* @param {import('./store.js').BlockStore} store
|
||
* @param {Embedder | null | undefined} embedder
|
||
* @param {string} query
|
||
* @param {object} [options]
|
||
* @param {number} [options.topK]
|
||
* @param {number} [options.poolSize]
|
||
* @param {AbortSignal} [options.signal]
|
||
* @returns {Promise<Array<object> & {degraded?: string|null}>}
|
||
*/
|
||
export async function recallBlocksHybrid(store, embedder, query, options = {}) {
|
||
// ★ The active space is resolved ONCE and injected into `options` before any
|
||
// branch can run. Every one of the six degradation paths below funnels into
|
||
// `recallBlocks(store, query, options)`, so injecting it here means the space
|
||
// filter applies on ALL of them — the symbol path included. Setting it at each
|
||
// call site would leave exactly one path unfiltered the first time somebody adds
|
||
// a seventh, which is how HA's symbol path ended up unfiltered
|
||
// (`fusion.go:481-495`, recorded there as "一个真实的安全缺陷").
|
||
//
|
||
// When the embedder cannot name its space, `fingerprint` stays undefined and the
|
||
// filter is inert — recall degrades to its pre-F2 behaviour rather than
|
||
// silently excluding every vectorised block.
|
||
const space = embedder && typeof embedder.info === 'function' ? embedder.info() : undefined
|
||
const scoped = {
|
||
...options,
|
||
fingerprint: typeof space?.fingerprint === 'string' && space.fingerprint !== ''
|
||
? space.fingerprint
|
||
: undefined,
|
||
// The declared width travels with the space so the recall path can apply the
|
||
// F4b dimension filter — without it, `skipped_dim` could only ever be 0.
|
||
dimension: typeof space?.dimension === 'number' && space.dimension > 0
|
||
? space.dimension
|
||
: undefined,
|
||
}
|
||
|
||
if (embedder === null || embedder === undefined || typeof embedder.embedText !== 'function') {
|
||
return withDiagnosis(recallBlocks(store, query, scoped), 'no_embedder')
|
||
}
|
||
|
||
const info = space
|
||
if (info === undefined || typeof info.fingerprint !== 'string' || info.fingerprint === '') {
|
||
// An embedder that cannot name its own space cannot produce comparable
|
||
// vectors — treat it as absent rather than silently comparing across spaces.
|
||
return withDiagnosis(recallBlocks(store, query, scoped), 'no_embedder')
|
||
}
|
||
|
||
let queryVector
|
||
try {
|
||
queryVector = await embedder.embedText(String(query ?? ''), options.signal)
|
||
} catch (error) {
|
||
// Decision §0.4: never let the vector side take the whole recall down.
|
||
//
|
||
// ★ Cancellation and failure are different facts and must not share a label.
|
||
// `aborted` means "we stopped on purpose", so the model must not be told the
|
||
// search was incomplete in a way that invites a retry; `embed_error` means
|
||
// "the provider failed", where absence of evidence really is suspicious.
|
||
// Collapsing them made the notice say "search FAILED" for a deliberate
|
||
// cancellation — misleading in exactly the case where the user is watching.
|
||
const aborted =
|
||
options.signal?.aborted === true ||
|
||
(error instanceof Error && (error.name === 'AbortError' || /abort/i.test(error.message)))
|
||
return withDiagnosis(recallBlocks(store, query, scoped), aborted ? 'aborted' : 'embed_error', error)
|
||
}
|
||
|
||
if (!Array.isArray(queryVector) || queryVector.length === 0) {
|
||
return withDiagnosis(recallBlocks(store, query, scoped), 'embed_error',
|
||
new Error(`embedder returned ${Array.isArray(queryVector) ? 'an empty vector' : 'a non-array'}`))
|
||
}
|
||
if (Number.isInteger(info.dimension) && queryVector.length !== info.dimension) {
|
||
return withDiagnosis(recallBlocks(store, query, scoped), 'embed_error',
|
||
new Error(`embedder returned dim ${queryVector.length}, declared ${info.dimension}`))
|
||
}
|
||
|
||
// ── the wiring ────────────────────────────────────────────────────────────
|
||
//
|
||
// This is the line P1 exists for. Until now the query vector was computed,
|
||
// dimension-checked, and then DISCARDED: the call below passed `scoped`, which
|
||
// carries `topK` / `signal` / `fingerprint` / `dimension` and nothing else. So
|
||
// every V1–V7 scoring branch in `recallBlocks` was reachable only from unit
|
||
// tests that hand-built a `vectorScores` Map; production recall was pure
|
||
// symbol/FTS.
|
||
//
|
||
// ★ BOTH options have to be passed, and the reason is structural rather than
|
||
// stylistic:
|
||
// · `vectorScores` alone reaches only the V1 exact-band tie adjustment
|
||
// (`:438`), because the fused candidate assembly at `:233` is gated on
|
||
// `Number.isFinite(minScore)`; without it the vector route can never
|
||
// admit a block the symbol path missed, which is exactly what V5 needs.
|
||
// · `minScore` alone does nothing at all, because `:299` additionally
|
||
// requires `vectorScores` to BE a Map before it builds vector candidates.
|
||
// The two are one seam split across three guards, so they travel together.
|
||
//
|
||
// ★ Scheme A (score the existing space) over scheme B (teach `recallBlocks` to
|
||
// accept a query vector). B would change a signature that phase A's golden
|
||
// fixture and nine call sites depend on being synchronous and vector-free, and
|
||
// it would move cosine arithmetic into a function whose contract is "store +
|
||
// query text". A leaves `recallBlocks` completely untouched: the seam it
|
||
// already exposes is the seam we drive.
|
||
//
|
||
// Cost, measured on this box before choosing (2000 blocks x 512 dims, the
|
||
// profile's default space width): narrow `SELECT id, vector` 21 ms + parse and
|
||
// cosine 60 ms ≈ 81 ms per recall; at 8000 blocks ≈ 310 ms. That is affordable
|
||
// because the scan is bounded by the space and width filters below and runs
|
||
// once per TURN (pre-step), not per candidate. Two design choices keep it from
|
||
// growing worse than linear:
|
||
// · the space/width filters are pushed into SQL, so blocks from a retired
|
||
// space or a wrong width are never decoded;
|
||
// · a space holding no comparable vectors scores nothing at all, so a database
|
||
// with no embedded blocks (every phase-A store) pays one indexed SELECT that
|
||
// returns no rows and takes the symbol path untouched.
|
||
// The remaining risk is a very large single space, which is a ceiling to
|
||
// measure before raising `topK`/space size — not a reason to ship a vector path
|
||
// that never runs. Note the fail-closed consequence (§0.2/F5): blocks whose
|
||
// fingerprint is empty are NOT scored, by design.
|
||
//
|
||
// ★ This does not introduce a NEW order of cost. `recallBlocks` already walks
|
||
// every block for symbol union when `minScore` is finite (the `fusedCandidates`
|
||
// step, `allBlocks(-1)`), so the fused path was always O(total blocks); what is
|
||
// added here is one narrow indexed read plus the cosine arithmetic, and the
|
||
// decode is bounded to the active space and width instead of the whole table.
|
||
// ── D2: blocks that could not be scored at all ──────────────────────────
|
||
//
|
||
// `scoreVectorCandidates` skips a row when it cannot produce a number: bad
|
||
// JSON, wrong width, a non-finite component, a zero-norm vector, or — since C4
|
||
// — a vector that centring collapses to dust. Such a block never enters
|
||
// `vectorScores`, so `recallBlocks` never puts it in the candidate pool, so
|
||
// none of the downstream counters can see it.
|
||
//
|
||
// Measured: three identical blocks with centring active produced
|
||
// `hits=0 skipped=0 skippedDim=0 skippedScore=0 skippedNoEvidence=0 truncated=0`
|
||
// — the whole space vanished while every counter read zero. That is exactly the
|
||
// silent branch D2 forbids, one stage EARLIER than where D2 first looked.
|
||
//
|
||
// The counter is threaded by reference so it can be reported on both the fused
|
||
// path and the early `no_vector_space` return.
|
||
const unscoreable = { count: 0 }
|
||
|
||
const vectorScores = scoreVectorCandidates(store, queryVector, scoped, unscoreable)
|
||
const vectorRoute = vectorScores.size > 0
|
||
if (!vectorRoute) {
|
||
// An empty map would still open the fused candidate assembly (minScore is
|
||
// finite) and then admit symbol matches with a vector weight of 0 — the
|
||
// `0.7*0 + 0.3*coverage` regression V2 forbids. Reporting the honest reason
|
||
// and taking the untouched symbol path is both correct and cheaper.
|
||
return withDiagnosis(
|
||
recallBlocks(store, query, { ...scoped, skippedUnscoreable: unscoreable.count }),
|
||
'no_vector_space',
|
||
)
|
||
}
|
||
|
||
// `minScore` is only defaulted when the caller did not decide: an explicit
|
||
// `0` is a deliberate "no threshold" and must survive (it is how the V7 and
|
||
// pool-truncation criteria disable the gate).
|
||
const minScore = Number.isFinite(scoped.minScore) ? scoped.minScore : DEFAULT_MIN_SCORE
|
||
|
||
return withDiagnosis(
|
||
recallBlocks(store, query, { ...scoped, vectorScores, minScore, skippedUnscoreable: unscoreable.count }),
|
||
// 'hybrid' from here on is a statement of fact, not an aspiration: the vector
|
||
// route contributed at least one scoreable candidate to the same call that
|
||
// produced these hits. Group A shipped `vector_ready_unused` precisely so this
|
||
// rename could not happen before the wiring existed.
|
||
'hybrid',
|
||
)
|
||
}
|
||
|
||
/**
|
||
* Cosine similarity of the query vector against every block in its space.
|
||
*
|
||
* Returns the `Map<id, score>` that `recallBlocks` consumes. An empty Map means
|
||
* "this space has nothing to score", which the caller must treat as the symbol
|
||
* path rather than as a vector route with score 0 everywhere.
|
||
*
|
||
* Vectors are stored as JSON text (`memory_blocks.vector`, per the DDL comment:
|
||
* "vector is a JSON array, matching HA's memory_blocks.vector"). A row that fails
|
||
* to parse is SKIPPED rather than thrown on: one corrupt payload must not take
|
||
* the whole recall down (decision §0.4), and the migration path already records
|
||
* corrupt widths as a negative `vector_dim` so they are filterable.
|
||
*
|
||
* ★ Centering lives at the BOTTOM of this function (see the C4 block by the
|
||
* scoring loop): the query is centred once, and each block vector is centred by
|
||
* the same mean inside the loop. Nothing above this line knows about it.
|
||
*
|
||
* ★ Not L2-normalised on the way out. Cosine already divides by both norms, and
|
||
* `putBlock` does not promise unit vectors, so normalising would change nothing
|
||
* except to hide a zero vector behind a NaN.
|
||
*
|
||
* @param {import('./store.js').BlockStore} store
|
||
* @param {number[]} queryVector already dimension-checked by the caller
|
||
* @param {{fingerprint?: string, dimension?: number}} options
|
||
* @returns {Map<string, number>}
|
||
*/
|
||
function scoreVectorCandidates(store, queryVector, { fingerprint, dimension } = {}, unscoreable = { count: 0 }) {
|
||
const scores = new Map()
|
||
// Without a space identity there is nothing to compare against: the vectors in
|
||
// the table may belong to several incompatible spaces (§0.1), and a cosine
|
||
// across two of them is a number without a meaning.
|
||
if (typeof fingerprint !== 'string' || fingerprint === '') return scores
|
||
|
||
const rows = store.vectorCandidates(fingerprint, dimension)
|
||
if (rows.length === 0) return scores
|
||
|
||
// ── C4 · 中心化校正 ────────────────────────────────────────────────────
|
||
//
|
||
// 一次索引查询取本空间的中心，仅当它被判定为各向异性、且宽度与查询一致
|
||
// 时才启用。维度不符时**不校正也不报错**：按错误维度截断或补齐造出来的
|
||
// 余弦，看起来像分数，实际没有意义。
|
||
//
|
||
// 门控条件照 C2：`isAnomalous === false` 的空间不校正。这不是优化开关。
|
||
// 各向同性空间的均值范数本来就低，减掉它只会放大噪声 —— 实测 8 块正交
|
||
// （mean norm 0.3536）在无条件校正下命中数 8 → 1，且因为 V5 的结构性门
|
||
// （vectorScore <= 0 且 symbolHit === 0 直接丢弃）无法靠 minScore 调参救回。
|
||
// 所以这条 if 是正确性的一部分，不是性能的一部分。
|
||
//
|
||
// ★ loadCentroid 是纯读的一次索引查询，不进写路径、不触发重建。
|
||
// 重建挂在写入侧（lib/index.js 的 commit 之后），因为它是 O(n·d)，
|
||
// 放进每轮 pre-step 就是灾难；C3-k 也明令计数查询不得进热路径。
|
||
const centroid = store.loadCentroid(fingerprint)
|
||
// ★ C4's order: a round that judges the centroid STALE still returns results,
|
||
// but with UNCORRECTED vectors. The verdict is read from the row (`stats.stale`),
|
||
// written by the write path — the recall side must not run `centroidStatus`,
|
||
// whose first line is a COUNT (C3-k).
|
||
//
|
||
// Why, stated honestly: a mean built from a fraction of the space describes a
|
||
// space that no longer exists, and applying it is unjustifiable on principle.
|
||
// An attempt to measure the damage could not separate the cases: with a
|
||
// near-duplicate late corpus, scores saturate near 0.99 whether the mean is
|
||
// stale (0.997) or absent (0.994); only a freshly rebuilt mean spread them
|
||
// (0.978..0.970). So this is a safety choice; the measurable win belongs to
|
||
// keeping the centroid FRESH, not to skipping it.
|
||
// ★ `!centroid.stats?.stale` and NOT `centroid.stats?.stale !== true`:
|
||
// the flag is written as the JSON number 1 by `markCentroidStale`, and in JS
|
||
// `1 !== true` — a strict comparison across types is always true, so the strict
|
||
// form silently made this guard a no-op. Caught by a fixture where centring is
|
||
// observable; the suite was 305/305 green with the bug in place.
|
||
const mean = centroid?.isAnomalous === true
|
||
&& centroid.dim === queryVector.length
|
||
&& !centroid.stats?.stale
|
||
? centroid.mean
|
||
: null
|
||
const centeredQuery = mean === null ? null : centerVector(queryVector, mean)
|
||
// `mean !== null` 但 `centeredQuery === null` 表示减完是零向量（没有方向），
|
||
// 同样退回未校正路径，而不是拿零向量去算余弦。
|
||
const centering = centeredQuery !== null
|
||
|
||
const scoringQuery = centering ? centeredQuery : queryVector
|
||
let queryNorm = 0
|
||
for (const value of scoringQuery) queryNorm += value * value
|
||
queryNorm = Math.sqrt(queryNorm)
|
||
if (!(queryNorm > 0)) return scores
|
||
|
||
for (const row of rows) {
|
||
let vector
|
||
try {
|
||
vector = JSON.parse(String(row.vector ?? ''))
|
||
} catch {
|
||
unscoreable.count += 1
|
||
continue
|
||
}
|
||
if (!Array.isArray(vector) || vector.length !== queryVector.length) {
|
||
unscoreable.count += 1
|
||
continue
|
||
}
|
||
|
||
// ★ 两边都中心化。只处理一边 = 一边带公共分量、另一边不带，算出来的
|
||
// 余弦是两种量纲的混合（HA centroid_store.go:439-442 的原话）。
|
||
// 实测：只减 query 时，C4b 那个 5 块 fixture 上同值对的余弦从 1.0 掉到
|
||
// 0.1897（余弦对称，只减块侧同值）—— 那就不是在量夹角了。
|
||
const scoringVector = centering ? centerVector(vector, mean) : vector
|
||
if (scoringVector === null) {
|
||
// ★ C4's dust guard fired: the block sits on the space mean, so it has no
|
||
// direction to compare against. Counted explicitly, because otherwise this
|
||
// block disappears BEFORE the candidate pool is assembled and no downstream
|
||
// counter can see it — measured as "the whole space vanished while every
|
||
// counter read zero".
|
||
unscoreable.count += 1
|
||
continue
|
||
}
|
||
|
||
let dot = 0
|
||
let norm = 0
|
||
for (let index = 0; index < scoringVector.length; index += 1) {
|
||
const value = Number(scoringVector[index])
|
||
// A non-numeric element makes the whole score meaningless. Skip the row
|
||
// instead of letting NaN propagate into the sort, where it would silently
|
||
// randomise the order (NaN comparisons are always false).
|
||
if (!Number.isFinite(value)) {
|
||
dot = Number.NaN
|
||
break
|
||
}
|
||
dot += value * scoringQuery[index]
|
||
norm += value * value
|
||
}
|
||
if (!Number.isFinite(dot) || !(norm > 0)) {
|
||
unscoreable.count += 1
|
||
continue
|
||
}
|
||
scores.set(row.id, dot / (queryNorm * Math.sqrt(norm)))
|
||
}
|
||
return scores
|
||
}
|
||
|
||
/**
|
||
* 减均值再 L2 归一化（HA `Centroid.CenterVector`，`centroid.go:185-188`）。
|
||
*
|
||
* ★ 归一化不能省。减去均值会改变模长，不归一化就是拿两把不同的尺子量。
|
||
* 实测：同一对相同向量，两侧都归一化时余弦恒为 1；只减不归一时它会变成 0.61
|
||
* —— 看起来仍是个「分数」，但已经不是在算夹角了。
|
||
*
|
||
* ★ 零向量返回 `null`。减完均值恰好没有方向（向量就躺在中心上）时，
|
||
* 余弦无定义。调用方必须跳过这一行，而不是拿零向量去算出一个 0 分
|
||
* 再让它参与排序 —— 0 分在 V5 的门里等于「未命中」，会把一次真实的
|
||
* 无法比较伪装成一次「不相关」。
|
||
*
|
||
* @param {number[]} vector
|
||
* @param {number[]} mean the space centroid mean, same length
|
||
* @returns {number[]|null} centered unit vector, or null when it has no direction
|
||
*/
|
||
function centerVector(vector, mean) {
|
||
const out = new Array(vector.length)
|
||
let norm = 0
|
||
for (let index = 0; index < vector.length; index += 1) {
|
||
const delta = Number(vector[index]) - mean[index]
|
||
// A non-numeric component poisons the whole direction: bail out rather than
|
||
// let NaN reach the sort (NaN comparisons are always false ⇒ silent shuffle).
|
||
if (!Number.isFinite(delta)) return null
|
||
out[index] = delta
|
||
norm += delta * delta
|
||
}
|
||
norm = Math.sqrt(norm)
|
||
// ★ Not `norm > 0`.
|
||
//
|
||
// A block that IS the mean differs from it only by floating-point dust
|
||
// (~1e-17). `norm > 0` accepts that dust, and the renormalisation below turns
|
||
// it into a unit vector pointing in an arbitrary direction — which then scores
|
||
// against the query as if it were signal. Measured on a space of two identical
|
||
// blocks: cosine 0.2183 out of dust, where the raw cosine was 0.8608.
|
||
//
|
||
// Treating dust as "no direction" also matches the Go side, where the cosine of
|
||
// a zero vector is 0 rather than a random angle.
|
||
if (!(norm > CENTER_EPSILON)) return null
|
||
for (let index = 0; index < out.length; index += 1) out[index] /= norm
|
||
return out
|
||
}
|
||
|
||
/**
|
||
* Render recall hits as the model-facing text injected into the turn.
|
||
*
|
||
* Shape follows HologramAgent's `crossModalMarkdown`
|
||
* (internal/agent/core/crossmodal.go:256): a header that says what the content
|
||
* IS, then one line per hit. The header matters — the measured failure mode in
|
||
* `docs/zh/memory-usability-plan.md` was that the model ignored injected memory
|
||
* because nothing told it these were *historical* fragments rather than the
|
||
* current conversation.
|
||
*
|
||
* ★ A degraded recall renders a notice line. `hits.degraded` is non-enumerable
|
||
* (so it cannot disturb the phase-A golden comparison), which means it is
|
||
* invisible to JSON.stringify, Object.keys, and every serialised log. Asserting
|
||
* it in a test therefore proves nothing about what the MODEL sees — this text is
|
||
* the only channel by which the model learns that recall was partial.
|
||
*
|
||
* @param {Array<{text: string, exact: boolean, score: number, degraded?: string}>} hits
|
||
* @param {string} [sourceLabel]
|
||
* @returns {string} rendered block, or '' when there is nothing to say
|
||
*/
|
||
export function renderRecall(hits, sourceLabel = 'stablemem') {
|
||
if (!Array.isArray(hits)) return ''
|
||
|
||
const lines = hits.map((hit, index) => {
|
||
const tag = hit.exact ? 'exact' : 'text'
|
||
const body = String(hit.text).replace(/\s+/g, ' ').trim().slice(0, 200)
|
||
return `[${index + 1}] (${tag} ${hit.score.toFixed(2)}) ${body}`
|
||
})
|
||
const notice = degradationNotice(hits)
|
||
|
||
// ★ "no entries" and "nothing to say" are different questions.
|
||
//
|
||
// A degraded recall with zero lexical hits still owes the model an explanation:
|
||
// otherwise silence means "memory has nothing on this", when the truth is
|
||
// "memory could not be searched fully". Staying silent there is the false
|
||
// negative the notice exists to prevent.
|
||
//
|
||
// The converse must hold too: a HEALTHY empty recall stays silent. Emitting the
|
||
// header on every empty turn would burn tokens on the commonest case and train
|
||
// the model to ignore the memory section entirely.
|
||
if (lines.length === 0 && notice === '') return ''
|
||
|
||
const header =
|
||
`[Memory · ${sourceLabel}] Recalled from durable block memory — these are PAST notes, not the current conversation.`
|
||
return [header, ...(notice === '' ? [] : [notice]), ...lines].join('\n')
|
||
}
|
||
|
||
/**
|
||
* Human-readable text for a degraded recall, or '' when nothing was degraded.
|
||
*
|
||
* Wording matters: the model must be able to tell "memory has nothing on this"
|
||
* apart from "memory could not be searched fully", because in the second case
|
||
* absence of evidence is not evidence of absence.
|
||
*
|
||
* @param {Array<object> & {degraded?: string|null}} hits
|
||
* @returns {string}
|
||
*/
|
||
export function degradationNotice(hits) {
|
||
const mode = hits?.degraded
|
||
if (mode === undefined || mode === null) return ''
|
||
// One question, asked once: are there entries below? Every branch shares it.
|
||
// Wording contract (pinned by W11f, table-driven): nothing === true => no branch
|
||
// may say "matches below"; nothing === false => the branch must point at them.
|
||
const nothing = !Array.isArray(hits) || hits.length === 0
|
||
switch (mode) {
|
||
case 'hybrid':
|
||
// NOT a degradation: this is the healthy, fully-wired path (symbol + vector
|
||
// fused in one scoring pass). It carries no notice for the same reason `ok`
|
||
// would not — there is nothing the model needs to know.
|
||
//
|
||
// ★ Kept as an explicit case rather than relying on the `default:` branch.
|
||
// `default:` exists to make an UNKNOWN mode fail loudly-safe ("unknown, not
|
||
// absent"), so routing the healthiest mode through it would print a false
|
||
// degradation notice on every single turn that had a vector hit.
|
||
return ''
|
||
case 'no_embedder':
|
||
// Permanent configuration state, not a per-turn failure. The constant index
|
||
// (systemPrompt.context -> renderIndex) already tells the model that block
|
||
// memory exists and is searched lexically. Repeating it on every zero-hit
|
||
// turn costs ~213 chars on the COMMONEST case — measured 10/10 ordinary
|
||
// turns under the shipping default (no embedder) — and trains the model to
|
||
// skip the memory section, which is exactly what W11 exists to prevent for
|
||
// real degradations. W11e pins the silence.
|
||
return ''
|
||
case 'embed_error': {
|
||
// Two wordings, because "matches below are …" is a lie when nothing was
|
||
// found. The distinction matters: with zero entries the model must know the
|
||
// search was incomplete AND empty, not merely incomplete.
|
||
return nothing
|
||
? '[Memory] Semantic search FAILED for this turn and the lexical fallback found nothing. Absence of a memory here does not prove it is unrecorded.'
|
||
: '[Memory] Semantic search FAILED for this turn; matches below are lexical/symbolic only. Absence of a memory here does not prove it is unrecorded.'
|
||
}
|
||
case 'aborted':
|
||
// Deliberate stop, not a fault: say so plainly and do NOT imply the memory
|
||
// is incomplete, which would invite a pointless retry. Same two-wording rule
|
||
// as embed_error — a cancelled turn that ALSO found nothing is the commonest
|
||
// shape of an abort (the user stops, and their query matches nothing), so
|
||
// "matches below are …" is a lie in exactly the case that happens most.
|
||
return nothing
|
||
? '[Memory] Semantic search was cancelled for this turn and the lexical fallback found nothing.'
|
||
: '[Memory] Semantic search was cancelled for this turn; matches below are lexical/symbolic only.'
|
||
case 'vector_ready_unused':
|
||
// Group A state: the vector was computed but does not take part in scoring
|
||
// yet. Not something the model needs to hear about — but keeping the case
|
||
// explicit means a future mode cannot silently fall through to ''.
|
||
return ''
|
||
case 'no_vector_space':
|
||
// A space IS configured and the embedder answered, but no block in that
|
||
// space carries a usable vector (a fresh space, or every row pre-dates the
|
||
// embedder). Unlike `no_embedder` this is a transitional DATA state, and it
|
||
// is the one an operator most often mistakes for "semantic recall is on".
|
||
// Still silent for the model: the fallback it received is the correct one,
|
||
// and `stablemem_stats` is where the operator is told why.
|
||
return ''
|
||
default:
|
||
// A mode nobody anticipated. "Not found" must not read as "not recorded":
|
||
// this is the ONLY path a future mode can take, so it is the worst possible
|
||
// place to start lying. Carries the same obligation as embed_error.
|
||
return nothing
|
||
? `[Memory] Recall degraded (${mode}); nothing was found, so treat this as "unknown", not "absent".`
|
||
: `[Memory] Recall degraded (${mode}); matches below may be incomplete.`
|
||
}
|
||
}
|
||
|
||
/** Fixed text heuristic matching dsh-token-meter: 4 chars/token plus framing. */
|
||
export function estimateMemoryTokens(text) {
|
||
return Math.ceil(String(text ?? "").length / 4) + 8
|
||
}
|
||
|
||
/** Estimate rendered hit cost from its note body, not serialized JSON. */
|
||
export function estimateRecallHitTokens(hit) {
|
||
const body = String(hit?.text ?? "").replace(/\s+/g, " ").trim().slice(0, 200)
|
||
return estimateMemoryTokens(body) + 20
|
||
}
|
||
|
||
/** Select a recall count under explicit, host-derived, or fallback budget. */
|
||
export function segmentRecallBudget(hits, options = {}) {
|
||
const topK = Number.isFinite(options.topK) ? Math.max(1, Math.floor(options.topK)) : 3
|
||
const explicit = Number.isFinite(options.budgetTokens) && options.budgetTokens > 0 ? options.budgetTokens : undefined
|
||
const window = Number.isFinite(options.contextWindow) && options.contextWindow > 0 ? options.contextWindow : undefined
|
||
const source = explicit !== undefined ? "explicit" : window !== undefined ? "host" : "fallback"
|
||
const budget = explicit ?? (window === undefined ? 4096 : Math.max(1, Math.floor(window * 0.8)))
|
||
const rows = Array.isArray(hits) ? hits : []
|
||
if (rows.length === 0) return { hits: rows, budgetTokens: budget, source, singleHitOverBudget: false }
|
||
const costs = rows.map(estimateRecallHitTokens)
|
||
const average = costs.reduce((sum, cost) => sum + cost, 0) / costs.length
|
||
const count = Math.max(1, Math.min(topK, Math.floor(budget / Math.max(1, average))))
|
||
const retained = rows.slice(0, count)
|
||
// ★ `slice` copies indexed elements only. `degraded` / `degradedError` /
|
||
// `skipped` are NON-enumerable by design (so the phase-A golden fixture and
|
||
// every serialised log keep their shape), which means a plain slice silently
|
||
// drops them — and the degradation notice would then never reach the model.
|
||
// Re-attach every out-of-band channel the recall path is known to carry.
|
||
for (const key of ['degraded', 'degradedError', 'skipped', 'skippedDim']) {
|
||
if (Object.prototype.hasOwnProperty.call(rows, key)) {
|
||
Object.defineProperty(retained, key, {
|
||
value: rows[key],
|
||
enumerable: false,
|
||
configurable: true,
|
||
})
|
||
}
|
||
}
|
||
const singleHitOverBudget = costs[0] > budget
|
||
Object.defineProperty(retained, "budgetNotice", { value: singleHitOverBudget ? `[Memory] Top note estimate ${costs[0]} tokens exceeds memory budget ${budget}; retained without truncation.` : count < rows.length ? `[Memory] Recall limited to ${count} note(s) by memory budget ${budget}.` : "", enumerable: false })
|
||
return { hits: retained, budgetTokens: budget, source, singleHitOverBudget }
|
||
}
|