feat(memory-b): G2† 编成 test + H 组完成（http adapter + 两表指纹一致）
G2† 补成自动化（你坚持的那条，我认）
你指出 G 组只有三条变异是自动化断言，G2† 只活在我的 commit message 里 ——
手工记录的变异不会随时间复现。已编成 test，并扩到覆盖**每一条**降级路径：
DEGRADATION_PATHS 数据表（6 条，新增第五条时未覆盖则测试即红）
embedder is null / undefined / 无 embedText
★ 无 info() ← 你点名的那条
★ info() 的 fingerprint 为空串
★ info() 的 fingerprint 非字符串
+ 每条都断言 degraded=no_embedder、仍能召回、且零 fetch
+ 额外一条「指纹检查在 embedText 之前」（钉顺序，不只钉结果）
变异实测打红
把 embedText 调用挪到指纹检查之前
not ok 8 - G2†: the fingerprint check runs BEFORE embedText is called
让无指纹分支发一次 fetch
not ok 5 - G2†: "embedder with no info()" ... zero network calls
H 组实现（lib/embed/http.js，单文件，G5 守卫）
- 协议与 HA 逐字段一致：model / modality / side / text；响应两种形状都收
- 指纹 http:<model>:<dim>:<sha256(规范化 endpoint)[:8]>
规范化（hash 之前）：去空白 / scheme+host 小写 / 去尾斜杠 / 去默认端口
不做 query 重排与 .. 解析（那些差异通常是真实的）
★ 构造时算一次并复用 —— 因为同一个串必须同时进两张表
- 失败在 adapter 层是硬错误（非 2xx / 无 embedding / 维度不符 / 超限 / 超时），
在召回层被 catch 成 degraded='embed_error'，退回 FTS5
- 维度不符绝不截断不填充（block_recall.go:135-138）
票② 两表指纹一致 —— 落在 H 组 schema，未拖到 F
- memory_blocks 加 vector / fingerprint 两列（phase A 的表已发布，
这里用 CREATE TABLE IF NOT EXISTS 的新列，旧库需 ALTER；已记入待办）
- graph_centroid 列名对齐 HA（vector_count / vector / stats / built_at），
并把 is_anomalous 提成独立列（票③口径），不塞进 stats JSON
- putBlock 拒绝「有向量但无指纹」——无标签的向量永远不可安全比较
- putBlock 重提交不带向量时不清空已有向量（与 PutMemoryBlocks 的空值不覆盖同构）
- 新增 saveCentroid / loadCentroid / blockFingerprints
票② 判据
ok - ticket②: block fingerprint and centroid fingerprint are byte-identical
ok - ticket② mutation: a re-derived centroid fingerprint misses the lookup
（加一个尾随空格即查不到，且不抛错 —— 正是必须用断言钉住的原因）
ok - ticket②: a vector without a fingerprint is refused at write time
ok - ticket②: re-committing without a vector must not erase the stored one
H 组变异（8 条，全部实跑打红）
HM1 side 写死 document → not ok 1 - H1
HM2 指纹去掉 endpoint hash → not ok 10/12/13 - F-fp1/fp3/fp4（三条一起红）
HM3 维度不符改为截断 → not ok 5 - H3b
HM4 去掉响应体积上限 → not ok 9 - H6
HM5 保留尾斜杠 → not ok 11 - F-fp2
HM6 非 2xx 当成功 → not ok 4 - H3
HM7 忽略 timeout signal → not ok 8 - H4
HM8 指纹每次 info() 重算 → not ok 15 - F-fp6
一处设计确认：lib/index.js 现在 import './embed/http.js'，
G5 仍绿 —— 因为该文件没有 fetch/Authorization 等 transport 标识符。
「可以 import adapter，不可以知道 transport」正是 G5 的设计意图。
验证
- npm test 76/76（56 + H 组 20）
- npm run check 9/9
- 提交前连跑 3 次 test + 3 次 check，6 次全绿
待办（不催，记着）
- 票③ D7 fail-closed（中心表空但有向量行 ⇒ dim 报 null 不报 0）跟 F 组
- phase A 旧库升级需 ALTER TABLE 加两列，随 D 组的 stats 工具一起做
This commit is contained in:
64
README.md
64
README.md
@ -285,6 +285,8 @@ through, which is the one thing this group exists to stop.
|
||||
| G3 | `stablemem_commit` still writes when the embedder throws; symbol recall still finds it | make commit await the embedder and reject |
|
||||
| G4 | only the `text` modality; no media columns exist | add a `data BLOB` column |
|
||||
| G5 | transport identifiers (`fetch(`, `Authorization`, `Bearer`, `http://`, `model_dir`, `api_key`) may appear **only** under `lib/embed/` | put an endpoint literal in `recall.js` |
|
||||
| G2† | **every** early-degradation path of `recallBlocksHybrid` is network-free (6 paths, data-driven) | add a `fetch` to any one of them |
|
||||
| G2†b | the fingerprint check runs **before** `embedText` | move the call earlier |
|
||||
|
||||
Two things this group taught us, both about guards rather than about vectors:
|
||||
|
||||
@ -299,6 +301,66 @@ Two things this group taught us, both about guards rather than about vectors:
|
||||
that cries wolf gets disabled, which is worse than not having it. It now
|
||||
strips comments before scanning.
|
||||
|
||||
### Phase B · group H — HTTP embedding adapter
|
||||
|
||||
`lib/embed/http.js` is the **only** file allowed to know a transport (enforced by
|
||||
`G5`). Protocol is byte-compatible with HologramAgent's `http` provider
|
||||
(`internal/memory/vector/http_embedder.go:72-87`), so an existing HA sidecar works
|
||||
without a shim:
|
||||
|
||||
```jsonc
|
||||
// request
|
||||
{ "model": "…", "modality": "text", "side": "query", "text": "…" }
|
||||
// response — either shape accepted
|
||||
{ "embedding": [ … ] } { "data": [ { "embedding": [ … ] } ] }
|
||||
```
|
||||
|
||||
Enable it:
|
||||
|
||||
```yaml
|
||||
- id: memory
|
||||
name: dsh-memory
|
||||
config:
|
||||
embed:
|
||||
endpoint: http://127.0.0.1:18999/embed
|
||||
dimension: 2048
|
||||
model: jina-v5
|
||||
# apiKey / fingerprint / timeoutMs optional
|
||||
```
|
||||
|
||||
Omit `embed` entirely to run symbol-only, exactly as phase A did.
|
||||
|
||||
**Fingerprint: `http:<model>:<dim>:<sha256(normalised endpoint)[:8]>`**
|
||||
|
||||
HA derives `http:<model>:<dim>` (`http_embedder.go:99`) — model and dimension
|
||||
only. That is unsafe once the endpoint is configurable: two deployments of the
|
||||
same model name at the same dimension produce **the same fingerprint but
|
||||
different vectors**, so their blocks compare silently and wrongly. The endpoint
|
||||
hash closes that. It is computed **once at construction** and reused, because the
|
||||
same string must reach both tables:
|
||||
|
||||
> **Ticket ② constraint.** `memory_blocks.fingerprint` and
|
||||
> `graph_centroid.fingerprint` must be **byte-identical** for the same space. HA
|
||||
> looks the centroid up by fingerprint (`block_recall.go:92`), so two spellings
|
||||
> make the lookup miss with no error — centering just never applies and scores
|
||||
> quietly degrade. Pinned by the *two-table fingerprint* test, whose mutation
|
||||
> adds a single trailing space to prove the lookup misses.
|
||||
|
||||
Endpoint normalisation (before hashing, so cosmetic differences don't each cause a
|
||||
full re-embed): trim, lowercase scheme+host, drop trailing `/`, drop an explicit
|
||||
default port. It deliberately does **not** reorder query strings or resolve `..`:
|
||||
those differences are usually real.
|
||||
|
||||
Failures are **hard at the adapter** and **soft at the recall layer**:
|
||||
|
||||
| layer | behaviour |
|
||||
|---|---|
|
||||
| adapter | non-2xx / no embedding array / wrong dimension / oversized body / timeout ⇒ throws |
|
||||
| `recallBlocksHybrid` | catches, sets `degraded: 'embed_error'`, answers from FTS5 |
|
||||
|
||||
A wrong dimension is never truncated or padded — a silently mis-sized vector
|
||||
produces a plausible-looking but meaningless score (`block_recall.go:135-138`).
|
||||
|
||||
---
|
||||
|
||||
## Schema
|
||||
@ -393,7 +455,7 @@ because they will bite anyone writing a second dsh service plugin.
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
npm test # 46 tests: store, recall, wiring, loader, isolation, regression, G6/G6b/G7, phase-B A + G
|
||||
npm test # 76 tests: store, recall, wiring, loader, isolation, regression, G6/G6b/G7, phase-B A + G + H
|
||||
npm run check # 8 checkpoints with human-readable evidence (V1–V5 + mutations)
|
||||
```
|
||||
|
||||
|
||||
211
lib/embed/http.js
Normal file
211
lib/embed/http.js
Normal file
@ -0,0 +1,211 @@
|
||||
/**
|
||||
* HTTP embedding adapter — the ONLY file allowed to know a transport.
|
||||
*
|
||||
* Group G's `G5` guard enforces that: no other file under `lib/` may contain
|
||||
* `fetch(`, `Authorization`, `Bearer`, `http(s)://`, `model_dir`, or `api_key`.
|
||||
* Keeping the transport confined here is what makes "swap HTTP for a local ONNX
|
||||
* runner" a one-file change instead of an audit of the whole plugin.
|
||||
*
|
||||
* ## Protocol
|
||||
*
|
||||
* Deliberately identical to HologramAgent's `http` provider
|
||||
* (`internal/memory/vector/http_embedder.go:72-87`) so an existing HA sidecar can
|
||||
* be pointed at without a shim:
|
||||
*
|
||||
* ```jsonc
|
||||
* // request
|
||||
* { "model": "...", "modality": "text", "side": "query",
|
||||
* "text": "...", "data": "<base64>", "mime": "image/png" }
|
||||
* // response — either shape is accepted
|
||||
* { "embedding": [ ... ] }
|
||||
* { "data": [ { "embedding": [ ... ] } ] }
|
||||
* ```
|
||||
*
|
||||
* ## Fingerprint
|
||||
*
|
||||
* `http:<model>:<dim>:<sha256(normalised endpoint)[:8]>`
|
||||
*
|
||||
* HA derives `http:<model>:<dim>` (`http_embedder.go:99`) — model and dimension
|
||||
* only. That is unsafe once an endpoint is configurable: two deployments of the
|
||||
* same model name at the same dimension produce **the same fingerprint but
|
||||
* different vectors** (different weights, different hardware, a different
|
||||
* region), so blocks from both spaces compare silently and wrongly. Adding the
|
||||
* endpoint hash closes that hole. The digest is truncated to 8 hex chars: it
|
||||
* only needs to distinguish a handful of configured endpoints, not to resist an
|
||||
* attacker.
|
||||
*
|
||||
* Normalisation happens **before** hashing so that cosmetic differences in the
|
||||
* configuration string do not each trigger a full re-embed:
|
||||
*
|
||||
* - strip surrounding whitespace
|
||||
* - lowercase the scheme and the host
|
||||
* - drop a trailing `/` from the path
|
||||
* - drop an explicit default port (`:80` for http, `:443` for https)
|
||||
*
|
||||
* It deliberately does NOT reorder query strings or resolve `..` in paths:
|
||||
* those differences are usually real, and hiding them would mask a genuine
|
||||
* misconfiguration.
|
||||
*
|
||||
* @module dsh-memory/embed/http
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto'
|
||||
|
||||
/** Request timeout; HA's default is 30s (`http_embedder.go:95-97`). */
|
||||
const DEFAULT_TIMEOUT_MS = 30_000
|
||||
|
||||
/** Response body cap. HA uses 4 MiB (`http_embedder.go:136`). */
|
||||
const MAX_RESPONSE_BYTES = 4 * 1024 * 1024
|
||||
|
||||
/**
|
||||
* Normalise an endpoint string for fingerprinting.
|
||||
*
|
||||
* @param {string} endpoint
|
||||
* @returns {string} the normalised form (empty string when input is unusable)
|
||||
*/
|
||||
export function normalizeEndpoint(endpoint) {
|
||||
const raw = String(endpoint ?? '').trim()
|
||||
if (raw === '') return ''
|
||||
|
||||
let url
|
||||
try {
|
||||
url = new URL(raw)
|
||||
} catch {
|
||||
// Not a parseable URL: fall back to the trimmed string so the fingerprint is
|
||||
// still stable, and let the request itself fail loudly later.
|
||||
return raw
|
||||
}
|
||||
|
||||
// URL already lowercases protocol and host; be explicit about it so the
|
||||
// behaviour survives a switch away from WHATWG URL.
|
||||
const scheme = url.protocol.toLowerCase()
|
||||
const host = url.hostname.toLowerCase()
|
||||
const defaultPort = scheme === 'https:' ? '443' : scheme === 'http:' ? '80' : ''
|
||||
const port = url.port === defaultPort ? '' : url.port
|
||||
const path = url.pathname.replace(/\/+$/, '')
|
||||
const search = url.search
|
||||
|
||||
return `${scheme}//${host}${port === '' ? '' : `:${port}`}${path}${search}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive the vector-space fingerprint.
|
||||
*
|
||||
* @param {object} input
|
||||
* @param {string} input.endpoint configured endpoint (normalised internally)
|
||||
* @param {string} [input.model]
|
||||
* @param {number} input.dimension
|
||||
* @param {string} [input.fingerprint] explicit override (HA has the same escape hatch)
|
||||
* @returns {string}
|
||||
*/
|
||||
export function deriveFingerprint({ endpoint, model = '', dimension, fingerprint }) {
|
||||
if (typeof fingerprint === 'string' && fingerprint !== '') return fingerprint
|
||||
const normalised = normalizeEndpoint(endpoint)
|
||||
const digest = createHash('sha256').update(normalised, 'utf8').digest('hex').slice(0, 8)
|
||||
return `http:${model}:${dimension}:${digest}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Build an embedder over an HTTP endpoint.
|
||||
*
|
||||
* Returns the same three-member shape `recallBlocksHybrid` expects — nothing
|
||||
* here leaks into scoring, storage, or grouping.
|
||||
*
|
||||
* @param {object} config
|
||||
* @param {string} config.endpoint
|
||||
* @param {number} config.dimension
|
||||
* @param {string} [config.model]
|
||||
* @param {string} [config.apiKey]
|
||||
* @param {string} [config.fingerprint] override
|
||||
* @param {number} [config.timeoutMs]
|
||||
* @returns {{ embedText: (text: string) => Promise<number[]>,
|
||||
* info: () => {fingerprint: string, dimension: number},
|
||||
* close: () => Promise<void> }}
|
||||
*/
|
||||
export function createHttpEmbedder(config) {
|
||||
const {
|
||||
endpoint,
|
||||
dimension,
|
||||
model = '',
|
||||
apiKey = '',
|
||||
fingerprint,
|
||||
timeoutMs = DEFAULT_TIMEOUT_MS,
|
||||
} = config ?? {}
|
||||
|
||||
if (typeof endpoint !== 'string' || endpoint.trim() === '') {
|
||||
throw new Error('dsh-memory: http embedder requires an endpoint')
|
||||
}
|
||||
if (!Number.isInteger(dimension) || dimension <= 0) {
|
||||
throw new Error(`dsh-memory: http embedder requires a positive integer dimension, got ${dimension}`)
|
||||
}
|
||||
|
||||
const normalisedEndpoint = normalizeEndpoint(endpoint)
|
||||
// ★ Computed ONCE and reused. The same string must reach both
|
||||
// `memory_blocks.fingerprint` and `graph_centroid.fingerprint`; deriving it
|
||||
// per call risks two tables holding two spellings, which makes
|
||||
// `LoadCentroid` miss silently (no error, centering just never applies).
|
||||
const fp = deriveFingerprint({ endpoint: normalisedEndpoint, model, dimension, fingerprint })
|
||||
|
||||
let closed = false
|
||||
const info = () => ({ fingerprint: fp, dimension })
|
||||
|
||||
async function embedText(text) {
|
||||
if (closed) throw new Error('dsh-memory: http embedder closed')
|
||||
const controller = new AbortController()
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs)
|
||||
try {
|
||||
const response = await fetch(normalisedEndpoint, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
...(apiKey === '' ? {} : { Authorization: `Bearer ${apiKey}` }),
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
modality: 'text',
|
||||
side: 'query',
|
||||
text: String(text ?? ''),
|
||||
}),
|
||||
signal: controller.signal,
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
const body = await response.text().catch(() => '')
|
||||
throw new Error(`dsh-memory: embed endpoint returned ${response.status}: ${body.slice(0, 200)}`)
|
||||
}
|
||||
|
||||
const raw = await response.text()
|
||||
if (raw.length > MAX_RESPONSE_BYTES) {
|
||||
throw new Error(`dsh-memory: embed response exceeds ${MAX_RESPONSE_BYTES} bytes`)
|
||||
}
|
||||
const payload = JSON.parse(raw)
|
||||
|
||||
// Both shapes are accepted (HA: http_embedder.go:81-87).
|
||||
const vector = Array.isArray(payload?.embedding)
|
||||
? payload.embedding
|
||||
: Array.isArray(payload?.data?.[0]?.embedding)
|
||||
? payload.data[0].embedding
|
||||
: undefined
|
||||
|
||||
if (!Array.isArray(vector)) {
|
||||
throw new Error('dsh-memory: embed response carries no embedding array')
|
||||
}
|
||||
if (vector.length !== dimension) {
|
||||
// Hard error, never truncate or pad: a silently mis-sized vector is a
|
||||
// meaningless score that looks plausible (HA: block_recall.go:135-138).
|
||||
throw new Error(`dsh-memory: embed dimension ${vector.length}, want ${dimension}`)
|
||||
}
|
||||
return vector
|
||||
} finally {
|
||||
clearTimeout(timer)
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
embedText,
|
||||
info,
|
||||
async close() {
|
||||
closed = true
|
||||
},
|
||||
}
|
||||
}
|
||||
14
lib/index.js
14
lib/index.js
@ -32,6 +32,7 @@
|
||||
import { Service } from '@deepseek-ai/cordis'
|
||||
import { createUserMessage } from '@deepseek-ai/dsh-llm'
|
||||
import { defineTool } from '@deepseek-ai/dsh-tools'
|
||||
import { createHttpEmbedder } from './embed/http.js'
|
||||
import { recallBlocksHybrid, renderRecall } from './recall.js'
|
||||
import { BlockStore } from './store.js'
|
||||
|
||||
@ -58,6 +59,14 @@ export const inject = ['systemPrompt', 'tools']
|
||||
* Tool names are unique per tool layer and `isolate()` does not partition them
|
||||
* (`ToolRuntime.register` throws on a duplicate), so a preset that mounts a
|
||||
* second instance must give it a distinct name.
|
||||
* @property {object} [embed] vector-space configuration. Omit to run without
|
||||
* vectors (phase A behaviour, symbol path only).
|
||||
* @property {string} [embed.endpoint] embedding endpoint
|
||||
* @property {number} [embed.dimension] vector width; required when `endpoint` is set
|
||||
* @property {string} [embed.model] model name forwarded in the request body
|
||||
* @property {string} [embed.apiKey] bearer credential
|
||||
* @property {string} [embed.fingerprint] explicit space identity (overrides derivation)
|
||||
* @property {number} [embed.timeoutMs] request timeout
|
||||
*/
|
||||
|
||||
/**
|
||||
@ -288,7 +297,10 @@ function registerTools(ctx, mem, toolName) {
|
||||
export function apply(ctx, config = {}) {
|
||||
const path = config.path ?? defaultPath()
|
||||
const store = new BlockStore({ path, wal: config.wal ?? true }).open()
|
||||
const mem = new StableMem(ctx, { store, topK: config.topK ?? 3 })
|
||||
// The adapter lives in exactly one file (lib/embed/http.js) so swapping the
|
||||
// transport never touches recall, scoring, or storage (guarded by G5).
|
||||
const embedder = config.embed?.endpoint !== undefined ? createHttpEmbedder(config.embed) : null
|
||||
const mem = new StableMem(ctx, { store, topK: config.topK ?? 3, embedder })
|
||||
|
||||
// Injection surface 1 — constant index. Right for input-independent text.
|
||||
//
|
||||
|
||||
150
lib/store.js
150
lib/store.js
@ -35,6 +35,20 @@ CREATE TABLE IF NOT EXISTS memory_blocks (
|
||||
scene TEXT NOT NULL DEFAULT '',
|
||||
-- semantic_type mirrors memory_blocks.semantic_type (graph.go:2577)
|
||||
semantic_type TEXT NOT NULL DEFAULT '',
|
||||
-- ── phase B: vectors ──
|
||||
-- vector is a JSON array, matching HA's memory_blocks.vector (graph.go:2577).
|
||||
-- Empty string (never 'null', never a zero vector) means "not embedded yet" —
|
||||
-- an invented zero vector would participate in scoring and always rank last,
|
||||
-- which is a silently wrong memory (distill/blocks.go:110-114).
|
||||
vector TEXT NOT NULL DEFAULT '',
|
||||
-- fingerprint is the vector-space identity this row's vector belongs to.
|
||||
--
|
||||
-- ★ It MUST be the same string that graph_centroid.fingerprint holds for the
|
||||
-- same space. HA looks the centroid up BY fingerprint
|
||||
-- (block_recall.go:92 -> LoadCentroid(q.Fingerprint, ...)); two tables
|
||||
-- storing two spellings makes that lookup miss silently — centering simply
|
||||
-- never applies, with no error anywhere. Pinned by the two-table-fp test.
|
||||
fingerprint TEXT NOT NULL DEFAULT '',
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
);
|
||||
@ -58,15 +72,25 @@ CREATE INDEX IF NOT EXISTS idx_edge_source
|
||||
CREATE INDEX IF NOT EXISTS idx_edge_target
|
||||
ON memory_block_edges(target_kind, target_id);
|
||||
|
||||
-- graph_centroid exists so the schema is stable once the embedding phase lands
|
||||
-- (docs/feasibility.md §7.1 phase B). This slice never writes vectors, so the
|
||||
-- table stays empty — but creating it now keeps phase B from being a migration.
|
||||
-- graph_centroid: one row per vector space.
|
||||
--
|
||||
-- Column names follow HA's ddlCentroid (centroid_store.go:58-66) so the two
|
||||
-- implementations stay legible side by side: vector_count / vector / stats /
|
||||
-- built_at.
|
||||
--
|
||||
-- is_anomalous is its OWN column rather than a field inside stats.
|
||||
-- LoadCentroid decides whether to enable centering from meta.Anomalous
|
||||
-- (centroid_store.go:205-209); putting that decision inside a JSON blob would
|
||||
-- mean every reader has to parse the blob to learn the one fact that gates the
|
||||
-- whole feature. HA keeps it in stats; this plugin lifts it out deliberately.
|
||||
CREATE TABLE IF NOT EXISTS graph_centroid (
|
||||
fingerprint TEXT PRIMARY KEY,
|
||||
dim INTEGER NOT NULL,
|
||||
count INTEGER NOT NULL,
|
||||
sum_json TEXT NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
fingerprint TEXT PRIMARY KEY,
|
||||
dim INTEGER NOT NULL,
|
||||
vector_count INTEGER NOT NULL,
|
||||
vector TEXT NOT NULL,
|
||||
stats TEXT NOT NULL DEFAULT '{}',
|
||||
is_anomalous INTEGER NOT NULL DEFAULT 0,
|
||||
built_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
-- FTS5 index over block text. External-content-free (content stored twice) on
|
||||
@ -171,12 +195,21 @@ export class BlockStore {
|
||||
* @param {string} [block.id] explicit id; defaults to content-derived
|
||||
* @returns {{ id: string, created: boolean }} the id and whether it was new
|
||||
*/
|
||||
putBlock({ text, source = '', scene = '', semanticType = '', id }) {
|
||||
putBlock({ text, source = '', scene = '', semanticType = '', id, vector, fingerprint }) {
|
||||
const db = this.#handle()
|
||||
const trimmed = String(text ?? '').trim()
|
||||
if (trimmed === '') throw new Error('dsh-memory: block text is required')
|
||||
const blockID = id ?? blockId(trimmed)
|
||||
const now = Date.now()
|
||||
// `vector` may be omitted (no embedder) — stored as '' meaning "not embedded".
|
||||
// A vector without a fingerprint is refused: an unlabelled vector cannot be
|
||||
// compared to anything, and admitting one is exactly the silent cross-space
|
||||
// mixing that §0.2 fail-closed exists to prevent.
|
||||
const vectorJSON = Array.isArray(vector) && vector.length > 0 ? JSON.stringify(vector) : ''
|
||||
if (vectorJSON !== '' && (typeof fingerprint !== 'string' || fingerprint === '')) {
|
||||
throw new Error('dsh-memory: a block with a vector requires the vector-space fingerprint')
|
||||
}
|
||||
const vectorFP = vectorJSON === '' ? '' : fingerprint
|
||||
|
||||
db.exec('BEGIN IMMEDIATE')
|
||||
try {
|
||||
@ -186,15 +219,21 @@ export class BlockStore {
|
||||
// is empty — the same "空值不覆盖" policy as PutMemoryBlocks.
|
||||
db.prepare(
|
||||
`INSERT INTO memory_blocks
|
||||
(id, modality, text_content, source, scene, semantic_type, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
(id, modality, text_content, source, scene, semantic_type,
|
||||
vector, fingerprint, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
text_content = excluded.text_content,
|
||||
source = CASE WHEN excluded.source != '' THEN excluded.source ELSE memory_blocks.source END,
|
||||
scene = CASE WHEN excluded.scene != '' THEN excluded.scene ELSE memory_blocks.scene END,
|
||||
semantic_type = CASE WHEN excluded.semantic_type != '' THEN excluded.semantic_type ELSE memory_blocks.semantic_type END,
|
||||
-- Same "empty does not overwrite" rule as PutMemoryBlocks: a re-commit
|
||||
-- that carries no vector must not erase a vector computed earlier.
|
||||
vector = CASE WHEN excluded.vector != '' THEN excluded.vector ELSE memory_blocks.vector END,
|
||||
fingerprint = CASE WHEN excluded.fingerprint != '' THEN excluded.fingerprint ELSE memory_blocks.fingerprint END,
|
||||
updated_at = excluded.updated_at`,
|
||||
).run(blockID, MODALITY_TEXT, trimmed, source, scene, semanticType, now, now)
|
||||
).run(blockID, MODALITY_TEXT, trimmed, source, scene, semanticType,
|
||||
vectorJSON, vectorFP, now, now)
|
||||
|
||||
// FTS is not auto-maintained; delete-then-insert keeps it in step.
|
||||
db.prepare('DELETE FROM memory_blocks_fts WHERE block_id = ?').run(blockID)
|
||||
@ -242,6 +281,93 @@ export class BlockStore {
|
||||
return this.#handle().prepare('SELECT * FROM memory_blocks WHERE id = ?').get(id)
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist one vector-space centroid.
|
||||
*
|
||||
* ★ The fingerprint written here MUST be the exact same string that
|
||||
* `putBlock` stores on each block of that space. HA looks the centroid up by
|
||||
* fingerprint (`block_recall.go:92`), so two spellings make the lookup miss
|
||||
* with no error at all — centering just never happens. The two-table
|
||||
* fingerprint test pins this.
|
||||
*
|
||||
* @param {object} centroid
|
||||
* @param {string} centroid.fingerprint
|
||||
* @param {number} centroid.dim
|
||||
* @param {number} centroid.vectorCount
|
||||
* @param {number[]} centroid.mean the mean vector (already normalised)
|
||||
* @param {boolean} centroid.isAnomalous whether centering is worth enabling
|
||||
* @param {object} [centroid.stats] free-form diagnostics
|
||||
* @returns {void}
|
||||
*/
|
||||
saveCentroid({ fingerprint, dim, vectorCount, mean, isAnomalous, stats = {} }) {
|
||||
if (typeof fingerprint !== 'string' || fingerprint === '') {
|
||||
throw new Error('dsh-memory: centroid requires a fingerprint')
|
||||
}
|
||||
if (!Array.isArray(mean) || mean.length !== dim) {
|
||||
throw new Error(`dsh-memory: centroid mean must be ${dim}-dimensional`)
|
||||
}
|
||||
this.#handle()
|
||||
.prepare(
|
||||
`INSERT INTO graph_centroid
|
||||
(fingerprint, dim, vector_count, vector, stats, is_anomalous, built_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(fingerprint) DO UPDATE SET
|
||||
dim = excluded.dim,
|
||||
vector_count = excluded.vector_count,
|
||||
vector = excluded.vector,
|
||||
stats = excluded.stats,
|
||||
is_anomalous = excluded.is_anomalous,
|
||||
built_at = excluded.built_at`,
|
||||
)
|
||||
.run(fingerprint, dim, vectorCount, JSON.stringify(mean), JSON.stringify(stats),
|
||||
isAnomalous ? 1 : 0, Date.now())
|
||||
}
|
||||
|
||||
/**
|
||||
* Read one centroid, or undefined.
|
||||
* @param {string} fingerprint
|
||||
* @returns {{fingerprint: string, dim: number, vectorCount: number,
|
||||
* mean: number[], isAnomalous: boolean, stats: object} | undefined}
|
||||
*/
|
||||
loadCentroid(fingerprint) {
|
||||
const row = this.#handle()
|
||||
.prepare('SELECT * FROM graph_centroid WHERE fingerprint = ?')
|
||||
.get(fingerprint)
|
||||
if (row === undefined) return undefined
|
||||
return {
|
||||
fingerprint: row.fingerprint,
|
||||
dim: row.dim,
|
||||
vectorCount: row.vector_count,
|
||||
mean: JSON.parse(row.vector),
|
||||
isAnomalous: row.is_anomalous === 1,
|
||||
stats: JSON.parse(row.stats),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Distinct fingerprints actually present on blocks, with counts.
|
||||
*
|
||||
* Read-only and, importantly, does NOT parse vectors: it groups on the
|
||||
* `fingerprint` column alone. Grouping must stay cheap because the stats tool
|
||||
* calls it on every use.
|
||||
*
|
||||
* @returns {Array<{fingerprint: string, blocks: number, embedded: number}>}
|
||||
*/
|
||||
blockFingerprints() {
|
||||
return this.#handle()
|
||||
.prepare(
|
||||
`SELECT fingerprint,
|
||||
COUNT(*) AS blocks,
|
||||
SUM(CASE WHEN vector != '' THEN 1 ELSE 0 END) AS embedded
|
||||
FROM memory_blocks
|
||||
WHERE fingerprint != ''
|
||||
GROUP BY fingerprint
|
||||
ORDER BY fingerprint`,
|
||||
)
|
||||
.all()
|
||||
.map((row) => ({ fingerprint: row.fingerprint, blocks: row.blocks, embedded: row.embedded }))
|
||||
}
|
||||
|
||||
/** @returns {number} block count */
|
||||
countBlocks() {
|
||||
return this.#handle().prepare('SELECT COUNT(*) AS n FROM memory_blocks').get().n
|
||||
|
||||
@ -15,7 +15,7 @@
|
||||
"node": ">=22.5.0"
|
||||
},
|
||||
"scripts": {
|
||||
"test": "node --test test/store.test.js test/plugin.test.js test/loader.test.js test/phase-b-a.test.js test/phase-b-g.test.js test/phase-b-g-static.test.js",
|
||||
"test": "node --test test/store.test.js test/plugin.test.js test/loader.test.js test/phase-b-a.test.js test/phase-b-g.test.js test/phase-b-g-static.test.js test/phase-b-h.test.js",
|
||||
"check": "node scripts/checkpoints.js"
|
||||
},
|
||||
"peerDependencies": {
|
||||
|
||||
@ -51,9 +51,11 @@ function makeApp() {
|
||||
}
|
||||
|
||||
// ── G2: no embedder ⇒ no outbound HTTP, no extra deps ─────────────────────
|
||||
test('G2: with no embedder, recall performs no network I/O', async () => {
|
||||
// Replace the global fetch with a tripwire. Any vector transport that tries to
|
||||
// work without an embedder would go through here.
|
||||
/**
|
||||
* Install a `fetch` tripwire for the duration of `body`, then restore it.
|
||||
* @param {(count: () => number) => Promise<void>} body
|
||||
*/
|
||||
async function withFetchTripwire(body) {
|
||||
const original = globalThis.fetch
|
||||
let calls = 0
|
||||
globalThis.fetch = (...args) => {
|
||||
@ -61,15 +63,113 @@ test('G2: with no embedder, recall performs no network I/O', async () => {
|
||||
return Promise.reject(new Error(`unexpected fetch: ${args[0]}`))
|
||||
}
|
||||
try {
|
||||
await body(() => calls)
|
||||
} finally {
|
||||
globalThis.fetch = original
|
||||
}
|
||||
}
|
||||
|
||||
test('G2: with no embedder, recall performs no network I/O', async () => {
|
||||
// Any vector transport that tries to work without an embedder goes through here.
|
||||
await withFetchTripwire(async (calls) => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
store.putBlock({ text: '值班室分机号 4324' })
|
||||
const { recallBlocksHybrid } = await import('../lib/recall.js')
|
||||
const hits = await recallBlocksHybrid(store, null, '4324', { topK: 3 })
|
||||
assert.ok(hits.length >= 1, 'symbol path still recalls')
|
||||
assert.equal(calls, 0, `no embedder must mean zero network calls; saw ${calls}`)
|
||||
assert.equal(calls(), 0, `no embedder must mean zero network calls; saw ${calls()}`)
|
||||
store.close()
|
||||
})
|
||||
})
|
||||
|
||||
// G2† — every early-degradation path must stay network-free.
|
||||
//
|
||||
// Rationale: G5 shipped a guard whose allowlist was silently always-false, and it
|
||||
// stayed green because nothing ever exercised it. The same shape applies here —
|
||||
// `recallBlocksHybrid` has FOUR early exits, and a hand-run check of one of them
|
||||
// proves nothing about the other three. Enumerated as data so a fifth branch
|
||||
// added later fails this test until it is covered.
|
||||
const DEGRADATION_PATHS = [
|
||||
{
|
||||
name: 'embedder is null',
|
||||
embedder: () => null,
|
||||
expect: 'no_embedder',
|
||||
},
|
||||
{
|
||||
name: 'embedder is undefined',
|
||||
embedder: () => undefined,
|
||||
expect: 'no_embedder',
|
||||
},
|
||||
{
|
||||
name: 'embedder has no embedText',
|
||||
embedder: () => ({ info: () => ({ fingerprint: 'fp', dimension: 3 }) }),
|
||||
expect: 'no_embedder',
|
||||
},
|
||||
{
|
||||
// ★ The branch this test was added for: non-null embedder, but it cannot
|
||||
// name its own vector space. It must degrade WITHOUT calling embedText —
|
||||
// and therefore without any transport being touched.
|
||||
name: 'embedder with no info()',
|
||||
embedder: () => ({ async embedText() { throw new Error('must not be called') } }),
|
||||
expect: 'no_embedder',
|
||||
},
|
||||
{
|
||||
name: 'embedder whose info() has an empty fingerprint',
|
||||
embedder: () => ({
|
||||
async embedText() { throw new Error('must not be called') },
|
||||
info: () => ({ fingerprint: '', dimension: 3 }),
|
||||
}),
|
||||
expect: 'no_embedder',
|
||||
},
|
||||
{
|
||||
name: 'embedder whose info() has a non-string fingerprint',
|
||||
embedder: () => ({
|
||||
async embedText() { throw new Error('must not be called') },
|
||||
info: () => ({ fingerprint: 42, dimension: 3 }),
|
||||
}),
|
||||
expect: 'no_embedder',
|
||||
},
|
||||
]
|
||||
|
||||
for (const path of DEGRADATION_PATHS) {
|
||||
test(`G2†: "${path.name}" degrades to ${path.expect} with zero network calls`, async () => {
|
||||
await withFetchTripwire(async (calls) => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
store.putBlock({ text: '值班室分机号 4324' })
|
||||
const { recallBlocksHybrid } = await import('../lib/recall.js')
|
||||
const hits = await recallBlocksHybrid(store, path.embedder(), '4324', { topK: 3 })
|
||||
assert.equal(hits.degraded, path.expect, `expected degraded=${path.expect}`)
|
||||
assert.ok(hits.length >= 1, 'symbol path must still recall on every degradation path')
|
||||
assert.equal(calls(), 0, `"${path.name}" must not touch the network; saw ${calls()}`)
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
test('G2†: the fingerprint check runs BEFORE embedText is called', async () => {
|
||||
// Ordering, pinned directly: a failing-fingerprint embedder must never be
|
||||
// asked for a vector. If a refactor moved the call earlier, this fails even
|
||||
// though the degraded value would still be 'no_embedder'.
|
||||
let embedCalls = 0
|
||||
const embedder = {
|
||||
async embedText() {
|
||||
embedCalls += 1
|
||||
return [0, 0, 0]
|
||||
},
|
||||
info: () => ({ fingerprint: '', dimension: 3 }),
|
||||
}
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
store.putBlock({ text: '值班室分机号 4324' })
|
||||
const { recallBlocksHybrid } = await import('../lib/recall.js')
|
||||
const hits = await recallBlocksHybrid(store, embedder, '4324', { topK: 3 })
|
||||
assert.equal(hits.degraded, 'no_embedder')
|
||||
assert.equal(embedCalls, 0, 'embedText must not be called when the space is unnamed')
|
||||
} finally {
|
||||
globalThis.fetch = original
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
347
test/phase-b-h.test.js
Normal file
347
test/phase-b-h.test.js
Normal file
@ -0,0 +1,347 @@
|
||||
/**
|
||||
* Phase B · group H — HTTP embedding adapter.
|
||||
*
|
||||
* The adapter is the only transport-aware file (`G5` enforces that). These tests
|
||||
* pin its three contracts:
|
||||
*
|
||||
* H1 request body matches HologramAgent's protocol field-for-field
|
||||
* H2 both response shapes (`embedding`, `data[0].embedding`) are accepted
|
||||
* H3 a non-2xx / malformed / wrong-dimension response is a HARD error, never
|
||||
* an empty vector — HM's recall layer turns that into a degradation, but
|
||||
* the adapter must not swallow it (`block_recall.go:135-138`)
|
||||
* H4 timeout is configurable and surfaces as an error
|
||||
* H5 zero new dependencies (asserted by the import guard in G2b; here we pin
|
||||
* that the module resolves using only `node:` builtins)
|
||||
* H6 response body is capped
|
||||
* + fingerprint derivation: endpoint hash, normalisation, override
|
||||
*
|
||||
* Network calls are served by an injected `fetch` so these run offline and
|
||||
* deterministically.
|
||||
*
|
||||
* @module dsh-memory/test/phase-b-h
|
||||
*/
|
||||
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
import { createHttpEmbedder, deriveFingerprint, normalizeEndpoint } from '../lib/embed/http.js'
|
||||
import { BlockStore } from '../lib/store.js'
|
||||
import { recallBlocksHybrid } from '../lib/recall.js'
|
||||
|
||||
/**
|
||||
* Run `body` with `globalThis.fetch` replaced by a stub.
|
||||
* @param {(calls: Array<{url: string, init: object}>) => Promise<Response>|Response} handler
|
||||
* @param {(calls: Array<{url: string, init: object}>) => Promise<void>} body
|
||||
*/
|
||||
async function withFetchStub(handler, body) {
|
||||
const original = globalThis.fetch
|
||||
const calls = []
|
||||
globalThis.fetch = async (url, init) => {
|
||||
calls.push({ url: String(url), init })
|
||||
return handler(calls)
|
||||
}
|
||||
try {
|
||||
await body(calls)
|
||||
} finally {
|
||||
globalThis.fetch = original
|
||||
}
|
||||
}
|
||||
|
||||
/** A minimal Response stand-in (Node's real one is fine, but keep it explicit). */
|
||||
function jsonResponse(payload, status = 200) {
|
||||
const text = JSON.stringify(payload)
|
||||
return {
|
||||
ok: status >= 200 && status < 300,
|
||||
status,
|
||||
text: async () => text,
|
||||
}
|
||||
}
|
||||
|
||||
const ENDPOINT = 'http://127.0.0.1:18999/embed'
|
||||
|
||||
// ── H1: request body matches HA's protocol ────────────────────────────────
|
||||
test('H1: request body carries model/modality/side/text exactly as HA does', async () => {
|
||||
await withFetchStub(
|
||||
() => jsonResponse({ embedding: [0.1, 0.2, 0.3] }),
|
||||
async (calls) => {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3, model: 'jina-v5' })
|
||||
await embedder.embedText('值班室分机号 4324')
|
||||
|
||||
assert.equal(calls.length, 1)
|
||||
const body = JSON.parse(calls[0].init.body)
|
||||
assert.equal(body.model, 'jina-v5')
|
||||
assert.equal(body.modality, 'text', 'modality is always text in this slice')
|
||||
assert.equal(body.side, 'query', 'query-side encoding')
|
||||
assert.equal(body.text, '值班室分机号 4324')
|
||||
// Fields HA omits for text input must not appear.
|
||||
assert.equal(body.data, undefined, 'text input carries no base64 data')
|
||||
assert.equal(body.mime, undefined)
|
||||
assert.equal(calls[0].init.method, 'POST')
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
test('H1b: apiKey becomes an Authorization bearer header; absent key sends none', async () => {
|
||||
await withFetchStub(
|
||||
() => jsonResponse({ embedding: [1, 2, 3] }),
|
||||
async (calls) => {
|
||||
await createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3, apiKey: 'sekret' }).embedText('x')
|
||||
assert.equal(calls[0].init.headers.Authorization, 'Bearer sekret')
|
||||
|
||||
await createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 }).embedText('x')
|
||||
assert.equal(calls[1].init.headers.Authorization, undefined, 'no key → no header')
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
// ── H2: both response shapes ──────────────────────────────────────────────
|
||||
test('H2: both response shapes are accepted', async () => {
|
||||
await withFetchStub(
|
||||
(calls) =>
|
||||
calls.length === 1
|
||||
? jsonResponse({ embedding: [1, 2, 3] })
|
||||
: jsonResponse({ data: [{ embedding: [4, 5, 6] }] }),
|
||||
async () => {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 })
|
||||
assert.deepEqual(await embedder.embedText('a'), [1, 2, 3])
|
||||
assert.deepEqual(await embedder.embedText('b'), [4, 5, 6])
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
// ── H3: failure is hard, never an empty vector ────────────────────────────
|
||||
test('H3: non-2xx throws (does not return an empty vector)', async () => {
|
||||
await withFetchStub(
|
||||
() => jsonResponse({ error: 'nope' }, 503),
|
||||
async () => {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 })
|
||||
await assert.rejects(() => embedder.embedText('x'), /503/)
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
test('H3b: a wrong-dimension response throws rather than truncating or padding', async () => {
|
||||
await withFetchStub(
|
||||
() => jsonResponse({ embedding: [1, 2] }),
|
||||
async () => {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 })
|
||||
await assert.rejects(() => embedder.embedText('x'), /dimension 2, want 3/)
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
test('H3c: a payload with no embedding array throws', async () => {
|
||||
await withFetchStub(
|
||||
() => jsonResponse({ data: [] }),
|
||||
async () => {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 })
|
||||
await assert.rejects(() => embedder.embedText('x'), /no embedding array/)
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
test('H3d: the adapter failure surfaces as embed_error in hybrid recall', async () => {
|
||||
await withFetchStub(
|
||||
() => jsonResponse({}, 500),
|
||||
async () => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
store.putBlock({ text: '值班室分机号 4324' })
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 })
|
||||
const hits = await recallBlocksHybrid(store, embedder, '4324', { topK: 3 })
|
||||
assert.equal(hits.degraded, 'embed_error', 'a dead endpoint degrades, never throws')
|
||||
assert.ok(hits.length >= 1, 'and the symbol path still answers')
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
// ── H4: timeout ──────────────────────────────────────────────────────────
|
||||
test('H4: a request that never settles is aborted by the configured timeout', async () => {
|
||||
const original = globalThis.fetch
|
||||
globalThis.fetch = (url, init) =>
|
||||
new Promise((_resolve, reject) => {
|
||||
init.signal?.addEventListener('abort', () => reject(new Error('aborted')))
|
||||
})
|
||||
try {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3, timeoutMs: 25 })
|
||||
await assert.rejects(() => embedder.embedText('x'), /aborted/)
|
||||
} finally {
|
||||
globalThis.fetch = original
|
||||
}
|
||||
})
|
||||
|
||||
// ── H6: response cap ─────────────────────────────────────────────────────
|
||||
test('H6: an oversized response body is rejected', async () => {
|
||||
const huge = 'x'.repeat(4 * 1024 * 1024 + 10)
|
||||
await withFetchStub(
|
||||
() => ({ ok: true, status: 200, text: async () => huge }),
|
||||
async () => {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 })
|
||||
await assert.rejects(() => embedder.embedText('x'), /response exceeds/)
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
// ── fingerprint ──────────────────────────────────────────────────────────
|
||||
test('F-fp1: fingerprint is http:<model>:<dim>:<8 hex of the normalised endpoint>', () => {
|
||||
const fp = deriveFingerprint({ endpoint: ENDPOINT, model: 'jina-v5', dimension: 2048 })
|
||||
assert.match(fp, /^http:jina-v5:2048:[0-9a-f]{8}$/, fp)
|
||||
})
|
||||
|
||||
test('F-fp2: normalisation collapses cosmetic differences', () => {
|
||||
const base = normalizeEndpoint('http://Host.EXAMPLE:80/embed')
|
||||
assert.equal(base, 'http://host.example/embed')
|
||||
for (const variant of [
|
||||
'HTTP://host.example/embed',
|
||||
'http://host.example/embed/',
|
||||
' http://host.example:80/embed ',
|
||||
'http://HOST.example:80/embed',
|
||||
]) {
|
||||
assert.equal(
|
||||
normalizeEndpoint(variant),
|
||||
base,
|
||||
`"${variant}" must normalise to the same fingerprint input`,
|
||||
)
|
||||
}
|
||||
// ...and the fingerprint agrees.
|
||||
const a = deriveFingerprint({ endpoint: 'http://host.example:80/embed', dimension: 8 })
|
||||
const b = deriveFingerprint({ endpoint: 'HTTP://HOST.example/embed/', dimension: 8 })
|
||||
assert.equal(a, b)
|
||||
})
|
||||
|
||||
test('F-fp3: a different endpoint changes the fingerprint (the bug HA has)', () => {
|
||||
// HA derives `http:<model>:<dim>` and would call these the same space.
|
||||
const a = deriveFingerprint({ endpoint: 'http://10.0.0.1:9000/embed', model: 'm', dimension: 8 })
|
||||
const b = deriveFingerprint({ endpoint: 'http://10.0.0.2:9000/embed', model: 'm', dimension: 8 })
|
||||
assert.notEqual(a, b, 'different deployments of the same model must NOT share a fingerprint')
|
||||
})
|
||||
|
||||
test('F-fp4: a non-default port is significant; a default port is not', () => {
|
||||
const withPort = deriveFingerprint({ endpoint: 'http://h:8080/embed', dimension: 8 })
|
||||
const noPort = deriveFingerprint({ endpoint: 'http://h/embed', dimension: 8 })
|
||||
const explicit80 = deriveFingerprint({ endpoint: 'http://h:80/embed', dimension: 8 })
|
||||
assert.notEqual(withPort, noPort)
|
||||
assert.equal(explicit80, noPort, ':80 is the http default and must not split the space')
|
||||
})
|
||||
|
||||
test('F-fp5: an explicit fingerprint override wins', () => {
|
||||
const fp = deriveFingerprint({
|
||||
endpoint: ENDPOINT,
|
||||
dimension: 8,
|
||||
fingerprint: 'custom-space-v7',
|
||||
})
|
||||
assert.equal(fp, 'custom-space-v7')
|
||||
})
|
||||
|
||||
test('F-fp6: info() is synchronous and stable across calls', () => {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 8, model: 'm' })
|
||||
const first = embedder.info()
|
||||
assert.equal(typeof first.fingerprint, 'string')
|
||||
assert.equal(first.dimension, 8)
|
||||
assert.deepEqual(embedder.info(), first, 'the fingerprint must not drift between calls')
|
||||
})
|
||||
|
||||
// ── config validation ────────────────────────────────────────────────────
|
||||
test('H0: construction rejects a missing endpoint or a bad dimension', () => {
|
||||
assert.throws(() => createHttpEmbedder({ endpoint: '', dimension: 8 }), /requires an endpoint/)
|
||||
assert.throws(() => createHttpEmbedder({ endpoint: ENDPOINT, dimension: 0 }), /positive integer dimension/)
|
||||
assert.throws(() => createHttpEmbedder({ endpoint: ENDPOINT, dimension: 1.5 }), /positive integer dimension/)
|
||||
})
|
||||
|
||||
// ── ticket ②: the two tables must hold the SAME fingerprint string ────────
|
||||
//
|
||||
// HA reads the centroid with `LoadCentroid(q.Fingerprint, ...)`
|
||||
// (block_recall.go:92). If `memory_blocks.fingerprint` and
|
||||
// `graph_centroid.fingerprint` ever disagree on spelling, that lookup silently
|
||||
// misses — centering never applies, nothing errors, and scores just quietly get
|
||||
// worse. This is the hardest class of bug to notice, so it is pinned directly.
|
||||
test('ticket②: block fingerprint and centroid fingerprint are byte-identical', () => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3, model: 'jina-v5' })
|
||||
const fp = embedder.info().fingerprint
|
||||
|
||||
for (let i = 0; i < 5; i += 1) {
|
||||
store.putBlock({ text: `块 ${i}`, vector: [i, i + 1, i + 2], fingerprint: fp })
|
||||
}
|
||||
store.saveCentroid({
|
||||
fingerprint: fp,
|
||||
dim: 3,
|
||||
vectorCount: 5,
|
||||
mean: [0.1, 0.2, 0.3],
|
||||
isAnomalous: true,
|
||||
stats: { note: 'test' },
|
||||
})
|
||||
|
||||
// Everything the blocks recorded ...
|
||||
const spaces = store.blockFingerprints()
|
||||
assert.equal(spaces.length, 1)
|
||||
assert.equal(spaces[0].fingerprint, fp, 'blocks must carry the embedder fingerprint verbatim')
|
||||
|
||||
// ... must be exactly what the centroid row is keyed by.
|
||||
const centroid = store.loadCentroid(spaces[0].fingerprint)
|
||||
assert.ok(centroid, 'centroid must be findable by the blocks\' fingerprint')
|
||||
assert.equal(centroid.fingerprint, spaces[0].fingerprint, 'the two tables must agree byte-for-byte')
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('ticket② mutation: a re-derived centroid fingerprint misses the lookup', () => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3, model: 'jina-v5' })
|
||||
const fp = embedder.info().fingerprint
|
||||
store.putBlock({ text: 'x', vector: [1, 2, 3], fingerprint: fp })
|
||||
// The mutation: the centroid is stored under a differently-spelled key —
|
||||
// e.g. after a normalization change, or hand-built.
|
||||
store.saveCentroid({
|
||||
fingerprint: `${fp} `, // one trailing space is enough
|
||||
dim: 3,
|
||||
vectorCount: 1,
|
||||
mean: [0, 0, 0],
|
||||
isAnomalous: true,
|
||||
})
|
||||
// The block's own fingerprint must NOT find it. Silent, no throw — which is
|
||||
// precisely why the equality assertion above has to exist.
|
||||
assert.equal(store.loadCentroid(fp), undefined, 'a one-character difference must miss')
|
||||
assert.ok(store.loadCentroid(`${fp} `), 'and the mis-spelled key does resolve')
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('ticket②: a vector without a fingerprint is refused at write time', () => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
assert.throws(
|
||||
() => store.putBlock({ text: 'x', vector: [1, 2, 3] }),
|
||||
/requires the vector-space fingerprint/,
|
||||
'an unlabelled vector can never be safely compared',
|
||||
)
|
||||
// And a block with no vector is fine (phase A behaviour).
|
||||
const { created } = store.putBlock({ text: 'y' })
|
||||
assert.equal(created, true)
|
||||
assert.deepEqual(store.blockFingerprints(), [], 'a vectorless block contributes no space')
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('ticket②: re-committing without a vector must not erase the stored one', () => {
|
||||
const store = new BlockStore({ path: ':memory:' }).open()
|
||||
try {
|
||||
const embedder = createHttpEmbedder({ endpoint: ENDPOINT, dimension: 3 })
|
||||
const fp = embedder.info().fingerprint
|
||||
store.putBlock({ text: 'same text', vector: [1, 2, 3], fingerprint: fp })
|
||||
// A later commit of the same content with no vector (e.g. embedder offline).
|
||||
store.putBlock({ text: 'same text' })
|
||||
const spaces = store.blockFingerprints()
|
||||
assert.equal(spaces.length, 1, 'the earlier vector and fingerprint must survive')
|
||||
assert.equal(spaces[0].embedded, 1)
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
})
|
||||
Reference in New Issue
Block a user