JianFeeeee 0ca9e96290 fix(knowledge): 修复英文标识符检索不到(分词器 + RRF 融合)
## 故障

真实库 196 条里 154 条是 cangjie 文档(1.53MB / 88.6% 字节)。用户反馈
「语义检索命中偏弱,cangjie 霸榜」。实测三条带精确标识符的查询:

    PruneOldBackups   → top5 全是 cangjie,目标条目一次没出现
    minSceneEvidence  → 同上
    strip_path        → 同上

## 真因不是 cangjie,是分词器

cangjie 只是把问题放大的体量。真正的洞在 `memory.TokenizeWords`(jieba)
对英文标识符的处理 —— 实测:

    PruneOldBackups → [P r u n e O l d B a c k p s]   ← 切成单字符还去重
    strip_path      → [s t r i p _ a h]

于是任意两篇含英文字母的文档 token 重叠率都能到 50%,词法路彻底失效。
而稠密路(chineseclip)对「只在文中出现一次的专名」本就没有分辨力。

这解释了 2026-10-08 那次决策:当时把词法路从融合里去掉,理由是
「词法路对英文标识符失效」—— 观察对,**归因错**:失效的是分词器。

## 改法

1. `identtoken.go`(新):知识库词法路专用分词器
   - 补标识符**整词**:`PruneOldBackups` → `pruneoldbackups`
   - 补**分隔符部件**:`strip_path` → `strip` / `path`(仅当下划线等存在时)
   - **不拆 camelCase**:拆了会产出 `old`/`backups`,而 `old` 在
     `cangjie/tools/cmd-tools/cjcompat_manual` 里是 CLI 参数名 `--old`,
     出现 31 次 —— 实测评测中该文档得分 0.54 > 目标的 0.20,正好把无关文档顶上来
   - 丢掉 jieba 切出来的单字母残渣
   - 中文分词完全沿用 jieba,行为不变
   - 只注入知识库的 veczer,**不动** memory 索引/关键词提取

2. `fusion.go`(新):RRF 倒数排名融合
   - `score = 1.0/(60+rank_dense) + 0.5/(60+rank_lex)`
   - 不用当初那套「逐路按查询内最大值归一化」:那个做法把 0.02 的噪声抬到
     与 1.0 等权,正是 2026-10-08 淘汰融合的真凶。RRF 只看名次,与量纲无关
   - 稠密权重 1.0(语义主线)/ 词法 0.5(精确命名救助)

3. `Search` 文档注释重写:把两段历史都写清(2026-10-08 为何淘汰、2026-10-31
   为何加回但换成 RRF),避免下一个人重复踩。

## 判据

- `TestIdentifierTokenFeedsLex`:钉分词结果(必须产出整词、不得拆出 old)
- `TestIdentifierQueryBeatsVolume`:60 篇英文词密集的噪声文档 + 一条只在
  目标条目出现一次的标识符,top1 必须命中目标 —— 分词器一退化就红
- `TestSeparatorIdentifierNotBuried`:带 `_` 的标识符,整词要压过部件噪声
- `TestDensePathFeedsRanking`(原 `TestDensePathIsSoleRecall`):保留它真正的
  意图 —— 防「调用了 denseHits 却把结果丢掉」这个反模式,形状改为喂给 fuseRRF
- `TestSearchDocReflectsRRF`(原 `..DenseOnly`):钉文档与实现不漂移

## 验证

    go build ./...                        ✓
    go vet ./internal/knowledge/          ✓
    gofmt -l                              ✓
    go test ./internal/knowledge/         ✓(含 -race)
    go test ./... -count=1                全绿

⚠️ 生产实例需重启才能生效(lex 索引在启动时重建,无数据迁移)。
2026-10-31 16:42:36 +08:00

⚠️ AI-Assisted Programming Notice: Parts of this project's code, documentation, and commit history were generated or modified with AI assistance. Key changes have been human-reviewed, but please evaluate and verify before use.

HologramAgent

中文: README.md

Self-hosted agent kernel. Runs on your own hardware; plugins attach as separate processes.

make build build-cli
./build/homed -data /tmp/ha

API keys go in the web UI at http://localhost:8080. Requires Linux.

The main difference from typical agent frameworks is context handling: instead of truncating on overflow or asking the model to summarize, it scores history by relevance and sinks the low-scoring events into cold context, then distills those into a graph. Two other things differ: startup order is resolved at build time, and plugins launch as soon as their facilities are ready.

Architecture

                    homed (kernel)
        LLM orchestration · memory · retrieval
                       │
        ┌──────────────┼──────────────┐
        │              │              │
   stdio RPC       shared mem     event ring
     control       bulk data      notify
        │              │              │
        └──────────────┼──────────────┘
                       │
              plugins (separate processes)
   web UI · QQ · CLI · files · browser

The kernel process handles orchestration, memory and retrieval; HTTP, files and devices belong to plugins (internal/plugin). A crashed plugin is restarted with backoff 1s / 2s / 3s; the 4th crash within 5 minutes stops automatic restarting pending human intervention (cmd/homed/kmod_runtime.go).

Context window

internal/agent/core/context.go

Each event gets two vectors on entry: a dense vector sharing one space with media and cold context, and a TF-IDF word vector as fallback. Pruning scores history against the current input and keeps topK, with the most recent N events exempt from scoring.

N defaults to 10, bounded by budget: min(configured, ContextTokens / avg tokens per event).

That bound came from a measurement. At a 50k window with 5800 tokens per event and a 40000 budget, pinning 10 events needed 58k, more than the budget itself. Every prune met its target without freeing space; the budget was gone after two rounds and the task terminated, with recall falling from 44% to 22%. Small windows now tighten automatically; a 1M window uses the configured value.

Pruned events move to the cold-context layer together with the media blocks they hold. Block IDs stay the same; only the holding layer changes. Retrieval uses coldcontext_query: Peek to confirm, then Remove the source copy. There is no fetch-and-delete path, since deletion follows confirmation.

Three layers

internal/agent/core/context.go · internal/memory/coldcontext/ · internal/memory/graph.go

Layer Holds Reaches the model Cooling
L1 Context events + blocks directly, in the system prompt low relevance sinks
L2 Cold Context events + blocks (same shape as L1) inserted back into L1 order on recall ≤2 accesses and 72h idle, then distilled
L3 Stable triples, nodes are blocks on recall never leaves

A block has exactly one holder at any layer, with no reference counting. Images and audio are blocks in L2/L3 with their own vectors, searched in the same space as text.

Scenes

internal/memory/scene_emerge.go

A scene is built from five weighted features: channel 1.0, conversation partner 1.0, tools 0.8, topic 0.4, time of day 0.2. A scene is created on the second occurrence of the same fingerprint (minSceneEvidence = 2); one-off interactions don't create one. Join threshold 0.5, recall 0.35. Recurrence adds 1 to strength; scenes that stop recurring decay by half-life.

Two entry points run in parallel: a plugin declares the scene at an injection point (e.g. chan:qq/peer:group_1), or fingerprints cluster on their own. A declared scene derives features only from its own key and does not absorb the full fingerprint. Absorbing it would make the declared scene match nearly every later turn of that kind at close to 1.0 similarity, and the clustering path could never grow narrower scenes.

Seven stages

internal/agent/core/stages.go

Stage When Plugins can
on_input message arrives blocklist, rate limit, short-circuit reply
pre_action context ready, before LLM inject external data
post_action LLM returns text and tool calls filter, force redirect
before_toolcall before one tool runs audit, deny, rewrite args
after_toolcall after one tool runs redact, sort results
before_output final text ready format adaptation
after_output after sending statistics

The inner loop is post_action → [before_toolcall → run → after_toolcall] → post_action. It exits when the model stops calling tools, all calls are denied, or the limit is reached.

Everything else

  • Channels: web UI, CLI, QQ groups and DMs, A2A, ACP in your IDE
  • Tools: files, shell, code, browser (real Chrome), calendar, notes, RSS, music
  • Modalities: images, video, voice, screenshots, web pages

28 built-in plugins, 89 model-callable tools. External plugins live one per repo under github.com/HologramAgent.

Downloads

Releases ship three variants:

Variant Contents For
full homed + waiter + desktop GUI + systemd unit Single-machine, everything
server homed + waiter + systemd unit Servers (no desktop environment)
client waiter + desktop GUI Connecting to a remote HologramAgent
  • Linux: .deb (amd64/arm64), .rpm (x86_64), .tar.gz
  • Windows: HologramAgent_v1.2.0_{Full,Server,Client}_win64.exe (NSIS installer, includes the AGPL license page). Since v1.2.0 homed no longer supports native Windows (it relies on fd inheritance and in-segment offset dereferencing), so the installer bootstraps WSL2 and installs the Linux packages inside the distribution the same way a Linux host would.
  • Portable: homeagent-bin-<os>_<arch>.tar.gz (homed/waiter/initconfig)
  • Verification: SHA256SUMS

The macOS homed requires a native macOS build (CGO + sqlite3), so release packages ship only waiter/initconfig.

Build

make build build-cli    # Build daemon + CLI
make test               # go test ./...
make install            # Install to system

Dependencies: Go 1.25+, CGo (go-sqlite3), Linux.

homed requires Linux (it relies on fd inheritance and intra-segment offset dereferencing of the shared memory region; see cmd/homed/platform_windows.go). On Windows only waiter.exe is built and homed runs under WSL2 (see Downloads). macOS can build waiter/initconfig; homed must be built on native macOS.

License

AGPL-3.0-only. See LICENSE.

Bundled third-party components keep their own licenses; full texts ship in /usr/share/doc/hologramagent/licenses/.

Third-party components shipped with the packages

Component License Location
Chinese-CLIP ViT-B/16 (ONNX artifacts) Apache-2.0 /usr/lib/hologram/models/chinese-clip-vit-b16-onnx/
ONNX Runtime (libonnxruntime.so) MIT /usr/lib/hologram/onnxruntime/
jieba dictionary (embedded in the binary) MIT internal/memory/jiebadict/
Go dependencies (go-sqlite3, gojieba, bubbletea, …) MIT / BSD-3 / Apache-2.0 permissive, AGPL-3.0-compatible

The package metadata declares this package as AGPL-3.0-only.

Description
No description provided
Readme AGPL-3.0 190 MiB
Languages
Go 80.7%
JavaScript 9.1%
Python 2.9%
HTML 1.8%
CSS 1.4%
Other 4%