docs: update vectorization algorithm - LocalWordEmbedder (jieba + TF-IDF + PMI co-occurrence)

Context layer uses LocalWordEmbedder not plain TF-IDF:
- jieba segmentation, TF-IDF weights, sliding window PMI co-occurrence
- Document/Indexer layers still use char-bigram TF-IDF (TFIDFVectorizer)
- Add 'protect last 10 events' detail to Context pruning description
This commit is contained in:
root
2026-07-17 07:02:53 +08:00
parent dd01388e76
commit fb2a02af22
4 changed files with 38 additions and 12 deletions

View File

@ -117,13 +117,25 @@ Setting `ctx.Response` at any stage jumps to `after_output`.
→ triples → GraphDB.Commit
```
### TF-IDF Vectorization (char 1-2 gram)
### Vectorization: Two Strategies
TF-IDF is the core algorithm running through all three memory layers, used in 4 independent locations:
Vectorization is used in 4 independent locations with different strategies:
**Strategy A — Local Word Embedding** (`LocalWordEmbedder`, `internal/memory/embedder.go`), used by Context layer:
- **jieba tokenization** → removes stop words and single characters
- **TF-IDF** as base word weight
- **Sliding window (size=5)** counts word co-occurrence → **PMI (Pointwise Mutual Information)** → keeps top 50
- **Vectorization**: `vec[ctx] += TF-IDF × PMI` + self-tag `__w__` + TF-IDF
**Strategy B — char-bigram TF-IDF** (`TFIDFVectorizer`, `internal/memory/vector/`), used by Document and Indexer layers:
- **char bigram tokenization** (1-2 gram)
- **TF-IDF weights** + **inverted index**
| Location | File | Purpose | Algorithm |
|----------|------|---------|-----------|
| Context Prune | `context.go:161` | Trim low-relevance context events | CosineSimilarity(queryVec, evt.Vector) |
| Context Prune | `context.go:161` | Trim low-relevance context events | LocalWordEmbedder → CosineSimilarity(queryVec, evt.Vector) |
| DocStore Query | `document.go:198` | Recall related content from document memory | InvertedIndex + CosineSimilarity |
| Indexer Entity Search | `indexer.go:149` | Recall related entities from Graph | InvertedIndex + CosineSimilarity |
| Entity Similarity Detection | `agent.go:2297` | Detect similar entities in Graph | Bigram Jaccard (>0.75 → consolidation) |
@ -132,7 +144,8 @@ TF-IDF is the core algorithm running through all three memory layers, used in 4
`internal/agent/core/context.go``RelevanceContext`
- Maintains recent event list, writes JSON on each Append/Prune to prevent data loss
- TF-IDF relevance scoring on user input, keeps topK
- Word embedding relevance scoring on user input (LocalWordEmbedder → CosineSimilarity), keeps topK
- Protects last 10 events from eviction; excess candidates are sorted by relevance and archived to document memory
### Document Layer

View File

@ -19,7 +19,7 @@ Core architecture: a long-running kernel process (`homed`) that connects to vari
The significance: the kernel stays pure (zero IO, only orchestration and memory), plugins stay flexible (each does its job, hot-loadable), with no cross-contamination.
**Three-Layer Memory Architecture** — Solves the memory decay problem for long-running agents:
- **Context Layer**: In-memory TF-IDF scored event window, maintains recent context in real-time, low-relevance events automatically sink to the next layer
- **Context Layer**: In-memory local word embedding scored event window (jieba + TF-IDF + PMI → CosineSimilarity), maintains recent context in real-time, low-relevance events automatically sink to the next layer
- **Document Layer**: JSON files + TF-IDF vector-indexed temporary memory, supports explicit submission and implicit archival, cold data distills to Graph
- **Graph Layer**: SQLite graph database, persists entities and relations, BFS traversal recall, distillation pipeline extracts triples from conversations
@ -33,7 +33,7 @@ Code is in the project root, implemented in Go.
- Maintains a message loop (`eventLoop`), queuing input from the IO layer
- Each input goes through the full processing pipeline: memory recall → persona injection → LLM call → tool execution → output delivery
- LLM calls abstracted through Provider interface, supports 8 LLM sources with automatic fallback
- Context management (`context.go`) based on TF-IDF scoring, automatic pruning of low-relevance events
- Context management (`context.go`) based on word embedding scoring (LocalWordEmbedder → CosineSimilarity), automatic pruning of low-relevance events
**Memory System** (`internal/memory/`):
- **GraphDB** (`graph.go`) — SQLite, entities + relations tables, BFS traversal

View File

@ -117,13 +117,25 @@ eventLoop() → processTextInput()
→ 三元组 → GraphDB.Commit
```
### TF-IDF 向量化char 1-2 gram
### 向量化算法:两种策略
TF-IDF 是贯穿三层记忆的核心算法,在 4 个独立位置以不同方式使用:
向量化在 4 个独立位置以不同方式使用:
**策略 A — 局部词嵌入**`LocalWordEmbedder`, `internal/memory/embedder.go`),用于 Context 层:
- **jieba 分词** → 去除停用词和单字
- **TF-IDF** 作为基础词权重
- **滑动窗口size=5** 统计词对共现 → **PMI点互信息** → 保留 top 50
- **向量化**`vec[ctx] += TF-IDF × PMI` + 自身上标 `__w__` + TF-IDF
**策略 B — char-bigram TF-IDF**`TFIDFVectorizer`, `internal/memory/vector/`),用于 Document 和 Indexer 层:
- **char bigram 分词**1-2 gram
- **TF-IDF 权重** + **倒排索引**
| 位置 | 文件 | 用途 | 算法 |
|------|------|------|------|
| Context Prune | `context.go:161` | 裁剪低相关性上下文事件 | CosineSimilarity(queryVec, evt.Vector) |
| Context Prune | `context.go:161` | 裁剪低相关性上下文事件 | LocalWordEmbedder → CosineSimilarity(queryVec, evt.Vector) |
| DocStore Query | `document.go:198` | 从文档记忆召回相关内容 | InvertedIndex + CosineSimilarity |
| Indexer 实体搜索 | `indexer.go:149` | 从Graph召回相关实体 | InvertedIndex + CosineSimilarity |
| 实体相似度检测 | `agent.go:2297` | 检测Graph中相似实体 | Bigram Jaccard (>0.75 → consolidation) |
@ -132,7 +144,8 @@ TF-IDF 是贯穿三层记忆的核心算法,在 4 个独立位置以不同方
`internal/agent/core/context.go``RelevanceContext`
- 维护最近事件列表,每次 Append/Prune 写入 JSON 防丢
- 用户输入时做 TF-IDF 相关性评分,保留 topK
- 用户输入时做词嵌入相关性评分LocalWordEmbedder → CosineSimilarity,保留 topK
- 保护最近 10 条记录免于淘汰,超出部分按相关性排序归档到文档记忆
### Document 层

View File

@ -19,7 +19,7 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
这一划分的意义:内核保持纯粹(零 IO只做编排和记忆插件保持灵活各司其职热加载互不污染。
**三层记忆架构** — 解决 Agent 长期运行的记忆衰减:
- **Context 层**:内存中 TF-IDF 评分的事件窗口,实时维护最近上下文,低相关性事件自动下沉到下一层
- **Context 层**:内存中局部词嵌入评分的事件窗口jieba + TF-IDF + PMI → CosineSimilarity,实时维护最近上下文,低相关性事件自动下沉到下一层
- **Document 层**JSON 文件 + TF-IDF 向量索引的临时记忆,支持显式提交和隐式归档,冷数据蒸馏到 Graph
- **Graph 层**SQLite 图数据库持久化实体entities和关系relationsBFS 遍历召回,蒸馏管道从对话中提取三元组
@ -33,7 +33,7 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
- 维护一个消息循环(`eventLoop`),从 IO 层排队接收输入
- 每次输入走完整的处理管道:记忆召回 → 人格注入 → LLM 调用 → 工具执行 → 输出发送
- LLM 调用通过 Provider 接口抽象,支持 8 个 LLM 源自动降级
- 上下文管理(`context.go`)基于 TF-IDF 评分,自动剪枝低相关性事件
- 上下文管理(`context.go`)基于词嵌入评分LocalWordEmbedder → CosineSimilarity,自动剪枝低相关性事件
**记忆系统** (`internal/memory/`)
- **GraphDB** (`graph.go`) — SQLiteentities + relations 表BFS 遍历