From fb2a02af2254f692a348f4756b2c5d77fa5ecf79 Mon Sep 17 00:00:00 2001
From: root <root@qq.com>
Date: Fri, 17 Jul 2026 07:02:53 +0800
Subject: [PATCH] docs: update vectorization algorithm - LocalWordEmbedder
 (jieba + TF-IDF + PMI co-occurrence)

Context layer uses LocalWordEmbedder not plain TF-IDF:
- jieba segmentation, TF-IDF weights, sliding window PMI co-occurrence
- Document/Indexer layers still use char-bigram TF-IDF (TFIDFVectorizer)
- Add 'protect last 10 events' detail to Context pruning description
---
 docs/en/ARCHITECTURE.md | 21 +++++++++++++++++----
 docs/en/OVERVIEW.md     |  4 ++--
 docs/zh/ARCHITECTURE.md | 21 +++++++++++++++++----
 docs/zh/OVERVIEW.md     |  4 ++--
 4 files changed, 38 insertions(+), 12 deletions(-)

diff --git a/docs/en/ARCHITECTURE.md b/docs/en/ARCHITECTURE.md
index 1d8fddc..795e5f1 100644
--- a/docs/en/ARCHITECTURE.md
+++ b/docs/en/ARCHITECTURE.md
@@ -117,13 +117,25 @@ Setting `ctx.Response` at any stage jumps to `after_output`.
      → triples → GraphDB.Commit
 ```
 
-### TF-IDF Vectorization (char 1-2 gram)
+### Vectorization: Two Strategies
 
-TF-IDF is the core algorithm running through all three memory layers, used in 4 independent locations:
+Vectorization is used in 4 independent locations with different strategies:
+
+**Strategy A — Local Word Embedding** (`LocalWordEmbedder`, `internal/memory/embedder.go`), used by Context layer:
+
+- **jieba tokenization** → removes stop words and single characters
+- **TF-IDF** as base word weight
+- **Sliding window (size=5)** counts word co-occurrence → **PMI (Pointwise Mutual Information)** → keeps top 50
+- **Vectorization**: `vec[ctx] += TF-IDF × PMI` + self-tag `__w__` + TF-IDF
+
+**Strategy B — char-bigram TF-IDF** (`TFIDFVectorizer`, `internal/memory/vector/`), used by Document and Indexer layers:
+
+- **char bigram tokenization** (1-2 gram)
+- **TF-IDF weights** + **inverted index**
 
 | Location | File | Purpose | Algorithm |
 |----------|------|---------|-----------|
-| Context Prune | `context.go:161` | Trim low-relevance context events | CosineSimilarity(queryVec, evt.Vector) |
+| Context Prune | `context.go:161` | Trim low-relevance context events | LocalWordEmbedder → CosineSimilarity(queryVec, evt.Vector) |
 | DocStore Query | `document.go:198` | Recall related content from document memory | InvertedIndex + CosineSimilarity |
 | Indexer Entity Search | `indexer.go:149` | Recall related entities from Graph | InvertedIndex + CosineSimilarity |
 | Entity Similarity Detection | `agent.go:2297` | Detect similar entities in Graph | Bigram Jaccard (>0.75 → consolidation) |
@@ -132,7 +144,8 @@ TF-IDF is the core algorithm running through all three memory layers, used in 4
 
 `internal/agent/core/context.go` — `RelevanceContext`
 - Maintains recent event list, writes JSON on each Append/Prune to prevent data loss
-- TF-IDF relevance scoring on user input, keeps topK
+- Word embedding relevance scoring on user input (LocalWordEmbedder → CosineSimilarity), keeps topK
+- Protects last 10 events from eviction; excess candidates are sorted by relevance and archived to document memory
 
 ### Document Layer
 
diff --git a/docs/en/OVERVIEW.md b/docs/en/OVERVIEW.md
index 1af0364..c9a2599 100644
--- a/docs/en/OVERVIEW.md
+++ b/docs/en/OVERVIEW.md
@@ -19,7 +19,7 @@ Core architecture: a long-running kernel process (`homed`) that connects to vari
 The significance: the kernel stays pure (zero IO, only orchestration and memory), plugins stay flexible (each does its job, hot-loadable), with no cross-contamination.
 
 **Three-Layer Memory Architecture** — Solves the memory decay problem for long-running agents:
-- **Context Layer**: In-memory TF-IDF scored event window, maintains recent context in real-time, low-relevance events automatically sink to the next layer
+- **Context Layer**: In-memory local word embedding scored event window (jieba + TF-IDF + PMI → CosineSimilarity), maintains recent context in real-time, low-relevance events automatically sink to the next layer
 - **Document Layer**: JSON files + TF-IDF vector-indexed temporary memory, supports explicit submission and implicit archival, cold data distills to Graph
 - **Graph Layer**: SQLite graph database, persists entities and relations, BFS traversal recall, distillation pipeline extracts triples from conversations
 
@@ -33,7 +33,7 @@ Code is in the project root, implemented in Go.
 - Maintains a message loop (`eventLoop`), queuing input from the IO layer
 - Each input goes through the full processing pipeline: memory recall → persona injection → LLM call → tool execution → output delivery
 - LLM calls abstracted through Provider interface, supports 8 LLM sources with automatic fallback
-- Context management (`context.go`) based on TF-IDF scoring, automatic pruning of low-relevance events
+- Context management (`context.go`) based on word embedding scoring (LocalWordEmbedder → CosineSimilarity), automatic pruning of low-relevance events
 
 **Memory System** (`internal/memory/`):
 - **GraphDB** (`graph.go`) — SQLite, entities + relations tables, BFS traversal
diff --git a/docs/zh/ARCHITECTURE.md b/docs/zh/ARCHITECTURE.md
index d2f751f..5360d07 100644
--- a/docs/zh/ARCHITECTURE.md
+++ b/docs/zh/ARCHITECTURE.md
@@ -117,13 +117,25 @@ eventLoop() → processTextInput()
      → 三元组 → GraphDB.Commit
 ```
 
-### TF-IDF 向量化（char 1-2 gram）
+### 向量化算法：两种策略
 
-TF-IDF 是贯穿三层记忆的核心算法，在 4 个独立位置以不同方式使用：
+向量化在 4 个独立位置以不同方式使用：
+
+**策略 A — 局部词嵌入**（`LocalWordEmbedder`, `internal/memory/embedder.go`），用于 Context 层：
+
+- **jieba 分词** → 去除停用词和单字
+- **TF-IDF** 作为基础词权重
+- **滑动窗口（size=5）** 统计词对共现 → **PMI（点互信息）** → 保留 top 50
+- **向量化**：`vec[ctx] += TF-IDF × PMI` + 自身上标 `__w__` + TF-IDF
+
+**策略 B — char-bigram TF-IDF**（`TFIDFVectorizer`, `internal/memory/vector/`），用于 Document 和 Indexer 层：
+
+- **char bigram 分词**（1-2 gram）
+- **TF-IDF 权重** + **倒排索引**
 
 | 位置 | 文件 | 用途 | 算法 |
 |------|------|------|------|
-| Context Prune | `context.go:161` | 裁剪低相关性上下文事件 | CosineSimilarity(queryVec, evt.Vector) |
+| Context Prune | `context.go:161` | 裁剪低相关性上下文事件 | LocalWordEmbedder → CosineSimilarity(queryVec, evt.Vector) |
 | DocStore Query | `document.go:198` | 从文档记忆召回相关内容 | InvertedIndex + CosineSimilarity |
 | Indexer 实体搜索 | `indexer.go:149` | 从Graph召回相关实体 | InvertedIndex + CosineSimilarity |
 | 实体相似度检测 | `agent.go:2297` | 检测Graph中相似实体 | Bigram Jaccard (>0.75 → consolidation) |
@@ -132,7 +144,8 @@ TF-IDF 是贯穿三层记忆的核心算法，在 4 个独立位置以不同方
 
 `internal/agent/core/context.go` — `RelevanceContext`
 - 维护最近事件列表，每次 Append/Prune 写入 JSON 防丢
-- 用户输入时做 TF-IDF 相关性评分，保留 topK
+- 用户输入时做词嵌入相关性评分（LocalWordEmbedder → CosineSimilarity），保留 topK
+- 保护最近 10 条记录免于淘汰，超出部分按相关性排序归档到文档记忆
 
 ### Document 层
 
diff --git a/docs/zh/OVERVIEW.md b/docs/zh/OVERVIEW.md
index d9c0d42..c2fab68 100644
--- a/docs/zh/OVERVIEW.md
+++ b/docs/zh/OVERVIEW.md
@@ -19,7 +19,7 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
 这一划分的意义：内核保持纯粹（零 IO，只做编排和记忆），插件保持灵活（各司其职，热加载），互不污染。
 
 **三层记忆架构** — 解决 Agent 长期运行的记忆衰减：
-- **Context 层**：内存中 TF-IDF 评分的事件窗口，实时维护最近上下文，低相关性事件自动下沉到下一层
+- **Context 层**：内存中局部词嵌入评分的事件窗口（jieba + TF-IDF + PMI → CosineSimilarity），实时维护最近上下文，低相关性事件自动下沉到下一层
 - **Document 层**：JSON 文件 + TF-IDF 向量索引的临时记忆，支持显式提交和隐式归档，冷数据蒸馏到 Graph
 - **Graph 层**：SQLite 图数据库，持久化实体（entities）和关系（relations），BFS 遍历召回，蒸馏管道从对话中提取三元组
 
@@ -33,7 +33,7 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
 - 维护一个消息循环（`eventLoop`），从 IO 层排队接收输入
 - 每次输入走完整的处理管道：记忆召回 → 人格注入 → LLM 调用 → 工具执行 → 输出发送
 - LLM 调用通过 Provider 接口抽象，支持 8 个 LLM 源自动降级
-- 上下文管理（`context.go`）基于 TF-IDF 评分，自动剪枝低相关性事件
+- 上下文管理（`context.go`）基于词嵌入评分（LocalWordEmbedder → CosineSimilarity），自动剪枝低相关性事件
 
 **记忆系统** (`internal/memory/`)：
 - **GraphDB** (`graph.go`) — SQLite，entities + relations 表，BFS 遍历