mirror of
https://gitcode.com/JianFeeeee/HomeAgent.git
synced 2026-09-21 17:38:10 +00:00
docs: update vectorization algorithm - LocalWordEmbedder (jieba + TF-IDF + PMI co-occurrence)
Context layer uses LocalWordEmbedder not plain TF-IDF: - jieba segmentation, TF-IDF weights, sliding window PMI co-occurrence - Document/Indexer layers still use char-bigram TF-IDF (TFIDFVectorizer) - Add 'protect last 10 events' detail to Context pruning description
This commit is contained in:
@ -117,13 +117,25 @@ Setting `ctx.Response` at any stage jumps to `after_output`.
|
||||
→ triples → GraphDB.Commit
|
||||
```
|
||||
|
||||
### TF-IDF Vectorization (char 1-2 gram)
|
||||
### Vectorization: Two Strategies
|
||||
|
||||
TF-IDF is the core algorithm running through all three memory layers, used in 4 independent locations:
|
||||
Vectorization is used in 4 independent locations with different strategies:
|
||||
|
||||
**Strategy A — Local Word Embedding** (`LocalWordEmbedder`, `internal/memory/embedder.go`), used by Context layer:
|
||||
|
||||
- **jieba tokenization** → removes stop words and single characters
|
||||
- **TF-IDF** as base word weight
|
||||
- **Sliding window (size=5)** counts word co-occurrence → **PMI (Pointwise Mutual Information)** → keeps top 50
|
||||
- **Vectorization**: `vec[ctx] += TF-IDF × PMI` + self-tag `__w__` + TF-IDF
|
||||
|
||||
**Strategy B — char-bigram TF-IDF** (`TFIDFVectorizer`, `internal/memory/vector/`), used by Document and Indexer layers:
|
||||
|
||||
- **char bigram tokenization** (1-2 gram)
|
||||
- **TF-IDF weights** + **inverted index**
|
||||
|
||||
| Location | File | Purpose | Algorithm |
|
||||
|----------|------|---------|-----------|
|
||||
| Context Prune | `context.go:161` | Trim low-relevance context events | CosineSimilarity(queryVec, evt.Vector) |
|
||||
| Context Prune | `context.go:161` | Trim low-relevance context events | LocalWordEmbedder → CosineSimilarity(queryVec, evt.Vector) |
|
||||
| DocStore Query | `document.go:198` | Recall related content from document memory | InvertedIndex + CosineSimilarity |
|
||||
| Indexer Entity Search | `indexer.go:149` | Recall related entities from Graph | InvertedIndex + CosineSimilarity |
|
||||
| Entity Similarity Detection | `agent.go:2297` | Detect similar entities in Graph | Bigram Jaccard (>0.75 → consolidation) |
|
||||
@ -132,7 +144,8 @@ TF-IDF is the core algorithm running through all three memory layers, used in 4
|
||||
|
||||
`internal/agent/core/context.go` — `RelevanceContext`
|
||||
- Maintains recent event list, writes JSON on each Append/Prune to prevent data loss
|
||||
- TF-IDF relevance scoring on user input, keeps topK
|
||||
- Word embedding relevance scoring on user input (LocalWordEmbedder → CosineSimilarity), keeps topK
|
||||
- Protects last 10 events from eviction; excess candidates are sorted by relevance and archived to document memory
|
||||
|
||||
### Document Layer
|
||||
|
||||
|
||||
@ -19,7 +19,7 @@ Core architecture: a long-running kernel process (`homed`) that connects to vari
|
||||
The significance: the kernel stays pure (zero IO, only orchestration and memory), plugins stay flexible (each does its job, hot-loadable), with no cross-contamination.
|
||||
|
||||
**Three-Layer Memory Architecture** — Solves the memory decay problem for long-running agents:
|
||||
- **Context Layer**: In-memory TF-IDF scored event window, maintains recent context in real-time, low-relevance events automatically sink to the next layer
|
||||
- **Context Layer**: In-memory local word embedding scored event window (jieba + TF-IDF + PMI → CosineSimilarity), maintains recent context in real-time, low-relevance events automatically sink to the next layer
|
||||
- **Document Layer**: JSON files + TF-IDF vector-indexed temporary memory, supports explicit submission and implicit archival, cold data distills to Graph
|
||||
- **Graph Layer**: SQLite graph database, persists entities and relations, BFS traversal recall, distillation pipeline extracts triples from conversations
|
||||
|
||||
@ -33,7 +33,7 @@ Code is in the project root, implemented in Go.
|
||||
- Maintains a message loop (`eventLoop`), queuing input from the IO layer
|
||||
- Each input goes through the full processing pipeline: memory recall → persona injection → LLM call → tool execution → output delivery
|
||||
- LLM calls abstracted through Provider interface, supports 8 LLM sources with automatic fallback
|
||||
- Context management (`context.go`) based on TF-IDF scoring, automatic pruning of low-relevance events
|
||||
- Context management (`context.go`) based on word embedding scoring (LocalWordEmbedder → CosineSimilarity), automatic pruning of low-relevance events
|
||||
|
||||
**Memory System** (`internal/memory/`):
|
||||
- **GraphDB** (`graph.go`) — SQLite, entities + relations tables, BFS traversal
|
||||
|
||||
@ -117,13 +117,25 @@ eventLoop() → processTextInput()
|
||||
→ 三元组 → GraphDB.Commit
|
||||
```
|
||||
|
||||
### TF-IDF 向量化(char 1-2 gram)
|
||||
### 向量化算法:两种策略
|
||||
|
||||
TF-IDF 是贯穿三层记忆的核心算法,在 4 个独立位置以不同方式使用:
|
||||
向量化在 4 个独立位置以不同方式使用:
|
||||
|
||||
**策略 A — 局部词嵌入**(`LocalWordEmbedder`, `internal/memory/embedder.go`),用于 Context 层:
|
||||
|
||||
- **jieba 分词** → 去除停用词和单字
|
||||
- **TF-IDF** 作为基础词权重
|
||||
- **滑动窗口(size=5)** 统计词对共现 → **PMI(点互信息)** → 保留 top 50
|
||||
- **向量化**:`vec[ctx] += TF-IDF × PMI` + 自身上标 `__w__` + TF-IDF
|
||||
|
||||
**策略 B — char-bigram TF-IDF**(`TFIDFVectorizer`, `internal/memory/vector/`),用于 Document 和 Indexer 层:
|
||||
|
||||
- **char bigram 分词**(1-2 gram)
|
||||
- **TF-IDF 权重** + **倒排索引**
|
||||
|
||||
| 位置 | 文件 | 用途 | 算法 |
|
||||
|------|------|------|------|
|
||||
| Context Prune | `context.go:161` | 裁剪低相关性上下文事件 | CosineSimilarity(queryVec, evt.Vector) |
|
||||
| Context Prune | `context.go:161` | 裁剪低相关性上下文事件 | LocalWordEmbedder → CosineSimilarity(queryVec, evt.Vector) |
|
||||
| DocStore Query | `document.go:198` | 从文档记忆召回相关内容 | InvertedIndex + CosineSimilarity |
|
||||
| Indexer 实体搜索 | `indexer.go:149` | 从Graph召回相关实体 | InvertedIndex + CosineSimilarity |
|
||||
| 实体相似度检测 | `agent.go:2297` | 检测Graph中相似实体 | Bigram Jaccard (>0.75 → consolidation) |
|
||||
@ -132,7 +144,8 @@ TF-IDF 是贯穿三层记忆的核心算法,在 4 个独立位置以不同方
|
||||
|
||||
`internal/agent/core/context.go` — `RelevanceContext`
|
||||
- 维护最近事件列表,每次 Append/Prune 写入 JSON 防丢
|
||||
- 用户输入时做 TF-IDF 相关性评分,保留 topK
|
||||
- 用户输入时做词嵌入相关性评分(LocalWordEmbedder → CosineSimilarity),保留 topK
|
||||
- 保护最近 10 条记录免于淘汰,超出部分按相关性排序归档到文档记忆
|
||||
|
||||
### Document 层
|
||||
|
||||
|
||||
@ -19,7 +19,7 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
|
||||
这一划分的意义:内核保持纯粹(零 IO,只做编排和记忆),插件保持灵活(各司其职,热加载),互不污染。
|
||||
|
||||
**三层记忆架构** — 解决 Agent 长期运行的记忆衰减:
|
||||
- **Context 层**:内存中 TF-IDF 评分的事件窗口,实时维护最近上下文,低相关性事件自动下沉到下一层
|
||||
- **Context 层**:内存中局部词嵌入评分的事件窗口(jieba + TF-IDF + PMI → CosineSimilarity),实时维护最近上下文,低相关性事件自动下沉到下一层
|
||||
- **Document 层**:JSON 文件 + TF-IDF 向量索引的临时记忆,支持显式提交和隐式归档,冷数据蒸馏到 Graph
|
||||
- **Graph 层**:SQLite 图数据库,持久化实体(entities)和关系(relations),BFS 遍历召回,蒸馏管道从对话中提取三元组
|
||||
|
||||
@ -33,7 +33,7 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
|
||||
- 维护一个消息循环(`eventLoop`),从 IO 层排队接收输入
|
||||
- 每次输入走完整的处理管道:记忆召回 → 人格注入 → LLM 调用 → 工具执行 → 输出发送
|
||||
- LLM 调用通过 Provider 接口抽象,支持 8 个 LLM 源自动降级
|
||||
- 上下文管理(`context.go`)基于 TF-IDF 评分,自动剪枝低相关性事件
|
||||
- 上下文管理(`context.go`)基于词嵌入评分(LocalWordEmbedder → CosineSimilarity),自动剪枝低相关性事件
|
||||
|
||||
**记忆系统** (`internal/memory/`):
|
||||
- **GraphDB** (`graph.go`) — SQLite,entities + relations 表,BFS 遍历
|
||||
|
||||
Reference in New Issue
Block a user