docs: update vectorization algorithm - LocalWordEmbedder (jieba + TF-IDF + PMI co-occurrence)

Context layer uses LocalWordEmbedder not plain TF-IDF:
- jieba segmentation, TF-IDF weights, sliding window PMI co-occurrence
- Document/Indexer layers still use char-bigram TF-IDF (TFIDFVectorizer)
- Add 'protect last 10 events' detail to Context pruning description
This commit is contained in:
root
2026-07-17 07:02:53 +08:00
parent dd01388e76
commit fb2a02af22
4 changed files with 38 additions and 12 deletions

View File

@ -117,13 +117,25 @@ Setting `ctx.Response` at any stage jumps to `after_output`.
→ triples → GraphDB.Commit
```
### TF-IDF Vectorization (char 1-2 gram)
### Vectorization: Two Strategies
TF-IDF is the core algorithm running through all three memory layers, used in 4 independent locations:
Vectorization is used in 4 independent locations with different strategies:
**Strategy A — Local Word Embedding** (`LocalWordEmbedder`, `internal/memory/embedder.go`), used by Context layer:
- **jieba tokenization** → removes stop words and single characters
- **TF-IDF** as base word weight
- **Sliding window (size=5)** counts word co-occurrence → **PMI (Pointwise Mutual Information)** → keeps top 50
- **Vectorization**: `vec[ctx] += TF-IDF × PMI` + self-tag `__w__` + TF-IDF
**Strategy B — char-bigram TF-IDF** (`TFIDFVectorizer`, `internal/memory/vector/`), used by Document and Indexer layers:
- **char bigram tokenization** (1-2 gram)
- **TF-IDF weights** + **inverted index**
| Location | File | Purpose | Algorithm |
|----------|------|---------|-----------|
| Context Prune | `context.go:161` | Trim low-relevance context events | CosineSimilarity(queryVec, evt.Vector) |
| Context Prune | `context.go:161` | Trim low-relevance context events | LocalWordEmbedder → CosineSimilarity(queryVec, evt.Vector) |
| DocStore Query | `document.go:198` | Recall related content from document memory | InvertedIndex + CosineSimilarity |
| Indexer Entity Search | `indexer.go:149` | Recall related entities from Graph | InvertedIndex + CosineSimilarity |
| Entity Similarity Detection | `agent.go:2297` | Detect similar entities in Graph | Bigram Jaccard (>0.75 → consolidation) |
@ -132,7 +144,8 @@ TF-IDF is the core algorithm running through all three memory layers, used in 4
`internal/agent/core/context.go` — `RelevanceContext`
- Maintains recent event list, writes JSON on each Append/Prune to prevent data loss
- TF-IDF relevance scoring on user input, keeps topK
- Word embedding relevance scoring on user input (LocalWordEmbedder → CosineSimilarity), keeps topK
- Protects last 10 events from eviction; excess candidates are sorted by relevance and archived to document memory
### Document Layer

View File

@ -19,7 +19,7 @@ Core architecture: a long-running kernel process (`homed`) that connects to vari
The significance: the kernel stays pure (zero IO, only orchestration and memory), plugins stay flexible (each does its job, hot-loadable), with no cross-contamination.
**Three-Layer Memory Architecture** — Solves the memory decay problem for long-running agents:
- **Context Layer**: In-memory TF-IDF scored event window, maintains recent context in real-time, low-relevance events automatically sink to the next layer
- **Context Layer**: In-memory local word embedding scored event window (jieba + TF-IDF + PMI → CosineSimilarity), maintains recent context in real-time, low-relevance events automatically sink to the next layer
- **Document Layer**: JSON files + TF-IDF vector-indexed temporary memory, supports explicit submission and implicit archival, cold data distills to Graph
- **Graph Layer**: SQLite graph database, persists entities and relations, BFS traversal recall, distillation pipeline extracts triples from conversations
@ -33,7 +33,7 @@ Code is in the project root, implemented in Go.
- Maintains a message loop (`eventLoop`), queuing input from the IO layer
- Each input goes through the full processing pipeline: memory recall → persona injection → LLM call → tool execution → output delivery
- LLM calls abstracted through Provider interface, supports 8 LLM sources with automatic fallback
- Context management (`context.go`) based on TF-IDF scoring, automatic pruning of low-relevance events
- Context management (`context.go`) based on word embedding scoring (LocalWordEmbedder → CosineSimilarity), automatic pruning of low-relevance events
**Memory System** (`internal/memory/`):
- **GraphDB** (`graph.go`) — SQLite, entities + relations tables, BFS traversal