mirror of
https://gitcode.com/JianFeeeee/HomeAgent.git
synced 2026-09-21 01:18:08 +00:00
docs: 同步仓库文档到 v1.1.1,补媒体记忆与接口扩展规则
五处文档此前停在 v1.0.0,而 v1.1.0/v1.1.1 都已发布并在现网运行。 本轮补齐三层记忆的媒体架构、模型工具的媒体参数、通信面 method 数, 以及冻结解除后的替代约束。 ## 中英双语 OVERVIEW.md 三层记忆段只写了 Context/Document/Graph 三层,而 v1.1.0 起图片/音频是 三层里的一类节点。补上媒体记忆的架构概要:CAS + 引用计数 GC + 标记格式 (描述文本才是持久语义记忆,blob 是可淘汰的缓存)+ 插件边界贯通。 ## 中英双语 ARCHITECTURE.md - 「记忆工具」表补 `memory_commit`/`doc_commit` 的 `media_digests` 与 `sentence_text`(后者从未暴露给模型,而它是媒体绑定链的必经环节) - 「三层记忆」段新增媒体记忆子节:CAS 设计表(寻址/完整性/写入原子性/引用/GC)、 标记格式、可选性(`enabled=false` 时整条链路静默退化) - 「三个通信面」从 51 改为 55 个 method,新增四个 media method 的说明 - 「SDK 四通道」代码示例补媒体注入三方法 + 与 SetToolBlocks 的区别 ## plugin-interface-matrix.md - 状态从「完成 v2」改「完成 v3」,v3 记录 v1.1.1 的接口扩展 - §二 A3 DocMemoryAPI 补 InsertWithMedia - §二 A4 补三个媒体注入方法 + 为何不能搭 SetToolBlocks 的车 - §二 A5 补 MediaAttachment 类型 + Triple/Doc/TextEvent 的扩展字段 - §六「新获得的能力」补 SetToolBlocks 已落地、媒体入记忆、插件主动发起 带媒体的对话 - §七 冻结检查点补第 5 条(冻结已解除,取代它的是 §九) - 新增 §九「v1.1.x 的接口扩展规则」:冻结解除后的三条硬约束 (只增不减签名不改 / 新增方法方向 / 模板接线六处失败链)+ 验证方式 (存量插件 17/17、旧产物 4/4 建链、模板断言、压测 -race) ## 为何分立两笔 commit 前一笔(SDK 仓 README + 内核 README)改的是给**插件开发者**看的文本, 本笔改的是给**架构师与维护者**看的技术文档。受众与改动层次不同, 放在同一个 commit 会让追溯时看不出"文档在哪一层跟上了代码"。
This commit is contained in:
@ -189,6 +189,39 @@ All vectorization unified under `StaticEmbedder` (`internal/memory/static_embedd
|
||||
| `doc_query` | Search from Document |
|
||||
| `doc_commit` | Write to Document |
|
||||
|
||||
Since v1.1.1 `memory_commit` and `doc_commit` accept `media_digests`, and the kernel appends the
|
||||
`[<mime> <short digest>] <description>` marker into the sentence/body — **the kernel builds the
|
||||
marker, the model only supplies the digest**. Requiring the caller to know the format would mean a
|
||||
single typo silently breaks reference binding with no error anywhere in the chain. `memory_commit`
|
||||
also gained `sentence_text`: media references hang off a sentence, so with no sentence there is
|
||||
nowhere to attach them.
|
||||
|
||||
### Media Memory (since v1.1.0)
|
||||
|
||||
`internal/memory/media/` — `Store`, content-addressed (CAS)
|
||||
|
||||
| Concern | Approach | Why |
|
||||
|---|---|---|
|
||||
| Addressing | sha256 digest; metadata in SQLite, blobs on disk | Identical bytes stored once; metadata must be queryable, blobs must not live in the database |
|
||||
| Integrity | Every `Get` re-verifies the digest | Silently returning corrupt data on disk damage is far worse than an error |
|
||||
| Write atomicity | `.tmp` + rename | A half-written file taken as complete content would permanently poison that digest |
|
||||
| References | `owner_kind/owner_id/digest` composite primary key, `AddRef` idempotent | Three owner kinds: `context` (context events), `document`, `graph_sentence` |
|
||||
| GC | Two-stage with `minAge`, **referenced items are never deleted** | Description text stays in the text layers while blobs may be evicted — semantic memory and byte cache are decoupled |
|
||||
|
||||
**How media is represented in plain-text memory** is the marker `[<mime> <short digest>] <description>`:
|
||||
|
||||
```
|
||||
[image/png a1b2c3d4e5f6] a purple-blue-red three-band chart
|
||||
```
|
||||
|
||||
Why it must ride on text: `Doc.Content`, `sentences.text` and text memory's `Input` are all strings
|
||||
— there is no field to carry structured data. **The description text is the durable semantic
|
||||
memory** (retrieval uses it); the digest is the key back to the bytes (reverse lookup uses it).
|
||||
After capacity GC evicts a blob, the description remains in the L0/L2/L3 text.
|
||||
|
||||
The media store is **optional throughout**: with `core.memory.media.enabled=false` or no
|
||||
configuration, the whole chain silently degrades to plain-text behaviour — no errors, no panics.
|
||||
|
||||
### Other Memory Layers
|
||||
|
||||
- **Social** (`internal/memory/social/social.go`) — Persona traits and relationship network, wraps GraphDB entity types
|
||||
@ -296,7 +329,7 @@ Common ground:
|
||||
|
||||
| Plane | Mechanism | Why this choice |
|
||||
|---|---|---|
|
||||
| Control | stdio JSON-RPC (NDJSON frames), 51 `core.*` methods | The process boundary *is* the ABI boundary—no need to maintain three platform-specific dynamic-library loaders |
|
||||
| Control | stdio JSON-RPC (NDJSON frames), 55 `core.*` methods | The process boundary *is* the ABI boundary—no need to maintain three platform-specific dynamic-library loaders |
|
||||
| Data | Shared memory segment, **one segment shared by all subprocesses** | One segment per plugin would degrade "kernel ctx → segment → plugin mutates → read back" into the copy model under concurrency, reproducing lost updates exactly |
|
||||
| Notification | Event ring + platform notify (Linux eventfd / macOS pipe / Windows Event) | The kernel must never block on a consumer: streaming output publishes per token, so any wait shows up as stutter |
|
||||
|
||||
@ -334,8 +367,23 @@ sdk.Memory().Recall/Commit
|
||||
sdk.Knowledge().Search/Create
|
||||
sdk.Settings().Get/Set/List
|
||||
sdk.RegisterOutputChannel("qq", sdk.CapText|sdk.CapAudio|sdk.CapImage, "QQ channel, see output_send__qq_help for details", handler)
|
||||
|
||||
// v1.1.1 media APIs (all additive, no signature changes)
|
||||
sdk.DocMemory().InsertWithMedia(doc, attachments) // attachments with Data land in CAS; Digest-only ones reference existing content
|
||||
sdk.InjectInputMedia(source, channel, text, blocks) // media reaches the model in *this* turn
|
||||
sdk.InjectInputMediaSync(...) // same, and waits for the reply
|
||||
sdk.InjectInterruptMedia(...) // media-bearing interrupt, can preempt current processing
|
||||
```
|
||||
|
||||
How media injection differs from `SetToolBlocks`: the latter is only callable inside a tool handler
|
||||
and its media reaches the model with the **next** tool message; these three let a plugin
|
||||
**initiate a turn that carries media** — it goes out with this turn's message and is automatically
|
||||
stored in CAS with a memory reference attached. `Triple` and `Doc` gained `MediaDigests` /
|
||||
`Attachments` correspondingly.
|
||||
|
||||
`internal/sdk/` is the bridge implementation for this layer and is not subject to the public
|
||||
interface freeze (see `docs/git-branching.md` §6).
|
||||
|
||||
### Plugin Interface
|
||||
|
||||
```go
|
||||
|
||||
@ -25,6 +25,19 @@ The significance lies in clear responsibility boundaries: the kernel focuses on
|
||||
|
||||
Three progressive layers — context, cold archive, long-term graph memory — form an information decay and consolidation pipeline from short-term to persistent storage.
|
||||
|
||||
**Media Memory (since v1.1.0)** — Images and audio are not attachments; they are a kind of node in all three layers:
|
||||
- **Content-addressed store (CAS)**: addressed by digest, metadata in SQLite and blobs on disk, identical bytes
|
||||
stored once. Every `Get` re-verifies the digest (silently returning corrupt data is worse than an error).
|
||||
- **Reference-counted GC**: `owner_kind/owner_id/digest` is the primary key; context events, documents and graph
|
||||
sentences each hold their own references. **Referenced items are never deleted** — only unowned content past
|
||||
`minAge` is reclaimed.
|
||||
- **The description text is the durable semantic memory**: what the vision model produced is written into
|
||||
plain-text memory as a `[<mime> <short digest>] <description>` marker and participates in vector retrieval and
|
||||
distillation; the blob is only a cache that capacity GC may evict. Months later "that purple-blue-red
|
||||
three-band chart" is still findable — via the description, not the bytes.
|
||||
- **Reaches the plugin boundary since v1.1.1**: plugins read and write media through `InsertWithMedia` /
|
||||
`InjectInputMedia`; the model attaches media via the `media_digests` argument of `memory_commit` / `doc_commit`.
|
||||
|
||||
## What It Actually Does
|
||||
|
||||
Code is in the project root, implemented in Go.
|
||||
|
||||
@ -189,6 +189,36 @@ eventLoop() → processTextInput()
|
||||
| `doc_query` | 从 Document 搜索 |
|
||||
| `doc_commit` | 写入 Document |
|
||||
|
||||
`memory_commit` 与 `doc_commit` 自 v1.1.1 起接受 `media_digests`,并由内核把
|
||||
`[<mime> <短digest>] <描述>` 标记补进句子/正文——**标记由内核拼,模型只给 digest**。
|
||||
要求调用方知道格式,等于让一个拼写错误静默切断引用绑定而全链路无人报错。
|
||||
`memory_commit` 同时新增 `sentence_text`:媒体引用挂在句子上,没有句子就无处可挂。
|
||||
|
||||
### 媒体记忆(v1.1.0 起)
|
||||
|
||||
`internal/memory/media/` — `Store`,内容寻址(CAS)
|
||||
|
||||
| 关注点 | 做法 | 为何 |
|
||||
|---|---|---|
|
||||
| 寻址 | sha256 digest,元数据在 SQLite、blob 在磁盘 | 相同字节只存一份;元数据要可查询,blob 不该进数据库 |
|
||||
| 完整性 | 每次 `Get` 重校 digest | 磁盘损坏时静默返回脏数据比报错危险得多 |
|
||||
| 写入原子性 | `.tmp` + rename | 半个文件被当成完整内容会永久污染那个 digest |
|
||||
| 引用 | `owner_kind/owner_id/digest` 三元组主键,`AddRef` 幂等 | 三个 owner 类型:`context`(上下文事件)、`document`(文档)、`graph_sentence`(图谱句子) |
|
||||
| GC | 两阶段 + `minAge`,**有引用者绝不删** | 描述文本留在文本层,blob 可淘汰——语义记忆与字节缓存分离 |
|
||||
|
||||
**媒体在纯文本记忆里的表示**是标记 `[<mime> <短digest>] <描述>`:
|
||||
|
||||
```
|
||||
[image/png a1b2c3d4e5f6] 一张紫蓝红三色带图
|
||||
```
|
||||
|
||||
之所以必须借文本承载:`Doc.Content`、`sentences.text`、文本记忆的 `Input` 全是字符串,
|
||||
没有字段能挂结构化数据。**描述文本才是持久的语义记忆**(检索靠它),digest 是回到字节的
|
||||
钥匙(反查靠它)。blob 被容量 GC 淘汰后,描述仍留在 L0/L2/L3 的文本里。
|
||||
|
||||
媒体存储**全程可选**:`core.memory.media.enabled=false` 或未配置时,整条链路静默退化为
|
||||
纯文本行为,不报错不 panic。
|
||||
|
||||
### 其他记忆层
|
||||
|
||||
- **Social** (`internal/memory/social/social.go`) — 人格特质和关系网,包装 GraphDB 实体类型
|
||||
@ -294,10 +324,14 @@ Lua 脚本插件加载:`internal/plugin/` → gopher-lua 解释器执行 `main
|
||||
|
||||
| 面 | 机制 | 为何这么选 |
|
||||
|---|---|---|
|
||||
| 控制面 | stdio JSON-RPC(NDJSON 帧),51 个 `core.*` method | 进程边界即 ABI 边界,无需维护三套平台特定的动态库加载代码 |
|
||||
| 控制面 | stdio JSON-RPC(NDJSON 帧),55 个 `core.*` method | 进程边界即 ABI 边界,无需维护三套平台特定的动态库加载代码 |
|
||||
| 数据面 | 共享内存段,**全部子进程共用一块** | 每插件一段会让「内核 ctx → 段 → 插件改 → 回读 ctx」在多插件下退化成副本模型,lost update 原样复现 |
|
||||
| 通知面 | 事件环 + 平台通知(Linux eventfd / macOS pipe / Windows Event) | 内核发事件绕不等消费者,流式输出逐 token 发布时任何等待都会造成卡顿 |
|
||||
|
||||
v1.1.1 新增 4 个 method(51 → 55):`doc.insertWithMedia`、`io.injectMedia`、
|
||||
`io.injectMediaSync`、`io.injectInterruptMedia`。**媒体块走 JSON 而非共享段二进制通道**——
|
||||
data URL 本身已是 base64 文本,包进二进制传输省不了空间,还要跟其余 51 个 method 分道。
|
||||
|
||||
**子进程生命周期管理**:
|
||||
- 每子进程一根专职 `waitLoop`(`cmd.Wait()` 唯一调用点)——不依赖 stdout EOF,
|
||||
因为插件 fork 的孙子进程(browser 拉 chromium、editdoc 拉 python)继承同一 stdout,
|
||||
@ -330,8 +364,20 @@ sdk.Memory().Recall/Commit
|
||||
sdk.Knowledge().Search/Create
|
||||
sdk.Settings().Get/Set/List
|
||||
sdk.RegisterOutputChannel("qq", sdk.CapText|sdk.CapAudio|sdk.CapImage, "QQ消息通道,详见 output_send__qq_help", handler)
|
||||
|
||||
// v1.1.1 媒体接口(全部新增,无签名变更)
|
||||
sdk.DocMemory().InsertWithMedia(doc, attachments) // 带 Data 的落进 CAS,只给 Digest 的引用已有内容
|
||||
sdk.InjectInputMedia(source, channel, text, blocks) // 媒体在「本轮」就发给模型
|
||||
sdk.InjectInputMediaSync(...) // 同上并同步等回复
|
||||
sdk.InjectInterruptMedia(...) // 带媒体的中断,可抢占当前处理
|
||||
```
|
||||
|
||||
媒体注入与 `SetToolBlocks` 的区别:后者只能在工具处理函数内部调用,且媒体要等**下一条**
|
||||
tool message 才到模型手上;前三个是插件**主动发起一轮带媒体的对话**,媒体随本轮消息发出,
|
||||
并自动落进 CAS、挂上媒体记忆引用。`Triple` 与 `Doc` 相应新增 `MediaDigests`、`Attachments`。
|
||||
|
||||
`internal/sdk/` 是这层的桥接实现,不受公开接口冻结约束(见 `docs/git-branching.md` §六)。
|
||||
|
||||
### Plugin 接口
|
||||
|
||||
```go
|
||||
|
||||
@ -25,6 +25,17 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
|
||||
|
||||
三层递进:上下文 → 冷归档 → 长期图记忆,构成从短期到持久的信息衰减与整合管道。
|
||||
|
||||
**媒体记忆(v1.1.0 起)** — 图片/音频不是附属物,而是三层里的一类节点:
|
||||
- **内容寻址存储(CAS)**:digest 寻址,元数据在 SQLite、blob 在磁盘,相同字节只存一份,
|
||||
每次 `Get` 重校 digest(磁盘损坏静默返回脏数据比报错更危险)
|
||||
- **引用计数 GC**:`owner_kind/owner_id/digest` 三元组为主键,上下文事件/文档/图谱句子各自持引用;
|
||||
**有引用者绝不删除**,仅回收无主且超过 `minAge` 的内容
|
||||
- **描述文本才是持久语义记忆**:视觉模型生成的描述以
|
||||
`[<mime> <短digest>] <描述>` 标记形式写进纯文本记忆,参与向量检索与蒸馏;
|
||||
blob 只是可被容量 GC 淘汰的缓存。几个月后“那张紫蓝红三色带图”仍可检索,靠的是描述而不是字节
|
||||
- **v1.1.1 起贯通插件边界**:插件可通过 `InsertWithMedia` / `InjectInputMedia` 读写媒体,
|
||||
模型可用 `memory_commit` / `doc_commit` 的 `media_digests` 参数关联媒体
|
||||
|
||||
## 它实际做了什么
|
||||
|
||||
代码位于项目仓库根目录,Go 语言实现。
|
||||
|
||||
Reference in New Issue
Block a user