docs: 同步仓库文档到 v1.1.1,补媒体记忆与接口扩展规则

五处文档此前停在 v1.0.0,而 v1.1.0/v1.1.1 都已发布并在现网运行。
本轮补齐三层记忆的媒体架构、模型工具的媒体参数、通信面 method 数,
以及冻结解除后的替代约束。

## 中英双语 OVERVIEW.md

三层记忆段只写了 Context/Document/Graph 三层,而 v1.1.0 起图片/音频是
三层里的一类节点。补上媒体记忆的架构概要:CAS + 引用计数 GC + 标记格式
(描述文本才是持久语义记忆,blob 是可淘汰的缓存)+ 插件边界贯通。

## 中英双语 ARCHITECTURE.md

- 「记忆工具」表补 `memory_commit`/`doc_commit` 的 `media_digests` 与
  `sentence_text`(后者从未暴露给模型,而它是媒体绑定链的必经环节)
- 「三层记忆」段新增媒体记忆子节:CAS 设计表(寻址/完整性/写入原子性/引用/GC)、
  标记格式、可选性(`enabled=false` 时整条链路静默退化)
- 「三个通信面」从 51 改为 55 个 method,新增四个 media method 的说明
- 「SDK 四通道」代码示例补媒体注入三方法 + 与 SetToolBlocks 的区别

## plugin-interface-matrix.md

- 状态从「完成 v2」改「完成 v3」,v3 记录 v1.1.1 的接口扩展
- §二 A3 DocMemoryAPI 补 InsertWithMedia
- §二 A4 补三个媒体注入方法 + 为何不能搭 SetToolBlocks 的车
- §二 A5 补 MediaAttachment 类型 + Triple/Doc/TextEvent 的扩展字段
- §六「新获得的能力」补 SetToolBlocks 已落地、媒体入记忆、插件主动发起
  带媒体的对话
- §七 冻结检查点补第 5 条(冻结已解除,取代它的是 §九)
- 新增 §九「v1.1.x 的接口扩展规则」:冻结解除后的三条硬约束
  (只增不减签名不改 / 新增方法方向 / 模板接线六处失败链)+ 验证方式
  (存量插件 17/17、旧产物 4/4 建链、模板断言、压测 -race)

## 为何分立两笔 commit

前一笔(SDK 仓 README + 内核 README)改的是给**插件开发者**看的文本,
本笔改的是给**架构师与维护者**看的技术文档。受众与改动层次不同,
放在同一个 commit 会让追溯时看不出"文档在哪一层跟上了代码"。
This commit is contained in:
JianFeeeee
2026-09-06 15:07:14 +08:00
parent 8cd884027f
commit b889934bba
5 changed files with 198 additions and 6 deletions

View File

@ -189,6 +189,39 @@ All vectorization unified under `StaticEmbedder` (`internal/memory/static_embedd
| `doc_query` | Search from Document |
| `doc_commit` | Write to Document |
Since v1.1.1 `memory_commit` and `doc_commit` accept `media_digests`, and the kernel appends the
`[<mime> <short digest>] <description>` marker into the sentence/body — **the kernel builds the
marker, the model only supplies the digest**. Requiring the caller to know the format would mean a
single typo silently breaks reference binding with no error anywhere in the chain. `memory_commit`
also gained `sentence_text`: media references hang off a sentence, so with no sentence there is
nowhere to attach them.
### Media Memory (since v1.1.0)
`internal/memory/media/``Store`, content-addressed (CAS)
| Concern | Approach | Why |
|---|---|---|
| Addressing | sha256 digest; metadata in SQLite, blobs on disk | Identical bytes stored once; metadata must be queryable, blobs must not live in the database |
| Integrity | Every `Get` re-verifies the digest | Silently returning corrupt data on disk damage is far worse than an error |
| Write atomicity | `.tmp` + rename | A half-written file taken as complete content would permanently poison that digest |
| References | `owner_kind/owner_id/digest` composite primary key, `AddRef` idempotent | Three owner kinds: `context` (context events), `document`, `graph_sentence` |
| GC | Two-stage with `minAge`, **referenced items are never deleted** | Description text stays in the text layers while blobs may be evicted — semantic memory and byte cache are decoupled |
**How media is represented in plain-text memory** is the marker `[<mime> <short digest>] <description>`:
```
[image/png a1b2c3d4e5f6] a purple-blue-red three-band chart
```
Why it must ride on text: `Doc.Content`, `sentences.text` and text memory's `Input` are all strings
— there is no field to carry structured data. **The description text is the durable semantic
memory** (retrieval uses it); the digest is the key back to the bytes (reverse lookup uses it).
After capacity GC evicts a blob, the description remains in the L0/L2/L3 text.
The media store is **optional throughout**: with `core.memory.media.enabled=false` or no
configuration, the whole chain silently degrades to plain-text behaviour — no errors, no panics.
### Other Memory Layers
- **Social** (`internal/memory/social/social.go`) — Persona traits and relationship network, wraps GraphDB entity types
@ -296,7 +329,7 @@ Common ground:
| Plane | Mechanism | Why this choice |
|---|---|---|
| Control | stdio JSON-RPC (NDJSON frames), 51 `core.*` methods | The process boundary *is* the ABI boundary—no need to maintain three platform-specific dynamic-library loaders |
| Control | stdio JSON-RPC (NDJSON frames), 55 `core.*` methods | The process boundary *is* the ABI boundary—no need to maintain three platform-specific dynamic-library loaders |
| Data | Shared memory segment, **one segment shared by all subprocesses** | One segment per plugin would degrade "kernel ctx → segment → plugin mutates → read back" into the copy model under concurrency, reproducing lost updates exactly |
| Notification | Event ring + platform notify (Linux eventfd / macOS pipe / Windows Event) | The kernel must never block on a consumer: streaming output publishes per token, so any wait shows up as stutter |
@ -334,8 +367,23 @@ sdk.Memory().Recall/Commit
sdk.Knowledge().Search/Create
sdk.Settings().Get/Set/List
sdk.RegisterOutputChannel("qq", sdk.CapText|sdk.CapAudio|sdk.CapImage, "QQ channel, see output_send__qq_help for details", handler)
// v1.1.1 media APIs (all additive, no signature changes)
sdk.DocMemory().InsertWithMedia(doc, attachments) // attachments with Data land in CAS; Digest-only ones reference existing content
sdk.InjectInputMedia(source, channel, text, blocks) // media reaches the model in *this* turn
sdk.InjectInputMediaSync(...) // same, and waits for the reply
sdk.InjectInterruptMedia(...) // media-bearing interrupt, can preempt current processing
```
How media injection differs from `SetToolBlocks`: the latter is only callable inside a tool handler
and its media reaches the model with the **next** tool message; these three let a plugin
**initiate a turn that carries media** — it goes out with this turn's message and is automatically
stored in CAS with a memory reference attached. `Triple` and `Doc` gained `MediaDigests` /
`Attachments` correspondingly.
`internal/sdk/` is the bridge implementation for this layer and is not subject to the public
interface freeze (see `docs/git-branching.md` §6).
### Plugin Interface
```go

View File

@ -25,6 +25,19 @@ The significance lies in clear responsibility boundaries: the kernel focuses on
Three progressive layers — context, cold archive, long-term graph memory — form an information decay and consolidation pipeline from short-term to persistent storage.
**Media Memory (since v1.1.0)** — Images and audio are not attachments; they are a kind of node in all three layers:
- **Content-addressed store (CAS)**: addressed by digest, metadata in SQLite and blobs on disk, identical bytes
stored once. Every `Get` re-verifies the digest (silently returning corrupt data is worse than an error).
- **Reference-counted GC**: `owner_kind/owner_id/digest` is the primary key; context events, documents and graph
sentences each hold their own references. **Referenced items are never deleted** — only unowned content past
`minAge` is reclaimed.
- **The description text is the durable semantic memory**: what the vision model produced is written into
plain-text memory as a `[<mime> <short digest>] <description>` marker and participates in vector retrieval and
distillation; the blob is only a cache that capacity GC may evict. Months later "that purple-blue-red
three-band chart" is still findable — via the description, not the bytes.
- **Reaches the plugin boundary since v1.1.1**: plugins read and write media through `InsertWithMedia` /
`InjectInputMedia`; the model attaches media via the `media_digests` argument of `memory_commit` / `doc_commit`.
## What It Actually Does
Code is in the project root, implemented in Go.

View File

@ -189,6 +189,36 @@ eventLoop() → processTextInput()
| `doc_query` | 从 Document 搜索 |
| `doc_commit` | 写入 Document |
`memory_commit``doc_commit` 自 v1.1.1 起接受 `media_digests`,并由内核把
`[<mime> <短digest>] <描述>` 标记补进句子/正文——**标记由内核拼,模型只给 digest**。
要求调用方知道格式,等于让一个拼写错误静默切断引用绑定而全链路无人报错。
`memory_commit` 同时新增 `sentence_text`:媒体引用挂在句子上,没有句子就无处可挂。
### 媒体记忆v1.1.0 起)
`internal/memory/media/``Store`内容寻址CAS
| 关注点 | 做法 | 为何 |
|---|---|---|
| 寻址 | sha256 digest元数据在 SQLite、blob 在磁盘 | 相同字节只存一份元数据要可查询blob 不该进数据库 |
| 完整性 | 每次 `Get` 重校 digest | 磁盘损坏时静默返回脏数据比报错危险得多 |
| 写入原子性 | `.tmp` + rename | 半个文件被当成完整内容会永久污染那个 digest |
| 引用 | `owner_kind/owner_id/digest` 三元组主键,`AddRef` 幂等 | 三个 owner 类型:`context`(上下文事件)、`document`(文档)、`graph_sentence`(图谱句子) |
| GC | 两阶段 + `minAge`**有引用者绝不删** | 描述文本留在文本层blob 可淘汰——语义记忆与字节缓存分离 |
**媒体在纯文本记忆里的表示**是标记 `[<mime> <短digest>] <描述>`
```
[image/png a1b2c3d4e5f6] 一张紫蓝红三色带图
```
之所以必须借文本承载:`Doc.Content``sentences.text`、文本记忆的 `Input` 全是字符串,
没有字段能挂结构化数据。**描述文本才是持久的语义记忆**检索靠它digest 是回到字节的
钥匙反查靠它。blob 被容量 GC 淘汰后,描述仍留在 L0/L2/L3 的文本里。
媒体存储**全程可选**`core.memory.media.enabled=false` 或未配置时,整条链路静默退化为
纯文本行为,不报错不 panic。
### 其他记忆层
- **Social** (`internal/memory/social/social.go`) — 人格特质和关系网,包装 GraphDB 实体类型
@ -294,10 +324,14 @@ Lua 脚本插件加载:`internal/plugin/` → gopher-lua 解释器执行 `main
| 面 | 机制 | 为何这么选 |
|---|---|---|
| 控制面 | stdio JSON-RPCNDJSON 帧51`core.*` method | 进程边界即 ABI 边界,无需维护三套平台特定的动态库加载代码 |
| 控制面 | stdio JSON-RPCNDJSON 帧55`core.*` method | 进程边界即 ABI 边界,无需维护三套平台特定的动态库加载代码 |
| 数据面 | 共享内存段,**全部子进程共用一块** | 每插件一段会让「内核 ctx → 段 → 插件改 → 回读 ctx」在多插件下退化成副本模型lost update 原样复现 |
| 通知面 | 事件环 + 平台通知Linux eventfd / macOS pipe / Windows Event | 内核发事件绕不等消费者,流式输出逐 token 发布时任何等待都会造成卡顿 |
v1.1.1 新增 4 个 method51 → 55`doc.insertWithMedia``io.injectMedia`
`io.injectMediaSync``io.injectInterruptMedia`。**媒体块走 JSON 而非共享段二进制通道**——
data URL 本身已是 base64 文本,包进二进制传输省不了空间,还要跟其余 51 个 method 分道。
**子进程生命周期管理**
- 每子进程一根专职 `waitLoop``cmd.Wait()` 唯一调用点)——不依赖 stdout EOF
因为插件 fork 的孙子进程browser 拉 chromium、editdoc 拉 python继承同一 stdout
@ -330,8 +364,20 @@ sdk.Memory().Recall/Commit
sdk.Knowledge().Search/Create
sdk.Settings().Get/Set/List
sdk.RegisterOutputChannel("qq", sdk.CapText|sdk.CapAudio|sdk.CapImage, "QQ消息通道详见 output_send__qq_help", handler)
// v1.1.1 媒体接口(全部新增,无签名变更)
sdk.DocMemory().InsertWithMedia(doc, attachments) // 带 Data 的落进 CAS只给 Digest 的引用已有内容
sdk.InjectInputMedia(source, channel, text, blocks) // 媒体在「本轮」就发给模型
sdk.InjectInputMediaSync(...) // 同上并同步等回复
sdk.InjectInterruptMedia(...) // 带媒体的中断,可抢占当前处理
```
媒体注入与 `SetToolBlocks` 的区别:后者只能在工具处理函数内部调用,且媒体要等**下一条**
tool message 才到模型手上;前三个是插件**主动发起一轮带媒体的对话**,媒体随本轮消息发出,
并自动落进 CAS、挂上媒体记忆引用。`Triple``Doc` 相应新增 `MediaDigests``Attachments`
`internal/sdk/` 是这层的桥接实现,不受公开接口冻结约束(见 `docs/git-branching.md` §六)。
### Plugin 接口
```go

View File

@ -25,6 +25,17 @@ HomeAgent 是一个持续运行的个人智能 Agent 框架。
三层递进:上下文 → 冷归档 → 长期图记忆,构成从短期到持久的信息衰减与整合管道。
**媒体记忆v1.1.0 起)** — 图片/音频不是附属物,而是三层里的一类节点:
- **内容寻址存储CAS**digest 寻址,元数据在 SQLite、blob 在磁盘,相同字节只存一份,
每次 `Get` 重校 digest磁盘损坏静默返回脏数据比报错更危险
- **引用计数 GC**`owner_kind/owner_id/digest` 三元组为主键,上下文事件/文档/图谱句子各自持引用;
**有引用者绝不删除**,仅回收无主且超过 `minAge` 的内容
- **描述文本才是持久语义记忆**:视觉模型生成的描述以
`[<mime> <短digest>] <描述>` 标记形式写进纯文本记忆,参与向量检索与蒸馏;
blob 只是可被容量 GC 淘汰的缓存。几个月后“那张紫蓝红三色带图”仍可检索,靠的是描述而不是字节
- **v1.1.1 起贯通插件边界**:插件可通过 `InsertWithMedia` / `InjectInputMedia` 读写媒体,
模型可用 `memory_commit` / `doc_commit``media_digests` 参数关联媒体
## 它实际做了什么
代码位于项目仓库根目录Go 语言实现。