mirror of
https://gitcode.com/JianFeeeee/HomeAgent.git
synced 2026-09-22 09:58:06 +00:00
Compare commits
5 Commits
v1.2.0-bet
...
v1.1.1
| Author | SHA1 | Date | |
|---|---|---|---|
| 22a5af6534 | |||
| 56af9e2053 | |||
| 9a61efb173 | |||
| 7a57a14c6f | |||
| 579d7dbaee |
@ -193,6 +193,8 @@ internal/
|
||||
|
||||
## 项目状态
|
||||
|
||||
**v1.1.0** — 记忆系统支持二进制多媒体节点。此前四层记忆(L0 活跃上下文 / L1 文本 / L2 文档 / L3 图库)全部只存文字,图片音频经视觉模型转成描述后原始字节即丢弃,"那张紫蓝红三色带图"再也取不回来。本版新增内容寻址媒体存储(CAS,`internal/memory/media`):元数据进 SQLite、blob 按 sha256 落盘去重,L0/L2/L3 各层只记 digest 并通过 `media_refs` 维护引用计数,容量上限由后台 GC 真正兑现(被引用的内容即便超限也永不删除)。**描述文本才是持久语义记忆,blob 只是缓存**——描述随记忆各层一直留存并可检索,原始字节可被容量 GC 淘汰,因此几个月后仍能从图库句子反查到那张图(若尚在则逐字节取回)。描述由后台循环经视觉源生成(默认关闭,开启后每 30s 最多 4 条,不与对话抢配额),放在对话路径上会给每张图的回复加十几秒而收益为零——那一轮模型本来就直接看着图。同时修五个缺陷:`core.New` 漏接 `rc.SetMediaStore` 致 L0→L2 引用转移在生产静默失效;三元组全被实体名校验拒绝时仍释放引用并删除文档(数据丢失,已反向验证);媒体入图库曾依赖 NLP 提取器碰巧提出合规三元组而时好时坏,改为按媒体标记确定性产出;L3 媒体检索一度没有任何调用方(能存进去、agent 拿不出来);`remotedevice` 网关与 `agentcli` 终端各一处数据竞争。配套 `-tags medialive` 自动触发链实测:只注入一个图片事件,落盘/描述/归档/图库绑定/GC 保护/二轮召回七个阶段全由生产代码自行触发,真实视觉模型下 agent 在不给图的第二轮准确答出三条色带的颜色与近似 hex。插件 ABI 未变(`SDKCompatibleVersion` 仍为 1.0.0),存量 `plugin.bin` 无需重编。
|
||||
|
||||
**v1.0.0** — 外部插件从 C ABI 动态库迁移到**子进程 + 共享内存**。首个不再加载 `.so`/`.dll` 的版本,与 0.9.x 不兼容(存量插件须用新版 `plugindev` 重编为 `plugin.bin`,**业务代码零改动**)。消除 6 类此前在生产造成故障的缺陷:热重载失效(`DF_1_NODELETE` 让 `dlclose` 成 no-op)、崩溃隔离缺失(插件 panic 带崩 homed)、stage lost update(副本模型丢失 35.8~36.8%)、cgo 超时不可中断(线程线性泄漏)、`output_send` 假成功(模型收到「已发送」而消息未送达)、Windows 能力断层(只见 3 个 stage 字段且无法写回)。三面通信:stdio JSON-RPC(控制)+ 共享内存段(数据)+ 事件环(通知);权限梯度显式化为三道闸。RPC 往返 p50 24.1µs,崩溃到恢复 <1s。
|
||||
|
||||
**v0.9.0** — C ABI v2:外部插件 Stage 回调支持写回(`invoke_stage` 增加 result 输出,插件可在 OnInput/AfterToolcall/PostAction 修改 RawMessage/LLMText/ToolResults 等并同步回内核),ABI 版本随内核 minor 对齐(v0.9.x → ABIVersion=2,`version_min=1` 向后兼容旧插件)。同步修复工具循环 zen 兼容补位误伤首轮 system 上下文的问题。配套 SDK 提供增强版 sanitizer 示例(坏 UTF-8/U+FFFD/ANSI 转义全链路清洗)。**该 ABI 已随 v1.0.0 退场。**
|
||||
@ -218,7 +220,7 @@ internal/
|
||||
| **client** | waiter + 桌面 GUI | 连接远程 HomeAgent |
|
||||
|
||||
- Linux:`.deb`(amd64/arm64)、`.rpm`(x86_64)、`.tar.gz`
|
||||
- Windows:`HomeAgent_v1.0.0_{Full,Server,Client}_win64.exe`(NSIS 安装向导)
|
||||
- Windows:`HomeAgent_v1.1.0_{Full,Server,Client}_win64.exe`(NSIS 安装向导)
|
||||
- 免安装:`homeagent-bin-<os>_<arch>.tar.gz`(含 homed/waiter/initconfig)
|
||||
- 校验:`SHA256SUMS`
|
||||
|
||||
|
||||
@ -179,6 +179,8 @@ External plugin development: see [homeagent-sdk](https://gitcode.com/JianFeeeee/
|
||||
|
||||
## Project Status
|
||||
|
||||
**v1.1.0** — Binary/multimedia nodes in the memory system. All four tiers (L0 active context / L1 text / L2 documents / L3 graph) previously stored text only: an image or audio clip was turned into a description by a vision model and the original bytes were dropped, so "that purple-blue-red banded image" could never be retrieved again. This release adds a content-addressed media store (CAS, `internal/memory/media`): metadata in SQLite, blobs deduplicated on disk by sha256, with every tier holding only digests and reference counts maintained through `media_refs`, so the capacity cap is finally enforced by a background GC (referenced content is never deleted, even over the limit). **The description text is the durable semantic memory; the blob is only a cache** — descriptions persist across all tiers and stay searchable while raw bytes may be evicted, so months later a graph sentence still resolves back to that image (byte-for-byte if it survives). Descriptions are generated by a background loop through a vision source (off by default; at most 4 items per 30s when enabled, so it never competes with conversations for quota) — doing it inline would add tens of seconds to every image reply for no gain, since the model is looking at the image in that turn anyway. Five defects fixed as well: `core.New` never called `rc.SetMediaStore`, silently disabling L0→L2 reference transfer in production; references were released and the document deleted even when every triple was rejected by entity-name validation (data loss, reverse-verified); media entering the graph depended on the NLP extractor happening to produce valid triples and was therefore intermittent, now replaced by deterministic triples derived from media markers; L3 media lookup had no callers at all (stored fine, unreachable by the agent); and one data race each in the `remotedevice` gateway and the `agentcli` terminal. Ships with a `-tags medialive` auto-trigger integration test: a single injected image event drives all seven stages — CAS write, description, archival, graph binding, GC protection, second-turn recall — entirely through production code paths, and with a real vision model the agent names all three band colours and their approximate hex values in a second turn that includes no image. Plugin ABI unchanged (`SDKCompatibleVersion` stays 1.0.0); existing `plugin.bin` files need no rebuild.
|
||||
|
||||
**v1.0.0** — External plugins moved from C ABI shared libraries to **subprocess + shared memory**. The first release that no longer loads `.so`/`.dll`, and it is incompatible with 0.9.x (existing plugins must be rebuilt into `plugin.bin` with the new `plugindev`, though **business code needs zero changes**). Eliminates 6 classes of defects that had caused production incidents: hot-reload silently failing (`DF_1_NODELETE` making `dlclose` a no-op), no crash isolation (a plugin panic took down homed), stage lost updates (35.8~36.8% loss under the copy model), uncancellable cgo timeouts (linear OS-thread leaks), `output_send` reporting false success (the model was told "sent" while the message never went out), and Windows capability degradation (only 3 stage fields visible, no write-back). Three communication planes: stdio JSON-RPC (control) + shared memory segment (data) + event ring (notification); the privilege gradient is now enforced by three explicit gates. RPC round-trip p50 24.1µs; crash-to-recovery under 1s.
|
||||
|
||||
**v0.9.0** — C ABI v2: external plugin Stage callbacks can now write back (`invoke_stage` gained a result out-param; plugins may mutate RawMessage/LLMText/ToolResults etc. in OnInput/AfterToolcall/PostAction and have them synced to the core). ABI version now tracks core minor releases (v0.9.x → ABIVersion=2, `version_min=1` keeps old plugins loadable). Also fixes the tool-loop zen-compat placeholder that wrongly fired on first-turn system context tail. The SDK ships an enhanced sanitizer example (bad-UTF-8 / U+FFFD / ANSI-escape scrub across the whole pipeline). **This ABI retired with v1.0.0.**
|
||||
@ -204,7 +206,7 @@ External plugin development: see [homeagent-sdk](https://gitcode.com/JianFeeeee/
|
||||
| **client** | waiter + desktop GUI | Connecting to a remote HomeAgent |
|
||||
|
||||
- Linux: `.deb` (amd64/arm64), `.rpm` (x86_64), `.tar.gz`
|
||||
- Windows: `HomeAgent_v1.0.0_{Full,Server,Client}_win64.exe` (NSIS installer)
|
||||
- Windows: `HomeAgent_v1.1.0_{Full,Server,Client}_win64.exe` (NSIS installer)
|
||||
- Portable: `homeagent-bin-<os>_<arch>.tar.gz` (homed/waiter/initconfig)
|
||||
- Verification: `SHA256SUMS`
|
||||
|
||||
|
||||
@ -378,6 +378,7 @@ func main() {
|
||||
pluginReg.SetMemory(memDB)
|
||||
pluginReg.SetTextMemory(textMem)
|
||||
pluginReg.SetDocStore(docStore)
|
||||
pluginReg.SetMediaStore(mediaStore) // 插件写入的记忆也走媒体链路;nil 时静默降级
|
||||
pluginReg.SetKnowledge(ks)
|
||||
pluginReg.SetProviderManager(providerMgr)
|
||||
pluginReg.SetConfigRegistry(cfgReg)
|
||||
|
||||
@ -14,7 +14,7 @@
|
||||
# 此前硬编码 0.8.0 而 release 已到 1.0.0,装出来的包在「添加/删除程序」里
|
||||
# 会显示错误版本(DisplayVersion 也取自这个宏)。
|
||||
!ifndef PRODUCT_VERSION
|
||||
!define PRODUCT_VERSION "1.0.0"
|
||||
!define PRODUCT_VERSION "1.1.0"
|
||||
!endif
|
||||
|
||||
!if "${VARIANT}" == "full"
|
||||
|
||||
@ -9,14 +9,20 @@ PACKAGE_ROOT="${PROJECT_ROOT}/deploy/packaging/linux"
|
||||
GO="${GO:-$(command -v go 2>/dev/null || echo "go")}"
|
||||
|
||||
ARCH="${1:-amd64}" # amd64 or arm64
|
||||
|
||||
# electron 官方发布物用 x64/arm64 命名,而 Debian 用 amd64/arm64。
|
||||
# 两者在 arm64 上恰好同名,amd64 上不同——此前缓存查找统一用 TAR_ARCH
|
||||
# (amd64),于是 electron-v*-linux-x64.zip 永远命中不到,amd64 GUI 只能
|
||||
# 靠"回退到 host node_modules"这条路组装。干净 worktree 里没有完整
|
||||
# node_modules,GUI 就被静默跳过。故单独映射。
|
||||
ACTION="${2:-all}" # all, build, deb, tar, rpm
|
||||
|
||||
DEB_ARCH="$ARCH"
|
||||
RPM_ARCH="$ARCH"
|
||||
TAR_ARCH="$ARCH"
|
||||
case "$ARCH" in
|
||||
amd64) DEB_ARCH="amd64"; RPM_ARCH="x86_64"; TAR_ARCH="amd64" ;;
|
||||
arm64) DEB_ARCH="arm64"; RPM_ARCH="aarch64"; TAR_ARCH="arm64" ;;
|
||||
amd64) DEB_ARCH="amd64"; RPM_ARCH="x86_64"; TAR_ARCH="amd64"; ELECTRON_ARCH="x64" ;;
|
||||
arm64) DEB_ARCH="arm64"; RPM_ARCH="aarch64"; TAR_ARCH="arm64"; ELECTRON_ARCH="arm64" ;;
|
||||
*) echo "Unknown arch: $ARCH (use amd64 or arm64)"; exit 1 ;;
|
||||
esac
|
||||
|
||||
@ -142,24 +148,48 @@ build_gui() {
|
||||
|
||||
echo ">>> Building GUI directory for linux/$ARCH..."
|
||||
|
||||
if [ ! -d "$gui_dir/node_modules" ]; then
|
||||
# 判据是 electron 包本身在不在,而不是 node_modules 目录在不在。
|
||||
#
|
||||
# npm install 失败(离线、网络受限)会留下一个只有一两个条目的空壳
|
||||
# node_modules,目录存在但 electron 缺失。只看目录会以为"已安装",
|
||||
# 于是 ever 读不到版本、缓存匹配退化、最后走到"host dist 也没有"而
|
||||
# 静默跳过 GUI——包名和目录名全都正确,只是没有 GUI,没有任何一步报错。
|
||||
if [ ! -f "$gui_dir/node_modules/electron/package.json" ]; then
|
||||
if [ -d "$gui_dir/node_modules" ]; then
|
||||
echo " node_modules 存在但 electron 缺失(疑似上次 npm install 未完成)"
|
||||
fi
|
||||
echo " npm install..."
|
||||
(cd "$gui_dir" && npm install --production)
|
||||
if ! (cd "$gui_dir" && npm install --production); then
|
||||
echo " WARNING: npm install 失败——离线环境下这是预期的。"
|
||||
echo " GUI 需要 cmd/gui/node_modules/electron 或 ~/.cache/electron 缓存。"
|
||||
fi
|
||||
fi
|
||||
|
||||
# electron 版本从已安装的包里读,保证运行时与 app 依赖一致
|
||||
# electron 版本优先从已安装的包里读,保证运行时与 app 依赖一致。
|
||||
# 读不到时退而从 package.json 的依赖声明里取数字部分(它可能写成
|
||||
# "^33.0.0" 这类范围,只用于给缓存匹配一个提示,匹配不上仍会走通配)。
|
||||
local ever
|
||||
ever=$(python3 -c "import json;print(json.load(open('$gui_dir/node_modules/electron/package.json'))['version'])" 2>/dev/null || true)
|
||||
if [ -z "$ever" ]; then
|
||||
ever=$(python3 -c "
|
||||
import json, re
|
||||
d = json.load(open('$gui_dir/package.json'))
|
||||
spec = (d.get('devDependencies', {}) or {}).get('electron') or (d.get('dependencies', {}) or {}).get('electron') or ''
|
||||
m = re.search(r'(\\d+(?:\\.\\d+)*)', spec)
|
||||
print(m.group(1) if m else '')
|
||||
" 2>/dev/null || true)
|
||||
[ -n "$ever" ] && echo " electron 版本取自 package.json 依赖声明: $ever(非精确)"
|
||||
fi
|
||||
|
||||
mkdir -p "$gui_out"
|
||||
|
||||
# 优先:缓存里的目标架构 zip(~/.cache/electron/<hash>/electron-v<ver>-linux-<arch>.zip)
|
||||
local zip=""
|
||||
if [ -n "$ever" ]; then
|
||||
zip=$(find "$HOME/.cache/electron" -name "electron-v${ever}-linux-${TAR_ARCH}.zip" 2>/dev/null | head -1)
|
||||
zip=$(find "$HOME/.cache/electron" -name "electron-v${ever}-linux-${ELECTRON_ARCH}.zip" 2>/dev/null | head -1)
|
||||
fi
|
||||
if [ -z "$zip" ]; then
|
||||
zip=$(find "$HOME/.cache/electron" -name "electron-v*-linux-${TAR_ARCH}.zip" 2>/dev/null | head -1)
|
||||
zip=$(find "$HOME/.cache/electron" -name "electron-v*-linux-${ELECTRON_ARCH}.zip" 2>/dev/null | head -1)
|
||||
fi
|
||||
|
||||
if [ -n "$zip" ]; then
|
||||
@ -174,9 +204,9 @@ build_gui() {
|
||||
*) host_arch=unknown ;;
|
||||
esac
|
||||
if [ "$TAR_ARCH" != "$host_arch" ]; then
|
||||
echo " WARNING: 缺 electron-v*-linux-${TAR_ARCH}.zip 缓存,且目标架构与 host"
|
||||
echo " WARNING: 缺 electron-v*-linux-${ELECTRON_ARCH}.zip 缓存,且目标架构与 host"
|
||||
echo " ($host_arch) 不同——不能用 host 的 electron 冒充。跳过 GUI。"
|
||||
echo " 解法:下载 electron-v${ever:-<ver>}-linux-${TAR_ARCH}.zip 到"
|
||||
echo " 解法:下载 electron-v${ever:-<ver>}-linux-${ELECTRON_ARCH}.zip 到"
|
||||
echo " ~/.cache/electron/<任意子目录>/ 后重跑。"
|
||||
rm -rf "$gui_out"
|
||||
return
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Git 分支管理规范
|
||||
|
||||
> 生效:2026-08-31,2026-09-04 修订(三级发布通道 + 单条发布分支)。
|
||||
> 生效:2026-08-31,2026-09-04 修订(三级发布通道 + 单条发布分支),2026-09-06 修订(SDK 仓版本语义与发版联动,见 §七)。
|
||||
> 适用:**本仓(TrueAgent/HomeAgent)与 third_party/homeagent-sdk(SDK 仓)**——两仓协作时分支策略必须一致,本规范两仓同用。
|
||||
> 核心原则一句话:**main 唯一长命、永远可部署;一切新工作在特性分支;一个中版本一条发布分支,alpha/beta/正式由 tag 区分;hotfix 只进发布分支并 cherry-pick 回 main。**
|
||||
|
||||
@ -83,6 +83,10 @@ main ──────────────── E ────────
|
||||
- alpha/beta 的构建产物**可以上传 release 附件**,但必须在 gitcode release 上勾选
|
||||
"预发布"标记,且发布说明首行标注通道与已知风险。
|
||||
- **beta 未清零的严重问题不得进正式**:正式 tag 意味着"我们认为它能上 24/7 现网"。
|
||||
- **发版动作只在发布分支上做**:版本号 bump、打 tag、构建产物、上传 release 附件,
|
||||
全部发生在 `release/vX.Y.x` 上。**main 永远不是发版分支**——即使某个改动刚刚合进 main、
|
||||
即使 main 此刻可部署,也不从 main 打 tag、不拿 main 的构建产物发布。
|
||||
main 的版本号是「下一个未发布中版本」的路牌,不是任何一次发布的版本号。
|
||||
|
||||
### 5. hotfix(发布后发现的严重 bug)
|
||||
|
||||
@ -132,12 +136,21 @@ main ──────────────── E ────────
|
||||
|
||||
| 分支 | 状态 | 处理 |
|
||||
|---|---|---|
|
||||
| `main` | 含全部 hotfix(逐个 cherry-pick),`meta.Version` = 下一个未发布版本 | ✅ 保持 |
|
||||
| `main` | 含全部 hotfix(逐个 cherry-pick),`meta.Version` = 下一个未发布中版本(现为 `1.2.0`) | ✅ 保持 |
|
||||
| `release/v1.0.x` | 承载 v1.0.0 / v1.0.1 / v1.0.3 全部 tag | ✅ **由 `release/v1.0.1` 重命名而来**(2026-09-04) |
|
||||
| `release/v1.0.0` | `9b92a04`,已被 1.0.x 线完全包含(`merge-base --is-ancestor` 验证通过) | 🗑️ **已删除**(本地 + 远端),tag `v1.0.0` 保留全部历史 |
|
||||
| `release/v1.0.1` | 旧 patch 号命名 | 🗑️ **已重命名为 `release/v1.0.x`**(远端旧名删除) |
|
||||
| `feature/memory-media` | 记忆系统媒体(多模态)支持,进行中 | ⏳ 完成后合回 main 并删除 |
|
||||
| `feature/plugin-proc-migration` | 已合入 main(`525aa1f`) | ⏳ 待删(规范要求合回后删除) |
|
||||
| `release/v1.1.x` | 承载 v1.1.0 / v1.1.0-beta.1 / v1.1.1 全部 tag | ✅ 1.1 线的唯一发布分支 |
|
||||
|
||||
### SDK 仓(homeagent-sdk)
|
||||
|
||||
| 分支 | 状态 | 处理 |
|
||||
|---|---|---|
|
||||
| `main` | `meta.Version` = 下一个未发布中版本(现为 `1.2.0`) | ✅ 保持 |
|
||||
| `release/v1.1.x` | `meta.Version` = `1.1.0`,承载 tag `v1.1.0` | ✅ 与核心 `release/v1.1.x` 对应 |
|
||||
| `release/v1.0.0` | 旧 patch 号命名形态,内容已被 main 完全包含 | 📦 保留(供追溯 1.0 线构建) |
|
||||
|
||||
### 1.0.x 发布线 tag 历史
|
||||
|
||||
@ -149,6 +162,18 @@ main ──────────────── E ────────
|
||||
|
||||
> `v1.0.2` 未使用:该号从未发布也无 tag,留空以免与任何本地构建混淆。
|
||||
|
||||
### 1.1.x 发布线 tag 历史
|
||||
|
||||
| tag | 提交 | 通道 | SDK | 说明 |
|
||||
|---|---|---|---|---|
|
||||
| `v1.1.0` | `579d7db` | 正式 | 1.0.0 | 记忆系统支持二进制多媒体节点(CAS 媒体存储 + L0/L2/L3 贯通) |
|
||||
| `v1.1.0-beta.1` | `7a57a14` | beta | 不发 | 打包链路验证(GUI 架构污染 + 空壳 node_modules)。按 §七.2,beta 不伴随 SDK 发版 |
|
||||
| `v1.1.1` | 见发布说明 | 正式 | **1.1.0** | 多模态贯通插件边界;SDK 首次随核心正式版发布 |
|
||||
|
||||
> `v1.1.0-beta.1` 的提交序在 `v1.1.0` **之后**(它多含一个打包修复),
|
||||
> 而 semver 预发布语义里 `1.1.0-beta.1 < 1.1.0`。这是「一条发布分支 + tag 区分通道」的
|
||||
> 已知代价:beta 是为验证**打包链路**而补打的,不代表源码更旧。发布说明里已注明。
|
||||
|
||||
---
|
||||
|
||||
## 四、现网部署与版本对应(运维纪律)
|
||||
@ -164,7 +189,7 @@ main ──────────────── E ────────
|
||||
- **改造期间现网不得部署 main 或 feature 的中间态**——只有发版才用发布分支的 tag。
|
||||
- alpha/beta tag 的产物**不上现网**(现网是 24/7 服务,预发布通道的存在就是为了不拿它冒险)。
|
||||
- 涉及 SDK 仓时:主仓 `go.mod` 的 `replace => ./third_party/homeagent-sdk` 指向本地 vendored 副本,
|
||||
发版前确认 vendored SDK 与 SDK 仓 release tag 一致(**两仓版本对齐是第一优先级**)。
|
||||
发版前确认 vendored SDK 与 SDK 仓 release tag 一致(**两仓中版本对齐是第一优先级**,见 §七)。
|
||||
|
||||
---
|
||||
|
||||
@ -199,6 +224,20 @@ git cherry-pick <修复①的sha> # ③ 只挑修复
|
||||
# 若有活跃 feature 分支,也 pick 过去
|
||||
git checkout feature/xxx && git cherry-pick <main 上那个 pick 的 sha>
|
||||
|
||||
# 公开 SDK 接口改动(feature,不是 hotfix):先进 main,再 pick 到发布分支
|
||||
git checkout -b feature/sdk-xxx main
|
||||
# ... 改 third_party/homeagent-sdk/sdk/ 与内核桥接层 ...
|
||||
git checkout main && git merge --no-ff feature/sdk-xxx
|
||||
git checkout release/v1.1.x
|
||||
git cherry-pick <feature 的各 sha> # 只挑改动,不挑 main 的版本号
|
||||
git commit -am "chore(release): bump v1.1.1" # 发布分支自己的版本号
|
||||
git tag -a v1.1.1 -m "..."
|
||||
# SDK 仓同步(仅在核心打正式 tag 时,见 §七.2/§七.3)
|
||||
cd third_party/homeagent-sdk
|
||||
git checkout -b release/v1.1.x main
|
||||
git commit -am "chore(release): SDK 1.1.0(1.1.x 线全程共用)"
|
||||
git tag -a v1.1.0 -m "..."
|
||||
|
||||
# 发布分支退役(下个中版本发布后,可选)
|
||||
git branch -d release/v1.0.x # tag 已保存历史,删分支不丢东西
|
||||
```
|
||||
@ -211,3 +250,53 @@ git branch -d release/v1.0.x # tag 已保存历史,
|
||||
- `internal/sdk` **不受冻结约束**,可自由扩展;冻结只针对公开 SDK 接口(`third_party/homeagent-sdk/sdk/`)。
|
||||
- 若整改确需突破公开接口,走变更评审(见 `docs/zh/plugin-interface-matrix.md` §七),
|
||||
并同步 `SDKCompatibleVersion` 与 SDK 仓的 release tag。
|
||||
- **公开接口的改动本身是 feature,不是发布准备**:它必须走 `feature/xxx` → 合回 main 的路径,
|
||||
再 cherry-pick 到发布分支。不允许把接口新增当成"发布分支上的 bug 修复"直接提交进 release
|
||||
——发布分支冻结功能(§2.3),接口是最典型的功能面。
|
||||
|
||||
---
|
||||
|
||||
## 七、SDK 仓的版本语义与发版联动
|
||||
|
||||
### 1. SDK 版本号跟随核心的中版本,patch 位恒为 `.0`
|
||||
|
||||
| 核心版本 | 对应 SDK 版本 |
|
||||
|---|---|
|
||||
| 1.1.0 / 1.1.1 / 1.1.2 / … / 1.1.N | **1.1.0**(全线共用,不随核心 patch 变动) |
|
||||
| 1.2.0 起 | **1.2.0** |
|
||||
|
||||
- 核心的 patch 位(`x`)专用于 **bugfix 与漏洞修复**,这类改动不触碰公开 SDK 接口,
|
||||
因此 SDK 版本号没有理由跟着动。
|
||||
- **为什么不逐位对齐**:SDK 版本号是插件开发者的依赖声明。若核心每发一个 bugfix 就把 SDK
|
||||
也推一个新号,开发者要么被迫跟版、要么怀疑自己版本过时,而接口其实一个字都没变。
|
||||
让 SDK 号只在**接口可能变化的中版本边界**上跳,开发者只需关心「我在为哪个中版本写插件」。
|
||||
- 因此「两仓版本对齐」在本规范里指**中版本对齐**(核心 1.1.x ↔ SDK 1.1.0),
|
||||
不是三位全等。核心 1.1.1 配 SDK 1.1.0 就是对齐状态。
|
||||
|
||||
### 2. beta 阶段不发 SDK
|
||||
|
||||
- **核心的 alpha/beta tag 不伴随 SDK 仓发版**:SDK 仓在这一阶段**不打 tag、不建 release**。
|
||||
- **为什么**:beta 是核心自己的测试阶段,此时 SDK 接口尚未固定。若此刻给 SDK 发版,
|
||||
插件开发者会照着一个还会变的接口写代码——**那是无效开发**。接口没定就没有可依赖的契约,
|
||||
发出去的版本号是一个假承诺。
|
||||
- 这条约束的对象是 **SDK 仓的发版动作**,不是核心二进制里有没有 SDK 代码。
|
||||
主仓 `go.mod` 用 `replace => ./third_party/homeagent-sdk`,任何核心构建都必然含 vendored
|
||||
SDK 源码,这是构建机制决定的,不在本条约束范围内。
|
||||
|
||||
### 3. 正式发布时 SDK 随核心一起发
|
||||
|
||||
核心打**正式 tag**(`vX.Y.Z`,无预发布后缀)时,SDK 仓同步执行:
|
||||
|
||||
1. SDK 仓也有自己的 `release/vX.Y.x`(与核心同名,一个中版本一条);
|
||||
2. 在该分支上把 `meta.Version` 定为 `X.Y.0`;
|
||||
3. 打 tag `vX.Y.0`(首次进入该中版本时),并建 gitcode release;
|
||||
4. 上传 5 平台 plugindev 产物 + `SHA256SUMS`。
|
||||
|
||||
同一中版本内的后续核心 patch(1.1.1 → 1.1.2 …)**不重复发 SDK**——SDK 已经是 1.1.0,
|
||||
没有新东西要发。只有接口再次变化并进入下一个中版本时,SDK 才发 1.2.0。
|
||||
|
||||
### 4. 版本号在两仓 main 上的含义
|
||||
|
||||
两仓的 `main` 都遵守 §2.1:`meta.Version` 是**下一个未发布中版本**。
|
||||
所以在 1.1.x 线发布期间,两仓 main 上的值都是 `1.2.0`——它标记「main 正在积攒 1.2 的东西」,
|
||||
而不是「1.2.0 已经存在」。已发布的版本号一律看对应 `release/vX.Y.x` 分支与 tag。
|
||||
|
||||
@ -10,6 +10,7 @@ import (
|
||||
agentIO "gitcode.com/JianFeeeee/HomeAgent/internal/agent/io"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/events"
|
||||
sdk "gitcode.com/JianFeeeee/HomeAgent/internal/sdk"
|
||||
pubsdk "gitcode.com/JianFeeeee/homeagent-sdk/sdk"
|
||||
)
|
||||
|
||||
func (a *Agent) eventLoop() {
|
||||
@ -125,30 +126,18 @@ func (a *Agent) handleSelfInput(msg selfInputMsg) {
|
||||
if msg.channel == "" {
|
||||
msg.channel = channelConsolidation // 兼容空值:默认走整理路径
|
||||
}
|
||||
a.processTextInput(&agentIO.InputEvent{
|
||||
a.processInput(&agentIO.InputEvent{
|
||||
Source: "system",
|
||||
Type: "text",
|
||||
Payload: map[string]interface{}{"content": msg.text},
|
||||
OutputChannel: msg.channel,
|
||||
}, msg.text)
|
||||
})
|
||||
}
|
||||
|
||||
func (a *Agent) handleInput(evt *agentIO.InputEvent) {
|
||||
switch evt.Type {
|
||||
case "text":
|
||||
input, _ := evt.Payload["content"].(string)
|
||||
if input == "" {
|
||||
return
|
||||
}
|
||||
// 去重:webui/GUI 断线重连会重放未确认消息,短窗口内同来源同内容丢弃,避免轰炸
|
||||
if a.isDuplicateInput(evt.Source, input) {
|
||||
log.Printf("[agent] dropped duplicate input from %s: %s", evt.Source, truncateStr(input, 60))
|
||||
return
|
||||
}
|
||||
a.processTextInput(evt, input)
|
||||
|
||||
case "image", "audio":
|
||||
a.processMediaInput(evt)
|
||||
case "text", "image", "audio":
|
||||
a.processInput(evt)
|
||||
|
||||
case "event":
|
||||
log.Printf("[agent] event from %s: %v", evt.Source, evt.Payload)
|
||||
@ -162,87 +151,97 @@ func (a *Agent) handleInput(evt *agentIO.InputEvent) {
|
||||
}
|
||||
}
|
||||
|
||||
func (a *Agent) processMediaInput(evt *agentIO.InputEvent) {
|
||||
start := time.Now()
|
||||
a.pendingMedia = evt.Payload
|
||||
defer func() { a.pendingMedia = nil }()
|
||||
// inputPayload 是一次输入在「模态」这个维度上的全部内容。
|
||||
//
|
||||
// 拆出这个结构,是为了让 processInput 只有一条主干:模态不再决定走哪个函数,
|
||||
// 只决定这里的字段填不填。此前 text 与 image/audio 各有一个 process 函数,
|
||||
// 媒体那条缺了去重、no_memory、通道 Cleaner、中断语义、EventRawInput 五项——
|
||||
// 不是因为媒体不需要,而是复制粘贴之后文本那条继续演进、媒体那条没跟上。
|
||||
type inputPayload struct {
|
||||
// text 是进 LLM 与记忆的文本。纯媒体输入时它是 mediaToBlocks 给的 alt 文案。
|
||||
text string
|
||||
// blocks 非空表示本轮带多模态内容,随当前轮的 message 一起发给模型。
|
||||
blocks []agentAPI.ContentBlock
|
||||
// mediaType 供插件在 stage 里判断本轮媒体的模态。
|
||||
mediaType string
|
||||
// captureTool 是媒体落进 CAS 时记录的来源标签。
|
||||
captureTool string
|
||||
}
|
||||
|
||||
a.currentOutputChannel = evt.OutputChannel
|
||||
if a.currentOutputChannel == "" {
|
||||
a.currentOutputChannel = evt.Source
|
||||
// resolveInput 把 InputEvent 归一成 inputPayload。
|
||||
//
|
||||
// 三种来源在这里合流:
|
||||
// 1. evt.Type 是 image/audio —— 用户直接发的媒体,payload 里是 data/url;
|
||||
// 2. evt.Type 是 text 且 payload 带 media_blocks —— 插件经 IOInjector 的
|
||||
// InjectInputMedia / InjectInputMediaSync / InjectInterruptMedia 注入的
|
||||
// 媒体,块已经是成品;
|
||||
// 3. 纯文本。
|
||||
//
|
||||
// 第 2 种此前无处可去:注入方把块放进 payload,而文本路径不看这个键,
|
||||
// 于是插件注入的媒体到 payload 就断了,且不报错。
|
||||
func (a *Agent) resolveInput(evt *agentIO.InputEvent) (inputPayload, bool) {
|
||||
switch evt.Type {
|
||||
case "image", "audio":
|
||||
blocks, alt := a.mediaToBlocks(evt.Payload, evt.Type, evt.Source)
|
||||
return inputPayload{
|
||||
text: alt,
|
||||
blocks: blocks,
|
||||
mediaType: evt.Type,
|
||||
captureTool: "input_" + evt.Type,
|
||||
}, true
|
||||
}
|
||||
|
||||
blocks, fallback := a.mediaToBlocks(evt.Payload, evt.Type, evt.Source)
|
||||
|
||||
// 用户直接发来的媒体:先落进 CAS。
|
||||
// 不存的后果是 ContextEvent.Input 只剩一句 alt 文本
|
||||
//("[从 qq 收到了 image]"),base64 随 message 数组发给模型后就丢了。
|
||||
a.stageMediaDigests(a.captureBlockMedia(blocks, "input_"+evt.Type)...)
|
||||
|
||||
stageCtx := a.stageCtxFromInput(fallback, evt.Source, "")
|
||||
stageCtx.Extra = map[string]interface{}{
|
||||
"media_blocks": blocks,
|
||||
"media_type": evt.Type,
|
||||
"input_source": evt.Source,
|
||||
"output_channel": evt.OutputChannel,
|
||||
text, _ := evt.Payload["content"].(string)
|
||||
blocks, mediaType := injectedBlocks(evt.Payload)
|
||||
// 文本与媒体都空才算无效输入:只带图不带字是合法的(插件注入常这样)。
|
||||
if text == "" && len(blocks) == 0 {
|
||||
return inputPayload{}, false
|
||||
}
|
||||
a.injectSourceContext(stageCtx, evt)
|
||||
return inputPayload{
|
||||
text: text,
|
||||
blocks: blocks,
|
||||
mediaType: mediaType,
|
||||
captureTool: "inject_" + evt.Source,
|
||||
}, true
|
||||
}
|
||||
|
||||
if a.runStage(sdk.StageOnInput, stageCtx) {
|
||||
a.emitResponse(evt, *stageCtx.Response)
|
||||
return
|
||||
// injectedBlocks 取出 payload 里插件注入的多模态块。
|
||||
//
|
||||
// 两种静态类型都要认:内核内部注入直接给 []agentAPI.ContentBlock,
|
||||
// 而经公共 SDK 的 IOInjector 过来的是 []pubsdk.ContentBlock。两者字段完全一致,
|
||||
// 但 Go 不会自动转换,只认一种的后果是另一种被静默丢弃。
|
||||
func injectedBlocks(payload map[string]interface{}) ([]agentAPI.ContentBlock, string) {
|
||||
var blocks []agentAPI.ContentBlock
|
||||
switch v := payload["media_blocks"].(type) {
|
||||
case []agentAPI.ContentBlock:
|
||||
blocks = v
|
||||
case []pubsdk.ContentBlock:
|
||||
blocks = make([]agentAPI.ContentBlock, 0, len(v))
|
||||
for _, b := range v {
|
||||
nb := agentAPI.ContentBlock{Type: b.Type, Text: b.Text}
|
||||
if b.ImageURL != nil {
|
||||
nb.ImageURL = &agentAPI.ImageURL{URL: b.ImageURL.URL, Detail: b.ImageURL.Detail}
|
||||
}
|
||||
if b.AudioURL != nil {
|
||||
nb.AudioURL = &agentAPI.AudioURL{URL: b.AudioURL.URL}
|
||||
}
|
||||
blocks = append(blocks, nb)
|
||||
}
|
||||
}
|
||||
|
||||
a.publishEvent(events.EventRawInput, map[string]interface{}{
|
||||
"content": evt.Payload,
|
||||
"source": evt.Source,
|
||||
})
|
||||
|
||||
archived := a.context.Prune(fallback, a.maxContextSize-1, a.docStore)
|
||||
if archived > 0 {
|
||||
log.Printf("[agent] pruned %d low-relevance events to document memory", archived)
|
||||
if len(blocks) == 0 {
|
||||
return nil, ""
|
||||
}
|
||||
|
||||
a.context.Append(ContextEvent{
|
||||
Timestamp: start,
|
||||
Source: evt.Source,
|
||||
Input: fallback,
|
||||
})
|
||||
|
||||
response, toolsUsed, toolResults, err := a.process(fallback, stageCtx)
|
||||
if err != nil {
|
||||
log.Printf("[agent] process media error: %v", err)
|
||||
resp := fmt.Sprintf("处理错误: %v", err)
|
||||
a.emitResponse(evt, resp)
|
||||
a.context.Append(ContextEvent{Timestamp: time.Now(), Source: "agent", Input: fallback, Response: resp})
|
||||
return
|
||||
}
|
||||
|
||||
elapsed := time.Since(start)
|
||||
log.Printf("[agent] %s from %s → response (%dms, tools=%v)", evt.Type, evt.Source, elapsed.Milliseconds(), toolsUsed)
|
||||
|
||||
// 本轮捕获的媒体(用户发的 + 工具注入的)挂到这条事件上。
|
||||
// 媒体描述并进 Input:描述文本才是持久语义记忆,blob 只是缓存。
|
||||
digests := a.drainMediaDigests()
|
||||
mediaEvt := ContextEvent{
|
||||
Timestamp: time.Now(),
|
||||
Source: "agent",
|
||||
Input: fallback,
|
||||
Response: response,
|
||||
ToolsUsed: toolsUsed,
|
||||
ToolResults: toolResults,
|
||||
}
|
||||
a.bindEventMedia(&mediaEvt, digests)
|
||||
if s := a.mediaSummaryForEvent(mediaEvt.Media); s != "" {
|
||||
mediaEvt.Input = mediaEvt.Input + "\n" + s
|
||||
}
|
||||
a.context.Append(mediaEvt)
|
||||
|
||||
a.emitResponse(evt, response)
|
||||
|
||||
if !stageCtx.NoMemory {
|
||||
a.emitMemoryCandidate(evt.Source, fallback, response, toolResults, toolsUsed)
|
||||
// 模态由块自身判定,注入方不必额外声明。图优先:一次注入里图片是主体。
|
||||
mediaType := ""
|
||||
for _, b := range blocks {
|
||||
if b.ImageURL != nil {
|
||||
return blocks, "image"
|
||||
}
|
||||
if b.AudioURL != nil {
|
||||
mediaType = "audio"
|
||||
}
|
||||
}
|
||||
return blocks, mediaType
|
||||
}
|
||||
|
||||
func (a *Agent) mediaToBlocks(payload map[string]interface{}, mediaType string, source string) ([]agentAPI.ContentBlock, string) {
|
||||
@ -298,19 +297,51 @@ func (a *Agent) mediaToBlocks(payload map[string]interface{}, mediaType string,
|
||||
return blocks, alt
|
||||
}
|
||||
|
||||
func (a *Agent) processTextInput(evt *agentIO.InputEvent, input string) {
|
||||
// processInput 是全部模态输入的唯一主干。
|
||||
//
|
||||
// 文本、用户上传的图/音频、插件注入的多模态块走同一条路径,因此去重、
|
||||
// no_memory、通道 Cleaner、中断语义、EventRawInput、媒体入 CAS、媒体记忆绑定
|
||||
// 对所有模态一致——不会再出现「文本路径加了功能、媒体路径没跟上」。
|
||||
func (a *Agent) processInput(evt *agentIO.InputEvent) {
|
||||
start := time.Now()
|
||||
|
||||
in, ok := a.resolveInput(evt)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
// 去重按文本做:webui/GUI 断线重连会重放未确认消息。
|
||||
// 带媒体时跳过——媒体输入的 alt 文案("[从 qq 收到了 image]")对不同图片
|
||||
// 是同一句,拿它去重会把连发的两张图误判成重复。
|
||||
if len(in.blocks) == 0 && a.isDuplicateInput(evt.Source, in.text) {
|
||||
log.Printf("[agent] dropped duplicate input from %s: %s", evt.Source, truncateStr(in.text, 60))
|
||||
return
|
||||
}
|
||||
|
||||
a.currentOutputChannel = evt.OutputChannel
|
||||
if a.currentOutputChannel == "" {
|
||||
a.currentOutputChannel = evt.Source
|
||||
}
|
||||
|
||||
if evt.OutputChannel == "_consolidation_" {
|
||||
a.processConsolidation(evt, input)
|
||||
a.processConsolidation(evt, in.text)
|
||||
return
|
||||
}
|
||||
|
||||
// pendingMedia 让 describe_image / transcribe_audio / ocr_image 拿到本轮媒体的
|
||||
// 原始 data/url,也是这三个工具是否出现在工具表里的开关。仅对用户直接上传成立
|
||||
//(payload 里才有 data/url);插件注入的是成品 block,取不到原始数据。
|
||||
if evt.Type == "image" || evt.Type == "audio" {
|
||||
a.pendingMedia = evt.Payload
|
||||
defer func() { a.pendingMedia = nil }()
|
||||
}
|
||||
|
||||
// 媒体先落进 CAS。不存的后果是 ContextEvent.Input 只剩一句 alt 文本,
|
||||
// base64 随 message 数组发给模型后就丢了。
|
||||
if len(in.blocks) > 0 {
|
||||
a.stageMediaDigests(a.captureBlockMedia(in.blocks, in.captureTool)...)
|
||||
}
|
||||
|
||||
noMemory := false
|
||||
if v, ok := evt.Payload["no_memory"].(bool); ok {
|
||||
noMemory = v
|
||||
@ -331,9 +362,13 @@ func (a *Agent) processTextInput(evt *agentIO.InputEvent, input string) {
|
||||
noMemory = true
|
||||
}
|
||||
|
||||
stageCtx := a.stageCtxFromInput(input, evt.Source, "")
|
||||
stageCtx := a.stageCtxFromInput(in.text, evt.Source, "")
|
||||
stageCtx.Extra["input_source"] = evt.Source
|
||||
stageCtx.Extra["output_channel"] = evt.OutputChannel
|
||||
if len(in.blocks) > 0 {
|
||||
stageCtx.Extra["media_blocks"] = in.blocks
|
||||
stageCtx.Extra["media_type"] = in.mediaType
|
||||
}
|
||||
if noMemory {
|
||||
stageCtx.NoMemory = true
|
||||
}
|
||||
@ -344,7 +379,7 @@ func (a *Agent) processTextInput(evt *agentIO.InputEvent, input string) {
|
||||
return
|
||||
}
|
||||
|
||||
input = stageCtx.RawMessage
|
||||
input := stageCtx.RawMessage
|
||||
|
||||
// 计算层用的清洗文本(不改原文):通道 Cleaner 提取语义内容后用于向量化/提关键词
|
||||
cleanInput := input
|
||||
@ -354,10 +389,16 @@ func (a *Agent) processTextInput(evt *agentIO.InputEvent, input string) {
|
||||
}
|
||||
}
|
||||
|
||||
a.publishEvent(events.EventRawInput, map[string]interface{}{
|
||||
"content": input,
|
||||
"source": evt.Source,
|
||||
})
|
||||
// upload_* 字段一并转发:webui 的 EventRawInput 订阅方靠它们还原附件卡片。
|
||||
// 媒体路径此前把整个 payload 塞进 content(一个 map),订阅方按 string 断言
|
||||
// 直接失败 → 用户发的图从不出现在聊天记录里。
|
||||
rawPayload := map[string]interface{}{"content": input, "source": evt.Source}
|
||||
for _, k := range []string{"upload_url", "upload_type", "upload_size", "upload_name"} {
|
||||
if v, ok := evt.Payload[k]; ok {
|
||||
rawPayload[k] = v
|
||||
}
|
||||
}
|
||||
a.publishEvent(events.EventRawInput, rawPayload)
|
||||
|
||||
archived := a.context.Prune(cleanInput, a.maxContextSize-1, a.docStore)
|
||||
if archived > 0 {
|
||||
@ -374,7 +415,7 @@ func (a *Agent) processTextInput(evt *agentIO.InputEvent, input string) {
|
||||
|
||||
response, toolsUsed, toolResults, err := a.process(input, stageCtx)
|
||||
if err != nil {
|
||||
log.Printf("[agent] process error: %v", err)
|
||||
log.Printf("[agent] process %s error: %v", evt.Type, err)
|
||||
resp := fmt.Sprintf("处理错误: %v", err)
|
||||
a.emitResponse(evt, resp)
|
||||
a.context.Append(ContextEvent{Timestamp: time.Now(), Source: "agent", Input: input, Response: resp})
|
||||
@ -382,11 +423,12 @@ func (a *Agent) processTextInput(evt *agentIO.InputEvent, input string) {
|
||||
}
|
||||
|
||||
elapsed := time.Since(start)
|
||||
log.Printf("[agent] input from %s → response (%dms, tools=%v)", evt.Source, elapsed.Milliseconds(), toolsUsed)
|
||||
log.Printf("[agent] %s from %s → response (%dms, tools=%v)", evt.Type, evt.Source, elapsed.Milliseconds(), toolsUsed)
|
||||
|
||||
// 纯文本输入也可能产生媒体:模型调 multimodal_see_picture / see_video 等工具时,
|
||||
// 插件经 SetToolBlocks 注入的块已在 process() 里被捕获。
|
||||
textEvt := ContextEvent{
|
||||
// 本轮捕获的媒体一起挂到这条事件上:用户上传的、插件注入的,以及模型调
|
||||
// multimodal_see_picture / see_video 时经 SetToolBlocks 注入的(后者在
|
||||
// process() 里被捕获,纯文本输入也会有)。
|
||||
turnEvt := ContextEvent{
|
||||
Timestamp: time.Now(),
|
||||
Source: "agent",
|
||||
Input: cleanInput,
|
||||
@ -394,11 +436,11 @@ func (a *Agent) processTextInput(evt *agentIO.InputEvent, input string) {
|
||||
ToolsUsed: toolsUsed,
|
||||
ToolResults: toolResults,
|
||||
}
|
||||
a.bindEventMedia(&textEvt, a.drainMediaDigests())
|
||||
if s := a.mediaSummaryForEvent(textEvt.Media); s != "" {
|
||||
textEvt.Input = textEvt.Input + "\n" + s
|
||||
a.bindEventMedia(&turnEvt, a.drainMediaDigests())
|
||||
if s := a.mediaSummaryForEvent(turnEvt.Media); s != "" {
|
||||
turnEvt.Input = turnEvt.Input + "\n" + s
|
||||
}
|
||||
a.context.Append(textEvt)
|
||||
a.context.Append(turnEvt)
|
||||
|
||||
a.emitResponse(evt, response)
|
||||
|
||||
|
||||
@ -255,6 +255,130 @@ func (a *Agent) bindSentenceMedia(sentenceIDs map[string]int64) int {
|
||||
return bound
|
||||
}
|
||||
|
||||
// sentenceWithMediaMarkers 保证句子文本里带上这些 digest 的媒体标记。
|
||||
//
|
||||
// 存在的理由:媒体的绑定链是 SentenceText → sentences 表 → sentence_id →
|
||||
// media_refs。模型只知道 digest(从 memory_recall 的「关联媒体」或对话里的
|
||||
// 媒体标记读到),不该要求它自己按内核格式拼标记——格式写错的后果是引用
|
||||
// 静默挂不上,模型也无从察觉。
|
||||
//
|
||||
// 已出现过的 digest 不重复追加:模型可能既写了标记又填了 media_digests。
|
||||
func (a *Agent) sentenceWithMediaMarkers(sentence string, digests []string) string {
|
||||
if a.mediaStore == nil || len(digests) == 0 {
|
||||
return sentence
|
||||
}
|
||||
present := make(map[string]bool)
|
||||
for _, d := range extractMediaDigests(sentence) {
|
||||
present[d] = true
|
||||
}
|
||||
|
||||
var add []string
|
||||
for _, d := range digests {
|
||||
if d == "" || present[shortDigest(d)] {
|
||||
continue
|
||||
}
|
||||
// 模型给的多半是短 digest(它在上下文里看到的就是短的),补全成完整
|
||||
// digest 才能进 media_refs 主键。补不上就跳过:内容可能已被 GC 清掉。
|
||||
full, err := a.mediaStore.ResolvePrefix(d)
|
||||
if err != nil {
|
||||
log.Printf("[media] 模型提交的 digest %s 无法解析: %v", d, err)
|
||||
continue
|
||||
}
|
||||
if line := a.mediaMarkerLine(full); line != "" {
|
||||
add = append(add, line)
|
||||
present[shortDigest(full)] = true
|
||||
}
|
||||
}
|
||||
if len(add) == 0 {
|
||||
return sentence
|
||||
}
|
||||
if sentence == "" {
|
||||
return strings.Join(add, "\n")
|
||||
}
|
||||
return sentence + "\n" + strings.Join(add, "\n")
|
||||
}
|
||||
|
||||
// docMediaContext 为一篇文档产出媒体说明,供 doc_query 拼进工具返回值。
|
||||
//
|
||||
// 优先读 media_refs(权威:谁挂上去的就是谁),为空时退回解析正文标记——
|
||||
// 历史文档与经旧版路径写入的文档只有标记、没有引用。
|
||||
func (a *Agent) docMediaContext(docID, content string) string {
|
||||
if a.mediaStore == nil {
|
||||
return ""
|
||||
}
|
||||
digests, err := a.mediaStore.Refs(media.OwnerDocument, docID)
|
||||
if err != nil {
|
||||
log.Printf("[media] 读取文档 %s 的媒体引用失败: %v", docID, err)
|
||||
}
|
||||
if len(digests) == 0 {
|
||||
for _, short := range extractMediaDigests(content) {
|
||||
full, err := a.mediaStore.ResolvePrefix(short)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
digests = append(digests, full)
|
||||
}
|
||||
}
|
||||
var lines []string
|
||||
for _, d := range digests {
|
||||
if line := a.mediaMarkerLine(d); line != "" {
|
||||
lines = append(lines, line)
|
||||
}
|
||||
}
|
||||
if len(lines) == 0 {
|
||||
return ""
|
||||
}
|
||||
return strings.Join(lines, ";")
|
||||
}
|
||||
|
||||
// resolveMediaDigests 把模型给的(多为短)digest 补全成完整 digest。
|
||||
//
|
||||
// 补不上就丢弃那一条并记日志:模型可能凭印象编了个 digest,也可能内容已被
|
||||
// 容量 GC 淘汰。挂一条对不上的引用比不挂更糟——digest 进了 media_refs 主键,
|
||||
// 错了则 DropOwner 永远匹配不到它,那是一条永久泄漏的引用。
|
||||
func (a *Agent) resolveMediaDigests(digests []string) []string {
|
||||
if a.mediaStore == nil || len(digests) == 0 {
|
||||
return nil
|
||||
}
|
||||
seen := make(map[string]bool, len(digests))
|
||||
var out []string
|
||||
for _, d := range digests {
|
||||
full, err := a.mediaStore.ResolvePrefix(d)
|
||||
if err != nil {
|
||||
log.Printf("[media] 模型给的 digest %s 无法解析: %v", d, err)
|
||||
continue
|
||||
}
|
||||
if seen[full] {
|
||||
continue
|
||||
}
|
||||
seen[full] = true
|
||||
out = append(out, full)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// bindDocMedia 把一组完整 digest 挂到文档 owner 上,返回成功条数。
|
||||
//
|
||||
// 与 releaseDocMedia 成对:文档归档进 L3 时释放,文档写入时绑定。
|
||||
// 只绑不放会让磁盘只增不减,只放不绑会让 GC 误删仍被引用的内容。
|
||||
func (a *Agent) bindDocMedia(docID string, digests []string) int {
|
||||
if a.mediaStore == nil || docID == "" || len(digests) == 0 {
|
||||
return 0
|
||||
}
|
||||
bound := 0
|
||||
for _, d := range digests {
|
||||
if err := a.mediaStore.AddRef(d, media.OwnerDocument, docID); err != nil {
|
||||
log.Printf("[media] 文档引用绑定失败 (%s → doc %s): %v", shortDigest(d), docID, err)
|
||||
continue
|
||||
}
|
||||
bound++
|
||||
}
|
||||
if bound > 0 {
|
||||
log.Printf("[media] 文档 %s 绑定 %d 个媒体引用", docID, bound)
|
||||
}
|
||||
return bound
|
||||
}
|
||||
|
||||
// commitTriplesWithMedia 提交三元组并绑定句子里的媒体引用。
|
||||
//
|
||||
// 包一层是为了让所有「三元组入库」的调用点用同一条路径拿到媒体绑定,
|
||||
@ -357,19 +481,9 @@ func (a *Agent) mediaContextForSentences(sentenceIDs []int64) string {
|
||||
}
|
||||
var parts []string
|
||||
for _, d := range digests {
|
||||
it, err := a.mediaStore.Stat(d)
|
||||
if err != nil || it == nil {
|
||||
continue
|
||||
if line := a.mediaMarkerLine(d); line != "" {
|
||||
parts = append(parts, line)
|
||||
}
|
||||
label := string(it.Kind)
|
||||
if it.MIME != "" {
|
||||
label = it.MIME
|
||||
}
|
||||
desc := it.Description
|
||||
if desc == "" {
|
||||
desc = "(未描述)"
|
||||
}
|
||||
parts = append(parts, fmt.Sprintf("[%s %s] %s", label, shortDigest(d), desc))
|
||||
}
|
||||
if len(parts) > 0 {
|
||||
lines = append(lines, fmt.Sprintf("句子 #%d 关联媒体:%s", sid, strings.Join(parts, ";")))
|
||||
|
||||
587
internal/agent/core/inputunify_test.go
Normal file
587
internal/agent/core/inputunify_test.go
Normal file
@ -0,0 +1,587 @@
|
||||
package core
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
agentAPI "gitcode.com/JianFeeeee/HomeAgent/internal/agent/api"
|
||||
agentIO "gitcode.com/JianFeeeee/HomeAgent/internal/agent/io"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/document"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/media"
|
||||
pubsdk "gitcode.com/JianFeeeee/homeagent-sdk/sdk"
|
||||
)
|
||||
|
||||
// 统一输入主干(processInput / resolveInput / injectedBlocks)与
|
||||
// 模型可调用工具的媒体接线测试。
|
||||
//
|
||||
// 这一层此前的结构性缺陷:text 与 image/audio 各有一个 process 函数,
|
||||
// 媒体那条缺了去重、no_memory、通道 Cleaner、中断语义、EventRawInput 五项。
|
||||
// 归一成一条主干后,这些行为对所有模态一致——下面的断言就是这个不变量。
|
||||
|
||||
func newInputTestAgent(t *testing.T) (*Agent, *media.Store) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
|
||||
ms, err := media.New(filepath.Join(dir, "media"), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("media.New: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { ms.Close() })
|
||||
|
||||
return &Agent{mediaStore: ms}, ms
|
||||
}
|
||||
|
||||
// ---------- injectedBlocks ----------
|
||||
|
||||
// 内核内部注入直接给 []agentAPI.ContentBlock;经公共 SDK 的 IOInjector 过来的是
|
||||
// []pubsdk.ContentBlock。两者字段一致但 Go 不会自动转换,只认一种的后果是
|
||||
// 另一种被静默丢弃——插件注入的图到 payload 就断了,且不报错。
|
||||
func TestInjectedBlocks_AcceptsBothStaticTypes(t *testing.T) {
|
||||
t.Run("内核类型", func(t *testing.T) {
|
||||
blocks, kind := injectedBlocks(map[string]interface{}{
|
||||
"media_blocks": []agentAPI.ContentBlock{
|
||||
{Type: "text", Text: "看图"},
|
||||
{Type: "image_url", ImageURL: &agentAPI.ImageURL{URL: "data:image/png;base64,AAA"}},
|
||||
},
|
||||
})
|
||||
if len(blocks) != 2 {
|
||||
t.Fatalf("blocks = %d,期望 2", len(blocks))
|
||||
}
|
||||
if kind != "image" {
|
||||
t.Errorf("mediaType = %q,期望 image", kind)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("公共SDK类型", func(t *testing.T) {
|
||||
blocks, kind := injectedBlocks(map[string]interface{}{
|
||||
"media_blocks": []pubsdk.ContentBlock{
|
||||
{Type: "text", Text: "听音频"},
|
||||
{Type: "audio_url", AudioURL: &pubsdk.AudioURL{URL: "data:audio/wav;base64,BBB"}},
|
||||
},
|
||||
})
|
||||
if len(blocks) != 2 {
|
||||
t.Fatalf("blocks = %d,期望 2(公共 SDK 类型被静默丢弃)", len(blocks))
|
||||
}
|
||||
if kind != "audio" {
|
||||
t.Errorf("mediaType = %q,期望 audio", kind)
|
||||
}
|
||||
// 转换必须保留 URL,否则块到了模型手上是空的
|
||||
if blocks[1].AudioURL == nil || blocks[1].AudioURL.URL != "data:audio/wav;base64,BBB" {
|
||||
t.Errorf("AudioURL 转换丢失: %+v", blocks[1].AudioURL)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("图优先于音频", func(t *testing.T) {
|
||||
_, kind := injectedBlocks(map[string]interface{}{
|
||||
"media_blocks": []agentAPI.ContentBlock{
|
||||
{Type: "audio_url", AudioURL: &agentAPI.AudioURL{URL: "a"}},
|
||||
{Type: "image_url", ImageURL: &agentAPI.ImageURL{URL: "b"}},
|
||||
},
|
||||
})
|
||||
if kind != "image" {
|
||||
t.Errorf("mediaType = %q,期望 image", kind)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("无媒体块", func(t *testing.T) {
|
||||
blocks, kind := injectedBlocks(map[string]interface{}{"content": "纯文本"})
|
||||
if blocks != nil || kind != "" {
|
||||
t.Errorf("无 media_blocks 时应返回 (nil,\"\"),实际 (%v,%q)", blocks, kind)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("ImageURL 的 Detail 透传", func(t *testing.T) {
|
||||
blocks, _ := injectedBlocks(map[string]interface{}{
|
||||
"media_blocks": []pubsdk.ContentBlock{
|
||||
{Type: "image_url", ImageURL: &pubsdk.ImageURL{URL: "u", Detail: "high"}},
|
||||
},
|
||||
})
|
||||
if len(blocks) != 1 || blocks[0].ImageURL.Detail != "high" {
|
||||
t.Errorf("Detail 未透传: %+v", blocks)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ---------- resolveInput ----------
|
||||
|
||||
func TestResolveInput_UnifiesAllModalities(t *testing.T) {
|
||||
a, _ := newInputTestAgent(t)
|
||||
|
||||
t.Run("用户上传图片", func(t *testing.T) {
|
||||
in, ok := a.resolveInput(&agentIO.InputEvent{
|
||||
Source: "qq",
|
||||
Type: "image",
|
||||
Payload: map[string]interface{}{"data": "AAAA", "mime": "image/png"},
|
||||
})
|
||||
if !ok {
|
||||
t.Fatal("图片输入被判为无效")
|
||||
}
|
||||
if in.mediaType != "image" || in.captureTool != "input_image" {
|
||||
t.Errorf("mediaType=%q captureTool=%q", in.mediaType, in.captureTool)
|
||||
}
|
||||
if in.text == "" {
|
||||
t.Error("纯媒体输入应有 alt 文案作为文本落点")
|
||||
}
|
||||
if len(in.blocks) == 0 {
|
||||
t.Error("图片应转成内容块")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("插件注入的媒体", func(t *testing.T) {
|
||||
in, ok := a.resolveInput(&agentIO.InputEvent{
|
||||
Source: "myplugin",
|
||||
Type: "text",
|
||||
Payload: map[string]interface{}{
|
||||
"content": "帮我看看这张图",
|
||||
"media_blocks": []pubsdk.ContentBlock{
|
||||
{Type: "image_url", ImageURL: &pubsdk.ImageURL{URL: "data:image/png;base64,AAA"}},
|
||||
},
|
||||
},
|
||||
})
|
||||
if !ok {
|
||||
t.Fatal("带媒体的文本输入被判为无效")
|
||||
}
|
||||
if in.text != "帮我看看这张图" {
|
||||
t.Errorf("text = %q", in.text)
|
||||
}
|
||||
if len(in.blocks) != 1 || in.mediaType != "image" {
|
||||
t.Errorf("blocks=%d mediaType=%q —— 插件注入的媒体到 payload 就断了", len(in.blocks), in.mediaType)
|
||||
}
|
||||
if in.captureTool != "inject_myplugin" {
|
||||
t.Errorf("captureTool = %q,期望带来源便于溯源", in.captureTool)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("只带图不带字也合法", func(t *testing.T) {
|
||||
_, ok := a.resolveInput(&agentIO.InputEvent{
|
||||
Source: "myplugin",
|
||||
Type: "text",
|
||||
Payload: map[string]interface{}{
|
||||
"media_blocks": []agentAPI.ContentBlock{
|
||||
{Type: "image_url", ImageURL: &agentAPI.ImageURL{URL: "u"}},
|
||||
},
|
||||
},
|
||||
})
|
||||
if !ok {
|
||||
t.Error("只带媒体不带文本应视为有效输入(插件注入常这样)")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("文本与媒体都空才无效", func(t *testing.T) {
|
||||
if _, ok := a.resolveInput(&agentIO.InputEvent{
|
||||
Source: "cli",
|
||||
Type: "text",
|
||||
Payload: map[string]interface{}{"content": ""},
|
||||
}); ok {
|
||||
t.Error("空输入应被拒")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("纯文本", func(t *testing.T) {
|
||||
in, ok := a.resolveInput(&agentIO.InputEvent{
|
||||
Source: "cli",
|
||||
Type: "text",
|
||||
Payload: map[string]interface{}{"content": "你好"},
|
||||
})
|
||||
if !ok || in.text != "你好" || len(in.blocks) != 0 || in.mediaType != "" {
|
||||
t.Errorf("纯文本路径异常: ok=%v in=%+v", ok, in)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ---------- 模型工具侧:sentenceWithMediaMarkers ----------
|
||||
|
||||
// 模型只知道 digest(从对话或 memory_recall 的「关联媒体」读到),
|
||||
// 不该要求它自己按内核格式拼标记——格式写错的后果是引用静默挂不上。
|
||||
func TestSentenceWithMediaMarkers(t *testing.T) {
|
||||
a, ms := newInputTestAgent(t)
|
||||
digest, err := ms.Put([]byte("marker-bytes"), media.Item{
|
||||
MIME: "image/png", Description: "一张紫蓝红三色带图",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Put: %v", err)
|
||||
}
|
||||
|
||||
t.Run("短digest补全并生成标记", func(t *testing.T) {
|
||||
got := a.sentenceWithMediaMarkers("用户发来一张图。", []string{digest[:12]})
|
||||
if !strings.Contains(got, "三色带图") {
|
||||
t.Errorf("描述未并入句子: %q", got)
|
||||
}
|
||||
if !strings.Contains(got, digest[:12]) {
|
||||
t.Errorf("digest 未并入句子(反查会失效): %q", got)
|
||||
}
|
||||
// 反解必须成功,否则 bindSentenceMedia 挂不上引用
|
||||
if got := extractMediaDigests(got); len(got) != 1 {
|
||||
t.Errorf("生成的标记无法被 extractMediaDigests 反解: %v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("模型已写标记时不重复追加", func(t *testing.T) {
|
||||
sentence := "看这个 [image/png " + digest[:12] + "] 三色带图"
|
||||
got := a.sentenceWithMediaMarkers(sentence, []string{digest[:12]})
|
||||
if n := strings.Count(got, digest[:12]); n != 1 {
|
||||
t.Errorf("digest 出现 %d 次,期望 1 次: %q", n, got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("空句子时标记本身充当句子", func(t *testing.T) {
|
||||
got := a.sentenceWithMediaMarkers("", []string{digest})
|
||||
if got == "" {
|
||||
t.Error("媒体必须有句子落点,否则 media_refs 无从挂起")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("无法解析的digest被跳过", func(t *testing.T) {
|
||||
got := a.sentenceWithMediaMarkers("原句。", []string{"ffffffffffff"})
|
||||
if got != "原句。" {
|
||||
t.Errorf("不存在的 digest 不该造出标记: %q", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("无媒体存储时原样返回", func(t *testing.T) {
|
||||
bare := &Agent{}
|
||||
if got := bare.sentenceWithMediaMarkers("原句。", []string{digest}); got != "原句。" {
|
||||
t.Errorf("无媒体存储时应原样返回: %q", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ---------- resolveMediaDigests ----------
|
||||
|
||||
func TestResolveMediaDigests(t *testing.T) {
|
||||
a, ms := newInputTestAgent(t)
|
||||
d1, _ := ms.Put([]byte("one"), media.Item{MIME: "image/png"})
|
||||
d2, _ := ms.Put([]byte("two"), media.Item{MIME: "image/png"})
|
||||
|
||||
got := a.resolveMediaDigests([]string{d1[:10], d2, d1, "ffffffffffff"})
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("got = %v,期望 2 条(去重 + 丢弃无法解析的)", got)
|
||||
}
|
||||
for _, d := range got {
|
||||
if len(d) != 64 {
|
||||
t.Errorf("应返回完整 digest,实际 %q", d)
|
||||
}
|
||||
}
|
||||
|
||||
if a.resolveMediaDigests(nil) != nil {
|
||||
t.Error("空输入应返回 nil")
|
||||
}
|
||||
bare := &Agent{}
|
||||
if bare.resolveMediaDigests([]string{d1}) != nil {
|
||||
t.Error("无媒体存储时应返回 nil")
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- bindDocMedia ----------
|
||||
|
||||
func TestBindDocMedia(t *testing.T) {
|
||||
a, ms := newInputTestAgent(t)
|
||||
d1, _ := ms.Put([]byte("doc-one"), media.Item{MIME: "image/png"})
|
||||
d2, _ := ms.Put([]byte("doc-two"), media.Item{MIME: "image/png"})
|
||||
|
||||
if n := a.bindDocMedia("doc_x", []string{d1, d2}); n != 2 {
|
||||
t.Fatalf("绑定 %d 条,期望 2", n)
|
||||
}
|
||||
refs, err := ms.Refs(media.OwnerDocument, "doc_x")
|
||||
if err != nil {
|
||||
t.Fatalf("Refs: %v", err)
|
||||
}
|
||||
if len(refs) != 2 {
|
||||
t.Errorf("引用 = %v,期望 2 条", refs)
|
||||
}
|
||||
|
||||
if n := a.bindDocMedia("", []string{d1}); n != 0 {
|
||||
t.Error("空 docID 不该绑定")
|
||||
}
|
||||
bare := &Agent{}
|
||||
if n := bare.bindDocMedia("doc_y", []string{d1}); n != 0 {
|
||||
t.Error("无媒体存储时不该绑定")
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- docMediaContext ----------
|
||||
|
||||
func TestDocMediaContext(t *testing.T) {
|
||||
a, ms := newInputTestAgent(t)
|
||||
digest, _ := ms.Put([]byte("ctx-bytes"), media.Item{
|
||||
MIME: "image/png", Description: "文档里的配图",
|
||||
})
|
||||
|
||||
t.Run("优先用media_refs", func(t *testing.T) {
|
||||
if err := ms.AddRef(digest, media.OwnerDocument, "doc_refs"); err != nil {
|
||||
t.Fatalf("AddRef: %v", err)
|
||||
}
|
||||
got := a.docMediaContext("doc_refs", "正文里没有任何标记")
|
||||
if !strings.Contains(got, "文档里的配图") {
|
||||
t.Errorf("未从 media_refs 取到媒体说明: %q", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("无引用时回退解析正文标记", func(t *testing.T) {
|
||||
content := "旧正文 [image/png " + digest[:12] + "] 文档里的配图"
|
||||
got := a.docMediaContext("doc_legacy", content)
|
||||
if !strings.Contains(got, "文档里的配图") {
|
||||
t.Errorf("历史文档只有标记时应回退解析: %q", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("既无引用也无标记", func(t *testing.T) {
|
||||
if got := a.docMediaContext("doc_empty", "普通正文"); got != "" {
|
||||
t.Errorf("应返回空串,实际 %q", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("无媒体存储", func(t *testing.T) {
|
||||
bare := &Agent{}
|
||||
if got := bare.docMediaContext("doc_x", "任意"); got != "" {
|
||||
t.Errorf("无媒体存储时应返回空串,实际 %q", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ---------- mediaMarkerLine ----------
|
||||
|
||||
// 标记格式的唯一生成处。此前 mediaSummaryForEvent 与 mediaContextForSentences
|
||||
// 各拼一份,改动截断长度或分隔符时只改一处,另一处写出的标记就再也解析不回来。
|
||||
func TestMediaMarkerLine(t *testing.T) {
|
||||
a, ms := newInputTestAgent(t)
|
||||
|
||||
described, _ := ms.Put([]byte("with-desc"), media.Item{
|
||||
MIME: "image/png", Description: "已描述的图",
|
||||
})
|
||||
if got := a.mediaMarkerLine(described); !strings.Contains(got, "已描述的图") {
|
||||
t.Errorf("有描述时应带描述: %q", got)
|
||||
}
|
||||
|
||||
// 「已入库但还没描述」与「压根没有媒体」必须可区分
|
||||
bare, _ := ms.Put([]byte("no-desc"), media.Item{MIME: "image/png"})
|
||||
got := a.mediaMarkerLine(bare)
|
||||
if !strings.Contains(got, "(未描述)") {
|
||||
t.Errorf("无描述时应有占位符: %q", got)
|
||||
}
|
||||
if !strings.Contains(got, shortDigest(bare)) {
|
||||
t.Errorf("必须带短 digest 供反查: %q", got)
|
||||
}
|
||||
|
||||
// 查不到返回空串:媒体可能已被容量 GC 淘汰,此时不该造出指向虚无的标记
|
||||
if got := a.mediaMarkerLine("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"); got != "" {
|
||||
t.Errorf("查不到的 digest 应返回空串,实际 %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 模型工具端到端:memory_commit / doc_commit / doc_query ----------
|
||||
|
||||
func newToolTestAgent(t *testing.T) (*Agent, *media.Store) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
|
||||
g, err := memory.NewGraphDB(filepath.Join(dir, "graph.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("NewGraphDB: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { g.Close() })
|
||||
|
||||
ds := document.NewStore(filepath.Join(dir, "documents"))
|
||||
if err := ds.Start(); err != nil {
|
||||
t.Fatalf("doc store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { ds.Stop() })
|
||||
|
||||
ms, err := media.New(filepath.Join(dir, "media"), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("media.New: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { ms.Close() })
|
||||
|
||||
emb := memory.NewStaticEmbedder("")
|
||||
a := &Agent{
|
||||
id: "tester",
|
||||
memory: g,
|
||||
docStore: ds,
|
||||
mediaStore: ms,
|
||||
context: NewRelevanceContext("", emb),
|
||||
}
|
||||
return a, ms
|
||||
}
|
||||
|
||||
// memory_commit 带 media_digests:三元组入库后必须能从句子反查回那份字节。
|
||||
func TestToolMemoryCommit_BindsMedia(t *testing.T) {
|
||||
a, ms := newToolTestAgent(t)
|
||||
digest, _ := ms.Put([]byte("commit-bytes"), media.Item{
|
||||
MIME: "image/png", Description: "提交时关联的图",
|
||||
})
|
||||
|
||||
out := a.executeMemoryTool(agentAPI.ToolCall{
|
||||
Name: "memory_commit",
|
||||
Arguments: map[string]interface{}{
|
||||
"triples": []interface{}{
|
||||
map[string]interface{}{
|
||||
"subject": "配色方案",
|
||||
"relation": "参考",
|
||||
"object": "三色带图",
|
||||
"media_digests": []interface{}{digest[:12]},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if !strings.Contains(out, "关联") {
|
||||
t.Errorf("返回值应告知模型媒体已关联: %q", out)
|
||||
}
|
||||
|
||||
res, err := a.memory.Recall([]string{"配色方案"}, nil, 2, "")
|
||||
if err != nil {
|
||||
t.Fatalf("Recall: %v", err)
|
||||
}
|
||||
if len(res.Relations) == 0 || res.Relations[0].SentenceID == 0 {
|
||||
t.Fatal("没有句子落点 —— 媒体引用无从挂起")
|
||||
}
|
||||
refs, _ := ms.Refs(media.OwnerGraphSentence, strconv.FormatInt(res.Relations[0].SentenceID, 10))
|
||||
if len(refs) != 1 || refs[0] != digest {
|
||||
t.Errorf("句子引用 = %v,期望 [%s]", refs, digest)
|
||||
}
|
||||
}
|
||||
|
||||
// 不带 media_digests 时行为与本特性上线前一致(不多写句子、不报错)。
|
||||
func TestToolMemoryCommit_WithoutMedia(t *testing.T) {
|
||||
a, _ := newToolTestAgent(t)
|
||||
out := a.executeMemoryTool(agentAPI.ToolCall{
|
||||
Name: "memory_commit",
|
||||
Arguments: map[string]interface{}{
|
||||
"triples": []interface{}{
|
||||
map[string]interface{}{"subject": "甲方", "relation": "签署", "object": "合同"},
|
||||
},
|
||||
},
|
||||
})
|
||||
if strings.Contains(out, "失败") {
|
||||
t.Errorf("普通提交不该失败: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "关联") {
|
||||
t.Errorf("无媒体时不该提媒体: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
// sentence_text 必须透传:丢了它,图谱就回不到原文。
|
||||
func TestToolMemoryCommit_CarriesSentenceText(t *testing.T) {
|
||||
a, _ := newToolTestAgent(t)
|
||||
a.executeMemoryTool(agentAPI.ToolCall{
|
||||
Name: "memory_commit",
|
||||
Arguments: map[string]interface{}{
|
||||
"triples": []interface{}{
|
||||
map[string]interface{}{
|
||||
"subject": "李四",
|
||||
"relation": "住在",
|
||||
"object": "杭州",
|
||||
"sentence_text": "李四搬到杭州已经三年了。",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
res, _ := a.memory.Recall([]string{"李四"}, nil, 2, "")
|
||||
if len(res.Relations) == 0 {
|
||||
t.Fatal("召回为空")
|
||||
}
|
||||
if res.Relations[0].SentenceText != "李四搬到杭州已经三年了。" {
|
||||
t.Errorf("SentenceText = %q", res.Relations[0].SentenceText)
|
||||
}
|
||||
}
|
||||
|
||||
// doc_commit 带 media_digests:标记进正文(否则检索不到)+ 引用挂文档 owner(否则 GC 会清)。
|
||||
func TestToolDocCommit_BindsMedia(t *testing.T) {
|
||||
a, ms := newToolTestAgent(t)
|
||||
digest, _ := ms.Put([]byte("doc-commit-bytes"), media.Item{
|
||||
MIME: "image/png", Description: "笔记里的插图",
|
||||
})
|
||||
|
||||
out := a.executeDocTool(agentAPI.ToolCall{
|
||||
Name: "doc_commit",
|
||||
Arguments: map[string]interface{}{
|
||||
"content": "这是一篇带图的笔记正文。",
|
||||
"summary": "带图笔记",
|
||||
"media_digests": []interface{}{digest[:12]},
|
||||
},
|
||||
})
|
||||
if !strings.Contains(out, "关联") {
|
||||
t.Errorf("返回值应告知模型媒体已关联: %q", out)
|
||||
}
|
||||
|
||||
docs := a.docStore.RecentDocs(5)
|
||||
if len(docs) == 0 {
|
||||
t.Fatal("文档未写入")
|
||||
}
|
||||
d := docs[0]
|
||||
if !strings.Contains(d.Content, "笔记里的插图") {
|
||||
t.Errorf("标记未进正文(向量索引看不到这份媒体): %q", d.Content)
|
||||
}
|
||||
refs, _ := ms.Refs(media.OwnerDocument, d.ID)
|
||||
if len(refs) != 1 || refs[0] != digest {
|
||||
t.Errorf("文档引用 = %v,期望 [%s]", refs, digest)
|
||||
}
|
||||
}
|
||||
|
||||
// doc_query 必须把媒体说明附在返回值里,否则模型检索到带图文档也不知道有图。
|
||||
func TestToolDocQuery_ShowsMedia(t *testing.T) {
|
||||
a, ms := newToolTestAgent(t)
|
||||
digest, _ := ms.Put([]byte("query-bytes"), media.Item{
|
||||
MIME: "image/png", Description: "检索命中的配图",
|
||||
})
|
||||
|
||||
a.executeDocTool(agentAPI.ToolCall{
|
||||
Name: "doc_commit",
|
||||
Arguments: map[string]interface{}{
|
||||
"content": "紫蓝红三色带配色说明正文",
|
||||
"summary": "紫蓝红三色带",
|
||||
"media_digests": []interface{}{digest},
|
||||
},
|
||||
})
|
||||
|
||||
a.executeDocTool(agentAPI.ToolCall{
|
||||
Name: "doc_query",
|
||||
Arguments: map[string]interface{}{"query": "紫蓝红三色带 配色说明", "top_k": float64(3)},
|
||||
})
|
||||
|
||||
// 正文进的是 cold_storage 事件(工具返回值只给引用编号),媒体说明也在那里。
|
||||
var found bool
|
||||
for _, e := range a.context.Recent(10) {
|
||||
if strings.Contains(e.Response, "检索命中的配图") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("doc_query 未把媒体说明带进上下文 —— 模型不知道这篇文档带过图")
|
||||
}
|
||||
}
|
||||
|
||||
// 无媒体存储时三个工具的行为与本特性上线前完全一致。
|
||||
func TestTools_NilMediaStoreDegrades(t *testing.T) {
|
||||
a, _ := newToolTestAgent(t)
|
||||
a.mediaStore = nil
|
||||
|
||||
out := a.executeMemoryTool(agentAPI.ToolCall{
|
||||
Name: "memory_commit",
|
||||
Arguments: map[string]interface{}{
|
||||
"triples": []interface{}{
|
||||
map[string]interface{}{
|
||||
"subject": "无存储", "relation": "仍可", "object": "提交",
|
||||
"media_digests": []interface{}{"aabbccddeeff"},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
if strings.Contains(out, "失败") {
|
||||
t.Errorf("无媒体存储时提交不该失败: %q", out)
|
||||
}
|
||||
|
||||
out = a.executeDocTool(agentAPI.ToolCall{
|
||||
Name: "doc_commit",
|
||||
Arguments: map[string]interface{}{
|
||||
"content": "无媒体存储的文档",
|
||||
"media_digests": []interface{}{"aabbccddeeff"},
|
||||
},
|
||||
})
|
||||
if strings.Contains(out, "失败") {
|
||||
t.Errorf("无媒体存储时文档写入不该失败: %q", out)
|
||||
}
|
||||
}
|
||||
@ -15,7 +15,7 @@ import (
|
||||
//
|
||||
// 为何需要这一层:媒体进入对话有两条路,两条都只把**文字**留给记忆——
|
||||
//
|
||||
// 1. 用户直接发图 → processMediaInput → mediaToBlocks
|
||||
// 1. 用户直接发图 → processInput/resolveInput → mediaToBlocks
|
||||
// ContextEvent.Input 只存 alt 文本("[从 qq 收到了 image]"),
|
||||
// base64 随 message 数组发给模型后就丢了。
|
||||
// 2. 插件注入 → SetToolBlocks → process.go 的 mediaMsg
|
||||
@ -123,18 +123,8 @@ func (a *Agent) mediaSummaryForEvent(digests []string) string {
|
||||
}
|
||||
var lines []string
|
||||
for _, d := range digests {
|
||||
it, err := a.mediaStore.Stat(d)
|
||||
if err != nil || it == nil {
|
||||
continue
|
||||
}
|
||||
label := string(it.Kind)
|
||||
if it.MIME != "" {
|
||||
label = it.MIME
|
||||
}
|
||||
if it.Description != "" {
|
||||
lines = append(lines, fmt.Sprintf("[%s %s] %s", label, shortDigest(d), it.Description))
|
||||
} else {
|
||||
lines = append(lines, fmt.Sprintf("[%s %s] (未描述)", label, shortDigest(d)))
|
||||
if line := a.mediaMarkerLine(d); line != "" {
|
||||
lines = append(lines, line)
|
||||
}
|
||||
}
|
||||
if len(lines) == 0 {
|
||||
@ -143,6 +133,34 @@ func (a *Agent) mediaSummaryForEvent(digests []string) string {
|
||||
return "媒体内容:\n" + strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// mediaMarkerLine 为一份媒体生成一行标记文本 `[<mime> <短digest>] <描述>`。
|
||||
//
|
||||
// 这是媒体标记格式的唯一生成处。此前 mediaSummaryForEvent 与
|
||||
// mediaContextForSentences 各拼一份,改动截断长度或分隔符时只改一处,
|
||||
// 另一处写出的标记就再也解析不回来——而解析失败是静默的(引用挂不上)。
|
||||
//
|
||||
// 查不到返回空串:媒体可能已被容量 GC 淘汰,此时不该造出一条指向虚无的标记。
|
||||
func (a *Agent) mediaMarkerLine(digest string) string {
|
||||
if a.mediaStore == nil {
|
||||
return ""
|
||||
}
|
||||
it, err := a.mediaStore.Stat(digest)
|
||||
if err != nil || it == nil {
|
||||
return ""
|
||||
}
|
||||
label := string(it.Kind)
|
||||
if it.MIME != "" {
|
||||
label = it.MIME
|
||||
}
|
||||
desc := it.Description
|
||||
if desc == "" {
|
||||
// 「已入库但还没描述」与「压根没有媒体」必须可区分:描述由后台循环
|
||||
// 异步补齐,占位符保证补齐前这份媒体也不会从文本里消失。
|
||||
desc = "(未描述)"
|
||||
}
|
||||
return fmt.Sprintf("[%s %s] %s", label, shortDigest(digest), desc)
|
||||
}
|
||||
|
||||
// newEventID 生成 ContextEvent 的稳定标识。
|
||||
//
|
||||
// 沿用 document.Store 的 doc_<unixnano> 手法(同一份代码库里保持一致,
|
||||
|
||||
@ -185,9 +185,16 @@ func (a *Agent) executeMemoryTool(tc agentAPI.ToolCall) string {
|
||||
for _, td := range triplesData {
|
||||
if m, ok := td.(map[string]interface{}); ok {
|
||||
t := memory.Triple{
|
||||
Subject: getString(m, "subject"),
|
||||
Relation: getString(m, "relation"),
|
||||
Object: getString(m, "object"),
|
||||
Subject: getString(m, "subject"),
|
||||
Relation: getString(m, "relation"),
|
||||
Object: getString(m, "object"),
|
||||
SentenceText: getString(m, "sentence_text"),
|
||||
}
|
||||
// 模型显式关联的媒体:标记由内核补进句子文本,模型不必知道格式。
|
||||
// 没有 sentence_text 时 sentenceWithMediaMarkers 会用标记本身
|
||||
// 充当句子——媒体必须有句子落点,否则 media_refs 无从挂起。
|
||||
if digests := getStringSlice(m, "media_digests"); len(digests) > 0 {
|
||||
t.SentenceText = a.sentenceWithMediaMarkers(t.SentenceText, digests)
|
||||
}
|
||||
if t.Subject != "" && t.Relation != "" && t.Object != "" {
|
||||
triples = append(triples, t)
|
||||
@ -199,10 +206,13 @@ func (a *Agent) executeMemoryTool(tc agentAPI.ToolCall) string {
|
||||
}
|
||||
// remember 工具是用户/模型显式写入,不涉及归档删除,
|
||||
// 因此不需要 mediaBound——没有旧引用要释放。
|
||||
ec, rc, _, err := a.commitTriplesWithMedia(triples, string(a.id), 0)
|
||||
ec, rc, mb, err := a.commitTriplesWithMedia(triples, string(a.id), 0)
|
||||
if err != nil {
|
||||
return fmt.Sprintf("记忆写入失败: %v", err)
|
||||
}
|
||||
if mb > 0 {
|
||||
return fmt.Sprintf("已写入 %d 个实体和 %d 条关系,关联 %d 份媒体", ec, rc, mb)
|
||||
}
|
||||
return fmt.Sprintf("已写入 %d 个实体和 %d 条关系", ec, rc)
|
||||
|
||||
case "memory_introspect":
|
||||
@ -521,6 +531,11 @@ func (a *Agent) executeDocTool(tc agentAPI.ToolCall) string {
|
||||
if len(content) > 2000 {
|
||||
content = content[:2000] + "..."
|
||||
}
|
||||
// 媒体说明单独一行进冷存事件:正文可能被上面的 2000 字截断,
|
||||
// 而媒体标记往往在文档末尾——截掉之后模型就不知道这篇文档带过图。
|
||||
if mc := a.docMediaContext(d.ID, d.Content); mc != "" {
|
||||
content = content + "\n关联媒体: " + mc
|
||||
}
|
||||
a.context.InsertByTimestamp(ContextEvent{
|
||||
Timestamp: d.CreatedAt,
|
||||
Source: "cold_storage",
|
||||
@ -556,9 +571,21 @@ func (a *Agent) executeDocTool(tc agentAPI.ToolCall) string {
|
||||
Tags: tags,
|
||||
Source: "manual",
|
||||
}
|
||||
|
||||
// 模型显式关联的媒体:标记补进正文后再写入。顺序关键——向量索引用
|
||||
// Summary+Content 计算,标记进不去正文就检索不到这份媒体。
|
||||
mediaDigests := a.resolveMediaDigests(getStringSlice(tc.Arguments, "media_digests"))
|
||||
doc.Content = a.sentenceWithMediaMarkers(doc.Content, mediaDigests)
|
||||
|
||||
if err := a.docStore.Insert(doc); err != nil {
|
||||
return fmt.Sprintf("文档写入失败: %v", err)
|
||||
}
|
||||
// 引用必须在拿到 doc.ID 之后挂:owner_id 就是文档 id。
|
||||
// 不挂的后果是这些媒体在文档里可见却无主,下一轮 GC 会把它们清掉。
|
||||
bound := a.bindDocMedia(doc.ID, mediaDigests)
|
||||
if bound > 0 {
|
||||
return fmt.Sprintf("文档已提交 (id: %s, 摘要: %s, 关联 %d 份媒体)", doc.ID, summary, bound)
|
||||
}
|
||||
return fmt.Sprintf("文档已提交 (id: %s, 摘要: %s)", doc.ID, summary)
|
||||
|
||||
default:
|
||||
|
||||
@ -374,6 +374,11 @@ func (a *Agent) buildToolDefs() []interface{} {
|
||||
"description": "标签列表",
|
||||
"items": map[string]interface{}{"type": "string"},
|
||||
},
|
||||
"media_digests": map[string]interface{}{
|
||||
"type": "array",
|
||||
"description": "可选:这篇文档关联的媒体 digest(对话或 memory_recall 的「关联媒体」里显示的十六进制串,短的即可)。填了以后检索到这篇文档就能看到并取回原图/音频。",
|
||||
"items": map[string]interface{}{"type": "string"},
|
||||
},
|
||||
},
|
||||
"required": []string{"content"},
|
||||
},
|
||||
|
||||
@ -29,6 +29,24 @@ func getFloat(m map[string]interface{}, key string) float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
// getStringSlice 从工具参数里取字符串数组。
|
||||
//
|
||||
// 需要单独一个 helper 而不是直接断言 []string:LLM 的参数经 JSON 解码后是
|
||||
// []interface{},直接断言 []string 恒失败——静默拿到 nil,参数像没传一样。
|
||||
func getStringSlice(m map[string]interface{}, key string) []string {
|
||||
raw, ok := m[key].([]interface{})
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, v := range raw {
|
||||
if s, ok := v.(string); ok && s != "" {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func truncateStr(s string, max int) string {
|
||||
if utf8.RuneCountInString(s) <= max {
|
||||
return s
|
||||
|
||||
@ -174,7 +174,9 @@ func (idx *Indexer) BuildToolPrompt() string {
|
||||
### memory_commit
|
||||
将三元组写入图记忆。
|
||||
参数:
|
||||
- triples: [{"subject": "实体名", "relation": "关系类型", "object": "目标实体"}]
|
||||
- triples: [{"subject": "实体名", "relation": "关系类型", "object": "目标实体",
|
||||
"sentence_text": "原始句子(可选)", "media_digests": ["图片digest(可选)"]}]
|
||||
填了 media_digests,日后从这条记忆就能取回当时那张图/那段音频。
|
||||
|
||||
### memory_introspect
|
||||
查看记忆统计信息。
|
||||
@ -261,6 +263,15 @@ func (idx *Indexer) GetToolDefinitions() []map[string]interface{} {
|
||||
"subject": map[string]interface{}{"type": "string"},
|
||||
"relation": map[string]interface{}{"type": "string"},
|
||||
"object": map[string]interface{}{"type": "string"},
|
||||
"sentence_text": map[string]interface{}{
|
||||
"type": "string",
|
||||
"description": "可选:这条三元组的原始句子。填了才能日后从图谱回到原文。",
|
||||
},
|
||||
"media_digests": map[string]interface{}{
|
||||
"type": "array",
|
||||
"description": "可选:这条记忆关联的媒体 digest(对话或 memory_recall 的「关联媒体」里显示的十六进制串,短的即可)。填了以后从这条记忆能取回原图/音频。",
|
||||
"items": map[string]interface{}{"type": "string"},
|
||||
},
|
||||
},
|
||||
"required": []string{"subject", "relation", "object"},
|
||||
},
|
||||
|
||||
@ -10,7 +10,15 @@ var (
|
||||
// 1.0.0:外部插件从 C ABI 动态库迁到子进程 + 共享内存。
|
||||
// 这是首个不再加载 `.so`/`.dll` 的版本,与 0.9.x 不兼容(存量插件必须
|
||||
// 用新版 plugindev 重编),故跃到主版本号。
|
||||
Version = "1.0.0"
|
||||
// 1.1.0:记忆系统支持二进制多媒体节点——CAS 媒体存储 + L0/L2/L3 贯通。
|
||||
// 1.1.1:多模态贯通**插件边界**。内核实现公开 SDK 1.1.0 新增的媒体接口
|
||||
// (doc.insertWithMedia、io.injectMedia / injectMediaSync /
|
||||
// injectInterruptMedia),并把 text/image/audio 三条输入路径归一成
|
||||
// 一条 processInput 主干。
|
||||
//
|
||||
// 发布分支上此值是**本条发布线当前的版本号**;main 上则是下一个未发布中版本
|
||||
//(见 docs/git-branching.md §2.1 与 §四)。
|
||||
Version = "1.1.1"
|
||||
|
||||
// Commit 是构建时的 Git commit hash。
|
||||
Commit = "unknown"
|
||||
@ -22,7 +30,11 @@ var (
|
||||
KernelName = "HomeAgent"
|
||||
|
||||
// SDKCompatibleVersion 是此内核可兼容的最高 SDK 版本(semver)。
|
||||
SDKCompatibleVersion = "1.0.0"
|
||||
//
|
||||
// 1.1.0:本内核实现了 SDK 1.1.0 的全部新增方法。
|
||||
// 用 SDK 1.0.0 编的存量插件照旧可用——新增方法由**插件调用、内核实现**,
|
||||
// 不调就不受影响,无需重编。
|
||||
SDKCompatibleVersion = "1.1.0"
|
||||
)
|
||||
|
||||
// FullVersion 返回完整的版本字符串。
|
||||
|
||||
@ -101,6 +101,11 @@ var methodCapability = map[string]Capability{
|
||||
MethodIOInjectInterrupt: CapIO,
|
||||
MethodIOInjectTextNoMem: CapIO,
|
||||
MethodIOInjectSync: CapIO,
|
||||
// 带媒体的注入与纯文本注入同一权限组:能不能发起一轮对话是 IO 能力,
|
||||
// 带不带图不改变这个判断。
|
||||
MethodIOInjectMedia: CapIO,
|
||||
MethodIOInjectMediaSync: CapIO,
|
||||
MethodIOInjectInterruptMedia: CapIO,
|
||||
|
||||
// ---- 图记忆 ----
|
||||
MethodMemoryRecall: CapMemory,
|
||||
@ -114,6 +119,8 @@ var methodCapability = map[string]Capability{
|
||||
MethodDocInsert: CapDocMemory,
|
||||
MethodDocRemove: CapDocMemory,
|
||||
MethodDocStats: CapDocMemory,
|
||||
// 带媒体写入与普通写入同权限:都是往文档记忆里写东西。
|
||||
MethodDocInsertMedia: CapDocMemory,
|
||||
|
||||
// ---- 知识库 ----
|
||||
MethodKnowledgeSearch: CapKnowledge,
|
||||
|
||||
@ -34,10 +34,12 @@ func TestCapability_AllMethodsClassified(t *testing.T) {
|
||||
MethodAPIRegister, MethodInputRegister,
|
||||
MethodIOInjectText, MethodIOInjectInterrupt, MethodIOInjectTextNoMem,
|
||||
MethodIOInjectSync, MethodIOSetToolBlocks,
|
||||
MethodIOInjectMedia, MethodIOInjectMediaSync, MethodIOInjectInterruptMedia,
|
||||
MethodLifecycleAutoRestart,
|
||||
MethodMemoryRecall, MethodMemoryCommit, MethodMemoryIntrospect,
|
||||
MethodMemoryMerge, MethodMemoryPurge,
|
||||
MethodDocQuery, MethodDocInsert, MethodDocRemove, MethodDocStats,
|
||||
MethodDocInsertMedia,
|
||||
MethodKnowledgeSearch, MethodKnowledgeAdd, MethodKnowledgeList,
|
||||
MethodTextMemoryAppend,
|
||||
MethodSettingsGet, MethodSettingsSet, MethodSettingsRegisterDef,
|
||||
|
||||
@ -91,6 +91,10 @@ type CoreSDK interface {
|
||||
InjectInterruptText(source, channel, text string)
|
||||
InjectTextNoMemory(source, channel, text string)
|
||||
InjectInputSync(source, channel, text string) string
|
||||
// 带媒体的注入:子进程插件也能主动发起一轮带图/音频的对话。
|
||||
InjectInputMedia(source, channel, text string, blocks []pubsdk.ContentBlock)
|
||||
InjectInputMediaSync(source, channel, text string, blocks []pubsdk.ContentBlock) string
|
||||
InjectInterruptMedia(source, channel, text string, blocks []pubsdk.ContentBlock)
|
||||
|
||||
SetAutoRestart(enabled bool)
|
||||
}
|
||||
@ -163,6 +167,30 @@ func (h *coreHandler) Handle(method string, params json.RawMessage) (interface{}
|
||||
}
|
||||
return map[string]interface{}{"reply": h.sdk.InjectInputSync(p.Source, p.Channel, p.Text)}, nil
|
||||
|
||||
case MethodIOInjectMedia:
|
||||
var p injectMediaParams
|
||||
if err := unmarshal(params, &p); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h.sdk.InjectInputMedia(p.Source, p.Channel, p.Text, p.Blocks)
|
||||
return nil, nil
|
||||
|
||||
case MethodIOInjectMediaSync:
|
||||
var p injectMediaParams
|
||||
if err := unmarshal(params, &p); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reply := h.sdk.InjectInputMediaSync(p.Source, p.Channel, p.Text, p.Blocks)
|
||||
return map[string]interface{}{"reply": reply}, nil
|
||||
|
||||
case MethodIOInjectInterruptMedia:
|
||||
var p injectMediaParams
|
||||
if err := unmarshal(params, &p); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h.sdk.InjectInterruptMedia(p.Source, p.Channel, p.Text, p.Blocks)
|
||||
return nil, nil
|
||||
|
||||
// ---- 生命周期(原 case 8)----
|
||||
case MethodLifecycleAutoRestart:
|
||||
var p struct {
|
||||
@ -293,6 +321,28 @@ func (h *coreHandler) Handle(method string, params json.RawMessage) (interface{}
|
||||
}
|
||||
return nil, dm.Insert(p.Doc)
|
||||
|
||||
case MethodDocInsertMedia:
|
||||
dm := h.sdk.DocMemory()
|
||||
if dm == nil {
|
||||
return nil, errUnavailable("doc memory")
|
||||
}
|
||||
var p struct {
|
||||
Doc *pubsdk.Doc `json:"doc"`
|
||||
Attachments []pubsdk.MediaAttachment `json:"attachments"`
|
||||
}
|
||||
if err := unmarshal(params, &p); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if p.Doc == nil {
|
||||
return nil, fmt.Errorf("doc.insertWithMedia: 缺少 doc 字段")
|
||||
}
|
||||
if err := dm.InsertWithMedia(p.Doc, p.Attachments); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// 回传内核补过的字段:ID 新建时才生成,Content 含内核补的媒体标记,
|
||||
// MediaDigests 是附件落盘后的完整 digest——插件靠它们后续引用同一份媒体。
|
||||
return map[string]interface{}{"doc": p.Doc}, nil
|
||||
|
||||
case MethodDocRemove:
|
||||
dm := h.sdk.DocMemory()
|
||||
if dm == nil {
|
||||
@ -505,6 +555,17 @@ type injectParams struct {
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
// injectMediaParams 是带媒体注入的参数。
|
||||
//
|
||||
// blocks 走 JSON(而非共享段二进制通道):data URL 已经是 base64 文本,
|
||||
// 再套一层二进制传输不会更小,而 JSON 让这条路径与其他 method 一致。
|
||||
type injectMediaParams struct {
|
||||
Source string `json:"source"`
|
||||
Channel string `json:"channel"`
|
||||
Text string `json:"text"`
|
||||
Blocks []pubsdk.ContentBlock `json:"blocks"`
|
||||
}
|
||||
|
||||
func unmarshal(params json.RawMessage, out interface{}) error {
|
||||
if len(params) == 0 {
|
||||
return nil
|
||||
|
||||
@ -34,21 +34,26 @@ func newFakeCore() *fakeCoreSDK {
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fakeCoreSDK) PluginName() string { return "fake" }
|
||||
func (f *fakeCoreSDK) Settings() pubsdk.SettingsAPI { return nil }
|
||||
func (f *fakeCoreSDK) Memory() pubsdk.MemoryAPI { return nil }
|
||||
func (f *fakeCoreSDK) TextMemory() pubsdk.TextMemoryAPI { return nil }
|
||||
func (f *fakeCoreSDK) DocMemory() pubsdk.DocMemoryAPI { return nil }
|
||||
func (f *fakeCoreSDK) Knowledge() pubsdk.KnowledgeAPI { return nil }
|
||||
func (f *fakeCoreSDK) LLM() pubsdk.LLMAPI { return nil }
|
||||
func (f *fakeCoreSDK) Social() pubsdk.SocialAPI { return nil }
|
||||
func (f *fakeCoreSDK) PluginMgr() pubsdk.PluginMgrAPI { return nil }
|
||||
func (f *fakeCoreSDK) RegisterPluginAPI(name string) error { return nil }
|
||||
func (f *fakeCoreSDK) InjectText(s, c, t string) {}
|
||||
func (f *fakeCoreSDK) InjectInterruptText(s, c, t string) {}
|
||||
func (f *fakeCoreSDK) InjectTextNoMemory(s, c, t string) {}
|
||||
func (f *fakeCoreSDK) InjectInputSync(s, c, t string) string { return "" }
|
||||
func (f *fakeCoreSDK) SetAutoRestart(enabled bool) { f.autoStart = enabled }
|
||||
func (f *fakeCoreSDK) PluginName() string { return "fake" }
|
||||
func (f *fakeCoreSDK) Settings() pubsdk.SettingsAPI { return nil }
|
||||
func (f *fakeCoreSDK) Memory() pubsdk.MemoryAPI { return nil }
|
||||
func (f *fakeCoreSDK) TextMemory() pubsdk.TextMemoryAPI { return nil }
|
||||
func (f *fakeCoreSDK) DocMemory() pubsdk.DocMemoryAPI { return nil }
|
||||
func (f *fakeCoreSDK) Knowledge() pubsdk.KnowledgeAPI { return nil }
|
||||
func (f *fakeCoreSDK) LLM() pubsdk.LLMAPI { return nil }
|
||||
func (f *fakeCoreSDK) Social() pubsdk.SocialAPI { return nil }
|
||||
func (f *fakeCoreSDK) PluginMgr() pubsdk.PluginMgrAPI { return nil }
|
||||
func (f *fakeCoreSDK) RegisterPluginAPI(name string) error { return nil }
|
||||
func (f *fakeCoreSDK) InjectText(s, c, t string) {}
|
||||
func (f *fakeCoreSDK) InjectInterruptText(s, c, t string) {}
|
||||
func (f *fakeCoreSDK) InjectTextNoMemory(s, c, t string) {}
|
||||
func (f *fakeCoreSDK) InjectInputSync(s, c, t string) string { return "" }
|
||||
func (f *fakeCoreSDK) InjectInputMedia(s, c, t string, b []pubsdk.ContentBlock) {}
|
||||
func (f *fakeCoreSDK) InjectInputMediaSync(s, c, t string, b []pubsdk.ContentBlock) string {
|
||||
return ""
|
||||
}
|
||||
func (f *fakeCoreSDK) InjectInterruptMedia(s, c, t string, b []pubsdk.ContentBlock) {}
|
||||
func (f *fakeCoreSDK) SetAutoRestart(enabled bool) { f.autoStart = enabled }
|
||||
|
||||
func (f *fakeCoreSDK) RegisterTool(name string, def pubsdk.ToolDef, h pubsdk.ToolHandler) error {
|
||||
f.mu.Lock()
|
||||
|
||||
@ -74,6 +74,13 @@ const (
|
||||
MethodIOInjectInterrupt = "io.injectInterrupt" // 6 CORE_INJECT_INTERRUPT_TEXT
|
||||
MethodIOInjectTextNoMem = "io.injectTextNoMem" // 7 CORE_INJECT_TEXT_NO_MEMORY
|
||||
MethodIOInjectSync = "io.injectInputSync" // 47 CORE_INJECT_INPUT_SYNC
|
||||
// 带媒体的注入:blocks 随参数 JSON 一并过来,内核侧转成
|
||||
// payload["media_blocks"],由 resolveInput 归一进统一输入主干。
|
||||
// 与 SetToolBlocks 的区别:这三个是「主动发起一轮带图的对话」,
|
||||
// 后者是「工具返回值里带图」,只能在工具调用内部用。
|
||||
MethodIOInjectMedia = "io.injectMedia"
|
||||
MethodIOInjectMediaSync = "io.injectMediaSync"
|
||||
MethodIOInjectInterruptMedia = "io.injectInterruptMedia"
|
||||
// MethodIOSetToolBlocks 多模态注入——今日 C ABI 侧是空实现(§1.4),
|
||||
// 子进程下二进制落 arena、描述符回传,首次真正可用。
|
||||
MethodIOSetToolBlocks = "io.setToolBlocks"
|
||||
@ -93,6 +100,9 @@ const (
|
||||
MethodDocInsert = "doc.insert" // 32
|
||||
MethodDocRemove = "doc.remove" // 33
|
||||
MethodDocStats = "doc.stats" // 34
|
||||
// MethodDocInsertMedia 写入文档并关联媒体(附件带 data 则落盘去重,
|
||||
// 只带 digest 则引用已有内容)。
|
||||
MethodDocInsertMedia = "doc.insertWithMedia"
|
||||
|
||||
// 知识库(原 case 15/35/36)
|
||||
MethodKnowledgeSearch = "knowledge.search" // 15
|
||||
|
||||
@ -142,6 +142,23 @@ func (c procCore) InjectInputSync(source, channel, text string) string {
|
||||
return reply
|
||||
}
|
||||
|
||||
// ---- 带媒体的 IO 注入 ----
|
||||
//
|
||||
// 三个方法都直接转调 internal/sdk 的同名方法:那一层已经是三参数 + blocks
|
||||
// 的公开形态,不像 InjectInputSync 需要收窄。
|
||||
|
||||
func (c procCore) InjectInputMedia(source, channel, text string, blocks []pubsdk.ContentBlock) {
|
||||
c.sdk.InjectInputMedia(source, channel, text, blocks)
|
||||
}
|
||||
|
||||
func (c procCore) InjectInputMediaSync(source, channel, text string, blocks []pubsdk.ContentBlock) string {
|
||||
return c.sdk.InjectInputMediaSync(source, channel, text, blocks)
|
||||
}
|
||||
|
||||
func (c procCore) InjectInterruptMedia(source, channel, text string, blocks []pubsdk.ContentBlock) {
|
||||
c.sdk.InjectInterruptMedia(source, channel, text, blocks)
|
||||
}
|
||||
|
||||
// ---- 生命周期 ----
|
||||
|
||||
func (c procCore) SetAutoRestart(enabled bool) { c.sdk.SetAutoRestart(enabled) }
|
||||
|
||||
@ -22,6 +22,7 @@ import (
|
||||
luaVM "gitcode.com/JianFeeeee/HomeAgent/internal/lua"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory"
|
||||
doc "gitcode.com/JianFeeeee/HomeAgent/internal/memory/document"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/media"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/text"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/plugin/proc"
|
||||
sdk "gitcode.com/JianFeeeee/HomeAgent/internal/sdk"
|
||||
@ -88,13 +89,17 @@ type Registry struct {
|
||||
memDB *memory.GraphDB
|
||||
textMem *text.Memory
|
||||
docStore *doc.Store
|
||||
ks *knowledge.Store
|
||||
mgr *agentAPI.ProviderManager
|
||||
cfgReg *internalConfig.ConfigRegistry
|
||||
plgDir string
|
||||
dataDir string // 守护进程数据目录(注入给插件 SettingsAPI.DataDir)
|
||||
lua *luaVM.VM
|
||||
baseKey string
|
||||
// mediaStore 让插件写入的记忆也能带媒体。
|
||||
// 为 nil 时(配置关闭或初始化失败)插件侧记忆包装退化为纯文本行为,
|
||||
// 与本特性上线前完全一致。
|
||||
mediaStore *media.Store
|
||||
ks *knowledge.Store
|
||||
mgr *agentAPI.ProviderManager
|
||||
cfgReg *internalConfig.ConfigRegistry
|
||||
plgDir string
|
||||
dataDir string // 守护进程数据目录(注入给插件 SettingsAPI.DataDir)
|
||||
lua *luaVM.VM
|
||||
baseKey string
|
||||
|
||||
regTool sdk.ToolRegistrar
|
||||
regStage sdk.StageRegistrar
|
||||
@ -184,6 +189,7 @@ func (r *Registry) SetEventBus(evBus *events.Bus) { r.
|
||||
func (r *Registry) SetMemory(memDB *memory.GraphDB) { r.memDB = memDB }
|
||||
func (r *Registry) SetTextMemory(tm *text.Memory) { r.textMem = tm }
|
||||
func (r *Registry) SetDocStore(ds *doc.Store) { r.docStore = ds }
|
||||
func (r *Registry) SetMediaStore(ms *media.Store) { r.mediaStore = ms }
|
||||
func (r *Registry) SetKnowledge(ks *knowledge.Store) { r.ks = ks }
|
||||
func (r *Registry) SetProviderManager(mgr *agentAPI.ProviderManager) { r.mgr = mgr }
|
||||
func (r *Registry) SetConfigRegistry(cfgReg *internalConfig.ConfigRegistry) { r.cfgReg = cfgReg }
|
||||
@ -311,11 +317,13 @@ func (r *Registry) buildSDK(name string) *sdk.PluginSDK {
|
||||
}
|
||||
|
||||
return sdk.New(name, sdk.SDKConfig{
|
||||
IOManager: r.iom,
|
||||
EventBus: r.evBus,
|
||||
Memory: sdk.NewGraphMemory(r.memDB),
|
||||
TextMemory: sdk.NewTextMemory(r.textMem),
|
||||
DocMemory: sdk.NewDocMemory(r.docStore),
|
||||
IOManager: r.iom,
|
||||
EventBus: r.evBus,
|
||||
// 带 media 的包装:插件提交的三元组/文档/文本事件里的媒体会落进 CAS
|
||||
// 并挂上引用。传入插件名仅用于日志溯源(哪个插件写的媒体)。
|
||||
Memory: sdk.NewGraphMemoryWithMedia(name, r.memDB, r.mediaStore),
|
||||
TextMemory: sdk.NewTextMemoryWithMedia(name, r.textMem, r.mediaStore),
|
||||
DocMemory: sdk.NewDocMemoryWithMedia(name, r.docStore, r.mediaStore),
|
||||
Knowledge: sdk.NewKnowledge(r.ks),
|
||||
LLM: sdk.NewLLM(r.mgr, r.cfgReg, r.lua, r.baseKey),
|
||||
Settings: sett,
|
||||
|
||||
@ -32,7 +32,7 @@ func (tc *toolCapture) RegisterTool(name string, def sdk.ToolDef, handler sdk.To
|
||||
return nil
|
||||
}
|
||||
func (tc *toolCapture) RegisterStage(stage sdk.Stage, handler sdk.StageHandler) {}
|
||||
func (tc *toolCapture) RegisterAPI(name string) error { return nil }
|
||||
func (tc *toolCapture) RegisterAPI(name string) error { return nil }
|
||||
|
||||
func setupPlugin() (*Plugin, *toolCapture, error) {
|
||||
p := New("agentcli")
|
||||
@ -422,6 +422,24 @@ func (c *injectCapture) SetToolBlocks(blocks []sdkpub.ContentBlock) {
|
||||
}
|
||||
func (c *injectCapture) InjectInputSync(source, channel, text string) string { return "" }
|
||||
|
||||
// 三个带媒体的注入方法同样记录文本:本测试只关心「注入了什么话」,
|
||||
// 媒体块的转发在 core 的 injectedBlocks 测试里覆盖。
|
||||
func (c *injectCapture) InjectInputMedia(source, channel, text string, blocks []sdkpub.ContentBlock) {
|
||||
c.mu.Lock()
|
||||
c.texts = append(c.texts, text)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
func (c *injectCapture) InjectInputMediaSync(source, channel, text string, blocks []sdkpub.ContentBlock) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (c *injectCapture) InjectInterruptMedia(source, channel, text string, blocks []sdkpub.ContentBlock) {
|
||||
c.mu.Lock()
|
||||
c.texts = append(c.texts, text)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
func (c *injectCapture) snapshot() []string {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
@ -3148,6 +3148,19 @@
|
||||
'" ' +
|
||||
'alt="image" loading="lazy" style="max-width:320px;max-height:240px;border-radius:10px;display:block;cursor:zoom-in" ' +
|
||||
'onerror="this.parentElement.innerHTML=\'<span class=\\"att-err\\">图片加载失败</span>\'"/></a>';
|
||||
} else if (att.type === "audio") {
|
||||
// 音频用原生播放器:与图片同理,附件能在聊天里直接消费才算可见。
|
||||
// preload="metadata" 只拉时长不拉全部字节,避免历史消息满屏时并发下载。
|
||||
attHtml =
|
||||
'<audio class="chat-attachment-audio" controls preload="metadata" src="' +
|
||||
escHtml(att.url) +
|
||||
'" style="max-width:320px;display:block"></audio>' +
|
||||
'<a href="' +
|
||||
escHtml(att.url) +
|
||||
'" download style="font-size:12px;color:var(--text-muted,#888);text-decoration:none">' +
|
||||
escHtml(att.name || "audio") +
|
||||
(att.size ? " (" + formatBytes(att.size) + ")" : "") +
|
||||
"</a>";
|
||||
} else {
|
||||
var sizeStr = att.size ? formatBytes(att.size) : "";
|
||||
attHtml =
|
||||
@ -3929,7 +3942,13 @@
|
||||
role: "user",
|
||||
content: message,
|
||||
attachment: {
|
||||
type: /^image\//.test(fileObj.type) ? "image" : "file",
|
||||
// 与服务端的 attType 判定保持一致(image/audio/file),
|
||||
// 否则乐观渲染的卡片会在 SSE 回流后变成另一种样式。
|
||||
type: /^image\//.test(fileObj.type)
|
||||
? "image"
|
||||
: /^audio\//.test(fileObj.type)
|
||||
? "audio"
|
||||
: "file",
|
||||
url: URL.createObjectURL(fileObj),
|
||||
name: fileObj.name,
|
||||
size: fileObj.size,
|
||||
@ -4578,7 +4597,11 @@
|
||||
if (p.kind === "channel_output") {
|
||||
// 附件输出(output_type=image/file):渲染为图片预览/下载卡片
|
||||
var att = null;
|
||||
if (p.output_type === "image" || p.output_type === "file") {
|
||||
if (
|
||||
p.output_type === "image" ||
|
||||
p.output_type === "audio" ||
|
||||
p.output_type === "file"
|
||||
) {
|
||||
att = {
|
||||
type: p.output_type,
|
||||
url: p.url || p.content,
|
||||
|
||||
@ -5,6 +5,7 @@ import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"embed"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
@ -1446,6 +1447,13 @@ func parseIntDefault(s string, def int) int {
|
||||
return n
|
||||
}
|
||||
|
||||
// maxInlineMediaBytes 是上传媒体内联进 LLM 请求的字节上限。
|
||||
//
|
||||
// base64 会胀大 4/3,8MB 原图变成 ~11MB 文本;再加上网关的请求体上限与
|
||||
// 模型的图像 token 预算,超过这个量级多半会被上游 413 拒掉。
|
||||
// 超限时退回按路径处理(模型可用 describe_image 主动看)而不是报错。
|
||||
const maxInlineMediaBytes = 8 << 20
|
||||
|
||||
// handleChatFile 处理用户经 webui 上传文件并附带消息注入 agent。
|
||||
// 设计对齐 qq 插件收文件模式:文件落盘到固定目录(<data>/uploads),
|
||||
// 注入文本带「文件名 + 保存路径」,agent 用 files_read 等工具按路径消费。
|
||||
@ -1511,8 +1519,45 @@ func (h *Handler) handleChatFile(w http.ResponseWriter, r *http.Request) {
|
||||
dlURL := "/uploads/" + filepath.Base(savePath)
|
||||
attType := "file"
|
||||
ct := hdr.Header.Get("Content-Type")
|
||||
if strings.HasPrefix(ct, "image/") {
|
||||
if ct == "" {
|
||||
// 部分客户端(curl -F、某些移动端)不带 Content-Type,退回按扩展名判定。
|
||||
// 判错的后果不只是卡片样式:图片被当普通文件就走不进视觉链路,模型看不到图。
|
||||
ct = contentTypeByExt(strings.ToLower(filepath.Ext(savePath)))
|
||||
}
|
||||
switch {
|
||||
case strings.HasPrefix(ct, "image/"):
|
||||
attType = "image"
|
||||
case strings.HasPrefix(ct, "audio/"):
|
||||
attType = "audio"
|
||||
}
|
||||
|
||||
// 图片/音频直接进多模态链路:读回字节拼 data URL,随本轮 message 发给模型。
|
||||
//
|
||||
// 此前只注入一句「文件已保存到 <路径>」,指望模型自己调 files_read——
|
||||
// 但 files_read 返回的是文本,图片的字节对模型永远不可见,除非它想到再调
|
||||
// describe_image。走 InjectInputMedia 后与用户在 qq 发图走同一条统一输入主干:
|
||||
// 自动落进 CAS、挂上媒体记忆引用,且模型「本轮」就看得到图。
|
||||
var mediaBlocks []sdk.ContentBlock
|
||||
if attType == "image" || attType == "audio" {
|
||||
if sz > maxInlineMediaBytes {
|
||||
log.Printf("[webui] %s %s 有 %s,超过 %s 内联上限,退回按路径处理",
|
||||
attType, base, formatBytesGo(sz), formatBytesGo(maxInlineMediaBytes))
|
||||
} else if raw, err := os.ReadFile(savePath); err != nil {
|
||||
log.Printf("[webui] 读回上传的%s失败,退回按路径处理: %v", attType, err)
|
||||
} else {
|
||||
dataURL := "data:" + ct + ";base64," + base64.StdEncoding.EncodeToString(raw)
|
||||
if attType == "image" {
|
||||
mediaBlocks = []sdk.ContentBlock{{
|
||||
Type: "image_url",
|
||||
ImageURL: &sdk.ImageURL{URL: dataURL, Detail: "auto"},
|
||||
}}
|
||||
} else {
|
||||
mediaBlocks = []sdk.ContentBlock{{
|
||||
Type: "audio_url",
|
||||
AudioURL: &sdk.AudioURL{URL: dataURL},
|
||||
}}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 注入 agent:文件元信息走 interrupt 通道(内核以 system 角色注入 LLM,
|
||||
@ -1524,8 +1569,17 @@ func (h *Handler) handleChatFile(w http.ResponseWriter, r *http.Request) {
|
||||
if deviceID != "" {
|
||||
source = "webui/" + deviceID
|
||||
}
|
||||
typeLabel := map[string]string{"image": "图片", "audio": "音频", "file": "文件"}[attType]
|
||||
if typeLabel == "" {
|
||||
typeLabel = "文件"
|
||||
}
|
||||
fileNote := fmt.Sprintf("[用户通过 webui 发送了%s: %s (%s)]\n文件已保存到: %s\n可用 files_read 等工具读取此路径处理。",
|
||||
map[string]string{"image": "图片", "file": "文件"}[attType], base, humanSize, savePath)
|
||||
typeLabel, base, humanSize, savePath)
|
||||
// 媒体已随本轮发给模型时不再叫它去读文件:那只会读到一堆二进制字节。
|
||||
if len(mediaBlocks) > 0 {
|
||||
fileNote = fmt.Sprintf("[用户通过 webui 发送了%s: %s (%s)]\n原文件保存在: %s",
|
||||
typeLabel, base, humanSize, savePath)
|
||||
}
|
||||
if message != "" {
|
||||
text := message
|
||||
go func() {
|
||||
@ -1538,14 +1592,15 @@ func (h *Handler) handleChatFile(w http.ResponseWriter, r *http.Request) {
|
||||
if clientMsgID != "" {
|
||||
payload2["client_msg_id"] = clientMsgID + "-note"
|
||||
}
|
||||
h.sdk.InjectInput(source, "webui", "text", func() map[string]interface{} {
|
||||
p := payload2
|
||||
p["upload_url"] = dlURL
|
||||
p["upload_type"] = attType
|
||||
p["upload_size"] = sz
|
||||
p["upload_name"] = base
|
||||
return p
|
||||
}())
|
||||
payload2["upload_url"] = dlURL
|
||||
payload2["upload_type"] = attType
|
||||
payload2["upload_size"] = sz
|
||||
payload2["upload_name"] = base
|
||||
// 媒体跟附言同一条注入:拆开会让模型先看到「帮我看看这张图」而图在下一轮才到。
|
||||
if len(mediaBlocks) > 0 {
|
||||
payload2["media_blocks"] = mediaBlocks
|
||||
}
|
||||
h.sdk.InjectInput(source, "webui", "text", payload2)
|
||||
}()
|
||||
time.Sleep(100 * time.Millisecond) // 保证附言先入队
|
||||
h.sdk.InjectInterrupt(source, "webui", "text", map[string]interface{}{"content": fileNote, "no_memory": true})
|
||||
@ -1563,6 +1618,9 @@ func (h *Handler) handleChatFile(w http.ResponseWriter, r *http.Request) {
|
||||
"upload_size": sz,
|
||||
"upload_name": base,
|
||||
}
|
||||
if len(mediaBlocks) > 0 {
|
||||
payload["media_blocks"] = mediaBlocks
|
||||
}
|
||||
if deviceID != "" {
|
||||
payload["device_id"] = deviceID
|
||||
payload["device_name"] = deviceName
|
||||
|
||||
@ -24,5 +24,9 @@ type TextMemoryAPI interface {
|
||||
|
||||
type TextEvent = pubsdk.TextEvent
|
||||
|
||||
// MediaAttachment 是记忆附件(媒体)在插件边界上的表示。
|
||||
// 与公共 SDK 同一类型,内置插件与外部插件用同一套字段。
|
||||
type MediaAttachment = pubsdk.MediaAttachment
|
||||
|
||||
type DocMemoryAPI = pubsdk.DocMemoryAPI
|
||||
type Doc = pubsdk.Doc
|
||||
|
||||
@ -1,110 +1,587 @@
|
||||
package sdk
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory"
|
||||
doc "gitcode.com/JianFeeeee/HomeAgent/internal/memory/document"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/media"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/text"
|
||||
)
|
||||
|
||||
type graphMemory struct{ db *memory.GraphDB }
|
||||
// 插件侧记忆接口的实现(SDK 桥接层)。
|
||||
//
|
||||
// 这一层原先的缺陷是**静默裁字段**:插件把 Triple / Doc 交进来,包装层只挑
|
||||
// 自己认识的几个字段转成内部结构,其余丢弃且不报错。两侧都中招:
|
||||
// - 图记忆:丢 Confidence/SubjectType/ObjectType/SentenceText,又走 Commit
|
||||
// 而非 CommitWithMedia,于是 sentences 表没有落点,媒体引用无从挂起;
|
||||
// - 知识库:Query 只回 ID/Title/Content,Insert 只写这三个,读写两个方向
|
||||
// 都把媒体元数据裁掉;Remove 不解引用,媒体永久算「被引用」,GC 收不掉。
|
||||
//
|
||||
// 现在的规则:内部结构有的字段一律透传;媒体一律走标记格式并挂到对应 owner。
|
||||
// 媒体存储为 nil 时整条链路静默降级为纯文本行为(媒体是记忆增强,不是必需品)。
|
||||
|
||||
// ---------- 媒体标记(本层内部) ----------
|
||||
//
|
||||
// 标记是媒体在**纯文本记忆**里的表示形式:
|
||||
//
|
||||
// [image/png a1b2c3d4e5f6] 一张紫蓝红三色带图
|
||||
// └ label └ 短 digest └ 描述
|
||||
//
|
||||
// 之所以必须借文本承载:Doc.Content、sentences.text、文本记忆的 Input 全是
|
||||
// 字符串,没有字段能挂结构化数据。描述文本是持久的语义记忆(检索靠它),
|
||||
// digest 是回到字节的钥匙(反查靠它)。
|
||||
//
|
||||
// 格式与内核侧 graphmedia.go 的 mediaSummaryForEvent 一致——两边必须能互读
|
||||
// 对方写下的标记,否则插件写入的媒体在内核归档时挂不上引用,且不报错。
|
||||
|
||||
const sdkShortDigestLen = 12
|
||||
|
||||
// sdkMarkerPattern 拆解一条标记,捕获组依次为 label、短 digest、该行剩余描述。
|
||||
// digest 放宽到 8-64 位以容忍完整 digest 手写的情况;描述取到行尾而非贪婪到底,
|
||||
// 因为一条记忆可能挂多份媒体、各占一行。
|
||||
var sdkMarkerPattern = regexp.MustCompile(`\[([^\[\]\s]+)\s+([0-9a-f]{8,64})\]([^\n]*)`)
|
||||
|
||||
func sdkShortDigest(d string) string {
|
||||
if len(d) > sdkShortDigestLen {
|
||||
return d[:sdkShortDigestLen]
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// sdkMarkerFor 为一份已入库的媒体生成标记行。查不到就返回空串——
|
||||
// 媒体可能已被 GC 清掉,此时不该凭空造出一条指向虚无的标记。
|
||||
func sdkMarkerFor(ms *media.Store, digest string) string {
|
||||
it, err := ms.Stat(digest)
|
||||
if err != nil || it == nil {
|
||||
return ""
|
||||
}
|
||||
label := string(it.Kind)
|
||||
if it.MIME != "" {
|
||||
label = it.MIME
|
||||
}
|
||||
if it.Description == "" {
|
||||
// 「已入库但还没描述」与「压根没有媒体」必须可区分:
|
||||
// 描述由后台循环异步补齐,占位符保证补齐前这份媒体也不会从文本里消失。
|
||||
return fmt.Sprintf("[%s %s] (未描述)", label, sdkShortDigest(digest))
|
||||
}
|
||||
return fmt.Sprintf("[%s %s] %s", label, sdkShortDigest(digest), it.Description)
|
||||
}
|
||||
|
||||
// sdkDigestsIn 返回文本里出现过的短 digest 集合,用于避免重复追加标记。
|
||||
func sdkDigestsIn(s string) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
for _, m := range sdkMarkerPattern.FindAllStringSubmatch(s, -1) {
|
||||
out[m[2]] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// sdkBindText 把文本里引用的媒体挂到 owner 上,返回新挂上的条数。
|
||||
//
|
||||
// 短 digest 补全失败(内容已 GC、或前缀有歧义)就跳过那一条:挂一条对不上的
|
||||
// 引用比不挂更糟——owner_kind/owner_id/digest 三者进了主键,digest 错了则
|
||||
// DropOwner 永远匹配不到它,那是一条永久泄漏的引用。
|
||||
//
|
||||
// done 记录本次已处理过的 digest。AddRef 幂等,重复挂不会多出一条引用,
|
||||
// 但会让计数虚高——文档路径先按附件挂一遍、再扫正文标记挂一遍,
|
||||
// 同一份媒体会被数两次,日志里「绑定 2 个」而实际只有 1 条引用。
|
||||
func sdkBindText(ms *media.Store, text, ownerKind, ownerID string, done map[string]bool) int {
|
||||
if ms == nil || text == "" || ownerID == "" {
|
||||
return 0
|
||||
}
|
||||
bound := 0
|
||||
for _, m := range sdkMarkerPattern.FindAllStringSubmatch(text, -1) {
|
||||
full, err := ms.ResolvePrefix(m[2])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if done != nil && done[full] {
|
||||
continue
|
||||
}
|
||||
if err := ms.AddRef(full, ownerKind, ownerID); err != nil {
|
||||
continue
|
||||
}
|
||||
if done != nil {
|
||||
done[full] = true
|
||||
}
|
||||
bound++
|
||||
}
|
||||
return bound
|
||||
}
|
||||
|
||||
// sdkPutAttachment 把一份附件解析成完整 digest。
|
||||
//
|
||||
// 两种入口:带 Data 的是新内容(落进 CAS,相同字节自动去重);
|
||||
// 只给 Digest 的是引用已有内容(补全前缀即可)。两者都不给则无效。
|
||||
func sdkPutAttachment(ms *media.Store, a MediaAttachment, tool string) (string, error) {
|
||||
if len(a.Data) > 0 {
|
||||
mime := a.MIME
|
||||
if mime == "" {
|
||||
mime = "application/octet-stream"
|
||||
}
|
||||
return ms.Put(a.Data, media.Item{
|
||||
MIME: mime,
|
||||
Tool: tool,
|
||||
OriginPath: a.Name,
|
||||
Description: a.Description,
|
||||
})
|
||||
}
|
||||
if a.Digest == "" {
|
||||
return "", fmt.Errorf("附件既无 data 也无 digest")
|
||||
}
|
||||
full, err := ms.ResolvePrefix(a.Digest)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("digest %s: %w", a.Digest, err)
|
||||
}
|
||||
return full, nil
|
||||
}
|
||||
|
||||
// sdkAttachmentsFromText 从文本标记反解出附件元数据(不含字节),
|
||||
// 让插件不必自己写正则去认标记。
|
||||
func sdkAttachmentsFromText(ms *media.Store, s string) []MediaAttachment {
|
||||
if ms == nil || s == "" {
|
||||
return nil
|
||||
}
|
||||
var out []MediaAttachment
|
||||
seen := map[string]bool{}
|
||||
for _, m := range sdkMarkerPattern.FindAllStringSubmatch(s, -1) {
|
||||
full, err := ms.ResolvePrefix(m[2])
|
||||
if err != nil || seen[full] {
|
||||
continue
|
||||
}
|
||||
seen[full] = true
|
||||
att := MediaAttachment{Digest: full, MIME: m[1], Description: strings.TrimSpace(m[3])}
|
||||
if it, err := ms.Stat(full); err == nil && it != nil {
|
||||
att.MIME = it.MIME
|
||||
if it.Description != "" {
|
||||
att.Description = it.Description
|
||||
}
|
||||
}
|
||||
out = append(out, att)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---------- 图记忆 ----------
|
||||
|
||||
type graphMemory struct {
|
||||
db *memory.GraphDB
|
||||
ms *media.Store
|
||||
plugin string
|
||||
}
|
||||
|
||||
func NewGraphMemory(db *memory.GraphDB) MemoryAPI { return &graphMemory{db: db} }
|
||||
|
||||
// NewGraphMemoryWithMedia 创建带媒体能力的图记忆包装。plugin 仅用于日志溯源。
|
||||
func NewGraphMemoryWithMedia(plugin string, db *memory.GraphDB, ms *media.Store) MemoryAPI {
|
||||
return &graphMemory{db: db, ms: ms, plugin: plugin}
|
||||
}
|
||||
|
||||
func (m *graphMemory) Recall(query []string, depth int) ([]Entity, []Relation, error) {
|
||||
if m.db == nil { return nil, nil, nil }
|
||||
if m.db == nil {
|
||||
return nil, nil, nil
|
||||
}
|
||||
result, err := m.db.Recall(query, nil, depth, "")
|
||||
if err != nil { return nil, nil, err }
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
entities := make([]Entity, len(result.Entities))
|
||||
for i, e := range result.Entities {
|
||||
entities[i] = Entity{Name: e.Name, Type: e.Type, MentionCount: e.MentionCount}
|
||||
}
|
||||
// Confidence 此前被丢弃:插件拿不到置信度就无法判断一条关系可不可信,
|
||||
// 只能把所有召回结果等同对待。
|
||||
relations := make([]Relation, len(result.Relations))
|
||||
for i, r := range result.Relations {
|
||||
relations[i] = Relation{SourceName: r.SourceName, TargetName: r.TargetName, RelationType: r.RelationType}
|
||||
relations[i] = Relation{
|
||||
SourceName: r.SourceName,
|
||||
TargetName: r.TargetName,
|
||||
RelationType: r.RelationType,
|
||||
Confidence: r.Confidence,
|
||||
}
|
||||
}
|
||||
return entities, relations, nil
|
||||
}
|
||||
|
||||
// Commit 把插件的三元组写入图库,并把三元组引用的媒体挂到句子上。
|
||||
//
|
||||
// 媒体的绑定链是 SentenceText → sentences 表 → sentence_id → media_refs。
|
||||
// 旧实现丢掉 SentenceText 又走 Commit(不回 sentenceIDs),这条链一步都走不通:
|
||||
// 插件即便按格式写好标记,媒体也永远挂不上。
|
||||
func (m *graphMemory) Commit(triples []Triple) error {
|
||||
if m.db == nil { return nil }
|
||||
ts := make([]memory.Triple, len(triples))
|
||||
for i, t := range triples {
|
||||
ts[i] = memory.Triple{Subject: t.Subject, Relation: t.Relation, Object: t.Object}
|
||||
if m.db == nil {
|
||||
return nil
|
||||
}
|
||||
ts := make([]memory.Triple, 0, len(triples))
|
||||
for _, t := range triples {
|
||||
mt := memory.Triple{
|
||||
Subject: t.Subject,
|
||||
Relation: t.Relation,
|
||||
Object: t.Object,
|
||||
Confidence: t.Confidence,
|
||||
SubjectType: t.SubjectType,
|
||||
ObjectType: t.ObjectType,
|
||||
SentenceText: t.SentenceText,
|
||||
}
|
||||
if len(t.MediaDigests) > 0 {
|
||||
mt.SentenceText = m.sentenceWithMedia(mt.SentenceText, t.MediaDigests)
|
||||
}
|
||||
ts = append(ts, mt)
|
||||
}
|
||||
|
||||
sentenceIDs, _, _, err := m.db.CommitWithMedia(ts, "plugin", 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
m.bindSentences(sentenceIDs)
|
||||
return nil
|
||||
}
|
||||
|
||||
// sentenceWithMedia 保证句子文本里带有这些 digest 的媒体标记。
|
||||
//
|
||||
// 让插件填 MediaDigests 就够,不必知道标记格式——否则格式写错的后果是
|
||||
// 引用静默挂不上。已出现过的 digest 不重复追加:插件可能既手写了标记又填了
|
||||
// MediaDigests,重复标记会让同一份媒体产生两条一样的句子引用。
|
||||
func (m *graphMemory) sentenceWithMedia(sentence string, digests []string) string {
|
||||
present := sdkDigestsIn(sentence)
|
||||
var add []string
|
||||
for _, d := range digests {
|
||||
if d == "" || present[sdkShortDigest(d)] {
|
||||
continue
|
||||
}
|
||||
if m.ms == nil {
|
||||
// 没有媒体存储时也把 digest 留在文本里:拿不到描述,
|
||||
// 但将来存储可用时这条记忆仍能反查回字节。
|
||||
add = append(add, fmt.Sprintf("[media %s] (未描述)", sdkShortDigest(d)))
|
||||
present[sdkShortDigest(d)] = true
|
||||
continue
|
||||
}
|
||||
full, err := m.ms.ResolvePrefix(d)
|
||||
if err != nil {
|
||||
log.Printf("[sdk media] 插件 %s 提交的 digest %s 无法解析: %v", m.plugin, d, err)
|
||||
continue
|
||||
}
|
||||
if line := sdkMarkerFor(m.ms, full); line != "" {
|
||||
add = append(add, line)
|
||||
present[sdkShortDigest(full)] = true
|
||||
}
|
||||
}
|
||||
if len(add) == 0 {
|
||||
return sentence
|
||||
}
|
||||
if sentence == "" {
|
||||
return strings.Join(add, "\n")
|
||||
}
|
||||
return sentence + "\n" + strings.Join(add, "\n")
|
||||
}
|
||||
|
||||
// bindSentences 把每条句子里引用的媒体挂到该句子的 graph_sentence owner 上。
|
||||
func (m *graphMemory) bindSentences(sentenceIDs map[string]int64) {
|
||||
if m.ms == nil || len(sentenceIDs) == 0 {
|
||||
return
|
||||
}
|
||||
bound := 0
|
||||
for text, sid := range sentenceIDs {
|
||||
if sid == 0 {
|
||||
continue
|
||||
}
|
||||
// 每条句子一个独立的 done 集:同一份媒体挂在不同句子上是两条
|
||||
// 合法引用(owner_id 不同),不该被跨句子去重。
|
||||
bound += sdkBindText(m.ms, text, media.OwnerGraphSentence,
|
||||
strconv.FormatInt(sid, 10), map[string]bool{})
|
||||
}
|
||||
if bound > 0 {
|
||||
log.Printf("[sdk media] 插件 %s 的三元组绑定 %d 个媒体引用", m.plugin, bound)
|
||||
}
|
||||
_, _, err := m.db.Commit(ts, "plugin", 0)
|
||||
return err
|
||||
}
|
||||
|
||||
func (m *graphMemory) Introspect() (map[string]interface{}, error) {
|
||||
if m.db == nil { return map[string]interface{}{}, nil }
|
||||
if m.db == nil {
|
||||
return map[string]interface{}{}, nil
|
||||
}
|
||||
return m.db.Introspect()
|
||||
}
|
||||
|
||||
func (m *graphMemory) MergeEntities(source, target string) (int, error) {
|
||||
if m.db == nil { return 0, nil }
|
||||
if m.db == nil {
|
||||
return 0, nil
|
||||
}
|
||||
return m.db.MergeEntities(source, target)
|
||||
}
|
||||
|
||||
func (m *graphMemory) Purge(criteria map[string]string, mode string) (int, error) {
|
||||
if m.db == nil { return 0, nil }
|
||||
if m.db == nil {
|
||||
return 0, nil
|
||||
}
|
||||
return m.db.Purge(criteria, mode)
|
||||
}
|
||||
|
||||
func (m *graphMemory) GraphData() (map[string]interface{}, error) {
|
||||
if m.db == nil { return map[string]interface{}{}, nil }
|
||||
if m.db == nil {
|
||||
return map[string]interface{}{}, nil
|
||||
}
|
||||
return m.db.GraphData()
|
||||
}
|
||||
|
||||
type textMemoryImpl struct{ tm *text.Memory }
|
||||
// ---------- 文本记忆 ----------
|
||||
|
||||
type textMemoryImpl struct {
|
||||
tm *text.Memory
|
||||
ms *media.Store
|
||||
plugin string
|
||||
}
|
||||
|
||||
func NewTextMemory(tm *text.Memory) TextMemoryAPI { return &textMemoryImpl{tm: tm} }
|
||||
|
||||
// NewTextMemoryWithMedia 创建带媒体能力的文本记忆包装。
|
||||
func NewTextMemoryWithMedia(plugin string, tm *text.Memory, ms *media.Store) TextMemoryAPI {
|
||||
return &textMemoryImpl{tm: tm, ms: ms, plugin: plugin}
|
||||
}
|
||||
|
||||
// Append 追加一条文本事件;带附件时把媒体标记并进正文。
|
||||
//
|
||||
// 文本记忆是追加写 JSONL,没有稳定 owner_id 可挂 media_refs,所以媒体在这一层
|
||||
// 只能以标记形式存在。这不是妥协——描述文本才是持久的语义记忆,blob 只是缓存。
|
||||
func (m *textMemoryImpl) Append(evt TextEvent) error {
|
||||
if m.tm == nil { return nil }
|
||||
if m.tm == nil {
|
||||
return nil
|
||||
}
|
||||
content := evt.Content
|
||||
if len(evt.Attachments) > 0 && m.ms != nil {
|
||||
var lines []string
|
||||
for _, a := range evt.Attachments {
|
||||
d, err := sdkPutAttachment(m.ms, a, "plugin_text:"+m.plugin)
|
||||
if err != nil {
|
||||
log.Printf("[sdk media] 插件 %s 文本附件入库失败: %v", m.plugin, err)
|
||||
continue
|
||||
}
|
||||
if line := sdkMarkerFor(m.ms, d); line != "" {
|
||||
lines = append(lines, line)
|
||||
}
|
||||
}
|
||||
if len(lines) > 0 {
|
||||
if content == "" {
|
||||
content = strings.Join(lines, "\n")
|
||||
} else {
|
||||
content += "\n" + strings.Join(lines, "\n")
|
||||
}
|
||||
}
|
||||
}
|
||||
return m.tm.Append(text.Event{
|
||||
Timestamp: evt.Timestamp, Source: evt.Role, Input: evt.Content, AgentID: evt.Channel,
|
||||
Timestamp: evt.Timestamp, Source: evt.Role, Input: content, AgentID: evt.Channel,
|
||||
})
|
||||
}
|
||||
|
||||
func (m *textMemoryImpl) RecentEvents(n int) ([]TextEvent, error) {
|
||||
if m.tm == nil { return nil, nil }
|
||||
if m.tm == nil {
|
||||
return nil, nil
|
||||
}
|
||||
got, err := m.tm.RecentEvents(n)
|
||||
if err != nil { return nil, err }
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]TextEvent, len(got))
|
||||
for i, e := range got {
|
||||
out[i] = TextEvent{Role: e.Source, Content: e.Input, Timestamp: e.Timestamp, Channel: e.AgentID}
|
||||
out[i] = TextEvent{
|
||||
Role: e.Source, Content: e.Input, Timestamp: e.Timestamp, Channel: e.AgentID,
|
||||
Attachments: sdkAttachmentsFromText(m.ms, e.Input),
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (m *textMemoryImpl) Stats() map[string]interface{} {
|
||||
if m.tm == nil { return map[string]interface{}{} }
|
||||
if m.tm == nil {
|
||||
return map[string]interface{}{}
|
||||
}
|
||||
return m.tm.Stats()
|
||||
}
|
||||
|
||||
type docMemoryImpl struct{ ds *doc.Store }
|
||||
// ---------- 文档记忆(知识库) ----------
|
||||
|
||||
type docMemoryImpl struct {
|
||||
ds *doc.Store
|
||||
ms *media.Store
|
||||
plugin string
|
||||
}
|
||||
|
||||
func NewDocMemory(ds *doc.Store) DocMemoryAPI { return &docMemoryImpl{ds: ds} }
|
||||
|
||||
// NewDocMemoryWithMedia 创建带媒体能力的文档记忆包装。
|
||||
func NewDocMemoryWithMedia(plugin string, ds *doc.Store, ms *media.Store) DocMemoryAPI {
|
||||
return &docMemoryImpl{ds: ds, ms: ms, plugin: plugin}
|
||||
}
|
||||
|
||||
// Query 检索文档,并补齐媒体元数据。
|
||||
//
|
||||
// 旧实现只回 ID/Title/Content,插件即便拿到一篇带媒体的文档也看不出这里有
|
||||
// 几份媒体、分别是什么。现在同时给出完整 digest 列表与 mime+描述,
|
||||
// 但**不回字节**:一次检索可能命中几十份媒体,全塞回去会把跨进程消息撑爆,
|
||||
// 需要字节时按 digest 单取。
|
||||
func (m *docMemoryImpl) Query(text string, topK int) []*Doc {
|
||||
if m.ds == nil { return nil }
|
||||
if m.ds == nil {
|
||||
return nil
|
||||
}
|
||||
got := m.ds.Query(text, topK)
|
||||
out := make([]*Doc, len(got))
|
||||
for i, d := range got {
|
||||
out[i] = &Doc{ID: d.ID, Title: d.Summary, Content: d.Content}
|
||||
m.fillMedia(out[i])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (m *docMemoryImpl) Insert(d *Doc) error {
|
||||
if m.ds == nil { return nil }
|
||||
return m.ds.Insert(&doc.Doc{ID: d.ID, Summary: d.Title, Content: d.Content})
|
||||
// fillMedia 填充文档的媒体字段。
|
||||
//
|
||||
// 优先用 media_refs(权威:谁挂上去的就是谁),为空时退回解析正文标记——
|
||||
// 历史文档与经旧版插件写入的文档只有标记、没有引用。
|
||||
func (m *docMemoryImpl) fillMedia(out *Doc) {
|
||||
if m.ms == nil {
|
||||
return
|
||||
}
|
||||
digests, err := m.ms.Refs(media.OwnerDocument, out.ID)
|
||||
if err != nil {
|
||||
log.Printf("[sdk media] 读取文档 %s 的媒体引用失败: %v", out.ID, err)
|
||||
}
|
||||
if len(digests) == 0 {
|
||||
out.Attachments = sdkAttachmentsFromText(m.ms, out.Content)
|
||||
for _, a := range out.Attachments {
|
||||
out.MediaDigests = append(out.MediaDigests, a.Digest)
|
||||
}
|
||||
return
|
||||
}
|
||||
out.MediaDigests = digests
|
||||
for _, d := range digests {
|
||||
it, err := m.ms.Stat(d)
|
||||
if err != nil || it == nil {
|
||||
continue
|
||||
}
|
||||
out.Attachments = append(out.Attachments, MediaAttachment{
|
||||
Digest: it.Digest, MIME: it.MIME, Description: it.Description,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (m *docMemoryImpl) Remove(id string) { if m.ds != nil { m.ds.Remove(id) } }
|
||||
// Insert 写入文档。正文里已有的媒体标记会被挂成文档级引用,
|
||||
// 避免插件写进来的媒体在下一次 GC 时被当作无主内容清掉。
|
||||
func (m *docMemoryImpl) Insert(d *Doc) error { return m.InsertWithMedia(d, nil) }
|
||||
|
||||
// InsertWithMedia 写入文档并关联媒体。
|
||||
//
|
||||
// 标记由内核补进 Content——插件不必知道标记格式,也就不会因为格式写错导致
|
||||
// 引用挂不上。补标记必须在 ds.Insert 之前完成:向量索引用 Summary+Content
|
||||
// 计算,标记进不去正文就检索不到这份媒体。
|
||||
func (m *docMemoryImpl) InsertWithMedia(d *Doc, attachments []MediaAttachment) error {
|
||||
if m.ds == nil || d == nil {
|
||||
return nil
|
||||
}
|
||||
target := &doc.Doc{ID: d.ID, Summary: d.Title, Content: d.Content}
|
||||
if target.Source == "" {
|
||||
target.Source = "plugin:" + m.plugin
|
||||
}
|
||||
|
||||
digests := m.storeAttachments(attachments, &target.Content)
|
||||
|
||||
if err := m.ds.Insert(target); err != nil {
|
||||
return err
|
||||
}
|
||||
// 回填给调用方:ID 是新建时内核生成的,Content 含内核补的标记。
|
||||
d.ID = target.ID
|
||||
d.Content = target.Content
|
||||
|
||||
m.bindDocMedia(target, digests)
|
||||
return nil
|
||||
}
|
||||
|
||||
// storeAttachments 把附件落库并把标记追加进 content,返回全部完整 digest。
|
||||
func (m *docMemoryImpl) storeAttachments(atts []MediaAttachment, content *string) []string {
|
||||
if m.ms == nil || len(atts) == 0 {
|
||||
return nil
|
||||
}
|
||||
present := sdkDigestsIn(*content)
|
||||
var digests, lines []string
|
||||
for _, a := range atts {
|
||||
full, err := sdkPutAttachment(m.ms, a, "plugin_doc:"+m.plugin)
|
||||
if err != nil {
|
||||
// 媒体存不进去不该让文档写入失败——它是记忆增强,不是文档必需品
|
||||
log.Printf("[sdk media] 插件 %s 文档附件入库失败: %v", m.plugin, err)
|
||||
continue
|
||||
}
|
||||
digests = append(digests, full)
|
||||
if present[sdkShortDigest(full)] {
|
||||
continue // 插件自己写了标记,不重复追加
|
||||
}
|
||||
present[sdkShortDigest(full)] = true
|
||||
if line := sdkMarkerFor(m.ms, full); line != "" {
|
||||
lines = append(lines, line)
|
||||
}
|
||||
}
|
||||
if len(lines) > 0 {
|
||||
if *content == "" {
|
||||
*content = strings.Join(lines, "\n")
|
||||
} else {
|
||||
*content += "\n" + strings.Join(lines, "\n")
|
||||
}
|
||||
}
|
||||
return digests
|
||||
}
|
||||
|
||||
// bindDocMedia 把附件与正文标记引用的媒体一起挂到文档 owner 上。
|
||||
func (m *docMemoryImpl) bindDocMedia(target *doc.Doc, digests []string) {
|
||||
if m.ms == nil || target.ID == "" {
|
||||
return
|
||||
}
|
||||
bound := 0
|
||||
done := make(map[string]bool, len(digests))
|
||||
for _, full := range digests {
|
||||
if done[full] {
|
||||
continue
|
||||
}
|
||||
if err := m.ms.AddRef(full, media.OwnerDocument, target.ID); err != nil {
|
||||
log.Printf("[sdk media] 文档引用绑定失败 (%s → doc %s): %v",
|
||||
sdkShortDigest(full), target.ID, err)
|
||||
continue
|
||||
}
|
||||
done[full] = true
|
||||
bound++
|
||||
}
|
||||
// 插件手写在正文里的标记同样要挂上,否则那些媒体在文档里可见却无主。
|
||||
// 共用 done:附件刚挂过的那些是同一份媒体(内核自己把标记补进了正文)。
|
||||
bound += sdkBindText(m.ms, target.Content, media.OwnerDocument, target.ID, done)
|
||||
if bound > 0 {
|
||||
log.Printf("[sdk media] 插件 %s 写入文档 %s,绑定 %d 个媒体引用",
|
||||
m.plugin, target.ID, bound)
|
||||
}
|
||||
}
|
||||
|
||||
// Remove 删除文档,同时释放它持有的媒体引用。
|
||||
//
|
||||
// 旧实现只删文档不解引用,于是那些媒体永久处于「被引用」状态:GC 不回收,
|
||||
// 磁盘只增不减。内核的归档路径(distill 的 releaseDocMedia)做了这一步,
|
||||
// 插件路径漏了同一步。
|
||||
func (m *docMemoryImpl) Remove(id string) {
|
||||
if m.ds == nil {
|
||||
return
|
||||
}
|
||||
if m.ms != nil && id != "" {
|
||||
if n, err := m.ms.DropOwner(media.OwnerDocument, id); err != nil {
|
||||
log.Printf("[sdk media] 释放文档 %s 的媒体引用失败: %v", id, err)
|
||||
} else if n > 0 {
|
||||
log.Printf("[sdk media] 文档 %s 删除,释放 %d 个媒体引用", id, n)
|
||||
}
|
||||
}
|
||||
m.ds.Remove(id)
|
||||
}
|
||||
|
||||
func (m *docMemoryImpl) Stats() map[string]interface{} {
|
||||
if m.ds == nil { return map[string]interface{}{} }
|
||||
if m.ds == nil {
|
||||
return map[string]interface{}{}
|
||||
}
|
||||
return m.ds.Stats()
|
||||
}
|
||||
|
||||
|
||||
434
internal/sdk/memory_impl_test.go
Normal file
434
internal/sdk/memory_impl_test.go
Normal file
@ -0,0 +1,434 @@
|
||||
package sdk
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory"
|
||||
doc "gitcode.com/JianFeeeee/HomeAgent/internal/memory/document"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/media"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/text"
|
||||
)
|
||||
|
||||
// 插件边界的媒体透传测试。
|
||||
//
|
||||
// 断言的都是不变量,而不是"函数被调过":
|
||||
// 1. 插件填的字段一个都不许丢(旧实现静默裁掉 Confidence/类型/SentenceText);
|
||||
// 2. 插件不必知道媒体标记格式,内核负责补;
|
||||
// 3. 媒体引用挂到正确的 owner 上,删除时释放;
|
||||
// 4. mediaStore 为 nil 时整条链路退化成纯文本,不 panic 不报错。
|
||||
|
||||
func newTestStores(t *testing.T) (*memory.GraphDB, *doc.Store, *text.Memory, *media.Store) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
|
||||
g, err := memory.NewGraphDB(filepath.Join(dir, "graph.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("NewGraphDB: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { g.Close() })
|
||||
|
||||
ds := doc.NewStore(filepath.Join(dir, "documents"))
|
||||
if err := ds.Start(); err != nil {
|
||||
t.Fatalf("doc store start: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { ds.Stop() })
|
||||
|
||||
tm := text.New(filepath.Join(dir, "text"))
|
||||
if err := tm.Start(); err != nil {
|
||||
t.Fatalf("text memory start: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { tm.Stop() })
|
||||
|
||||
ms, err := media.New(filepath.Join(dir, "media"), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("media.New: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { ms.Close() })
|
||||
|
||||
return g, ds, tm, ms
|
||||
}
|
||||
|
||||
// putDescribed 存一份带描述的媒体,返回完整 digest。
|
||||
func putDescribed(t *testing.T, ms *media.Store, payload, desc string) string {
|
||||
t.Helper()
|
||||
d, err := ms.Put([]byte(payload), media.Item{MIME: "image/png", Description: desc})
|
||||
if err != nil {
|
||||
t.Fatalf("media.Put: %v", err)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// ---------- 图记忆 ----------
|
||||
|
||||
// 旧实现只搬 Subject/Relation/Object,其余字段静默丢弃:
|
||||
// 插件标注的类型全部落成默认 Concept,置信度全成 1.0,SentenceText 直接消失。
|
||||
func TestGraphCommit_CarriesAllFields(t *testing.T) {
|
||||
g, _, _, ms := newTestStores(t)
|
||||
m := NewGraphMemoryWithMedia("tester", g, ms)
|
||||
|
||||
err := m.Commit([]Triple{{
|
||||
Subject: "张三",
|
||||
Relation: "养",
|
||||
Object: "橘猫",
|
||||
Confidence: 0.75,
|
||||
SubjectType: "Person",
|
||||
ObjectType: "Animal",
|
||||
SentenceText: "张三养了一只橘猫。",
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("Commit: %v", err)
|
||||
}
|
||||
|
||||
res, err := g.Recall([]string{"张三"}, nil, 2, "")
|
||||
if err != nil {
|
||||
t.Fatalf("Recall: %v", err)
|
||||
}
|
||||
if len(res.Relations) == 0 {
|
||||
t.Fatal("召回不到刚提交的关系")
|
||||
}
|
||||
r := res.Relations[0]
|
||||
if r.Confidence != 0.75 {
|
||||
t.Errorf("Confidence = %v,期望 0.75(插件标注的置信度被丢弃)", r.Confidence)
|
||||
}
|
||||
if r.SentenceText != "张三养了一只橘猫。" {
|
||||
t.Errorf("SentenceText = %q,期望原句(丢了它媒体就没有落点)", r.SentenceText)
|
||||
}
|
||||
|
||||
var subjType, objType string
|
||||
for _, e := range res.Entities {
|
||||
switch e.Name {
|
||||
case "张三":
|
||||
subjType = e.Type
|
||||
case "橘猫":
|
||||
objType = e.Type
|
||||
}
|
||||
}
|
||||
if subjType != "Person" || objType != "Animal" {
|
||||
t.Errorf("实体类型 = (%q,%q),期望 (Person,Animal)", subjType, objType)
|
||||
}
|
||||
}
|
||||
|
||||
// 插件只给 digest,标记与句子由内核合成;引用必须挂到 graph_sentence owner 上。
|
||||
func TestGraphCommit_BindsMediaFromDigests(t *testing.T) {
|
||||
g, _, _, ms := newTestStores(t)
|
||||
digest := putDescribed(t, ms, "png-bytes", "一张紫蓝红三色带图")
|
||||
|
||||
m := NewGraphMemoryWithMedia("tester", g, ms)
|
||||
if err := m.Commit([]Triple{{
|
||||
Subject: "配色图",
|
||||
Relation: "包含",
|
||||
Object: "三色带",
|
||||
MediaDigests: []string{digest[:12]}, // 插件手里通常只有短 digest
|
||||
}}); err != nil {
|
||||
t.Fatalf("Commit: %v", err)
|
||||
}
|
||||
|
||||
res, err := g.Recall([]string{"配色图"}, nil, 2, "")
|
||||
if err != nil {
|
||||
t.Fatalf("Recall: %v", err)
|
||||
}
|
||||
if len(res.Relations) == 0 || res.Relations[0].SentenceID == 0 {
|
||||
t.Fatal("没有句子落点 —— 媒体引用无从挂起")
|
||||
}
|
||||
sid := res.Relations[0].SentenceID
|
||||
|
||||
// 描述必须进句子:描述文本才是持久语义记忆,检索靠它。
|
||||
if !strings.Contains(res.Relations[0].SentenceText, "三色带图") {
|
||||
t.Errorf("句子里没有媒体描述: %q", res.Relations[0].SentenceText)
|
||||
}
|
||||
|
||||
refs, err := ms.Refs(media.OwnerGraphSentence, strconv.FormatInt(sid, 10))
|
||||
if err != nil {
|
||||
t.Fatalf("Refs: %v", err)
|
||||
}
|
||||
if len(refs) != 1 || refs[0] != digest {
|
||||
t.Errorf("句子 #%d 的媒体引用 = %v,期望 [%s]", sid, refs, digest)
|
||||
}
|
||||
}
|
||||
|
||||
// 插件自己按格式写了标记又同时填了 MediaDigests,不能产生两条重复引用/两份标记。
|
||||
func TestGraphCommit_NoDuplicateMarker(t *testing.T) {
|
||||
g, _, _, ms := newTestStores(t)
|
||||
digest := putDescribed(t, ms, "dup-bytes", "重复标记测试图")
|
||||
short := digest[:12]
|
||||
|
||||
m := NewGraphMemoryWithMedia("tester", g, ms)
|
||||
if err := m.Commit([]Triple{{
|
||||
Subject: "重复图",
|
||||
Relation: "标记",
|
||||
Object: "一次",
|
||||
SentenceText: "看这个 [image/png " + short + "] 重复标记测试图",
|
||||
MediaDigests: []string{short},
|
||||
}}); err != nil {
|
||||
t.Fatalf("Commit: %v", err)
|
||||
}
|
||||
|
||||
res, _ := g.Recall([]string{"重复图"}, nil, 2, "")
|
||||
if len(res.Relations) == 0 {
|
||||
t.Fatal("召回不到关系")
|
||||
}
|
||||
if n := strings.Count(res.Relations[0].SentenceText, short); n != 1 {
|
||||
t.Errorf("句子里出现 %d 次 digest,期望 1 次: %q", n, res.Relations[0].SentenceText)
|
||||
}
|
||||
}
|
||||
|
||||
// mediaStore 为 nil 时仍要能提交(媒体是增强,不是必需品),digest 留在文本里备查。
|
||||
func TestGraphCommit_NilMediaStoreDegrades(t *testing.T) {
|
||||
g, _, _, _ := newTestStores(t)
|
||||
m := NewGraphMemory(g)
|
||||
|
||||
if err := m.Commit([]Triple{{
|
||||
Subject: "无存储",
|
||||
Relation: "仍可",
|
||||
Object: "提交",
|
||||
MediaDigests: []string{"aabbccddeeff"},
|
||||
}}); err != nil {
|
||||
t.Fatalf("Commit 在无媒体存储时不该失败: %v", err)
|
||||
}
|
||||
|
||||
res, _ := g.Recall([]string{"无存储"}, nil, 2, "")
|
||||
if len(res.Relations) == 0 {
|
||||
t.Fatal("召回不到关系")
|
||||
}
|
||||
if !strings.Contains(res.Relations[0].SentenceText, "aabbccddeeff") {
|
||||
t.Errorf("digest 应留在句子里以备将来反查: %q", res.Relations[0].SentenceText)
|
||||
}
|
||||
}
|
||||
|
||||
// Recall 必须把置信度带回插件:拿不到它,插件只能把所有召回结果等同看待。
|
||||
func TestGraphRecall_CarriesConfidence(t *testing.T) {
|
||||
g, _, _, ms := newTestStores(t)
|
||||
m := NewGraphMemoryWithMedia("tester", g, ms)
|
||||
|
||||
// 实体名至少两个字符:validEntityName 会静默跳过单字实体,
|
||||
// 那样 Commit 返回 nil 但什么都没写,测试会退化成假阳性。
|
||||
if err := m.Commit([]Triple{{
|
||||
Subject: "甲方", Relation: "疑似", Object: "乙方", Confidence: 0.3,
|
||||
}}); err != nil {
|
||||
t.Fatalf("Commit: %v", err)
|
||||
}
|
||||
|
||||
_, rels, err := m.Recall([]string{"甲方"}, 2)
|
||||
if err != nil {
|
||||
t.Fatalf("Recall: %v", err)
|
||||
}
|
||||
if len(rels) == 0 {
|
||||
t.Fatal("召回为空")
|
||||
}
|
||||
if rels[0].Confidence != 0.3 {
|
||||
t.Errorf("Confidence = %v,期望 0.3", rels[0].Confidence)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 文档记忆(知识库) ----------
|
||||
|
||||
// 附件带 Data → 落进 CAS、标记补进正文、引用挂到文档 owner。
|
||||
func TestDocInsertWithMedia_StoresAndBinds(t *testing.T) {
|
||||
_, ds, _, ms := newTestStores(t)
|
||||
dm := NewDocMemoryWithMedia("tester", ds, ms)
|
||||
|
||||
d := &Doc{Title: "带图笔记", Content: "这是正文。"}
|
||||
err := dm.InsertWithMedia(d, []MediaAttachment{{
|
||||
MIME: "image/png",
|
||||
Data: []byte("attachment-bytes"),
|
||||
Name: "chart.png",
|
||||
Description: "一张柱状图",
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("InsertWithMedia: %v", err)
|
||||
}
|
||||
if d.ID == "" {
|
||||
t.Fatal("ID 未回填 —— 插件拿不到刚写入文档的 id")
|
||||
}
|
||||
|
||||
// 标记必须进正文:向量索引用 Summary+Content 计算,
|
||||
// 标记进不去正文就永远检索不到这份媒体。
|
||||
if !strings.Contains(d.Content, "柱状图") {
|
||||
t.Errorf("正文里没有媒体标记: %q", d.Content)
|
||||
}
|
||||
|
||||
refs, err := ms.Refs(media.OwnerDocument, d.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("Refs: %v", err)
|
||||
}
|
||||
if len(refs) != 1 {
|
||||
t.Fatalf("文档媒体引用 = %v,期望 1 条", refs)
|
||||
}
|
||||
// 内容可读,说明真的落盘了而不只是记了个 digest。
|
||||
got, err := ms.Get(refs[0])
|
||||
if err != nil || string(got) != "attachment-bytes" {
|
||||
t.Errorf("媒体内容读回失败: %v / %q", err, got)
|
||||
}
|
||||
}
|
||||
|
||||
// 只给 Digest 的附件是「引用已有内容」,不该报错也不该重复落盘。
|
||||
func TestDocInsertWithMedia_DigestOnlyReference(t *testing.T) {
|
||||
_, ds, _, ms := newTestStores(t)
|
||||
digest := putDescribed(t, ms, "existing", "已有的图")
|
||||
before := ms.Stats()["count"]
|
||||
|
||||
dm := NewDocMemoryWithMedia("tester", ds, ms)
|
||||
d := &Doc{Title: "引用已有", Content: "正文"}
|
||||
if err := dm.InsertWithMedia(d, []MediaAttachment{{Digest: digest[:10]}}); err != nil {
|
||||
t.Fatalf("InsertWithMedia: %v", err)
|
||||
}
|
||||
|
||||
if after := ms.Stats()["count"]; after != before {
|
||||
t.Errorf("媒体条数从 %v 变成 %v —— 引用已有内容不该新增", before, after)
|
||||
}
|
||||
refs, _ := ms.Refs(media.OwnerDocument, d.ID)
|
||||
if len(refs) != 1 || refs[0] != digest {
|
||||
t.Errorf("引用 = %v,期望 [%s]", refs, digest)
|
||||
}
|
||||
}
|
||||
|
||||
// Query 必须回媒体元数据但**不回字节**:一次检索可能命中几十份媒体,
|
||||
// 全塞回插件会把跨进程消息撑爆。
|
||||
func TestDocQuery_FillsMediaMetadataWithoutBytes(t *testing.T) {
|
||||
_, ds, _, ms := newTestStores(t)
|
||||
dm := NewDocMemoryWithMedia("tester", ds, ms)
|
||||
|
||||
d := &Doc{Title: "紫蓝红三色带", Content: "配色说明"}
|
||||
if err := dm.InsertWithMedia(d, []MediaAttachment{{
|
||||
MIME: "image/png", Data: []byte("query-bytes"), Description: "三色带图",
|
||||
}}); err != nil {
|
||||
t.Fatalf("InsertWithMedia: %v", err)
|
||||
}
|
||||
|
||||
got := dm.Query("紫蓝红三色带 配色说明", 3)
|
||||
if len(got) == 0 {
|
||||
t.Fatal("检索不到刚写入的文档")
|
||||
}
|
||||
var hit *Doc
|
||||
for _, g := range got {
|
||||
if g.ID == d.ID {
|
||||
hit = g
|
||||
}
|
||||
}
|
||||
if hit == nil {
|
||||
t.Fatalf("检索结果里没有目标文档: %+v", got)
|
||||
}
|
||||
if len(hit.MediaDigests) != 1 {
|
||||
t.Errorf("MediaDigests = %v,期望 1 条", hit.MediaDigests)
|
||||
}
|
||||
if len(hit.Attachments) != 1 {
|
||||
t.Fatalf("Attachments = %v,期望 1 条", hit.Attachments)
|
||||
}
|
||||
att := hit.Attachments[0]
|
||||
if att.MIME != "image/png" || att.Description != "三色带图" {
|
||||
t.Errorf("附件元数据 = %+v,期望 mime=image/png desc=三色带图", att)
|
||||
}
|
||||
if len(att.Data) != 0 {
|
||||
t.Errorf("Attachments 不该带字节(%d 字节)—— 需要时按 digest 单取", len(att.Data))
|
||||
}
|
||||
}
|
||||
|
||||
// 历史文档只有标记、没有 media_refs(旧版插件写入的)。
|
||||
// 此时要能从正文标记反解出附件,否则那些文档的媒体对插件永远不可见。
|
||||
func TestDocQuery_FallsBackToMarkers(t *testing.T) {
|
||||
_, ds, _, ms := newTestStores(t)
|
||||
digest := putDescribed(t, ms, "legacy", "历史图片")
|
||||
|
||||
// 直接写底层 store,绕过 SDK 的绑定逻辑,模拟历史数据。
|
||||
if err := ds.Insert(&doc.Doc{
|
||||
Summary: "历史文档",
|
||||
Content: "旧正文 [image/png " + digest[:12] + "] 历史图片",
|
||||
}); err != nil {
|
||||
t.Fatalf("Insert: %v", err)
|
||||
}
|
||||
|
||||
dm := NewDocMemoryWithMedia("tester", ds, ms)
|
||||
got := dm.Query("历史文档 旧正文", 3)
|
||||
if len(got) == 0 {
|
||||
t.Fatal("检索不到历史文档")
|
||||
}
|
||||
if len(got[0].MediaDigests) != 1 || got[0].MediaDigests[0] != digest {
|
||||
t.Errorf("MediaDigests = %v,期望从标记反解出 [%s]", got[0].MediaDigests, digest)
|
||||
}
|
||||
}
|
||||
|
||||
// 旧实现删文档不解引用 → 媒体永久"被引用",GC 收不掉,磁盘只增不减。
|
||||
func TestDocRemove_ReleasesMediaRefs(t *testing.T) {
|
||||
_, ds, _, ms := newTestStores(t)
|
||||
dm := NewDocMemoryWithMedia("tester", ds, ms)
|
||||
|
||||
d := &Doc{Title: "待删除", Content: "正文"}
|
||||
if err := dm.InsertWithMedia(d, []MediaAttachment{{
|
||||
MIME: "image/png", Data: []byte("to-be-freed"), Description: "会被释放的图",
|
||||
}}); err != nil {
|
||||
t.Fatalf("InsertWithMedia: %v", err)
|
||||
}
|
||||
if refs, _ := ms.Refs(media.OwnerDocument, d.ID); len(refs) != 1 {
|
||||
t.Fatalf("前置条件不成立,引用 = %v", refs)
|
||||
}
|
||||
|
||||
dm.Remove(d.ID)
|
||||
|
||||
if refs, _ := ms.Refs(media.OwnerDocument, d.ID); len(refs) != 0 {
|
||||
t.Errorf("删除文档后仍有 %v 条引用 —— GC 永远收不掉这份媒体", refs)
|
||||
}
|
||||
}
|
||||
|
||||
// mediaStore 为 nil 时 Insert/Query/Remove 必须与本特性上线前完全一致。
|
||||
func TestDocMemory_NilMediaStoreDegrades(t *testing.T) {
|
||||
_, ds, _, _ := newTestStores(t)
|
||||
dm := NewDocMemory(ds)
|
||||
|
||||
d := &Doc{Title: "无媒体存储", Content: "正文照常写入"}
|
||||
if err := dm.InsertWithMedia(d, []MediaAttachment{{
|
||||
MIME: "image/png", Data: []byte("ignored"),
|
||||
}}); err != nil {
|
||||
t.Fatalf("无媒体存储时写入不该失败: %v", err)
|
||||
}
|
||||
if d.ID == "" {
|
||||
t.Error("ID 仍应回填")
|
||||
}
|
||||
got := dm.Query("无媒体存储 正文照常写入", 3)
|
||||
if len(got) == 0 {
|
||||
t.Fatal("检索不到文档")
|
||||
}
|
||||
if len(got[0].Attachments) != 0 {
|
||||
t.Errorf("无媒体存储时不该有附件: %+v", got[0].Attachments)
|
||||
}
|
||||
dm.Remove(d.ID) // 不该 panic
|
||||
}
|
||||
|
||||
// ---------- 文本记忆 ----------
|
||||
|
||||
// 文本记忆是追加写 JSONL,没有稳定 owner_id 可挂引用,
|
||||
// 媒体只能以标记形式留在正文里;读回时要能反解成结构化附件。
|
||||
func TestTextMemory_AttachmentRoundTrip(t *testing.T) {
|
||||
_, _, tm, ms := newTestStores(t)
|
||||
m := NewTextMemoryWithMedia("tester", tm, ms)
|
||||
|
||||
if err := m.Append(TextEvent{
|
||||
Role: "user",
|
||||
Content: "看这张图",
|
||||
Attachments: []MediaAttachment{{
|
||||
MIME: "image/png", Data: []byte("text-mem-bytes"), Description: "文本记忆里的图",
|
||||
}},
|
||||
}); err != nil {
|
||||
t.Fatalf("Append: %v", err)
|
||||
}
|
||||
|
||||
got, err := m.RecentEvents(5)
|
||||
if err != nil {
|
||||
t.Fatalf("RecentEvents: %v", err)
|
||||
}
|
||||
if len(got) == 0 {
|
||||
t.Fatal("读不到刚追加的事件")
|
||||
}
|
||||
last := got[len(got)-1]
|
||||
if !strings.Contains(last.Content, "文本记忆里的图") {
|
||||
t.Errorf("正文里没有媒体标记: %q", last.Content)
|
||||
}
|
||||
if len(last.Attachments) != 1 {
|
||||
t.Fatalf("Attachments = %+v,期望 1 条(标记应能反解)", last.Attachments)
|
||||
}
|
||||
if last.Attachments[0].Description != "文本记忆里的图" {
|
||||
t.Errorf("附件描述 = %q", last.Attachments[0].Description)
|
||||
}
|
||||
}
|
||||
@ -195,6 +195,57 @@ func (a ioAdapter) InjectTextNoMemory(source, channel, text string) {
|
||||
}
|
||||
}
|
||||
|
||||
// InjectInputMedia 注入带媒体内容块的输入。
|
||||
//
|
||||
// blocks 放在 payload 的 media_blocks 里,由 eventloop 取出转进
|
||||
// stageCtx.Extra——与用户直接发图走的是同一条通道,因此自动获得
|
||||
// CAS 落盘与媒体记忆绑定。与 SetToolBlocks 的区别:后者只能在工具
|
||||
// 调用内部用,且媒体要等到下一条 tool message 才到模型手上。
|
||||
func (a ioAdapter) InjectInputMedia(source, channel, text string, blocks []pubsdk.ContentBlock) {
|
||||
if a.iom != nil {
|
||||
a.iom.InjectInputTo(source, channel, "text", map[string]interface{}{
|
||||
"content": text,
|
||||
"media_blocks": blocks,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// InjectInputMediaSync 注入带媒体内容块的输入并同步等待回复。
|
||||
func (a ioAdapter) InjectInputMediaSync(source, channel, text string, blocks []pubsdk.ContentBlock) string {
|
||||
if a.iom == nil {
|
||||
return ""
|
||||
}
|
||||
out := a.iom.InjectInputSyncTo(source, channel, "text", map[string]interface{}{
|
||||
"content": text,
|
||||
"media_blocks": blocks,
|
||||
})
|
||||
if out == nil {
|
||||
return ""
|
||||
}
|
||||
reply, _ := out.Payload["content"].(string)
|
||||
return reply
|
||||
}
|
||||
|
||||
// InjectInterruptMedia 注入带媒体内容块的中断,可抢占当前 LLM 处理。
|
||||
func (a ioAdapter) InjectInterruptMedia(source, channel, text string, blocks []pubsdk.ContentBlock) {
|
||||
if a.iom != nil {
|
||||
a.iom.InjectInterrupt(source, channel, map[string]interface{}{
|
||||
"type": "text",
|
||||
"content": text,
|
||||
"media_blocks": blocks,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ContentBlock / ImageURL / AudioURL 是多模态内容块在插件边界上的类型。
|
||||
//
|
||||
// 别名到公共 SDK 而非另建一套:内置插件(webui/multimodal 等)与外部插件必须
|
||||
// 用同一套结构,否则 resolveInput 的类型分支要认第三种类型,而漏认的后果是
|
||||
// 媒体被静默丢弃。
|
||||
type ContentBlock = pubsdk.ContentBlock
|
||||
type ImageURL = pubsdk.ImageURL
|
||||
type AudioURL = pubsdk.AudioURL
|
||||
|
||||
// SDKConfig holds all dependencies for creating a PluginSDK.
|
||||
type SDKConfig struct {
|
||||
IOManager *agentIO.IOManager
|
||||
|
||||
32
third_party/homeagent-sdk/meta/meta.go
vendored
32
third_party/homeagent-sdk/meta/meta.go
vendored
@ -6,10 +6,31 @@ var (
|
||||
// Version 是 HomeAgent SDK 版本号。
|
||||
// 通过 `-ldflags="-X gitcode.com/JianFeeeee/homeagent-sdk/meta.Version=vX.Y.Z"` 注入。
|
||||
//
|
||||
// 版本号语义:**SDK 版本跟随核心的中版本,patch 位恒为 .0**。
|
||||
// 整条核心 1.1.x 线(1.1.0、1.1.1、1.1.7…)共用 SDK 1.1.0;
|
||||
// 只有核心进入 1.2.0 这种中版本跃迁时 SDK 才升到 1.2.0。
|
||||
// 这样插件开发者只需关心「我在为哪个中版本写插件」,
|
||||
// 不必跟着核心的每个 bugfix 换 SDK 依赖(见 核心仓 docs/git-branching.md §七)。
|
||||
//
|
||||
// 1.0.0:插件运行模型从 C ABI 动态库改为子进程 + 共享内存。
|
||||
// 公开 SDK 接口(sdk/ 目录)**零改动**——插件业务代码不需要改一行,
|
||||
// 但产物形态变了(plugin.so → plugin.bin),必须用新版 plugindev 重编。
|
||||
Version = "1.0.0"
|
||||
// 公开 SDK 接口零改动,但产物形态变了(plugin.so → plugin.bin)。
|
||||
// 1.1.0:多模态贯通插件边界。**全部是新增,无签名变更**:
|
||||
// - Triple.SentenceText / Triple.MediaDigests
|
||||
// - Doc.MediaDigests / Doc.Attachments、MediaAttachment
|
||||
// - TextEvent.Attachments
|
||||
// - DocMemoryAPI.InsertWithMedia
|
||||
// - IOInjector 的 InjectInputMedia / InjectInputMediaSync /
|
||||
// InjectInterruptMedia;PluginSDK 补上缺失的 SetToolBlocks 包装
|
||||
// 同版修掉两处并发竞态(sdk/stress_test.go 的 -race 实证,不是理论风险):
|
||||
// PluginSDK 的 API 字段与 autoRestart 标志此前无锁,而写方
|
||||
// (内核注入 API、插件 SetAutoRestart)与读方(插件后台 goroutine
|
||||
// 注入、内核 registry 读 AutoRestart)天然跨 goroutine。
|
||||
// 存量插件不需要改一行也不需要重编:新增方法由**插件调用、内核实现**,
|
||||
// 不调就不受影响。想用新字段的插件重编即可。
|
||||
//
|
||||
// ❗发布分支上此值是**本条发布线的 SDK 定版**;main 上则是下一个未发布中版本
|
||||
// (见 核心仓 docs/git-branching.md §2.1 与 §七.1)。
|
||||
Version = "1.1.0"
|
||||
|
||||
// Commit 是构建时的 Git commit hash。
|
||||
Commit = "unknown"
|
||||
@ -27,6 +48,11 @@ var (
|
||||
//
|
||||
// 1.0.0 是硬下限而非建议值:0.9.x 内核只会 dlopen `.so`,
|
||||
// 本版工具链产出的 `plugin.bin` 在旧内核上根本不会被识别。
|
||||
//
|
||||
// ⚠️ 1.1.0 新增的媒体接口需要核心 **1.1.1+**(更早的核心没有
|
||||
// doc.insertWithMedia / io.injectMedia* 这些 RPC,调用会返回 unknown method)。
|
||||
// 这里仍写 1.0.0,因为它是「SDK 能在其上运行」的下限;
|
||||
// 媒体接口是可选能力,不用就不受影响。
|
||||
CoreVersion = "1.0.0"
|
||||
)
|
||||
|
||||
|
||||
64
third_party/homeagent-sdk/sdk/memory.go
vendored
64
third_party/homeagent-sdk/sdk/memory.go
vendored
@ -25,13 +25,18 @@ type Relation struct {
|
||||
}
|
||||
|
||||
// Triple represents a subject-relation-object triple for the knowledge graph.
|
||||
//
|
||||
// SentenceText 是这条三元组的原句,会写进 sentences 表;媒体引用挂在句子上,
|
||||
// 所以 MediaDigests 非空时内核会保证句子存在(不给就自动合成一句)。
|
||||
type Triple struct {
|
||||
Subject string `json:"subject"`
|
||||
Relation string `json:"relation"`
|
||||
Object string `json:"object"`
|
||||
Confidence float64 `json:"confidence,omitempty"`
|
||||
SubjectType string `json:"subject_type,omitempty"`
|
||||
ObjectType string `json:"object_type,omitempty"`
|
||||
Subject string `json:"subject"`
|
||||
Relation string `json:"relation"`
|
||||
Object string `json:"object"`
|
||||
Confidence float64 `json:"confidence,omitempty"`
|
||||
SubjectType string `json:"subject_type,omitempty"`
|
||||
ObjectType string `json:"object_type,omitempty"`
|
||||
SentenceText string `json:"sentence_text,omitempty"`
|
||||
MediaDigests []string `json:"media_digests,omitempty"`
|
||||
}
|
||||
|
||||
// TextMemoryAPI provides access to chronological text event storage.
|
||||
@ -40,27 +45,52 @@ type TextMemoryAPI interface {
|
||||
}
|
||||
|
||||
// TextEvent represents a single text memory event.
|
||||
// MediaAttachment 描述一份与记忆关联的媒体。
|
||||
//
|
||||
// 两个方向共用一个类型:
|
||||
// - 写入(InsertWithMedia):给 Data + MIME 就是新内容;只给 Digest 则是引用已有内容。
|
||||
// - 读出(Query):内核只填 Digest/MIME/Description,**不回 Data**——
|
||||
// 一次检索可能命中几十张图,把字节全塞回插件会把 ABI 消息撑爆。
|
||||
// 需要字节时拿 Digest 单独取。
|
||||
type MediaAttachment struct {
|
||||
Digest string `json:"digest,omitempty"`
|
||||
MIME string `json:"mime,omitempty"`
|
||||
Data []byte `json:"data,omitempty"`
|
||||
Name string `json:"name,omitempty"`
|
||||
Description string `json:"description,omitempty"`
|
||||
}
|
||||
|
||||
type TextEvent struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
Timestamp int64 `json:"timestamp"`
|
||||
Channel string `json:"channel,omitempty"`
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
Timestamp int64 `json:"timestamp"`
|
||||
Channel string `json:"channel,omitempty"`
|
||||
Attachments []MediaAttachment `json:"attachments,omitempty"`
|
||||
}
|
||||
|
||||
// DocMemoryAPI provides access to the document vector store.
|
||||
type DocMemoryAPI interface {
|
||||
Query(text string, topK int) []*Doc
|
||||
Insert(doc *Doc) error
|
||||
// InsertWithMedia 写入文档并关联媒体。attachments 里带 Data 的会落进
|
||||
// 内容寻址存储(相同字节只存一份),只带 Digest 的直接引用已有内容。
|
||||
// 插件无需自己拼标记:内核会把 `[mime <短 digest>] <描述>` 补进 Content,
|
||||
// 让向量检索和后续蒸馏都能看到这份媒体。
|
||||
InsertWithMedia(doc *Doc, attachments []MediaAttachment) error
|
||||
Remove(id string)
|
||||
Stats() map[string]interface{}
|
||||
}
|
||||
|
||||
// Doc represents a document in the document store.
|
||||
//
|
||||
// MediaDigests / Attachments 在 Query 返回时由内核填充(仅元数据,不带字节)。
|
||||
type Doc struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Content string `json:"content"`
|
||||
Score float64 `json:"score,omitempty"`
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Content string `json:"content"`
|
||||
Score float64 `json:"score,omitempty"`
|
||||
MediaDigests []string `json:"media_digests,omitempty"`
|
||||
Attachments []MediaAttachment `json:"attachments,omitempty"`
|
||||
}
|
||||
|
||||
// SocialAPI provides read-only access to the social graph (person profiles and relationships).
|
||||
@ -75,9 +105,9 @@ type SocialAPI interface {
|
||||
|
||||
// PersonProfile represents a person's complete profile (traits + social relations).
|
||||
type PersonProfile struct {
|
||||
Name string `json:"name"`
|
||||
Traits map[string]string `json:"traits,omitempty"`
|
||||
Relations []SocialRelation `json:"relations,omitempty"`
|
||||
Name string `json:"name"`
|
||||
Traits map[string]string `json:"traits,omitempty"`
|
||||
Relations []SocialRelation `json:"relations,omitempty"`
|
||||
}
|
||||
|
||||
// SocialRelation represents a social relationship between two persons.
|
||||
|
||||
257
third_party/homeagent-sdk/sdk/plugin.go
vendored
257
third_party/homeagent-sdk/sdk/plugin.go
vendored
@ -61,14 +61,18 @@ type StageContext struct {
|
||||
Memory []MemItem
|
||||
NoMemory bool
|
||||
Extra map[string]interface{}
|
||||
Errors []string // 阶段处理过程中的错误信息
|
||||
Errors []string // 阶段处理过程中的错误信息
|
||||
}
|
||||
|
||||
func (c *StageContext) RLock() { c.mu.RLock() }
|
||||
func (c *StageContext) RUnlock() { c.mu.RUnlock() }
|
||||
func (c *StageContext) Lock() { c.mu.Lock() }
|
||||
func (c *StageContext) Unlock() { c.mu.Unlock() }
|
||||
func (c *StageContext) IsResponded() bool { c.mu.RLock(); defer c.mu.RUnlock(); return c.Response != nil }
|
||||
func (c *StageContext) RLock() { c.mu.RLock() }
|
||||
func (c *StageContext) RUnlock() { c.mu.RUnlock() }
|
||||
func (c *StageContext) Lock() { c.mu.Lock() }
|
||||
func (c *StageContext) Unlock() { c.mu.Unlock() }
|
||||
func (c *StageContext) IsResponded() bool {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return c.Response != nil
|
||||
}
|
||||
|
||||
// MemItem represents a memory item in stage context.
|
||||
type MemItem struct {
|
||||
@ -100,8 +104,8 @@ type ToolDef struct {
|
||||
Plugin string `json:"plugin,omitempty"`
|
||||
Description string `json:"description"`
|
||||
Parameters map[string]interface{} `json:"parameters"`
|
||||
NoMemory bool `json:"no_memory,omitempty"` // 此工具输出不参与记忆计算,但原文保留
|
||||
Cleaner func(string) string `json:"-"` // 计算层过滤函数,不改原文;仅在向量化/jieba/蒸馏时调用
|
||||
NoMemory bool `json:"no_memory,omitempty"` // 此工具输出不参与记忆计算,但原文保留
|
||||
Cleaner func(string) string `json:"-"` // 计算层过滤函数,不改原文;仅在向量化/jieba/蒸馏时调用
|
||||
}
|
||||
|
||||
// IOInjector provides methods for injecting input and interrupts into the agent pipeline.
|
||||
@ -117,6 +121,9 @@ type IOInjector interface {
|
||||
// SetToolBlocks 插件工具注入多模态内容块(image_url/audio_url),内核在下一条
|
||||
// tool message 的 content 数组里带上这些块,让模型在后续轮次看到图/听到音频。
|
||||
SetToolBlocks(blocks []ContentBlock)
|
||||
InjectInputMedia(source, channel, text string, blocks []ContentBlock)
|
||||
InjectInputMediaSync(source, channel, text string, blocks []ContentBlock) string
|
||||
InjectInterruptMedia(source, channel, text string, blocks []ContentBlock)
|
||||
}
|
||||
|
||||
// EventType identifies the kind of system event.
|
||||
@ -221,6 +228,24 @@ type PluginSDK struct {
|
||||
events EventSubscriber
|
||||
plgMgr PluginMgrAPI
|
||||
|
||||
// apiMu 保护上面这些由内核注入的 API 字段,以及 autoRestart。
|
||||
//
|
||||
// 这些字段的写方与读方天然跨 goroutine:
|
||||
// - 写方是内核(加载/重载插件时注入 API)与插件自己(SetAutoRestart);
|
||||
// - 读方是插件在 Start() 里起的后台 goroutine(轮询、监听、定时器
|
||||
// 都要拿 injector 往管道里注消息),以及内核 registry —— 它在
|
||||
// 另一个 goroutine 读 AutoRestart() 决定崩溃后是否重启。
|
||||
// SetAutoRestart 的文档用法本身就是「连接建立后再决定能否自动重启」,
|
||||
// 而连接建立通常发生在后台 goroutine 里,于是这对读写必然并发。
|
||||
//
|
||||
// sdk/stress_test.go 的 -race 实测确认这是真竞态,不是理论风险。
|
||||
// 未加锁时的生产表现是偶发 nil 解引用崩溃(读到半个接口值)。
|
||||
//
|
||||
// 约定:只在持锁期间取字段值,取完立刻释放再调用。
|
||||
// 持锁调用会把 InjectInputSync 这类阻塞到 agent 回复(可达数分钟)的
|
||||
// 方法与 SetIOInjector 串到一起,让插件重载卡死。
|
||||
apiMu sync.RWMutex
|
||||
|
||||
autoRestart bool
|
||||
|
||||
stopMu sync.Mutex
|
||||
@ -247,28 +272,57 @@ func New(name string, sett SettingsAPI, regTool ToolRegistrar, regStage StageReg
|
||||
func (s *PluginSDK) PluginName() string { return s.name }
|
||||
|
||||
// Settings returns the settings API for reading/writing plugin configuration.
|
||||
// sett 在 New 时一次性写入且无 setter,故不需要加锁。
|
||||
func (s *PluginSDK) Settings() SettingsAPI { return s.sett }
|
||||
|
||||
// Memory returns the graph memory API (may be nil if not available).
|
||||
func (s *PluginSDK) Memory() MemoryAPI { return s.mem }
|
||||
func (s *PluginSDK) Memory() MemoryAPI {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.mem
|
||||
}
|
||||
|
||||
// TextMemory returns the text memory API (may be nil if not available).
|
||||
func (s *PluginSDK) TextMemory() TextMemoryAPI { return s.textMem }
|
||||
func (s *PluginSDK) TextMemory() TextMemoryAPI {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.textMem
|
||||
}
|
||||
|
||||
// DocMemory returns the document memory API (may be nil if not available).
|
||||
func (s *PluginSDK) DocMemory() DocMemoryAPI { return s.docMem }
|
||||
func (s *PluginSDK) DocMemory() DocMemoryAPI {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.docMem
|
||||
}
|
||||
|
||||
// Knowledge returns the knowledge store API (may be nil if not available).
|
||||
func (s *PluginSDK) Knowledge() KnowledgeAPI { return s.know }
|
||||
func (s *PluginSDK) Knowledge() KnowledgeAPI {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.know
|
||||
}
|
||||
|
||||
// LLM returns the LLM provider API (may be nil if not available).
|
||||
func (s *PluginSDK) LLM() LLMAPI { return s.llm }
|
||||
func (s *PluginSDK) LLM() LLMAPI {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.llm
|
||||
}
|
||||
|
||||
// Social returns the social graph API (may be nil if not available).
|
||||
func (s *PluginSDK) Social() SocialAPI { return s.social }
|
||||
func (s *PluginSDK) Social() SocialAPI {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.social
|
||||
}
|
||||
|
||||
// Events returns the event subscriber for listening to kernel events (may be nil if not available).
|
||||
func (s *PluginSDK) Events() EventSubscriber { return s.events }
|
||||
func (s *PluginSDK) Events() EventSubscriber {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.events
|
||||
}
|
||||
|
||||
// RegisterTool registers a tool that the LLM can call.
|
||||
func (s *PluginSDK) RegisterTool(name string, def ToolDef, handler ToolHandler) error {
|
||||
@ -282,8 +336,9 @@ func (s *PluginSDK) RegisterTool(name string, def ToolDef, handler ToolHandler)
|
||||
}
|
||||
|
||||
// RegisterStage registers a handler for a pipeline stage.
|
||||
// scope: StageScopeGlobal (default) — receives all stage events.
|
||||
// StageScopeOwnTools — only before_toolcall/after_toolcall for this plugin's tools.
|
||||
//
|
||||
// scope: StageScopeGlobal (default) — receives all stage events.
|
||||
// StageScopeOwnTools — only before_toolcall/after_toolcall for this plugin's tools.
|
||||
func (s *PluginSDK) RegisterStage(stage Stage, handler StageHandler, scope ...StageScope) {
|
||||
if s.regStage == nil {
|
||||
return
|
||||
@ -333,8 +388,11 @@ func (s *PluginSDK) RegisterPluginAPI(name string) error {
|
||||
// def: 通道在记忆计算层的行为(NoMemory/Cleaner)
|
||||
// handler: receives args map with keys: payload (string), type (string), meta (string|optional)
|
||||
func (s *PluginSDK) RegisterOutputChannel(name string, caps int, desc string, def ChannelDef, handler ToolHandler) error {
|
||||
if s.regOutput != nil {
|
||||
return s.regOutput(name, caps, desc, def, handler)
|
||||
s.apiMu.RLock()
|
||||
reg := s.regOutput
|
||||
s.apiMu.RUnlock()
|
||||
if reg != nil {
|
||||
return reg(name, caps, desc, def, handler)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@ -343,57 +401,126 @@ func (s *PluginSDK) RegisterOutputChannel(name string, caps int, desc string, de
|
||||
// def.NoMemory: 此通道输入不参与记忆计算
|
||||
// def.Cleaner: 计算层对输入文本清洗后(不改原文)再向量化/提关键词
|
||||
func (s *PluginSDK) RegisterInputChannel(name string, def ChannelDef) error {
|
||||
if s.regInput != nil {
|
||||
return s.regInput(name, def)
|
||||
s.apiMu.RLock()
|
||||
reg := s.regInput
|
||||
s.apiMu.RUnlock()
|
||||
if reg != nil {
|
||||
return reg(name, def)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// 以下 setter 由内核在启动/重载时调用,与插件后台 goroutine 的读并发,故加锁。
|
||||
|
||||
// SetOutputChannelRegistrar sets the output channel registrar (called by the core at startup).
|
||||
func (s *PluginSDK) SetOutputChannelRegistrar(r OutputChannelRegistrar) { s.regOutput = r }
|
||||
func (s *PluginSDK) SetOutputChannelRegistrar(r OutputChannelRegistrar) {
|
||||
s.apiMu.Lock()
|
||||
s.regOutput = r
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
// SetInputChannelRegistrar sets the input channel registrar (called by the core at startup).
|
||||
func (s *PluginSDK) SetInputChannelRegistrar(r InputChannelRegistrar) { s.regInput = r }
|
||||
func (s *PluginSDK) SetInputChannelRegistrar(r InputChannelRegistrar) {
|
||||
s.apiMu.Lock()
|
||||
s.regInput = r
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
// SetIOInjector sets the IO injector (called by the core at startup).
|
||||
func (s *PluginSDK) SetIOInjector(io IOInjector) { s.io = io }
|
||||
func (s *PluginSDK) SetIOInjector(io IOInjector) {
|
||||
s.apiMu.Lock()
|
||||
s.io = io
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
// SetMemoryAPI sets the memory API (called by the core at startup).
|
||||
func (s *PluginSDK) SetMemoryAPI(mem MemoryAPI) { s.mem = mem }
|
||||
func (s *PluginSDK) SetTextMemoryAPI(tm TextMemoryAPI) { s.textMem = tm }
|
||||
func (s *PluginSDK) SetDocMemoryAPI(dm DocMemoryAPI) { s.docMem = dm }
|
||||
func (s *PluginSDK) SetKnowledgeAPI(kn KnowledgeAPI) { s.know = kn }
|
||||
func (s *PluginSDK) SetLLMAPI(llm LLMAPI) { s.llm = llm }
|
||||
func (s *PluginSDK) SetSocialAPI(social SocialAPI) { s.social = social }
|
||||
func (s *PluginSDK) SetEventSubscriber(es EventSubscriber) { s.events = es }
|
||||
func (s *PluginSDK) SetMemoryAPI(mem MemoryAPI) {
|
||||
s.apiMu.Lock()
|
||||
s.mem = mem
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
func (s *PluginSDK) SetTextMemoryAPI(tm TextMemoryAPI) {
|
||||
s.apiMu.Lock()
|
||||
s.textMem = tm
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
func (s *PluginSDK) SetDocMemoryAPI(dm DocMemoryAPI) {
|
||||
s.apiMu.Lock()
|
||||
s.docMem = dm
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
func (s *PluginSDK) SetKnowledgeAPI(kn KnowledgeAPI) {
|
||||
s.apiMu.Lock()
|
||||
s.know = kn
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
func (s *PluginSDK) SetLLMAPI(llm LLMAPI) {
|
||||
s.apiMu.Lock()
|
||||
s.llm = llm
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
func (s *PluginSDK) SetSocialAPI(social SocialAPI) {
|
||||
s.apiMu.Lock()
|
||||
s.social = social
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
func (s *PluginSDK) SetEventSubscriber(es EventSubscriber) {
|
||||
s.apiMu.Lock()
|
||||
s.events = es
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
// SetPluginMgrAPI sets the plugin manager API (called by the bridge at startup).
|
||||
func (s *PluginSDK) SetPluginMgrAPI(pm PluginMgrAPI) { s.plgMgr = pm }
|
||||
func (s *PluginSDK) SetPluginMgrAPI(pm PluginMgrAPI) {
|
||||
s.apiMu.Lock()
|
||||
s.plgMgr = pm
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
// PluginMgr returns the plugin manager API (ReloadOne / ReloadPlugins / list).
|
||||
// May be nil if the host did not wire it.
|
||||
func (s *PluginSDK) PluginMgr() PluginMgrAPI { return s.plgMgr }
|
||||
func (s *PluginSDK) PluginMgr() PluginMgrAPI {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.plgMgr
|
||||
}
|
||||
|
||||
// ---- IO Convenience Methods ----
|
||||
|
||||
// injector 取当前 injector 的快照。
|
||||
//
|
||||
// 取完即释放锁再调用:InjectInputSync 会阻塞到 agent 回复(可达数分钟),
|
||||
// 若持锁调用,插件重载时的 SetIOInjector 会一起卡住。
|
||||
func (s *PluginSDK) injector() IOInjector {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.io
|
||||
}
|
||||
|
||||
// InjectInterruptText injects a text interrupt that can preempt current LLM processing.
|
||||
func (s *PluginSDK) InjectInterruptText(source, channel, text string) {
|
||||
if s.io != nil {
|
||||
s.io.InjectInterruptText(source, channel, text)
|
||||
if io := s.injector(); io != nil {
|
||||
io.InjectInterruptText(source, channel, text)
|
||||
}
|
||||
}
|
||||
|
||||
// InjectText injects a text message into the agent pipeline.
|
||||
func (s *PluginSDK) InjectText(source, channel, text string) {
|
||||
if s.io != nil {
|
||||
s.io.InjectText(source, channel, text)
|
||||
if io := s.injector(); io != nil {
|
||||
io.InjectText(source, channel, text)
|
||||
}
|
||||
}
|
||||
|
||||
// InjectTextNoMemory injects a text message without generating memory.
|
||||
func (s *PluginSDK) InjectTextNoMemory(source, channel, text string) {
|
||||
if s.io != nil {
|
||||
s.io.InjectTextNoMemory(source, channel, text)
|
||||
if io := s.injector(); io != nil {
|
||||
io.InjectTextNoMemory(source, channel, text)
|
||||
}
|
||||
}
|
||||
|
||||
@ -401,18 +528,62 @@ func (s *PluginSDK) InjectTextNoMemory(source, channel, text string) {
|
||||
// returning the reply text (empty string if none). Replies must be dispatched back
|
||||
// to the source channel by the caller.
|
||||
func (s *PluginSDK) InjectInputSync(source, channel, text string) string {
|
||||
if s.io == nil {
|
||||
io := s.injector()
|
||||
if io == nil {
|
||||
return ""
|
||||
}
|
||||
return s.io.InjectInputSync(source, channel, text)
|
||||
return io.InjectInputSync(source, channel, text)
|
||||
}
|
||||
|
||||
// InjectInputMedia 注入带媒体内容块(image_url/audio_url)的输入。
|
||||
// blocks 会落进媒体存储被记忆引用捕获,同时作为当前轮 content 数组
|
||||
// 发给 LLM,让模型在「本轮」就看到图/听到音频——区别于 SetToolBlocks
|
||||
// 的「下一轮 tool message」语义。
|
||||
func (s *PluginSDK) InjectInputMedia(source, channel, text string, blocks []ContentBlock) {
|
||||
if io := s.injector(); io != nil {
|
||||
io.InjectInputMedia(source, channel, text, blocks)
|
||||
}
|
||||
}
|
||||
|
||||
// InjectInputMediaSync 注入带媒体内容块的输入并同步等待 agent 回复。
|
||||
func (s *PluginSDK) InjectInputMediaSync(source, channel, text string, blocks []ContentBlock) string {
|
||||
io := s.injector()
|
||||
if io == nil {
|
||||
return ""
|
||||
}
|
||||
return io.InjectInputMediaSync(source, channel, text, blocks)
|
||||
}
|
||||
|
||||
// InjectInterruptMedia 注入带媒体内容块的中断,可抢占当前 LLM 处理。
|
||||
// blocks 随中断消息一起发给模型。
|
||||
func (s *PluginSDK) InjectInterruptMedia(source, channel, text string, blocks []ContentBlock) {
|
||||
if io := s.injector(); io != nil {
|
||||
io.InjectInterruptMedia(source, channel, text, blocks)
|
||||
}
|
||||
}
|
||||
|
||||
// SetToolBlocks 在工具处理函数内注入多模态内容块,内核在下一条 tool message
|
||||
// 的 content 数组里带上它们。需要「本轮就让模型看到」时用 InjectInputMedia。
|
||||
func (s *PluginSDK) SetToolBlocks(blocks []ContentBlock) {
|
||||
if io := s.injector(); io != nil {
|
||||
io.SetToolBlocks(blocks)
|
||||
}
|
||||
}
|
||||
|
||||
// SetAutoRestart 设置插件是否允许内核自动重启(崩溃后自动重载)。
|
||||
// 默认 true。如果插件有无法恢复的状态(如外部连接),应设为 false。
|
||||
func (s *PluginSDK) SetAutoRestart(enabled bool) { s.autoRestart = enabled }
|
||||
func (s *PluginSDK) SetAutoRestart(enabled bool) {
|
||||
s.apiMu.Lock()
|
||||
s.autoRestart = enabled
|
||||
s.apiMu.Unlock()
|
||||
}
|
||||
|
||||
// AutoRestart 返回插件是否允许自动重启。
|
||||
func (s *PluginSDK) AutoRestart() bool { return s.autoRestart }
|
||||
func (s *PluginSDK) AutoRestart() bool {
|
||||
s.apiMu.RLock()
|
||||
defer s.apiMu.RUnlock()
|
||||
return s.autoRestart
|
||||
}
|
||||
|
||||
// RegisterStopHandler 注册插件停止阶段的清理回调。
|
||||
// 注册的 handler 会在插件 Stop() 之前按"后注册先执行"的顺序调用,
|
||||
|
||||
725
third_party/homeagent-sdk/sdk/stress_test.go
vendored
Normal file
725
third_party/homeagent-sdk/sdk/stress_test.go
vendored
Normal file
@ -0,0 +1,725 @@
|
||||
package sdk
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// SDK 公开接口的并发压力测试(1.1.0 媒体接口上线后新增)。
|
||||
//
|
||||
// 为什么这一层需要压测:SDK 是**被多个 goroutine 同时使用的共享对象**。
|
||||
// 一个插件的典型形态是 Start() 里起若干后台 goroutine(轮询、监听、定时器),
|
||||
// 它们各自持同一个 *PluginSDK 往里注入消息;内核侧同时还有 stage 扇出、
|
||||
// 工具调用、以及读 AutoRestart() 决定崩溃后是否重启。
|
||||
// 单线程单测全绿不代表这些并发路径成立。
|
||||
//
|
||||
// 关注点不是吞吐数字,而是不变量:
|
||||
// 1. 注入调用不丢、不串(媒体块必须与文本配对,不能张冠李戴)
|
||||
// 2. 状态字段的读写不产生数据竞争(-race 下必须干净)
|
||||
// 3. handler 注册/执行在并发下"恰好一次"
|
||||
// 4. 跨进程 JSON 序列化对新媒体类型必须字节级往返一致
|
||||
//
|
||||
// 媒体接口尤其需要 3 与 4:媒体块要经 JSON 过子进程边界,
|
||||
// 而 []byte 在 JSON 里是 base64,往返不一致的后果是图片静默损坏。
|
||||
|
||||
// ---------- 测试替身 ----------
|
||||
|
||||
// recordingInjector 记录每一次注入调用,用于验证"不丢不串"。
|
||||
type recordingInjector struct {
|
||||
mu sync.Mutex
|
||||
calls []injectCall
|
||||
|
||||
// 计数用原子量:并发路径上只增不减,可在不持锁时安全读。
|
||||
nText, nMedia, nInterrupt, nSync atomic.Int64
|
||||
}
|
||||
|
||||
type injectCall struct {
|
||||
kind string // text / media / interruptMedia / sync ...
|
||||
source string
|
||||
channel string
|
||||
text string
|
||||
blocks []ContentBlock
|
||||
}
|
||||
|
||||
func (r *recordingInjector) record(c injectCall) {
|
||||
r.mu.Lock()
|
||||
r.calls = append(r.calls, c)
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
func (r *recordingInjector) InjectInterruptText(s, c, t string) {
|
||||
r.nInterrupt.Add(1)
|
||||
r.record(injectCall{kind: "interruptText", source: s, channel: c, text: t})
|
||||
}
|
||||
|
||||
func (r *recordingInjector) InjectText(s, c, t string) {
|
||||
r.nText.Add(1)
|
||||
r.record(injectCall{kind: "text", source: s, channel: c, text: t})
|
||||
}
|
||||
|
||||
func (r *recordingInjector) InjectTextNoMemory(s, c, t string) {
|
||||
r.nText.Add(1)
|
||||
r.record(injectCall{kind: "textNoMem", source: s, channel: c, text: t})
|
||||
}
|
||||
|
||||
func (r *recordingInjector) InjectInputSync(s, c, t string) string {
|
||||
r.nSync.Add(1)
|
||||
r.record(injectCall{kind: "sync", source: s, channel: c, text: t})
|
||||
return "reply:" + t
|
||||
}
|
||||
|
||||
func (r *recordingInjector) SetToolBlocks(blocks []ContentBlock) {
|
||||
r.record(injectCall{kind: "toolBlocks", blocks: blocks})
|
||||
}
|
||||
|
||||
func (r *recordingInjector) InjectInputMedia(s, c, t string, b []ContentBlock) {
|
||||
r.nMedia.Add(1)
|
||||
r.record(injectCall{kind: "media", source: s, channel: c, text: t, blocks: b})
|
||||
}
|
||||
|
||||
func (r *recordingInjector) InjectInputMediaSync(s, c, t string, b []ContentBlock) string {
|
||||
r.nMedia.Add(1)
|
||||
r.nSync.Add(1)
|
||||
r.record(injectCall{kind: "mediaSync", source: s, channel: c, text: t, blocks: b})
|
||||
return "reply:" + t
|
||||
}
|
||||
|
||||
func (r *recordingInjector) InjectInterruptMedia(s, c, t string, b []ContentBlock) {
|
||||
r.nMedia.Add(1)
|
||||
r.record(injectCall{kind: "interruptMedia", source: s, channel: c, text: t, blocks: b})
|
||||
}
|
||||
|
||||
func (r *recordingInjector) snapshot() []injectCall {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return append([]injectCall{}, r.calls...)
|
||||
}
|
||||
|
||||
var _ IOInjector = (*recordingInjector)(nil)
|
||||
|
||||
// imageBlock 构造一个带可识别 URL 的图片块。
|
||||
func imageBlock(tag string) ContentBlock {
|
||||
return ContentBlock{
|
||||
Type: "image_url",
|
||||
ImageURL: &ImageURL{URL: "data:image/png;base64," + tag, Detail: "auto"},
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 1. 媒体注入并发不丢不串 ----------
|
||||
|
||||
// 三个媒体注入方法在高并发下必须:调用数精确、且每次调用的 text 与 blocks 配对不错。
|
||||
//
|
||||
// "不串"是这里的关键断言。注入是插件里最容易被后台 goroutine 并发调用的入口,
|
||||
// 若实现里出现任何共享中间状态(比如把 blocks 暂存到 SDK 字段再读出),
|
||||
// 高并发下就会出现 A 的文本配上 B 的图——而两者单独看都"成功"了,不报错。
|
||||
func TestStress_MediaInjectionConcurrentNoCrossTalk(t *testing.T) {
|
||||
const workers, perWorker = 32, 200
|
||||
|
||||
inj := &recordingInjector{}
|
||||
s := &PluginSDK{name: "stress"}
|
||||
s.SetIOInjector(inj)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func(w int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < perWorker; i++ {
|
||||
// tag 唯一标识这次调用,文本与图片 URL 里都带上它。
|
||||
tag := fmt.Sprintf("w%d-i%d", w, i)
|
||||
switch i % 3 {
|
||||
case 0:
|
||||
s.InjectInputMedia("src", "ch", tag, []ContentBlock{imageBlock(tag)})
|
||||
case 1:
|
||||
if got := s.InjectInputMediaSync("src", "ch", tag, []ContentBlock{imageBlock(tag)}); got != "reply:"+tag {
|
||||
t.Errorf("同步注入回复错位: got %q want %q", got, "reply:"+tag)
|
||||
}
|
||||
default:
|
||||
s.InjectInterruptMedia("src", "ch", tag, []ContentBlock{imageBlock(tag)})
|
||||
}
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
total := int64(workers * perWorker)
|
||||
if got := inj.nMedia.Load(); got != total {
|
||||
t.Fatalf("媒体注入调用数 = %d,期望 %d(有调用丢失)", got, total)
|
||||
}
|
||||
|
||||
// 逐条校验文本与媒体块配对:URL 必须含该次调用自己的 tag。
|
||||
seen := map[string]bool{}
|
||||
for _, c := range inj.snapshot() {
|
||||
if len(c.blocks) == 0 {
|
||||
continue
|
||||
}
|
||||
if c.blocks[0].ImageURL == nil {
|
||||
t.Fatalf("媒体块 ImageURL 丢失: %+v", c.blocks[0])
|
||||
}
|
||||
if !strings.HasSuffix(c.blocks[0].ImageURL.URL, c.text) {
|
||||
t.Fatalf("文本与媒体块错位: text=%q url=%q", c.text, c.blocks[0].ImageURL.URL)
|
||||
}
|
||||
if seen[c.text] {
|
||||
t.Fatalf("同一次调用被记录两次: %s", c.text)
|
||||
}
|
||||
seen[c.text] = true
|
||||
}
|
||||
if len(seen) != int(total) {
|
||||
t.Fatalf("去重后调用数 = %d,期望 %d", len(seen), total)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 2. 注入期间热替换 injector ----------
|
||||
|
||||
// 内核在插件运行期间可能重新注入 API(重载、恢复、子进程重连握手)。
|
||||
// 此时插件的后台 goroutine 仍在注入。这条路径若无同步就是对 s.io 的数据竞争,
|
||||
// 在 -race 下会被抓出;生产表现是偶发 nil 解引用崩溃。
|
||||
func TestStress_InjectorSwapDuringInjection(t *testing.T) {
|
||||
s := &PluginSDK{name: "stress"}
|
||||
s.SetIOInjector(&recordingInjector{})
|
||||
|
||||
stop := make(chan struct{})
|
||||
var injectors, swapper sync.WaitGroup
|
||||
|
||||
// 注入方:持续打直到 stop
|
||||
for w := 0; w < 8; w++ {
|
||||
injectors.Add(1)
|
||||
go func() {
|
||||
defer injectors.Done()
|
||||
for {
|
||||
select {
|
||||
case <-stop:
|
||||
return
|
||||
default:
|
||||
s.InjectInputMedia("src", "ch", "x", []ContentBlock{imageBlock("x")})
|
||||
s.InjectText("src", "ch", "y")
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// 替换方:反复换 injector(含换成 nil——内核卸载 API 时的真实状态)
|
||||
swapper.Add(1)
|
||||
go func() {
|
||||
defer swapper.Done()
|
||||
for i := 0; i < 500; i++ {
|
||||
if i%7 == 0 {
|
||||
s.SetIOInjector(nil)
|
||||
} else {
|
||||
s.SetIOInjector(&recordingInjector{})
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// 先等替换跑完,再告知注入方退出。
|
||||
// 顺序写反了就是死锁:注入方只依 close(stop) 退出。
|
||||
swapper.Wait()
|
||||
close(stop)
|
||||
injectors.Wait()
|
||||
// 断言就是「没崩、-race 没报」。nil injector 时必须静默跳过而非 panic。
|
||||
}
|
||||
|
||||
// ---------- 3. autoRestart 标志的并发读写 ----------
|
||||
|
||||
// SetAutoRestart 的文档用途是"插件有无法恢复的状态(如外部连接)时设为 false"——
|
||||
// 而连接建立本身通常是异步的,所以这个写入天然发生在后台 goroutine。
|
||||
// 内核侧 registry 在另一个 goroutine 读 AutoRestart() 决定崩溃后是否重启。
|
||||
// 这是一对跨 goroutine 的读写,必须同步。
|
||||
func TestStress_AutoRestartFlagConcurrent(t *testing.T) {
|
||||
s := &PluginSDK{name: "stress", autoRestart: true}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < 16; w++ {
|
||||
wg.Add(1)
|
||||
go func(w int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < 500; i++ {
|
||||
s.SetAutoRestart(i%2 == 0)
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
// 读方模拟内核 registry
|
||||
for r := 0; r < 8; r++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for i := 0; i < 500; i++ {
|
||||
_ = s.AutoRestart()
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// ---------- 4. stop / onRemove handler 的"恰好一次" ----------
|
||||
|
||||
// RunStopHandlers 的契约是"执行后清空,幂等"。内核在停止插件时可能并发触发
|
||||
// (超时强杀与正常 Stop 竞争),handler 里往往是关连接、落盘——
|
||||
// 执行两次的后果从"重复写文件"到"close 已关闭的 channel 直接 panic"。
|
||||
func TestStress_StopHandlersExactlyOnce(t *testing.T) {
|
||||
const n = 300
|
||||
s := &PluginSDK{name: "stress"}
|
||||
|
||||
var counters [n]atomic.Int64
|
||||
for i := 0; i < n; i++ {
|
||||
i := i
|
||||
s.RegisterStopHandler(func() { counters[i].Add(1) })
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < 16; w++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
s.RunStopHandlers()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
for i := 0; i < n; i++ {
|
||||
if got := counters[i].Load(); got != 1 {
|
||||
t.Fatalf("handler %d 执行 %d 次,期望恰好 1 次", i, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 注册与执行并发:已注册的 handler 一次都不能多跑,未跑到的也不能被丢。
|
||||
// 断言用"每个 handler 的执行次数 <= 1"而非"总数相等"——
|
||||
// 与 RunStopHandlers 竞争的注册可能落在快照之后,那属于合法的未执行。
|
||||
func TestStress_StopHandlersRegisterWhileRunning(t *testing.T) {
|
||||
s := &PluginSDK{name: "stress"}
|
||||
const n = 500
|
||||
var counters [n]atomic.Int64
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for i := 0; i < n; i++ {
|
||||
i := i
|
||||
s.RegisterStopHandler(func() { counters[i].Add(1) })
|
||||
}
|
||||
}()
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for i := 0; i < 50; i++ {
|
||||
s.RunStopHandlers()
|
||||
}
|
||||
}()
|
||||
wg.Wait()
|
||||
s.RunStopHandlers() // 收尾:把剩下的都跑掉
|
||||
|
||||
for i := 0; i < n; i++ {
|
||||
if got := counters[i].Load(); got > 1 {
|
||||
t.Fatalf("handler %d 被执行 %d 次(重复执行)", i, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStress_OnRemoveHandlersExactlyOnce(t *testing.T) {
|
||||
const n = 200
|
||||
s := &PluginSDK{name: "stress"}
|
||||
|
||||
var counters [n]atomic.Int64
|
||||
for i := 0; i < n; i++ {
|
||||
i := i
|
||||
s.RegisterOnRemoveHandler(func() { counters[i].Add(1) })
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < 12; w++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
s.RunOnRemoveHandlers()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
for i := 0; i < n; i++ {
|
||||
if got := counters[i].Load(); got != 1 {
|
||||
t.Fatalf("onRemove handler %d 执行 %d 次,期望恰好 1 次", i, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 5. StageContext 并发读改写 ----------
|
||||
|
||||
// StageContext 是全部 stage handler 共享的可变状态,字段全导出、靠调用方自觉
|
||||
// 持 Lock/RLock。媒体链路让 Extra 成为新热点(media_blocks 挂在这里),
|
||||
// 而 map 的并发写在 Go 里是直接 fatal,recover 都接不住。
|
||||
//
|
||||
// 这条测试锁定的不变量:按约定持锁的并发读改写不丢更新、不 fatal。
|
||||
func TestStress_StageContextConcurrentExtraAndFinalText(t *testing.T) {
|
||||
ctx := &StageContext{Extra: map[string]interface{}{}}
|
||||
|
||||
const workers, rounds = 16, 200
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func(w int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < rounds; i++ {
|
||||
// 写:模拟插件往 Extra 塞媒体块并追加文本(读-改-写)
|
||||
ctx.Lock()
|
||||
ctx.Extra[fmt.Sprintf("k%d-%d", w, i)] = []ContentBlock{imageBlock("x")}
|
||||
ctx.FinalText += "."
|
||||
ctx.Unlock()
|
||||
|
||||
// 读:模拟另一个 handler 检查是否已被响应
|
||||
_ = ctx.IsResponded()
|
||||
ctx.RLock()
|
||||
_ = len(ctx.Extra)
|
||||
ctx.RUnlock()
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
ctx.RLock()
|
||||
defer ctx.RUnlock()
|
||||
if len(ctx.Extra) != workers*rounds {
|
||||
t.Fatalf("Extra 键数 = %d,期望 %d(出现 lost update)", len(ctx.Extra), workers*rounds)
|
||||
}
|
||||
if len(ctx.FinalText) != workers*rounds {
|
||||
t.Fatalf("FinalText 长度 = %d,期望 %d(出现 lost update)", len(ctx.FinalText), workers*rounds)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 6. OwnTools scope 包装器的并发正确性 ----------
|
||||
|
||||
// StageScopeOwnTools 的包装闭环里要读 ctx.ToolCalls 判断归属。
|
||||
// 并发下若判断与执行之间状态被改写,就会出现"别人的工具触发了我的 handler"——
|
||||
// 后果是插件对不属于自己的工具结果动手,且没有任何错误。
|
||||
func TestStress_OwnToolsScopeNoCrossPluginLeak(t *testing.T) {
|
||||
var registered StageHandler
|
||||
s := &PluginSDK{
|
||||
name: "mine",
|
||||
regStage: func(stage Stage, h StageHandler) { registered = h },
|
||||
}
|
||||
|
||||
var fired atomic.Int64
|
||||
s.RegisterStage(StageBeforeToolcall, func(ctx *StageContext) error {
|
||||
fired.Add(1)
|
||||
ctx.RLock()
|
||||
defer ctx.RUnlock()
|
||||
// 触发了就必须确实是自己的工具
|
||||
if len(ctx.ToolCalls) == 0 || ctx.ToolCalls[0].Plugin != "mine" {
|
||||
t.Errorf("handler 被别的插件的工具触发: %+v", ctx.ToolCalls)
|
||||
}
|
||||
return nil
|
||||
}, StageScopeOwnTools)
|
||||
|
||||
if registered == nil {
|
||||
t.Fatal("handler 未注册")
|
||||
}
|
||||
|
||||
const workers, rounds = 16, 100
|
||||
var wg sync.WaitGroup
|
||||
var mineCount atomic.Int64
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func(w int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < rounds; i++ {
|
||||
// 每个 goroutine 用自己的 ctx——真实内核里 stage 扇出共享同一个
|
||||
// ctx,但那部分的并发由内核 host 仲裁;这里验证包装器本身。
|
||||
owner := "other"
|
||||
if (w+i)%2 == 0 {
|
||||
owner = "mine"
|
||||
mineCount.Add(1)
|
||||
}
|
||||
ctx := &StageContext{Extra: map[string]interface{}{}}
|
||||
ctx.ToolCalls = []ToolCall{{Plugin: owner, Name: "t"}}
|
||||
if err := registered(ctx); err != nil {
|
||||
t.Errorf("handler 返回错误: %v", err)
|
||||
}
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
if got, want := fired.Load(), mineCount.Load(); got != want {
|
||||
t.Fatalf("handler 触发 %d 次,期望 %d 次(漏触发或跨插件触发)", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 7. 媒体类型的 JSON 往返(跨进程边界的真实形态) ----------
|
||||
|
||||
// 媒体块与附件要经 JSON 过子进程边界。[]byte 在 JSON 里是 base64,
|
||||
// 往返不一致的后果是图片字节静默损坏——落进 CAS 后 digest 校验才会发现,
|
||||
// 而那时已经无从追查是谁改坏的。
|
||||
func TestStress_MediaTypesJSONRoundTripAtScale(t *testing.T) {
|
||||
// 覆盖真实会遇到的边界:空、单字节、含 0x00、全 0xFF、超过 base64 分组边界的长度
|
||||
sizes := []int{0, 1, 2, 3, 255, 256, 1023, 4096, 65537}
|
||||
for _, n := range sizes {
|
||||
data := make([]byte, n)
|
||||
for i := range data {
|
||||
data[i] = byte(i * 7 % 256)
|
||||
}
|
||||
att := MediaAttachment{
|
||||
Digest: strings.Repeat("a", 64),
|
||||
MIME: "image/png",
|
||||
Data: data,
|
||||
Name: "图片-名字 with space & 符号.png",
|
||||
Description: "一张紫蓝红三色带图,含 emoji 🎨 与换行\n第二行",
|
||||
}
|
||||
b, err := json.Marshal(att)
|
||||
if err != nil {
|
||||
t.Fatalf("size=%d marshal: %v", n, err)
|
||||
}
|
||||
var back MediaAttachment
|
||||
if err := json.Unmarshal(b, &back); err != nil {
|
||||
t.Fatalf("size=%d unmarshal: %v", n, err)
|
||||
}
|
||||
if len(back.Data) != n {
|
||||
t.Fatalf("size=%d 往返后长度 = %d", n, len(back.Data))
|
||||
}
|
||||
for i := range data {
|
||||
if back.Data[i] != data[i] {
|
||||
t.Fatalf("size=%d 第 %d 字节损坏: %02x != %02x", n, i, back.Data[i], data[i])
|
||||
}
|
||||
}
|
||||
if back.Name != att.Name || back.Description != att.Description || back.MIME != att.MIME || back.Digest != att.Digest {
|
||||
t.Fatalf("size=%d 元数据往返不一致: %+v", n, back)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// omitempty 必须真的生效:读路径上内核不回 Data,若序列化仍产出 "data":null
|
||||
// 之类的键,跨进程消息会凭空变大,且插件侧无法区分"没有字节"与"空字节"。
|
||||
func TestStress_MediaTypesOmitEmpty(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
v interface{}
|
||||
absent []string
|
||||
present []string
|
||||
}{
|
||||
{
|
||||
name: "Triple 无媒体",
|
||||
v: Triple{Subject: "甲方", Relation: "签署", Object: "合同"},
|
||||
absent: []string{"media_digests", "sentence_text", "confidence", "subject_type", "object_type"},
|
||||
present: []string{"subject", "relation", "object"},
|
||||
},
|
||||
{
|
||||
name: "Triple 带媒体",
|
||||
v: Triple{Subject: "甲方", Relation: "包含", Object: "图", MediaDigests: []string{"abc12345"}, SentenceText: "句子"},
|
||||
absent: []string{"confidence"},
|
||||
present: []string{"media_digests", "sentence_text"},
|
||||
},
|
||||
{
|
||||
name: "Doc 读路径无字节",
|
||||
v: Doc{ID: "d1", Title: "标题", Content: "正文", Attachments: []MediaAttachment{{Digest: "abc12345", MIME: "image/png"}}},
|
||||
absent: []string{"\"data\"", "media_digests", "score"},
|
||||
present: []string{"attachments", "digest", "mime"},
|
||||
},
|
||||
{
|
||||
name: "TextEvent 无附件",
|
||||
v: TextEvent{Role: "user", Content: "hi"},
|
||||
absent: []string{"attachments", "channel"},
|
||||
present: []string{"role", "content"},
|
||||
},
|
||||
{
|
||||
name: "ContentBlock 纯文本",
|
||||
v: ContentBlock{Type: "text", Text: "hi"},
|
||||
absent: []string{"image_url", "audio_url"},
|
||||
present: []string{"type", "text"},
|
||||
},
|
||||
{
|
||||
name: "ContentBlock 图片",
|
||||
v: imageBlock("AAA"),
|
||||
absent: []string{"audio_url", "\"text\""},
|
||||
present: []string{"image_url", "detail"},
|
||||
},
|
||||
}
|
||||
for _, c := range cases {
|
||||
b, err := json.Marshal(c.v)
|
||||
if err != nil {
|
||||
t.Fatalf("%s marshal: %v", c.name, err)
|
||||
}
|
||||
s := string(b)
|
||||
for _, k := range c.absent {
|
||||
if strings.Contains(s, k) {
|
||||
t.Errorf("%s: 不该出现的键 %s —— %s", c.name, k, s)
|
||||
}
|
||||
}
|
||||
for _, k := range c.present {
|
||||
if !strings.Contains(s, k) {
|
||||
t.Errorf("%s: 缺少键 %s —— %s", c.name, k, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 媒体块在并发序列化下必须各自独立:ImageURL/AudioURL 是指针,
|
||||
// 若某处复用同一个指针再改写,序列化结果会互相污染。
|
||||
func TestStress_ContentBlockConcurrentMarshal(t *testing.T) {
|
||||
const workers, rounds = 16, 300
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func(w int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < rounds; i++ {
|
||||
tag := fmt.Sprintf("w%d-i%d", w, i)
|
||||
blocks := []ContentBlock{
|
||||
{Type: "text", Text: tag},
|
||||
imageBlock(tag),
|
||||
{Type: "audio_url", AudioURL: &AudioURL{URL: "data:audio/wav;base64," + tag}},
|
||||
}
|
||||
b, err := json.Marshal(blocks)
|
||||
if err != nil {
|
||||
t.Errorf("marshal: %v", err)
|
||||
return
|
||||
}
|
||||
var back []ContentBlock
|
||||
if err := json.Unmarshal(b, &back); err != nil {
|
||||
t.Errorf("unmarshal: %v", err)
|
||||
return
|
||||
}
|
||||
if len(back) != 3 {
|
||||
t.Errorf("块数 = %d", len(back))
|
||||
return
|
||||
}
|
||||
if back[0].ImageURL != nil || back[0].AudioURL != nil {
|
||||
t.Errorf("文本块被填了媒体指针: %+v", back[0])
|
||||
}
|
||||
if back[1].ImageURL == nil || !strings.HasSuffix(back[1].ImageURL.URL, tag) {
|
||||
t.Errorf("图片块 URL 错位: %+v", back[1].ImageURL)
|
||||
}
|
||||
if back[1].AudioURL != nil {
|
||||
t.Errorf("图片块被填了音频指针")
|
||||
}
|
||||
if back[2].AudioURL == nil || !strings.HasSuffix(back[2].AudioURL.URL, tag) {
|
||||
t.Errorf("音频块 URL 错位: %+v", back[2].AudioURL)
|
||||
}
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// ---------- 8. 注册面的并发 ----------
|
||||
|
||||
// 插件在 Start() 里起多个 goroutine 分别注册工具是常见写法。
|
||||
// def.Plugin 的默认填充若不是每次调用独立的,就会出现工具归属错乱——
|
||||
// 表现是 OwnTools scope 失效、WebUI 里工具挂在别的插件名下。
|
||||
func TestStress_RegisterToolConcurrentPluginDefaulting(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
got := map[string]string{} // toolName -> def.Plugin
|
||||
|
||||
s := &PluginSDK{
|
||||
name: "mine",
|
||||
regTool: func(name string, def ToolDef, h ToolHandler) error {
|
||||
mu.Lock()
|
||||
got[name] = def.Plugin
|
||||
mu.Unlock()
|
||||
return nil
|
||||
},
|
||||
}
|
||||
|
||||
const workers, perWorker = 16, 100
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func(w int) {
|
||||
defer wg.Done()
|
||||
for i := 0; i < perWorker; i++ {
|
||||
name := fmt.Sprintf("tool_w%d_i%d", w, i)
|
||||
def := ToolDef{Description: "d", Parameters: map[string]interface{}{}}
|
||||
// 一半显式指定归属,一半靠 SDK 填默认值
|
||||
if i%2 == 0 {
|
||||
def.Plugin = "explicit"
|
||||
}
|
||||
if err := s.RegisterTool(name, def, func(map[string]interface{}) (interface{}, error) {
|
||||
return nil, nil
|
||||
}); err != nil {
|
||||
t.Errorf("RegisterTool: %v", err)
|
||||
}
|
||||
}
|
||||
}(w)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
if len(got) != workers*perWorker {
|
||||
t.Fatalf("注册工具数 = %d,期望 %d", len(got), workers*perWorker)
|
||||
}
|
||||
for name, owner := range got {
|
||||
want := "mine"
|
||||
if isEvenSuffix(name) {
|
||||
want = "explicit"
|
||||
}
|
||||
if owner != want {
|
||||
t.Fatalf("工具 %s 归属 = %q,期望 %q", name, owner, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// isEvenSuffix 判断 tool_wX_iY 里的 Y 是否为偶数。
|
||||
func isEvenSuffix(name string) bool {
|
||||
idx := strings.LastIndex(name, "_i")
|
||||
if idx < 0 {
|
||||
return false
|
||||
}
|
||||
n := 0
|
||||
if _, err := fmt.Sscanf(name[idx+2:], "%d", &n); err != nil {
|
||||
return false
|
||||
}
|
||||
return n%2 == 0
|
||||
}
|
||||
|
||||
// nil 依赖下所有便捷方法必须静默降级而非 panic。
|
||||
//
|
||||
// 这是"媒体存储可关闭"在 SDK 层的对应物:内核未注入某个 API 时
|
||||
// (精简部署、插件权限不足、子进程握手尚未完成),插件的调用不该崩。
|
||||
func TestStress_NilDependenciesDegradeSilently(t *testing.T) {
|
||||
s := &PluginSDK{name: "bare"}
|
||||
|
||||
const workers = 16
|
||||
var wg sync.WaitGroup
|
||||
for w := 0; w < workers; w++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for i := 0; i < 200; i++ {
|
||||
s.InjectText("s", "c", "t")
|
||||
s.InjectTextNoMemory("s", "c", "t")
|
||||
s.InjectInterruptText("s", "c", "t")
|
||||
if got := s.InjectInputSync("s", "c", "t"); got != "" {
|
||||
t.Errorf("无 injector 时同步注入应返回空串,got %q", got)
|
||||
}
|
||||
s.InjectInputMedia("s", "c", "t", []ContentBlock{imageBlock("x")})
|
||||
if got := s.InjectInputMediaSync("s", "c", "t", nil); got != "" {
|
||||
t.Errorf("无 injector 时媒体同步注入应返回空串,got %q", got)
|
||||
}
|
||||
s.InjectInterruptMedia("s", "c", "t", nil)
|
||||
|
||||
// getter 全部应返回 nil 而非 panic
|
||||
_ = s.Memory()
|
||||
_ = s.TextMemory()
|
||||
_ = s.DocMemory()
|
||||
_ = s.Knowledge()
|
||||
_ = s.LLM()
|
||||
_ = s.Social()
|
||||
_ = s.Events()
|
||||
_ = s.PluginMgr()
|
||||
_ = s.Settings()
|
||||
|
||||
// 注册面无 registrar 时应返回 nil error
|
||||
if err := s.RegisterTool("t", ToolDef{}, nil); err != nil {
|
||||
t.Errorf("无 registrar 时 RegisterTool 应返回 nil,got %v", err)
|
||||
}
|
||||
if err := s.RegisterPluginAPI("a"); err != nil {
|
||||
t.Errorf("无 registrar 时 RegisterPluginAPI 应返回 nil,got %v", err)
|
||||
}
|
||||
s.RegisterStage(StageOnInput, func(*StageContext) error { return nil })
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
Reference in New Issue
Block a user