mirror of
https://gitcode.com/JianFeeeee/HomeAgent.git
synced 2026-10-15 12:15:16 +00:00
chore: 根目录清理(第二轮)—— 移除过程文档、本机脚本、临时产物 (#3)
★ 上一轮清理被合并覆盖:GUI 分支含 DESKTOP-PET-NOTES.md /
SERVER-HANDOFF.md / GUI-PLAN.md,合并时取了我方版本,
但我方 main 已是 squash 产物 ⇒ 三个文件复活。
而我当时只核对了 benchmark.md 一个就宣布成功 —— 判断依据不足。
本轮逐个文件验证(不再抽样)。
删除(零代码引用)
DESKTOP-PET-NOTES.md 外部产品(Coopanion)拆解笔记
SERVER-HANDOFF.md 给服务端的待办清单,已过时
napcat-design-dna.json 设计 DNA 数据
tmp_jinabench.py 临时脚本
tmp_benchmark/ 临时跑分程序
归位(不是删)
plan.md → docs/zh/legacy-issues.md (遗留问题清单,有价值)
GUI-PLAN.md → cmd/gui/GUI-PLAN.md (含坐标换算/koffi 踩坑)
根目录那份与 cmd/gui/ 已有副本字节相同(285b84dd…),删前者
deploy-plan.sh → deploy/scripts/
deploy-waiter.sh → deploy/scripts/
deploy-sdk-site.sh → deploy/scripts/
脱敏(deploy-plan.sh 与 deploy-sdk-site.sh 含本机路径)
/home/newqqagent → ${HA_DATA}
/home/program/TrueAgent → ${REPO}
/home/program → ${HOME_DIR}
引用同步:27 处(docs/zh/deploy-runbook.md、toolcall-parallel-execution-plan.md、
scripts/capability-bench/PLAN.md、三个脚本自身的用法注释、ci.yml 注释)
根目录现状:仅 README.md / README_EN.md / LICENSE / Makefile /
go.mod / go.sum / .gitignore / .golangci.yml / .mailmap / .pi-lens.json
验证:go build ✅ 三个 sh 的 bash -n ✅
go test -short ./internal/... ./pkg/... ./cmd/... 全绿
根目录文件清单逐个核对(10 项,全为门面/配置)
Co-authored-by: pi <pi@local>
This commit is contained in:
2
.github/workflows/ci.yml
vendored
2
.github/workflows/ci.yml
vendored
@ -193,7 +193,7 @@ jobs:
|
||||
python-version: '3.12'
|
||||
- name: 校验站点配置可解析
|
||||
# 这里只做「配置与文档源没坏」的轻量校验,不做完整 mkdocs build
|
||||
# (站点发布有独立流水线,见 deploy-sdk-site.sh)。
|
||||
# (站点发布有独立流水线,见 deploy/scripts/deploy-sdk-site.sh)。
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -f mkdocs.yml ]; then
|
||||
|
||||
@ -1,229 +0,0 @@
|
||||
# 桌宠动作系统:Coopanion 拆解与可借鉴点
|
||||
|
||||
> 来源:`Pal-AI-Lab/Coopanion`(本机 clone 于 `/tmp/coop`)
|
||||
> 相关包:`packages/cortico-world-desktop-pet/`
|
||||
>
|
||||
> 本文只记录**桌宠的动作/表情/互动系统**,以及本仓(HomeAgent GUI)可借鉴什么。
|
||||
> 输入注入层(SendInput)的对比见 `GUI-PLAN.md` 与下文第七节。
|
||||
|
||||
---
|
||||
|
||||
## 一、它是什么
|
||||
|
||||
一个 C 形小生物,站在屏幕底边(任务栏上沿)。**用户看得见它的身体,看不见 agent 的消息正文** ——
|
||||
想让用户看到内容,只能通过它冒气泡。
|
||||
|
||||
这句话是整个设计的关键:
|
||||
|
||||
> 桌宠身体是 agent 的**唯一可见输出通道**,而不是装饰。
|
||||
|
||||
这和我们 GUI 的定位不同:我们的 mascot 只是「agent 光标」,不承载信息。这是一个可以借的点。
|
||||
|
||||
---
|
||||
|
||||
## 二、核心工具(4 个)
|
||||
|
||||
| 工具 | 作用 | 关键设计 |
|
||||
|---|---|---|
|
||||
| `pet_say` | 头顶冒气泡说话 | 脚本里可夹带**动作标记**,边说边做 |
|
||||
| `pet_ask` | 冒提问气泡 + 最多 3 个选项 | **立即返回**,回答通过 `[回答]` 事件异步送达 |
|
||||
| `pet_walk_to` | 沿屏幕底边走到某处 | 目标是 0–1 比例或 `left/center/right/cursor` |
|
||||
| `pet_act` | 不说话,只做一串动作 | `sit`/`sleep` 会**持续**到下个动作 |
|
||||
|
||||
### 值得注意的三个设计
|
||||
|
||||
1. **`pet_ask` 是异步的**。agent 提完问不阻塞,用户答了再通过事件唤醒。
|
||||
避免 agent 干等,也允许用户先不答。
|
||||
实用细节:新问题会**替换**还没回答的旧问题;关掉不答也会回一个事件,
|
||||
agent 不会永远等下去。
|
||||
|
||||
2. **`pet_walk_to` 支持 `cursor`** —— 走到鼠标当前横向位置。
|
||||
这是个很自然的"到我这儿来"表达,不需要坐标。
|
||||
|
||||
3. **`sit`/`sleep` 是持续态**,其他动作是一次性。
|
||||
这让 agent 能表达"待机"与"我在休息",而不只是瞬间动画。
|
||||
|
||||
---
|
||||
|
||||
## 三、动作词表(`src/script.ts` 的 `VOCAB`)
|
||||
|
||||
它把动作分成两类:**表情**(持续几秒后回到平常脸)与**动作**。
|
||||
|
||||
### 表情(expression,10 个)
|
||||
|
||||
| id | 中文 | 表现 |
|
||||
|---|---|---|
|
||||
| `neutral` | 平静 | 默认的脸 |
|
||||
| `happy` | 开心 / 高兴 | 眼睛弯成 ^ ^ |
|
||||
| `wink` | 眨眼 | 一只眼 ^ |
|
||||
| `love` | 喜欢 / 爱心 | 眼睛变心形,冒小心心 |
|
||||
| `shy` | 害羞 | 脸红,眼神躲开 |
|
||||
| `surprised` | 惊讶 / 吃惊 | 眼睛放大,头顶感叹号 |
|
||||
| `angry` | 生气 | 皱眉,头顶怒气符号,身体发抖 |
|
||||
| `sad` | 难过 / 伤心 | 八字眉,掉眼泪 |
|
||||
| `sleepy` | 犯困 / 困 | 眯眼打哈欠 |
|
||||
| `thinking` | 思考 / 想想 | 眼睛往上看,头顶冒圈 |
|
||||
|
||||
### 动作(motion,13 个)
|
||||
|
||||
| id | 中文 | 表现 |
|
||||
|---|---|---|
|
||||
| `stand` | 站起 / 站 | 站起来(坐着、睡着时) |
|
||||
| `jump` | 跳 / 跳起来 | 原地起跳 |
|
||||
| `hop` | 小跳 / 蹦 | 小小蹦一下 |
|
||||
| `look` | 张望 / 看看 | 左右张望 |
|
||||
| `turn` | 转身 | 转向另一边 |
|
||||
| `nod` | 点头 | 点两下头 |
|
||||
| `shake` | 摇头 | 摇头 |
|
||||
| `spin` | 转圈 | 原地转一圈 |
|
||||
| `sit` | 坐下 / 坐 | **坐下,持续到下个动作** |
|
||||
| `sleep` | 睡觉 / 睡 | **躺下睡觉,持续到下个动作** |
|
||||
| `dizzy` | 晕 / 转晕 | 头晕眼花几秒 |
|
||||
| `walk` | 走走 / 散步 | 随便走一段 |
|
||||
| `run` | 跑 / 跑起来 | 跑到屏幕另一头 |
|
||||
|
||||
### 词表设计的可借鉴处
|
||||
|
||||
- **每个词有多个中文别名**(`开心`/`高兴` 同一个 id)。LLM 输出自然语言时容错率更高。
|
||||
- **英文 id 与中文别名都进同一张 Map**(`BY_WORD`),不区分来源。
|
||||
- **未知词被丢弃而非报错**(`dropped` 数组收集但不中断)。
|
||||
对 LLM 输出的容错策略,值得学。
|
||||
- 词表通过 `pet.vocab` 变量**注入到 prompt**,让模型知道能做什么 ——
|
||||
而不是把 30 个动作硬编码进 tool schema 的 enum。
|
||||
|
||||
---
|
||||
|
||||
## 四、脚本标记(说明与动作混排)
|
||||
|
||||
`pet_say` 的 `script` 参数里可以夹带两种标记:
|
||||
|
||||
| 标记 | 作用 |
|
||||
|---|---|
|
||||
| `【词, 词】` | 先做这些动作,然后**换一个新气泡**继续说 |
|
||||
| `<词>` | 打字打到这里时做,**不换气泡** |
|
||||
|
||||
示例(源码里的原例):
|
||||
|
||||
```
|
||||
【开心】今天也辛苦啦!<眨眼>要不要歇一会儿?
|
||||
```
|
||||
|
||||
这个设计解决了一个真实问题:**让文字与动作在时间轴上对齐**。
|
||||
`<眨眼>` 落在句子中间,让"眨眼"发生在念到那个字的时候,而不是整句之前。
|
||||
比"先做动作再说话"生动得多,实现成本却很低(解析时切分即可)。
|
||||
|
||||
相关实现:`parseScript` / `parseActions` / `estimateSeconds`(`src/script.ts`),
|
||||
`estimateSeconds` 用于让回执报告"大约显示多久"。
|
||||
|
||||
---
|
||||
|
||||
## 五、自主行为与互动
|
||||
|
||||
### 待机(没有指令时)
|
||||
|
||||
> 没收到指令时身体会自己眨眼、张望、散步、坐下打盹;你的指令会先执行。
|
||||
|
||||
注意最后半句:**agent 指令优先于自主行为**。自主行为是填充,不是竞争。
|
||||
|
||||
### 用户互动(事件形式回给 agent)
|
||||
|
||||
| 事件 | 含义 |
|
||||
|---|---|
|
||||
| `[语音] 用户:…` | 麦克风听到并转写的话(可能同音错字) |
|
||||
| `[打字] 用户:…` | 在桌宠上双击或点悬停按钮打的字 |
|
||||
| `[回答] …` | 对 `pet_ask` 的回答 |
|
||||
| `[互动] …` | 戳、摸、拎起来甩出去,以及落地摔晕 |
|
||||
|
||||
细节值得注意:
|
||||
|
||||
- 互动事件**不单独唤醒 agent**,"跟着下一批事件到,可能已经过去一会儿"
|
||||
—— 避免用户随手一戳就把模型叫起来烧钱。
|
||||
- `asleep` 状态下被摸,文案不同(`摸了摸睡着的你`)。
|
||||
状态影响事件语义,而不只是影响动画。
|
||||
|
||||
---
|
||||
|
||||
## 六、对本仓(HomeAgent GUI)的可借鉴点
|
||||
|
||||
### 6.1 可以借鉴(成本低、收益明确)
|
||||
|
||||
| 点 | 现状 | 建议 |
|
||||
|---|---|---|
|
||||
| **动作与文字时间轴对齐**(`【】`/`<>`) | 我们的 mascot 只有移动/点击两种状态 | 若要表达"思考/等待/出错",可引入轻量标记 |
|
||||
| **姿态表达状态** | 「忙碌呼吸」一种 | 可区分:思考中(转圈)、等待用户(张望)、出错(晕) |
|
||||
| **持续态 vs 一次性**(`sit`/`sleep`) | 无持续态 | agent 空闲/被禁用时用持续态表达,而非静止 |
|
||||
| **LLM 输出容错**(未知词丢弃不报错) | 未处理 | 动作解析应容错,不能因一个错词整条失败 |
|
||||
| **待机自主行为** | 无 | 光标闲置时可轻微眨眼/呼吸,让"活着"可感知 |
|
||||
|
||||
### 6.2 不建议照搬
|
||||
|
||||
| 点 | 原因 |
|
||||
|---|---|
|
||||
| **桌宠承载全部对话输出** | 我们是桌面客户端,有完整聊天界面,不该退化到只能冒气泡 |
|
||||
| **语音输入(ASR)** | 与本期目标无关,且引入麦克风权限问题 |
|
||||
| **戳/摸/甩出去** | 需要模型 + 物理引擎,投入产出比低 |
|
||||
| **`pet.vocab` 注入 prompt** | 我们没把 mascot 交给 LLM 控,动作由 computeruse 隐式驱动 |
|
||||
|
||||
### 6.3 一个更重要的判断
|
||||
|
||||
**Coopanion 的桌宠和我们的"独立光标"解决的是不同问题:**
|
||||
|
||||
- 它的桌宠 = **输出通道**(让用户看见 agent 想说什么)
|
||||
- 我们的 mascot 光标 = **过程指示**(让用户看见 agent 在操作哪里)
|
||||
|
||||
两者不冲突,甚至可以合并 —— 让我们的 mascot 在 agent 思考时表现"思考"姿态,
|
||||
在执行 computeruse 时表现"移动/点击"。这正是 6.1 表格里那几条的价值所在。
|
||||
|
||||
---
|
||||
|
||||
## 七、接口层对比(顺带记录,详见 GUI-PLAN.md)
|
||||
|
||||
它和我们在**输入注入**上的关键分歧:
|
||||
|
||||
| | Coopanion | 本仓(HomeAgent GUI) |
|
||||
|---|---|---|
|
||||
| 注入 API | `SendInput` | `SendInput`(默认)/ `PostMessage`(可选) |
|
||||
| 能否操作 Chromium/游戏 | ✅ 能 | ✅ 能(sendinput 模式) |
|
||||
| 是否移动真实光标 | ✅ 会移动 | ✅ 会移动(sendinput)/ ❌ 不动(overlay) |
|
||||
| 用户操作时 | 等待空闲再注入 | 等待空闲再注入(**借鉴自它**) |
|
||||
| 自绘光标 | ❌ 没有 | ✅ 有(mascot + 点击脉冲) |
|
||||
|
||||
### 从它学到的关键一条
|
||||
|
||||
`GetLastInputInfo` 是**全局**的 —— agent 自己的 `SendInput` 也会被算作"用户输入"。
|
||||
若不排除自己的注入,`idleMs()` 每次注入后归零,于是**永远处于"用户正在操作"状态**,
|
||||
所有命令都被暂缓。
|
||||
|
||||
它的做法(我们已照此实现)是三重判定:
|
||||
|
||||
1. `ownTick` —— 自己最后一次注入的时刻,之后 40ms 内的输入忽略
|
||||
2. `ownCursor` —— 自己放下的光标位置,偏离 >2px 说明是用户动的
|
||||
3. 无符号差值 —— `GetTickCount` 每 2^32 ms 回绕,必须按无符号比较
|
||||
|
||||
这个坑我们实测踩到了(`deferMs:0` 仍被暂缓),教训已写进 `agent-inject.js` 注释。
|
||||
|
||||
---
|
||||
|
||||
## 八、其他独立印证
|
||||
|
||||
Coopanion 独立印证了我们上一轮的两个判断:
|
||||
|
||||
1. **坐标就是物理像素,不要换算** —— 它注释明写:
|
||||
> All coordinates are physical pixels of the primary display. The process declares
|
||||
> itself per-monitor DPI aware (v2) before anything else, so no API here returns scaled values.
|
||||
|
||||
我们曾错误地加了 `/scaleFactor`,已改回并在判据中钉住。
|
||||
|
||||
2. **必须显式声明 DPI 感知** —— 它显式调用 `SetProcessDpiAwarenessContext(-4)`,
|
||||
并注明 "before anything else"。我们是**依赖 Electron 默认值**,未显式设。
|
||||
虽然 Electron 默认已是 per-monitor aware,但显式声明更稳。**建议补上**(见待办)。
|
||||
|
||||
---
|
||||
|
||||
## 九、待办(由本文档派生)
|
||||
|
||||
- [ ] `main.js` 显式调用 `SetProcessDpiAwarenessContext(-4)`,不依赖 Electron 默认值
|
||||
- [ ] 修 `deferMs` 配置读取不生效的问题(实测 `deferMs:0` 仍走 3000ms 默认)
|
||||
- [ ] mascot 增加姿态:思考 / 等待 / 出错(参考第三节词表)
|
||||
- [ ] 动作解析容错:未知动作丢弃而非整条失败
|
||||
- [ ] (可选)mascot 待机自主行为:闲置时轻微眨眼/呼吸
|
||||
334
GUI-PLAN.md
334
GUI-PLAN.md
@ -1,334 +0,0 @@
|
||||
# GUI 改造计划与进度
|
||||
|
||||
> 本文件是 GUI(`cmd/gui`)当前改造的**单一事实来源**:做完了什么、为什么这么做、
|
||||
> 哪些结论是实测得到的、哪些还没做完。
|
||||
>
|
||||
> 相关文档:
|
||||
> - `cmd/gui/DESIGN-NOTES.md` —— 借鉴 PiDeck 的设计与实现
|
||||
> - `SERVER-HANDOFF.md` —— 需要服务端同步配合的部分
|
||||
> - 判据:`cmd/gui/*.test.mjs`,`npm test` 全绿为准(当前 127 条)
|
||||
|
||||
---
|
||||
|
||||
## 一、当前状态
|
||||
|
||||
分支:`feature/gui-starmap-pulse-vendor-align`(不合入 main)
|
||||
|
||||
最近提交:
|
||||
|
||||
| commit | 内容 |
|
||||
|---|---|
|
||||
| `02566f3` | 修启动期误判服务端未启动;keybd_event 类型名致组合键全废 |
|
||||
| `0d8b129` | computeruse 补高级操作(两端 9→18);修 HiDPI 坐标;记录 PiDeck 借鉴 |
|
||||
| `3059c99` | 加载动画改三点式 + 新增系统通知 |
|
||||
|
||||
判据:`cd cmd/gui && npm test` —— **127 条全绿**(其中 `computeruse.test.mjs` 43 条)。
|
||||
|
||||
---
|
||||
|
||||
## 二、已完成
|
||||
|
||||
### 2.1 computeruse 能力面:9 → 18 个动作
|
||||
|
||||
此前 GUI 侧(`cmd/gui/main.js`)与 waiter 侧(`cmd/waiter/device.go`)都只有
|
||||
9 个基础动作,与业界 computer-use 的基本盘差距明显。
|
||||
|
||||
GUI 侧现支持 **18 个**(已用脚本从 `main.js` 的 `case "computeruse"` 块精确提取核对):
|
||||
|
||||
```
|
||||
click doubleclick rightclick middleclick tripleclick
|
||||
move hover drag mousedown mouseup
|
||||
scroll keypress hotkey combo
|
||||
type wait sleep display
|
||||
```
|
||||
|
||||
waiter 侧(Linux)为 **20 个** —— 比 GUI 多 `middle`、`right` 两个
|
||||
「button 简写形式」,少 `physical` 相关处理;其余一致。
|
||||
|
||||
要点:
|
||||
|
||||
- **拖拽分步移动**(默认 12 步,上限 60)。一步到位会让 canvas 拖拽、列表排序、
|
||||
滑动条失效 —— 这类应用依赖中间 `mousemove` 事件。
|
||||
- **组合键归一化**(waiter 侧 `normKeySpec`)。旧实现把模型给的 key 字符串原样丢给
|
||||
`xdotool key`,于是 `"Ctrl+C"`、`"ctrl + c"`、`"CTRL-C"` **全部失效**。
|
||||
- **`type` 加 `--clearmodifiers`**(waiter 侧)。避免残留的 Ctrl/Alt 把后续输入
|
||||
变成快捷键。
|
||||
- **scroll 修正**(两端)。旧实现固定滚 `click 4/5`,忽略方向语义也不支持横向。
|
||||
|
||||
### 2.2 修 HiDPI 坐标错位 ⚠️ 重要,详见第五节
|
||||
|
||||
截图给模型的是**物理像素**,Win32 坐标 API 在 Electron 里也是物理像素,
|
||||
**不需要换算**。详见第五节的实测与踩坑记录。
|
||||
|
||||
### 2.3 修启动期误判「服务端未启动」
|
||||
|
||||
`isServerRunning()` 原硬编码 `http://localhost:8080/`。实测该部署下:
|
||||
|
||||
```
|
||||
localhost:8080 -> 000 连不上
|
||||
192.168.2.60:8080 -> 302 正常(homed 跑在 WSL 里)
|
||||
```
|
||||
|
||||
于是每次启动都:探测失败 → 误判服务没起 → 拉起 GUI 旁并不存在的 `homed.exe`
|
||||
→ 干等 8s 超时 → 打印 `homed failed to start within timeout`。
|
||||
该 ERR 曾累计出现数十次(实测 40+ 次,日志轮转后计数会变),
|
||||
且是 `gui.log` 里唯一的错误项。
|
||||
|
||||
改法:按 `connections.json` 里已配置的连接(当前优先)探测,命中即在线;
|
||||
服务可达时跳过 homed 自动拉起;无本地 homed 可执行文件时不再空等超时。
|
||||
|
||||
实测验证:修复后启动日志为 `server reachable, skip homed autostart`,该 ERR 消失。
|
||||
|
||||
> 这条很可能就是此前「聊天链路有问题」的根因 —— GUI 自以为服务端由自己托管,
|
||||
> 实际聊天依赖的是 `connections.json` 里的连接。
|
||||
|
||||
### 2.4 修 keybd_event 类型名致组合键全废
|
||||
|
||||
koffi 不认 Win32 头文件里的 `byte`:
|
||||
|
||||
```
|
||||
"void keybd_event(byte bVk, ...)" -> Error: Unknown or invalid type name 'byte'
|
||||
"void keybd_event(uint8 bVk, ...)" -> 正常
|
||||
```
|
||||
|
||||
原代码用的正是 `byte`,且整段包在**空 catch** 里,于是 `keybd_event` 恒为 null,
|
||||
所有组合键返回 `keybd_event unavailable`。
|
||||
|
||||
这个 bug 极难发现:单键 click/move/scroll 走 `mouse_event`,完全不受影响。
|
||||
现在签名解析失败会打日志,不再静默。
|
||||
|
||||
### 2.5 模式 A:agent 独立光标 + 后台注入(**代码已通,效果待验证**)
|
||||
|
||||
见第三节。
|
||||
|
||||
---
|
||||
|
||||
## 三、模式 A:agent 独立光标(进行中)
|
||||
|
||||
### 3.1 目标
|
||||
|
||||
现有 `computeruse` 走 `SetCursorPos` + `mouse_event`,**会抢走用户的真实鼠标**,
|
||||
agent 一干活用户就没法用电脑。
|
||||
|
||||
模式 A 的目标:GUI 自绘一个 agent 专属光标(mascot 形象),实际输入通过
|
||||
`PostMessage` 直接投递到目标窗口句柄,**用户真实鼠标全程不动**。
|
||||
|
||||
### 3.2 新增文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|---|---|
|
||||
| `cmd/gui/agent-cursor.js` | 透明置顶层窗口 + mascot 指针 + 移动/点击脉冲动画;记录上次命中的窗口句柄 |
|
||||
| `cmd/gui/agent-inject.js` | 两种注入路径:`SendInput`(默认)与 `PostMessage`(后台);含用户活动感知 |
|
||||
|
||||
### 3.3 配置
|
||||
|
||||
`gui-prefs.json` → `deviceBridge.agentCursor`:
|
||||
|
||||
| 键 | 值 | 行为 |
|
||||
|---|---|---|
|
||||
| `mode` | `"sendinput"` | **默认**。`SendInput` 注入系统输入队列,可靠性高(Chromium/游戏都能操作)。代价:会移动真实光标,故先等用户停手 |
|
||||
| `mode` | `"overlay"` | `PostMessage` 直投窗口句柄。用户鼠标纹丝不动,但自绘界面常忽略合成消息 |
|
||||
| `mode` | `"real"` | 保留旧代码路径(兼容) |
|
||||
| `deferMs` | 数字 | 用户活动时最多等多久(默认 3000,上限 10000)。0 = 不等待 |
|
||||
|
||||
> 为什么默认改成 `sendinput`:用户要求「可靠 + 用户正在操作则暂缓」。
|
||||
> 参考 `Pal-AI-Lab/Coopanion`(见 `DESKTOP-PET-NOTES.md` 第七节)。
|
||||
>
|
||||
> ⚠ 实测发现:`GetLastInputInfo` 是**全局**的,agent 自己的 `SendInput`
|
||||
> 也会被算作「用户输入」,若不排除则**永远处于「用户正在操作」状态**。
|
||||
> 已按 Coopanion 的三重判定修正(ownTick / ownCursor / 无符号差值)。
|
||||
|
||||
### 3.4 已验证(端到端)
|
||||
|
||||
用假网关(自实现 WS 服务端)下发 computeruse 命令,GUI 执行并回执。
|
||||
两轮合计:
|
||||
|
||||
**overlay(PostMessage)路径**:8/8 有响应,错误处理正确 ——
|
||||
无窗口时报「WindowFromPoint 返回空」,而不是静默回退到真实鼠标。
|
||||
|
||||
**sendinput 路径**(当前默认):8/8 有响应,回执形如
|
||||
`sendinput click @ (300,300)` / `sendinput typed 2 chars` /
|
||||
`sendinput drag (100,100) -> (400,300)`。
|
||||
|
||||
### 3.5 ⚠️ 未完成:真实点击效果未验证
|
||||
|
||||
当前执行环境**没有可供注入的可见目标窗口**
|
||||
(`WindowFromPoint` 对所有坐标返回 null,`GetCursorPos` 读回 0,0)。
|
||||
|
||||
**因此:「点击真的落到目标窗口上」这一步尚未被证明。** 需要在有交互桌面的
|
||||
会话里跑一个端到端测试:
|
||||
|
||||
> 打开记事本 → agent 点它 → 输入文字 → 截图回读验证
|
||||
|
||||
同时验证「`overlay` 模式下用户鼠标位置在全过程未变」。
|
||||
|
||||
### 3.6 设计取舍(明确的边界)
|
||||
|
||||
- **两种注入的适用范围不同**:
|
||||
- `sendinput`(默认):注入系统输入队列,等价真实硬件事件。
|
||||
Chromium/Electron 自绘界面、多数游戏都能接收 —— **可靠**。
|
||||
代价:会移动用户真实光标,故先等用户停手(`deferMs`)。
|
||||
- `overlay`:`PostMessage` 直投窗口句柄,用户鼠标纹丝不动。
|
||||
但自绘界面常忽略合成消息 ⇒ 只能操作原生 Win32 程序。
|
||||
|
||||
> 本仓曾把「自绘界面点不动」写成「Windows 消息模型的硬限制」——
|
||||
> **那个说法是错的**,它只是 `PostMessage` 这条 API 的限制。
|
||||
> 用 `SendInput` 就不受此限(参考 Coopanion)。
|
||||
|
||||
- **不做静默降级**。无论哪种模式,失败即 `status=error` 并说明原因,
|
||||
不会自作主张换成另一种注入方式。
|
||||
- **`sendinput` 模式下用户正在操作会暂缓**(而不是硬抢):
|
||||
轮询等到用户空闲,超时则跳过本次操作并报错。
|
||||
已在 `agent-inject.js` 中排除「自身注入被误判为用户操作」的陷阱。
|
||||
- **`overlay` 模式下键盘类动作依赖先前的鼠标操作**。
|
||||
`keypress`/`hotkey`/`type` 不带坐标,而该模式不移动真实光标,
|
||||
所以 `GetCursorPos` 拿到的是**用户自己**的光标位置;
|
||||
故记录「上次鼠标操作命中的 hwnd」,键盘动作投递到那里。从未点过则报错。
|
||||
|
||||
### 3.7 跨平台状态
|
||||
|
||||
| 平台 | 状态 |
|
||||
|---|---|
|
||||
| Windows | 已实现(sendinput 默认 / overlay 可选;真实效果待验证) |
|
||||
| Linux | waiter 侧仍是 `xdotool`,**会动真鼠标**。未做独立光标 |
|
||||
| macOS | 同上 |
|
||||
|
||||
---
|
||||
|
||||
## 四、待办
|
||||
|
||||
### 4.1 优先:修 `deferMs` 读取不生效
|
||||
|
||||
实测:`gui-prefs.json` 里 `agentCursor.deferMs = 0`,但命令仍按默认 3000ms
|
||||
暂缓(回执写「已等待 3000ms」)。`loadGuiPrefs()` 会重建对象且**不透传
|
||||
`agentCursor` 字段**,导致读取永远拿到默认值。
|
||||
|
||||
- [ ] `loadGuiPrefs()` 透传 `deviceBridge.agentCursor`
|
||||
- [ ] 判据:配置 `deferMs:0` 时不得进入暂缓路径
|
||||
|
||||
### 4.2 验证 sendinput 的真实效果
|
||||
|
||||
当前环境无可注入的可见目标窗口(`WindowFromPoint` 恒返回 null,
|
||||
`GetCursorPos` 读回 0,0),故「点击真的落到目标窗口上」尚未证明。
|
||||
|
||||
- [ ] 端到端测试:打开真实应用 → 点它 → 输入 → 截图回读
|
||||
- [ ] 断言:`overlay` 模式下全过程用户鼠标位置未变
|
||||
- [ ] 判据:固化进 `npm test`(需能在无桌面 CI 环境跳过)
|
||||
|
||||
### 4.3 服务端配合(详见 `SERVER-HANDOFF.md`)
|
||||
|
||||
- [ ] 放开 `device.go` 的 action 白名单(schema enum + switch default 双重限制)
|
||||
- [ ] 透传新参数 `tox`/`toy`/`steps`/`dx`/`ms`/`physical`
|
||||
- [ ] 统一 scroll 方向语义(服务端写「正=向下」,Windows 惯例是正=向上)
|
||||
- [ ] 把坐标约定写进注释(防重蹈 `/scale` 覆辙)
|
||||
|
||||
> ⚠ 不改白名单,则新增的 11 个动作**根本下发不到设备**。
|
||||
|
||||
### 4.4 设置页
|
||||
|
||||
- [ ] `agentCursor.mode` / `deferMs` 的 UI(目前只能手改 `gui-prefs.json`)
|
||||
|
||||
### 4.5 可选增强(详见 `DESKTOP-PET-NOTES.md` 第六、九节)
|
||||
|
||||
- [ ] `main.js` 显式调 `SetProcessDpiAwarenessContext(-4)`,不依赖 Electron 默认值
|
||||
- [ ] mascot 姿态:思考 / 等待 / 出错
|
||||
- [ ] 动作解析容错:未知动作丢弃而非整条失败
|
||||
- [ ] mascot 待机自主行为:闲置时轻微眨眼/呼吸
|
||||
- [ ] Linux 侧若也要「不抢鼠标」,需 Wayland + uinput 虚拟设备
|
||||
(X11 下做不到 —— XTEST 合成的是真实设备事件)
|
||||
|
||||
### 4.6 其它已定位但未修
|
||||
|
||||
- [ ] `overlay` 模式下键盘类动作「必须先点过」的限制可放宽:
|
||||
可改为「取当前前台窗口」作为兵底,而不是直接报错。
|
||||
- [ ] `real` 模式与 `sendinput` 模式代码重叠较多,可考虑合并;
|
||||
目前保留 `real` 仅为兼容旧路径,实际上与 `sendinput` 行为重复。
|
||||
- [ ] `p3.js` / `p4.js` 是本轮遗留的临时补丁脚本(未被跟踪),
|
||||
确认无价值后删除,避免占坑。
|
||||
|
||||
---
|
||||
|
||||
## 五、坐标空间:实测结论与踩坑记录 ⚠️
|
||||
|
||||
**结论:截图给模型的像素、Win32 坐标 API 的坐标,是同一个坐标系 —— 物理像素。
|
||||
不要换算。**
|
||||
|
||||
实测证据(Electron 主进程内,192 DPI / 200% 缩放屏):
|
||||
|
||||
```
|
||||
Electron screen API (DIP) : 1260 x 840 scaleFactor=2
|
||||
GetSystemMetrics : 2520 x 1680
|
||||
SetCursorPos(200,200) → GetCursorPos 读回 (200,200) ← 恒等,不缩放
|
||||
```
|
||||
|
||||
原因是 Electron 主进程是 **per-monitor DPI-aware**,此感知下 Win32 坐标不做虚拟化。
|
||||
|
||||
### 踩坑:曾经加错的 `/scaleFactor` 换算
|
||||
|
||||
本仓曾在 `main.js` 加过:
|
||||
|
||||
```js
|
||||
absX = ox + rawX / scaleFactor // ← 错的
|
||||
```
|
||||
|
||||
理由是「SetCursorPos 期望 DIP」—— **那个前提不成立**,DIP 行为只出现在
|
||||
DPI-*unaware* 进程里。加了之后反而把原本正确的点击改坏:200% 缩放屏上
|
||||
点 (600,400) 会被送到 (400,267),**越靠右下偏得越远**。
|
||||
|
||||
已改回 `physical ? rawX : rawX * scale`,并在 `computeruse.test.mjs` 加判据
|
||||
防止回退(检测 `/scale` 是否回来了)。
|
||||
|
||||
### 为什么会误判(重要)
|
||||
|
||||
**在普通 node 进程里测同一个函数,读到的是 1260×840**(虚拟化后的值),
|
||||
会让人以为 Win32 用 DIP。**必须在 Electron 内测** —— 同一份代码,
|
||||
不同进程的 DPI 感知状态不同,答案也不同。
|
||||
|
||||
同理,`python`/`go` 等非 DPI-aware 宿主里测同样不可作为依据。
|
||||
|
||||
---
|
||||
|
||||
## 六、端到端测试方法(可复用)
|
||||
|
||||
不依赖真实服务端,用自实现的假网关驱动 GUI:
|
||||
|
||||
1. 起一个最小 WS 服务端(自实现 HTTP `Upgrade` 握手 + 帧编解码)
|
||||
2. 改写 `gui-prefs.json` 指向它,并置 `deviceBridge.authorized = true`
|
||||
3. 启动 GUI,等 `hello` / `bind` 完成
|
||||
4. 下发 `{op:"cmd", cmd_type:"homeagent", command:"computeruse {...}"}`
|
||||
5. 读取 GUI 回的 `{op:"cmd_result", status, output, error}`
|
||||
|
||||
### 两个踩过的坑
|
||||
|
||||
- **服务端→客户端的帧不能加掩码。** RFC 6455 只要求客户端→服务端掩码。
|
||||
照抄 GUI 主进程(客户端方向)的 `sendDeviceFrame` 会导致 GUI 解不出。
|
||||
- **客户端→服务端的帧是掩码的**,解帧时必须先解掩码。
|
||||
|
||||
---
|
||||
|
||||
## 七、koffi 3.x 使用约定(踩坑记录)
|
||||
|
||||
| 事项 | 正确做法 | 错误做法 / 现象 |
|
||||
|---|---|---|
|
||||
| 类型名 | `uint8` / `int32_t` / `long` | `byte` → `Unknown or invalid type name` |
|
||||
| 指针参数 | 传 **Node `Buffer`** | 传普通 JS 对象 → **不写回**,读回仍是原值 |
|
||||
| 结构体 | `koffi.struct(name, {...})` 后传普通对象 | 重复 `koffi.struct("POINT")` → `Duplicate type name` |
|
||||
| 字符串 | `Buffer.from(s + "\0", "utf16le")` | `koffi.alloc(str)` 把 str 当**类型名** |
|
||||
|
||||
**指针参数必须用 Buffer** 这条尤其重要:`WindowFromPoint` 若用普通对象,
|
||||
会拿不到真实 hwnd ⇒ 点击落到错误窗口上。
|
||||
|
||||
跨进程/跨模块重复声明同名结构体会抛 `Duplicate type name`,需 try/catch 换名注册。
|
||||
|
||||
---
|
||||
|
||||
## 八、环境注意事项
|
||||
|
||||
- 本仓库文件多为 **CRLF**。用 Node 脚本打补丁时先 `replace(/\r\n/g, "\n")` 处理,
|
||||
写回时按原行尾还原。
|
||||
- `git status` 会把大量文件标成 `M`,但 `git diff --numstat` 显示 `0 0`
|
||||
—— 那是 CRLF 归一化噪声,不是真实改动。**提交前用 `--numstat` 甄别。**
|
||||
- 本机没有 Go 工具链(`go: command not found`),`cmd/waiter/` 的改动
|
||||
**未经过编译验证**,需在有 Go 环境的机器上 `go build ./cmd/waiter` 确认。
|
||||
- 本会话多数为无头/无交互桌面环境,`WindowFromPoint` / 真实光标相关行为
|
||||
**无法在此验证**,需在有桌面的会话实测。
|
||||
@ -1,294 +0,0 @@
|
||||
# 服务端交接:computeruse 协议对齐
|
||||
|
||||
> 面向服务端接手人。本文档描述 GUI 侧已完成的改动,以及**服务端必须同步修改**的部分。
|
||||
> 修改范围集中在 `internal/plugins/remotedevice/`(`device.go`)。
|
||||
> GUI 侧代码与判据见 `cmd/gui/`。
|
||||
|
||||
---
|
||||
|
||||
## 一、为什么需要这份文档
|
||||
|
||||
GUI 侧把 `computeruse` 从 9 个基础动作扩到 **18 个**,并新增了**模式 A(agent 独立光标 + 后台注入)**。
|
||||
但服务端 `device.go` 目前对 `action` 做了**两重硬限制**,导致新增能力根本下发不到设备:
|
||||
|
||||
1. **JSON Schema 的 `enum`**(`device.go:129`)—— 7 个值之外直接被 schema 校验拒绝
|
||||
2. **`switch` 的 `default` 分支**(`device.go:471`)—— 返回「不支持的操作类型」
|
||||
|
||||
所以:**即使 GUI 已实现,服务端不改则新功能等于不存在。**
|
||||
|
||||
---
|
||||
|
||||
## 二、必做项 1:放开 action 白名单
|
||||
|
||||
### 现状
|
||||
|
||||
```go
|
||||
// device.go:129 —— schema enum,只有 7 个
|
||||
"enum": []interface{}{"click", "doubleclick", "rightclick", "move", "scroll", "keypress", "type"},
|
||||
```
|
||||
|
||||
```go
|
||||
// device.go:471 —— default 直接报错
|
||||
default:
|
||||
return nil, fmt.Errorf("不支持的操作类型 %s(可选 click/doubleclick/rightclick/move/scroll/keypress/type)", action)
|
||||
```
|
||||
|
||||
### 需要改成
|
||||
|
||||
GUI 侧实际支持的 18 个 action(判据同源,`cmd/gui/computeruse.test.mjs` 会校验两端一致):
|
||||
|
||||
```
|
||||
click doubleclick rightclick middleclick tripleclick
|
||||
move hover drag mousedown mouseup
|
||||
scroll keypress hotkey combo
|
||||
type wait sleep display
|
||||
```
|
||||
|
||||
建议在 `device.go` 里提取成单一常量,schema 与 switch 共用,避免两处漂移:
|
||||
|
||||
```go
|
||||
// computeruseActions 是设备端支持的完整动作集合。
|
||||
// ★ 必须与 cmd/gui/main.js 的 computeruse switch 保持一致;
|
||||
// cmd/gui/computeruse.test.mjs 会校验 GUI 侧包含这些动作。
|
||||
var computeruseActions = []string{
|
||||
"click", "doubleclick", "rightclick", "middleclick", "tripleclick",
|
||||
"move", "hover", "drag", "mousedown", "mouseup",
|
||||
"scroll", "keypress", "hotkey", "combo", "type",
|
||||
"wait", "sleep", "display",
|
||||
}
|
||||
```
|
||||
|
||||
然后:
|
||||
- schema 的 `enum` 改为由 `computeruseActions` 生成
|
||||
- `switch` 的 `default` 分支的错误信息里列出这份清单
|
||||
|
||||
### 各动作的语义(供写 tool description 时参考)
|
||||
|
||||
| action | 必需参数 | 说明 |
|
||||
|---|---|---|
|
||||
| `click` | `x`,`y` | 可选 `button`(left/right/middle) |
|
||||
| `doubleclick` | `x`,`y` | |
|
||||
| `rightclick` | `x`,`y` | 等价 `click` + `button:right` |
|
||||
| `middleclick` | `x`,`y` | 中键 |
|
||||
| `tripleclick` | `x`,`y` | 三击选中整行 |
|
||||
| `move` / `hover` | `x`,`y` | 只移动不点击,用于触发 tooltip |
|
||||
| `mousedown` / `mouseup` | `x`,`y` | 按下/释放分离,配对可做「按住」 |
|
||||
| `drag` | `x`,`y`,`tox`,`toy` | 可选 `steps`(默认 12,上限 60) |
|
||||
| `scroll` | `dy` | 可选 `dx`(横向) |
|
||||
| `keypress` | `key` | 如 `ctrl+c`、`alt+f4` |
|
||||
| `hotkey` / `combo` | `key` | 与 keypress 同义 |
|
||||
| `type` | `text` | |
|
||||
| `wait` / `sleep` | `ms` | 显式等待,等 UI/动画完成 |
|
||||
| `display` | — | 返回各显示器 bounds + scaleFactor |
|
||||
|
||||
> ⚠ **`drag` 的分步移动不是可有可无的**:canvas 拖拽、列表排序、滑动条
|
||||
> 这类应用依赖中间 `mousemove` 事件,一步到位会失效。所以 `steps` 建议透传。
|
||||
|
||||
---
|
||||
|
||||
## 三、必做项 2:透传新参数
|
||||
|
||||
### 现状
|
||||
|
||||
`device.go` 的 `switch action`(第 439 行起)只转发这些:
|
||||
|
||||
```go
|
||||
params["x"], params["y"], params["button"]
|
||||
params["dy"]
|
||||
params["key"]
|
||||
params["text"]
|
||||
```
|
||||
|
||||
其余参数**被静默丢弃**,即使 LLM 传了也到不了设备。
|
||||
|
||||
### 需要补上
|
||||
|
||||
| 参数 | 哪些动作需要 | 缺失后果 |
|
||||
|---|---|---|
|
||||
| `tox` / `toy` | `drag` | 拖拽终点丢失,拖不动 |
|
||||
| `steps` | `drag` | 退化成一步到位,拖拽排序类操作失效 |
|
||||
| `dx` | `scroll` | 横向滚动不可用 |
|
||||
| `ms` / `duration` | `wait` / `sleep` | 等待时长不可控 |
|
||||
| `button` | `hover`/`mousedown`/`mouseup`/`tripleclick` | 中键/右键按不下 |
|
||||
| `physical` | 全部坐标类 | 无法声明坐标系(见第五节) |
|
||||
|
||||
建议实现(保持与现有风格一致):
|
||||
|
||||
```go
|
||||
// 透传辅助:从 args 取整数键,缺省则不写入 params。
|
||||
func putInt(params map[string]interface{}, args map[string]interface{}, key string) {
|
||||
if v, ok := args[key].(float64); ok {
|
||||
params[key] = int(v)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
在 `switch` 之后统一补一遍(避免每个 case 重复):
|
||||
|
||||
```go
|
||||
for _, k := range []string{"tox", "toy", "steps", "dx", "ms", "duration", "physical"} {
|
||||
putInt(params, args, k)
|
||||
}
|
||||
```
|
||||
|
||||
> ⚠ `physical` 的类型是布尔或字符串都可能,取决于 LLM 怎么填。
|
||||
> 建议 schema 里显式声明 `"type": "boolean"`,并在 `putInt` 之外单独处理。
|
||||
|
||||
---
|
||||
|
||||
## 四、必做项 3:统一 scroll 方向语义 ⚠️
|
||||
|
||||
### 现状:两侧约定相反
|
||||
|
||||
**服务端**(`device.go:130`):
|
||||
|
||||
```go
|
||||
"dy": map[string]interface{}{"type": "integer", "description": "scroll 滚动量:正=向下,负=向上"},
|
||||
```
|
||||
|
||||
**GUI / Windows 惯例**:正 = 向上。
|
||||
|
||||
这不是笔误级的小问题——**方向反了会让 agent 在页面上滚到完全相反的位置**,
|
||||
且因为「能滚」,不会报错,只会静默地做错事,比崩溃更难发现。
|
||||
|
||||
### 建议
|
||||
|
||||
统一为 **Windows 惯例:正 = 向上,负 = 向下**(与鼠标滚轮一致,也与 GUI 实现一致)。
|
||||
改 schema description:
|
||||
|
||||
```go
|
||||
"dy": map[string]interface{}{
|
||||
"type": "integer",
|
||||
"description": "scroll 垂直滚动量:正=向上滚,负=向下滚(与鼠标滚轮方向一致)。可选 dx 做横向滚动,正=向右",
|
||||
},
|
||||
```
|
||||
|
||||
GUI 侧对应实现(`cmd/gui/main.js`)已按此约定,无需改动。
|
||||
|
||||
> 如果服务端坚持保留「正=向下」,则必须**同步改 GUI**,否则两侧永远对不上。
|
||||
> 二选一,但必须写进注释并进判据。
|
||||
|
||||
---
|
||||
|
||||
## 五、建议项:把坐标约定写进注释(防重蹈)
|
||||
|
||||
这条不是功能需求,而是**防止后来人再犯同一个错**——我本人就在这一轮犯过。
|
||||
|
||||
### 事实
|
||||
|
||||
**截图给模型的像素、Win32 坐标 API 的坐标,是同一个坐标系:物理像素。不要换算。**
|
||||
|
||||
实测证据(Electron 主进程内,192 DPI / 200% 缩放屏):
|
||||
|
||||
```
|
||||
Electron screen API (DIP) : 1260 x 840 scaleFactor=2
|
||||
GetSystemMetrics : 2520 x 1680
|
||||
SetCursorPos(200,200) → GetCursorPos 读回 (200,200) ← 恒等,不缩放
|
||||
```
|
||||
|
||||
原因是 Electron 主进程是 **per-monitor DPI-aware**,此感知下 Win32 坐标不做虚拟化。
|
||||
|
||||
### 曾经的坑
|
||||
|
||||
本仓曾在 `cmd/gui/main.js` 加过:
|
||||
|
||||
```go
|
||||
absX = ox + rawX / scaleFactor // ← 错的
|
||||
```
|
||||
|
||||
理由是「SetCursorPos 期望 DIP」——**那个前提不成立**,DIP 行为只出现在
|
||||
DPI-*unaware* 进程里。加了之后反而把原本正确的点击改坏:200% 缩放屏上
|
||||
点 (600,400) 会被送到 (400,267),**越靠右下偏得越远**。
|
||||
|
||||
(易误判之处:在普通 node 进程里测同一个函数,读到的是 1260×840,
|
||||
那是虚拟化后的值,会把人带偏。必须在 Electron 内测。)
|
||||
|
||||
### 建议
|
||||
|
||||
在两处加注释:
|
||||
|
||||
1. `device.go` 的 `computeruse` 函数上方
|
||||
2. tool schema 里 `x`/`y` 的 description
|
||||
|
||||
```go
|
||||
// 坐标约定:x/y 为**物理像素**,与 screensee 截图坐标系一致,
|
||||
// 也与设备端 Win32 坐标一致,**不需要按 scaleFactor 换算**。
|
||||
// (设备端 GUI 是 per-monitor DPI-aware,Win32 坐标不做虚拟化。)
|
||||
// 仅当调用方显式传 physical=false 时,设备端才按 DIP 再乘回缩放比。
|
||||
```
|
||||
|
||||
`cmd/gui/computeruse.test.mjs` 已加判据防止 GUI 侧回退(会检测 `/scale` 是否回来了)。
|
||||
**服务端侧目前无对等判据**,建议补一个,锁定 schema 里的 description 不含
|
||||
「需换算」之类的误导表述。
|
||||
|
||||
---
|
||||
|
||||
## 六、模式 A(agent 独立光标 + 后台注入)需要服务端知道的事
|
||||
|
||||
### 行为变化
|
||||
|
||||
GUI 默认启用 `agentCursor.mode = "overlay"`:
|
||||
|
||||
- agent 操作时**不再移动用户的真实鼠标**,用户可同时正常使用电脑
|
||||
- GUI 会在目标位置绘制一个 mascot 形象作为 agent 的虚拟光标,并有移动/点击脉冲动画
|
||||
- 实际输入通过 `PostMessage` 直接投递到目标窗口句柄
|
||||
|
||||
配置项(`gui-prefs.json` → `deviceBridge.agentCursor.mode`):
|
||||
|
||||
| 值 | 行为 |
|
||||
|---|---|
|
||||
| `"overlay"` | **默认**。不动用户鼠标,后台注入 |
|
||||
| `"real"` | 旧行为。操作真实鼠标(`SetCursorPos` + `mouse_event`) |
|
||||
|
||||
### 服务端无需改动,但要知道三条限制
|
||||
|
||||
1. **只对「接受窗口消息的程序」可靠。**
|
||||
原生 Win32 程序(记事本、Excel、Office 等)可靠;Chromium/Electron 自绘界面、
|
||||
多数游戏、DirectX 程序会忽略合成消息。这是 Windows 消息模型的硬限制,
|
||||
不是实现缺陷。
|
||||
|
||||
2. **注入失败会明确报错,不静默回退。**
|
||||
用户既然选择「不抢鼠标」,就不该偷偷改成操作真实鼠标。所以失败时
|
||||
`status=error`,`error` 里说明原因(例如「该坐标下没有窗口」)。
|
||||
|
||||
3. **键盘类动作需要先有点击类动作。**
|
||||
`keypress`/`hotkey`/`type` 不带坐标,且后台注入**不移动真实光标**,
|
||||
所以 `GetCursorPos` 拿到的是**用户自己**的光标位置(通常不是 agent 要操作的地方)。
|
||||
设备端记录「上一次鼠标操作命中的窗口」,键盘动作投递到那里。
|
||||
**若从未点过,键盘动作会报错。** 服务端在 tool description 里说明这一依赖关系,
|
||||
可以减少 LLM 调错。
|
||||
|
||||
---
|
||||
|
||||
## 七、验收方式
|
||||
|
||||
GUI 侧判据(服务端改完后跑):
|
||||
|
||||
```bash
|
||||
cd cmd/gui && npm test
|
||||
```
|
||||
|
||||
含 `computeruse.test.mjs`,会校验两端 action 覆盖面的一致性。
|
||||
|
||||
服务端侧建议补的判据(放在 `internal/plugins/remotedevice/`):
|
||||
|
||||
1. schema `enum` 覆盖全部 18 个 action
|
||||
2. 每个 action 在 `switch` 里都有对应 case,不落 `default`
|
||||
3. `tox`/`toy`/`steps`/`dx`/`ms` 能透传到下发 JSON(可用现有
|
||||
`binary_test.go:471` 的 computeruse 下发测试作模板,参考它如何断言收到的 JSON)
|
||||
|
||||
---
|
||||
|
||||
## 八、附:本次 GUI 侧一并修掉的两个 bug(服务端可参考)
|
||||
|
||||
1. **GUI 误判「服务端未启动」**:原实现只探 `localhost:8080`,
|
||||
服务端在 WSL/远端时探不通 → 每次启动误判服务没起 → 去拉起 GUI 旁不存在的
|
||||
`homed.exe` → 干等 8s 超时。已改为按 `connections.json` 里已配置的连接探测。
|
||||
(服务端侧无改动,但这条能解释历史上 GUI 日志里刷屏的
|
||||
`homed failed to start within timeout`。)
|
||||
|
||||
2. **koffi 不认 Win32 的 `byte` 类型名**:
|
||||
`keybd_event(byte bVk, ...)` 会抛
|
||||
`Unknown or invalid type name 'byte'`,必须写 `uint8`。
|
||||
曾导致组合键整体返回 `keybd_event unavailable`,而单键 click 不受影响、极易漏掉。
|
||||
与服务端无直接关系,但若服务端将来也走 FFI 调 Win32,注意这一点。
|
||||
@ -2,10 +2,10 @@
|
||||
# 生产部署方案(**待确认,不自动执行**)
|
||||
#
|
||||
# 用法:
|
||||
# bash deploy-plan.sh check # 只做部署前检查,不改任何东西
|
||||
# bash deploy-plan.sh backup # 备份当前二进制与适配器
|
||||
# bash deploy-plan.sh deploy # 替换二进制并重启(需显式确认)
|
||||
# bash deploy-plan.sh rollback # 回滚到备份
|
||||
# bash deploy/scripts/deploy-plan.sh check # 只做部署前检查,不改任何东西
|
||||
# bash deploy/scripts/deploy-plan.sh backup # 备份当前二进制与适配器
|
||||
# bash deploy/scripts/deploy-plan.sh deploy # 替换二进制并重启(需显式确认)
|
||||
# bash deploy/scripts/deploy-plan.sh rollback # 回滚到备份
|
||||
#
|
||||
# ★ 本脚本刻意**不包含**任何"自动回滚"逻辑:回滚要不要做、什么时候做,
|
||||
# 是人的判断。脚本只负责把状态保全好,让回滚成为一条可执行的命令。
|
||||
@ -13,7 +13,7 @@
|
||||
set -uo pipefail
|
||||
|
||||
BIN=/usr/local/bin/homed
|
||||
DATA=/home/newqqagent
|
||||
DATA=${HA_DATA}
|
||||
BAK=/var/tmp/homed-backup-$(date +%Y%m%d-%H%M%S)
|
||||
NEW=${NEW_BIN:-/tmp/homed-ort}
|
||||
ADAPTERS=$DATA/adapters
|
||||
@ -22,15 +22,15 @@
|
||||
# 结论(打不了包)不变但**理由已变**,别照旧文字理解。
|
||||
#
|
||||
# 用法:
|
||||
# ./deploy-sdk-site.sh # 构建 + 部署 + 验证(两个站)
|
||||
# ./deploy-sdk-site.sh --check # 只核对线上与本地产物差异,不动线上
|
||||
# ./deploy-sdk-site.sh --introduce # 只部署 introduce 站(改 site/ 时用,跳过 SDK 站重建)
|
||||
# ./deploy-sdk-site.sh --rollback <备份目录名> # 回滚 sdk 站
|
||||
# ./deploy-sdk-site.sh --rollback-introduce <备份目录名> # 回滚 introduce 站
|
||||
# ./deploy/scripts/deploy-sdk-site.sh # 构建 + 部署 + 验证(两个站)
|
||||
# ./deploy/scripts/deploy-sdk-site.sh --check # 只核对线上与本地产物差异,不动线上
|
||||
# ./deploy/scripts/deploy-sdk-site.sh --introduce # 只部署 introduce 站(改 site/ 时用,跳过 SDK 站重建)
|
||||
# ./deploy/scripts/deploy-sdk-site.sh --rollback <备份目录名> # 回滚 sdk 站
|
||||
# ./deploy/scripts/deploy-sdk-site.sh --rollback-introduce <备份目录名> # 回滚 introduce 站
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SDK_DIR=/home/program/TrueAgent/third_party/homeagent-sdk
|
||||
SDK_DIR=${REPO}/third_party/homeagent-sdk
|
||||
BUILD_DIR="$SDK_DIR/site_build"
|
||||
HOST=192.168.2.106
|
||||
SSH="ssh -n -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=no admin@$HOST"
|
||||
@ -61,13 +61,13 @@ fi
|
||||
|
||||
# ── introduce 站 ──
|
||||
#
|
||||
# 零构建:/home/program/TrueAgent/site/ 里就是成品(site/README.md 自称
|
||||
# 零构建:${REPO}/site/ 里就是成品(site/README.md 自称
|
||||
# 「单文件、零构建、零依赖」),改完直接是产物,所以这条只需要同步。
|
||||
#
|
||||
# 部署时**故意不带上 README.md**:那是仓库里的说明文档,不是站点资源。
|
||||
# 线上现有 2 个文件(index.html + assets/logo.svg),带 README 会多出一个
|
||||
# 线上原本没有的文件,反而让 diff 比对永远不为零。
|
||||
INTRO_SRC=/home/program/TrueAgent/site
|
||||
INTRO_SRC=${REPO}/site
|
||||
INTRO_PKG_NAME=intro-site-$TS.tar.gz
|
||||
INTRO_PKG=/tmp/$INTRO_PKG_NAME
|
||||
|
||||
@ -15,9 +15,9 @@
|
||||
# · 验证项:进程活着 + 设备在网关侧注册成功
|
||||
#
|
||||
# 用法:
|
||||
# bash deploy-waiter.sh check 只读检查
|
||||
# bash deploy-waiter.sh deploy <ip> 更新指定一台
|
||||
# bash deploy-waiter.sh rollback <ip> 回滚指定一台
|
||||
# bash deploy/scripts/deploy-waiter.sh check 只读检查
|
||||
# bash deploy/scripts/deploy-waiter.sh deploy <ip> 更新指定一台
|
||||
# bash deploy/scripts/deploy-waiter.sh rollback <ip> 回滚指定一台
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
@ -53,7 +53,7 @@ cmp -s /tmp/d1 /tmp/d2; echo $? # ⇒ 1(cmp 未被污染)
|
||||
```
|
||||
|
||||
为什么这比“偶尔报错”危险得多:它让**部署判据变成假绿灯**。
|
||||
2026-09-29 实测:`deploy-sdk-site.sh --check` 对着两份**确实不同**的
|
||||
2026-09-29 实测:`deploy/scripts/deploy-sdk-site.sh --check` 对着两份**确实不同**的
|
||||
`index.html`(本地 `f8898cfc…` / 线上 `cc615cfa…`)报「✓ 逐字节一致」,
|
||||
于是永远判定「无需部署」——站点改完再也不会被更新,而且看不出来。
|
||||
已在 `547d28d` 修掉(钉绝对路径 + 启动自检)。
|
||||
@ -112,9 +112,9 @@ ls -l /tmp/homed-ort-new # 与 /usr/local/bin/homed
|
||||
### 1.3 部署
|
||||
|
||||
```bash
|
||||
bash deploy-plan.sh check # 只读
|
||||
NEW_BIN=/tmp/homed-ort-new bash deploy-plan.sh deploy # 需输入 yes
|
||||
bash deploy-plan.sh rollback # 回滚
|
||||
bash deploy/scripts/deploy-plan.sh check # 只读
|
||||
NEW_BIN=/tmp/homed-ort-new bash deploy/scripts/deploy-plan.sh deploy # 需输入 yes
|
||||
bash deploy/scripts/deploy-plan.sh rollback # 回滚
|
||||
```
|
||||
|
||||
- `check`:服务状态、onnxruntime 标签、`libonnxruntime.so`、模型资产、适配器清单
|
||||
@ -235,10 +235,10 @@ md5sum ${HA_DATA}/adapters/*.lua | md5sum
|
||||
### 2.2 更新
|
||||
|
||||
```bash
|
||||
bash deploy-waiter.sh check # 两台一起,只读
|
||||
bash deploy-waiter.sh deploy 192.168.2.106 # 需输入 yes
|
||||
bash deploy-waiter.sh deploy 192.168.2.30 # 上一台验证通过后再做
|
||||
bash deploy-waiter.sh rollback <ip>
|
||||
bash deploy/scripts/deploy-waiter.sh check # 两台一起,只读
|
||||
bash deploy/scripts/deploy-waiter.sh deploy 192.168.2.106 # 需输入 yes
|
||||
bash deploy/scripts/deploy-waiter.sh deploy 192.168.2.30 # 上一台验证通过后再做
|
||||
bash deploy/scripts/deploy-waiter.sh rollback <ip>
|
||||
```
|
||||
|
||||
**逐台更新,不要并行** —— 两台都连同一网关,同时重启会同时断链。
|
||||
@ -370,7 +370,7 @@ stage handler 入口自动加锁 + 锁不可重入所致,**插件侧问题**
|
||||
|
||||
## 5. 部署站点(SDK 文档站 / introduce)
|
||||
|
||||
`deploy-sdk-site.sh` 覆盖两个静态站,**与 homed / waiter 是独立部署单元**。
|
||||
`deploy/scripts/deploy-sdk-site.sh` 覆盖两个静态站,**与 homed / waiter 是独立部署单元**。
|
||||
|
||||
| 站 | 本地源 | 线上位置 | 构建 |
|
||||
| --- | --- | --- | --- |
|
||||
@ -385,9 +385,9 @@ introduce 是**零构建**的 —— `site/` 里就是成品,脚本部署时**
|
||||
### 5.1 用法
|
||||
|
||||
```bash
|
||||
bash deploy-sdk-site.sh --check # 只核对差异,不动线上
|
||||
bash deploy-sdk-site.sh # 构建 + 部署 + 验证
|
||||
bash deploy-sdk-site.sh --rollback .sdk-bak-<时间戳> # 回滚
|
||||
bash deploy/scripts/deploy-sdk-site.sh --check # 只核对差异,不动线上
|
||||
bash deploy/scripts/deploy-sdk-site.sh # 构建 + 部署 + 验证
|
||||
bash deploy/scripts/deploy-sdk-site.sh --rollback .sdk-bak-<时间戳> # 回滚
|
||||
```
|
||||
|
||||
`--check` 逐字节比对本地与线上,两个站都一致时可直接跳过部署。
|
||||
@ -441,7 +441,7 @@ cd third_party/homeagent-sdk
|
||||
# 3) 重新生成(含 docs/api/* 与 llms.txt 同步)
|
||||
tools/apidoc/build.sh
|
||||
# 4) 部署并验证
|
||||
bash ../../deploy-sdk-site.sh
|
||||
bash deploy/scripts/deploy-sdk-site.sh
|
||||
```
|
||||
|
||||
验证线上是否真的可访问(**别只看本地产物**):
|
||||
|
||||
@ -497,9 +497,9 @@ waiter 有**两条**设备桥启动路径:
|
||||
|
||||
### 部署脚本的一个坑
|
||||
|
||||
`deploy-waiter.sh` 里 `ssh` 会从 stdin 读,把后续 `read -p "确认更新"` 的输入吃掉:
|
||||
`deploy/scripts/deploy-waiter.sh 里 `ssh` 会从 stdin 读,把后续 `read -p "确认更新"` 的输入吃掉:
|
||||
|
||||
bash deploy-waiter.sh deploy <ip> <<< "yes" # 喂了 yes 却打印「已取消」
|
||||
bash deploy/scripts/deploy-waiter.sh deploy <ip> <<< "yes" # 喂了 yes 却打印「已取消」
|
||||
|
||||
脚本本身完全正常、备份逻辑没问题,只是"明明喂了 yes 却什么也没发生"。
|
||||
已给 5 处 `ssh` 统一加 `-n`。
|
||||
|
||||
132
internal/plugins/remotedevice/clipboardsue_content_test.go
Normal file
132
internal/plugins/remotedevice/clipboardsue_content_test.go
Normal file
@ -0,0 +1,132 @@
|
||||
package remotedevice
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// clipboardsue 的 content 参数:从**服务端可见的路径**读入剪切板。
|
||||
//
|
||||
// ## 为什么要它(2026-10-05)
|
||||
//
|
||||
// 长文本/代码块塞不进 `clipboardsue <文字>` 的命令行;让模型先生成一大段
|
||||
// 文本再原样传给设备,既烧 token 又容易在转述中失真。
|
||||
// 与 GUI 侧 screensue / speakeruse 的 `@<路径>` 是同一类问题的同一类解法。
|
||||
//
|
||||
// ## 这些判据钉住的行为
|
||||
//
|
||||
// ① text 与 content 都空 ⇒ 明确报错(不能静默写空剪切板)
|
||||
// ② content 指向不存在的文件 ⇒ 报错且**带上路径**(否则模型无从修正)
|
||||
// ③ content 超上限 ⇒ 报错(防止把会话撑爆)
|
||||
// ④ 设备不可用时优先报设备错,而不是先读文件
|
||||
// ——顺序很重要:不然一个离线设备会伪装成「文件不存在」
|
||||
//
|
||||
// 注:本文件只验**参数校验与错误路径**。真正读文件并下发需要一台在线设备,
|
||||
// 那由 binary_test.go 的 WS 级联调覆盖。
|
||||
|
||||
// newOfflineDevice 造一个 device,其 clipboardsue 必然在 clipboardCheck 阶段失败。
|
||||
func newOfflineDevice(t *testing.T) *devicectlDevice {
|
||||
t.Helper()
|
||||
return &devicectlDevice{reg: NewRegistry()}
|
||||
}
|
||||
|
||||
func TestClipboardsue_EmptyTextAndContentRejected(t *testing.T) {
|
||||
d := newOfflineDevice(t)
|
||||
_, err := d.clipboardsue(map[string]interface{}{"device_id": "x"})
|
||||
if err == nil {
|
||||
t.Fatal("text 与 content 都为空时应当报错(不能静默写空剪切板)")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "text") && !strings.Contains(err.Error(), "content") {
|
||||
t.Errorf("错误信息应指明缺哪个参数,实际: %v", err)
|
||||
}
|
||||
// 只有空白也视为空
|
||||
if _, err2 := d.clipboardsue(map[string]interface{}{
|
||||
"device_id": "x", "text": " ", "content": " ",
|
||||
}); err2 == nil {
|
||||
t.Error("空白 text/content 也应报错")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClipboardsue_DeviceCheckedBeforeReadingFile(t *testing.T) {
|
||||
// 设备不存在 ⇒ 必须先报「设备不存在」,
|
||||
// 而不是先读文件报「文件不存在」——否则离线设备会伪装成文件问题。
|
||||
d := newOfflineDevice(t)
|
||||
_, err := d.clipboardsue(map[string]interface{}{
|
||||
"device_id": "no-such-device",
|
||||
"content": filepath.Join(t.TempDir(), "nope.txt"),
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("设备不存在时应当报错")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no-such-device") {
|
||||
t.Errorf("应先报设备问题(设备 %s 不存在),实际: %v", "no-such-device", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClipboardsueMaxBytesIsTwoMB(t *testing.T) {
|
||||
// 上限与 GUI 侧 CAPABILITY_SOURCE_MAX_BYTES 对齐(2MB)。
|
||||
// 改这个常量时这条判据会红——提醒同步改 GUI。
|
||||
if clipboardsueMaxBytes != 2*1024*1024 {
|
||||
t.Errorf("clipboardsue 上限应为 2MB(与 GUI 侧一致),实际 %d", clipboardsueMaxBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClipboardsue_ContentFileReadable(t *testing.T) {
|
||||
// 造一个真文件,确认 os.ReadFile 能读到(判据本身的自检:
|
||||
// 若将来换成别的读法或路径处理变了,这里会先炸)。
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "note.txt")
|
||||
want := "行1\n行2\n"
|
||||
if err := os.WriteFile(p, []byte(want), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatalf("读取失败: %v", err)
|
||||
}
|
||||
if string(b) != want {
|
||||
t.Errorf("读回内容不符: %q", string(b))
|
||||
}
|
||||
}
|
||||
|
||||
// TestClipboardsue_ContentOversizeRejected 钉住**上限真的被用上**。
|
||||
//
|
||||
// ★ 这条判据是补来的:我先前只写了「常量等于 2MB」那条,
|
||||
//
|
||||
// 把 `if len(b) > clipboardsueMaxBytes` 改成 `if false` 它照样绿——
|
||||
// **测的是常量、不是闸**。凡是要验「闸真的关着」,
|
||||
// 必须造一个越过阈值的数据去撞它,而不是断言阈值的数值。
|
||||
func TestClipboardsue_ContentOversizeRejected(t *testing.T) {
|
||||
// 造一个在线且已授权的设备,让 clipboardCheck 过;
|
||||
// 随后超限文件必须在**读文件后的校验**处被拦下。
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "big.txt")
|
||||
big := make([]byte, clipboardsueMaxBytes+1)
|
||||
for i := range big {
|
||||
big[i] = 'x'
|
||||
}
|
||||
if err := os.WriteFile(p, big, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reg := NewRegistry()
|
||||
// 注册一个在线设备并授 capability:clipboardsue。
|
||||
reg.devices["d1"] = &DeviceMeta{
|
||||
DeviceID: "d1", Name: "d", Kind: "computer", Online: true,
|
||||
Caps: []string{"clipboardsue"},
|
||||
}
|
||||
d := &devicectlDevice{reg: reg}
|
||||
|
||||
_, err := d.clipboardsue(map[string]interface{}{
|
||||
"device_id": "d1",
|
||||
"content": p,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("超 2MB 的 content 应当被拒绝(把上限改成 if false 时这条会红)")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "过大") && !strings.Contains(err.Error(), "上限") {
|
||||
t.Errorf("错误信息应说明超限,实际: %v", err)
|
||||
}
|
||||
}
|
||||
@ -1,348 +0,0 @@
|
||||
{
|
||||
"meta": {
|
||||
"name": "NapCat WebUI Design DNA",
|
||||
"description": "Design DNA extracted from NapCat WebUI (napcat-webui-frontend) — ACG/cherry-blossom aesthetic with glassmorphism, motion/react animations, HeroUI components, sakura/frost color palette",
|
||||
"source_references": [
|
||||
"https://github.com/NapNeko/NapCatQQ/tree/main/packages/napcat-webui-frontend",
|
||||
"packages/napcat-webui-frontend/tailwind.config.js",
|
||||
"packages/napcat-webui-frontend/src/styles/globals.css",
|
||||
"packages/napcat-webui-frontend/src/components/page_background.tsx",
|
||||
"packages/napcat-webui-frontend/src/components/effect_card.tsx",
|
||||
"packages/napcat-webui-frontend/src/layouts/default.tsx",
|
||||
"packages/napcat-webui-frontend/src/layouts/pure.tsx",
|
||||
"packages/napcat-webui-frontend/src/pages/web_login.tsx",
|
||||
"packages/napcat-webui-frontend/src/pages/dashboard/index.tsx",
|
||||
"packages/napcat-webui-frontend/src/components/sidebar/index.tsx",
|
||||
"packages/napcat-webui-frontend/src/components/display_network_item.tsx",
|
||||
"packages/napcat-webui-frontend/src/components/system_status_display.tsx",
|
||||
"packages/napcat-webui-frontend/src/config/site.tsx"
|
||||
],
|
||||
"created_at": "2026-08-12T00:00:00Z"
|
||||
},
|
||||
"design_system": {
|
||||
"color": {
|
||||
"palette_type": "split-complementary",
|
||||
"primary": {
|
||||
"hex": "#FF7FAC",
|
||||
"role": "Sakura pink - dominant accent, CTAs, focus rings, interactive elements"
|
||||
},
|
||||
"secondary": {
|
||||
"hex": "#88C0D0",
|
||||
"role": "Frost blue - supporting accent, secondary actions, info states"
|
||||
},
|
||||
"accent": {
|
||||
"hex": "#F33B7C",
|
||||
"role": "Deep rose - hover/active primary, destructive actions"
|
||||
},
|
||||
"neutral": {
|
||||
"scale": {
|
||||
"50": "#FFF0F5",
|
||||
"100": "#FFE4E9",
|
||||
"200": "#FFCDD9",
|
||||
"300": "#FF9EB5",
|
||||
"400": "#FF7FAC",
|
||||
"500": "#F33B7C",
|
||||
"600": "#C92462",
|
||||
"700": "#991B4B",
|
||||
"800": "#691233",
|
||||
"900": "#380A1B"
|
||||
},
|
||||
"usage": "Pink-tinted neutrals for light mode; inverted for dark mode"
|
||||
},
|
||||
"semantic": {
|
||||
"success": "#17a964",
|
||||
"warning": "#b7791f",
|
||||
"error": "#DB3694",
|
||||
"info": "#3f6ef5"
|
||||
},
|
||||
"surface": {
|
||||
"background": "gradient-to-br from-indigo-50 via-white to-pink-50 (light) / dark:from-gray-900 dark:via-gray-800 dark:to-gray-900",
|
||||
"card": "bg-white/60 dark:bg-black/40 with backdrop-blur-sm border border-white/40 dark:border-white/10",
|
||||
"elevated": "bg-white/70 dark:bg-black/30 hover:bg-white/80 dark:hover:bg-black/20 with shadow-sm"
|
||||
},
|
||||
"contrast_strategy": "subtle layers with glassmorphism — semi-transparent cards over gradient backgrounds, white/black text with opacity modulation"
|
||||
},
|
||||
"typography": {
|
||||
"type_scale": {
|
||||
"display": {
|
||||
"size": "4xl-5xl (36-48px)",
|
||||
"weight": "bold",
|
||||
"line_height": "1.1",
|
||||
"tracking": "-0.02em"
|
||||
},
|
||||
"heading_1": {
|
||||
"size": "2xl-3xl (24-30px)",
|
||||
"weight": "semibold",
|
||||
"line_height": "1.2",
|
||||
"tracking": "-0.02em"
|
||||
},
|
||||
"heading_2": {
|
||||
"size": "xl-2xl (20-24px)",
|
||||
"weight": "semibold",
|
||||
"line_height": "1.3",
|
||||
"tracking": "-0.01em"
|
||||
},
|
||||
"heading_3": {
|
||||
"size": "lg-xl (18-20px)",
|
||||
"weight": "medium",
|
||||
"line_height": "1.4",
|
||||
"tracking": "0"
|
||||
},
|
||||
"body": {
|
||||
"size": "base (16px)",
|
||||
"weight": "normal",
|
||||
"line_height": "1.55",
|
||||
"tracking": "0.02em"
|
||||
},
|
||||
"body_small": {
|
||||
"size": "sm (14px)",
|
||||
"weight": "normal",
|
||||
"line_height": "1.5",
|
||||
"tracking": "0.01em"
|
||||
},
|
||||
"caption": {
|
||||
"size": "xs (12px)",
|
||||
"weight": "normal",
|
||||
"line_height": "1.4",
|
||||
"tracking": "0.02em"
|
||||
},
|
||||
"overline": {
|
||||
"size": "xs (12px)",
|
||||
"weight": "medium",
|
||||
"line_height": "1.3",
|
||||
"tracking": "0.08em"
|
||||
}
|
||||
},
|
||||
"font_families": {
|
||||
"heading": "'Quicksand', 'Nunito', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Microsoft YaHei', sans-serif",
|
||||
"body": "'Quicksand', 'Nunito', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'PingFang SC', 'Microsoft YaHei', sans-serif",
|
||||
"mono": "'JetBrains Mono', 'ui-monospace', 'SFMono-Regular', 'SF Mono', 'Menlo', 'Consolas', 'Liberation Mono', monospace"
|
||||
},
|
||||
"font_style_notes": "Rounded geometric sans (Quicksand/Nunito) for friendly ACG feel; JetBrains Mono for code/terminal; letter-spacing 0.02em body, negative for headings; -webkit-font-smoothing antialiased"
|
||||
},
|
||||
"spacing": {
|
||||
"base_unit": "4px (0.25rem)",
|
||||
"scale": [0, 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32, 40, 48, 64],
|
||||
"content_density": "comfortable with generous breathing room; cards use p-4/p-5/p-6; sections gap-4/gap-6",
|
||||
"section_rhythm": "Consistent 4-unit (16px) base rhythm; md:gap-4 (16px) / lg:gap-6 (24px) for grid gaps; py-5 (20px) vertical section padding"
|
||||
},
|
||||
"layout": {
|
||||
"grid_system": "CSS Grid + Flexbox hybrid; 8-col mobile / 3-col md / 6-col lg for dashboards; sidebar 16rem (256px) fixed",
|
||||
"max_content_width": "1000px (md:max-w-[1000px]) for main content; full-width for login/pure layouts",
|
||||
"columns": "Responsive: 1col < 3col md < 6col lg",
|
||||
"gutter": "gap-1 (4px) sm / gap-4 (16px) md",
|
||||
"breakpoints": "sm: 640px, md: 768px, lg: 1024px, xl: 1280px (Tailwind defaults)",
|
||||
"alignment_tendency": "centered for auth/pure layouts; left-aligned grid for dashboards; sticky top breadcrumbs"
|
||||
},
|
||||
"shape": {
|
||||
"border_radius": {
|
||||
"small": "0.375rem (6px) - inputs, buttons",
|
||||
"medium": "0.5rem (8px) - cards, modals",
|
||||
"large": "0.75rem (12px) - HeroUI default radius (--heroui-radius: 0.75rem)",
|
||||
"pill": "9999px (full) - primary CTAs, pills, tags"
|
||||
},
|
||||
"border_usage": "subtle 1px border-white/40 dark:border-white/10 on glass cards; no hard borders elsewhere",
|
||||
"divider_style": "hairline via backdrop-blur edges; HeroUI divider component for lists"
|
||||
},
|
||||
"elevation": {
|
||||
"shadow_style": "soft diffused with backdrop-blur; layering via transparency not heavy shadows",
|
||||
"levels": {
|
||||
"low": "shadow-sm (0 1px 2px rgba(0,0,0,0.05)) + backdrop-blur-sm",
|
||||
"medium": "shadow-md (0 4px 6px rgba(0,0,0,0.07)) + backdrop-blur-md",
|
||||
"high": "shadow-xl (0 20px 25px rgba(0,0,0,0.1)) + backdrop-blur-xl + backdrop-saturate-200"
|
||||
},
|
||||
"depth_cues": ["backdrop-blur", "transparency/opacity", "subtle borders", "hover elevation increase", "motion scale on interaction"]
|
||||
},
|
||||
"iconography": {
|
||||
"style": "Lucide React (Lu*) icons — clean 2px stroke, rounded caps; react-icons for brand icons (Gi, Bi, Io, Md)",
|
||||
"stroke_weight": "2px equivalent (w-5 h-5 default, w-4 h-4 small, w-6 h-6 large)",
|
||||
"size_scale": "xs: 14px, sm: 16px, md: 20px, lg: 24px, xl: 32px",
|
||||
"preferred_set": "Lucide React for UI; react-icons for brand/logos"
|
||||
},
|
||||
"motion": {
|
||||
"easing": "spring (stiffness: 120-150, damping: 15-20) for layout; ease-out for micro-interactions",
|
||||
"duration_scale": {
|
||||
"micro": "150-200ms (hover, focus, tap)",
|
||||
"normal": "300-400ms (page transitions, sidebar, modals)",
|
||||
"macro": "500ms (initial entrance, spring animations)"
|
||||
},
|
||||
"entrance_pattern": "fade-up (opacity 0→1, y: 20→0) + scale (0.95→1) with spring; staggered children via AnimatePresence",
|
||||
"exit_pattern": "fade-down (opacity 1→0, y: 0→10) fast tween (150ms)",
|
||||
"philosophy": "playful spring physics for layout; snappy functional for interactions; motion/redux (framer-motion) for complex choreography"
|
||||
},
|
||||
"components": {
|
||||
"button_style": "HeroUI Button variants: shadow (primary CTA), flat (secondary), light (ghost), bordered; pill radius (full) for primary; size lg with py-7; startContent icons; isLoading state",
|
||||
"input_style": "HeroUI Input: backdrop-blur-xl bg-transparent innerWrapper; shadow-xl inputWrapper with focus glow; radius lg; startContent icons; isClearable for token fields",
|
||||
"card_style": "HeroUI Card: backdrop-blur-sm border border-white/40 dark:border-white/10 shadow-sm; bg-white/60 dark:bg-black/40; hover elevation; optional background image overlay with opacity-30",
|
||||
"navigation_pattern": "Collapsible sidebar (16rem→0) with spring animation; breadcrumbs with motion stagger; mobile overlay backdrop-blur",
|
||||
"modal_style": "HeroUI Dialog/Modal with backdrop-blur; HoverEffectCard for login (3D tilt on hover); spring entrance",
|
||||
"list_style": "Network item cards: grid layout, large mono count + label; System status: grid 2-col with icon headers; UsagePie canvas charts",
|
||||
"component_notes": "HoverEffectCard: 3D perspective tilt (rotateX/rotateY) + moving gradient light blob on hover; PageBackground: static gradient blobs (primary/secondary/pink) with massive blur; Toast: react-hot-toast top-right"
|
||||
}
|
||||
},
|
||||
"design_style": {
|
||||
"aesthetic": {
|
||||
"mood": ["playful", "dreamy", "technical", "friendly", "modern"],
|
||||
"visual_metaphor": "Cherry blossoms in digital frost — ACG aesthetic meets developer tooling",
|
||||
"era_influence": "Modern web (2020s) with anime/moe culture influences; glassmorphism 2.0",
|
||||
"genre": "Developer dashboard with ACG personality",
|
||||
"personality_traits": ["approachable", "meticulous", "whimsical", "competent"],
|
||||
"adjectives": ["sakura", "frost", "glass", "animated", "responsive"]
|
||||
},
|
||||
"visual_language": {
|
||||
"complexity": "moderate — rich but not cluttered; breathing room via glass cards",
|
||||
"ornamentation": "subtle accents — gradient blobs, 3D tilt cards, animated breadcrumbs, scrollbar theming",
|
||||
"whitespace_usage": "generous — cards have internal padding, grid gaps 16-24px, section padding 20px+",
|
||||
"visual_weight_distribution": "balanced — sidebar + main content; dashboard cards equal weight; login centered hero",
|
||||
"focal_strategy": "progressive reveal — entrance animations, hover discoveries, layered depth",
|
||||
"contrast_level": "medium — relies on opacity layers not high contrast; dark mode inverts surfaces",
|
||||
"texture_usage": "glassmorphism (backdrop-blur), gradient mesh blobs, subtle noise via saturation"
|
||||
},
|
||||
"composition": {
|
||||
"hierarchy_method": "scale contrast + typographic hierarchy + spatial isolation (cards)",
|
||||
"balance_type": "asymmetric — sidebar fixed, main fluid; dashboard grid balanced",
|
||||
"flow_direction": "vertical primary (sidebar→content), horizontal secondary (breadcrumbs, card grids)",
|
||||
"grouping_strategy": "cards as containers; grid for peer items; sections separated by gap",
|
||||
"negative_space_role": "breathing room between cards; focus isolation for modals/login"
|
||||
},
|
||||
"imagery": {
|
||||
"photo_treatment": "custom background image upload (base64) with overlay opacity-30 object-contain",
|
||||
"illustration_style": "ACG/moe logo (NapCat cat), gradient blob art",
|
||||
"graphic_elements": "gradient orbs (primary/secondary/pink), moving light blob on hover cards",
|
||||
"pattern_usage": "none explicit; glassmorphism creates implied pattern",
|
||||
"image_shape": "full-bleed background; logo 7em height on login"
|
||||
},
|
||||
"interaction_feel": {
|
||||
"feedback_style": "immediate visual — 3D tilt on cards, light blob follows cursor, spring scale on buttons",
|
||||
"hover_behavior": "card: perspective rotateX/rotateY + gradient light follows mouse; button: shadow elevation + scale; input: focus glow ring",
|
||||
"transition_personality": "spring physics (stiffness 120-150) for layout; snappy ease-out for micro",
|
||||
"loading_style": "PageLoading full-screen overlay; Skeleton cards for dashboard; Suspense boundaries per route",
|
||||
"microinteraction_density": "high — every interactive element has hover/tap/transition choreography"
|
||||
},
|
||||
"brand_voice_in_ui": {
|
||||
"tone": "friendly technical — '猫猫日志', '喵~', emoji in hints",
|
||||
"formality": "casual developer — Chinese labels with English terms mixed",
|
||||
"cta_style": "direct imperative with playful twist — '登录' with logo animation, '切换主题', '重启进程'",
|
||||
"empty_state_approach": "illustrated (ring logo) + welcoming copy + hint",
|
||||
"error_tone": "helpful not alarming — toast errors, confirm dialogs for destructive actions"
|
||||
}
|
||||
},
|
||||
"visual_effects": {
|
||||
"overview": {
|
||||
"effect_intensity": "moderate",
|
||||
"performance_tier": "medium",
|
||||
"fallback_strategy": "reduce to CSS (disable motion spring, keep glassmorphism)",
|
||||
"primary_technology": "CSS (Tailwind) + motion/react (Framer Motion) + Canvas 2D (UsagePie charts) + HeroUI"
|
||||
},
|
||||
"background_effects": {
|
||||
"type": "gradient-mesh-blobs",
|
||||
"description": "Three large radial gradient blobs (primary-200/40, secondary-200/40, pink-200/30) with massive blur (90-110px) positioned at corners, static but imply depth",
|
||||
"technology": "CSS (absolute positioned divs with blur)",
|
||||
"params": {
|
||||
"color_palette": ["#FF7FAC", "#88C0D0", "#FF9EB5"],
|
||||
"speed": "static",
|
||||
"density": "3 blobs",
|
||||
"opacity": "0.3-0.4",
|
||||
"blend_mode": "normal (opacity stacking)"
|
||||
}
|
||||
},
|
||||
"particle_systems": {
|
||||
"enabled": false
|
||||
},
|
||||
"3d_elements": {
|
||||
"enabled": true,
|
||||
"type": "card-3d-tilt",
|
||||
"description": "HoverEffectCard uses CSS 3D transforms (perspective 1000px, rotateX/rotateY up to ±5deg) with a moving gradient light blob that follows cursor",
|
||||
"technology": "CSS 3D + React mouse tracking",
|
||||
"params": {
|
||||
"renderer": "CSS transform3d",
|
||||
"lighting": "moving gradient blob (from-primary-400 to-secondary-400) follows cursor",
|
||||
"camera": "perspective(1000px)",
|
||||
"materials": "glassmorphism card surface",
|
||||
"geometry": "flat card with perspective rotation",
|
||||
"post_processing": "blur-[100px] on light blob",
|
||||
"interaction_model": "mouse-move tracking within card bounds"
|
||||
}
|
||||
},
|
||||
"shader_effects": {
|
||||
"enabled": false
|
||||
},
|
||||
"scroll_effects": {
|
||||
"parallax": {
|
||||
"enabled": false
|
||||
},
|
||||
"scroll_triggered_animations": {
|
||||
"enabled": true,
|
||||
"trigger_points": "route mount (Suspense), sidebar open/close, breadcrumb items",
|
||||
"animation_type": "fade-up + scale (spring stagger)",
|
||||
"scrub_behavior": "none (time-based)"
|
||||
},
|
||||
"scroll_morphing": {
|
||||
"enabled": false
|
||||
}
|
||||
},
|
||||
"text_effects": {
|
||||
"type": "gradient-text",
|
||||
"description": "HeroUI title component supports color='violet' for gradient text; login page 'Web Login' split with violet accent",
|
||||
"technology": "CSS background-clip-text via HeroUI",
|
||||
"params": {
|
||||
"split_strategy": "by-word (Web / Login)",
|
||||
"animation_per_unit": "none",
|
||||
"stagger": "none",
|
||||
"effect_style": "gradient fill"
|
||||
}
|
||||
},
|
||||
"cursor_effects": {
|
||||
"enabled": true,
|
||||
"type": "magnetic-light-blob",
|
||||
"description": "On HoverEffectCard, a gradient radial blur blob follows cursor position within card bounds",
|
||||
"technology": "CSS absolute positioned div + React mouse tracking",
|
||||
"params": {
|
||||
"shape": "rounded-full",
|
||||
"size": "150px",
|
||||
"blend_mode": "normal",
|
||||
"trail": "opacity transition 300ms",
|
||||
"interaction_zone": "card bounding box"
|
||||
}
|
||||
},
|
||||
"image_effects": {
|
||||
"type": "background-overlay",
|
||||
"description": "User-uploaded background image rendered full-bleed behind layout with opacity-30 object-contain",
|
||||
"technology": "CSS background-image on layout root",
|
||||
"params": {
|
||||
"filter_pipeline": "opacity-30",
|
||||
"hover_transform": "none",
|
||||
"reveal_animation": "none",
|
||||
"distortion_type": "none"
|
||||
}
|
||||
},
|
||||
"glassmorphism_neumorphism": {
|
||||
"enabled": true,
|
||||
"style": "glass",
|
||||
"params": {
|
||||
"blur_radius": "backdrop-blur-sm (4px) to backdrop-blur-xl (24px)",
|
||||
"transparency": "bg-white/10-70 dark:bg-black/10-40",
|
||||
"border_treatment": "border border-white/40 dark:border-white/10",
|
||||
"shadow_type": "shadow-sm to shadow-xl layered",
|
||||
"light_source_angle": "implied top-left via gradient blobs"
|
||||
}
|
||||
},
|
||||
"canvas_drawings": {
|
||||
"enabled": true,
|
||||
"type": "data-visualization-pie",
|
||||
"description": "UsagePie component renders CPU/memory donut charts using Canvas 2D with animated sweep",
|
||||
"technology": "Canvas 2D API",
|
||||
"params": {
|
||||
"draw_method": "arc segments with gradient fill",
|
||||
"animation_loop": "on mount sweep animation",
|
||||
"color_scheme": "primary/secondary/palette",
|
||||
"responsiveness": "container-sized",
|
||||
"interaction": "none (static display)"
|
||||
}
|
||||
},
|
||||
"svg_animations": {
|
||||
"enabled": false
|
||||
},
|
||||
"composite_notes": "The design layers: 1) Static gradient mesh background (CSS), 2) Glass cards with backdrop-blur (CSS), 3) 3D tilt cards on hover (CSS 3D + React), 4) Magnetic light blob follows cursor (React + CSS), 5) Spring animations for layout transitions (motion/react), 6) Canvas pie charts for data viz, 7) Themed scrollbars (CSS), 8) Custom background image overlay (CSS). All effects are lightweight — no WebGL, no heavy JS animation libraries beyond motion/react. HeroUI provides accessible component primitives. Dark mode inverts surfaces and adjusts blob colors. Performance is good on modern browsers; fallback is static CSS (disable motion, keep glass)."
|
||||
}
|
||||
}
|
||||
@ -1,254 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/document"
|
||||
"gitcode.com/JianFeeeee/HomeAgent/internal/memory/vector"
|
||||
)
|
||||
|
||||
type benchDoc struct {
|
||||
ID string `json:"id"`
|
||||
Summary string `json:"summary"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type queryCase struct {
|
||||
Name string
|
||||
Query string
|
||||
Seeds []string
|
||||
}
|
||||
|
||||
type hit struct {
|
||||
ID string `json:"id"`
|
||||
Score float64 `json:"score"`
|
||||
}
|
||||
|
||||
type result struct {
|
||||
Name string `json:"name"`
|
||||
Query string `json:"query"`
|
||||
Relevant int `json:"relevant"`
|
||||
Rank int `json:"rank"`
|
||||
Reciprocal float64 `json:"reciprocal_rank"`
|
||||
HitAt1 bool `json:"hit_at_1"`
|
||||
HitAt5 bool `json:"hit_at_5"`
|
||||
LatencyMS float64 `json:"latency_ms"`
|
||||
Top []hit `json:"top"`
|
||||
}
|
||||
|
||||
type report struct {
|
||||
Documents int `json:"documents"`
|
||||
Queries []queryCase `json:"queries"`
|
||||
Methods map[string][]result `json:"methods"`
|
||||
}
|
||||
|
||||
var cases = []queryCase{
|
||||
{Name: "mail-semantic", Query: "邮件代理是否已经成功接入", Seeds: []string{"AgentMail 接入验证"}},
|
||||
{Name: "fox-cross-language", Query: "生成一张雪地红狐狸的图片", Seeds: []string{"red fox in snowy forest"}},
|
||||
{Name: "plugin-semantic", Query: "升级安装 QQ 插件包", Seeds: []string{"plugin_install"}},
|
||||
{Name: "weather-paraphrase", Query: "我所在城市的天气预报", Seeds: []string{"河南新乡"}},
|
||||
{Name: "textarea-paraphrase", Query: "聊天输入区域文字多了会不会自动增高", Seeds: []string{"输入框在内容超过一行"}},
|
||||
{Name: "devices-paraphrase", Query: "检查当前接入了哪些终端设备", Seeds: []string{"你看看现在你都有哪些设备"}},
|
||||
{Name: "memory-health", Query: "长期文档记忆功能是否健康", Seeds: []string{"文档记忆系统是否正常工作"}},
|
||||
{Name: "reload-plugins", Query: "重新加载全部扩展组件", Seeds: []string{"热重载所有插件", "plgreload"}},
|
||||
{Name: "exact-agentmail", Query: "AgentMail 接入验证", Seeds: []string{"AgentMail 接入验证"}},
|
||||
{Name: "exact-plugin", Query: "plugin_install", Seeds: []string{"plugin_install"}},
|
||||
}
|
||||
|
||||
func main() {
|
||||
docs, err := loadDocs("/data/homeagent/memory/documents")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "loaded %d production documents\n", len(docs))
|
||||
|
||||
rel := relevantSets(docs)
|
||||
for i, c := range cases {
|
||||
fmt.Fprintf(os.Stderr, "case %-20s relevant=%d query=%q\n", c.Name, len(rel[i]), c.Query)
|
||||
}
|
||||
|
||||
r := report{Documents: len(docs), Queries: cases, Methods: make(map[string][]result)}
|
||||
|
||||
// 方案 A:纯 TF-IDF。完整训练在生产文档上,保留 IDF 高频抑制与倒排候选剪枝。
|
||||
tfidf := vector.NewTFIDFVectorizer(memory.TokenizeWords)
|
||||
texts := make([]string, len(docs))
|
||||
for i, d := range docs {
|
||||
texts[i] = d.Summary + "\n" + d.Content
|
||||
}
|
||||
tfidf.Train(texts)
|
||||
tfStore := buildStore(docs, tfidf)
|
||||
r.Methods["tfidf"] = runCases(tfStore, tfidf, rel)
|
||||
|
||||
// 方案 B:当前生产 fastText(中英各 20/37 万词,300 维平均词向量)。
|
||||
fast := memory.NewStaticEmbedder("/data/cc.zh.top200k.vec", "/data/cc.en.top200k.vec")
|
||||
fastStore := buildStore(docs, fast)
|
||||
r.Methods["fasttext"] = runCases(fastStore, fast, rel)
|
||||
|
||||
// 方案 C:旧通道混合。RRF 不要求两种分数处于同一标尺,避免拍脑袋设绝对权重。
|
||||
r.Methods["tfidf_fasttext_rrf"] = runHybrid(tfStore, tfidf, fastStore, fast, rel)
|
||||
|
||||
out, _ := json.MarshalIndent(r, "", " ")
|
||||
if err := os.WriteFile("/tmp/homeagent-old-retrieval.json", out, 0644); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
printSummary(r)
|
||||
}
|
||||
|
||||
func loadDocs(dir string) ([]benchDoc, error) {
|
||||
ents, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var docs []benchDoc
|
||||
for _, e := range ents {
|
||||
if e.IsDir() || !strings.HasPrefix(e.Name(), "doc_") || filepath.Ext(e.Name()) != ".json" {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, e.Name()))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var d document.Doc
|
||||
if json.Unmarshal(b, &d) != nil || d.ID == "" {
|
||||
continue
|
||||
}
|
||||
docs = append(docs, benchDoc{ID: d.ID, Summary: d.Summary, Content: d.Content})
|
||||
}
|
||||
sort.Slice(docs, func(i, j int) bool { return docs[i].ID < docs[j].ID })
|
||||
return docs, nil
|
||||
}
|
||||
|
||||
func relevantSets(docs []benchDoc) []map[string]bool {
|
||||
sets := make([]map[string]bool, len(cases))
|
||||
for i, c := range cases {
|
||||
sets[i] = make(map[string]bool)
|
||||
for _, d := range docs {
|
||||
text := strings.ToLower(d.Summary + "\n" + d.Content)
|
||||
for _, seed := range c.Seeds {
|
||||
if strings.Contains(text, strings.ToLower(seed)) {
|
||||
sets[i][d.ID] = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return sets
|
||||
}
|
||||
|
||||
type textVectorizer interface {
|
||||
Vectorize(text string) vector.Vector
|
||||
}
|
||||
|
||||
func buildStore(docs []benchDoc, v textVectorizer) *vector.Store {
|
||||
s := vector.NewStore()
|
||||
for _, d := range docs {
|
||||
s.Insert(d.ID, d.Summary, v.Vectorize(d.Summary+"\n"+d.Content), nil)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func runCases(s *vector.Store, v textVectorizer, rel []map[string]bool) []result {
|
||||
out := make([]result, 0, len(cases))
|
||||
for i, c := range cases {
|
||||
start := time.Now()
|
||||
hs := s.SearchScored(v.Vectorize(c.Query), s.Size())
|
||||
lat := time.Since(start)
|
||||
ids := make([]hit, len(hs))
|
||||
for j, h := range hs {
|
||||
ids[j] = hit{ID: h.Doc.ID, Score: h.Score}
|
||||
}
|
||||
out = append(out, measure(c, ids, rel[i], lat))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func runHybrid(a *vector.Store, av textVectorizer, b *vector.Store, bv textVectorizer, rel []map[string]bool) []result {
|
||||
out := make([]result, 0, len(cases))
|
||||
for i, c := range cases {
|
||||
start := time.Now()
|
||||
ah := a.SearchScored(av.Vectorize(c.Query), a.Size())
|
||||
bh := b.SearchScored(bv.Vectorize(c.Query), b.Size())
|
||||
scores := make(map[string]float64)
|
||||
const k = 60.0
|
||||
for rank, h := range ah {
|
||||
scores[h.Doc.ID] += 1 / (k + float64(rank+1))
|
||||
}
|
||||
for rank, h := range bh {
|
||||
scores[h.Doc.ID] += 1 / (k + float64(rank+1))
|
||||
}
|
||||
ids := make([]hit, 0, len(scores))
|
||||
for id, score := range scores {
|
||||
ids = append(ids, hit{ID: id, Score: score})
|
||||
}
|
||||
sort.Slice(ids, func(i, j int) bool {
|
||||
if ids[i].Score == ids[j].Score {
|
||||
return ids[i].ID < ids[j].ID
|
||||
}
|
||||
return ids[i].Score > ids[j].Score
|
||||
})
|
||||
out = append(out, measure(c, ids, rel[i], time.Since(start)))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func measure(c queryCase, ranked []hit, relevant map[string]bool, latency time.Duration) result {
|
||||
rank := 0
|
||||
for i, h := range ranked {
|
||||
if relevant[h.ID] {
|
||||
rank = i + 1
|
||||
break
|
||||
}
|
||||
}
|
||||
topN := 5
|
||||
if len(ranked) < topN {
|
||||
topN = len(ranked)
|
||||
}
|
||||
r := result{Name: c.Name, Query: c.Query, Relevant: len(relevant), Rank: rank, LatencyMS: float64(latency.Microseconds()) / 1000, Top: append([]hit(nil), ranked[:topN]...)}
|
||||
if rank > 0 {
|
||||
r.Reciprocal = 1 / float64(rank)
|
||||
r.HitAt1 = rank <= 1
|
||||
r.HitAt5 = rank <= 5
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func printSummary(r report) {
|
||||
fmt.Printf("documents=%d queries=%d\n", r.Documents, len(r.Queries))
|
||||
names := make([]string, 0, len(r.Methods))
|
||||
for name := range r.Methods {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
rs := r.Methods[name]
|
||||
var h1, h5 int
|
||||
var mrr, lat float64
|
||||
for _, x := range rs {
|
||||
if x.HitAt1 {
|
||||
h1++
|
||||
}
|
||||
if x.HitAt5 {
|
||||
h5++
|
||||
}
|
||||
mrr += x.Reciprocal
|
||||
lat += x.LatencyMS
|
||||
}
|
||||
fmt.Printf("%-24s Hit@1=%d/%d Hit@5=%d/%d MRR=%.4f avg-query=%.3fms\n", name, h1, len(rs), h5, len(rs), mrr/float64(len(rs)), lat/float64(len(rs)))
|
||||
for _, x := range rs {
|
||||
fmt.Printf(" %-20s rank=%-4d latency=%7.3fms", x.Name, x.Rank, x.LatencyMS)
|
||||
if len(x.Top) > 0 {
|
||||
fmt.Printf(" top=%s score=%.4g", x.Top[0].ID, x.Top[0].Score)
|
||||
}
|
||||
fmt.Println()
|
||||
}
|
||||
}
|
||||
_ = math.MaxFloat64
|
||||
}
|
||||
204
tmp_jinabench.py
204
tmp_jinabench.py
@ -1,204 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Real Jina v5-omni benchmark on HomeAgent production documents and media.
|
||||
|
||||
Inputs never leave the host. Reports Recall/Hit@K, MRR, margins, and latency for:
|
||||
- Jina unified dense space: text query -> text document
|
||||
- Jina unified dense space: text query -> production image
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from PIL import Image
|
||||
from transformers import AutoModel, AutoProcessor
|
||||
|
||||
MODEL = "/home/newqqagent/models/jina-v5-omni-nano"
|
||||
DOC_DIR = "/home/newqqagent/memory/documents"
|
||||
MEDIA_DB = "/home/newqqagent/memory/media/media.db"
|
||||
BLOB_ROOT = "/home/newqqagent/memory/media/blobs"
|
||||
|
||||
CASES = [
|
||||
("mail-semantic", "邮件代理是否已经成功接入", ["AgentMail 接入验证"]),
|
||||
("fox-cross-language", "生成一张雪地红狐狸的图片", ["red fox in snowy forest"]),
|
||||
("plugin-semantic", "升级安装 QQ 插件包", ["plugin_install"]),
|
||||
("weather-paraphrase", "我所在城市的天气预报", ["河南新乡"]),
|
||||
("textarea-paraphrase", "聊天输入区域文字多了会不会自动增高", ["输入框在内容超过一行"]),
|
||||
("devices-paraphrase", "检查当前接入了哪些终端设备", ["你看看现在你都有哪些设备"]),
|
||||
("memory-health", "长期文档记忆功能是否健康", ["文档记忆系统是否正常工作"]),
|
||||
("reload-plugins", "重新加载全部扩展组件", ["热重载所有插件", "plgreload"]),
|
||||
("exact-agentmail", "AgentMail 接入验证", ["AgentMail 接入验证"]),
|
||||
("exact-plugin", "plugin_install", ["plugin_install"]),
|
||||
]
|
||||
|
||||
MEDIA_QUERIES = [
|
||||
("captcha-cn", "一张带有干扰线和字符的验证码图片", "7e689211"),
|
||||
("captcha-en", "captcha with distorted letters and noise", "7e689211"),
|
||||
("news-cn", "中文新闻报道页面截图", "f6f17229"),
|
||||
("news-en", "a screenshot of a Chinese news article", "f6f17229"),
|
||||
("notes-cn", "手机深色模式备忘录截图", "ee5bdb2a"),
|
||||
("notes-en", "dark mode phone notes application screenshot", "ee5bdb2a"),
|
||||
]
|
||||
|
||||
|
||||
def load_docs():
|
||||
docs = []
|
||||
for p in sorted(glob.glob(DOC_DIR + "/doc_*.json")):
|
||||
try:
|
||||
d = json.load(open(p, encoding="utf-8"))
|
||||
except Exception:
|
||||
continue
|
||||
if not d.get("id"):
|
||||
continue
|
||||
docs.append({"id": d["id"], "text": d.get("summary", "") + "\n" + d.get("content", "")})
|
||||
return docs
|
||||
|
||||
|
||||
def relevant(docs, seeds):
|
||||
seeds = [s.lower() for s in seeds]
|
||||
return {d["id"] for d in docs if any(s in d["text"].lower() for s in seeds)}
|
||||
|
||||
|
||||
def embed_text(model, proc, texts, side, batch_size=4):
|
||||
result = []
|
||||
prefix = "Query: " if side == "query" else "Document: "
|
||||
for start in range(0, len(texts), batch_size):
|
||||
batch = [prefix + x for x in texts[start:start + batch_size]]
|
||||
inp = proc(text=batch, padding=True, truncation=True, max_length=1024, return_tensors="pt")
|
||||
with torch.inference_mode():
|
||||
vec = model.embed(**inp)
|
||||
result.append(vec.float().cpu().numpy())
|
||||
return np.concatenate(result, axis=0)
|
||||
|
||||
|
||||
def measure_text(docs, doc_vecs, model, proc):
|
||||
out = []
|
||||
for name, query, seeds in CASES:
|
||||
rel = relevant(docs, seeds)
|
||||
t0 = time.perf_counter()
|
||||
qv = embed_text(model, proc, [query], "query", 1)[0]
|
||||
latency = (time.perf_counter() - t0) * 1000
|
||||
scores = doc_vecs @ qv
|
||||
order = np.argsort(-scores)
|
||||
rank = 0
|
||||
for pos, idx in enumerate(order, 1):
|
||||
if docs[int(idx)]["id"] in rel:
|
||||
rank = pos
|
||||
break
|
||||
best_rel = max((float(scores[i]) for i, d in enumerate(docs) if d["id"] in rel), default=float("nan"))
|
||||
best_neg = max((float(scores[i]) for i, d in enumerate(docs) if d["id"] not in rel), default=float("nan"))
|
||||
out.append({
|
||||
"name": name, "query": query, "relevant": len(rel), "rank": rank,
|
||||
"reciprocal_rank": 1.0 / rank if rank else 0.0,
|
||||
"hit_at_1": 0 < rank <= 1, "hit_at_5": 0 < rank <= 5,
|
||||
"query_latency_ms": latency,
|
||||
"best_relevant": best_rel, "best_negative": best_neg,
|
||||
"margin": best_rel - best_neg,
|
||||
"top": [{"id": docs[int(i)]["id"], "score": float(scores[i])} for i in order[:5]],
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def load_images():
|
||||
rows = [
|
||||
("7e689211", "7e/689211bf78bc2f6366405b2a1f2fe0f7ec9ccdc59f368c13175c421c8e4018"),
|
||||
("ee5bdb2a", "ee/5bdb2a5fa3927ede79384ee6ec7657428896e2690b622f2409daa4b3fa9fa3"),
|
||||
("f6f17229", "f6/f17229b3513ea1f9c660353ee4ab021bec88e2b423780e8ed4445bc9aaf2fb"),
|
||||
]
|
||||
return [(name, Image.open(os.path.join(BLOB_ROOT, rel)).convert("RGB")) for name, rel in rows]
|
||||
|
||||
|
||||
def embed_image(model, proc, img):
|
||||
# Retrieval document side; nano's image geometry is nearly side-invariant, but use the documented prefix.
|
||||
inp = proc(images=img, text="Document: <image>", return_tensors="pt")
|
||||
with torch.inference_mode():
|
||||
return model.embed(**inp).float().cpu().numpy()[0]
|
||||
|
||||
|
||||
def measure_media(images, image_vecs, model, proc):
|
||||
out = []
|
||||
names = [x[0] for x in images]
|
||||
for name, query, want in MEDIA_QUERIES:
|
||||
t0 = time.perf_counter()
|
||||
qv = embed_text(model, proc, [query], "query", 1)[0]
|
||||
latency = (time.perf_counter() - t0) * 1000
|
||||
scores = image_vecs @ qv
|
||||
order = np.argsort(-scores)
|
||||
want_idx = names.index(want)
|
||||
rank = int(np.where(order == want_idx)[0][0]) + 1
|
||||
best_neg = max(float(scores[i]) for i in range(len(names)) if i != want_idx)
|
||||
out.append({
|
||||
"name": name, "query": query, "want": want, "rank": rank,
|
||||
"query_latency_ms": latency, "positive": float(scores[want_idx]),
|
||||
"best_negative": best_neg, "margin": float(scores[want_idx]) - best_neg,
|
||||
"ranking": [{"id": names[int(i)], "score": float(scores[i])} for i in order],
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def summary(rows):
|
||||
return {
|
||||
"queries": len(rows),
|
||||
"hit_at_1": sum(r["hit_at_1"] for r in rows) / len(rows),
|
||||
"hit_at_5": sum(r["hit_at_5"] for r in rows) / len(rows),
|
||||
"mrr": sum(r["reciprocal_rank"] for r in rows) / len(rows),
|
||||
"mean_query_latency_ms": sum(r["query_latency_ms"] for r in rows) / len(rows),
|
||||
"mean_margin": sum(r["margin"] for r in rows) / len(rows),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
torch.set_num_threads(min(8, os.cpu_count() or 1))
|
||||
print("loading Jina v5-omni-nano (vision + text)...", flush=True)
|
||||
t0 = time.perf_counter()
|
||||
model = AutoModel.from_pretrained(
|
||||
MODEL, trust_remote_code=True, local_files_only=True,
|
||||
default_task="retrieval", modality="vision", dtype=torch.float32,
|
||||
).eval()
|
||||
proc = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True, local_files_only=True)
|
||||
load_s = time.perf_counter() - t0
|
||||
docs = load_docs()
|
||||
print(f"loaded model in {load_s:.2f}s, embedding {len(docs)} docs", flush=True)
|
||||
t0 = time.perf_counter()
|
||||
doc_vecs = embed_text(model, proc, [d["text"] for d in docs], "document", batch_size=4)
|
||||
doc_embed_s = time.perf_counter() - t0
|
||||
text_rows = measure_text(docs, doc_vecs, model, proc)
|
||||
|
||||
images = load_images()
|
||||
t0 = time.perf_counter()
|
||||
image_vecs = np.stack([embed_image(model, proc, img) for _, img in images])
|
||||
image_embed_s = time.perf_counter() - t0
|
||||
media_rows = measure_media(images, image_vecs, model, proc)
|
||||
|
||||
report = {
|
||||
"model": "jina-v5-omni-nano", "dimension": int(doc_vecs.shape[1]),
|
||||
"model_load_seconds": load_s, "documents": len(docs),
|
||||
"document_embedding_seconds": doc_embed_s,
|
||||
"document_embedding_ms_per_doc": doc_embed_s * 1000 / len(docs),
|
||||
"text_retrieval": text_rows, "text_summary": summary(text_rows),
|
||||
"images": len(images), "image_embedding_seconds": image_embed_s,
|
||||
"image_embedding_ms_per_image": image_embed_s * 1000 / len(images),
|
||||
"media_retrieval": media_rows,
|
||||
"media_summary": {
|
||||
"queries": len(media_rows),
|
||||
"hit_at_1": sum(r["rank"] == 1 for r in media_rows) / len(media_rows),
|
||||
"mrr": sum(1.0 / r["rank"] for r in media_rows) / len(media_rows),
|
||||
"mean_margin": sum(r["margin"] for r in media_rows) / len(media_rows),
|
||||
"mean_query_latency_ms": sum(r["query_latency_ms"] for r in media_rows) / len(media_rows),
|
||||
},
|
||||
}
|
||||
Path("/tmp/homeagent-jina-benchmark.json").write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(json.dumps({k: report[k] for k in ["model", "dimension", "model_load_seconds", "documents", "document_embedding_seconds", "document_embedding_ms_per_doc", "text_summary", "images", "image_embedding_seconds", "image_embedding_ms_per_image", "media_summary"]}, ensure_ascii=False, indent=2))
|
||||
for r in text_rows:
|
||||
print(f"TEXT {r['name']:20s} rank={r['rank']:3d} margin={r['margin']:+.4f} latency={r['query_latency_ms']:.1f}ms")
|
||||
for r in media_rows:
|
||||
print(f"MEDIA {r['name']:20s} rank={r['rank']} pos={r['positive']:+.4f} neg={r['best_negative']:+.4f} margin={r['margin']:+.4f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user