mirror of
https://gitcode.com/JianFeeeee/HomeAgent.git
synced 2026-09-21 17:38:10 +00:00
feat: screensee 工具 — agent 查看远程设备屏幕内容
与 screensue(向用户屏幕显示)配对: screensue 是给用户看, screensee 是 agent 看。 服务端实现: 1. remotedevice 新增 screensee 工具: - 下发 homeagent-screensee 命令 → 设备截屏回传 jpeg base64 - seeHandler 回调(agent 核心注入)用视觉模型自动描述屏幕内容 - 未授权/离线/超时完整错误路径; 结果留档 cmdresult 2. SDK LLMMessage 扩展多模态 Blocks(text/image_url): - llm_impl 转换为 agentAPI.ContentBlock, 视觉模型可看图 3. describeScreen: 默认提示词描述窗口/文字/界面状态; provider 参数可指定视觉源(临时切换后恢复) GUI 端需配套(已发群): onDeviceMsg 加 case "screensee", desktopCapturer 截屏 → jpeg base64 data URL 回执(同 camerasue 抓拍模式)。 测试: 端到端模拟设备截屏回传+视觉回调验证; 全项目 go test 通过
This commit is contained in:
@ -71,6 +71,18 @@ func (l *llmImpl) Chat(ctx context.Context, req *LLMCompletionRequest) (*LLMComp
|
||||
ReasoningContent: m.ReasoningContent,
|
||||
ToolCallID: m.ToolCallID,
|
||||
}
|
||||
// 多模态 Blocks:text/image_url → agentAPI.ContentBlock
|
||||
for _, b := range m.Blocks {
|
||||
switch b.Type {
|
||||
case "text":
|
||||
msg.Blocks = append(msg.Blocks, agentAPI.ContentBlock{Type: "text", Text: b.Text})
|
||||
case "image_url":
|
||||
msg.Blocks = append(msg.Blocks, agentAPI.ContentBlock{
|
||||
Type: "image_url",
|
||||
ImageURL: &agentAPI.ImageURL{URL: b.ImageURL, Detail: "high"},
|
||||
})
|
||||
}
|
||||
}
|
||||
if len(m.ToolCalls) > 0 {
|
||||
msg.ToolCalls = make([]agentAPI.ToolCall, len(m.ToolCalls))
|
||||
for j, tc := range m.ToolCalls {
|
||||
|
||||
Reference in New Issue
Block a user