JianFeeeee 75dfff5406 fix(multimodal): 修多模态假成功 + 落地视觉回退链 + see_video 帧数语义
## 起因

生产盲测:模型调 multimodal_see_picture 后声称看到了图,实际一个字
都没收到。工具却返回「[已将图片注入后续对话]」。

链路:core.llm.model=AUTO → llmsproxy 按优先级选 big-pickle(prio=100)
→ 转 opencode zen。llmsproxy 的 opencode.lua 明写着:

    -- zen 上游 schema 只接受 text content part(无视觉/音频能力)
    if part.type ~= nil and part.type ~= "text" then  -- 丢弃

判据:256x256 纯红 PNG,带图与不带图的 prompt_tokens 都是 256。
图片贡献零 token,即根本没进上游。

内核序列化与注入链本身是对的(Message.MarshalJSON 正确产出 content
数组,SetToolBlocks → IOManager → ConsumeToolBlocks → toolMsg.Blocks
全通)。缺的是「主模型能否消费这些块」这一判断——内核此前完全没有
多模态能力的概念(grep supportsVision|multimodal 在 agent/ 零命中)。

这与 v1.0.0 修的 output_send 假成功同类:告诉调用方成功而实际未送达。

## 1. 能力声明

新增 core.llm.sources.<name>.vision / .audio(走既有 sourceFieldDefs,
WebUI 配置页自动出现),types.LLMSource 与 api.BaseConfig 同步加字段。

新增 agentAPI.ModalProvider 接口 + ProviderSupportsVision/Audio 判定:
未实现该接口的 provider 一律按不支持处理。保守侧是刻意的——宁可多走
一次文字回退,也不能把图默默扔给会剥掉它的上游。

为何是声明而非探测:探测需额外真实调用且结果不稳定(取决于 AUTO 当次
路由到哪);而 200 响应 + 相同 token 数从响应侧无法区分「看到了但没
内容」和「被剥掉了」。

## 2. 回退链(modalfallback.go)

实现了 config/registry.go 里注册但从未被读取的 image/audio
fallback_provider + fallback_model(此前 0 处读取点)。

prepareToolBlocks 在 process.go 注入前判定:能直视就原样透传;不能就
调声明了该能力的源转写成文字,带 [由 X 转写,非当前模型直接感知] 标注。

几处刻意的设计:
- 逐模态判定,不一刀切。很多视觉模型能看图但听不到音频,全部降级会
  白白把可直视的图变成二手描述
- 混合场景下转写文字作为 text 块并入 native,两部分同时到达模型
- 配置指向未声明能力的源时拒绝并继续找——照用只会重演静默剥离
- 未配 fallback_provider 但某源声明了 vision 时自动扫出来用;静默失败
  比多找一个能用的源更糟
- 空回复算失败。上游剥掉媒体后模型往往回「我没看到图片」或空串,两种
  都说明回退链也没真看到
- 多媒体块按模态合包为一次请求(见下)

## 3. 批量合包(生产实测驱动的返工)

首版逐块调用,生产 see_video 6 帧实测:4 帧里 3 帧超时,整轮 363 秒。
改为按模态合包一次请求后同一用例 131 秒、6/6 成功。

顺带把 modalFallbackTimeout 从 90s 提到 180s:生产经网关转
claude-opus-5 看一张 400x400 图要 ~81s,90s 贴着上限。
多张时 detail 默认 low 控体积,单张用 high 看细节;插件显式给了
detail 则尊重它。

## 4. see_video 帧数语义

fps=1/N 是频率(每 N 秒一帧)不是数量。20s 视频实测:
frames=4 → 5 帧、frames=10 → 2 帧、frames=1 → 20 帧,要得越多拿得越少;
长视频下 frames=4 会产出 时长/4 帧,靠 i>=9 的 break 兜着才没炸上下文,
而那个 break 用的是 ReadDir 索引,跳过条目后与实际帧数错位。

改为 ffprobe 取时长 → fps=N/时长 + -frames:v N 硬封顶。
0.4s/3s/20s/120s × frames=1/2/4/7/10 全部精确。

极短视频的坑:fps=1 在 0.4s 素材上产出 0 帧(不足一秒抽不出),所以
时长探测失败时不能退化成 fps=1,改为不传 -vf 只靠 -frames:v。

## 验证

- modalfallback_test.go 14 例:直视透传 / 回退转写 / 无源如实报告 /
  未实现接口按不支持 / 混合模态拆分 / 空回复算失败 / 块数上限 /
  多图合一次调用 / detail 策略 / 拒绝未声明能力的源 / 未配置时自动扫源
- go test ./... 全绿,go vet 无警告
- 生产盲测(答案预先封存、生成时不读):随机三色带 → 模型答
  「紫、蓝、红」,与封存答案完全一致
- 负向验证:拿掉回退源后模型如实回答「没看到图片内容」并引用工具返回
  的配置提示,且主动纠正了上一轮的答案
- 生产 see_video 6 帧:单次转写,模型正确描述测试图卡的计数器递增与
  彩虹带滚动
2026-09-04 06:25:51 +08:00
2026-07-29 15:45:24 +08:00

⚠️ AI-Assisted Programming Notice: Parts of this project's code, documentation, and commit history were generated or modified with AI assistance. Key changes have been human-reviewed, but please evaluate and verify before use.

HomeAgent

中文: README.md

An Agent framework designed around separation of core domain and application domain. The kernel enforces a zero-IO policy — all external interaction (WebUI, QQ, CLI, file operations, web search, memos, etc.) is handled by the plugin layer; the kernel performs no direct IO operations.

Combined with a three-layer memory architecture (Context → Document → Graph), it maintains contextual coherence across long-running single-conversation sessions through tiered storage and automated archival.

homed (kernel, zero IO) ← PluginSDK → plugins (all IO capabilities)

Since v1.0.0 external plugins are independent subprocesses, communicating with the kernel over stdio JSON-RPC (control plane) + a shared memory segment (data plane) + an event ring (notification plane). A plugin crash cannot take down the kernel and it restarts automatically; swapping plugin.bin gives true hot-reload.

Design Principles

Separation of Core Domain and Application Domain — The kernel's responsibilities are limited to LLM orchestration, memory management, and knowledge retrieval; all IO capabilities (message send/receive, file read/write, network requests, hardware interaction, etc.) are implemented by plugins. This separation defines domain boundaries at the Agent framework level, with distinct responsibility scopes for the kernel and plugins.

Three-Layer Memory Architecture — Manages information retention in long-running agents through a tiered storage strategy:

  • Context Layer: Pretrained word embedding / TF-IDF fallback relevance-scored event window, protects last 10 entries, maintains topK context entries
  • Document Layer: Temporary memory with automatic cold data sinking, also supports user-initiated submissions
  • Graph Layer: SQLite graph database, persists entity relationships and semantic memory, supports distillation pipelines to extract triples from conversations

Architecture Diagrams

1. Message Processing Sequence

sequenceDiagram
    participant U as User/Plugin
    participant IO as IOManager
    participant EV as eventLoop
    participant CTX as RelevanceContext
    participant LLM as LLM+Tool Loop
    participant ST as StageHost
    participant MEM as Three-Layer Memory

    U->>IO: InjectInput(type, payload)
    IO->>EV: inputCh
    rect lavender
        Note over EV: processTextInput
        EV->>ST: StageOnInput  Plugin can rewrite/short-circuit
        EV->>CTX: Prune(input,topK)  StaticEmbedder/TF-IDF cosine pruning
        CTX->>MEM: Low-score events archived to Document (original timestamp)
        EV->>CTX: Append(input)  CleanTemplateText→three-branch vector→5s write
    end
    rect lightgreen
        Note over EV,LLM: process()
        EV->>MEM: buildMemoryContext  Indexer recalls from Graph (vector+jieba→BFS depth=2)
        EV->>MEM: buildSystemPrompt  DocQuery summary+Graph memory index+Persona+Skills
        EV->>ST: StagePreAction  Plugin can pre-intercept
        loop Tool loop
            LLM->>LLM: drainInterrupts
            LLM->>LLM: LLM Chat
            LLM->>ST: StagePostAction  Plugin can modify/short-circuit
            alt No tool call
                LLM-->>EV: Returns response
            else
                loop Each tool
                    ST->>ST: StageBeforeToolcall  Plugin can reject
                    LLM->>LLM: executeToolCall
                    ST->>ST: StageAfterToolcall
                end
            end
        end
    end
    rect lightpink
        Note over EV: emitResponse
        CTX->>CTX: Append(response)
        ST->>ST: StageBeforeOutput  Plugin can rewrite
        EV-->>U: ResponseCh CLI sync
        EV-->>EV: Event bus WebUI SSE
        ST->>ST: StageAfterOutput  Read-only
        EV->>MEM: emitMemoryCandidate
    end

2. Stage Pipeline

flowchart LR
    S1[① on_input] --> S2[② pre_action]
    S2 --> S3[③ post_action]
    S3 --> Q{Has tool?}
    Q -->|Yes| S4[④ before_toolcall]
    S4 --> T[executeToolCall]
    T --> S5[⑤ after_toolcall]
    S5 --> S3
    Q -->|No| S6[⑥ before_output]
    S6 --> S7[⑦ after_output]
    style S1 fill:#e1f5fe
    style S3 fill:#fff3e0
    style S6 fill:#e8f5e9

3. Three-Layer Memory

flowchart TB
    subgraph C[① Context Working Window]
        RC[RelevanceContext]
        A[Append] -->|CleanTemplateText→three-branch vector| RC
        P[Prune StaticEmbedder/TF-IDF Cosine] -->|Low score original timestamp| D
        P -->|Keep| TL[timeline→chronological→system prompt]
    end
    subgraph D[② Document File Memory]
        DS[DocStore JSON+TF-IDF]
        Q1[Query summary auto-inject] -->|[Related Memory Docs]| SP
        Q2[doc_query LLM active recall] -->|Consume+delete source| DS
        Q2 -->|Original timestamp write to context| RC
        CD[FindColdDocs 72h] -->|docToTriples| G
    end
    subgraph G[③ Graph Database]
        DB[(SQLite)]
        IDX[Indexer vector+jieba→BFS depth=2] -->|[Memory Index]| SP
        MEM[memory_recall/commit/merge/purge/edit]
        SOC[person_query/set_trait]
    end
    subgraph H[④ Heartbeat Distillation]
        REORG -->|Step3 Cold docs| CD
        REORG -->|Step4 Bigram Jaccard| CONS[consolidation]
        PIPE[Pipeline regex] -->|Name/Address/Likes/Age/Job| DB
    end
    SP[System Prompt] -->|Sequential assembly| LLM
    LLM[LLM] -->|doc_query| Q2
    LLM -->|memory_recall| MEM

See assets/docs/en/ARCHITECTURE.md for details.

Web Mascot

HomeAgent Web Mascot Xiaozhai

Xiaozhai — HomeAgent Web Mascot

Quick Start

make build build-cli
./build/homed -data /tmp/ha
# Interactive mode
./build/waiter

# Or single message
echo "Hello, remember that I like coffee" | ./build/waiter

API keys are configured via WebUI http://localhost:8080 settings page, persisted in SQLite.

Code Structure

cmd/homed/          Daemon entry, assembles all subsystems
cmd/waiter/         CLI client (Unix socket)
internal/
├── agent/core/     Agent core: event loop, LLM tool loop, 7-stage pipeline
├── agent/api/      LLM Provider + 8 Lua adapters
├── memory/         Three-layer memory: Graph(SQLite) / Document(JSON+TF-IDF) / Text(JSONL) + StaticEmbedder(pretrained word embedding/TF-IDF fallback) + CleanTemplateText(de-template)
├── knowledge/      Knowledge base (filesystem + TF-IDF)
├── plugin/         Plugin registry + subprocess loader (stdio RPC + shared memory segment + event ring)
├── plugins/        11 built-in plugins (webui/cli/timer/cmd/mcp/clawhubadapter/agentcli/healthcheck/pluginmgr/files/cfgmgr)
├── sdk/            PluginSDK (Tool/Stage/Event three channels)
├── config/         SQLite config center
├── events/         Event bus
└── internal/lua/adapters/   8 LLM protocol adapter scripts
External plugin development: see [homeagent-sdk](https://gitcode.com/JianFeeeee/homeagent-sdk) repo, use `plugindev` toolchain, refer to Go and Lua examples in `example/`

Project Status

v1.0.0 — External plugins moved from C ABI shared libraries to subprocess + shared memory. The first release that no longer loads .so/.dll, and it is incompatible with 0.9.x (existing plugins must be rebuilt into plugin.bin with the new plugindev, though business code needs zero changes). Eliminates 6 classes of defects that had caused production incidents: hot-reload silently failing (DF_1_NODELETE making dlclose a no-op), no crash isolation (a plugin panic took down homed), stage lost updates (35.8~36.8% loss under the copy model), uncancellable cgo timeouts (linear OS-thread leaks), output_send reporting false success (the model was told "sent" while the message never went out), and Windows capability degradation (only 3 stage fields visible, no write-back). Three communication planes: stdio JSON-RPC (control) + shared memory segment (data) + event ring (notification); the privilege gradient is now enforced by three explicit gates. RPC round-trip p50 24.1µs; crash-to-recovery under 1s.

v0.9.0 — C ABI v2: external plugin Stage callbacks can now write back (invoke_stage gained a result out-param; plugins may mutate RawMessage/LLMText/ToolResults etc. in OnInput/AfterToolcall/PostAction and have them synced to the core). ABI version now tracks core minor releases (v0.9.x → ABIVersion=2, version_min=1 keeps old plugins loadable). Also fixes the tool-loop zen-compat placeholder that wrongly fired on first-turn system context tail. The SDK ships an enhanced sanitizer example (bad-UTF-8 / U+FFFD / ANSI-escape scrub across the whole pipeline). This ABI retired with v1.0.0.

v0.8.0 — Core is functional, plugin system enhanced. 20+ built-in plugins. External plugin development via homeagent-sdk repo. Added input channel NoMemory/Cleaner, ChannelDef, plugin disable/enable system (CLI + WebUI), plugindev toolchain C ABI ChannelDef support.

Documentation

Downloads

Releases ship three variants:

Variant Contents For
full homed + waiter + desktop GUI + systemd unit Single-machine, everything
server homed + waiter + systemd unit Servers (no desktop environment)
client waiter + desktop GUI Connecting to a remote HomeAgent
  • Linux: .deb (amd64/arm64), .rpm (x86_64), .tar.gz
  • Windows: HomeAgent_v1.0.0_{Full,Server,Client}_win64.exe (NSIS installer)
  • Portable: homeagent-bin-<os>_<arch>.tar.gz (homed/waiter/initconfig)
  • Verification: SHA256SUMS

The macOS homed requires a native macOS build (CGO + sqlite3), so release packages ship only waiter/initconfig.

Build

make build build-cli    # Build daemon + CLI
make test               # go test ./...
make install            # Install to system

Dependencies: Go 1.25+, CGo (go-sqlite3), Linux/Windows.

Description
No description provided
Readme AGPL-3.0 87 MiB
Languages
Go 74.8%
JavaScript 11.1%
HTML 3.1%
Python 2.8%
CSS 2.4%
Other 5.6%