mirror of
https://gitcode.com/JianFeeeee/homeagent-sdk.git
synced 2026-09-20 08:58:03 +00:00
feat(deepsearch): 联网检索插件 + SearXNG 生命周期托管
把原 websearch 示例改名为 deepsearch(目录/go.mod/plg.json/工具前缀/README 全量对齐, 工具名 websearch_* → deepsearch_*)。 新增 searxng.go:插件自己托管搜索后端 - 启动探 healthz:已在跑则直接接管(不重启),没跑就 docker compose up -d 并等就绪 - 关闭动作注册为 stop handler(幂等、限时 4s < 内核 5s 宽限期) - 配置 manage_searxng / searxng_dir / stop_searxng_on_exit - 崩溃/被 kill 时不关后端(下次启动接管):安全失败方向 - 可注入 cmdRunner + 时间预算,8 项单测离线覆盖接管/拉起/失败/幂等/保留 契约依据(internal/plugin/proc):停止插件 = plugin.stop → RunStopHandlers(LIFO、 幂等)→ Stop() → exit(0),宽限期 5s;stdin 关闭同路径。 验证:19 项单测(18 通过 + 1 联调跳过)、-race 干净、vet/gofmt 干净;内核 E2E 真调用 返回 58 条/37 条、58/58 带摘要;线上 daemon 实测 stop handler 与冷启动(约 3s)。
This commit is contained in:
100
example/deepsearch/README.md
Normal file
100
example/deepsearch/README.md
Normal file
@ -0,0 +1,100 @@
|
||||
# 联网检索插件(HomeAgent)
|
||||
|
||||
给 agent 补上**真正的信息检索**能力:检索交给本地 SearXNG(多引擎聚合、结构化 JSON),
|
||||
并补上「读完前 K 篇再回答」的深检索。
|
||||
|
||||
## 为什么需要它(背景)
|
||||
|
||||
agent 原本只有 `browser_*` 那套浏览器工具,联网检索实际只有 `browser_search` 一个入口,而它是
|
||||
**「抓 Bing HTML + 正则解析」**:
|
||||
|
||||
| 缺陷 | 实测结果 |
|
||||
|---|---|
|
||||
| 标题取的是结果块里**第一个 `<a>`** | 拿到的是 Bing 的「来源行」而非标题 → `deepin.orghttps://www.deepin.org` |
|
||||
| 摘要正则 `<div class="b_caption">.*?<p>` | 对现代 Bing **命中 0/10**(摘要已迁到 `p.b_lineclamp*`)→ 结果**完全没有摘要** |
|
||||
| 用 `www.bing.com` | 程序化请求直接 302;`cn.bing.com` 才返回 10 个结果块 |
|
||||
| 单引擎、无兜底、无去重、无站点读取 | 模型只能反复换词重搜(日志里 8 秒 6 连击) |
|
||||
|
||||
结果就是日志里那句用户反馈:**「你的搜索能力好像不太行啊」**。
|
||||
|
||||
## 依赖:本地 SearXNG(由本插件托管)
|
||||
|
||||
插件会**自己管后端**:
|
||||
|
||||
- **启动时**:探 `healthz`;已在跑就**直接接管**(不重启),没跑就 `docker compose up -d` 并等就绪(上限 6s)
|
||||
- **停止时**:跑 `docker compose stop -t 2` 关闭它
|
||||
|
||||
配置项 `manage_searxng`(默认 true)与 `searxng_dir`(默认 `/root/searxng-agent`)控制这套行为;
|
||||
`stop_searxng_on_exit`(默认 true)设 false 可让后端在插件停止后继续跑(**插件重载频繁时建议设 false**,
|
||||
否则每次重载都会把后端重启一遍)。
|
||||
|
||||
### 生命周期契约(依据内核源码,非猜测)
|
||||
|
||||
| 环节 | 内核行为 |
|
||||
|---|---|
|
||||
| 停止插件 | 发 `plugin.stop` → 插件先跑 **RunStopHandlers(LIFO、幂等)** → 再 `Stop()` → `exit(0)` |
|
||||
| 宽限期 | **5 秒**;未退出则直接 SIGKILL —— 所以关闭动作限时 4s(`searxShutdownBudget`) |
|
||||
| stdin 关闭 | 同样会跑 handlers + `Stop()` |
|
||||
| 崩溃/被 kill | 关闭动作不会执行,后端会留在运行态;下次启动探测到就直接接管(**更安全的失败方向**) |
|
||||
| 自动重启 | `SetAutoRestart(true)` 由注入的 runtime 在 `plugin.start` 后经 `lifecycle.autoRestart` **显式上报**内核 |
|
||||
|
||||
### SearXNG 侧配置
|
||||
|
||||
部署在 **.60**,`127.0.0.1:8888`:
|
||||
|
||||
```
|
||||
/root/searxng-agent/docker-compose.yml # host 网络(要访问宿主 clash)
|
||||
/root/searxng-agent/settings.yml # json 输出 + limiter 关闭 + 出站走 clash
|
||||
```
|
||||
|
||||
两个必须知道的坑:
|
||||
|
||||
1. **`search.formats` 必须含 `json`**,否则 `/search?format=json` 返回 **403**(看起来像网络问题,其实是配置)。
|
||||
2. 该镜像默认 `GRANIAN_PORT=8080`,而 granian 的 `GRANIAN_*` **优先级高于 settings.yml**:
|
||||
.60 上 8080 被 homeagent 占用 → 不改 `SEARXNG_PORT` 就是无休止的 `Address already in use` 崩溃循环。
|
||||
|
||||
实测可用的引擎(2026-09-12):`duckduckgo`、`brave`、`google cse`;`quark` 时好时坏;
|
||||
`baidu`/`google` 经代理出口触发 CAPTCHA,`sogou` 崩溃,`wikidata` 报 HTTP error(已关)。
|
||||
|
||||
## 工具
|
||||
|
||||
| 工具 | 说明 |
|
||||
|---|---|
|
||||
| `deepsearch_search` | 联网检索(首选):标题 + URL + 摘要 + 发布时间,支持 `engines`/`category`/`time_range`/`language`,自动按 URL 去重并按分数排序;会回报**引擎覆盖度与无响应引擎** |
|
||||
| `deepsearch_news` | 新闻检索:`news` 类别 + 默认最近一周;新闻为空时自动回退 general + 时间范围 |
|
||||
| `deepsearch_fetch` | 抓单个网页并抽正文(去脚本/样式/导航),返回标题 + 纯文本,可设截断长度 |
|
||||
| `deepsearch_deep` | **深检索**:检索 → 并行抓前 K 篇正文 → 一次返回「候选清单 + 证据正文」;单篇失败不影响整体 |
|
||||
| `deepsearch_status` | 自检:healthz、json 是否可用、延迟、**哪些引擎真的在返回结果**(检索出问题先跑这个) |
|
||||
|
||||
## 配置项
|
||||
|
||||
| 键 | 默认 | 说明 |
|
||||
|---|---|---|
|
||||
| `searxng_url` | `http://127.0.0.1:8888` | 本地 SearXNG 地址 |
|
||||
| `max_results` | `8` | 默认条数(控制上下文体积) |
|
||||
| `language` | `zh-CN` | 检索语言 |
|
||||
| `safesearch` | `0` | 0 关 / 1 中 / 2 严 |
|
||||
| `request_timeout` | `20` | 单次请求超时(秒) |
|
||||
| `fetch_max_chars` | `4000` | `deepsearch_fetch` 正文上限 |
|
||||
| `proxy` | 空 | 仅作用于本插件直连抓取(搜索出网由 SearXNG 侧负责) |
|
||||
| `user_agent` | Chrome UA | 抓取用 |
|
||||
|
||||
每次调用前重读配置,改完即时生效。
|
||||
|
||||
## 开发与验证
|
||||
|
||||
```bash
|
||||
go test -count=1 -race ./... # 11 项测试(httptest 打桩 SearXNG)
|
||||
|
||||
# 真实后端联调(默认跳过):跑的就是当初失败的那条查询
|
||||
DEEPSEARCH_LIVE_SEARXNG=http://127.0.0.1:8888 go test -run TestLiveSearxng -v ./...
|
||||
|
||||
hmapdev build # 产出 dist/deep_search_bundle.hmap
|
||||
```
|
||||
|
||||
## 已知边界
|
||||
|
||||
- **知乎等站点对直连抓取返回 403**(反爬),`deepsearch_deep` 会如实标注该篇抓取失败并继续;
|
||||
这类页面请改用浏览器工具(`browser_navigate` + `browser_render`)。
|
||||
- 引擎可用性随出口 IP 与目标站点风控变化;`deepsearch_status` 与每次结果里的「覆盖度」行就是给这个用的。
|
||||
- 未做正文去重/相似度合并:同一事件的多篇转载会各占一条(摘要已能区分)。
|
||||
17
example/deepsearch/go.mod
Normal file
17
example/deepsearch/go.mod
Normal file
@ -0,0 +1,17 @@
|
||||
module deepsearch-plugin
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require gitcode.com/JianFeeeee/homeagent-sdk v1.2.0
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
replace gitcode.com/JianFeeeee/homeagent-sdk => /root/.homeagent/hmapdev/sdk/v1.2.0
|
||||
81
example/deepsearch/live_test.go
Normal file
81
example/deepsearch/live_test.go
Normal file
@ -0,0 +1,81 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 真实后端联调(默认跳过,需显式指定地址):
|
||||
//
|
||||
// DEEPSEARCH_LIVE_SEARXNG=http://127.0.0.1:8888 go test -run TestLiveSearxng -v ./...
|
||||
//
|
||||
// 它跑的就是当初失败的场景(日志里那条「你的搜索能力好像不太行啊」对应的查询),
|
||||
// 用来回答一个具体问题:换了后端之后,模型拿到的是不是「带摘要的相关结果」。
|
||||
func TestLiveSearxng(t *testing.T) {
|
||||
base := os.Getenv("DEEPSEARCH_LIVE_SEARXNG")
|
||||
if base == "" {
|
||||
t.Skip("未设置 DEEPSEARCH_LIVE_SEARXNG,跳过真实后端联调")
|
||||
}
|
||||
p := &Plugin{
|
||||
name: "deepsearch",
|
||||
searxURL: strings.TrimRight(base, "/"),
|
||||
maxItems: 6,
|
||||
language: "zh-CN",
|
||||
fetchMax: 1200,
|
||||
userAgent: defaultUA,
|
||||
}
|
||||
p.ensure()
|
||||
|
||||
// 1) 自检
|
||||
st, err := p.handleStatus(map[string]interface{}{})
|
||||
if err != nil {
|
||||
t.Fatalf("status: %v", err)
|
||||
}
|
||||
t.Logf("status: %v", st)
|
||||
|
||||
// 2) 当初失败的那条查询
|
||||
res, err := p.handleSearch(map[string]interface{}{"query": "深度科技 deepin 开发者 被开除"})
|
||||
if err != nil {
|
||||
t.Fatalf("search: %v", err)
|
||||
}
|
||||
txt := res.(map[string]interface{})["content"].(string)
|
||||
t.Logf("检索结果:\n%s", txt)
|
||||
if !strings.Contains(txt, "摘要:") {
|
||||
t.Errorf("结果里应当有摘要(这正是原实现缺失的东西)")
|
||||
}
|
||||
if !strings.Contains(txt, "覆盖:") {
|
||||
t.Errorf("应报告引擎覆盖度")
|
||||
}
|
||||
|
||||
// 3) 正文抓取(取第一条结果的 URL)
|
||||
var firstURL string
|
||||
for _, line := range strings.Split(txt, "\n") {
|
||||
l := strings.TrimSpace(line)
|
||||
if strings.HasPrefix(l, "http") {
|
||||
firstURL = l
|
||||
break
|
||||
}
|
||||
}
|
||||
if firstURL == "" {
|
||||
t.Fatal("未从结果中解析出 URL")
|
||||
}
|
||||
page, err := p.handleFetch(map[string]interface{}{"url": firstURL, "max_chars": float64(600)})
|
||||
if err != nil {
|
||||
t.Logf("抓取 %s 失败(真实站点有反爬/需 JS 属正常):%v", firstURL, err)
|
||||
} else {
|
||||
body := page.(map[string]interface{})["content"].(string)
|
||||
t.Logf("抓取 %s 正文前 400 字:%s", firstURL, oneLine(body, 400))
|
||||
}
|
||||
|
||||
// 4) 深检索
|
||||
deep, err := p.handleDeep(map[string]interface{}{"query": "统信 UOS 内核工程师 西装 事件", "top_k": float64(2)})
|
||||
if err != nil {
|
||||
t.Fatalf("deep: %v", err)
|
||||
}
|
||||
dTxt := deep.(map[string]interface{})["content"].(string)
|
||||
if !strings.Contains(dTxt, "候选清单") || !strings.Contains(dTxt, "正文证据") {
|
||||
t.Errorf("深检索输出结构不对")
|
||||
}
|
||||
t.Logf("深检索输出前 800 字:\n%s", oneLine(dTxt, 800))
|
||||
}
|
||||
12
example/deepsearch/plg.json
Normal file
12
example/deepsearch/plg.json
Normal file
@ -0,0 +1,12 @@
|
||||
{
|
||||
"name": "deepsearch",
|
||||
"name_zh": "联网检索",
|
||||
"name_en": "Deep Search",
|
||||
"version": "1.1.0",
|
||||
"description": "为 agent 提供真正的联网信息检索:本地 SearXNG 聚合多引擎(返回标题/URL/摘要/时间),支持新闻、时间范围、指定引擎;并提供网页正文抽取与「搜索+读前K篇」的深检索",
|
||||
"author": "HomeAgent",
|
||||
"entry": "plugin.bin",
|
||||
"sdk": "1.2.0",
|
||||
"tags": ["search", "web", "searxng", "retrieval", "news"],
|
||||
"targets": "linux/amd64"
|
||||
}
|
||||
1000
example/deepsearch/plugin.go
Normal file
1000
example/deepsearch/plugin.go
Normal file
File diff suppressed because it is too large
Load Diff
288
example/deepsearch/plugin_test.go
Normal file
288
example/deepsearch/plugin_test.go
Normal file
@ -0,0 +1,288 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func newTestPlugin(t *testing.T, h http.HandlerFunc) (*Plugin, *httptest.Server) {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(h)
|
||||
t.Cleanup(srv.Close)
|
||||
p := &Plugin{
|
||||
name: "deepsearch",
|
||||
searxURL: srv.URL,
|
||||
maxItems: 5,
|
||||
language: "zh-CN",
|
||||
fetchMax: 1000,
|
||||
userAgent: "test-agent",
|
||||
http: srv.Client(),
|
||||
}
|
||||
return p, srv
|
||||
}
|
||||
|
||||
// 一份贴近真实 SearXNG 的响应:含重复 URL、缺摘要、多引擎、无响应引擎
|
||||
const sampleResponse = `{
|
||||
"query": "deepin 被开除",
|
||||
"results": [
|
||||
{"url":"https://www.zhihu.com/question/1?utm_source=x","title":"网传统信内核开发工程师因没穿西服被开除","content":"截止1月9日最新情况…","engines":["duckduckgo","brave"],"score":9.5,"publishedDate":"2026-09-10T00:00:00"},
|
||||
{"url":"https://www.zhihu.com/question/1","title":"网传统信内核开发工程师因没穿西服被开除(重复项)","content":"重复条目","engines":["brave"],"score":1.0},
|
||||
{"url":"https://www.163.com/dy/article/KIQURODQ.html","title":"离谱!传某信创操作系统大厂因西装开除核心开发者","content":"一位负责Linux内核开发的核心工程师…","engines":["brave","quark"],"score":7.2},
|
||||
{"url":"https://bbs.deepin.org.cn/zh","title":"deepin官方论坛","content":"","engines":["duckduckgo"],"score":2.0}
|
||||
],
|
||||
"answers": [],
|
||||
"suggestions": ["deepin 王勇 离职"],
|
||||
"unresponsive_engines": [["baidu","CAPTCHA"],["sogou","unexpected crash"]],
|
||||
"timings": {"search": 1.2}
|
||||
}`
|
||||
|
||||
// 1) 检索:去重 + 按分数排序 + 摘要/覆盖度输出
|
||||
func TestSearchDedupAndFormat(t *testing.T) {
|
||||
var gotQuery url.Values
|
||||
p, _ := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/search" {
|
||||
gotQuery = r.URL.Query()
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(sampleResponse))
|
||||
return
|
||||
}
|
||||
http.NotFound(w, r)
|
||||
})
|
||||
res, err := p.handleSearch(map[string]interface{}{"query": "deepin 被开除", "count": float64(5)})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
if gotQuery.Get("format") != "json" {
|
||||
t.Errorf("必须要求 json 输出,实际 %q", gotQuery.Get("format"))
|
||||
}
|
||||
if gotQuery.Get("limit") != "5" {
|
||||
t.Errorf("limit 未生效: %q", gotQuery.Get("limit"))
|
||||
}
|
||||
txt := res.(map[string]interface{})["content"].(string)
|
||||
// utm_source 应被规范化掉,重复项只剩一条
|
||||
if n := strings.Count(txt, "zhihu.com/question/1"); n != 1 {
|
||||
t.Errorf("URL 未正确去重(出现 %d 次):\n%s", n, txt)
|
||||
}
|
||||
if !strings.Contains(txt, "网传统信内核开发工程师") {
|
||||
t.Errorf("缺少标题: %s", txt)
|
||||
}
|
||||
if !strings.Contains(txt, "摘要:") {
|
||||
t.Errorf("应输出摘要: %s", txt)
|
||||
}
|
||||
if !strings.Contains(txt, "baidu(CAPTCHA)") {
|
||||
t.Errorf("应回报无响应引擎(让模型知道覆盖度): %s", txt)
|
||||
}
|
||||
if !strings.Contains(txt, "duckduckgo") || !strings.Contains(txt, "quark") {
|
||||
t.Errorf("应回报引擎覆盖: %s", txt)
|
||||
}
|
||||
// 高分条目应排在前面
|
||||
if strings.Index(txt, "统信内核开发工程师") > strings.Index(txt, "离谱!") {
|
||||
t.Errorf("未按分数排序:\n%s", txt)
|
||||
}
|
||||
}
|
||||
|
||||
// 2) 403(未开 json)必须给出可操作提示,而不是裸错误
|
||||
func TestSearchForbiddenHint(t *testing.T) {
|
||||
p, _ := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusForbidden)
|
||||
_, _ = w.Write([]byte("Forbidden"))
|
||||
})
|
||||
_, err := p.handleSearch(map[string]interface{}{"query": "x"})
|
||||
if err == nil {
|
||||
t.Fatal("应返回错误")
|
||||
}
|
||||
msg := err.Error()
|
||||
if !strings.Contains(msg, "403") || !strings.Contains(msg, "formats") {
|
||||
t.Errorf("403 提示应指向 json/limiter 配置,实际: %s", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// 3) 空结果:要给出原因与下一步建议
|
||||
func TestSearchEmptyHint(t *testing.T) {
|
||||
p, _ := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"query":"x","results":[],"suggestions":["换个词"],"unresponsive_engines":[["google","CAPTCHA"]]}`))
|
||||
})
|
||||
res, err := p.handleSearch(map[string]interface{}{"query": "x"})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
txt := res.(map[string]interface{})["content"].(string)
|
||||
for _, want := range []string{"未返回结果", "google(CAPTCHA)", "换个词", "deepsearch_news"} {
|
||||
if !strings.Contains(txt, want) {
|
||||
t.Errorf("空结果提示缺少 %q: %s", want, txt)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 4) 新闻:应带 categories=news 与 time_range=week;新闻为空时回退 general
|
||||
func TestNewsParamsAndFallback(t *testing.T) {
|
||||
var calls []url.Values
|
||||
p, _ := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
calls = append(calls, r.URL.Query())
|
||||
if r.URL.Query().Get("categories") == "news" {
|
||||
_, _ = w.Write([]byte(`{"query":"n","results":[]}`))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"query":"n","results":[{"url":"https://a.com/1","title":"回退结果","content":"内容","engines":["brave"],"score":1}]}`))
|
||||
})
|
||||
res, err := p.handleNews(map[string]interface{}{"query": "某事"})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
if len(calls) != 2 {
|
||||
t.Fatalf("新闻为空时应回退 general,实际调用 %d 次", len(calls))
|
||||
}
|
||||
if calls[0].Get("categories") != "news" || calls[0].Get("time_range") != "week" {
|
||||
t.Errorf("首次应为 news + week,实际 categories=%q time_range=%q", calls[0].Get("categories"), calls[0].Get("time_range"))
|
||||
}
|
||||
if tmp := res.(map[string]interface{})["content"].(string); !strings.Contains(tmp, "回退结果") {
|
||||
t.Errorf("回退结果未被采用: %s", tmp)
|
||||
}
|
||||
}
|
||||
|
||||
// 5) 正文抽取:去脚本/样式/导航,保留 article
|
||||
func TestFetchExtractsArticle(t *testing.T) {
|
||||
page := `<!doctype html><html><head><title>测试标题 - 站点</title>
|
||||
<style>.x{color:red}</style><script>var secret="SHOULD_NOT_APPEAR";</script></head>
|
||||
<body><nav>导航链接</nav><article>
|
||||
<p>第一段正文,包含关键事实。</p><p>第二段正文。</p>
|
||||
</article><footer>页脚</footer></body></html>`
|
||||
var srvURL string
|
||||
p, srv := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
_, _ = w.Write([]byte(page))
|
||||
})
|
||||
srvURL = srv.URL
|
||||
// 注意:不要用 example.com 之类真实域名——本机 DNS/proxy 会把它们转走,测试会飘
|
||||
res, err := p.handleFetch(map[string]interface{}{"url": srvURL + "/a"})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
txt := res.(map[string]interface{})["content"].(string)
|
||||
if !strings.Contains(txt, "第一段正文") {
|
||||
t.Errorf("正文丢失: %s", txt)
|
||||
}
|
||||
if strings.Contains(txt, "SHOULD_NOT_APPEAR") {
|
||||
t.Errorf("脚本内容不应出现: %s", txt)
|
||||
}
|
||||
if strings.Contains(txt, "导航链接") || strings.Contains(txt, "页脚") {
|
||||
t.Errorf("导航/页脚应被剥离: %s", txt)
|
||||
}
|
||||
if !strings.Contains(txt, "测试标题") {
|
||||
t.Errorf("标题应被提取: %s", txt)
|
||||
}
|
||||
}
|
||||
|
||||
// 6) 深检索:候选 + 正文证据;单篇失败不应导致整体失败
|
||||
func TestDeepSearch(t *testing.T) {
|
||||
var srvURL string // 处理函数先于 server 存在,故用闭包变量回填
|
||||
p, srv := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/search":
|
||||
_, _ = w.Write([]byte(`{"query":"d","results":[
|
||||
{"url":"` + srvURL + `/ok1","title":"好文一","content":"摘要一","engines":["brave"],"score":3},
|
||||
{"url":"` + srvURL + `/bad","title":"打不开的","content":"摘要二","engines":["brave"],"score":2},
|
||||
{"url":"` + srvURL + `/ok2","title":"好文二","content":"摘要三","engines":["brave"],"score":1}]}`))
|
||||
case "/ok1", "/ok2":
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte("<html><body><article><p>正文内容 " + r.URL.Path + "</p></article></body></html>"))
|
||||
case "/bad":
|
||||
w.WriteHeader(http.StatusForbidden)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
})
|
||||
srvURL = srv.URL
|
||||
res, err := p.handleDeep(map[string]interface{}{"query": "d", "top_k": float64(3), "max_chars": float64(500)})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
txt := res.(map[string]interface{})["content"].(string)
|
||||
for _, want := range []string{"候选清单", "正文证据", "正文内容 /ok1", "正文内容 /ok2", "抓取失败"} {
|
||||
if !strings.Contains(txt, want) {
|
||||
t.Errorf("深检索输出缺少 %q:\n%s", want, txt)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 7) 自检:健康检查 + 探测检索 + 引擎覆盖统计
|
||||
func TestStatusReportsEngines(t *testing.T) {
|
||||
p, _ := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/healthz" {
|
||||
_, _ = w.Write([]byte("OK"))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(sampleResponse))
|
||||
})
|
||||
res, err := p.handleStatus(map[string]interface{}{})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
m := res.(map[string]interface{})
|
||||
if m["healthz"] != 200 {
|
||||
t.Errorf("healthz 应为 200,实际 %v", m["healthz"])
|
||||
}
|
||||
if m["search_ok"] != true {
|
||||
t.Errorf("search_ok 应为 true:%v", m["search_ok"])
|
||||
}
|
||||
engs, ok := m["engines_returning_results"].(map[string]int)
|
||||
if !ok || engs["brave"] == 0 || engs["quark"] == 0 {
|
||||
t.Errorf("引擎统计不正确: %#v", m["engines_returning_results"])
|
||||
}
|
||||
}
|
||||
|
||||
// 8) 摘要压成一行并按字符截断(避免巨长摘要吃掉上下文)
|
||||
func TestOneLineTruncate(t *testing.T) {
|
||||
got := oneLine("第一行\n第二行\t第三行", 5)
|
||||
if strings.Contains(got, "\n") {
|
||||
t.Errorf("应为单行: %q", got)
|
||||
}
|
||||
if r := []rune(got); len(r) != 6 { // 5 字符 + 省略号
|
||||
t.Errorf("截断长度不符: %q (%d runes)", got, len(r))
|
||||
}
|
||||
}
|
||||
|
||||
// 9) 正文抽取长度上限生效
|
||||
func TestHtmlToTextTruncation(t *testing.T) {
|
||||
long := strings.Repeat("字", 5000)
|
||||
_, text := htmlToText("<html><body><article><p>"+long+"</p></article></body></html>", 100)
|
||||
if !strings.Contains(text, "已截断") {
|
||||
t.Errorf("超长正文应被截断: %d", len([]rune(text)))
|
||||
}
|
||||
}
|
||||
|
||||
// 10) 非 http(s) 协议应被拒绝
|
||||
func TestFetchRejectsBadScheme(t *testing.T) {
|
||||
p, _ := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {})
|
||||
if _, err := p.handleFetch(map[string]interface{}{"url": "file:///etc/passwd"}); err == nil {
|
||||
t.Fatal("file:// 应被拒绝")
|
||||
}
|
||||
if _, err := p.handleFetch(map[string]interface{}{"url": "javascript:alert(1)"}); err == nil {
|
||||
t.Fatal("javascript: 应被拒绝")
|
||||
}
|
||||
}
|
||||
|
||||
// 11) raw 模式返回结构化 JSON(排查用)
|
||||
func TestSearchRawMode(t *testing.T) {
|
||||
p, _ := newTestPlugin(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(sampleResponse))
|
||||
})
|
||||
res, err := p.handleSearch(map[string]interface{}{"query": "q", "raw": true})
|
||||
if err != nil {
|
||||
t.Fatalf("err: %v", err)
|
||||
}
|
||||
m, ok := res.(*searxResponse)
|
||||
if !ok {
|
||||
t.Fatalf("raw 应返回结构化响应,实际 %T", res)
|
||||
}
|
||||
if len(m.Results) != 4 {
|
||||
t.Errorf("结果数应为 4(raw 不去重),实际 %d", len(m.Results))
|
||||
}
|
||||
if _, err := json.Marshal(m); err != nil {
|
||||
t.Errorf("结构化结果应可序列化: %v", err)
|
||||
}
|
||||
}
|
||||
159
example/deepsearch/searxng.go
Normal file
159
example/deepsearch/searxng.go
Normal file
@ -0,0 +1,159 @@
|
||||
package main
|
||||
|
||||
// SearXNG 生命周期托管:插件启动时拉起搜索后端,插件停止时关闭它。
|
||||
//
|
||||
// 契约依据(内核侧 internal/plugin/proc/*,已逐行核对):
|
||||
// - 内核停止插件:发 `plugin.stop` → 插件先跑 RunStopHandlers(LIFO、幂等)→ 再 Stop() → exit(0)
|
||||
// - 若插件未在 stopGracePeriod(**5 秒**)内退出,内核直接 SIGKILL
|
||||
// - stdin 关闭(内核消失)同样会跑 handlers + Stop()
|
||||
//
|
||||
// 因此这里的关闭动作必须**有界**:searxShutdownBudget 取 4s,留 1s 余量。
|
||||
// 若插件是被 kill -9 / OOM 带走的,关闭动作不会执行 —— SearXNG 会留在运行态;
|
||||
// 下次 Start 探测到它在跑就直接接管,这是更安全的失败方向。
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
cfgManageSearx = "manage_searxng"
|
||||
cfgSearxDir = "searxng_dir"
|
||||
cfgStopOnExit = "stop_searxng_on_exit"
|
||||
|
||||
defaultSearxDir = "/root/searxng-agent"
|
||||
|
||||
searxProbeTimeout = 1500 * time.Millisecond // 单次 healthz 探测
|
||||
searxUpBudget = 20 * time.Second // docker compose up -d 的上限(正常 1s 内返回)
|
||||
searxReadyBudget = 6 * time.Second // up 之后等 healthz 就绪的上限
|
||||
searxShutdownBudget = 4 * time.Second // 必须 < 内核 5s 宽限期
|
||||
)
|
||||
|
||||
// searxBudget 把四个时间预算收拢,便于单测注入短值(否则测试要真等就绪窗口)。
|
||||
type searxBudget struct {
|
||||
probe time.Duration
|
||||
up time.Duration
|
||||
ready time.Duration
|
||||
shutdown time.Duration
|
||||
}
|
||||
|
||||
func (p *Plugin) budget() searxBudget {
|
||||
b := p.bud
|
||||
if b.probe == 0 {
|
||||
b.probe = searxProbeTimeout
|
||||
}
|
||||
if b.up == 0 {
|
||||
b.up = searxUpBudget
|
||||
}
|
||||
if b.ready == 0 {
|
||||
b.ready = searxReadyBudget
|
||||
}
|
||||
if b.shutdown == 0 {
|
||||
b.shutdown = searxShutdownBudget
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// cmdRunner 抽出来是为了让生命周期逻辑可单测:注入假执行器,不起真容器。
|
||||
type cmdRunner func(ctx context.Context, dir, name string, args ...string) (string, error)
|
||||
|
||||
func defaultRunner(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
cmd.Dir = dir
|
||||
out, err := cmd.CombinedOutput()
|
||||
return string(out), err
|
||||
}
|
||||
|
||||
// searxReachable 探测搜索后端是否可用(只看 healthz,不发检索请求)。
|
||||
func (p *Plugin) searxReachable(timeout time.Duration) bool {
|
||||
if p.searxURL == "" {
|
||||
return false
|
||||
}
|
||||
base := p.http
|
||||
if base == nil {
|
||||
base = &http.Client{}
|
||||
}
|
||||
cl := *base // 复制一份,避免改到共享 client 的超时
|
||||
cl.Timeout = timeout
|
||||
req, err := http.NewRequest(http.MethodGet, p.searxURL+"/healthz", nil)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
req.Header.Set("User-Agent", p.userAgent)
|
||||
resp, err := cl.Do(req)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
return resp.StatusCode < 400
|
||||
}
|
||||
|
||||
// ensureSearxng 在插件启动时确保搜索后端在跑;已在跑则直接接管,不重启。
|
||||
func (p *Plugin) ensureSearxng() {
|
||||
b := p.budget()
|
||||
if !p.manageSearx {
|
||||
log.Printf("[%s] 未启用 SearXNG 托管(manage_searxng=false),假定 %s 由外部维护", p.name, p.searxURL)
|
||||
return
|
||||
}
|
||||
if p.searxReachable(b.probe) {
|
||||
log.Printf("[%s] SearXNG 已在运行(%s),直接接管", p.name, p.searxURL)
|
||||
p.markSearxOwned()
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), b.up)
|
||||
out, err := p.run(ctx, p.searxDir, "docker", "compose", "up", "-d")
|
||||
cancel()
|
||||
if err != nil {
|
||||
log.Printf("[%s] 拉起 SearXNG 失败(dir=%s,请检查 manage_searxng/searxng_dir 配置): %v;输出: %s",
|
||||
p.name, p.searxDir, err, oneLine(out, 300))
|
||||
return
|
||||
}
|
||||
log.Printf("[%s] 已执行 docker compose up -d(%s):%s", p.name, p.searxDir, oneLine(out, 200))
|
||||
|
||||
deadline := time.Now().Add(b.ready)
|
||||
for time.Now().Before(deadline) {
|
||||
if p.searxReachable(800 * time.Millisecond) {
|
||||
log.Printf("[%s] SearXNG 就绪", p.name)
|
||||
p.markSearxOwned()
|
||||
return
|
||||
}
|
||||
time.Sleep(600 * time.Millisecond)
|
||||
}
|
||||
log.Printf("[%s] SearXNG 已启动但 %s 内未就绪;首次检索会自动等待", p.name, b.ready)
|
||||
p.markSearxOwned()
|
||||
}
|
||||
|
||||
func (p *Plugin) markSearxOwned() {
|
||||
p.searxMu.Lock()
|
||||
p.searxOwned = true
|
||||
p.searxMu.Unlock()
|
||||
}
|
||||
|
||||
// shutdownSearxng 关闭搜索后端。幂等,且有界(内核宽限期 5s,这里最多 4s)。
|
||||
func (p *Plugin) shutdownSearxng() {
|
||||
b := p.budget()
|
||||
p.searxMu.Lock()
|
||||
owned := p.searxOwned
|
||||
p.searxOwned = false
|
||||
p.searxMu.Unlock()
|
||||
|
||||
if !owned {
|
||||
return // 不是我们拉起来的 / 已经关过
|
||||
}
|
||||
if !p.manageSearx || !p.stopOnExit {
|
||||
log.Printf("[%s] 保留 SearXNG 运行(stop_searxng_on_exit=false)", p.name)
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), b.shutdown)
|
||||
defer cancel()
|
||||
out, err := p.run(ctx, p.searxDir, "docker", "compose", "stop", "-t", "2")
|
||||
if err != nil {
|
||||
// 故意只记日志:这里再重试就会拖过内核宽限期,被 SIGKILL 更糟
|
||||
log.Printf("[%s] 关闭 SearXNG 失败(忽略): %v;输出: %s", p.name, err, oneLine(out, 200))
|
||||
return
|
||||
}
|
||||
log.Printf("[%s] 已关闭 SearXNG", p.name)
|
||||
}
|
||||
197
example/deepsearch/searxng_test.go
Normal file
197
example/deepsearch/searxng_test.go
Normal file
@ -0,0 +1,197 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type fakeCall struct {
|
||||
dir string
|
||||
name string
|
||||
args []string
|
||||
}
|
||||
|
||||
func (c fakeCall) String() string { return c.name + " " + strings.Join(c.args, " ") }
|
||||
|
||||
// newFakeRunner 记录调用并返回预设结果
|
||||
func newFakeRunner(calls *[]fakeCall, out string, err error) cmdRunner {
|
||||
var mu sync.Mutex
|
||||
return func(ctx context.Context, dir, name string, args ...string) (string, error) {
|
||||
mu.Lock()
|
||||
*calls = append(*calls, fakeCall{dir: dir, name: name, args: args})
|
||||
mu.Unlock()
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
|
||||
// fastBudget 把就绪窗口压到毫秒级,避免单测真等
|
||||
func fastBudget() searxBudget {
|
||||
return searxBudget{
|
||||
probe: 50 * time.Millisecond,
|
||||
up: time.Second,
|
||||
ready: 200 * time.Millisecond,
|
||||
shutdown: time.Second,
|
||||
}
|
||||
}
|
||||
|
||||
// 1) 后端没跑 → 应执行 docker compose up -d,并认领关闭责任
|
||||
func TestEnsureSearxngStartsWhenUnreachable(t *testing.T) {
|
||||
var calls []fakeCall
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxURL: "http://127.0.0.1:1", searxDir: "/tmp/fake-searx",
|
||||
manageSearx: true, stopOnExit: true, userAgent: "test",
|
||||
bud: fastBudget(), run: newFakeRunner(&calls, "Container searxng-agent Started", nil),
|
||||
}
|
||||
p.ensureSearxng()
|
||||
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("应恰好拉起一次,实际 %d 次:%v", len(calls), calls)
|
||||
}
|
||||
got := calls[0]
|
||||
if got.name != "docker" || strings.Join(got.args, " ") != "compose up -d" {
|
||||
t.Errorf("命令不对:%s", got)
|
||||
}
|
||||
if got.dir != "/tmp/fake-searx" {
|
||||
t.Errorf("工作目录应为配置的 compose 目录,实际 %q", got.dir)
|
||||
}
|
||||
if !p.searxOwned {
|
||||
t.Error("既然是我们拉起的,就应认领关闭责任")
|
||||
}
|
||||
}
|
||||
|
||||
// 2) 后端已在跑 → 不重启,直接接管
|
||||
func TestEnsureSearxngAdoptsRunningBackend(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/healthz" {
|
||||
_, _ = w.Write([]byte("OK"))
|
||||
return
|
||||
}
|
||||
http.NotFound(w, r)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var calls []fakeCall
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxURL: srv.URL, searxDir: "/tmp/fake-searx",
|
||||
manageSearx: true, stopOnExit: true, userAgent: "test",
|
||||
bud: fastBudget(), run: newFakeRunner(&calls, "", nil),
|
||||
}
|
||||
p.ensureSearxng()
|
||||
|
||||
if len(calls) != 0 {
|
||||
t.Errorf("已在跑就不该重启它,实际执行了:%v", calls)
|
||||
}
|
||||
if !p.searxOwned {
|
||||
t.Error("接管后也应负责停止(与「不重启」不冲突)")
|
||||
}
|
||||
}
|
||||
|
||||
// 3) 关掉托管 → 完全不碰 docker
|
||||
func TestEnsureSearxngDisabled(t *testing.T) {
|
||||
var calls []fakeCall
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxURL: "http://127.0.0.1:1", searxDir: "/tmp/fake-searx",
|
||||
manageSearx: false, stopOnExit: true, userAgent: "test",
|
||||
bud: fastBudget(), run: newFakeRunner(&calls, "", nil),
|
||||
}
|
||||
p.ensureSearxng()
|
||||
if len(calls) != 0 || p.searxOwned {
|
||||
t.Errorf("manage_searxng=false 时不该有任何动作:calls=%v owned=%v", calls, p.searxOwned)
|
||||
}
|
||||
}
|
||||
|
||||
// 4) 拉起失败不能让插件起不来(记日志即可)
|
||||
func TestEnsureSearxngFailureNonFatal(t *testing.T) {
|
||||
var calls []fakeCall
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxURL: "http://127.0.0.1:1", searxDir: "/tmp/fake-searx",
|
||||
manageSearx: true, stopOnExit: true, userAgent: "test",
|
||||
bud: fastBudget(), run: newFakeRunner(&calls, "Cannot connect to the Docker daemon", errors.New("exit status 1")),
|
||||
}
|
||||
p.ensureSearxng() // 不应 panic
|
||||
if p.searxOwned {
|
||||
t.Error("没拉起来就不该认领关闭责任(否则停止时会去关一个不是我们起的服务)")
|
||||
}
|
||||
}
|
||||
|
||||
// 5) 停止:关掉我们拉起的后端,且幂等
|
||||
func TestShutdownStopsOwnedBackend(t *testing.T) {
|
||||
var calls []fakeCall
|
||||
runner := newFakeRunner(&calls, "ok", nil)
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxURL: "http://127.0.0.1:1", searxDir: "/tmp/fake-searx",
|
||||
manageSearx: true, stopOnExit: true, userAgent: "test",
|
||||
bud: fastBudget(), run: runner,
|
||||
}
|
||||
p.ensureSearxng()
|
||||
calls = nil
|
||||
|
||||
p.shutdownSearxng()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("应执行一次 compose stop,实际 %v", calls)
|
||||
}
|
||||
if got := strings.Join(calls[0].args, " "); !strings.HasPrefix(got, "compose stop") {
|
||||
t.Errorf("停止命令不对:%s", got)
|
||||
}
|
||||
if p.searxOwned {
|
||||
t.Error("停止后应清掉认领标记")
|
||||
}
|
||||
|
||||
p.shutdownSearxng() // 幂等:不应再调一次
|
||||
if len(calls) != 1 {
|
||||
t.Errorf("重复停止应无副作用,实际 %v", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// 6) 不是我们拉起的 → 停止时不许动它
|
||||
func TestShutdownSkippedWhenNotOwned(t *testing.T) {
|
||||
var calls []fakeCall
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxDir: "/tmp/fake-searx", manageSearx: true, stopOnExit: true,
|
||||
bud: fastBudget(), run: newFakeRunner(&calls, "", nil),
|
||||
}
|
||||
p.shutdownSearxng()
|
||||
if len(calls) != 0 {
|
||||
t.Errorf("不该去停一个我们没起的服务:%v", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// 7) 配了「停止时保留」→ 认领过也不关
|
||||
func TestShutdownKeepsBackendWhenConfigured(t *testing.T) {
|
||||
var calls []fakeCall
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxURL: "http://127.0.0.1:1", searxDir: "/tmp/fake-searx",
|
||||
manageSearx: true, stopOnExit: false, userAgent: "test",
|
||||
bud: fastBudget(), run: newFakeRunner(&calls, "", nil),
|
||||
}
|
||||
p.ensureSearxng()
|
||||
calls = nil
|
||||
p.shutdownSearxng()
|
||||
if len(calls) != 0 {
|
||||
t.Errorf("stop_searxng_on_exit=false 时不应关闭:%v", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// 8) Stop() 自身也要收尾(内核 stdin 关闭路径不会走 stop handler 的注册顺序之外)
|
||||
func TestStopTriggersShutdown(t *testing.T) {
|
||||
var calls []fakeCall
|
||||
p := &Plugin{
|
||||
name: "deepsearch", searxURL: "http://127.0.0.1:1", searxDir: "/tmp/fake-searx",
|
||||
manageSearx: true, stopOnExit: true, userAgent: "test",
|
||||
bud: fastBudget(), run: newFakeRunner(&calls, "", nil),
|
||||
}
|
||||
p.ensureSearxng()
|
||||
calls = nil
|
||||
if err := p.Stop(); err != nil {
|
||||
t.Fatalf("Stop 返回错误: %v", err)
|
||||
}
|
||||
if len(calls) != 1 {
|
||||
t.Errorf("Stop 应触发一次关闭,实际 %v", calls)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user