mirror of
https://gitcode.com/JianFeeeee/HomeAgent.git
synced 2026-10-03 15:53:56 +00:00
feat(c-core): 内核编解码层 C 化第一刀 —— L1 纯函数层落地并打通构建链
第一刀只做三个纯函数(窗口推断 / token 估算 / token 截断), 价值不在功能(Go 版没问题),而在打通「Go → cgo → C」全链路并 建立可复现的对照范式,后面每扩一个函数都复用它。 ## 为什么是这三个 按「无状态 → 有状态」分层,L1 协议编解码最安全:纯 string in → struct out, 不碰网络、不碰 Lua、不碰 goroutine。三个函数更是同一组纯算术,最小可验证切片。 ## 关键设计:包内符号链接,不链接静态库、不 include 包外源 三种做法都实测过,只有一种同时满足「可构建 + 可交叉编译 + 缓存可跟踪」: 1. ❌ 链接 `csrc/build/libha_codec.a`(原方案) - .a 是构建产物、不入库(.gitignore 的 build/ 命中 csrc/build/), 而发布脚本原先并不产出它 ⇒「不入库 + 不生成」两头空, 实测报 `cannot find .../libha_codec.a` - 交叉编译 linux/arm64(homed 真实发布目标)时,宿主 x86-64 的 .a 被链进目标产物,实测报 `file in wrong format` 2. ❌ `#include "../../../csrc/src/ha_codec.c"`(包外相对包含) ★ Go 构建缓存**不跟踪包外被 #include 的 C 文件**。实测:包外源把返回值 7→8,`go test` 依然通过(缓存命中、静默沿用旧代码);同样改动落在包内 文件时立即判红。对「逐步推进 C 化」这是致命的——改 C 源码不生效且无报错。 (包内 shim `#include` 包外源同样漏跟踪,已实测排除。) 3. ✅ 包内符号链接 `internal/agent/api/ha_codec.{c,h}` → `csrc/` 文件在包目录内 ⇒ 缓存按内容正确跟踪;只有一份权威源 ⇒ 无副本漂移, 也不需要「同步 C 源」的 make 目标。 ## 不需要额外 build tag ha_codec 是零依赖纯 C99 源码内联编译,不需要外部库或工具链前提; 而 homed 本就强制 cgo(sqlite3 + gojieba),故 C 路径自然生效。 只用 `cgo` / `!cgo` 一组约束(对比 onnxruntime:那个需运行期 .so,故必须显式 tag)。 ## 顺带修掉的既有缺陷(非 C 化引入,但一直缺覆盖) - Makefile 的 arm64 目标缺 CC/CXX:cgo 回退到宿主 g++,报 `gcc_arm64.S: no such instruction: 'stp x29,x30,[sp,'` (deploy/packaging/build.sh:49 一直是对的,Makefile 漏了) - Makefile 的 arm64 目标缺 .syso 隔离:cmd/{homed,waiter}/*.syso 是 Windows COFF 资源对象,Go 会把同目录 .syso 无条件链进任何目标,交叉到非 Windows 平台报 `file format not recognized`(build.sh 有 hide_syso_for_target) ## 验证(每条可复现) - `make build-linux-arm64` → ELF 64-bit LSB executable, ARM aarch64(82MB) - `bash deploy/packaging/build.sh linux/arm64 homed` → ELF aarch64(79MB) - 移走 csrc/build/ 后 homed(cgo)与 waiter(CGO=0)均能构建 - 变异 C 源(131072→777)后**同一缓存**下 go test 立即 FAIL(改前:仍报 ok) - `make check-codec-paths` 两条路径 OK;`make csrc-test` C 契约测试 100% - 黄金对照:手写用例 + 2000 次随机对拍,C 与纯 Go 逐值相等 - 全量 `go test -count=1 ./...` → 57 包:38 ok + 19 无测试 + 0 FAIL 新增 `make check-codec-paths` 防回归:只测一条路径时,另一条的破坏不会被发现。 ## 文档 - `docs/zh/c-core/llm-orchestration-c.md` 同步为「已落地」,并更正因 「Windows 原生已放弃」而过时的 §2.3(回退路径的理由需重述) - plan.md 的 P0-1 标记为已修复,附实测证据
This commit is contained in:
57
csrc/CMakeLists.txt
Normal file
57
csrc/CMakeLists.txt
Normal file
@ -0,0 +1,57 @@
|
||||
cmake_minimum_required(VERSION 3.10)
|
||||
project(ha_codec VERSION 0.1.0 LANGUAGES C)
|
||||
|
||||
# ============================================================
|
||||
# ha_codec — HomeAgent 内核编解码层(C 实现)
|
||||
#
|
||||
# 零外部依赖,纯 C99。产出静态库供 homed 经 cgo 链接,
|
||||
# 同时可独立用于其他端(鸿蒙 / 嵌入式 / C SDK)。
|
||||
#
|
||||
# 使用方式:
|
||||
# add_subdirectory(path/to/csrc)
|
||||
# target_link_libraries(my_app ha_codec)
|
||||
# target_include_directories(my_app PRIVATE ${HA_CODEC_INCLUDE_DIR})
|
||||
# ============================================================
|
||||
|
||||
option(BUILD_SHARED_LIBS "Build ha_codec as shared library" OFF)
|
||||
option(BUILD_TESTS "Build ha_codec tests" OFF)
|
||||
|
||||
set(HA_CODEC_SRC
|
||||
src/ha_codec.c
|
||||
)
|
||||
|
||||
if(BUILD_SHARED_LIBS)
|
||||
add_library(ha_codec SHARED ${HA_CODEC_SRC})
|
||||
if(WIN32)
|
||||
set_target_properties(ha_codec PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS ON)
|
||||
endif()
|
||||
else()
|
||||
add_library(ha_codec STATIC ${HA_CODEC_SRC})
|
||||
endif()
|
||||
|
||||
set(HA_CODEC_INCLUDE ${CMAKE_CURRENT_SOURCE_DIR}/include)
|
||||
target_include_directories(ha_codec PUBLIC ${HA_CODEC_INCLUDE})
|
||||
|
||||
# 不链接任何外部库 —— 保持与 ha_remotedevice 同一克制标准
|
||||
target_link_libraries(ha_codec PRIVATE)
|
||||
|
||||
set(HA_CODEC_INCLUDE_DIR ${HA_CODEC_INCLUDE} CACHE INTERNAL "ha_codec include directories")
|
||||
|
||||
install(TARGETS ha_codec
|
||||
EXPORT ha_codec-targets
|
||||
LIBRARY DESTINATION lib
|
||||
ARCHIVE DESTINATION lib
|
||||
RUNTIME DESTINATION bin
|
||||
INCLUDES DESTINATION include
|
||||
)
|
||||
install(DIRECTORY include/ DESTINATION include)
|
||||
|
||||
# ============================================================
|
||||
# 测试
|
||||
# ============================================================
|
||||
if(BUILD_TESTS)
|
||||
add_executable(ha_codec_test test/test_ha_codec.c)
|
||||
target_link_libraries(ha_codec_test PRIVATE ha_codec)
|
||||
enable_testing()
|
||||
add_test(NAME ha_codec_test COMMAND ha_codec_test)
|
||||
endif()
|
||||
67
csrc/include/ha_codec.h
Normal file
67
csrc/include/ha_codec.h
Normal file
@ -0,0 +1,67 @@
|
||||
#ifndef HA_CODEC_H
|
||||
#define HA_CODEC_H
|
||||
|
||||
/*
|
||||
* ha_codec — HomeAgent 内核编解码层(C 实现)
|
||||
*
|
||||
* ============================ 接口冻结声明 ============================
|
||||
* 本头文件是对外契约。函数签名、语义、返回值一经发布即为冻结接口,
|
||||
* 修改必须走大版本流程(与 third_party/homeagent-sdk 同一冻结标准)。
|
||||
*
|
||||
* 设计约束(见 docs/zh/c-core/llm-orchestration-c.md §四):
|
||||
* 1. 只吃 const char* + 长度,出数值/JSON 串
|
||||
* 2. 不回调 Go、不传 Go 指针
|
||||
* 3. 不长期持有 malloc 内存;需要出参的用调用方缓冲区
|
||||
* 4. 无状态、纯函数、线程安全(不写全局可变状态)
|
||||
*
|
||||
* 当前覆盖:L1 协议编解码层中的纯计算部分(第一个最小切片)。
|
||||
*/
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* ==================== 模型上下文窗口推断 ==================== */
|
||||
|
||||
/* 无法从模型名推断时的哨兵值(与 Go 侧一致)。
|
||||
*
|
||||
* 为什么返回哨兵而不是直接给兜底值:调用方需要区分「真推断出了」与
|
||||
* 「推断不出、只能兜底」——后者要打一行日志(窗口被低估必须可见),
|
||||
* 并提示部署方用 per-source context_window 显式声明。
|
||||
* 若 C 侧直接返回兜底值,调用方就永远分不清这两种情况。 */
|
||||
#define HA_CODEC_CONTEXT_WINDOW_UNKNOWN (-1)
|
||||
|
||||
/* 由模型名推断最大上下文窗口(token 数);推断不出返回
|
||||
* HA_CODEC_CONTEXT_WINDOW_UNKNOWN。model 为 NULL 时同样返回 UNKNOWN。
|
||||
*
|
||||
* model 为 UTF-8 字符串,匹配大小写不敏感。
|
||||
* 语义必须与 Go 侧 modelContextWindowPure 逐值一致(黄金对照测试钉死)。 */
|
||||
int ha_codec_model_context_window(const char *model);
|
||||
|
||||
/* ==================== token 估算与截断 ==================== */
|
||||
|
||||
/* 粗略估算 token 数。
|
||||
*
|
||||
* 规则(与 Go 侧 EstimateTokens 一致):中文 ~1.5 token/字、英文 ~0.3 token/字符,
|
||||
* 保守取 max(1, runeCount * 2)。text 为 NULL 或空串返回 0。
|
||||
*
|
||||
* 注意:按 UTF-8 **字符数**(rune)计,不是字节数。 */
|
||||
int ha_codec_estimate_tokens(const char *text);
|
||||
|
||||
/* 截断字符串至不超过 maxTokens 估计值,返回写入 out 的字节数(不含结尾 NUL)。
|
||||
*
|
||||
* 语义与 Go 侧 TruncateByTokens 一致:从开头保留 maxTokens/2 个字符。
|
||||
* maxTokens <= 0 或 text 为空时写入空串。
|
||||
*
|
||||
* out 由调用方提供,容量须为 outCap(含结尾 NUL);函数保证 NUL 结尾、
|
||||
* 不越界写。返回值是实际写入的字节数(可能因 outCap 不足而短于完整截断结果)。 */
|
||||
size_t ha_codec_truncate_by_tokens(const char *text, int max_tokens,
|
||||
char *out, size_t out_cap);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* HA_CODEC_H */
|
||||
191
csrc/src/ha_codec.c
Normal file
191
csrc/src/ha_codec.c
Normal file
@ -0,0 +1,191 @@
|
||||
/*
|
||||
* ha_codec.c — HomeAgent 内核编解码层(C 实现)
|
||||
*
|
||||
* 第一个最小切片:模型窗口推断 + token 估算/截断。
|
||||
* 语义必须与 Go 侧实现逐值一致,由黄金对照测试钉死。
|
||||
*/
|
||||
|
||||
#include "ha_codec.h"
|
||||
|
||||
#include <ctype.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
/* ---------------------------------------------------------------- */
|
||||
/* 小工具 */
|
||||
/* ---------------------------------------------------------------- */
|
||||
|
||||
/* 在 s 中查找子串 sub(子串已小写)。s 需已是小写。找不到返回 NULL。 */
|
||||
static const char *find_sub(const char *s, const char *sub) {
|
||||
return strstr(s, sub);
|
||||
}
|
||||
|
||||
/* 分配一份小写副本。调用方负责 free。失败返回 NULL。 */
|
||||
static char *lower_dup(const char *s) {
|
||||
if (s == NULL) {
|
||||
return NULL;
|
||||
}
|
||||
size_t n = strlen(s);
|
||||
char *p = (char *)malloc(n + 1);
|
||||
if (p == NULL) {
|
||||
return NULL;
|
||||
}
|
||||
for (size_t i = 0; i < n; i++) {
|
||||
/* 只对 ASCII 做小写;UTF-8 多字节原样保留(与 Go strings.ToLower 对
|
||||
* 中文不改变结果一致——Go 会把非 ASCII 也处理,但模型名都是 ASCII)。 */
|
||||
unsigned char c = (unsigned char)s[i];
|
||||
p[i] = (char)((c < 0x80) ? tolower(c) : c);
|
||||
}
|
||||
p[n] = '\0';
|
||||
return p;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- */
|
||||
/* 模型上下文窗口推断 */
|
||||
/* ---------------------------------------------------------------- */
|
||||
|
||||
int ha_codec_model_context_window(const char *model) {
|
||||
if (model == NULL) {
|
||||
return HA_CODEC_CONTEXT_WINDOW_UNKNOWN;
|
||||
}
|
||||
|
||||
char *m = lower_dup(model);
|
||||
if (m == NULL) {
|
||||
return HA_CODEC_CONTEXT_WINDOW_UNKNOWN;
|
||||
}
|
||||
|
||||
int result = HA_CODEC_CONTEXT_WINDOW_UNKNOWN;
|
||||
|
||||
/* 顺序与 Go 侧 switch 分支**严格一致**:先匹配到的分支胜出。
|
||||
* 这不是「随便一组 if」,顺序错了就会给出不同窗口。 */
|
||||
if (find_sub(m, "deepseek-v4") || find_sub(m, "deepseek-v3")) {
|
||||
result = 1048576;
|
||||
} else if (find_sub(m, "deepseek-r1") || find_sub(m, "deepseek-chat")) {
|
||||
result = 65536;
|
||||
} else if (find_sub(m, "gpt-4") &&
|
||||
(find_sub(m, "turbo") || find_sub(m, "mini") || find_sub(m, "omni"))) {
|
||||
result = 128000;
|
||||
} else if (find_sub(m, "gpt-4")) {
|
||||
result = 8192;
|
||||
} else if (find_sub(m, "gpt-3.5")) {
|
||||
result = 16384;
|
||||
} else if (find_sub(m, "claude-3.5") || find_sub(m, "claude-3")) {
|
||||
result = 200000;
|
||||
} else if (find_sub(m, "claude")) {
|
||||
result = 100000;
|
||||
} else if (find_sub(m, "gemini-1.5") || find_sub(m, "gemini-2")) {
|
||||
result = 1048576;
|
||||
} else if (find_sub(m, "gemini")) {
|
||||
result = 32768;
|
||||
} else if (find_sub(m, "qwen")) {
|
||||
result = 131072;
|
||||
} else if (find_sub(m, "glm") || find_sub(m, "chatglm")) {
|
||||
result = 131072;
|
||||
} else if (find_sub(m, "llama-3")) {
|
||||
result = 8192;
|
||||
} else if (find_sub(m, "llama-2")) {
|
||||
result = 4096;
|
||||
} else if (find_sub(m, "mistral") || find_sub(m, "mixtral")) {
|
||||
result = 32768;
|
||||
} else if (find_sub(m, "yi-") || find_sub(m, "零一")) {
|
||||
result = 200000;
|
||||
} else if (find_sub(m, "moonshot") || find_sub(m, "kimi")) {
|
||||
result = 131072;
|
||||
}
|
||||
|
||||
free(m);
|
||||
return result;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- */
|
||||
/* token 估算 */
|
||||
/* ---------------------------------------------------------------- */
|
||||
|
||||
/* 计 UTF-8 字符数(rune 数)并返回下一字符起点。
|
||||
* 非法字节按 1 字符前进(不吞字节),保证不会死循环。 */
|
||||
static size_t utf8_next(const char *s, size_t remaining) {
|
||||
unsigned char c = (unsigned char)s[0];
|
||||
size_t len = 1;
|
||||
if (c >= 0xF0 && remaining >= 4) {
|
||||
len = 4;
|
||||
} else if (c >= 0xE0 && remaining >= 3) {
|
||||
len = 3;
|
||||
} else if (c >= 0xC0 && remaining >= 2) {
|
||||
len = 2;
|
||||
}
|
||||
return len;
|
||||
}
|
||||
|
||||
int ha_codec_estimate_tokens(const char *text) {
|
||||
if (text == NULL || text[0] == '\0') {
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t n = strlen(text);
|
||||
size_t runes = 0;
|
||||
size_t i = 0;
|
||||
while (i < n) {
|
||||
i += utf8_next(text + i, n - i);
|
||||
runes++;
|
||||
}
|
||||
|
||||
/* 与 Go 侧一致:t = runeCount * 2;t < 1 时取 1。
|
||||
* runes > 0 时 t >= 2,故只需处理溢出与下限。 */
|
||||
if (runes > (size_t)0x3FFFFFFF) { /* 防 int 溢出 */
|
||||
return 0x7FFFFFFF;
|
||||
}
|
||||
int t = (int)(runes * 2);
|
||||
if (t < 1) {
|
||||
return 1;
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
/* ---------------------------------------------------------------- */
|
||||
/* 按 token 截断 */
|
||||
/* ---------------------------------------------------------------- */
|
||||
|
||||
size_t ha_codec_truncate_by_tokens(const char *text, int max_tokens,
|
||||
char *out, size_t out_cap) {
|
||||
if (out == NULL || out_cap == 0) {
|
||||
return 0;
|
||||
}
|
||||
out[0] = '\0';
|
||||
|
||||
if (max_tokens <= 0 || text == NULL || text[0] == '\0') {
|
||||
return 0;
|
||||
}
|
||||
|
||||
size_t n = strlen(text);
|
||||
|
||||
/* 先算 rune 数:与 Go 侧 len([]rune(s))*2 <= maxTokens 的短路一致 */
|
||||
size_t runes = 0;
|
||||
size_t i = 0;
|
||||
while (i < n) {
|
||||
i += utf8_next(text + i, n - i);
|
||||
runes++;
|
||||
}
|
||||
|
||||
/* 未超限:整体返回 */
|
||||
if (runes <= (size_t)0x3FFFFFFF && (int)(runes * 2) <= max_tokens) {
|
||||
size_t copy = (n < out_cap - 1) ? n : (out_cap - 1);
|
||||
memcpy(out, text, copy);
|
||||
out[copy] = '\0';
|
||||
return copy;
|
||||
}
|
||||
|
||||
/* 保留 maxTokens/2 个字符(与 Go 一致:keep := maxTokens / 2,整数除法) */
|
||||
size_t keep = (size_t)(max_tokens / 2);
|
||||
|
||||
size_t byte_end = 0;
|
||||
size_t kept = 0;
|
||||
while (kept < keep && byte_end < n) {
|
||||
byte_end += utf8_next(text + byte_end, n - byte_end);
|
||||
kept++;
|
||||
}
|
||||
|
||||
size_t copy = (byte_end < out_cap - 1) ? byte_end : (out_cap - 1);
|
||||
memcpy(out, text, copy);
|
||||
out[copy] = '\0';
|
||||
return copy;
|
||||
}
|
||||
131
csrc/test/test_ha_codec.c
Normal file
131
csrc/test/test_ha_codec.c
Normal file
@ -0,0 +1,131 @@
|
||||
/*
|
||||
* test_ha_codec.c — ha_codec C 侧契约测试
|
||||
*
|
||||
* 编译运行(无 cmake 亦可):
|
||||
* gcc -std=c99 -I../include ../src/ha_codec.c test_ha_codec.c -o test_ha_codec && ./test_ha_codec
|
||||
*
|
||||
* 这一层钉死 C 实现的语义;与 Go 的逐值一致由黄金对照测试负责(双保险)。
|
||||
*/
|
||||
|
||||
#include "ha_codec.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
|
||||
static int g_fail = 0;
|
||||
static int g_pass = 0;
|
||||
|
||||
static void check_int(const char *what, int got, int want) {
|
||||
if (got != want) {
|
||||
printf(" [FAIL] %s: got %d, want %d\n", what, got, want);
|
||||
g_fail++;
|
||||
} else {
|
||||
g_pass++;
|
||||
}
|
||||
}
|
||||
|
||||
static void check_str(const char *what, const char *got, const char *want) {
|
||||
if (strcmp(got, want) != 0) {
|
||||
printf(" [FAIL] %s: got \"%s\", want \"%s\"\n", what, got, want);
|
||||
g_fail++;
|
||||
} else {
|
||||
g_pass++;
|
||||
}
|
||||
}
|
||||
|
||||
static void check_size(const char *what, size_t got, size_t want) {
|
||||
if (got != want) {
|
||||
printf(" [FAIL] %s: got %zu, want %zu\n", what, got, want);
|
||||
g_fail++;
|
||||
} else {
|
||||
g_pass++;
|
||||
}
|
||||
}
|
||||
|
||||
static void test_context_window(void) {
|
||||
printf("model_context_window:\n");
|
||||
check_int("deepseek-v4.1-flash",
|
||||
ha_codec_model_context_window("deepseek/deepseek-v4.1-flash"), 1048576);
|
||||
check_int("deepseek-v4-flash",
|
||||
ha_codec_model_context_window("deepseek-v4-flash"), 1048576);
|
||||
check_int("deepseek-chat",
|
||||
ha_codec_model_context_window("deepseek-chat"), 65536);
|
||||
check_int("claude-opus-5",
|
||||
ha_codec_model_context_window("claude-opus-5"), 100000);
|
||||
check_int("gpt-4-turbo",
|
||||
ha_codec_model_context_window("gpt-4-turbo"), 128000);
|
||||
check_int("llama-3-70b",
|
||||
ha_codec_model_context_window("llama-3-70b"), 8192);
|
||||
check_int("AUTO (unknown)",
|
||||
ha_codec_model_context_window("AUTO"), HA_CODEC_CONTEXT_WINDOW_UNKNOWN);
|
||||
check_int("NULL (unknown)",
|
||||
ha_codec_model_context_window(NULL), HA_CODEC_CONTEXT_WINDOW_UNKNOWN);
|
||||
check_int("case-insensitive",
|
||||
ha_codec_model_context_window("QWEN-MAX"), 131072);
|
||||
check_int("moonshot",
|
||||
ha_codec_model_context_window("moonshot-v1-128k"), 131072);
|
||||
/* 分支顺序:gpt-4-turbo 必须先于裸 gpt-4 命中 */
|
||||
check_int("gpt-4-mini (branch order)",
|
||||
ha_codec_model_context_window("gpt-4-mini"), 128000);
|
||||
check_int("gpt-4 (bare)",
|
||||
ha_codec_model_context_window("gpt-4"), 8192);
|
||||
/* claude-3 必须先于裸 claude */
|
||||
check_int("claude-3-opus (branch order)",
|
||||
ha_codec_model_context_window("claude-3-opus"), 200000);
|
||||
}
|
||||
|
||||
static void test_estimate_tokens(void) {
|
||||
printf("estimate_tokens:\n");
|
||||
check_int("empty", ha_codec_estimate_tokens(""), 0);
|
||||
check_int("NULL", ha_codec_estimate_tokens(NULL), 0);
|
||||
/* "abc" = 3 rune * 2 = 6 */
|
||||
check_int("ascii abc", ha_codec_estimate_tokens("abc"), 6);
|
||||
/* "你好" = 2 rune * 2 = 4(注意:不是字节数 6) */
|
||||
check_int("chinese 2 chars", ha_codec_estimate_tokens("你好"), 4);
|
||||
/* 混合 "a你" = 2 rune * 2 = 4 */
|
||||
check_int("mixed", ha_codec_estimate_tokens("a你"), 4);
|
||||
/* 4 字节 emoji:1 rune * 2 = 2 */
|
||||
check_int("emoji", ha_codec_estimate_tokens("\xF0\x9F\x98\x80"), 2);
|
||||
}
|
||||
|
||||
static void test_truncate(void) {
|
||||
printf("truncate_by_tokens:\n");
|
||||
char buf[64];
|
||||
|
||||
/* max_tokens<=0 → 空 */
|
||||
ha_codec_truncate_by_tokens("hello", 0, buf, sizeof(buf));
|
||||
check_str("max_tokens=0", buf, "");
|
||||
|
||||
/* 未超限 → 原样返回 */
|
||||
size_t n = ha_codec_truncate_by_tokens("abc", 100, buf, sizeof(buf));
|
||||
check_str("no truncation", buf, "abc");
|
||||
check_size("no truncation len", n, 3);
|
||||
|
||||
/* "abcdefghij" = 10 rune → 20 tokens;max=8 → keep=4 → "abcd" */
|
||||
n = ha_codec_truncate_by_tokens("abcdefghij", 8, buf, sizeof(buf));
|
||||
check_str("keep 4", buf, "abcd");
|
||||
check_size("keep 4 len", n, 4);
|
||||
|
||||
/* 中文按 rune 截断,不切碎 UTF-8:"你好世界" 4 rune,max=4 → keep=2 → "你好" */
|
||||
n = ha_codec_truncate_by_tokens("你好世界", 4, buf, sizeof(buf));
|
||||
check_str("chinese keep 2", buf, "你好");
|
||||
check_size("chinese keep 2 len (bytes)", n, 6);
|
||||
|
||||
/* 缓冲区不足:必须 NUL 结尾且不越界 */
|
||||
char tiny[4];
|
||||
n = ha_codec_truncate_by_tokens("abcdefghij", 100, tiny, sizeof(tiny));
|
||||
check_size("tiny buf len", n, 3);
|
||||
check_str("tiny buf NUL-terminated", tiny, "abc");
|
||||
|
||||
/* out_cap=0 不写 */
|
||||
check_size("zero cap", ha_codec_truncate_by_tokens("abc", 100, tiny, 0), 0);
|
||||
}
|
||||
|
||||
int main(void) {
|
||||
printf("=== ha_codec 契约测试 ===\n\n");
|
||||
test_context_window();
|
||||
test_estimate_tokens();
|
||||
test_truncate();
|
||||
printf("\n=== 结果: %d passed, %d failed ===\n", g_pass, g_fail);
|
||||
return g_fail == 0 ? 0 : 1;
|
||||
}
|
||||
Reference in New Issue
Block a user