Commit Graph

4 Commits

Author SHA1 Message Date
3069cfce4e fix(gateway): pass through upstream token usage in streams for all adapters
The prior usage-passthrough fix only covered openai/opencode; the same
empty-choices+usage drop bug remained in the 5 sibling OpenAI-compatible
adapters, and non-OpenAI providers (anthropic/gemini/ollama) never surfaced
streaming usage at all.

- deepseek/github/groq/kimicode/mistral: preserve usage on empty-choices
  chunks and attach it to normal chunks (same pattern as openai.lua)
- anthropic: emit usage from message_start (prompt) and message_delta
  (completion); gateway merges split usage additively
- gemini: read usageMetadata in the stream path
- ollama: fix non-streaming key (usage -> token_usage, matches
  UnifiedResponse json tag) and read prompt_eval_count/eval_count;
  surface counts from the done stream chunk
- gateway: mergeUsage combines usage across chunks (non-zero fields win,
  total recomputed from prompt+completion) so split usage doesn't lose
  the prompt half; single-chunk case (OpenAI) preserved exactly
- usage-only chunks: done=false (no redundant terminal stop), matching
  the Go fallback standardSSEChunk
2026-08-18 23:24:01 +08:00
83e6d88813 feat(gateway): pass through exact upstream token usage in streams
Streaming responses now carry the upstream's real token usage instead of
gateway estimates:
- UnifiedChunk gains an optional Usage field; adapters (opencode, openai)
  extract usage from upstream stream chunks (including the final chunk with
  empty choices) and pass it through.
- standardSSEChunk preserves usage for passthrough adapters.
- Gateway emits the exact usage in the final stream chunk when available,
  falling back to estimates only when the upstream provided none.

Non-streaming usage was already fixed to emit OpenAI-standard keys.
2026-08-18 19:04:44 +08:00
5d50b69153 fix: tool call anchor & wire format, streaming chunk passthrough, WebUI narrow-screen, docs bilingual 2026-08-08 11:26:58 +08:00
f7f76e097d feat: ModelRouter — unified OpenAI-compatible multi-source LLM gateway
- Lua adapters per upstream (transform_request/response/stream_chunk, build_headers signing hooks)
- AUTO priority routing with per-model kind (chat/image), explicit source/model routing
- Per-source concurrency caps with queueing, exponential backoff, AUTO failover
- OpenAI-compatible API: chat completions, SSE streaming, image generations, models
- Gateway key auth, web UI for adapter/source management, runtime persistence
- e2e test running the real binary against mocked upstreams
2026-08-05 15:25:47 +08:00