OpenCode Go (and other strict OpenAI-compatible upstreams) reject a
non-streaming request that carries stream_options with
"stream_options should be set along with stream". The adapter attached it
unconditionally, so every non-stream call through the opencode adapter
failed on those upstreams.
Verified against OpenCode Go: 25/25 configured models now pass a real
completion through the gateway (they previously 400'd).
Test: TestOpenCodeStreamOptionsOnlyWhenStreaming (absent when
non-streaming, include_usage present when streaming).
Three related forwarding defects found by auditing every adapter with a
tool-calling replay (assistant turn with content:[] + tool_calls).
1) content:[] -> content:{} (all 12 openai-adapter sources, plus
deepseek/trae/sensenova/agentrouter/github/groq/kimicode/mistral)
Lua adapters json.decode the request and re-encode it, and an empty Lua
table is indistinguishable from an empty JSON array — the encoder emits
{} for both. Agent clients serialise a tool-calling assistant turn with
no text as content:[], so every pass-through adapter rewrote it to
content:{} — not valid OpenAI (content is string|array|null). Verified
against a live upstream: content:[] produced "400 invalid arguments"
while content:"" was accepted.
Fixed once at the decode boundary (types.ChatMessage.UnmarshalJSON):
empty-array content normalises to "" and an empty tool_calls array is
dropped, so every adapter — including future ones — sees a valid shape.
2) gemini dropped tool_calls and never emitted functionCall /
functionResponse; the tool role also stayed as an invalid role inside
contents and system was not moved to systemInstruction.
3) ollama copied only role/content, dropping tool_calls and the call
attribution entirely (it needs tool_name, not tool_call_id).
Test: TestAdaptersPreserveToolCalls asserts, for every adapter, that the
call id (or function name where the wire format has no id), the function
name, the tool result and the trailing user turn all survive, plus a
negative control for plain text.
An agent client (pi) serialises an assistant turn whose content is only
[thinking, toolCall] as content:[] with tool_calls. The multimodal-strip
pass treated an empty content array as 'nothing left, drop the whole
message' and discarded the tool_calls with it.
The next message is that call's tool result, so it arrived orphaned: the
model saw a result for a call it had never made and re-issued the same
call on every turn — an endless repeated-tool-call loop. Reproduced
against a capture sink: content:[] lost tool_calls, while content:"" and
content:null kept them.
Only messages with neither usable content NOR a tool call now get
dropped. Content that collapses to empty but still has tool_calls or a
tool_call_id is emitted as "" instead.
Test: TestOpenCodeKeepsToolCallWithEmptyContent (plus a negative control
that an image-only message without tool calls is still dropped).
A key with scope=[AUTO] could previously:
1. request ANY concrete model id directly (hasScopeModel/checkModelScope
treated AUTO as a wildcard)
2. see the full 56-model list on /v1/models (intersectModels considered
AUTO as grant-everything)
AUTO now only authorizes the AUTO routing mode. Direct requests to a
specific model require an explicit scope entry.
Also carries agentrouter.lua WAF fingerprint headers (Origin/Referer/
X-Requested-With) already staged on this branch.
Tests: TestHasScopeModelWithSourcePrefix updated; full suite green.
Full-repo audit after the anthropic/openai fix: agentrouter exposes
claude-opus-4-8, so it inherits Anthropic's tool id rule
^[a-zA-Z0-9_-]{1,64}$ and rejects the whole request on a violation, exactly
like justwoker/tabitoken/扇贝. It was the only remaining adapter serving Claude
models without the sanitizer, so a client that had picked up a dirty id (e.g.
"bash:0" from moonshotai/kimi-k3) would still lose every turn here.
Same shape as the other two — inbound tool_calls[].id + tool_call_id, outbound
non-streaming ids and the first streamed fragment — and the test now asserts
all three adapters rewrite an identical input identically, so a client mixing
sources within one session cannot end up with unpaired tool calls.
Audit result: every source exposing a claude/opus/sonnet model (qijiar, toter,
juziai, agentrouter, justwoker, api456, tabitoken) now routes through a
sanitizing adapter.
Anthropic and OpenAI disagree on what the prompt count means:
Anthropic: input_tokens EXCLUDES cached blocks; cache_read_input_tokens and
cache_creation_input_tokens are separate, additive, billed input.
OpenAI: prompt_tokens INCLUDES its cached_tokens subset.
anthropic.lua mapped input_tokens straight onto prompt, so a cache-heavy turn
was doubly wrong: the billed prompt was undercounted by the entire cache
portion, and cached_tokens could exceed prompt_tokens — a cache hit rate above
100% for any client that divides one by the other. cache_creation_input_tokens
was never read at all, so a cache-write turn silently lost those billed tokens.
Worse, the streaming path dropped the cache split entirely: message_delta
carries the FINAL usage and only mapped input/output, so every streamed
response reported no cache information even when the upstream sent it.
All three counts are now summed into prompt, with the read half exposed as
prompt_tokens_details.cached_tokens plus the DeepSeek-legacy hit/miss pair, via
one shared map_usage() used by transform_response, message_start and
message_delta. A reported zero stays distinguishable from "never reported": the
split is emitted whenever either cache field is present, and omitted entirely
when the upstream mentions neither (justwoker reports only input/output plus its
own cost fields, so its output is byte-identical to before). map_usage returns
nil for a countless object, preserving "no usage in this chunk means say
nothing" rather than reporting zeros.
message_start's placeholder count is still emitted: justwoker reports 160 there
and the real 6931 in message_delta, and the gateway's mergeUsage lets the later
non-zero value win.
Anthropic requires tool_use.id / tool_result.tool_use_id to match
^[a-zA-Z0-9_-]{1,64}$ and rejects the WHOLE request otherwise with
REQUEST_BODY_INVALID / "Invalid tool use format". OpenAI has no such rule, so
an OpenAI-compatible model can mint an id like "bash:0"
(xinjianya/moonshotai/kimi-k3 does exactly that).
In a fan-out router that id does not stay local: the client stores it in its
history and replays it to every other source. One such id therefore kills
every Claude slot at once — justwoker, tabitoken and 扇贝 are all
Claude-behind-{OpenAI,Anthropic} — and an AUTO request falls through all four
tiers to whatever tolerant model is left. Observed live: 4 consecutive 503
"all N auto providers failed" with tier 1/2/3 each reporting the same 400.
Both directions are sanitized, in both adapters:
- request: tool_calls[].id and tool_call_id, so poisoned history recovers
- response: non-streaming tool_calls[].id and the first streamed fragment,
so a bad id never enters a client session in the first place
safe_tool_id is pure and deterministic, so a call and its result are rewritten
identically within one request. A rewritten id keeps an 8-hex digest of the
original, without which distinct ids could collapse ("a:b" and "a_b") into a
duplicate/unpaired tool_use. Already-legal ids pass through byte-identical, so
well-behaved traffic is unaffected. openai.lua carries its own copy because
Lua adapters have no shared prelude.
Streamed argument fragments carry no id and must stay id-less, otherwise
index-based accumulation on the client breaks; a test pins that.
Root cause: trae-local-api is deployed to users without the "Fold past
assistant tool_calls into [Called tool: name({...})]" text format that
their own histories already contained. Trae-Local-API-LLM then mimics this
format in subsequent responses. The trae.lua parser only recognized
<tool_call>...</tool_call> or <toolcall>...</toolcall> tags, so
[Called tool: ...] responses were left unparsed and the client received
plain text where a structured tool_calls array should be.
Fix: Add a legacy pattern match at the END of parse_text_tool_calls to
catch the [Called tool: name({args})] shape and emit proper tool_calls.
This is a fallback; models should emit <tool_call> tags per system prompt,
but we tolerate the mimicked form for robustness.
Four adapters handled tool_calls in transform_stream_chunk but lost them in
transform_response, so any NON-streaming tool-using conversation broke on its
second request: the client received finish_reason:"tool_calls" with no
tool_calls payload, replayed an assistant message whose function
name/arguments were empty, and the upstream rejected the next turn with
400 invalid tool_call function, function/name/arguments cannot be empty
The production audit trail shows 46 such failures on sensenova alone.
- sensenova.lua: forward message.tool_calls, decoding the arguments JSON string
into an object as the unified shape expects.
- gemini.lua: collect functionCall parts from candidates[].content.parts. Also
correct finish_reason, since Gemini reports "STOP" even when it emitted a
function call and clients keyed on it treat that as a finished answer.
- ollama.lua: the field was initialized to an empty table and never filled;
fill it and likewise correct done_reason "stop" -> "tool_calls".
trae is a different failure with the same symptom: trae-local-api's OpenAI
endpoint (/v1/chat/completions, src/server.js:353) never reads the request's
`tools` array — only its Anthropic endpoint does — so the relayed model is never
told the tool schema and instead PRINTS a <tool_call>{...}</tool_call> block into
content, leaving message.tool_calls null and finish_reason "stop". An OpenAI
client sees an ordinary completion and its agent loop ends mid-conversation.
trae.lua now recovers the structured call from that text, strips the block from
user-visible content, and corrects finish_reason. Both tag spellings
(<tool_call>/<toolcall>, the latter is what the same codebase's Anthropic prompt
asks for) and all three argument key names (arguments/params/input) are accepted.
This is a defensive fallback: fixing the upstream shim to honour `tools` remains
the real fix, since the model still guesses parameter names.
Tests: TestNonStreamToolCallsPreserved covers all ten OpenAI-shaped adapters,
TestGeminiNonStreamToolCalls and TestOllamaNonStreamToolCalls cover their native
shapes, TestTraeTextToolCallRecovery covers both tag spellings, prose around the
block, and asserts a plain text answer never gains tool_calls.
Verified end-to-end against mock upstreams reproducing each shape: a full
two-round agent loop (tool call -> tool result -> final answer) now completes for
both the structured and the text-emitted variants.
Production oscillated between ~46 MB and ~56 MB RSS with the openai pool
cycling 1 -> 10..12 -> 1 states every couple of minutes, while peak_in_use
never went above 2.
Cause: the batch prewarm sized itself purely on the adapter's ceiling. With
max_concurrent summing to 76, growStep is 8, so any two overlapping requests
warmed 8 states — 6 more than anything was waiting for. A minute later the
janitor correctly reclaimed the surplus, the next pair of overlapping requests
warmed 8 again, and the pool churned boot/discard forever. The elasticity was
working; the growth signal was simply wrong.
Prewarm is now bounded by BOTH limits: the ceiling still caps the step, but the
batch never exceeds p.waiting, the number of goroutines actually blocked on the
pool. Overlapping-but-not-queued traffic (the common case) creates exactly the
states it uses; a genuinely queued burst still ramps in one jump.
TestContentionBatchPrewarms is rewritten to queue real waiters instead of
relying on the ceiling to imply demand, and TestNoPrewarmWithoutWaiters pins the
production shape: two overlapping requests against a 76-wide adapter must create
exactly 2 states.
Production after the first deploy showed the openai pool stuck at 9 states with
peak_in_use=1: a startup burst grew it, and then it never shrank again. The
shrink grace counter was reset by every CHECKOUT, so on a gateway that always
has a request in flight the counter never reached shrinkGraceRounds and the
burst's leftover states were pinned indefinitely — the same monotonic-growth
behaviour this series set out to remove, just with an extra step.
Grace is now reset by GROWTH (a miss that had to boot a state), which is the
actual signal that capacity is short. Ordinary sequential traffic no longer
defers reclaim, while two consecutive quiet-ish rounds are still required so a
gap between two bursts does not tear the pool down.
TestCheckoutResetsGrace asserted the old behaviour and is replaced by
TestGrowthResetsGrace (sequential traffic must NOT defer, growth must) plus
TestBurstLeftoverIsReclaimedUnderSteadyTraffic, which reproduces the production
shape: 9 concurrent holds, then one request per janitor round, and the pool must
still fall back to the resident floor.
sensenova: the deployed /etc/llmsproxy/adapters/sensenova.lua carried a fix that
never made it back into the repo — sensenova-6.8-flash-lite reports its chain of
thought in `reasoning` rather than `reasoning_content`, in both the single-shot
response and the stream deltas. Without this the model's output looked empty.
Repo and deployment now match byte for byte.
trae: the adapter was in use on this deployment but untracked, so a fresh
install had no way to serve the trae source.
ConfigureConcurrency() set each adapter pool's target to the sum of
max_concurrent over its sources (108 on this deployment) and `created` only
ever went UP: once a Lua state was booted it was parked forever, so a
long-running gateway's resident state count was a high-water mark of all
traffic it had ever seen, never of what it currently needs.
Pools are now sized from three live inputs:
* the adapter's MAX CONCURRENCY (sum of max_concurrent) is a ceiling, not a
preallocation, and it sets the growth step:
growStep = clamp(ceil(maxW/8), 1, 8). A 64-wide adapter warms 8 states at
once on a spike; an 8-wide one creeps up one at a time.
* the LIVE CONNECTION COUNT (inUse, i.e. checked-out states) sets the shrink
step: shrinkStep = clamp(ceil(excess/(1+inUse)), 1, excess). With no
connections the slack collapses in a single round; a busy adapter gives up
one state per round so the hot path keeps its warm states.
* how many states already exist (created / len(idle)) decides how much room
is left to grow and how much can be reclaimed.
Batch prewarm only fires on genuine contention (a miss while every existing
state is checked out), so a single sequential caller keeps reusing one state
rather than burning a whole grow step on a cold start. A single VM-level
janitor goroutine (not one per adapter) reclaims idle states every 30s, and
shrinkGraceRounds=2 plus idleHeadroom=1 keep a gap between requests from being
mistaken for the end of a load period; any checkout resets the grace counter.
`created` now decrements on reclaim and on shutdown, and release() closes a
state outright when the ceiling was lowered underneath it, so shrinking
max_concurrent in the config gives memory back immediately instead of parking
orphans until restart.
PoolStats() exposes created/idle/in_use/waiting/max/resident/grow_step/
shrink_step/peak_in_use for the status API.
Measured on the test instance (single mock source, max_concurrent=64):
idle created=1; 50 concurrent requests -> created=10 (peak_in_use=5, ceiling
respected); after 95s of silence -> created=1. On production after deploy: 13
adapters, 1 resident Lua state total with ceilings up to 76.
Live testing across the zen pool showed models report
prompt_tokens_details.cached_tokens even when the hit count is 0 (e.g.
nemotron-3-ultra-free returns cached_tokens:0, audio_tokens:0,
cache_write_tokens:0). The previous >0 guard dropped those objects, so a
cache-enabled upstream looked identical to one without cache support.
- types: PromptTokensDetails.CachedTokens always emitted (drop inner
omitempty) so clients see cached_tokens:0 explicitly; dsh reads it as
a 0% hit instead of 'no data'
- adapters (9): forward prompt_tokens_details whenever the upstream
provides it (presence check instead of >0)
- Req: add cache_reported flag set when usage carried cache accounting;
WebUI shows an amber 0% tag for reported-but-missed rows and keeps
the em-dash only for sources that never report cache data
Live testing proved both sensenova and zen DO return cache fields:
- zen laguna-s-2.1-free: usage.prompt_tokens_details.cached_tokens = 32
(real hit), plus cache_write_tokens/audio_tokens
- sensenova glm-5.2: prompt_tokens_details.cached_tokens present (0 on
short prompts)
The previous round only patched deepseek/openai/anthropic/gemini.lua;
sensenova/opencode (localzen!) and the other adapters still dropped them.
- sensenova/opencode/groq/mistral/github/kimicode: stream + response
cache passthrough (same pattern as openai.lua)
- agentrouter: response passthrough + NEW stream usage forwarding (it
previously dropped the terminal usage-only chunk entirely)
- ollama skipped intentionally: its native API has no cache fields
Verified end-to-end through the gateway: localzen/laguna-s-2.1-free now
returns prompt_tokens_details.cached_tokens=32 to clients, and the request
record carries cache_hit_tokens (both chat and stream paths).
dsh displays cache-hit %, but llmsproxy dropped every upstream's cache
fields — deepseek prompt_cache_hit_tokens, OpenAI prompt_tokens_details.
cached_tokens, anthropic cache_read_input_tokens, gemini cachedContentTokenCount.
Changes:
- TokenUsage: add PromptTokensDetails (with CachedTokens) + PromptCacheHit/Miss
- MarshalJSON: emit prompt_tokens_details.cached_tokens (OpenAI v2 standard)
and prompt_cache_hit/miss_tokens (DeepSeek legacy) — dsh reads the former
first, falls back to the latter
- mergeUsage: preserve cache fields across stream chunks
- standardSSEChunk: parse the upstream raw prompt_tokens_details too
- deepseek.lua: forward prompt_cache_hit/miss_tokens + create
prompt_tokens_details from them
- openai.lua: forward prompt_tokens_details.cached_tokens and legacy
prompt_cache_hit/miss_tokens; normalize legacy hits into the standard
object so dsh sees them regardless of upstream format
- anthropic.lua: map cache_read_input_tokens → prompt_tokens_details
- gemini.lua: map cachedContentTokenCount → prompt_tokens_details
Every upstream formats errors differently, which is adapter territory:
the protocol gains an optional transform_error(status, body) hook and all
built-in adapters implement their own envelope parsing (zen free-pool
labels, anthropic/gemini/ollama/mistral shapes, sensenova quota notes,
agentrouter WAF pages). The core keeps a single uniform fallback: when no
hook yields a reason clients get "api error <status>: unknown error" and
the raw body goes to server logs only.
The gateway hardcoded "stop" on every terminating stream chunk, so
tool-call rounds reported finish_reason=stop and length caps were
invisible to clients. UnifiedChunk now carries finish_reason; adapters
emit it (with empty-string finish reasons like sensenova treated as
non-terminal), standardSSEChunk passes it through for un-adapted
upstreams, [DONE] no longer emits a duplicate reason-less done chunk,
and both streaming paths emit the real reason with "stop" as fallback.
Also vendor sensenova/agentrouter adapters into the repo: they were
WebUI-only uploads and a deploy sync silently removed them while live
AUTO-chain slots still referenced them.
zen fingerprints clients via UA + x-opencode-client/session/request/project
headers; requests without them are routed as anonymous and fail with a
single-chunk network_error stream (tools+stream) or 503 Endpoint is
unavailable (non-stream), which the adapter laundered into empty-but-valid
replies. Derive per-request session/request ids from build_headers meta
(sandbox has no os/math), bump UA to the real client format, map zen
reasoning field to reasoning_content, and set stream_options.include_usage.
The prior usage-passthrough fix only covered openai/opencode; the same
empty-choices+usage drop bug remained in the 5 sibling OpenAI-compatible
adapters, and non-OpenAI providers (anthropic/gemini/ollama) never surfaced
streaming usage at all.
- deepseek/github/groq/kimicode/mistral: preserve usage on empty-choices
chunks and attach it to normal chunks (same pattern as openai.lua)
- anthropic: emit usage from message_start (prompt) and message_delta
(completion); gateway merges split usage additively
- gemini: read usageMetadata in the stream path
- ollama: fix non-streaming key (usage -> token_usage, matches
UnifiedResponse json tag) and read prompt_eval_count/eval_count;
surface counts from the done stream chunk
- gateway: mergeUsage combines usage across chunks (non-zero fields win,
total recomputed from prompt+completion) so split usage doesn't lose
the prompt half; single-chunk case (OpenAI) preserved exactly
- usage-only chunks: done=false (no redundant terminal stop), matching
the Go fallback standardSSEChunk
Streaming responses now carry the upstream's real token usage instead of
gateway estimates:
- UnifiedChunk gains an optional Usage field; adapters (opencode, openai)
extract usage from upstream stream chunks (including the final chunk with
empty choices) and pass it through.
- standardSSEChunk preserves usage for passthrough adapters.
- Gateway emits the exact usage in the final stream chunk when available,
falling back to estimates only when the upstream provided none.
Non-streaming usage was already fixed to emit OpenAI-standard keys.
- adapters/opencode.lua: opencode.ai zen free pool adapter — sends the
opencode client User-Agent (zen fingerprints clients by UA; non-official
clients hit FreeUsageLimitError); pairs with api_key: public
- config: no config file ships in the repo; first run generates a default
config at the -config path with a random admin key, loopback listen and a
keyless zen source (config.EnsureDefault); remove config.example.yaml
- lua: seed bundled adapters from the embedded FS instead of a hardcoded
name list
- ui: widen model kind select (chat was clipped to 'cha')
- phase 5 bugfixes: stats ms/s bucket mixing, cleanScopes nil, ctx.Err
guards, direct-path ModelAvailable, empty stream body failure,
bestImageModel rewrite, transform failure recording, Core.mu, timer,
effective model for tool-calls
