Follow-on to the session-stability fix. "Per source" already made the
prefix cache hit, but it puts every conversation into one upstream session.
Using the client's own session id is not possible: capturing real agent
traffic (tcpdump on 127.0.0.1:8081) shows generic OpenAI clients send NO
session identifier at all — no user / session_id / conversation_id /
metadata in the body, and no session header (only X-Stainless-* plus
User-Agent: pi). The x-opencode-session the Go endpoint asks for is an
OpenCode native-client concept that a generic client cannot forward.
Since history is replayed every turn, the FIRST user message is invariant
for the life of a conversation, so it is used as the conversation
fingerprint. The session becomes stable within a conversation and distinct
across conversations; requests with no user message fall back to per-source
stability.
Measured through the gateway (same 5.7k-token prompt): 2nd call
cached_tokens=5504, and an unrelated conversation gets its own session.
Test: TestOpenCodeSessionIsStableForCache covers same-conversation
stability, cross-conversation separation, per-request request ids and the
sessionless fallback.
The opencode adapters derived x-opencode-session from meta.timestamp, i.e. a
brand new session on every request. The upstream prefix cache is
session-scoped, so no request could ever hit it, and the cache fields the
endpoint does report (prompt_tokens_details.cached_tokens,
prompt_cache_hit_tokens/prompt_cache_miss_tokens) always came back 0/absent.
Measured against the live endpoint, same 6032-token prompt:
fixed session id -> 2nd call: hit 5888, miss 144
rotating session id -> every call: hit 0, miss 6032
Fix: derive the session from the source name (stable), matching how
x-opencode-project is already derived. x-opencode-request stays unique per
request — it is only a request identifier, not part of the cache key.
Applied to both opencodego and opencodezen.
Through the gateway the same prompt now reports, on the 2nd call:
details={'cached_tokens': 5888} hit=5888 miss=144 (non-streaming)
prompt_tokens_details={'cached_tokens': 5888} (streaming)
Test: TestOpenCodeSessionIsStableForCache asserts the session is stable
across requests for one source while the request id differs.
v1.5.4 stopped stripping reasoning_content, which fixes clients that send
it — but most agent clients (pi included) never store or replay their
reasoning, keeping only the tool call. OpenCode Go validates the field on
any assistant turn that carries tool_calls and rejects the whole request:
400 invalid_request_error: The `reasoning_content` in the thinking mode
must be passed back to the API.
Verified against the live endpoint that an EMPTY string satisfies the
check, so the adapter now fills in "" when a tool-calling assistant turn
has no reasoning_content. Nothing is fabricated: the reasoning shown to
the client is still exactly what the upstream returned for that turn.
Measured: with a tool_call + tool_result history and no reasoning_content,
all 25 configured Go models returned 400 before and all 25 answer
correctly now.
Test: TestOpenCodeGoVsZenReasoning also pins that a plain assistant turn
(no tool calls) must NOT gain the field.
Zen (https://opencode.ai/zen/v1) and Go (https://opencode.ai/zen/go/v1)
are different services with different requirements, and one shared adapter
could not satisfy both.
The decisive difference is reasoning_content:
* OpenCode Go runs thinking models and REQUIRES the assistant turn's
reasoning_content to be echoed back. The shared adapter stripped it
(msg.reasoning_content = nil), so every replay of a thinking turn
failed with:
400 invalid_request_error: The `reasoning_content` in the thinking
mode must be passed back to the API.
Reproduced directly: the same request with reasoning_content -> 200,
without -> 400. That is why the Go tier never worked in an agent loop.
* The Zen free pool must not receive it, so it keeps stripping.
Both adapters keep the earlier fixes they share (never drop an assistant
turn carrying tool_calls; send stream_options only when streaming; role
whitelist; multimodal strip) and the opencode client fingerprint headers —
the Go endpoint additionally REQUIRES x-opencode-session, which the
adapter already sends.
config: localzen -> opencodezen, gozen -> opencodego.
Verified: all 25 gozen models answer correctly through the gateway with a
thinking + tool_call + tool_result history (was 0/25 before), streaming
included; the Zen free models still pass.
Test: TestOpenCodeGoVsZenReasoning pins the Go-keeps / Zen-strips split.
OpenCode Go (and other strict OpenAI-compatible upstreams) reject a
non-streaming request that carries stream_options with
"stream_options should be set along with stream". The adapter attached it
unconditionally, so every non-stream call through the opencode adapter
failed on those upstreams.
Verified against OpenCode Go: 25/25 configured models now pass a real
completion through the gateway (they previously 400'd).
Test: TestOpenCodeStreamOptionsOnlyWhenStreaming (absent when
non-streaming, include_usage present when streaming).
Three related forwarding defects found by auditing every adapter with a
tool-calling replay (assistant turn with content:[] + tool_calls).
1) content:[] -> content:{} (all 12 openai-adapter sources, plus
deepseek/trae/sensenova/agentrouter/github/groq/kimicode/mistral)
Lua adapters json.decode the request and re-encode it, and an empty Lua
table is indistinguishable from an empty JSON array — the encoder emits
{} for both. Agent clients serialise a tool-calling assistant turn with
no text as content:[], so every pass-through adapter rewrote it to
content:{} — not valid OpenAI (content is string|array|null). Verified
against a live upstream: content:[] produced "400 invalid arguments"
while content:"" was accepted.
Fixed once at the decode boundary (types.ChatMessage.UnmarshalJSON):
empty-array content normalises to "" and an empty tool_calls array is
dropped, so every adapter — including future ones — sees a valid shape.
2) gemini dropped tool_calls and never emitted functionCall /
functionResponse; the tool role also stayed as an invalid role inside
contents and system was not moved to systemInstruction.
3) ollama copied only role/content, dropping tool_calls and the call
attribution entirely (it needs tool_name, not tool_call_id).
Test: TestAdaptersPreserveToolCalls asserts, for every adapter, that the
call id (or function name where the wire format has no id), the function
name, the tool result and the trailing user turn all survive, plus a
negative control for plain text.
An agent client (pi) serialises an assistant turn whose content is only
[thinking, toolCall] as content:[] with tool_calls. The multimodal-strip
pass treated an empty content array as 'nothing left, drop the whole
message' and discarded the tool_calls with it.
The next message is that call's tool result, so it arrived orphaned: the
model saw a result for a call it had never made and re-issued the same
call on every turn — an endless repeated-tool-call loop. Reproduced
against a capture sink: content:[] lost tool_calls, while content:"" and
content:null kept them.
Only messages with neither usable content NOR a tool call now get
dropped. Content that collapses to empty but still has tool_calls or a
tool_call_id is emitted as "" instead.
Test: TestOpenCodeKeepsToolCallWithEmptyContent (plus a negative control
that an image-only message without tool calls is still dropped).
A key with scope=[AUTO] could previously:
1. request ANY concrete model id directly (hasScopeModel/checkModelScope
treated AUTO as a wildcard)
2. see the full 56-model list on /v1/models (intersectModels considered
AUTO as grant-everything)
AUTO now only authorizes the AUTO routing mode. Direct requests to a
specific model require an explicit scope entry.
Also carries agentrouter.lua WAF fingerprint headers (Origin/Referer/
X-Requested-With) already staged on this branch.
Tests: TestHasScopeModelWithSourcePrefix updated; full suite green.
Full-repo audit after the anthropic/openai fix: agentrouter exposes
claude-opus-4-8, so it inherits Anthropic's tool id rule
^[a-zA-Z0-9_-]{1,64}$ and rejects the whole request on a violation, exactly
like justwoker/tabitoken/扇贝. It was the only remaining adapter serving Claude
models without the sanitizer, so a client that had picked up a dirty id (e.g.
"bash:0" from moonshotai/kimi-k3) would still lose every turn here.
Same shape as the other two — inbound tool_calls[].id + tool_call_id, outbound
non-streaming ids and the first streamed fragment — and the test now asserts
all three adapters rewrite an identical input identically, so a client mixing
sources within one session cannot end up with unpaired tool calls.
Audit result: every source exposing a claude/opus/sonnet model (qijiar, toter,
juziai, agentrouter, justwoker, api456, tabitoken) now routes through a
sanitizing adapter.
Anthropic and OpenAI disagree on what the prompt count means:
Anthropic: input_tokens EXCLUDES cached blocks; cache_read_input_tokens and
cache_creation_input_tokens are separate, additive, billed input.
OpenAI: prompt_tokens INCLUDES its cached_tokens subset.
anthropic.lua mapped input_tokens straight onto prompt, so a cache-heavy turn
was doubly wrong: the billed prompt was undercounted by the entire cache
portion, and cached_tokens could exceed prompt_tokens — a cache hit rate above
100% for any client that divides one by the other. cache_creation_input_tokens
was never read at all, so a cache-write turn silently lost those billed tokens.
Worse, the streaming path dropped the cache split entirely: message_delta
carries the FINAL usage and only mapped input/output, so every streamed
response reported no cache information even when the upstream sent it.
All three counts are now summed into prompt, with the read half exposed as
prompt_tokens_details.cached_tokens plus the DeepSeek-legacy hit/miss pair, via
one shared map_usage() used by transform_response, message_start and
message_delta. A reported zero stays distinguishable from "never reported": the
split is emitted whenever either cache field is present, and omitted entirely
when the upstream mentions neither (justwoker reports only input/output plus its
own cost fields, so its output is byte-identical to before). map_usage returns
nil for a countless object, preserving "no usage in this chunk means say
nothing" rather than reporting zeros.
message_start's placeholder count is still emitted: justwoker reports 160 there
and the real 6931 in message_delta, and the gateway's mergeUsage lets the later
non-zero value win.
Anthropic requires tool_use.id / tool_result.tool_use_id to match
^[a-zA-Z0-9_-]{1,64}$ and rejects the WHOLE request otherwise with
REQUEST_BODY_INVALID / "Invalid tool use format". OpenAI has no such rule, so
an OpenAI-compatible model can mint an id like "bash:0"
(xinjianya/moonshotai/kimi-k3 does exactly that).
In a fan-out router that id does not stay local: the client stores it in its
history and replays it to every other source. One such id therefore kills
every Claude slot at once — justwoker, tabitoken and 扇贝 are all
Claude-behind-{OpenAI,Anthropic} — and an AUTO request falls through all four
tiers to whatever tolerant model is left. Observed live: 4 consecutive 503
"all N auto providers failed" with tier 1/2/3 each reporting the same 400.
Both directions are sanitized, in both adapters:
- request: tool_calls[].id and tool_call_id, so poisoned history recovers
- response: non-streaming tool_calls[].id and the first streamed fragment,
so a bad id never enters a client session in the first place
safe_tool_id is pure and deterministic, so a call and its result are rewritten
identically within one request. A rewritten id keeps an 8-hex digest of the
original, without which distinct ids could collapse ("a:b" and "a_b") into a
duplicate/unpaired tool_use. Already-legal ids pass through byte-identical, so
well-behaved traffic is unaffected. openai.lua carries its own copy because
Lua adapters have no shared prelude.
Streamed argument fragments carry no id and must stay id-less, otherwise
index-based accumulation on the client breaks; a test pins that.
Upstreams like justwoker/tabitoken sit behind Cloudflare geo/IP blocks and
only respond through a proxy. A global env proxy is wrong (intranet sources
trae/localzen must stay direct), so add an explicit per-source proxy_url
that overrides http.ProxyFromEnvironment for just that source. Sources
without proxy_url keep the existing env/direct behavior.
Also accepts a User-Agent header per source (config already supported
headers) so CF-fronted resellers can be reached.
Verified e2e: tabitoken and justwoker now return tool_calls through
127.0.0.1:7890 (clash); trae stays direct. Full go test -tags luajit
passes.
Root cause: trae-local-api is deployed to users without the "Fold past
assistant tool_calls into [Called tool: name({...})]" text format that
their own histories already contained. Trae-Local-API-LLM then mimics this
format in subsequent responses. The trae.lua parser only recognized
<tool_call>...</tool_call> or <toolcall>...</toolcall> tags, so
[Called tool: ...] responses were left unparsed and the client received
plain text where a structured tool_calls array should be.
Fix: Add a legacy pattern match at the END of parse_text_tool_calls to
catch the [Called tool: name({args})] shape and emit proper tool_calls.
This is a fallback; models should emit <tool_call> tags per system prompt,
but we tolerate the mimicked form for robustness.
Report: an allowance-metered source (sensenova) had its deepseek model sitting at
pref -10 while the WebUI showed zero failures. All three numbers were accurate,
and they exposed three compounding problems.
1. Quota exhaustion cost the SAME score as a real failure. RecordQuotaExhausted
intentionally avoids failCount (an exhausted allowance is not a fault), so the
UI showed fails=0 / cooling=false — yet it deducted the full prefFailStep (5).
For a metered source, running out of budget is an everyday event, so the score
drifted deep negative with no visible cause. Now quota and 429 events cost
prefQuotaStep (1): the cooldown already keeps the slot out of rotation until
the window resets, the score only needs a mild preference for slots with budget.
2. Recovery was 5:1 asymmetric. A failure cost -5 but a success only +1, so a
slot at -10 needed ten consecutive successes just to reach neutral — which it
could never get, because a low score makes the scheduler not pick it in the
first place (starvation). Success now rewards prefSuccessStep (2): recovery
from -10 needs five successes, while a real failure still outweighs one.
3. No idle decay. A penalised slot kept its negative score forever once it stopped
being selected. Pref() now applies lazy decay: after prefDecayAfter (2 min) of
no outcome, the score drifts one step back toward 0 per interval (never past
0, never touches positive scores). Applied in Pref() and TryProbe(), so a
naturally-recovered idle slot is schedulable again without needing a probe.
The failure penalty itself is unchanged (prefFailStep=5), so genuinely broken
upstreams are still marked as clearly worse than healthy ones.
Tests: quota penalty lighter than failure; 5 quota resets stay well above the
floor with failCount untouched; 429 is quota-class; recovery from -10 needs <=5;
idle decay rehabilitates a written-off slot, stops at 0, and never drags a
positive score; decay repeatedly lifts a slot off the prefMin floor. Updated the
two pre-existing tests that asserted the old -5/129 +1 values.
Four adapters handled tool_calls in transform_stream_chunk but lost them in
transform_response, so any NON-streaming tool-using conversation broke on its
second request: the client received finish_reason:"tool_calls" with no
tool_calls payload, replayed an assistant message whose function
name/arguments were empty, and the upstream rejected the next turn with
400 invalid tool_call function, function/name/arguments cannot be empty
The production audit trail shows 46 such failures on sensenova alone.
- sensenova.lua: forward message.tool_calls, decoding the arguments JSON string
into an object as the unified shape expects.
- gemini.lua: collect functionCall parts from candidates[].content.parts. Also
correct finish_reason, since Gemini reports "STOP" even when it emitted a
function call and clients keyed on it treat that as a finished answer.
- ollama.lua: the field was initialized to an empty table and never filled;
fill it and likewise correct done_reason "stop" -> "tool_calls".
trae is a different failure with the same symptom: trae-local-api's OpenAI
endpoint (/v1/chat/completions, src/server.js:353) never reads the request's
`tools` array — only its Anthropic endpoint does — so the relayed model is never
told the tool schema and instead PRINTS a <tool_call>{...}</tool_call> block into
content, leaving message.tool_calls null and finish_reason "stop". An OpenAI
client sees an ordinary completion and its agent loop ends mid-conversation.
trae.lua now recovers the structured call from that text, strips the block from
user-visible content, and corrects finish_reason. Both tag spellings
(<tool_call>/<toolcall>, the latter is what the same codebase's Anthropic prompt
asks for) and all three argument key names (arguments/params/input) are accepted.
This is a defensive fallback: fixing the upstream shim to honour `tools` remains
the real fix, since the model still guesses parameter names.
Tests: TestNonStreamToolCallsPreserved covers all ten OpenAI-shaped adapters,
TestGeminiNonStreamToolCalls and TestOllamaNonStreamToolCalls cover their native
shapes, TestTraeTextToolCallRecovery covers both tag spellings, prose around the
block, and asserts a plain text answer never gains tool_calls.
Verified end-to-end against mock upstreams reproducing each shape: a full
two-round agent loop (tool call -> tool result -> final answer) now completes for
both the structured and the text-emitted variants.
Adds a fourth accent alongside sakura/ocean/violet. Unlike those it is not just a
different hue: the coloured themes are glass surfaces (translucent cards with
backdrop-filter) floating over an animated gradient-mesh background, so setting
--card:#ffffff there still renders as a tinted grey. Mono therefore also switches
off the translucency and hides the blobs, so #ffffff is actually #ffffff and
#000000 is actually #000000, with greys carrying the hierarchy that hue carries
elsewhere. A side effect worth having: no backdrop-filter and no animated blobs
makes it the cheapest theme to render, which helps on weak GPUs and over remote
desktops.
Both light and dark variable blocks are defined, so the existing light/dark
toggle drives it with no extra wiring: light -> white, dark -> black.
Also fixes a latent theme bug found while checking contrast on black: the active
chart's grid baseline assigned the literal string "var(--line)" to
ctx.strokeStyle. Canvas 2D does not resolve CSS custom properties, so that was an
invalid colour the browser ignored, leaving the previous fillStyle (black) — an
invisible baseline on every dark theme. Colours used on a canvas now go through a
cssVar() helper.
Tests: TestUIThemeMatrix asserts every accent defines BOTH a light and a dark
block plus a picker button (a half-defined theme shows up as unreadable text, not
as an error); TestUIMonoThemeIsFlat pins the opaque surfaces and disabled blobs;
TestUICanvasColorsResolveVars fails if any ctx.strokeStyle/fillStyle is handed a
raw var().
Unrelated packaging fix in the same commit: dist:linux only built deb+AppImage
while build.linux.target listed rpm too, so `make gui-dist` silently skipped the
rpm that release builds are expected to produce. Makefile/README wording updated
to match.
The gateway still built Stats with NewStats(10000), a leftover from when the
ring WAS the source of the dashboard's numbers. With aggregates now computed
from the full audit history, a 10000-entry ring only means 10000 resident Req
structs — the live instance came up at 64 MB instead of ~40 MB, putting back
most of the memory the on-demand paging work removed.
The ring's only jobs are the status page's 5-minute SourceRecent /
SourceAverages windows and the dashboard's first screen, both of which fit in
defaultRingSize (500). Older records are paged from disk.
TestGatewayRingStaysSmall pins it so the constant cannot drift back up.
Two problems reported after the on-demand log work landed.
1. Dashboard totals were wrong. LoadAudit only replayed the last 4 MB of the
audit file, so requests/tokens/per-key rows reflected a window instead of all
time — a regression in reported numbers, not just in presentation.
The aggregates are now built by streaming EVERY audit file (oldest first, so
the hourly quota buckets keep their intended trailing window) and keeping
nothing per record: aggregate maps are keyed by key/model/source, so their
size is bounded by cardinality. Measured on the production host: 29 MB /
221k lines / 37k requests in ~260 ms at startup.
What stays bounded is the RAW-record ring: a fixed-size reqRing keeps only the
newest maxRecs records, so the ~25 MB that used to be spent appending every
record into a slice is still saved. auditReplayBytes is gone, and
replayPartial now means "an audit file could not be read", which is the only
remaining way for the totals to be incomplete.
2. Scrolling to the bottom stopped loading more records. Two independent causes:
* paintRecords rebuilt the entire table on every 5s poll whenever the row
count did not exceed the first screen — the "is a paged view live?" test
compared row counts and matched exactly on the first refresh — wiping loaded
pages and resetting scroll position.
* IntersectionObserver only fires on TRANSITIONS. With a short list, or after a
page whose rows all duplicated the first screen, the sentinel stayed visible
and never fired again.
paintRecords now builds once (recsState.built) and later polls PREPEND only
genuinely new rows; attachRecsObserver adds a scroll-position fallback;
fillRecordsViewport loads until the list actually overflows; and
loadMoreRecords chains (bounded) when a page yields no new rows, since the
first fetch necessarily overlaps the first screen.
TestUIRecordsPagingWiring pins all four mechanisms structurally, since none of
them is reachable from Go. Test names/comments referring to bounded replay are
updated to describe the bounded RING instead, and both READMEs now state that
totals come from the full history while records are paged.
Deleting a gateway key from the admin UI did nothing and reported
"use GET /api/keys": delKey() called api() with an empty options object, so
fetch defaulted to GET and the request landed in the GET branch of
handleKeysAPI. delAdapter() had the identical bug and reported
"adapter code not exposed; edit in UI".
This is the third instance of the same mistake — ab20f1b fixed delSource and
delTemplate, missing these two — so it is now pinned by tests instead of by
review:
* TestUIAPICallsDeclareMethod walks every api() call in the embedded
index.html and fails if one passes an options object without a method
(an AbortSignal-only read is allowed, being a deliberate GET).
* TestUIDeleteHelpersUseDelete / TestUIMutatingHelpersUseWriteMethods pin the
verb of each removal and write helper by name.
* TestKeyDeleteRoundTrip covers create -> DELETE -> gone -> second DELETE is a
clean 404, and TestCannotDeleteOwnKey keeps the lockout guard.
The 404 bodies for GET /api/keys/{key} and GET /api/adapters/{name} now name the
verb to use ("DELETE /api/keys/{key} to remove"), because that message is what a
mis-methoded client actually shows its user; "use GET /api/keys" read as though
the caller had done nothing wrong.
delSource's indentation, broken by ab20f1b, is also straightened out.
Production oscillated between ~46 MB and ~56 MB RSS with the openai pool
cycling 1 -> 10..12 -> 1 states every couple of minutes, while peak_in_use
never went above 2.
Cause: the batch prewarm sized itself purely on the adapter's ceiling. With
max_concurrent summing to 76, growStep is 8, so any two overlapping requests
warmed 8 states — 6 more than anything was waiting for. A minute later the
janitor correctly reclaimed the surplus, the next pair of overlapping requests
warmed 8 again, and the pool churned boot/discard forever. The elasticity was
working; the growth signal was simply wrong.
Prewarm is now bounded by BOTH limits: the ceiling still caps the step, but the
batch never exceeds p.waiting, the number of goroutines actually blocked on the
pool. Overlapping-but-not-queued traffic (the common case) creates exactly the
states it uses; a genuinely queued burst still ramps in one jump.
TestContentionBatchPrewarms is rewritten to queue real waiters instead of
relying on the ceiling to imply demand, and TestNoPrewarmWithoutWaiters pins the
production shape: two overlapping requests against a 76-wide adapter must create
exactly 2 states.
Production after the first deploy showed the openai pool stuck at 9 states with
peak_in_use=1: a startup burst grew it, and then it never shrank again. The
shrink grace counter was reset by every CHECKOUT, so on a gateway that always
has a request in flight the counter never reached shrinkGraceRounds and the
burst's leftover states were pinned indefinitely — the same monotonic-growth
behaviour this series set out to remove, just with an extra step.
Grace is now reset by GROWTH (a miss that had to boot a state), which is the
actual signal that capacity is short. Ordinary sequential traffic no longer
defers reclaim, while two consecutive quiet-ish rounds are still required so a
gap between two bursts does not tear the pool down.
TestCheckoutResetsGrace asserted the old behaviour and is replaced by
TestGrowthResetsGrace (sequential traffic must NOT defer, growth must) plus
TestBurstLeftoverIsReclaimedUnderSteadyTraffic, which reproduces the production
shape: 9 concurrent holds, then one request per janitor round, and the pool must
still fall back to the resident floor.
sensenova: the deployed /etc/llmsproxy/adapters/sensenova.lua carried a fix that
never made it back into the repo — sensenova-6.8-flash-lite reports its chain of
thought in `reasoning` rather than `reasoning_content`, in both the single-shot
response and the stream deltas. Without this the model's output looked empty.
Repo and deployment now match byte for byte.
trae: the adapter was in use on this deployment but untracked, so a fresh
install had no way to serve the trae source.
Front-end half of the on-demand log loading plus the observability for the two
new scheduling mechanisms.
Records table:
* the first screen comes from the dashboard poll, and an IntersectionObserver
sentinel below the last row pulls the next page from /api/stats/records as
the user scrolls.
* rendered rows are capped at RECS_MAX_DOM=1000 (oldest rendered rows are
dropped) so a long scroll cannot grow the DOM without bound, with a "back
to newest" button to reset cheaply.
* releaseRecords() drops the buffer, disconnects the observer and aborts the
in-flight fetch (AbortController) on tab switch, on key-filter change and
on pagehide/beforeunload — leaving the page releases everything at once.
* the poll no longer rebuilds the table once extra pages are loaded, so the
5s refresh cannot throw away scrolled history.
* when the server reports replay_partial, the filter row states that the
aggregates cover the recent audit tail and points at CSV for full history.
Adapters page shows each pool as created/max plus in_use/idle and the live
grow/shrink steps, with the sizing rule in the hover text.
Priority page health tags now distinguish cooling / probe-ready / probing
instead of a flat "cooling", and the tooltip spells out when the window opened,
when the single probe is allowed through and when the slot clears completely.
core.AutoSlotState carries cooldown_from / probe_after / probing / probeable to
feed this.
Startup RSS on this deployment was 56 MB with a 29 MB audit log and ~10 MB
without one: LoadAudit() json-unmarshalled the ENTIRE file into the aggregates
and kept a 10000-entry ring of raw records. Two more paths had the same shape —
AuditRecords() materialized a whole export window into a []Req before sorting
it, and a dashboard poll serialized the full ring so the browser could render
300 rows of it.
The audit file is now the source of truth and memory only holds the live
window:
* LoadAudit replays only the last auditReplayBytes (4 MB) and drops the
truncated first line; the ring default drops 10000 -> 500, which still
covers both of its consumers (the status page's 5-minute SourceRecent /
SourceAverages windows and the first screen of the records table).
replayPartial is exported so the UI can say the totals cover a window
rather than all time. Token-quota accounting is unaffected: it reads the
modelHour buckets, not the ring (pinned by a test).
* AuditPage(cursor, limit, key) pages records straight off disk, reading the
newest file backwards in 64 KB chunks and returning as soon as the page is
full. The cursor is "<file>:<offset>" and walks into rotated .old files;
a cursor whose file rotated away reports rotated=true so the client can
reset instead of silently skipping records. No state is cached between
requests and the file handle is closed before responding, so "release when
the user leaves the page" is guaranteed by never retaining anything.
* StreamAuditRecords(from,to,key,fn) replaces the accumulate-then-sort export
path; the CSV handler writes rows as they are read and flushes every 1000,
and a write error (client gone) aborts the walk. Export memory is O(1)
regardless of the window. AuditRecords is kept as a test-only wrapper.
* Snapshot ships one screen (firstScreenRecords=100) by default; aggregates
are untouched.
* Audit rotation 64 MB x 10 -> 16 MB x 16: same 256 MB total budget, but a
smaller newest file keeps the first reverse page cheap.
New route: GET /api/stats/records?before=&limit=&key= (non-admins are pinned to
their own key by exportKey). /api/status additionally reports adapter_pools for
admins.
Measured with production's 29 MB audit copied to the test instance: startup RSS
19.0 MB (was 56 MB); scrolling 10 pages (1000 records) +0.7 MB; exporting the
full history (36441 rows / 4.4 MB CSV) +0.1 MB with no residual growth.
ConfigureConcurrency() set each adapter pool's target to the sum of
max_concurrent over its sources (108 on this deployment) and `created` only
ever went UP: once a Lua state was booted it was parked forever, so a
long-running gateway's resident state count was a high-water mark of all
traffic it had ever seen, never of what it currently needs.
Pools are now sized from three live inputs:
* the adapter's MAX CONCURRENCY (sum of max_concurrent) is a ceiling, not a
preallocation, and it sets the growth step:
growStep = clamp(ceil(maxW/8), 1, 8). A 64-wide adapter warms 8 states at
once on a spike; an 8-wide one creeps up one at a time.
* the LIVE CONNECTION COUNT (inUse, i.e. checked-out states) sets the shrink
step: shrinkStep = clamp(ceil(excess/(1+inUse)), 1, excess). With no
connections the slack collapses in a single round; a busy adapter gives up
one state per round so the hot path keeps its warm states.
* how many states already exist (created / len(idle)) decides how much room
is left to grow and how much can be reclaimed.
Batch prewarm only fires on genuine contention (a miss while every existing
state is checked out), so a single sequential caller keeps reusing one state
rather than burning a whole grow step on a cold start. A single VM-level
janitor goroutine (not one per adapter) reclaims idle states every 30s, and
shrinkGraceRounds=2 plus idleHeadroom=1 keep a gap between requests from being
mistaken for the end of a load period; any checkout resets the grace counter.
`created` now decrements on reclaim and on shutdown, and release() closes a
state outright when the ceiling was lowered underneath it, so shrinking
max_concurrent in the config gives memory back immediately instead of parking
orphans until restart.
PoolStats() exposes created/idle/in_use/waiting/max/resident/grow_step/
shrink_step/peak_in_use for the status API.
Measured on the test instance (single mock source, max_concurrent=64):
idle created=1; 50 concurrent requests -> created=10 (peak_in_use=5, ceiling
respected); after 95s of silence -> created=1. On production after deploy: 13
adapters, 1 resident Lua state total with ceilings up to 76.
A slot that failed ten times landed in a 30-minute cooldown and was a hard
skip for the whole window: ModelAvailable() said no, the scheduler dropped the
candidate, and the ONLY way back was a success that could never happen. In
practice an upstream that blipped for a minute — or whose quota reset in
seconds — stayed unusable for half an hour.
Cooldown is now a window with a probe gate instead of a wall:
* backoffCap 30min -> 5min. 401/403 no longer implicitly "jump to the cap"
but get an explicit authCooldown (10min); 429 keeps its 30s fixed window.
* ModelState records cooldownFrom alongside cooldownUntil, so the window has
a measurable midpoint, plus a single-token probe permit (probeInFlight).
* TryProbe() hands out that permit only in the SECOND half of the window: a
freshly failed slot stays completely silent, and past the midpoint exactly
one request may go through as a probe.
* ProbeSlots() = clamp(max_concurrent/10, 1, 2), so a source configured for
max_concurrent=10 lets exactly one request probe.
* A probe is a REAL request: success runs RecordSuccess(), which clears the
cooldown and failCount on the spot and returns the slot to full rotation.
Failure reopens the window, pushing the next probe to the NEW midpoint
instead of retrying immediately.
Quota exhaustion is modelled separately (RecordQuotaExhausted): running out of
allowance is not the upstream being broken, so it must not feed the exponential
ladder or inflate failCount. It cools until the allowance window can plausibly
have reset (capped at 30min so a manual top-up is noticed) and is recognised
from the adapter-normalized error text via ReportStatusReason.
Scheduler side: collectCands() splits a tier into normal + probe candidates
with probes strictly LAST, so probe traffic is what a tier falls back to, never
what it prefers, and the round-robin cursor rotates over the normal head only.
releaseProbes() returns every permit on all exits (success, hard failure, ctx
cancel, tier fallthrough); the 90-line inlined busy-poll loop is extracted into
pollBusyTier() to keep those exits auditable. Direct Chat/ChatStream/Image take
the same gate.
Also fixes a real blacklist bug found on the way: a slot whose pref sank to
prefMin was refused by ModelAvailable forever, long after every cooldown had
expired. Such a slot is now probeable, and one success lifts it off the floor.
Verified live against a controllable upstream (tier1 flaky, tier2 healthy):
failure ladder 5s/9s/20s/40s/79s; after healing the upstream, 10 AUTO requests
across the silent half produced ZERO upstream calls; past the midpoint a single
AUTO request produced exactly ONE upstream call, cleared the remaining 39s of
cooldown, and the next three requests were served by the recovered slot.
gofmt -l reported 13 files with misaligned struct tags / stale formatting.
This commit contains ONLY formatting: no behaviour change, no logic touched.
Files that also carry real changes in this series are formatted by their own
commits.
delSource and delTemplate called api() without a method, so fetch
defaulted to GET; the DELETE handlers never ran and the UI silently
left the item in place.
A template stores every source field except name and api_key, so
operators spin up N key-bearing sources from one shared skeleton
instead of duplicating the whole source block N times.
- config: SourceTemplate type + RuntimeConfig.SourceTemplates stored
in runtime.json alongside runtime sources
- store: UpsertTemplate / ListTemplates / RemoveTemplate
- core: Templates / SaveTemplate / RemoveTemplate
- gateway: GET/POST/DELETE /api/source_templates
- webui: source list gains a Templates button opening a manager with
per-template edit/delete; the add-source dialog gains 'from
template' (event-delegated picker) and 'as template' (card modal)
buttons in its header; z-index fixed so the template editor layers
above the manager
Image models previously could not be scheduled through a priority
chain: the chat AUTO chain explicitly skips image-kind slots, and
AUTO image requests fell back to unordered registry discovery.
- config: add auto_image rules (auto_image yaml / image_rules json);
legacy auto rules keep their meaning as the chat chain
- core: buildAutoImageChain mirrors buildAutoChain with inverted kind
filter (image-only); SaveAutoImageRules + AutoImageRules/AutoImageChain
- scheduler: ChainImage walks the chain tier-by-tier with round-robin
and preference ordering, skipping cooling slots
- gateway: handleImage AUTO now runs down AutoImageChain when one is
configured (falls back to legacy discovery otherwise) and records
the actual served model; handleAutoAPI GET returns image_rules and
PUT accepts image_rules independently of rules
- webui: priority page gains a chat/image toggle editing two
independent lane sets; add-slot picker filters by active kind;
persistAuto writes only the active chain's field
The priority page and key-scope pickers skipped kind=image models,
so image sources (e.g. Kwai-Kolors/Kolors) could not be placed in
the AUTO chain or granted per-key. The backend already supports it:
handleImage resolves AUTO through the chain then filters with
imageOnly, while chat requests are protected by chatOnly, so image
slots never receive chat traffic.
Image models now appear in the priority canvas, the add-slot picker
(labelled ' (image)'), and the key scope dialog.
handleImage recorded the raw request model id, so AUTO image
generations showed up as model=AUTO in the request records and
by-model aggregates instead of the image model actually served
(e.g. Kwai-Kolors/Kolors).
UnifiedResponse gains an optional Model field; Provider.Image fills
it with the resolved id (AUTO resolves to the source's best image
model), and handleImage prefers it when writing the audit record.
6 call sites used the fetch default GET while sending a request body,
which browsers reject outright ('Request with GET/HEAD method cannot
have body'). Affected flows: create key, save source, save auto rules,
upload adapter, update key model scope, and /api/chat streaming.
All now send the method their backend handlers require (POST or PUT)
with an explicit Content-Type.
Prerequisite for removing the nginx global-auth layer in front of the
gateway: llmsproxy must defend its own login endpoint.
Design:
- per-IP failure counter: 5 consecutive failures trigger an exponential
lockout (30s base, doubling per extra burst, capped at 30min); 15min of
quiet forgives the counter
- global budget: max 100 failures/minute across all IPs so a distributed
spray cannot outrun per-IP windows
- locked-out and over-budget attempts get the SAME 'invalid gateway api
key' 401 as normal failures — no oracle to probe lockout state, no info
leak on key validity timing
- successful login clears the IP's counter entirely
- clientIP(): prefers X-Real-IP (trusted nginx proxy), falls back to
RemoteAddr host
Tests: lockout engages at threshold with identical replies, valid keys
rejected while locked, other IPs unaffected, success resets counters,
X-Real-IP extraction.
Live testing across the zen pool showed models report
prompt_tokens_details.cached_tokens even when the hit count is 0 (e.g.
nemotron-3-ultra-free returns cached_tokens:0, audio_tokens:0,
cache_write_tokens:0). The previous >0 guard dropped those objects, so a
cache-enabled upstream looked identical to one without cache support.
- types: PromptTokensDetails.CachedTokens always emitted (drop inner
omitempty) so clients see cached_tokens:0 explicitly; dsh reads it as
a 0% hit instead of 'no data'
- adapters (9): forward prompt_tokens_details whenever the upstream
provides it (presence check instead of >0)
- Req: add cache_reported flag set when usage carried cache accounting;
WebUI shows an amber 0% tag for reported-but-missed rows and keeps
the em-dash only for sources that never report cache data
Live testing proved both sensenova and zen DO return cache fields:
- zen laguna-s-2.1-free: usage.prompt_tokens_details.cached_tokens = 32
(real hit), plus cache_write_tokens/audio_tokens
- sensenova glm-5.2: prompt_tokens_details.cached_tokens present (0 on
short prompts)
The previous round only patched deepseek/openai/anthropic/gemini.lua;
sensenova/opencode (localzen!) and the other adapters still dropped them.
- sensenova/opencode/groq/mistral/github/kimicode: stream + response
cache passthrough (same pattern as openai.lua)
- agentrouter: response passthrough + NEW stream usage forwarding (it
previously dropped the terminal usage-only chunk entirely)
- ollama skipped intentionally: its native API has no cache fields
Verified end-to-end through the gateway: localzen/laguna-s-2.1-free now
returns prompt_tokens_details.cached_tokens=32 to clients, and the request
record carries cache_hit_tokens (both chat and stream paths).
- Req: add CacheHit and CacheMiss fields (carrying upstream cache
accounting from either prompt_tokens_details.cached_tokens or legacy
prompt_cache_hit_tokens)
- recordChatUsage (non-streaming): copy cache fields from resp.TokenUsage
- pumpStream (streaming): write lastUsage cache fields back onto rec at
stream end, so streaming requests carry cache data too
- CSV export: add first_byte_ms, cache_hit_tokens, cache_miss_tokens
columns alongside the existing latency/prompt/completion
- WebUI request-records table: add a Cache column showing hit% per row
(green/amber tag with tooltip hit/miss breakdown; em-dash when the
upstream reported no cache data)
- Req: add FirstByteMs field (ms to first byte, tracked for streaming)
- Stat: add FirstByteSum for aggregation
- SourceAverages(): new method computing per-source avg TTFB and tokens/s
from the in-memory ring (300s window)
- SourceStatus: add AvgFirstByteMs and AvgTokPerS fields
- pumpStream: record FirstByteMs after first SSE chunk sent to client
- singleChat/singleChatAuto: set FirstByteMs = LatMs (non-streaming)
- handleStatusAPI: populate the new SourceStatus fields from SourceAverages()
- WebUI source table: two new columns showing TTFB (s) and Tokens/s
When only the legacy DeepSeek fields (prompt_cache_hit_tokens) are set but
the OpenAI-standard prompt_tokens_details is nil, MarshalJSON now auto-
generates the nested object. This ensures dsh (which reads the standard
format first) sees cache hit data regardless of which adapter format the
upstream uses.
dsh displays cache-hit %, but llmsproxy dropped every upstream's cache
fields — deepseek prompt_cache_hit_tokens, OpenAI prompt_tokens_details.
cached_tokens, anthropic cache_read_input_tokens, gemini cachedContentTokenCount.
Changes:
- TokenUsage: add PromptTokensDetails (with CachedTokens) + PromptCacheHit/Miss
- MarshalJSON: emit prompt_tokens_details.cached_tokens (OpenAI v2 standard)
and prompt_cache_hit/miss_tokens (DeepSeek legacy) — dsh reads the former
first, falls back to the latter
- mergeUsage: preserve cache fields across stream chunks
- standardSSEChunk: parse the upstream raw prompt_tokens_details too
- deepseek.lua: forward prompt_cache_hit/miss_tokens + create
prompt_tokens_details from them
- openai.lua: forward prompt_tokens_details.cached_tokens and legacy
prompt_cache_hit/miss_tokens; normalize legacy hits into the standard
object so dsh sees them regardless of upstream format
- anthropic.lua: map cache_read_input_tokens → prompt_tokens_details
- gemini.lua: map cachedContentTokenCount → prompt_tokens_details
WebUI (internal/gateway/ui):
- BUG: the export modal's custom-range button called
downloadStatsCsvFromForm() which was never defined — clicking it threw a
ReferenceError and nothing downloaded. Implement it: reads #exp-from /
#exp-to date inputs and forwards to downloadStatsCsv.
- BUG-adjacent: clearChat() existed but was reachable from no control —
add a Clear button to the chat composer so conversation reset is actually
possible (+ cClear i18n zh/en).
- remove byte-identical duplicate html[data-theme=dark] CSS block (15 lines)
- remove 8 dead CSS rules (.keys-grid .m-model-row .scope-add/.scope-box/
.scope-chips .scr-blocks .tag-warn .twrap) and the never-consumed
--accent custom property
- remove 3 dead JS functions (activeTab/findSlots/scopeUncomb; lastTab decl kept)
- remove 24 dead i18n keys x zh/en (~55 lines) — legacy of the replaced
key-scope editor, matching the removed .scope-* styles
GUI (cmd/gui/main.js):
- BUG: stopCore() set app.isQuitting=true and nothing reset it — after using
tray 'stop core', closing the window quit the whole app instead of hiding
to tray, and core crash auto-restart stayed disabled. isQuitting now only
flips in restartCore (scoped) and before-quit.
Verified: go vet/test green; node --check on all three GUI js files and the
WebUI inline script.
- extract csvHeaders() (Content-Type + Content-Disposition) shared by both
export branches
- extract exportKey(): the identical admin/user key-filter logic existed
twice (JSON path + keys-csv); now all three call sites share one function
- inline the nine single-use intermediate variables in the keys-csv loop
singleChat / singleChatAuto / streamChat / streamChatAuto shared ~260 near-
identical lines (diff after stripping comments was empty). Extract three
shared bodies and shrink all four entry points to thin dispatchers:
- failChat: error → record + writeError, shared ChainErr extraction
(errors.As returns false for direct-path errors, so the two paths stay
equivalent without a branch)
- recordChatUsage: exact upstream numbers win, byte estimates fill gaps
- writeChatCompletion: unified ChatCompletion rendering (model name is the
only direct/auto difference, passed in)
- pumpStream: the full SSE pump (preamble, delta loop, terminal finish +
usage chunk, [DONE]) — shared by both streaming entries
Behavior change (pinned by TestDirectStreamFailoverAuditSource): direct
streams now pin rec.Source to the source that actually served the stream
after a failover, instead of discarding it (_, usedModel). The audit row
previously recorded the first candidate, which was wrong on failover.
chat.go: 982 → 896 lines (−86).
provider.oneLineStr and gateway.oneLine had byte-identical bodies (flatten
whitespace + cap length). Move the single implementation into the types
package, which both layers already depend on, and delete both locals.
- provider.truncate: zero callers (oneLineStr is the used superset)
- gateway.normalizeModel: zero callers
- config.resolvedAPIKey: zero callers (core.resolveSourceKey is the live equivalent)
- Stats.Records: zero callers (CSV export uses AuditRecords)
- store.containsString: zero callers
- Config.MaxConcurrent: global inflight-cap field never read; per-source
MaxConcurrent is what actually drives semaphores. Legacy configs carrying a
top-level max_concurrent key still load (yaml.v3 ignores unknown fields —
verified by test).
- gui renderer esc(): zero callers; renderer uses textContent, and the embedded
WebUI has its own esc()
statusRecorder (the access-log wrapper) did not implement http.Flusher, so
w.(http.Flusher) inside streamChat/streamChatAuto returned nil and every SSE
chunk stayed buffered until the response ended. Add Flush() that delegates to
the underlying writer when it supports flushing.
Regression test: TestStatusRecorderFlusher pins the interface assertion and
the forwarding path.
