Deleting a gateway key from the admin UI did nothing and reported
"use GET /api/keys": delKey() called api() with an empty options object, so
fetch defaulted to GET and the request landed in the GET branch of
handleKeysAPI. delAdapter() had the identical bug and reported
"adapter code not exposed; edit in UI".
This is the third instance of the same mistake — ab20f1b fixed delSource and
delTemplate, missing these two — so it is now pinned by tests instead of by
review:
* TestUIAPICallsDeclareMethod walks every api() call in the embedded
index.html and fails if one passes an options object without a method
(an AbortSignal-only read is allowed, being a deliberate GET).
* TestUIDeleteHelpersUseDelete / TestUIMutatingHelpersUseWriteMethods pin the
verb of each removal and write helper by name.
* TestKeyDeleteRoundTrip covers create -> DELETE -> gone -> second DELETE is a
clean 404, and TestCannotDeleteOwnKey keeps the lockout guard.
The 404 bodies for GET /api/keys/{key} and GET /api/adapters/{name} now name the
verb to use ("DELETE /api/keys/{key} to remove"), because that message is what a
mis-methoded client actually shows its user; "use GET /api/keys" read as though
the caller had done nothing wrong.
delSource's indentation, broken by ab20f1b, is also straightened out.
Production oscillated between ~46 MB and ~56 MB RSS with the openai pool
cycling 1 -> 10..12 -> 1 states every couple of minutes, while peak_in_use
never went above 2.
Cause: the batch prewarm sized itself purely on the adapter's ceiling. With
max_concurrent summing to 76, growStep is 8, so any two overlapping requests
warmed 8 states — 6 more than anything was waiting for. A minute later the
janitor correctly reclaimed the surplus, the next pair of overlapping requests
warmed 8 again, and the pool churned boot/discard forever. The elasticity was
working; the growth signal was simply wrong.
Prewarm is now bounded by BOTH limits: the ceiling still caps the step, but the
batch never exceeds p.waiting, the number of goroutines actually blocked on the
pool. Overlapping-but-not-queued traffic (the common case) creates exactly the
states it uses; a genuinely queued burst still ramps in one jump.
TestContentionBatchPrewarms is rewritten to queue real waiters instead of
relying on the ceiling to imply demand, and TestNoPrewarmWithoutWaiters pins the
production shape: two overlapping requests against a 76-wide adapter must create
exactly 2 states.
Production after the first deploy showed the openai pool stuck at 9 states with
peak_in_use=1: a startup burst grew it, and then it never shrank again. The
shrink grace counter was reset by every CHECKOUT, so on a gateway that always
has a request in flight the counter never reached shrinkGraceRounds and the
burst's leftover states were pinned indefinitely — the same monotonic-growth
behaviour this series set out to remove, just with an extra step.
Grace is now reset by GROWTH (a miss that had to boot a state), which is the
actual signal that capacity is short. Ordinary sequential traffic no longer
defers reclaim, while two consecutive quiet-ish rounds are still required so a
gap between two bursts does not tear the pool down.
TestCheckoutResetsGrace asserted the old behaviour and is replaced by
TestGrowthResetsGrace (sequential traffic must NOT defer, growth must) plus
TestBurstLeftoverIsReclaimedUnderSteadyTraffic, which reproduces the production
shape: 9 concurrent holds, then one request per janitor round, and the pool must
still fall back to the resident floor.
README: new "冷却与半冷却探测（自愈调度）" section with the per-class cooldown
table (5xx exponential to 5min / 401-403 10min / 429 30s fixed / quota aligned
to its window), the probe-slot formula and the ordering rule that probes are
tried last. The LuaJIT section now says the pool is an elastic ceiling rather
than a preallocation and documents both step formulas. Headline figures replaced
with measured ones: idle ~10 MB, ~19 MB starting with a 29 MB audit log, and a
new bullet for on-demand log loading.
package.json also loses a `\u2014` escape and the broken indentation that an
earlier edit left in the electron-builder block.
.gitignore: add .omo/ (an AI-assisted-coding local state dir that was being
tracked — .omo/run-continuation/*.json is session scratch, not source) and
/plan.md (local working notes).
make clean: also remove cmd/build/ and cmd/gui/dist/. Those hold the packaged
installers (1.8 GB and 386 MB on this machine) and were never cleaned, so
`make clean` left almost all of the build output behind.
sensenova: the deployed /etc/llmsproxy/adapters/sensenova.lua carried a fix that
never made it back into the repo — sensenova-6.8-flash-lite reports its chain of
thought in `reasoning` rather than `reasoning_content`, in both the single-shot
response and the stream deltas. Without this the model's output looked empty.
Repo and deployment now match byte for byte.
trae: the adapter was in use on this deployment but untracked, so a fresh
install had no way to serve the trae source.
Front-end half of the on-demand log loading plus the observability for the two
new scheduling mechanisms.
Records table:
* the first screen comes from the dashboard poll, and an IntersectionObserver
sentinel below the last row pulls the next page from /api/stats/records as
the user scrolls.
* rendered rows are capped at RECS_MAX_DOM=1000 (oldest rendered rows are
dropped) so a long scroll cannot grow the DOM without bound, with a "back
to newest" button to reset cheaply.
* releaseRecords() drops the buffer, disconnects the observer and aborts the
in-flight fetch (AbortController) on tab switch, on key-filter change and
on pagehide/beforeunload — leaving the page releases everything at once.
* the poll no longer rebuilds the table once extra pages are loaded, so the
5s refresh cannot throw away scrolled history.
* when the server reports replay_partial, the filter row states that the
aggregates cover the recent audit tail and points at CSV for full history.
Adapters page shows each pool as created/max plus in_use/idle and the live
grow/shrink steps, with the sizing rule in the hover text.
Priority page health tags now distinguish cooling / probe-ready / probing
instead of a flat "cooling", and the tooltip spells out when the window opened,
when the single probe is allowed through and when the slot clears completely.
core.AutoSlotState carries cooldown_from / probe_after / probing / probeable to
feed this.
Startup RSS on this deployment was 56 MB with a 29 MB audit log and ~10 MB
without one: LoadAudit() json-unmarshalled the ENTIRE file into the aggregates
and kept a 10000-entry ring of raw records. Two more paths had the same shape —
AuditRecords() materialized a whole export window into a []Req before sorting
it, and a dashboard poll serialized the full ring so the browser could render
300 rows of it.
The audit file is now the source of truth and memory only holds the live
window:
* LoadAudit replays only the last auditReplayBytes (4 MB) and drops the
truncated first line; the ring default drops 10000 -> 500, which still
covers both of its consumers (the status page's 5-minute SourceRecent /
SourceAverages windows and the first screen of the records table).
replayPartial is exported so the UI can say the totals cover a window
rather than all time. Token-quota accounting is unaffected: it reads the
modelHour buckets, not the ring (pinned by a test).
* AuditPage(cursor, limit, key) pages records straight off disk, reading the
newest file backwards in 64 KB chunks and returning as soon as the page is
full. The cursor is "<file>:<offset>" and walks into rotated .old files;
a cursor whose file rotated away reports rotated=true so the client can
reset instead of silently skipping records. No state is cached between
requests and the file handle is closed before responding, so "release when
the user leaves the page" is guaranteed by never retaining anything.
* StreamAuditRecords(from,to,key,fn) replaces the accumulate-then-sort export
path; the CSV handler writes rows as they are read and flushes every 1000,
and a write error (client gone) aborts the walk. Export memory is O(1)
regardless of the window. AuditRecords is kept as a test-only wrapper.
* Snapshot ships one screen (firstScreenRecords=100) by default; aggregates
are untouched.
* Audit rotation 64 MB x 10 -> 16 MB x 16: same 256 MB total budget, but a
smaller newest file keeps the first reverse page cheap.
New route: GET /api/stats/records?before=&limit=&key= (non-admins are pinned to
their own key by exportKey). /api/status additionally reports adapter_pools for
admins.
Measured with production's 29 MB audit copied to the test instance: startup RSS
19.0 MB (was 56 MB); scrolling 10 pages (1000 records) +0.7 MB; exporting the
full history (36441 rows / 4.4 MB CSV) +0.1 MB with no residual growth.
ConfigureConcurrency() set each adapter pool's target to the sum of
max_concurrent over its sources (108 on this deployment) and `created` only
ever went UP: once a Lua state was booted it was parked forever, so a
long-running gateway's resident state count was a high-water mark of all
traffic it had ever seen, never of what it currently needs.
Pools are now sized from three live inputs:
* the adapter's MAX CONCURRENCY (sum of max_concurrent) is a ceiling, not a
preallocation, and it sets the growth step:
growStep = clamp(ceil(maxW/8), 1, 8). A 64-wide adapter warms 8 states at
once on a spike; an 8-wide one creeps up one at a time.
* the LIVE CONNECTION COUNT (inUse, i.e. checked-out states) sets the shrink
step: shrinkStep = clamp(ceil(excess/(1+inUse)), 1, excess). With no
connections the slack collapses in a single round; a busy adapter gives up
one state per round so the hot path keeps its warm states.
* how many states already exist (created / len(idle)) decides how much room
is left to grow and how much can be reclaimed.
Batch prewarm only fires on genuine contention (a miss while every existing
state is checked out), so a single sequential caller keeps reusing one state
rather than burning a whole grow step on a cold start. A single VM-level
janitor goroutine (not one per adapter) reclaims idle states every 30s, and
shrinkGraceRounds=2 plus idleHeadroom=1 keep a gap between requests from being
mistaken for the end of a load period; any checkout resets the grace counter.
`created` now decrements on reclaim and on shutdown, and release() closes a
state outright when the ceiling was lowered underneath it, so shrinking
max_concurrent in the config gives memory back immediately instead of parking
orphans until restart.
PoolStats() exposes created/idle/in_use/waiting/max/resident/grow_step/
shrink_step/peak_in_use for the status API.
Measured on the test instance (single mock source, max_concurrent=64):
idle created=1; 50 concurrent requests -> created=10 (peak_in_use=5, ceiling
respected); after 95s of silence -> created=1. On production after deploy: 13
adapters, 1 resident Lua state total with ceilings up to 76.
A slot that failed ten times landed in a 30-minute cooldown and was a hard
skip for the whole window: ModelAvailable() said no, the scheduler dropped the
candidate, and the ONLY way back was a success that could never happen. In
practice an upstream that blipped for a minute — or whose quota reset in
seconds — stayed unusable for half an hour.
Cooldown is now a window with a probe gate instead of a wall:
* backoffCap 30min -> 5min. 401/403 no longer implicitly "jump to the cap"
but get an explicit authCooldown (10min); 429 keeps its 30s fixed window.
* ModelState records cooldownFrom alongside cooldownUntil, so the window has
a measurable midpoint, plus a single-token probe permit (probeInFlight).
* TryProbe() hands out that permit only in the SECOND half of the window: a
freshly failed slot stays completely silent, and past the midpoint exactly
one request may go through as a probe.
* ProbeSlots() = clamp(max_concurrent/10, 1, 2), so a source configured for
max_concurrent=10 lets exactly one request probe.
* A probe is a REAL request: success runs RecordSuccess(), which clears the
cooldown and failCount on the spot and returns the slot to full rotation.
Failure reopens the window, pushing the next probe to the NEW midpoint
instead of retrying immediately.
Quota exhaustion is modelled separately (RecordQuotaExhausted): running out of
allowance is not the upstream being broken, so it must not feed the exponential
ladder or inflate failCount. It cools until the allowance window can plausibly
have reset (capped at 30min so a manual top-up is noticed) and is recognised
from the adapter-normalized error text via ReportStatusReason.
Scheduler side: collectCands() splits a tier into normal + probe candidates
with probes strictly LAST, so probe traffic is what a tier falls back to, never
what it prefers, and the round-robin cursor rotates over the normal head only.
releaseProbes() returns every permit on all exits (success, hard failure, ctx
cancel, tier fallthrough); the 90-line inlined busy-poll loop is extracted into
pollBusyTier() to keep those exits auditable. Direct Chat/ChatStream/Image take
the same gate.
Also fixes a real blacklist bug found on the way: a slot whose pref sank to
prefMin was refused by ModelAvailable forever, long after every cooldown had
expired. Such a slot is now probeable, and one success lifts it off the floor.
Verified live against a controllable upstream (tier1 flaky, tier2 healthy):
failure ladder 5s/9s/20s/40s/79s; after healing the upstream, 10 AUTO requests
across the silent half produced ZERO upstream calls; past the midpoint a single
AUTO request produced exactly ONE upstream call, cleared the remaining 39s of
cooldown, and the next three requests were served by the recovered slot.
gofmt -l reported 13 files with misaligned struct tags / stale formatting.
This commit contains ONLY formatting: no behaviour change, no logic touched.
Files that also carry real changes in this series are formatted by their own
commits.
- AUTO chain section: chat and image generation are now two independent
chains (auto / auto_image) toggled in the Priority page; legacy image
discovery fallback documented
- new Source templates section: multi-key balancing via reusable templates
(Templates manager, From template / As template in the add-source dialog)
- Desktop GUI section: two installer kinds — Headless (server, plain binary)
and Desktop (Electron GUI); release artifact naming ModelRouter-Headless-* /
ModelRouter-Desktop-*; stale 1.0.0 version pins replaced with <ver>
delSource and delTemplate called api() without a method, so fetch
defaulted to GET; the DELETE handlers never ran and the UI silently
left the item in place.
A template stores every source field except name and api_key, so
operators spin up N key-bearing sources from one shared skeleton
instead of duplicating the whole source block N times.
- config: SourceTemplate type + RuntimeConfig.SourceTemplates stored
in runtime.json alongside runtime sources
- store: UpsertTemplate / ListTemplates / RemoveTemplate
- core: Templates / SaveTemplate / RemoveTemplate
- gateway: GET/POST/DELETE /api/source_templates
- webui: source list gains a Templates button opening a manager with
per-template edit/delete; the add-source dialog gains 'from
template' (event-delegated picker) and 'as template' (card modal)
buttons in its header; z-index fixed so the template editor layers
above the manager
Image models previously could not be scheduled through a priority
chain: the chat AUTO chain explicitly skips image-kind slots, and
AUTO image requests fell back to unordered registry discovery.
- config: add auto_image rules (auto_image yaml / image_rules json);
legacy auto rules keep their meaning as the chat chain
- core: buildAutoImageChain mirrors buildAutoChain with inverted kind
filter (image-only); SaveAutoImageRules + AutoImageRules/AutoImageChain
- scheduler: ChainImage walks the chain tier-by-tier with round-robin
and preference ordering, skipping cooling slots
- gateway: handleImage AUTO now runs down AutoImageChain when one is
configured (falls back to legacy discovery otherwise) and records
the actual served model; handleAutoAPI GET returns image_rules and
PUT accepts image_rules independently of rules
- webui: priority page gains a chat/image toggle editing two
independent lane sets; add-slot picker filters by active kind;
persistAuto writes only the active chain's field
The priority page and key-scope pickers skipped kind=image models,
so image sources (e.g. Kwai-Kolors/Kolors) could not be placed in
the AUTO chain or granted per-key. The backend already supports it:
handleImage resolves AUTO through the chain then filters with
imageOnly, while chat requests are protected by chatOnly, so image
slots never receive chat traffic.
Image models now appear in the priority canvas, the add-slot picker
(labelled ' (image)'), and the key scope dialog.
handleImage recorded the raw request model id, so AUTO image
generations showed up as model=AUTO in the request records and
by-model aggregates instead of the image model actually served
(e.g. Kwai-Kolors/Kolors).
UnifiedResponse gains an optional Model field; Provider.Image fills
it with the resolved id (AUTO resolves to the source's best image
model), and handleImage prefers it when writing the audit record.
6 call sites used the fetch default GET while sending a request body,
which browsers reject outright ('Request with GET/HEAD method cannot
have body'). Affected flows: create key, save source, save auto rules,
upload adapter, update key model scope, and /api/chat streaming.
All now send the method their backend handlers require (POST or PUT)
with an explicit Content-Type.
Prerequisite for removing the nginx global-auth layer in front of the
gateway: llmsproxy must defend its own login endpoint.
Design:
- per-IP failure counter: 5 consecutive failures trigger an exponential
lockout (30s base, doubling per extra burst, capped at 30min); 15min of
quiet forgives the counter
- global budget: max 100 failures/minute across all IPs so a distributed
spray cannot outrun per-IP windows
- locked-out and over-budget attempts get the SAME 'invalid gateway api
key' 401 as normal failures — no oracle to probe lockout state, no info
leak on key validity timing
- successful login clears the IP's counter entirely
- clientIP(): prefers X-Real-IP (trusted nginx proxy), falls back to
RemoteAddr host
Tests: lockout engages at threshold with identical replies, valid keys
rejected while locked, other IPs unaffected, success resets counters,
X-Real-IP extraction.
Live testing across the zen pool showed models report
prompt_tokens_details.cached_tokens even when the hit count is 0 (e.g.
nemotron-3-ultra-free returns cached_tokens:0, audio_tokens:0,
cache_write_tokens:0). The previous >0 guard dropped those objects, so a
cache-enabled upstream looked identical to one without cache support.
- types: PromptTokensDetails.CachedTokens always emitted (drop inner
omitempty) so clients see cached_tokens:0 explicitly; dsh reads it as
a 0% hit instead of 'no data'
- adapters (9): forward prompt_tokens_details whenever the upstream
provides it (presence check instead of >0)
- Req: add cache_reported flag set when usage carried cache accounting;
WebUI shows an amber 0% tag for reported-but-missed rows and keeps
the em-dash only for sources that never report cache data
Live testing proved both sensenova and zen DO return cache fields:
- zen laguna-s-2.1-free: usage.prompt_tokens_details.cached_tokens = 32
(real hit), plus cache_write_tokens/audio_tokens
- sensenova glm-5.2: prompt_tokens_details.cached_tokens present (0 on
short prompts)
The previous round only patched deepseek/openai/anthropic/gemini.lua;
sensenova/opencode (localzen!) and the other adapters still dropped them.
- sensenova/opencode/groq/mistral/github/kimicode: stream + response
cache passthrough (same pattern as openai.lua)
- agentrouter: response passthrough + NEW stream usage forwarding (it
previously dropped the terminal usage-only chunk entirely)
- ollama skipped intentionally: its native API has no cache fields
Verified end-to-end through the gateway: localzen/laguna-s-2.1-free now
returns prompt_tokens_details.cached_tokens=32 to clients, and the request
record carries cache_hit_tokens (both chat and stream paths).
- Req: add CacheHit and CacheMiss fields (carrying upstream cache
accounting from either prompt_tokens_details.cached_tokens or legacy
prompt_cache_hit_tokens)
- recordChatUsage (non-streaming): copy cache fields from resp.TokenUsage
- pumpStream (streaming): write lastUsage cache fields back onto rec at
stream end, so streaming requests carry cache data too
- CSV export: add first_byte_ms, cache_hit_tokens, cache_miss_tokens
columns alongside the existing latency/prompt/completion
- WebUI request-records table: add a Cache column showing hit% per row
(green/amber tag with tooltip hit/miss breakdown; em-dash when the
upstream reported no cache data)
- Req: add FirstByteMs field (ms to first byte, tracked for streaming)
- Stat: add FirstByteSum for aggregation
- SourceAverages(): new method computing per-source avg TTFB and tokens/s
from the in-memory ring (300s window)
- SourceStatus: add AvgFirstByteMs and AvgTokPerS fields
- pumpStream: record FirstByteMs after first SSE chunk sent to client
- singleChat/singleChatAuto: set FirstByteMs = LatMs (non-streaming)
- handleStatusAPI: populate the new SourceStatus fields from SourceAverages()
- WebUI source table: two new columns showing TTFB (s) and Tokens/s
When only the legacy DeepSeek fields (prompt_cache_hit_tokens) are set but
the OpenAI-standard prompt_tokens_details is nil, MarshalJSON now auto-
generates the nested object. This ensures dsh (which reads the standard
format first) sees cache hit data regardless of which adapter format the
upstream uses.
dsh displays cache-hit %, but llmsproxy dropped every upstream's cache
fields — deepseek prompt_cache_hit_tokens, OpenAI prompt_tokens_details.
cached_tokens, anthropic cache_read_input_tokens, gemini cachedContentTokenCount.
Changes:
- TokenUsage: add PromptTokensDetails (with CachedTokens) + PromptCacheHit/Miss
- MarshalJSON: emit prompt_tokens_details.cached_tokens (OpenAI v2 standard)
and prompt_cache_hit/miss_tokens (DeepSeek legacy) — dsh reads the former
first, falls back to the latter
- mergeUsage: preserve cache fields across stream chunks
- standardSSEChunk: parse the upstream raw prompt_tokens_details too
- deepseek.lua: forward prompt_cache_hit/miss_tokens + create
prompt_tokens_details from them
- openai.lua: forward prompt_tokens_details.cached_tokens and legacy
prompt_cache_hit/miss_tokens; normalize legacy hits into the standard
object so dsh sees them regardless of upstream format
- anthropic.lua: map cache_read_input_tokens → prompt_tokens_details
- gemini.lua: map cachedContentTokenCount → prompt_tokens_details
WebUI (internal/gateway/ui):
- BUG: the export modal's custom-range button called
downloadStatsCsvFromForm() which was never defined — clicking it threw a
ReferenceError and nothing downloaded. Implement it: reads #exp-from /
#exp-to date inputs and forwards to downloadStatsCsv.
- BUG-adjacent: clearChat() existed but was reachable from no control —
add a Clear button to the chat composer so conversation reset is actually
possible (+ cClear i18n zh/en).
- remove byte-identical duplicate html[data-theme=dark] CSS block (15 lines)
- remove 8 dead CSS rules (.keys-grid .m-model-row .scope-add/.scope-box/
.scope-chips .scr-blocks .tag-warn .twrap) and the never-consumed
--accent custom property
- remove 3 dead JS functions (activeTab/findSlots/scopeUncomb; lastTab decl kept)
- remove 24 dead i18n keys x zh/en (~55 lines) — legacy of the replaced
key-scope editor, matching the removed .scope-* styles
GUI (cmd/gui/main.js):
- BUG: stopCore() set app.isQuitting=true and nothing reset it — after using
tray 'stop core', closing the window quit the whole app instead of hiding
to tray, and core crash auto-restart stayed disabled. isQuitting now only
flips in restartCore (scoped) and before-quit.
Verified: go vet/test green; node --check on all three GUI js files and the
WebUI inline script.
- extract csvHeaders() (Content-Type + Content-Disposition) shared by both
export branches
- extract exportKey(): the identical admin/user key-filter logic existed
twice (JSON path + keys-csv); now all three call sites share one function
- inline the nine single-use intermediate variables in the keys-csv loop
singleChat / singleChatAuto / streamChat / streamChatAuto shared ~260 near-
identical lines (diff after stripping comments was empty). Extract three
shared bodies and shrink all four entry points to thin dispatchers:
- failChat: error → record + writeError, shared ChainErr extraction
(errors.As returns false for direct-path errors, so the two paths stay
equivalent without a branch)
- recordChatUsage: exact upstream numbers win, byte estimates fill gaps
- writeChatCompletion: unified ChatCompletion rendering (model name is the
only direct/auto difference, passed in)
- pumpStream: the full SSE pump (preamble, delta loop, terminal finish +
usage chunk, [DONE]) — shared by both streaming entries
Behavior change (pinned by TestDirectStreamFailoverAuditSource): direct
streams now pin rec.Source to the source that actually served the stream
after a failover, instead of discarding it (_, usedModel). The audit row
previously recorded the first candidate, which was wrong on failover.
chat.go: 982 → 896 lines (−86).
provider.oneLineStr and gateway.oneLine had byte-identical bodies (flatten
whitespace + cap length). Move the single implementation into the types
package, which both layers already depend on, and delete both locals.
- provider.truncate: zero callers (oneLineStr is the used superset)
- gateway.normalizeModel: zero callers
- config.resolvedAPIKey: zero callers (core.resolveSourceKey is the live equivalent)
- Stats.Records: zero callers (CSV export uses AuditRecords)
- store.containsString: zero callers
- Config.MaxConcurrent: global inflight-cap field never read; per-source
MaxConcurrent is what actually drives semaphores. Legacy configs carrying a
top-level max_concurrent key still load (yaml.v3 ignores unknown fields —
verified by test).
- gui renderer esc(): zero callers; renderer uses textContent, and the embedded
WebUI has its own esc()
statusRecorder (the access-log wrapper) did not implement http.Flusher, so
w.(http.Flusher) inside streamChat/streamChatAuto returned nil and every SSE
chunk stayed buffered until the response ended. Add Flush() that delegates to
the underlying writer when it supports flushing.
Regression test: TestStatusRecorderFlusher pins the interface assertion and
the forwarding path.
Every upstream formats errors differently, which is adapter territory:
the protocol gains an optional transform_error(status, body) hook and all
built-in adapters implement their own envelope parsing (zen free-pool
labels, anthropic/gemini/ollama/mistral shapes, sensenova quota notes,
agentrouter WAF pages). The core keeps a single uniform fallback: when no
hook yields a reason clients get "api error <status>: unknown error" and
the raw body goes to server logs only.
api error strings embedded raw upstream response bodies, so JSON quota
payloads and WAF HTML pages leaked through to clients (and through the
per-tier chain summary). shortAPIError extracts the envelope reason
(error.message / message / msg), collapses HTML blocklist pages to a
marker, and caps everything at one line.
WindowTokens buckets are whole unix hours, so a fixture built from raw
offsets like "1 minute ago" landed in the previous hour bucket whenever
the suite ran in the first minute of an hour — the 1h-window assertion
then saw 0 tokens and failed. Anchor rows to hour boundaries instead
(H-2 / H-1 / current bucket) and assert the exact current-bucket value;
the suite is now deterministic for any run time.
Clients saw the full per-tier chain error including upstream HTML WAF
pages and JSON quota payloads. The response now carries one capped
one-line reason per tier (quota/cooling skips preserved); full detail
remains in rec.Err / stats API and is logged server-side.
Some upstreams (zen free pool) answer HTTP 200 with a single-chunk stream
whose only payload is finish_reason:"network_error" and an empty delta.
ChatStream used to hand that channel up as a success, so clients received
a laundered empty reply (agents ended their turn mid-loop) and the broken
slot kept its scheduling preference. ChatStream now holds back the first
chunk: an error-only done chunk fails the candidate before any byte
reaches the gateway, letting the scheduler fall through to the next
source. Standard finish reasons (including instant-empty "stop") are never
classified as errors.
The gateway hardcoded "stop" on every terminating stream chunk, so
tool-call rounds reported finish_reason=stop and length caps were
invisible to clients. UnifiedChunk now carries finish_reason; adapters
emit it (with empty-string finish reasons like sensenova treated as
non-terminal), standardSSEChunk passes it through for un-adapted
upstreams, [DONE] no longer emits a duplicate reason-less done chunk,
and both streaming paths emit the real reason with "stop" as fallback.
Also vendor sensenova/agentrouter adapters into the repo: they were
WebUI-only uploads and a deploy sync silently removed them while live
AUTO-chain slots still referenced them.
zen fingerprints clients via UA + x-opencode-client/session/request/project
headers; requests without them are routed as anonymous and fail with a
single-chunk network_error stream (tools+stream) or 503 Endpoint is
unavailable (non-stream), which the adapter laundered into empty-but-valid
replies. Derive per-request session/request ids from build_headers meta
(sandbox has no os/math), bump UA to the real client format, map zen
reasoning field to reasoning_content, and set stream_options.include_usage.
The prior usage-passthrough fix only covered openai/opencode; the same
empty-choices+usage drop bug remained in the 5 sibling OpenAI-compatible
adapters, and non-OpenAI providers (anthropic/gemini/ollama) never surfaced
streaming usage at all.
- deepseek/github/groq/kimicode/mistral: preserve usage on empty-choices
chunks and attach it to normal chunks (same pattern as openai.lua)
- anthropic: emit usage from message_start (prompt) and message_delta
(completion); gateway merges split usage additively
- gemini: read usageMetadata in the stream path
- ollama: fix non-streaming key (usage -> token_usage, matches
UnifiedResponse json tag) and read prompt_eval_count/eval_count;
surface counts from the done stream chunk
- gateway: mergeUsage combines usage across chunks (non-zero fields win,
total recomputed from prompt+completion) so split usage doesn't lose
the prompt half; single-chunk case (OpenAI) preserved exactly
- usage-only chunks: done=false (no redundant terminal stop), matching
the Go fallback standardSSEChunk
Streaming responses now carry the upstream's real token usage instead of
gateway estimates:
- UnifiedChunk gains an optional Usage field; adapters (opencode, openai)
extract usage from upstream stream chunks (including the final chunk with
empty choices) and pass it through.
- standardSSEChunk preserves usage for passthrough adapters.
- Gateway emits the exact usage in the final stream chunk when available,
falling back to estimates only when the upstream provided none.
Non-streaming usage was already fixed to emit OpenAI-standard keys.
- TokenUsage.MarshalJSON now emits both standard (prompt_tokens,
completion_tokens, total_tokens) and legacy (prompt, completion, total)
keys, so OpenAI-compatible clients (DSH, DevEco Code, etc.) can read
token usage.
- ChatChunk gains an optional Usage field; stream responses now send a
final usage chunk (empty choices) before [DONE].
Refs: usage not visible in clients because the gateway serialized only the
internal short keys and never emitted a streaming usage chunk.
