A local mock LLM API where a typed word triggers each failure

Build a local fake of {{provider}}, for this app to point at in development. Every failure the app must handle should be reproducible by typing a trigger word into the app's own input: no API key, no network, no cost.

First read how the app ({{app_stack}}) calls the API: base URL, auth header, request body, streaming or not, timeouts and retries, where the user's input lands in the request, and how each error reaches the user today. Then build the following.

1. The mock: dev/mock-upstream.mjs
- One file, plain Node 22, no dependencies. Listen on 127.0.0.1 only. Port from MOCK_PORT, delay between chunks from MOCK_DELAY_MS.
- Match the real API exactly: paths, methods, status codes, headers, error bodies and error type names, and the streaming event sequence and framing. For the Anthropic Messages API that is message_start, content_block_start, ping, content_block_delta (text_delta), content_block_stop, message_delta (stop_reason, usage), message_stop, each written as "event: <name>\ndata: <json>\n\n". When thinking is on (the request asks for it, or leaves out the thinking field on a model that thinks by default), a thinking block comes first: with display "omitted" (the default on current models), one empty thinking_delta, then a signature_delta. The text block is then index 1. The app's SDK or parser must not be able to tell the mock from the real thing.
- With no trigger word, stream a good answer: the app's own sample or fixture output whose file name best matches the input, else a short built-in one, so the mock works in a fresh checkout. Split it into 40–80 character deltas on code-point boundaries, seeded from a hash of the input, so the same input always streams the same way.
- Answer a request with no auth header with the provider's 401. Serve the non-streaming form too if the app uses it.
- Log one line per request: path, status, mode, which sample it streamed, and the shape of the call (model, stream, max tokens, thinking or effort settings, beta headers, system and user sizes, cache markers). Never log the key.
- Stop writing when the client disconnects. An exception in a handler returns the provider's 500 error body or, once streaming has started, its error event: never a hung socket.

2. Trigger words
Read them only from the user's own text in the request (for example the line of the user message that holds what the user typed): case-insensitive, whole words, first match in the list wins. Never match against the system prompt or the whole body: they can contain "400" or "drop" by accident.
- "401": HTTP 401 with the provider's authentication error.
- "429": HTTP 429 rate-limit error with retry-after. If the SDK retries on its own, add the header that stops it (Anthropic: x-should-retry: false), or every test waits through backoff.
- "400": HTTP 400 invalid-request error.
- "529" and "500": the provider's overloaded and server errors as HTTP statuses, before any stream, with the same no-retry header.
- "overload": a few lines of the good answer, then the provider's mid-stream error event (Anthropic: event: error with overloaded_error). No stop event.
- "drop": a few lines, then destroy the socket. No error event, no stop event.
- "slow": hold the first byte for MOCK_SLOW_MS, longer than the app's upstream timeout. Make that timeout an env var so the test can shorten both. If the app has no timeout of its own, add one. If the SDK retries timeouts, turn retries off in the test env.
- "malformed": a few lines, then a data line that is not valid JSON.
Add these app-specific triggers too:
{{failure_modes}}
Mid-stream failures always come after real content, so the UI has a half-written answer to deal with. If the app falls back through a list of models, add a trigger that fails only the first model, so the fall-through runs. If the app calls other upstreams (search, retrieval, a second provider), mock them in the same file with their own triggers, such as "missing" for no result.

3. Wiring
- The app reads the base URL from an env var (for Anthropic, ANTHROPIC_BASE_URL) and uses the real host when it is unset. The mock's URL never appears in app code.
- Commit an example env file that points at the mock with an obviously fake key, so a copy of it runs against the mock as written, and git-ignore the copy. To call the real API locally, delete the base-URL line in the copy and add a real key. In production the key lives in the host's secret store, never in a file.

4. Tests
- One end-to-end test per trigger word, through the real UI (Playwright if there is a browser front end, else the app's own HTTP endpoint): start the mock on a free port, start the app pointed at it, enter the trigger, assert what the user sees.
- After each failure, assert nothing was cached or saved: send the same input again and check the mock logged it a second time. Then assert that a normal input succeeds.
- If the app caches good answers, clear the cache between tests: a cached answer never reaches the mock.
- If the app rate-limits, gates or bot-checks its own users, give the tests a documented way past it (a separate client IP or access code per test, a bot check mocked in the same file that passes a known token) and list it under the gotchas.
- Fix what the tests expose in the app, never by bending the mock.

5. Docs: a "Running against the mock" section in the dev README
- The commands to run the app against the mock.
- A table with one row per trigger word: trigger | what the mock does | what the user should see. The tests and the table cover the same rows.
- Two or three curl checks, and the gotchas: caching, the app's own rate limit or gates, and restarting the app after editing the env file.

Finish by starting the mock and the app pointed at it (from the example env file). Curl one normal request and one trigger word through the app's own endpoint and show what the client receives, then add one raw curl against the mock with its output.

Show all 43 lines