# Harness incidents and fallbacks Runbook for support and operators: every fallback reason with symptom, diagnosis and fix, replay with a chosen engine, the AI kill switch, escalations and audit-chain verification. Every agent run ends with an engine (`ai` or `rules`) and, if the AI engine was asked for but did not finish the run, a fallback reason. A fallback is not an outage: the rules engine finishes the run, and every proposed write still waits for a person. This page says how to read a fallback, what caused it and what to change. The design is in [ADR-0014](/docs/adr/0014-harness-engines-and-approvals); how to change tools, models and budgets is in [Extending the harness](/docs/features/harness-handover). ## Where to look [#where-to-look] | Place | What it tells you | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Runs list `/harness/laeufe` | Engine badge with "Grund für den Fallback: …", status, cost per run, and "KI-Kosten heute" (today's spend against `AI_BUDGET_PER_DAY`). Filters `surface`, `engine`, `status`, `seite`, `mandant` live in the URL; there is no filter by reason | | Run detail `/harness/laeufe/` | Header: engine badge and reason, "Angefordert" ("Automatisch" or "Nur Regeln"), "Engine je Phase" ("KI verworfen → Regeln" on a phase the AI lost). Trace: each step with kind, status and raw JSON. "KI-Verbrauch": every model call with tokens and cost, "Antwort verworfen" on a failed one | | Ticket page `/support/tickets/` | The latest run's engine and reason; "Automatisch" disabled with a reason when no key is configured | | Database | `agent_run.fallback_reason`, `requested_engine`, `engine`, `engine_detail`, `usage` (JSON), `ai_cost` (generated from `usage`); `agent_step` rows (`kind`, `status`, `title_de`, `output`); `agent_audit_event` `run.finished` with `after.fallbackReason` | The harness writes **no log line** for an AI failure: the failure is converted to a value and stored in the run. The run row and its steps are the record. Server logs only carry refused procedure calls (`console.debug("[orpc]", code, message, { requestId, data })` for expected 4xx errors, with the harness reason in `data.reason`) and unexpected errors (`console.error`). The `x-request-id` response header matches the `requestId` in those lines. Useful queries (run them with a connection that is not the `app_tenant` role, which only sees one Organization): Fallback reasons and spend today (Europe/Berlin day, as the budget counts it): ```sql select fallback_reason, engine, count(*), sum(ai_cost) from agent_run where started_at >= (date_trunc('day', now() at time zone 'Europe/Berlin') at time zone 'Europe/Berlin') group by 1, 2; ``` The step that explains a fallback: ```sql select seq, phase, kind, status, title_de, output from agent_step where run_id = '' and kind = 'guardrail' order by seq; ``` A step's input and output over 8 KB are stored as `{ "truncated": true, "preview": "…" }`. ## How a fallback is recorded [#how-a-fallback-is-recorded] The ticket engine keeps the **first** reason of a run (`fallbackReason ??= reason` in `harness.engine.ticket.ts`). Where it failed decides how much of the run is rules: - **Triage** fails: the reason is set, no step of its own is recorded (except the budget step below), and the whole run continues on rules. The triage step is a "Regelentscheidung". - **Diagnose or plan** fails: a `guardrail` step with status `blocked`, phase `diagnose` or `plan`, titled `KI-Ausgabe verworfen (): Regel-Engine übernimmt`, output `{ reason, message }`. The rules engine re-runs that phase. - **Command** (practice) fails: a `guardrail` step with status `blocked`, phase `triage`, titled `KI-Befehl verworfen (): Regel-Parser übernimmt`. The fallback reasons are `FALLBACK_REASONS` in `packages/db/src/schema/harness.enums.ts`: `no_key`, `timeout`, `invalid_output`, `guardrail`, `error`, `budget_exceeded`, `kill_switch`. Each has one section below. The UI labels are `AGENT_FALLBACK_*` in `apps/web/src/locales/agent.{de,en,fr}.ts`. ## `no_key` [#no_key] **Symptom.** "Grund für den Fallback: Kein KI-Schlüssel konfiguriert". The ticket page disables "Automatisch" and starts every run as "Nur Regeln", so those runs carry **no** reason. `no_key` appears on runs that were requested as "Automatisch" while no key was configured: a run started before the key was removed and executed after, or an "Automatisch" replay. **Diagnosis.** `agent_run.requested_engine = 'auto'`, `engine = 'rules'`, `fallback_reason = 'no_key'`, and no AI step in the trace. `harness.engine.info` returns `ai: null`. `aiSettings()` (`packages/api/src/platform/ai/ai.settings.ts`) returns `null` when `AI_PROVIDER` is unset or the matching key (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`) is empty. **Fix.** Set `AI_PROVIDER` and its key in the Worker's variables and secrets (locally `apps/web/.dev.vars`) and restart or redeploy. Check that the ticket page offers "Automatisch" again. Never paste the key into a ticket, a trace or these docs. ## `timeout` [#timeout] **Symptom.** "KI hat nicht rechtzeitig geantwortet". In "KI-Verbrauch" the timed-out call is not free: without usage from the provider it is charged at the estimate made before it started (marked as estimated). **Diagnosis.** Three causes, told apart by the step's `output.message`: - `nach ms`: one model call hit its cap, 25 s (`CALL_CAP_MS` in `harness.ai.ts`) or less if the run had less time left; - `Zeitbudget des Laufs erschöpft`: under 2 s were left of the run's 45 s deadline (set by `aiDeps()` in `harness.procedures.ts` when `harness.runs.execute` starts) before the next call; - `abgebrochen`: the call was interrupted, for example because the request was cancelled. `AI_TIMEOUT_MS` does not apply to the harness. Compare the run's `duration_ms` and the `durationMs` of its steps: slow tool calls eat the same 45 s. **Fix.** One slow answer: replay with "Automatisch". Repeated timeouts: check the provider's status page and latency; a larger model may need a smaller prompt or a faster model. The caps are constants in code, not env vars. ## `invalid_output` [#invalid_output] **Symptom.** "KI-Ausgabe hat die Prüfungen nicht bestanden". **Diagnosis.** The answer did not pass a check. Read `output.message` of the `KI-Ausgabe verworfen (invalid_output)` step: - the answer does not fit the structured-output schema; - diagnose: the root cause does not belong to the ticket's category, or an evidence quote does not appear verbatim in the step it cites (the message names the step it does appear in, if any); - plan: the actions are not the ones `REQUIRED_ACTIONS` demands for the root cause (and not a single escalation), a tool input is not valid JSON for its schema, the reply is longer than 180 words, uses du-form, or names another practice; - command: the parameters do not fit the intent ("Parameter passen nicht zum Intent"). Every check except a reply that names another practice (that one is final) and a planned tool outside the skill or scope (`guardrail`) gets **one** correction retry. A step "KI-Ausgabe korrigiert (1 Versuch)" with status `ok` means the retry passed and the run stayed on AI; `invalid_output` means the retry failed too. In "KI-Verbrauch" a rejected call shows "Antwort verworfen". **Fix.** A single case: replay with "Automatisch" and compare. A pattern (same check, many runs): run the AI evals on `/harness/evals` and see which check fails; adjust the skill's `instructions_md` (re-seed with `db:seed:reset-demo`) or the model. Do not loosen the checks to make a model pass. ## `guardrail` [#guardrail] **Symptom.** "Von einer Schutzregel blockiert". **Diagnosis.** Set only by the ticket plan check (`harness.engine.ticket-ai.ts`): the model planned a tool that is not a write tool of the run's skill (` ist nicht freigegeben`), or an action whose input targets another practice or ticket (`… zielt auf eine fremde Praxis`), or, on a ticket with a suspected prompt injection, any action other than the escalation (`Verdacht auf Prompt-Injection: aus diesem Ticket wird keine Korrektur geplant`; the engine does not ask the model for a plan on such a ticket, so this only guards other callers of `aiPlan`). The message is in the `KI-Ausgabe verworfen (guardrail)` step. Do not confuse it with guardrail **flags** on a run (`prompt_injection_suspected`, `tool_not_allowed`, `tool_disabled`, `target_out_of_scope`, `input_invalid`, `too_many_writes`, `evidence_unverified`). Flags are recorded as `guardrail` steps of either engine and can lead to an escalation without any fallback. **Fix.** Usually none: the guardrail did its job and the rules plan is in place. If a tool should be allowed, add it to the skill's `tools` ([Add a skill](/docs/features/harness-handover#add-a-skill)). If the ticket text tried to steer the agent, leave it to the escalation. ## `error` [#error] **Symptom.** "KI nicht erreichbar". **Diagnosis.** Any other provider failure (`AiUnavailable` in `packages/api/src/platform/ai/ai.structured.ts`): network error, an HTTP 4xx or 5xx (wrong key, unknown model, quota, rate limit), a refusal or a stream error. The message is in the step's `output.message` (up to 300 characters); on a triage failure there is no step, only the reason. Not tested: a read tool that throws inside the model's tool loop most likely also ends as `error`. **Fix.** A rejected key or an unknown `AI_MODEL`: correct the variable. Quota or rate limit: check the provider account. A provider outage: switch to the rules engine for the duration (see [Turning the AI off](#turning-the-ai-off)). ## `budget_exceeded` [#budget_exceeded] **Symptom.** "KI-Budget erreicht". The runs list's "KI-Kosten heute" tile turns orange from 80 % of the daily budget. **Diagnosis.** Before every model call the meter (`harness.usage.ts`) estimates the call's cost and refuses it if the run total plus the estimate would exceed `AI_BUDGET_PER_RUN`, or today's spend of the Organization plus the run plus the estimate would exceed `AI_BUDGET_PER_DAY` (USD, Europe/Berlin day). A refused call is not made and not charged. The trace has a `guardrail` step with status `blocked`, titled "KI-Budget des Laufs erreicht: Regel-Engine übernimmt" or "KI-Tagesbudget erreicht: Regel-Engine übernimmt", with output `{ reason: "budget_exceeded", scope: "run" | "day", spent, estimate, budget, currency }`. Today's spend is the sum of `agent_run.ai_cost`, including the spend rows of AI evals (`kind = 'eval'`, not shown in the runs list). AI evals go through the same meter: a refused call makes that case fall back to the rules with `budget_exceeded` in its eval result. A call that ended without reported usage (timeout, abort) is charged at its pre-call estimate (`unreported: true` in `usage.calls`), so it counts too. To see what spent today's budget, by kind (`eval` rows are the AI eval runs): ```sql select kind, count(*), sum(ai_cost) from agent_run where started_at >= date_trunc('day', now() at time zone 'Europe/Berlin') at time zone 'Europe/Berlin' group by kind; ``` Runs that hit a budget: ```sql select r.id, r.started_at, s.phase, s.output from agent_step s join agent_run r on r.id = s.run_id where s.kind = 'guardrail' and s.output ->> 'reason' = 'budget_exceeded' order by r.started_at desc; ``` **Fix.** `scope: "day"`: wait for the next Berlin day or raise `AI_BUDGET_PER_DAY`. `scope: "run"`: the model is pricier than the run budget allows; raise `AI_BUDGET_PER_RUN`. If the cost is marked as estimated, the model is missing from `MODEL_PRICES` and is priced at the high fallback rate: add its price ([Swap the model](/docs/features/harness-handover#swap-the-model-or-provider)). ## `kill_switch` [#kill_switch] **Symptom.** "Grund für den Fallback: KI deaktiviert". Every harness page and the runs list show the notice "KI deaktiviert · Alle Läufe nutzen die Regel-Engine." with the admin's reason; the run detail header adds `KI deaktiviert: (von , )`. **Diagnosis.** The platform AI kill switch is on (`harness_setting.ai_disabled = true`, see [Turning the AI off](#turning-the-ai-off)). The run has `requested_engine = 'auto'`, `engine = 'rules'`, `fallback_reason = 'kill_switch'`, no model step and no "KI-Verbrauch" (no model call was made, no budget was asked). Its first step is a `guardrail` step (phase `triage`, status `blocked`) titled `KI deaktiviert: (von ,