Status: accepted · Date: 2026-10-21 · Related: ADR-0003, ADR-0005, ADR-0007, ADR-0009
The HASOMED demo runs an agent on two surfaces: support tickets (triage, diagnose, plan a fix and a reply) and practice commands in the Elefant assistant. The agent reads customer data and proposes changes to practice configurations, accounts, exports, appointments and tickets. It has to work without a model key, explain every step to a support lead, never change data on its own, stay within a known spend, and be reachable from external agents without a second implementation. The template's assistant (ADR-0007) approves writes per tool call in a chat thread. That does not fit a run with several dependent fixes, a four-eyes rule and an audit trail across runs, so the harness has its own model. How to extend it is in Extending the harness; how to operate it is in Harness incidents and fallbacks.
Decision
Two engines, AI first, rules always. Every run has a requested engine (auto or rules) and ends with the engine that produced it (ai or rules). The rules engine (RULES_ENGINE_LABEL "Regelbasiert (Demo-Fallback)", detail "Regel-Engine v1") is deterministic and complete: it triages, diagnoses per category, plans and writes the reply without a model. The AI engine goes through the template's AI_PROVIDER seam (TanStack AI structured outputs; default model gpt-6-luna for OpenAI). Our code checks every answer: the schema, the root cause against the category, every evidence quote verbatim against the cited tool output, the plan against REQUIRED_ACTIONS for the root cause, the reply's length, formal address and that it names no other practice. Diagnose and plan issues get one correction retry; a reply naming another practice is final. A failed check is a value, not an exception: the phase is re-run by the rules engine and the run records the first FALLBACK_REASONS value (no_key, timeout, invalid_output, guardrail, error, budget_exceeded, kill_switch); diagnose, plan, command, budget and kill-switch fallbacks also record a blocked guardrail step. The UI shows the engine per phase. A model call never runs inside a database transaction, and each call is bounded (25 s per call, 45 s per run).
One tool registry: code for behaviour, rows for policy. The 23 tools are named in packages/db/src/schema/harness.enums.ts, typed by one zod input schema each (TOOL_INPUTS) and implemented once in TOOLS (packages/api/src/features/harness/harness.tools.ts): a read tool has run, a write tool has a pure preview and an execute. Their German title, description, approval policy, risk and enabled flag live in harness_tool rows; skills (harness_skill) are allowlists of tools plus instructions; context sources document data origin and PII level. Both engines call the same tools through the same checks (makeCall, proposeActions): allowlisted by the skill, enabled, input valid, target inside the ticket's practice, at most MAX_WRITE_ACTIONS writes. Names are text columns, so a new tool, skill or context source needs no migration. A test keeps the rows and the code in step.
Approval policies: auto, confirm, four_eyes; writes are never auto. auto is for read tools: they run during the run and are recorded as trace steps. A write tool is confirm (any member of the Workspace approves) or four_eyes (an owner or admin who is not the requester). updatePolicy rejects auto for a write tool and anything but auto for a read tool; a policy change goes through the platform-admin procedure admin.harness.policies.update (the UI offers it to a support lead who is also platform admin) and is audited as policy.updated.
Writes become pending actions. An engine never executes a write. Its plan becomes agent_action rows with status proposed, the tool's input, a rationale, a preview (before, after, conflicts) computed without writing, a snapshot of approval and risk, the evidence steps and an optional dependency on an earlier action. harness.actions.decide locks the action, applies the stricter of the snapshot and the current policy, checks the dependency has executed, the tool is still enabled and the input still targets the run's practice and ticket, then runs execute in a savepoint of the decision's transaction. A failed execution marks the action failed, changes nothing, and supersedes its dependants. Replays run the original input again on current data; their actions have status replay and can never be approved.
Escalation is a tool, not a shortcut. When an engine is unsure (no practice or category, confidence below 0.5), when the ticket tripped a guardrail, when a guardrail dropped a planned fix, or when the model chooses to, the plan ends with support_ticket_escalate instead of ticket_resolve. It is a write tool with policy confirm: a person approves the hand-over like any other action, and its execution sets the ticket to escalated with an assignee and a coded reason and appends ticket.escalated. A suspected prompt injection proposes no fix at all, only the escalation: the AI engine does not ask the model for a plan, and a fix the rules engine found is held back as a blocked guardrail step. The preview already checks a named assignee against the support Workspace's members; approval checks again and refuses with AssigneeNotMember. People escalate and reassign by hand through support.tickets.escalate and support.tickets.reassign on the same columns.
Cost is metered per call and capped per run and per day. Every model call of a run goes through one meter (harness.usage.ts). Before the call it estimates the cost and refuses the call if the run would exceed AI_BUDGET_PER_RUN or the Organization's Europe/Berlin day would exceed AI_BUDGET_PER_DAY (defaults 0.50 and 10.00 USD); a refusal is a budget_exceeded fallback with its own trace step. After the call it records input, cached, reasoning and output tokens and the cost from MODEL_PRICES (packages/env/src/ai-cost.ts, USD per 1M tokens); an unknown model is priced at a deliberately high fallback rate and labelled as an estimate; a call that ends without reported usage (timeout, abort) is charged at its pre-call estimate, not at 0. Totals are stored on agent_run.usage after every call; agent_run.ai_cost is a generated column over it, so the daily check sums one column and sees parallel runs. AI evals go through the same meter: each eval run writes one spend row (agent_run kind eval, no steps or actions, left out of the run lists) that counts against the daily budget, and each case gets the run budget. Rules runs cost 0.
One kill switch, checked at one seam. The platform admin can force every run onto the rules engine. The switch is one row in the database (harness_setting, read by every trusted session, written only by admin.harness.aiSwitch.set on the system path), not an env var, so it takes effect on the next run without a redeploy. Every flip needs a reason and appends ai.disabled or ai.enabled to the support Workspace's hash chain in the same transaction. The engine router (routeEngine in harness.engine.ts) is the only reader: every engine entry (ticket, command, replay, eval case) and the MCP proposal path asks it once, before any model call and before the budget meter. With the switch on, a run requested as auto goes on rules with kill_switch and a blocked guardrail step that carries the admin's reason; an MCP write proposal is refused (AiDisabled), since it was written by an outside model and no engine of ours can stand in for it. The template's feature flags were not reused: a flag row has no reason, its edits are audited in the template log and not in this chain, and a disabled flag row falls back to the code default, which would silently turn the switch off.
MCP uses the same seam. /api/mcp registers the harness tools from the registry rows of the caller's surface, with the same input schemas, through the same procedures and Workspace guards. Reads run under the tool policy; a write only creates one proposed action (harness.external.service.ts) for a person to approve in the UI. It joins the ticket's run while that run still waits for decisions, leaves an escalated or waiting ticket's status alone, and is refused on a resolved ticket (TicketClosed, CONFLICT). A token works in one demo Workspace, chosen deterministically (support before practice). There is no MCP tool to approve, reject or execute. A tool added to the registry appears on MCP without MCP code; a disabled tool disappears.
Every decision is in a hash chain. Runs, proposals, decisions, executions, failures, policy changes and ticket transitions append agent_audit_event rows in the writer's transaction. Each row's hash is sha256(prev_hash ‖ canonical JSON of the row), seq is gap-free per Organization under a transaction-scoped advisory lock, and a trigger refuses UPDATE, DELETE and TRUNCATE (except during the demo reset). support.audit.verify recomputes the chain and reports the first broken seq. Like every tenant table, the harness tables are scoped by Organization and row-level security (ADR-0003, ADR-0009); the registry tables are global reference rows.
Alternatives rejected
- AI only, with retries. The demo must run without a key and with an exhausted budget, and a support lead must be able to compare. A deterministic engine is also the eval baseline (18 frozen cases, the rules engine must pass all).
- Rules as a separate product path. Both engines share tools, checks, proposals and the trace, so a fallback mid-run is a phase swap, not a second implementation.
- Tool approval inside the model loop (TanStack
needsApproval, as in ADR-0007). It binds approval to a chat turn; the harness needs dependent multi-action plans, four-eyes across people and days, and proposals from MCP, the agent and replays on one record. - Policies only in code. A lead must be able to tighten a policy at runtime; the code owns behaviour and the rows own policy.
- A database constraint for "writes never auto". It would need the tool kind in the row and a migration per change; the rule is enforced in the one service that edits policies, and engines only read policies.
Consequences
- The kill switch stops new model calls, not ones in flight: a run that already passed the router finishes on the engine it started with. It also does not cover the template chat assistant (ADR-0007), which is not a harness run and stays behind the
ai.assistantflag and the provider key. Without the app, the env levers remain: removing the provider key (no_key) or a zero budget (budget_exceeded), both with a redeploy. - A failed AI triage makes the whole run rules; only diagnose and plan fall back per phase.
- A disabled read tool makes a rules run fail instead of falling back; a disabled write tool leads to an escalation.
- The harness uses its own time limits and ignores the assistant's
AI_MAX_STEPS,AI_TIMEOUT_MSandAI_REQUESTS_PER_MINUTE. - AI failures are stored in the run (reason, trace step, usage), not in logs.
- Adding a tool touches the enum, the input schema, the seed, the implementation and the registry test; MCP, the plan schema and the registry pages follow from these.