# Extending the harness

Handover guide: add a tool, skill, context source, policy or eval case; swap the model or provider; run evals and replays; how MCP picks up new tools.

This guide is for the team that takes the harness over. It names every file a change touches, in the order you touch them. The design and its reasons are in [ADR-0014](/docs/adr/0014-harness-engines-and-approvals); what to do when a run falls back is in [Harness incidents and fallbacks](/docs/features/harness-runbook). The pages that show the result are described in [Harness view and evals](/docs/features/harness).

Paths are relative to the repository root. `H/` stands for `packages/api/src/features/harness/`.

## Where things live [#where-things-live]

| Concern                             | File                                                                                               |
| ----------------------------------- | -------------------------------------------------------------------------------------------------- |
| Vocabulary (tools, skills, reasons) | `packages/db/src/schema/harness.enums.ts` (leaf module, imports nothing at runtime)                |
| Tool inputs                         | `packages/db/src/schema/harness.tool-inputs.enums.ts` (`TOOL_INPUTS`, one zod schema per tool)     |
| Registry seed (metadata and policy) | `packages/db/src/seed/hasomed/registry.ts` (`TOOL_SEEDS`, `SKILL_SEEDS`, `CONTEXT_SEEDS`)          |
| Tool implementations                | `H/harness.tools.<area>.ts`, collected in `H/harness.tools.ts` (`TOOLS`)                           |
| Engines                             | `H/harness.engine.ticket.ts`, `H/harness.engine.command.ts`, shared core `H/harness.engine.ts`     |
| AI path                             | `H/harness.ai.ts` (`ask`, schemas), `H/harness.engine.ticket-ai.ts` (checks, retry)                |
| Rules path                          | `H/harness.engine.triage.ts` (`SKILL_BY_CATEGORY`, `RULES`), `H/harness.rules.*.ts`                |
| Usage, cost, budget                 | `H/harness.usage.ts`, price table `packages/env/src/ai-cost.ts`                                    |
| Approval and execution              | `H/harness.engine.propose.ts` (policy snapshot), `H/harness.actions.service.ts` (`decideAction`)   |
| Eval cases                          | `packages/db/src/seed/hasomed/evals.ts`; runner `H/harness.evals.ts`, `H/harness.evals.service.ts` |
| MCP exposure                        | `packages/api/src/features/mcp/mcp.harness-catalog.ts`                                             |

Tool, skill and context source names are `text` columns in Postgres (`harness_tool.name`, `harness_skill.tools text[]`, `agent_action.tool`, …), not database enums. Adding any of the three needs **no migration**. A migration is only needed when a write tool's `execute` writes columns or tables that do not exist yet (as `support_ticket_escalate` did with the `support_ticket` escalation columns).

## Add a tool [#add-a-tool]

The worked example is `support_ticket_escalate`, the newest write tool, next to the read tool `kb_search`.

<Steps>
<Step title="Name it">
Add the name to one of the four lists in `packages/db/src/schema/harness.enums.ts`: `SUPPORT_READ_TOOLS`, `SUPPORT_WRITE_TOOLS`, `PRACTICE_READ_TOOLS` or `PRACTICE_WRITE_TOOLS`. `TOOL_NAMES` spreads all four and is the key type of every registry map. If the tool is the fix for a root cause, add it to `REQUIRED_ACTIONS` for that `RootCause`; an AI plan for that cause must then contain exactly these tools.
</Step>
<Step title="Define its input">
Add a `z.object({ … }).strict()` under the tool's name in `TOOL_INPUTS` (`harness.tool-inputs.enums.ts`). This one schema is used everywhere: `parseToolInput` validates every call and proposal with it, the seed stores it as the JSON Schema in `harness_tool.input_schema` (`z.toJSONSchema`), the AI read tools and the plan prompt are built from it, and MCP uses it as the tool's input schema.
</Step>
<Step title="Seed its metadata and policy">
Add an entry to `TOOL_SEEDS` in `packages/db/src/seed/hasomed/registry.ts`:

- a read tool with `read(name, surface, titleDe, descriptionDe, contextSources)`, which always sets `approval: "auto"` and `risk: "low"`;
- a write tool with `write(name, surface, titleDe, descriptionDe, approval, risk, contextSources)`, where `approval` is `"confirm"` or `"four_eyes"` (`support_ticket_escalate` is `"confirm"`, `"low"`, `["support.ticket"]`).

Then add the tool to the `tools` of every skill in `SKILL_SEEDS` that may use it. A tool that is in no skill can never be called by an engine.

</Step>
<Step title="Implement it">
Write the implementation in a `H/harness.tools.<area>.ts` file, typed by `H/harness.tool-types.ts`:

- `ReadTool`: `run(input, ctx: ToolContext)` returns JSON-serialisable data. It reads through `ctx.data` (`HarnessData`), so the same tool runs against the database and against eval fixtures.
- `WriteTool`: `preview(input, ctx)` is pure and returns an `ActionPreview` (before, after, conflicts, `summaryDe`); `execute(input, exec: ExecContext)` runs only from an approved action, inside the decision's transaction (`exec.tx`), and returns an `ExecResult`.

Spread the new array into `TOOLS` in `H/harness.tools.ts`. Importing that module throws `Tool <name> has no implementation` for any `TOOL_NAMES` entry that is missing.

Execution after approval is generic (`TOOLS[action.tool].execute` in `H/harness.actions.service.ts`). Add code there only if the tool emits an audit event of its own, as `support_ticket_escalate` (`ticket.escalated`, `ticket.reassigned`) and `ticket_resolve` do.

</Step>
<Step title="Wire the rules engine">
A rules diagnoser (`H/harness.rules.<category>.ts`, registered in `RULES` in `H/harness.engine.triage.ts`) reads through `ctx.call(tool, input)` and returns a `RulesOutcome` whose `actions` are `PlannedAction`s (`H/harness.rules.shared.ts`). The engine appends `ticket_resolve` itself. Escalation is not a rule: the engine closes with `support_ticket_escalate` instead of `ticket_resolve` when it is unsure (`H/harness.engine.escalate.ts`).
</Step>
<Step title="Wire the AI engine">
Nothing to register by hand: the AI engine derives its tools from the skill. In `H/harness.engine.ticket-ai.ts` the diagnose call gets the skill's read tools (descriptions from `harness_tool.title_de`), and the plan schema is built from the skill's write tools except `ticket_resolve` (`planSchema` in `H/harness.ai.ts`). Every call still passes the allowlist, input and scope checks in `makeCall` and `proposeActions`. Commands (practice) map intents to skills in `SKILL_BY_INTENT` and call practice tools per intent in `H/harness.engine.command.ts`.
</Step>
<Step title="Show it">
Registry pages show `title_de` and `description_de` from the database; there is no locale key per support tool. Trace steps, action cards and the audit show the tool's name. Only the practice write tools have labels (`TOOL_TITLE` in `apps/web/src/features/elefant/elefant.preview-card.tsx`, keys `ELEFANT_TOOL_*` in `apps/web/src/locales/elefant.{de,en,fr}.ts`).
</Step>
<Step title="Test it">
- `H/harness.flow.test.ts` ("the registry is public, matches the code and admin policy edits are audited") asserts the registry counts (today 23 tools, 13 read, 10 write, 10 skills, 12 context sources) and that `[name, kind, surface]` in the database equals `TOOLS`. Update the counts. It is an integration test: run it with `pnpm test:integration`, which starts a disposable local Postgres and sets `TEST_DATABASE_URL`.
- Unit-test the tool's `run` or `preview` against `fixtureSource` data (`H/harness.data.fixture.ts`; add fields there if the fixture rows lack them).
- If the tool changes what the rules engine plans for the open demo tickets, re-record the seed pre-run (see [Eval cases](#add-an-eval-case)) and update `H/harness.prerun.test.ts`, `H/harness.rules.eval.test.ts` and, for AI plans, `H/harness.ai-plan.test.ts`.
</Step>
<Step title="Write the rows">
`pnpm --filter @template-cloudflare/db db:seed:reset-demo` inserts tools, skills and context sources that are new in code. Existing tool rows keep their policy, title and schema; skills take the code's `tools` and `instructionsMd`. It also rewrites the demo data (keeping the logins). `db:seed` instead deletes and re-inserts the whole registry and loses policy edits. Both refuse a non-local database unless `SEED_ALLOW_REMOTE=true`; the demo database is shared, so coordinate before seeding it.
</Step>
</Steps>

## Add a skill [#add-a-skill]

1. Add the name to `SKILL_NAMES` in `harness.enums.ts`.
2. Add a `SkillSeed` to `SKILL_SEEDS` in `registry.ts`: `name`, `surface`, `titleDe`, `descriptionDe`, `instructionsMd`, `tools`, `ruleSet`, `examples`, `sort`. `tools` is the allowlist: `makeCall` refuses any other tool with the guardrail `tool_not_allowed`, and `proposeActions` drops any other write.
3. Route to it. Support tickets map a `TicketCategory` to a skill in `SKILL_BY_CATEGORY` (`H/harness.engine.triage.ts`), and each category needs a diagnoser in `RULES`. Practice commands map a `CommandIntent` in `SKILL_BY_INTENT` (`H/harness.engine.command.ts`).
4. Write the prompt. `instructions_md` is the prefix of the AI diagnose and plan prompts; the seeds embed a shared `RULES` text. The triage and command prompts are inline in the engine and do not use skill instructions. A command skill's `examples` are listed in the reply to an unknown command.

The registry page lists skills with their tools (`apps/web/src/features/harness/harness.skills.tsx`).

## Add a context source [#add-a-context-source]

Context sources document where a tool's data comes from and how personal it is; the engines never read them. Add a `ctx(sort, name, surface, kind, titleDe, descriptionDe, tables, pii)` entry to `CONTEXT_SEEDS` in `registry.ts`:

- `kind`: `database`, `knowledge`, `policy` or `conversation` (`CONTEXT_SOURCE_KINDS`);
- `pii`: `none`, `pseudonymised` or `contact_data` (`PII_LEVELS`);
- `freshnessDe` is set to `"live"`.

Reference it by name in the tools' `contextSources`. The name is a plain string with no foreign key and nothing validates it: a misspelt name shows in the tool sheet as the bare name without a title. Check the tool sheet and the "Kontext" tab (`/harness?tab=kontext`) after seeding.

## Change a policy [#change-a-policy]

A tool's policy is `harness_tool.approval` (`auto`, `confirm`, `four_eyes`), `risk` and `enabled`. Change it in the UI on `/harness?tab=richtlinien` (editable for a support lead who is also platform admin) or through `admin.harness.policies.update` (`H/harness.admin.procedures.ts`, a platform-admin procedure). The service (`updatePolicy` in `H/harness.registry.service.ts`) is where "writes are never `auto`" is enforced: a write tool with `auto`, or a read tool with anything else, fails with `PolicyInvalid`. There is no database constraint, so do not edit the rows by SQL. Every change appends `policy.updated` to the support Workspace's audit chain.

When an engine proposes an action, `proposeActions` copies the tool's current `approval` and `risk` onto `agent_action`. At decision time `decideAction` applies the stricter of that snapshot and the current policy, so tightening a policy to `four_eyes` also covers proposals that are already open.

To change a default for fresh databases, change the `write(…)` call in `TOOL_SEEDS` and re-record the seed pre-run: the recorded actions carry their approval.

## Add an eval case [#add-an-eval-case]

Cases are defined in `packages/db/src/seed/hasomed/evals.ts`: `TICKET_EXPECTATIONS` (one tuple `[ticket number, slug, expected, tags]` per open ticket) and `commandExpectations(now)` (practice commands); `buildEvalCases(now)` builds the `eval_case` rows, 18 today. `expected` is an `EvalExpected` (`harness.enums.ts`): `category`, `priority`, `rootCause`, `actions`, `forbiddenActions`, `intent`, `params`, `guardrails`. Each field becomes one check (`EVAL_CHECKS`); `actions` are compared as sorted lists.

A ticket case needs its ticket and the practice data in the frozen fixture (`packages/db/src/seed/hasomed/fixtures.ts`, `supportFixture`), because evals run against `fixtureSource`, never against live rows, and never execute a write.

After adding a case:

- `H/harness.rules.eval.test.ts` requires the rules engine to pass every case at two pinned times. Update the expected count (`has 18 cases`).
- The seed writes the cases (`db:seed` or `db:seed:reset-demo`).
- If the rules output for an open ticket changed, re-record the seed pre-run: run `H/harness.prerun.test.ts` with `UPDATE_PRERUN=1` (it rewrites `packages/db/src/seed/hasomed/prerun.recorded.json`), then format the file with `pnpm exec oxfmt --write`.

## Swap the model or provider [#swap-the-model-or-provider]

The AI engine goes through the template's provider seam, `packages/api/src/platform/ai/`:

| Setting       | Env var (typed in `packages/env/src/server.ts`)                     | Notes                                                                                                                           |
| ------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Provider      | `AI_PROVIDER` = `anthropic`, `openai` or `gemini`                   | Adapters in `ai.provider.ts` (`createTextAdapter`)                                                                              |
| Key           | `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or `GEMINI_API_KEY`           | The one that matches the provider. Without it `aiSettings()` returns `null` and runs use the rules engine                       |
| Model         | `AI_MODEL`                                                          | Defaults in `DEFAULT_AI_MODELS` (`ai.settings.ts`): `claude-sonnet-5-5`, `gpt-6-luna`, `gemini-4-argon`                         |
| Budgets (USD) | `AI_BUDGET_PER_RUN` (default 0.5), `AI_BUDGET_PER_DAY` (default 10) | Schema in `packages/env/src/ai-cost.ts`, documented in `apps/web/.env.example`; `0` refuses every model call, AI evals included |

Locally the values live in `apps/web/.dev.vars` (gitignored); in a deployment they are Worker variables and secrets. `harness.engine.info` reports the active provider and model (`ai: null` without a key), and the ticket page offers "Automatisch" only when it is set.

`AI_MAX_STEPS`, `AI_TIMEOUT_MS` and `AI_REQUESTS_PER_MINUTE` belong to the template assistant; the harness does not read them. Its limits are in code: 25 s per model call (`CALL_CAP_MS` in `H/harness.ai.ts`), 45 s per run (`aiDeps` in `H/harness.procedures.ts`) and at most 6 tool-loop iterations per call (`ai.structured.ts`).

**Add a price.** Cost is computed from `MODEL_PRICES` in `packages/env/src/ai-cost.ts`: USD per 1 million input, cached input and output tokens, keyed by model id (a dated snapshot such as `gpt-6-luna-2026-08-01` matches its base id). A model that is not in the table is priced with `FALLBACK_PRICE` (2.50 / 0.25 / 10 USD) and every cost shown for it is marked as estimated ("Modell nicht in der Preistabelle: Kosten geschätzt."). Add an entry with the provider's list price when you change `AI_MODEL`. Reasoning tokens are billed as output.

**Check the budgets.** The run budget must leave room for a full ticket run (triage, diagnose with its tool loop, plan, one correction retry). Before every call the meter estimates the call's cost from the prompt (tool-loop calls at four times the prompt's input and twice the output) and refuses it when the run or the day would exceed its budget. With a pricier model, raise `AI_BUDGET_PER_RUN` or expect `budget_exceeded` fallbacks.

After a swap, run the AI evals and compare them with the previous model in the trend chart. Then run the real-model smoke test against your dev server: `node scripts/demo/harness-ai-smoke.mjs --base http://localhost:<port>`. It signs in with the demo logins, runs DEMO-1004 and a practice command with engine "auto", prints the engine and model that answered, and writes demo runs.

## Run evals [#run-evals]

- **UI.** `/harness/evals`: "Evals ausführen" for the rules or the AI engine, results per case and check, history and score trend.
- **Procedure.** `harness.evals.run` (support Workspace) with `{ engine: "rules" | "ai", caseIds? }`. `ai` without a configured key fails with `AiNotConfigured`. Each case gets 45 s and the run budget (`AI_BUDGET_PER_RUN`). An AI eval is metered like a run: one spend row (`agent_run` kind `eval`, hidden from the runs lists) carries the calls of all cases, counts in today's spend and is refused by `AI_BUDGET_PER_DAY`. A refused call makes that case fall back to the rules with `budget_exceeded` in its result.
- **Tests.** The rules engine is checked on every case in `pnpm test` (`H/harness.rules.eval.test.ts`). `H/harness.ai-recorded.test.ts` replays hand-written `gpt-6-luna` answers through the `fakeText` adapter from `@tanstack/ai/testing`; it does not call a model, and there is no re-record switch. There is no eval CLI script.

## Replay a run [#replay-a-run]

On a run's detail page (`/harness/laeufe/<runId>`, support runs only) the replay panel runs the original input again on current data with the engine you choose ("Nur Regeln" by default, or "Automatisch"). It calls `harness.runs.replay` with `{ runId, engine: "auto" | "rules" }`, which creates a run of kind `replay` with `replay_of` set, then `harness.runs.execute`. A replay's actions get the status `replay` and can never be approved; the ticket is not changed. The panel then compares root cause, actions, step count, engine and status with the original.

## MCP picks up tools automatically [#mcp-picks-up-tools-automatically]

Nothing in the MCP code names a harness tool. `registerHarnessTools` (`packages/api/src/features/mcp/mcp.harness-catalog.ts`) loads the registry (`harness.registry.get`), keeps the rows of the caller's surface that are `enabled`, and registers each with its `title_de`, `description_de` and the input schema from `TOOL_INPUTS`. Read tools run through the harness read adapter; write tools only propose an action (`harness.external.service.ts`) that a person approves in the UI. `HARNESS_WRITE_TOOL_NAMES`, which decides which calls need `data:write`, is derived from `TOOLS`. A new tool is therefore on `/api/mcp` as soon as its registry row exists. Disabling a tool in the policies removes it from `tools/list`. Details: [MCP and agent connections](/docs/features/mcp).