Extending the harness

This guide is for the team that takes the harness over. It names every file a change touches, in the order you touch them. The design and its reasons are in ADR-0014; what to do when a run falls back is in Harness incidents and fallbacks. The pages that show the result are described in Harness view and evals.

Paths are relative to the repository root. H/ stands for packages/api/src/features/harness/.

Where things live

Tool, skill and context source names are text columns in Postgres (harness_tool.name, harness_skill.tools text[], agent_action.tool, …), not database enums. Adding any of the three needs no migration. A migration is only needed when a write tool's execute writes columns or tables that do not exist yet (as support_ticket_escalate did with the support_ticket escalation columns).

Add a tool

The worked example is support_ticket_escalate, the newest write tool, next to the read tool kb_search.

1. Name it

   Add the name to one of the four lists in packages/db/src/schema/harness.enums.ts: SUPPORT_READ_TOOLS, SUPPORT_WRITE_TOOLS, PRACTICE_READ_TOOLS or PRACTICE_WRITE_TOOLS. TOOL_NAMES spreads all four and is the key type of every registry map. If the tool is the fix for a root cause, add it to REQUIRED_ACTIONS for that RootCause; an AI plan for that cause must then contain exactly these tools.

2. Define its input

   Add a z.object({ … }).strict() under the tool's name in TOOL_INPUTS (harness.tool-inputs.enums.ts). This one schema is used everywhere: parseToolInput validates every call and proposal with it, the seed stores it as the JSON Schema in harness_tool.input_schema (z.toJSONSchema), the AI read tools and the plan prompt are built from it, and MCP uses it as the tool's input schema.

3. Seed its metadata and policy

   Add an entry to TOOL_SEEDS in packages/db/src/seed/hasomed/registry.ts:

   • a read tool with read(name, surface, titleDe, descriptionDe, contextSources), which always sets approval: "auto" and risk: "low";
   • a write tool with write(name, surface, titleDe, descriptionDe, approval, risk, contextSources), where approval is "confirm" or "four_eyes" (support_ticket_escalate is "confirm", "low", ["support.ticket"]).

   Then add the tool to the tools of every skill in SKILL_SEEDS that may use it. A tool that is in no skill can never be called by an engine.

4. Implement it

   Write the implementation in a H/harness.tools.<area>.ts file, typed by H/harness.tool-types.ts:

   • ReadTool: run(input, ctx: ToolContext) returns JSON-serialisable data. It reads through ctx.data (HarnessData), so the same tool runs against the database and against eval fixtures.
   • WriteTool: preview(input, ctx) is pure and returns an ActionPreview (before, after, conflicts, summaryDe); execute(input, exec: ExecContext) runs only from an approved action, inside the decision's transaction (exec.tx), and returns an ExecResult.

   Spread the new array into TOOLS in H/harness.tools.ts. Importing that module throws Tool <name> has no implementation for any TOOL_NAMES entry that is missing.

   Execution after approval is generic (TOOLS[action.tool].execute in H/harness.actions.service.ts). Add code there only if the tool emits an audit event of its own, as support_ticket_escalate (ticket.escalated, ticket.reassigned) and ticket_resolve do.

5. Wire the rules engine

   A rules diagnoser (H/harness.rules.<category>.ts, registered in RULES in H/harness.engine.triage.ts) reads through ctx.call(tool, input) and returns a RulesOutcome whose actions are PlannedActions (H/harness.rules.shared.ts). The engine appends ticket_resolve itself. Escalation is not a rule: the engine closes with support_ticket_escalate instead of ticket_resolve when it is unsure (H/harness.engine.escalate.ts).

6. Wire the AI engine

   Nothing to register by hand: the AI engine derives its tools from the skill. In H/harness.engine.ticket-ai.ts the diagnose call gets the skill's read tools (descriptions from harness_tool.title_de), and the plan schema is built from the skill's write tools except ticket_resolve (planSchema in H/harness.ai.ts). Every call still passes the allowlist, input and scope checks in makeCall and proposeActions. Commands (practice) map intents to skills in SKILL_BY_INTENT and call practice tools per intent in H/harness.engine.command.ts.

7. Show it

   Registry pages show title_de and description_de from the database; there is no locale key per support tool. Trace steps, action cards and the audit show the tool's name. Only the practice write tools have labels (TOOL_TITLE in apps/web/src/features/elefant/elefant.preview-card.tsx, keys ELEFANT_TOOL_* in apps/web/src/locales/elefant.{de,en,fr}.ts).

8. Test it

   • H/harness.flow.test.ts ("the registry is public, matches the code and admin policy edits are audited") asserts the registry counts (today 23 tools, 13 read, 10 write, 10 skills, 12 context sources) and that [name, kind, surface] in the database equals TOOLS. Update the counts. It is an integration test: run it with pnpm test:integration, which starts a disposable local Postgres and sets TEST_DATABASE_URL.
   • Unit-test the tool's run or preview against fixtureSource data (H/harness.data.fixture.ts; add fields there if the fixture rows lack them).
   • If the tool changes what the rules engine plans for the open demo tickets, re-record the seed pre-run (see Eval cases) and update H/harness.prerun.test.ts, H/harness.rules.eval.test.ts and, for AI plans, H/harness.ai-plan.test.ts.

9. Write the rows

   pnpm --filter @template-cloudflare/db db:seed:reset-demo inserts tools, skills and context sources that are new in code. Existing tool rows keep their policy, title and schema; skills take the code's tools and instructionsMd. It also rewrites the demo data (keeping the logins). db:seed instead deletes and re-inserts the whole registry and loses policy edits. Both refuse a non-local database unless SEED_ALLOW_REMOTE=true; the demo database is shared, so coordinate before seeding it.

Add a skill

1. Add the name to SKILL_NAMES in harness.enums.ts.
2. Add a SkillSeed to SKILL_SEEDS in registry.ts: name, surface, titleDe, descriptionDe, instructionsMd, tools, ruleSet, examples, sort. tools is the allowlist: makeCall refuses any other tool with the guardrail tool_not_allowed, and proposeActions drops any other write.
3. Route to it. Support tickets map a TicketCategory to a skill in SKILL_BY_CATEGORY (H/harness.engine.triage.ts), and each category needs a diagnoser in RULES. Practice commands map a CommandIntent in SKILL_BY_INTENT (H/harness.engine.command.ts).
4. Write the prompt. instructions_md is the prefix of the AI diagnose and plan prompts; the seeds embed a shared RULES text. The triage and command prompts are inline in the engine and do not use skill instructions. A command skill's examples are listed in the reply to an unknown command.

The registry page lists skills with their tools (apps/web/src/features/harness/harness.skills.tsx).

Add a context source

Context sources document where a tool's data comes from and how personal it is; the engines never read them. Add a ctx(sort, name, surface, kind, titleDe, descriptionDe, tables, pii) entry to CONTEXT_SEEDS in registry.ts:

• kind: database, knowledge, policy or conversation (CONTEXT_SOURCE_KINDS);
• pii: none, pseudonymised or contact_data (PII_LEVELS);
• freshnessDe is set to "live".

Reference it by name in the tools' contextSources. The name is a plain string with no foreign key and nothing validates it: a misspelt name shows in the tool sheet as the bare name without a title. Check the tool sheet and the "Kontext" tab (/harness?tab=kontext) after seeding.

Change a policy

A tool's policy is harness_tool.approval (auto, confirm, four_eyes), risk and enabled. Change it in the UI on /harness?tab=richtlinien (editable for a support lead who is also platform admin) or through admin.harness.policies.update (H/harness.admin.procedures.ts, a platform-admin procedure). The service (updatePolicy in H/harness.registry.service.ts) is where "writes are never auto" is enforced: a write tool with auto, or a read tool with anything else, fails with PolicyInvalid. There is no database constraint, so do not edit the rows by SQL. Every change appends policy.updated to the support Workspace's audit chain.

When an engine proposes an action, proposeActions copies the tool's current approval and risk onto agent_action. At decision time decideAction applies the stricter of that snapshot and the current policy, so tightening a policy to four_eyes also covers proposals that are already open.

To change a default for fresh databases, change the write(…) call in TOOL_SEEDS and re-record the seed pre-run: the recorded actions carry their approval.

Add an eval case

Cases are defined in packages/db/src/seed/hasomed/evals.ts: TICKET_EXPECTATIONS (one tuple [ticket number, slug, expected, tags] per open ticket) and commandExpectations(now) (practice commands); buildEvalCases(now) builds the eval_case rows, 18 today. expected is an EvalExpected (harness.enums.ts): category, priority, rootCause, actions, forbiddenActions, intent, params, guardrails. Each field becomes one check (EVAL_CHECKS); actions are compared as sorted lists.

A ticket case needs its ticket and the practice data in the frozen fixture (packages/db/src/seed/hasomed/fixtures.ts, supportFixture), because evals run against fixtureSource, never against live rows, and never execute a write.

After adding a case:

• H/harness.rules.eval.test.ts requires the rules engine to pass every case at two pinned times. Update the expected count (has 18 cases).
• The seed writes the cases (db:seed or db:seed:reset-demo).
• If the rules output for an open ticket changed, re-record the seed pre-run: run H/harness.prerun.test.ts with UPDATE_PRERUN=1 (it rewrites packages/db/src/seed/hasomed/prerun.recorded.json), then format the file with pnpm exec oxfmt --write.

Swap the model or provider

The AI engine goes through the template's provider seam, packages/api/src/platform/ai/:

Locally the values live in apps/web/.dev.vars (gitignored); in a deployment they are Worker variables and secrets. harness.engine.info reports the active provider and model (ai: null without a key), and the ticket page offers "Automatisch" only when it is set.

AI_MAX_STEPS, AI_TIMEOUT_MS and AI_REQUESTS_PER_MINUTE belong to the template assistant; the harness does not read them. Its limits are in code: 25 s per model call (CALL_CAP_MS in H/harness.ai.ts), 45 s per run (aiDeps in H/harness.procedures.ts) and at most 6 tool-loop iterations per call (ai.structured.ts).

Add a price. Cost is computed from MODEL_PRICES in packages/env/src/ai-cost.ts: USD per 1 million input, cached input and output tokens, keyed by model id (a dated snapshot such as gpt-6-luna-2026-08-01 matches its base id). A model that is not in the table is priced with FALLBACK_PRICE (2.50 / 0.25 / 10 USD) and every cost shown for it is marked as estimated ("Modell nicht in der Preistabelle: Kosten geschätzt."). Add an entry with the provider's list price when you change AI_MODEL. Reasoning tokens are billed as output.

Check the budgets. The run budget must leave room for a full ticket run (triage, diagnose with its tool loop, plan, one correction retry). Before every call the meter estimates the call's cost from the prompt (tool-loop calls at four times the prompt's input and twice the output) and refuses it when the run or the day would exceed its budget. With a pricier model, raise AI_BUDGET_PER_RUN or expect budget_exceeded fallbacks.

After a swap, run the AI evals and compare them with the previous model in the trend chart. Then run the real-model smoke test against your dev server: node scripts/demo/harness-ai-smoke.mjs --base http://localhost:<port>. It signs in with the demo logins, runs DEMO-1004 and a practice command with engine "auto", prints the engine and model that answered, and writes demo runs.

Run evals

• UI. /harness/evals: "Evals ausführen" for the rules or the AI engine, results per case and check, history and score trend.
• Procedure. harness.evals.run (support Workspace) with { engine: "rules" | "ai", caseIds? }. ai without a configured key fails with AiNotConfigured. Each case gets 45 s and the run budget (AI_BUDGET_PER_RUN). An AI eval is metered like a run: one spend row (agent_run kind eval, hidden from the runs lists) carries the calls of all cases, counts in today's spend and is refused by AI_BUDGET_PER_DAY. A refused call makes that case fall back to the rules with budget_exceeded in its result.
• Tests. The rules engine is checked on every case in pnpm test (H/harness.rules.eval.test.ts). H/harness.ai-recorded.test.ts replays hand-written gpt-6-luna answers through the fakeText adapter from @tanstack/ai/testing; it does not call a model, and there is no re-record switch. There is no eval CLI script.

Replay a run

On a run's detail page (/harness/laeufe/<runId>, support runs only) the replay panel runs the original input again on current data with the engine you choose ("Nur Regeln" by default, or "Automatisch"). It calls harness.runs.replay with { runId, engine: "auto" | "rules" }, which creates a run of kind replay with replay_of set, then harness.runs.execute. A replay's actions get the status replay and can never be approved; the ticket is not changed. The panel then compares root cause, actions, step count, engine and status with the original.

MCP picks up tools automatically

Nothing in the MCP code names a harness tool. registerHarnessTools (packages/api/src/features/mcp/mcp.harness-catalog.ts) loads the registry (harness.registry.get), keeps the rows of the caller's surface that are enabled, and registers each with its title_de, description_de and the input schema from TOOL_INPUTS. Read tools run through the harness read adapter; write tools only propose an action (harness.external.service.ts) that a person approves in the UI. HARNESS_WRITE_TOOL_NAMES, which decides which calls need data:write, is derived from TOOLS. A new tool is therefore on /api/mcp as soon as its registry row exists. Disabling a tool in the policies removes it from tools/list. Details: MCP and agent connections.