Zum Inhalt springen
Docs

Befehlspalette

Zu einer Seite oder Ansicht springen oder die Dokumentation durchsuchen.

Änderungen

Harness view and evals

The /harness pages: registry with editable approval policies, run history, run detail with replay diff, and evals with a score trend.

Markdown

The harness is the part of the demo a support lead takes over: which tools the agent may use, under which approval policy, what each run did, and how both engines score on frozen cases. Every page reads through the harness and admin.harness procedures; nothing is hard-coded in components. Only the support Workspace reaches these routes (_auth/harness/route.tsx).

Routes

RoutePageShows
/harness?tab=werkzeuge|skills|kontext|richtlinienharness.registry.page.tsxCounts, a tablist (arrow keys move between tabs), tools table with a sheet (description, context sources, skills, input JSON Schema), skill cards, context sources with PII level, approval policies
/harness/laeufe?surface=&engine=&status=&seite=&mandant=harness.runs.page.tsxRun history with engine and fallback reason, status, steps, duration and start; filters and pagination live in the URL
/harness/laeufe/$runIdharness.run.page.tsxInput, engine per phase, findings with evidence links, full trace with raw JSON per step, actions with previews, audit timeline, replay with engine choice and a diff to the original
/harness/evals?lauf=harness.evals.page.tsx"Evals ausführen" for rules and AI, progress, results matrix (case × check), run history as an SVG score trend plus a table, the eval cases with their expectations

Policies

Policy edits go through admin.harness.policies.update, a platform-admin procedure. The page enables editing only for a Workspace lead who is also platform admin (workspace.me(): isLead && isPlatformAdmin); everyone else sees the same table read-only with the reason. Write tools offer only confirm and four_eyes (auto is for reads). Every change is confirmed in a dialog, then the newest policy.updated audit event (sequence, short hash, actor, time) is shown next to the table.

Engine per phase

harness.run-model.ts reads the engine of each phase from the trace: a model step means the AI answered (the model id comes from the step title), a blocked guardrail step means the AI output was dropped and the rules engine re-ran the phase, anything else is the rules engine. The run header shows this next to the run's overall EngineBadge and its fallback reason.

Replay

Replay runs the original input again on current data, read-only (harness.runs.replay, then execute); its actions get status replay and can never be executed. The panel compares root cause, actions, step count, engine and status and says whether the original cause is gone.

Evals chart

harness.evals-trend.tsx is hand-written SVG (role="img", <title>, <desc> listing every point): score per completed eval run, oldest left, one line per engine. Rules are circles and AI squares, so the engine is not told by colour alone. The history table below it carries the same data and links each run to its results.

Shared agent components

apps/web/src/platform/agent/ holds the presentational pieces the support console and the Elefant assistant reuse: AgentTrace (optional phaseAside and highlightSeqs), AgentTraceStep, EngineBadge (optional stacked), ActionCard, DiffView, AuditTimeline, DomainChip and the chip specs in agent.labels.ts (categories, priorities, statuses, approval, risk, surface, run kind, PII, context kind, step kind and status; label keys for root causes, guardrails and eval checks).

Known gaps

  • RunSummary carries no "started by"; the detail page reads it from the run.started audit event, the list does not show it.
  • There is no organization list for platform admins; the Mandant filter lists the tenants seen in the newest 100 runs (admin.harness.runs.list), and that procedure filters by tenant and surface only.
  • RunDetail has no usage field, so token usage is not shown.
Auf GitHub bearbeitenDie Dokumentation ist auf Englisch.