orchestration-benchmark

Orchestration Benchmark (#1095)

Role: orchestrator. This command dispatches benchmark runs and aggregates their results — it never performs the benchmark tasks itself and never edits the code under test.

You have been invoked with the /orchestration-benchmark command.

Not /benchmark. /benchmark measures runtime performance of web pages (Core Web Vitals, resource sizes, load times). This skill measures orchestration value: whether the dev-team pipeline's delegation earns its overhead against a solo session on the same task at the same rigor. Different subject, different instruments, different report.

Orchestrator constraints

1. Matched rigor is non-negotiable. Every arm runs the identical task with identical acceptance checks. An arm may not skip or weaken verification — a cheap worker that reads less and passes weaker checks is a different product, not a cheaper version of the same one. A run that weakened its checks is a rigor violation: discard it, do not average it in.
2. The protocol is pre-registered. The decision rule below is fixed before the first run. Do not adjust thresholds, arms, or the task matrix after seeing results — report against the rule as written.
3. Never report a single run. Report median + spread per arm per task class. A difference smaller than the spread is a null result.
4. Write results to files. Per-run records and the final report go to files; present only the summary tables and verdict in chat.
5. Be concise. Tables and short sentences; no narration of each run.

Parse Arguments

Arguments: $ARGUMENTS

• --task-class <trivial|standard|complex>: Run only one task-matrix class (bucket names from knowledge/task-size-classifier.md). Default: all three.
• --runs <n>: Runs per arm per task class (default: 5). Fewer than 5 makes the result exploratory, not decidable — say so in the report and do not apply the decision rule to it.
• --dry-run: Print the run plan — interleaved schedule, selected fixtures, pinned band→model map, and a cost estimate — without dispatching anything.

Arms (3, matched rigor)

Arm	Configuration
A — Solo	Single session, no subagent dispatch; the session model does all reading and all work itself.
B — Current orchestration	The dev-team pipeline exactly as shipped.
C — Delegation-only sweep	Arm B plus the #1093 sweep rule: the orchestrator never reads beyond classification-sized probes; all bulk reading goes to low-band workers with scoped briefs per the #1092 batching rules.

All three arms run the identical task with identical acceptance checks (the fixture's eval assertions). See constraint 1: unequal rigor invalidates the run.

Task matrix

Sample across the knowledge/task-size-classifier.md buckets (trivial | standard | complex) — one task class per bucket:

Class	Bucket	Example	Expectation
Small single-file	trivial	One-file bug fix from an eval fixture	Expected worst case for delegation
Medium few-file	standard	Feature slice touching 3–5 files	Near the predicted crossover
Large sweep-heavy	complex	/code-review over a 20+ file diff; repo-wide audit	Expected best case for delegation

Reuse existing /agent-eval fixtures under evals/fixtures/ wherever possible; add fixtures only where a class has none.

Repetitions and isolation

• ≥5 runs per arm per task class (see --runs). Report median + spread (min/max or IQR) — never a single run. A difference smaller than the spread is a null result.
• Isolation via /headless-run (skills/headless-run/SKILL.md): each run dispatches through skills/headless-run/scripts/isolated_dispatch.py — fresh session id, clean HOME and config dir, scrubbed env — so runs are independent and no parent-session identity or state carries over.
• Cache parity: interleave arm order — A, B, C, A, B, C, … Never batch all runs of one arm together; batching gives later arms warmer caches.
• Model pinning: record the exact model ID(s) per run. All arms use the same band→model map — capture /model-routing-check output before the first run and include it in the report. If the map changes mid-experiment (ladder edit, routing bump), abort and restart; do not mix runs across maps.

Metrics (instrument named per metric)

Metric	Instrument
Dollar cost & tokens by model/thread	python3 ${CLAUDE_PLUGIN_ROOT}/hooks/lib/cost_meter.py report --transcript <path>
Tokens moved to low band (mechanism check)	The cost meter's by_model / by_thread split from the same report
Quality	The fixture's eval assertions — identical across arms
Rework	Fix-loop iterations; metrics/review-value.jsonl where review checkpoints ran
Wall-clock	Harness transcript timestamps

Mechanism check (mandatory)

Arm C's cost reduction must be produced by the claimed mechanism. If Arm C's cost drops but reading tokens did not shift to low-band subagent threads (the cost meter's by_model/by_thread split shows the movement), treat the result as a rigor violation, not a win — the savings came from reading less, not from delegating the reading.

Decision rule (pre-registered — fixed before the first run)

Apply per task class:

1. Adopt C over B only if all three hold:
   • median cost reduction ≥ 25%;
   • no quality regression — every run passes the fixture's acceptance checks / stays at-or-above its calibration floor (knowledge/calibration-floors.json);
   • the mechanism check passes.
2. Retain B over A only if B beats A on cost or quality. If A dominates B on a class, report it — that finding feeds /harness-audit — do not suppress it.
3. Crossover output: report the task-size threshold below which A/B beats C. That number feeds the #1093 sweep rule.

Threats to validity

Each of these must be explicitly addressed in the report — not silently skipped:

1. Unmatched rigor — an arm passed weaker checks; the comparison is void.
2. Fixed-overhead asymmetry — orchestration pays a constant dispatch tax that dominates small tasks and vanishes on large ones.
3. Cache warmth — runs batched by arm give later arms warmer caches; interleaving (above) is the control, state whether it held.
4. Model drift — the band→model map or an underlying model changed mid-experiment; the abort/restart rule (above) is the control.
5. Fixture overfitting — the crossover threshold generalizes only as far as the task matrix; say so wherever the threshold is quoted.

Report

Write the report to reports/orchestration-benchmark-<date>.md.

For the header block and closing Provenance section, follow knowledge/report-template.md; the sections below are this skill's own body.

Body sections, in order:

1. Per-class results — one table per task class: median + spread per arm for each metric (cost, tokens by band, quality pass rate, rework, wall-clock).
2. Crossover threshold — the task-size threshold below which A/B beats C, with the fixture-overfitting caveat attached.
3. Decision-rule outcomes — per class: adopt/retain verdicts against the pre-registered rule, including the mechanism-check result.
4. Threats to validity — all five items above, each with how this run addressed it.
5. Run log — pinned model map (/model-routing-check output), exact model IDs per run, interleaved schedule as executed, discarded runs and why.

Relationship to other tooling

• /benchmark is runtime-performance-only (web pages); this skill measures orchestration value. They share nothing but the name shape.
• /headless-run is the isolation mechanism for every dispatched run; /agent-eval fixtures supply the tasks and acceptance checks; /cost-report and hooks/lib/cost_meter.py are the cost instruments; /model-routing-check pins the model map; /harness-audit consumes any "A dominates B" finding.
• Instrumentation flip belongs to the run, not to this skill. After the first full documented protocol run (#1099), the instrumentation list in plugins/dev-team/CLAUDE.md moves "efficiency gains" from Not yet to Instrumented, naming this skill as the instrument. Shipping this skill does not perform that flip.