optimize-loop

Optimize Loop

An evaluator-optimizer loop with a pluggable correctness gate + minimized metric. The artifact is some editable thing (a code module or a SQL query); the feedback signal is two-part: a bound gate that must pass (behaviour/results unchanged) and a bound metric that must drop (the cost you minimize). You apply one change, check the gate, measure the metric, and keep the change only if the gate passes AND the metric improves — otherwise you revert. Repeat until the metric stops improving or the budget runs out. Once the loop starts, do not pause for permission.

Two ready bindings ship in tools/ (both vendored, stdlib-only):

• code mode — gate: <gate_cmd> (the test suite) exits 0; metric: tools/metrics.py prints complexity (primary), max_nesting, loc (lexicographic tie-breakers). Lower is better.
• sql mode — gate: the result-set hash from tools/bench.py matches the baseline; metric: the same tool's median_ms. Lower is better.

The gate is non-negotiable in both modes: a change that fails it is a regression, not an improvement. Never edit the ground truth (the tests / tools/metrics.py in code mode, the database / tools/bench.py in sql mode) — editing what measures you to move the number defeats the loop.

When to use

Use when there is a clear correctness bound to hold and a number to minimize: refactoring code that has a passing test suite (cut complexity), or tuning a SQL query that has a fixed result-set (cut latency). The default is the matching shipped tool; the escape hatch is to bind any <gate_cmd> that exits 0 on pass and any <metric_cmd> that prints a single number to minimize (e.g. a linter's issue count, or a non-SQLite engine's timing + result fingerprint). Not for adding features or fixing bugs — those intend to change behaviour, which this loop is built to forbid.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values back in one line, and skip to the loop. Otherwise pick <mode> first (it selects the gate + metric), then on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml and confirm the values before creating any other files.

<gate_cmd> and <metric_cmd> are the pluggable core: bind them per <mode> from the table. In sql mode one bench command supplies both — its hash is the gate, its median_ms is the metric.

binding	meaning	default	how to infer
<mode>	code (refactor under test) or sql (query, fixed results)	—	the artifact's kind
<editable_files>	the file(s) the loop may change	—	code: source files (not tests/configs/the tool); sql: the query file (+ optional indexes file)
<gate_cmd>	the bound gate that must PASS, else revert	—	code: the test command (exits 0 on pass); sql: implicit — candidate hash must equal the baseline hash from the bench command
<metric_cmd>	the bound metric printing a number to minimize	—	code: python3 <skill_dir>/tools/metrics.py <editable_files> (→ complexity, then max_nesting, loc); sql: python3 <skill_dir>/tools/bench.py --db <db> --query <query_file> --setup <indexes_file> --repeat 5 (→ median_ms, hash)
<sandbox_root>	where snapshots + the ledger live	./sandbox	—
<budget>	max iterations (hard cap)	8	—
<patience>	stop after N consecutive no-improvement iterations	3	—

<skill_dir> is this skill's installed folder; substitute the real path when writing loop.run.yaml. For non-default engines/languages, bind any <gate_cmd>/<metric_cmd> meeting the contract above. Two worked configs: examples/refactor.run.yaml (code) and examples/sql.run.yaml (sql).

The loop (until plateau or <budget>)

Copy this checklist and tick items off:

• Iteration 0 — baseline: run <gate_cmd> (code) — if not green, stop (the loop needs a passing gate to protect behaviour). Run <metric_cmd>; record the metric as the current best, and in sql mode record the baseline hash as the correctness reference. Log the baseline row.
• Snapshot every file in <editable_files> to <sandbox_root>/iter<N>/ so the iteration reverts.
• Apply one focused change (see the per-mode ideas below) — one idea per iteration so each metric delta is attributable.
• Check the gate: code — run <gate_cmd>; sql — read the candidate's hash from <metric_cmd>.
• If the gate fails (tests non-zero / hash ≠ baseline / the tool errored), discard: restore from the snapshot, log the reason, continue.
• Measure the metric and compare to the best: code — the triple (complexity, max_nesting, loc) lexicographically (complexity first; only on a tie consult max_nesting, then loc); sql — median_ms, keeping only on a margin clear of timing noise (default ≥ 3% relative).
• Keep if the metric strictly improves the best (update the best, leave the files in place), else discard (restore from the snapshot).
• Append a ledger row; stop on plateau (<patience>) or <budget>.

Change ideas — code mode: flatten nested if/else into guard clauses, replace a hand-rolled loop with a stdlib call (sum, min, max, statistics.*), collapse duplicated branches, remove dead code. Preserve public behaviour — names, signatures, return shapes, raised exceptions; the test suite is the contract.

Change ideas — sql mode: add an index to <indexes_file> covering filtered/joined/grouped columns; rewrite the query (correlated subquery → JOIN + GROUP BY, hoist a repeated computation, replace SELECT * with needed columns, push a filter earlier, drop a redundant DISTINCT/sort). The hash is over the multiset of rows, so it does not catch a changed row order — if ORDER BY is part of the contract, eyeball that the rewrite preserves it.

Lexicographic keep (code mode), current best (18, 3, 64): (15, 3, 45) keep (lower complexity); (18, 2, 70) keep (tie complexity, lower nesting); (18, 3, 61) keep (tie, fewer lines); (18, 3, 64) discard (no progress); (19, 1, 20) discard (higher complexity outweighs simpler nesting/loc).

Plateau counting: increment the no-improvement counter on every iteration that does not set a new best — discarded for a failed gate, a broken change, or an insufficient metric gain — and reset it to 0 on each keep. <patience> fruitless iterations in a row ends the run; <budget> is the hard cap. On stop, restore the working files to the best iteration (if the latest was a discard) and report: baseline vs best metric (and, in sql mode, the speedup factor), the trajectory, and the winning change set. If you run low on ideas before the budget, look harder rather than stopping early.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the description. status ∈ {keep, discard, baseline}. Use the columns for the active <mode>.

Code mode header iter complexity max_nesting loc status description:

iter	complexity	max_nesting	loc	status	description
0	23	7	86	baseline	unmodified module
1	19	5	78	keep	flatten summarize guard clauses
2	19	5	80	discard	extract helper (no complexity gain)
3	13	3	40	keep	use statistics + min/max/median

SQL mode header iter median_ms rows hash_ok status description (hash_ok ∈ {yes,no,-}):

iter	median_ms	rows	hash_ok	status	description
0	1121.06	10	-	baseline	correlated subquery no index
1	6.82	10	yes	keep	rewrite correlated subquery as JOIN + GROUP BY
2	1.18	10	yes	keep	add index orders(customer_id, amount)
4	0.40	10	no	discard	drop ORDER BY — changed result set

Report the best iteration, not necessarily the last.

Constraints

• Only edit files in <editable_files> — the gate's ground truth is read-only: the tests and tools/metrics.py (code), the database and tools/bench.py (sql). Editing what measures you to move the number invalidates the run.
• The correctness gate is non-negotiable. A change that fails it (tests red, or hash ≠ baseline) is a regression, not an optimization — revert it regardless of the metric. A green gate after a behaviour change means the gate is too weak, not that the change is safe; prefer holding behaviour identical over trusting a thin gate.
• One change per iteration, so each metric delta is attributable.
• Measure every candidate the same way (code: the same <gate_cmd>; sql: the same --repeat); compare the metric, not a single noisy run.
• Keep changes within the existing dependency set; do not add imports the project lacks (stdlib is fine).
• The sandbox is self-contained — no ../ escapes beyond the bound <sandbox_root>.
• Do not pause the loop to ask whether to continue; run until plateau or budget.