agents-shipgate

Agents Shipgate

Your coding agent changed what your AI agent can do — Agents Shipgate tells you whether it can merge.

The deterministic merge gate for AI-generated agent capability changes.

Local-first and static by default — no agent execution, tool calls, LLM calls, or network access.

[!IMPORTANT] Status: pre-1.0 (beta). The decision engine is deterministic and stable. This source tree is 0.16.0b6; install and GitHub Action examples remain pinned to the latest published tag, v0.15.0, until the beta is released. Real-history accuracy numbers (small n, published in full in benchmark/miner/README.md): across 361 mined rows from 8 distinct real agent repos, 336 (93%) organically skip the trigger; of the 19 unique labeled engine-engaged PRs, re-run on the released v0.15.0 engine (2026-W27-reeval), the gate never auto-passed an unsafe change — both real must_block PRs and every needs_human PR are held for human review (must_block_caught / needs_human_caught = 1.0). But it routes to review, it does not block: real-history blocked_recall is still 0.0 — both must_block PRs return human_review_required / review_required, not blocked. v0.15.0 cleared the 4 scan crashes and moved both must_block PRs off abstention (insufficient_evidence → review), at the cost of escalating 4 of 14 safe PRs to review (benign_escalation_rate 0.286 — largely a cold-start whole-repo-surface measurement artifact on large repos). Reliable blocked-recall (1.0) and zero benign escalation are proven on the constructed-adversarial stratum, not yet on real history. Labels are AI-adjudicated (0 disagreement), pending human spot-check. Treat it as an advisory gate while this work closes — see ROADMAP.md.

60 seconds: watch it block two PRs

Claude Code adds stripe.create_refund to your support agent and opens a PR. The diff looks fine to a human skimming it. Should it merge?

uvx agents-shipgate fixture run ai_generated_refund_pr

→ merge_verdict: blocked — the new refund capability has no declared approval policy and no idempotency evidence. The verifier explains both blockers and routes the PR to a human.

Now the move every reviewer fears — the agent deletes the Shipgate CI gate to make its PR pass:

uvx agents-shipgate fixture run agent_weakens_gate

→ merge_verdict: blocked, can_merge_without_human: false. The gate-removal checks are suppression-immune: the cheapest reward-hack is also the most visible one.

…and here's the failure mode. These two cases are constructed fixtures with a clear-cut answer, chosen to show the gate working. Real PRs are messier: when a change builds its tool surface dynamically — a toolkit factory, a config-bound allowlist, tools assembled at runtime — static extraction often can't enumerate the result, and Shipgate returns insufficient_evidence and routes to a human rather than emit a confident wrong verdict. That is the intended failure mode, not a bug; reducing how often it fires on real dynamic code is active work (see ROADMAP.md).

One engine decides (report.json.release_decision.decision); everything else — merge_verdict, PR comments, Check Runs, Action outputs — is a deterministic projection of it. Five-minute version: docs/mental-model.md.

Agents Shipgate is an open-source CLI and GitHub Action for local-first, static Tool-Use Readiness review. It scans MCP, OpenAPI, OpenAI Agents SDK, Anthropic Messages API, Google ADK, LangChain/LangGraph, CrewAI, OpenAI API, Codex repo config, Codex plugin, n8n, and Conductor OSS workflow artifacts, then writes a deterministic Tool-Use Readiness Report before your agent gets production-like permissions.