Quality Gate Strategy

A useful quality-gate contract is not a list of everything CI can run. It is a portfolio of independently scheduled proofs.

Each proof should answer five questions:

What does it prove?
  → when is that proof relevant?
  → how much does the full execution boundary cost?
  → can a coding agent normally repair its failure?
  → at which lifecycle boundary must the proof be fresh?

That sequence matters. Cost informs the design, but never decides policy by itself.

The working model

Proof shape	Selection	Schedule	Typical consequence
Fast, universal, source-repairable	always	after_phase(implement)	require + repair_loop
Moderate, subsystem-specific, source-repairable	paths or explicit task_kind	after_phase(implement)	require + repair_loop
Slow, release-relevant integration proof	paths, task_kind, or always when truly universal	before_delivery	require + handoff
Rare or operational proof	operator	manual_only	suggest or manual
Unmeasured or unpredictable proof	operator while calibrating	manual_only	no automatic execution

These are strong defaults, not hidden engine behavior. A slow universal test can still be mandatory after implementation. A fast diagnostic can still be advisory. The contract author makes that decision explicitly.

Start with executable proof, not tool names

Inventory the repository’s native commands before writing gate sets. Record the exact argv, working directory, environment, services, and result each command proves.

The useful unit is the smallest command that can be:

• selected independently;
• scheduled independently;
• failed and repaired independently;
• recorded in one immutable receipt;
• reused as fresh evidence without rerunning unrelated work.

Avoid hiding different proof costs behind one wrapper:

make qa
├── formatter check              5s
├── unit tests                  80s
├── full static analysis       4m
└── database integration       11m

If those are stable native commands, expose four command identities. Keep the wrapper only when its setup and outcome are genuinely atomic.

Measure the whole boundary

Use representative p90 wall time from the environment Orcho will execute. Include:

• dependency or worktree bootstrap that occurs for the gate;
• container and service startup;
• migrations and fixture loading;
• the check itself;
• cleanup or teardown required to settle the environment.

A useful initial calibration is:

Cost	Starting band
fast	usually under 30 seconds
moderate	roughly 30 seconds to 3 minutes
slow	usually over 3 minutes
unknown	not measured or too variable to predict

These bands are team guidance, not Orcho constants. Reclassify from evidence when the command or its environment changes.

Select by relevance, not by cost

Selection answers whether a proof matters for this run:

• always — every change can violate the invariant;
• paths — only specific subsystems or contract files can violate it;
• task_kind — an explicitly declared work class activates it;
• operator — a person or control client deliberately requests it.

Do not use paths merely to make a slow universal check disappear. Conversely, do not select a subsystem suite always merely because it is fast.

Schedule by feedback and transition

Use after_phase(implement) when a failure is actionable source feedback and a repair round can reasonably fix it. This is the only hook where repair_loop drives the real implement → repair transition.

Use before_delivery when the proof must be current at the release boundary, but failure should pause for judgment rather than automatically send a coding agent through another expensive loop.

Use manual_only for credentialed, destructive, rare, or still-unmeasured proof. Use on_resume only for assumptions that may have changed while the run was paused.

Avoid scheduling the same command at several hooks just to make it visible. Fresh receipts are reusable. Add another scheduled identity only when the later boundary requires a genuinely distinct proof.

Pattern 1: Python engine with a broad correctness floor

This pattern is distilled from a Python orchestration engine. Environment provenance and lint are fast. Domain-focused unit slices are moderate and selected by path. A broad non-e2e suite is slow but intentionally universal because changes to orchestration state have a large blast radius. End-to-end tests remain operator-owned.

The real project configuration pairs env-provenance with a pinned verification_envs interpreter and import-path assertions for the current checkout. The shortened command below records the imported paths; import success by itself is not provenance.

"commands": {
    "env-provenance": {
        "run": [
            "python", "-c",
            "import pipeline, sdk; print(pipeline.__file__, sdk.__file__)",
        ],
        "cost": "fast",
    },
    "lint": {
        "run": ["python", "-m", "ruff", "check", "."],
        "cost": "fast",
    },
    "verification-unit": {
        "run": [
            "python", "-m", "pytest", "-q",
            "tests/unit/pipeline/verification",
        ],
        "cost": "moderate",
    },
    "broad-non-e2e": {
        "run": [
            "python", "-m", "pytest", "-q",
            "-m", "not e2e and not packaging",
        ],
        "cost": "slow",
    },
    "e2e": {
        "run": ["python", "-m", "pytest", "-q", "-m", "e2e"],
        "cost": "slow",
    },
},
"gate_sets": {
    "provenance": {
        "commands": ["env-provenance"],
        "default_policy": "require",
    },
    "hygiene": {
        "commands": ["lint"],
        "default_policy": "require",
    },
    "verification": {
        "commands": ["verification-unit"],
        "default_policy": "require",
    },
    "broad": {
        "commands": ["broad-non-e2e"],
        "default_policy": "require",
    },
    "e2e": {
        "commands": ["e2e"],
        "default_policy": "suggest",
    },
},
"selection": [
    {"always": ["provenance", "hygiene", "broad"]},
    {
        "paths": [
            "pipeline/verification/**",
            "tests/unit/pipeline/verification/**",
        ],
        "include": ["verification"],
    },
    {"operator": ["e2e"]},
],
"schedule": [
    {
        "after_phase": "implement",
        "gate_sets": ["provenance"],
        "policy": "require",
        "action": "handoff",
    },
    {
        "after_phase": "implement",
        "gate_sets": ["hygiene", "verification", "broad"],
        "policy": "require",
        "action": "repair_loop",
    },
    {
        "manual_only": True,
        "gate_sets": ["e2e"],
        "policy": "suggest",
    },
],

The slow suite is not weakened to warn. Its cost is accepted because the proof is load-bearing. The rare e2e proof is separated so it cannot silently inflate every run.

Pattern 2: PHP backend with containers and a test database

This pattern is distilled from a modular PHP backend. Diff-scoped formatting, Rector, and static analysis give early feedback without starting the database. Full-project analysis and DB-bound suites are separate identities because their setup and failure shapes are different.

"commands": {
    "cs-diff": {
        "run": ["make", "cs-diff"],
        "cost": "fast",
    },
    "rector-diff": {
        "run": ["make", "rector-diff"],
        "cost": "fast",
    },
    "phpstan-diff": {
        "run": ["make", "phpstan-diff"],
        "cost": "moderate",
    },
    "test-unit": {
        "run": ["make", "test-unit"],
        "cost": "moderate",
    },
    "static-full": {
        "run": ["make", "qa-static"],
        "cost": "slow",
    },
    "test-db": {
        "run": ["make", "orcho-test-db"],
        "cost": "slow",
    },
},
"gate_sets": {
    "smoke": {
        "commands": ["cs-diff", "rector-diff", "test-unit"],
        "default_policy": "require",
    },
    "static-diff": {
        "commands": ["phpstan-diff"],
        "default_policy": "warn",
    },
    "static-full": {
        "commands": ["static-full"],
        "default_policy": "warn",
    },
    "database-suite": {
        "commands": ["test-db"],
        "default_policy": "warn",
    },
},
"selection": [
    {"always": ["smoke", "static-diff", "static-full"]},
    {
        "paths": [
            "src/**",
            "tests/Integration/**",
            "tests/Functional/**",
            "migrations/**",
        ],
        "include": ["database-suite"],
    },
],
"schedule": [
    {
        "after_phase": "implement",
        "gate_sets": ["smoke"],
        "action": "repair_loop",
    },
    {
        "after_phase": "implement",
        "gate_sets": ["static-diff"],
        "policy": "warn",
    },
    {
        "before_delivery": True,
        "gate_sets": ["static-full", "database-suite"],
        "policy": "warn",
    },
],

The important boundary is not “PHP versus Docker.” It is source-only feedback versus infrastructure-backed proof. Keeping them separate prevents a style failure from paying database startup cost and prevents an infrastructure failure from masquerading as a code-repair loop. This dogfood backend deliberately keeps full static analysis and DB proof as shipping-allowed warnings because they are memory-heavy or environment-bound; independent CI remains the downstream blocker. A project can promote either set to require + handoff once its environment is reliable enough to make that proof authoritative.

Pattern 3: TypeScript application with browser verification

This pattern is distilled from a TypeScript application that has both Node-based checks and Chromium-backed conformance tests. Typecheck and unit tests run after implementation. Browser proof runs at delivery, where a failure pauses instead of automatically repeating an expensive browser setup.

"commands": {
    "typecheck": {
        "run": ["npx", "tsc", "--noEmit"],
        "cost": "fast",
    },
    "vitest": {
        "run": ["npm", "run", "test:unit", "--", "--run"],
        "cost": "moderate",
    },
    "browser-e2e": {
        "run": ["python3", "-m", "pytest", "-q", "engine/verify/tests"],
        "cost": "slow",
    },
    "browser-conformance": {
        "run": [
            "python3", "-m", "pytest", "-q",
            "engine/tooling/conformance/tests",
        ],
        "cost": "slow",
    },
},
"gate_sets": {
    "smoke": {
        "commands": ["typecheck", "vitest"],
        "default_policy": "require",
    },
    "browser": {
        "commands": ["browser-e2e", "browser-conformance"],
        "default_policy": "require",
    },
},
"selection": [
    {"always": ["smoke", "browser"]},
],
"schedule": [
    {
        "after_phase": "implement",
        "gate_sets": ["smoke"],
        "action": "repair_loop",
    },
    {
        "before_delivery": True,
        "gate_sets": ["smoke", "browser"],
        "action": "handoff",
    },
],

Here browser proof is universal, so always is honest even though it is slow. Moving it to before_delivery changes when the cost is incurred and how failure is routed; it does not pretend the proof is irrelevant. Re-scheduling smoke creates a release-boundary freshness identity; when the verification subject is unchanged, Orcho reuses the fresh receipt instead of executing the command again.

What the three stacks teach

Decision	Python engine	PHP backend	TypeScript + browser
Cheapest feedback	provenance + Ruff	diff-scoped style and transforms	typecheck
Moderate feedback	domain unit slices	unit + diff static analysis	Vitest
Slow proof	broad non-e2e suite	full static + DB suite	browser e2e + conformance
Why slow work runs	universal high-blast-radius floor	full static is universal; DB proof is path-selected	universal release proof
Failure transition	repair for code proof; manual e2e	repair early; warn at the infrastructure boundary	repair smoke, handoff browser proof

The stack does not choose the design. The proof’s relevance, cost, repairability, and required freshness do.

Review the effective contract

Before the first real run:

orcho quality-gates --project .

Check every row:

1. Does the command identity represent one useful proof?
2. Is cost based on observed full-boundary duration?
3. Does selection match semantic relevance?
4. Is the hook the earliest useful feedback boundary?
5. Can the declared action really resolve this failure?
6. Is require, warn, suggest, or manual justified by risk rather than runtime cost?

Then run one small dogfood task that deliberately breaks a fast repairable gate. Confirm the engine records fail → repair → pass with immutable receipts. Run slow or external proof only when the test is safe and its environment is ready.

Anti-patterns

• one make qa identity for unrelated costs and failure shapes;
• every gate selected always;
• every expensive gate weakened to warn;
• full suites copied into implement acceptance criteria;
• the same command scheduled at several hooks without a distinct freshness requirement;
• DB, browser, network, or credential failures routed into blind code repair;
• unknown assigned a guessed cost so configuration looks complete;
• CI treated as proof for the current isolated run without a run receipt.

Deep reference

The canonical engineering docs live with the code:

• Scheduled verification guide
• Verification contract architecture
• plugin.py field reference
• Task-file authoring and verification ownership

Next

• Scheduled verification setup covers the complete field-level authoring path.
• Project tuning and plugins shows how an agent can draft a repository contract for engineer approval.
• Verification receipts explains the durable proof produced by each scheduled identity.