CI/CD conformance: adapter, security and parity tests

Orgabot's CI/CD runtime places each pipeline target on an execution backend: local Docker, a native macOS/Linux/Windows worker, local GitHub Actions compatibility (act), or GitHub Actions. The conformance kit holds every backend, the policy resolver and every migration to the same rules, so a green result means the same thing wherever the work ran.

The kit lives in framework/src/cicd/conformance/. Run it with:

orgabot ci-conformance run

It exits 0 only when every suite passes and every adapter report is complete.

The three rules

Every case in the kit enforces these:

• Failures stay failures. A non-zero exit, a timeout, a cancellation or a refused capability is never reported as success, and a late observation never changes a finished run. A test failure is never retried, and never sent to another provider in the hope that it passes there.
• Uncertain stays uncertain. A run or side effect whose outcome could not be observed is unknown. It holds the pipeline until a provider lookup answers, and it is never retried blind.
• No manufactured green. A success needs observed exit evidence for a pinned commit. A required check counts only from the expected reporter, for the exact commit, from a trusted workload.

Adapter conformance

Each adapter binds to one small port (ConformanceSubject in port.ts), and the same cases run against all of them:

Running it on real processes

orgabot ci-conformance run --binding native

--binding native runs the same adapter cases against this machine's real processes instead of a simulated backend. Each job is checked out from a real git repository at its pinned commit, each step runs in its own /bin/sh process group, timeouts and cancellation send real signals, and the infrastructure-fault case kills a real step with SIGKILL from outside Orgabot. Jobs get a clean environment, so nothing from your shell (including credentials) reaches them. This binding declares no emulation and no service containers, because it cannot verify either. It is the baseline for the native family. The native worker adapter (placement, worker leases, signing) binds to the same port when it lands.

The suite is proven to discriminate: each of the fourteen built-in mutants (an adapter that reports success on a non-zero exit, ignores cancellation, skips redaction, accepts a branch name, and so on) must fail it. A report also records complete: a binding that cannot inject faults gets not_exercised for those cases, which is never a pass.

Policy resolution

The policy suite checks organization, project and target resolution: inheritance is visible field by field, a narrower override can never weaken an organization restriction, a budget ceiling or an approval, an unreadable policy blocks, a project that still runs its own workflows resolves to them unchanged, and the CLI, dashboard, dashboard terminal, agent skills and mission verification all get the identical decision.

Adversarial cases

Failure recovery and scenarios

Faults injected and checked: worker restart, controller restart, durable runtime restart, Docker loss, webhook loss, duplicate webhooks, provider token expiry, rate limits, disk exhaustion, memory pressure, a remote dispatch accepted with its response lost, and unknown signing or deployment outcomes.

Scenarios checked: a clean PR, a failing PR, a fork PR, a merge queue (evidence for the PR head never counts for the merge commit), a tag release, a scheduled run with the Mac offline (blocked, not skipped), and release recovery after a crash around publication.

The hybrid release follows the Premail reference flow: local verification and macOS/Linux builds, a GitHub-hosted Windows installer build with its existing signing identity, and one release join. The join needs an approval bound to the exact commit and artifact digests, verifies every artifact's bytes and architecture, and publishes once. Cancellation stops the hosted run and publishes nothing; rollback reverts exactly once, across restarts. The hosted runner is dispatched only after local verification passes, so it is never billed while waiting on local work.

Mixed mission and CI load runs through the same admission rule missions use, and must stay inside the host's memory reserve, CPU and concurrency ceiling while every item still completes.

Migration parity and cutover

Before a target moves from its existing pipeline to Orgabot, the migration attaches parity evidence:

orgabot ci-conformance run --family docker --report-out report.json
orgabot ci-conformance parity --migration mig-site --project site --target site \
  --old-coverage old.json --new-coverage new.json \
  --old-artifacts old/ --new-artifacts new/ \
  --rollback rollback.json --conformance report.json --out evidence/parity.json
orgabot ci-conformance verify-evidence evidence/parity.json

Cutover is blocked unless all of these hold:

• Every required check still runs on every OS and architecture it ran on, still required. A renamed check needs an explicit entry in --renames.
• Artifacts are equivalent. Plain files compare by content. Archives compare entry by entry, ignoring timestamps. Signed or notarized binaries compare by code with the signature excluded, and both signatures must verify with the same identity and keep notarization. Re-signing alone is recorded, not treated as a regression.
• The target's adapter has a complete conformance report attached.
• Parity ran twice with identical results.
• Rollback restores the old triggers, was rehearsed, saw the old checks run again, and performed no production side effect.

Signatures are checked with codesign and xcrun stapler on macOS and osslsigncode for Windows binaries. If a tool is missing, that artifact is unverified, which blocks cutover. The evidence file is digest-stamped, so verify-evidence refuses a copy edited after it was written.

Test cost

No tier may publish, deploy, submit to a store, notify, or sign with a production identity.

What was exercised versus simulated

Only the adapter suite's native-process binding runs on real hardware today. Everything else in the kit runs on simulated backends:

EXERCISE_LEDGER in costBounds.ts is the source of this table, and a test fails if a new scenario is added without an entry. Update it only after a real run, and record where that run's evidence lives.