Durable pipelines: hybrid CI/CD that survives restarts

A pipeline is one dependency graph whose jobs run on different execution adapters: a local process tree, a container, a remote Windows runner. Orgabot runs it as part of a mission, and the mission's own lifecycle journal is where every pipeline fact is recorded. There is no separate pipeline database, so anything that can recover a mission can recover its pipeline.

What a pipeline definition holds

Each job names the adapters it may run on (in fallback order), what to run, and optionally:

• needs: the jobs it depends on.
• matrix: axes expanded into one job instance per combination, with include and exclude.
• if: run on success (the default), failure, always, or cancelled, plus conditions on a dependency's outputs.
• outputs: a job appends name=value lines to the file named by $ORGABOT_PIPELINE_OUTPUTS, and later jobs read them as ${{ needs.<job>.outputs.<name> }}.
• timeoutMs, and cleanup: true for a job that runs whenever its dependencies settle, even after a failure or a cancellation.
• effect: pure (verification, the default), or signing, publish, deploy, promotion for jobs with external side effects.

The pipeline also runs under a policy snapshot: which adapters are authorized, whether infrastructure failures may fall back to the next adapter, and which jobs or effects need a human approval. A definition naming an adapter the policy does not authorize is refused before anything runs.

Job states

A job is queued, approval_blocked, resource_blocked, dispatching, running, succeeded, failed, canceled, infrastructure_lost, outcome_unknown, or skipped. Only an observed succeeded is success. A job that was submitted, or whose runner vanished, is never counted as passed.

The mission view and the CI/CD view render the same summary of each pipeline, so a cancelled, failed, or blocked pipeline reads the same in both places.

Restarts and lost webhooks

Orgabot records its intent to dispatch a job before it calls the runner, and gives every attempt a stable operation id. After a restart, the next controller first reconciles every job the journal says is in flight: it reads the run back by its handle, or asks the provider for the operation id when the handle was never recorded. Only then does it start anything new. A job that already succeeded is never rebuilt.

Provider callbacks are signed (HMAC-SHA256 over the timestamp and body), must be fresh, and must name the operation id of the attempt they describe. A duplicate, late, or out-of-order callback is recorded as evidence but never moves a job backwards. A lost callback costs one poll interval, because the controller observes runs itself.

Two controllers at once

A controller must hold the pipeline's lease to act. Taking the lease over bumps an epoch, and the journal refuses any write from an older epoch. Because dispatch intent is written first and the journal admits exactly one dispatch per attempt, two controllers racing on the same job cannot both reach the runner. This is what keeps a promotion from happening twice.

Retries and fallback

• An assertion failure (the job ran and its checks failed) or a job timeout is final. It is never retried and never moved to another runner.
• An infrastructure failure (the runner refused the job, or lost it) may retry on the same adapter, then move down the job's authorized adapter list when the policy allows fallback.
• An unknown outcome on a pure job is simply run again. On a signing, publish, deploy, or promotion job it stays outcome_unknown and blocks the pipeline until it is reconciled, either by a provider lookup that proves the operation never arrived, or by an operator recording evidence:

orgabot pipeline reconcile <mission-id> <pipeline-id> <job-key> <attempt> --not-happened --evidence "<what you checked>"

Every attempt of a job passes the same idempotency key to the provider, so a provider that deduplicates on it refuses a second side effect even across attempts. Orgabot does not claim exactly-once delivery across arbitrary providers; it claims that it never retries a side effect it cannot account for.

Cancelling and draining

orgabot pipeline cancel records the request and propagates it: running local process trees and containers are terminated as a whole, remote runs are cancelled through their provider, jobs that had not started are marked cancelled, and jobs that already completed keep their evidence. A mission that ends cancels its pipelines the same way.

orgabot pipeline drain is for upgrades and shutdowns. It stops starting new work, waits for local jobs up to a deadline, and releases the lease so a successor can take over at once. With --stop-local it stops still-running local verification so the successor reruns it. It never kills a side-effecting job midway, since that would create exactly the unknown outcome the rules above exist to avoid.

Commands

orgabot pipeline list [--mission <id>] [--json]
orgabot pipeline status <mission-id> [<pipeline-id>] [--json]
orgabot pipeline run <mission-id> --definition <file.json> --policy <file.json> --sha <sha> [--callback-key-id <id>]
orgabot pipeline resume <mission-id> <pipeline-id>
orgabot pipeline cancel <mission-id> <pipeline-id> [--reason <text>]
orgabot pipeline drain <mission-id> <pipeline-id> [--wait-ms <n>] [--stop-local]
orgabot pipeline reconcile <mission-id> <pipeline-id> <job-key> <attempt> (--happened | --not-happened) --evidence <text>
orgabot pipeline callback <mission-id> --key-id <id> --timestamp <ms> --signature <hex> --body-file <file>

Adapters

• local runs a job as a process tree on this host, or inside a container when the job names a container image. Cancelling it stops the whole process group and kills the container by name. A local job does not inherit Orgabot's environment: it sees only basic variables such as PATH, HOME and the locale, plus its with values, so no callback secret or token reaches it. An upstream output written into a local job's run is shell-quoted, so it is always one literal word and never runs as a command. To use an output inside a larger string, pass it through with and read it as an environment variable.
• github-actions runs a job on a GitHub Actions runner, including Windows runners. The job's run names a workflow_dispatch workflow as <owner>/<repo>:<workflow-file>[@<ref>], and its with values become the workflow's inputs. Orgabot always adds orgabot_operation_id and orgabot_idempotency_key inputs, and the workflow must put the operation id in its run-name, for example run-name: "orgabot pipeline ${{ inputs.orgabot_operation_id }}". GitHub does not return a run id when a workflow is dispatched, so the run-name is how Orgabot finds the run again after a lost reply or a restart. GitHub's run list can lag, so a run Orgabot cannot find yet does not prove the dispatch never happened: a pure job is simply run again, and a side-effecting job stays blocked until the run appears or you reconcile it. Calls go through the GitHub App connected for that repository. Cancelling a pipeline cancels the run on GitHub. GitHub does not expose workflow outputs, so a remote job reports outputs with a signed callback, and its run artifacts are recorded as references.

Registering an adapter authorizes nothing. A pipeline may use only the adapters its policy snapshot lists.

Callbacks over HTTP

A remote runner can report a job's state and outputs by sending a POST to /webhooks/pipelines on the dashboard server, with the headers x-orgabot-pipeline-key-id, x-orgabot-pipeline-timestamp (milliseconds) and x-orgabot-pipeline-signature (hex HMAC-SHA256 of <timestamp>.<body>). The body is the same JSON that orgabot pipeline callback accepts. The signature is the only credential, so this route does not need the dashboard token. A callback signed with an unknown key, or one that is stale or forged, is refused before any mission is looked up. Each key's secret is read from ORGABOT_PIPELINE_CALLBACK_SECRET_<KEY_ID> (the key id in upper case, with every other character turned into _) and must be at least 16 characters. With no secret set, every callback is refused. A pipeline accepts callbacks only when it was started with --callback-key-id, and only signed with that key. A pipeline started without one accepts no callbacks at all and settles by polling, so the key one runner holds can never report on another runner's job.