Troubleshooting stuck missions, holds, and errors

Troubleshooting a stuck mission

Start with these three before anything else:

orgabot status              # what is running, held, or waiting right now
orgabot logs <mission-id>   # what that mission actually did
orgabot doctor              # host-level health: stale invocations, wedged state

Between them they cover "what's happening," "why," and "is Orgabot itself healthy." The sections below go deeper on each failure mode.

A mission is held

A hold is Orgabot refusing to guess. Find out which kind:

orgabot status
orgabot logs <mission-id>
orgabot session <mission-id> --json

A held (waiting) mission cannot be cancelled with orgabot cancel; cancel only ever ends a running mission. Aiming it at a held one refuses (exit code 2) and names the verb that actually applies: orgabot stop <mission-id> to end it, or orgabot mission resume <mission-id> to continue it, along with the hold's own code, reason, and remedy so you know which one to pick without a separate orgabot status round trip. orgabot stop on a held mission ends it the same way and prints the same hold detail, since ending it also clears the hold.

A mission also cannot be rescoped. The project is fixed at launch, so a follow-up that says "actually do this on the other repo" will hold forever. Relaunch on the right project instead.

A mission looks alive but is going nowhere

Heartbeat alone is not progress. The real signal is the last worker event time. Two common causes:

• A git hook in the target repo is hanging. A husky or similar pre-commit hook that never returns stalls Orgabot's own git commit forever, while the heartbeat keeps ticking.
• The process is wedged in delivery. A mission stuck in delivering can heartbeat indefinitely and ignore orgabot stop. orgabot cancel <mission-id> is what ends it.

orgabot long-runs lists missions past their alert threshold, and offers exactly three actions: steer (orgabot follow-up), end (orgabot stop), or acknowledge (orgabot ack-long-run). The --timeout value is an alert threshold, not a kill deadline: a healthy but slow worker is never killed.

The dashboard shows stale behavior after a pull

The dashboard is a long-running, non-hot-reloading process. Restart it. Separately, because the CLI runs the TypeScript sources through tsx, a change merged into Orgabot's own framework needs a git pull of your local clone; rebuilding dist/ is neither enough nor required.

You no longer have to notice this yourself: the dashboard checks its own checkout's HEAD on its own timer and raises a "Dashboard code is stale" incident (naming both commits and how many commits behind it is) with a one-click Restart dashboard button once it falls behind - that button drains in-flight requests, tears down the Restate components it owns, and re-execs against the current checkout, so the mission service picks up the fix too. orgabot stack restart (alias orgabot up --restart) does it from a terminal for the whole stack, including the orgabot up supervisor that runs the recovery sweep, and including a stack whose orgabot up has already exited: orgabot stack status reports that stack as orphaned and names what is stale. If stack restart prints NOT RESTARTED for the supervisor, the sweep is still on old code: run the command that line names (for a foreground orgabot up, Ctrl-C it and start it again). When it does restart, a foreground orgabot up continues DETACHED from its terminal: its output is in ~/.orgabot/logs/stack/supervisor.log and orgabot down stops it. orgabot up and orgabot review-loop's post-merge step both print the same notice when the dashboard they find is stale, and the System tab always shows the running commit next to Restate readiness.

The launch circuit is open and nothing will launch durably

A MissionDispatcher invocation for a mission that has since ended (or was never recorded) can be left paused/backing-off in Restate, and enough of those hold the durable-launch circuit open. A reaper on the dashboard's own timer kills exactly these - a stuck invocation whose mission is terminal or has no record at all - automatically, and orgabot stop/end/cancel and project retirement's mission deletion now also kill a mission's own invocation as they end it, so this should be rare. If orgabot doctor still reports stale invocations (it lists them separately from genuine LIVE-mission failures, with the manual DELETE /invocations/<id>?mode=Kill command for each), the reaper has not caught up to them yet - it runs on the same clock as the dashboard's other background passes.

Tests fail in a worktree but pass in your checkout

Worktrees start bare. Two classes of problem follow:

• Missing `node_modules`. A mission worktree gets a best-effort copy-on-write clone of the source checkout's installed node_modules (and of first-level package installs such as framework/ and site/), so it only starts empty when the source checkout has none. Installing may be approval-gated. Do not symlink the main checkout's node_modules into the worktree: a clean install through the link (npm ci removes node_modules first) deletes the shared tree.
• Stale or partial dependencies. A copy-on-write clone of node_modules can make npm ci skip work as "already present". Presence is not completeness; this is the usual reason a dependency-adding PR fails a pre-merge gate.
• Git LFS assets. Worktrees do not git lfs pull, so LFS-backed files arrive as pointer text and a worker can stall trying to read them.

Several unrelated missions hold on the same failing test

Suspect the base, not the missions. Post-merge verification re-checks the pre-merge baseline when the combined tree is red, so a mission that did not cause the break should complete with a note rather than be held. If many missions still show "post-merge verification needs attention" for the same failure, either the baseline could not be resolved (for example GitHub did not return mergeCommit.oid) or the base is still red for a reason that is being attributed conservatively. Fix the base first.

Related: when you compare two trees, verify the tree you measured. Confident-wrong conclusions come from stale refs; measure both sides in a fresh git worktree add --detach <ref>.

A merge landed but the feature is gone

A clean merge can delete a feature. There is no conflict, but the base branch's rewrite of the same region wins. The tell is a test file failing with zero failing test cases: the assertions vanished along with the code.

GitHub says rate limited

REST and GraphQL have separate limits. Before treating GitHub as unavailable, try the same operation over the other transport: gh pr create is GraphQL, gh api repos/O/R/pulls -X POST is REST: same identity, same authority. On real exhaustion, stop retrying the same command and report the failing path and the reset time.

Also check that the connector can actually see the repository. Authentication is not authorization: a token that can search a repo may still lack pull-request permission, and a token without an App installation or repository grant is not a working fallback at all.

Orgabot's own background readers back off rather than retrying into an exhausted allowance. An epic run that could not read GitHub is re-checked after five minutes, doubling up to twenty; one that is only waiting on, or held by, a child is re-checked about every ninety seconds. The dashboard's CI/CD backfill does not re-fetch a workflow run it already saw finish: it re-checks a failed run every fifteen minutes and a successful one every few hours, in case it was re-run without a webhook. A GitHub issue delivery whose context ingestion fails is retried on a growing, jittered delay from two minutes up to an hour; every recovery sweep reports it as a standing condition (which turns into a sweep failure after thirty minutes, like any other), and after eight failures it reports it as BLOCKED with the last reason and re-checks it once a day. orgabot issues context-retries lists every delivery that is backing off or blocked. Fix the cause it names (the Issue Curator grant, the repository configuration, or the issue itself), then run orgabot issues context-retries --clear <deliveryId> (or --clear-blocked) so the next sweep retries it. A delivery that falls out of the stored webhook journal before it could be ingested is reported once as LOST and its retry state removed. The dashboard shows the same list under Incidents, in the Issue context retries section (delivery id, repository, attempts, first failure, next retry, and last reason), with Retry now on each delivery and Retry all blocked and Retry all for the lot. They do exactly what --clear, --clear-blocked and --clear-all do: nothing is read from GitHub until the next recovery sweep retries the delivery.

Missions paused, or nothing launches, under memory pressure

The mission tab shows the pause glyph and its hover text reads "memory watchdog: ...", or launches queue with a reason naming the memory watchdog. This is the runtime memory watchdog protecting the host: at elevated pressure no new work starts, and at critical pressure running missions pause (their verify and build runs are killed and re-run later) until the host has been calm for a sustained window.

1. Run orgabot throttle watchdog to see the reading that decided it, which missions hold how much memory, and how much belongs to nothing Orgabot runs (a local model or a VM is the usual answer).
2. Free memory, or wait: paused missions resume on their own, one at a time. While one is still paused, the same output has a "resume held by" list naming what is keeping it paused. orgabot mission resume <mission-id> releases one early.
3. If the verdict mentions the Data volume, reclaim finished missions' worktrees with orgabot clean --attribute, then orgabot clean.
4. If the thresholds do not fit this machine, tune them (for example ORGABOT_MEMORY_WATCHDOG_SWAP_VOLUME_ELEVATED_GIB), or set ORGABOT_MEMORY_WATCHDOG=observe to record without acting; see Turning it down, or off.

If orgabot up is not running, nothing stays paused: a paused mission releases itself within 3 minutes of the watchdog's last reading, and admission reopens on the same bound.

A pause is not a failure and not a hold, so nothing needs to be retried; the mission continues from where it parked, with every gate still in force.

A round says "starting up" and nothing ever streams

Two different things used to look identical here.

The worker launched and never took a turn. A Claude Code worker on an exhausted account can start, print its session announcement, and then sit silent without erroring. Every worker now has a startup window (5 minutes by default, --startup-timeout <minutes>): if no turn streams inside it, the worker is stopped and the round continues on the next executor in the fallback chain, on the same branch and plan node. You do not need to stop it and relaunch it pinned to Codex. With no fallback configured, the round fails and says the executor never started.

The worker never launched at all. A round queued behind the host-wide worker budget prints that it is waiting for a host worker slot, and its long-run notice says so. orgabot verify slots shows who holds the slots. orgabot stop on a queued round takes effect immediately: the worker will not start.

The machine went to sleep while missions were running

Sleep is host-wide, so one suspension kills every running mission at once. Orgabot now handles it as its own case rather than as a mysterious verification failure.

It tries not to happen. Every mission asks macOS to withhold idle sleep for exactly as long as that mission runs. Lid close, low battery, and an explicit Sleep from the Apple menu still apply, deliberately: those are you saying stop. Each mission prints which it got, so an unattended overnight run never looks protected when it is not. ORGABOT_NO_SLEEP_GUARD=1 turns it off.

A suspension no longer eats a command's timeout. Timeouts are charged against time the process could actually run, so a suite that takes 67 seconds is not reported as a 10-minute timeout because the laptop was closed for an hour in the middle of it.

A suspension no longer reads as a hung worker either. The idle watchdog charges its silence budget the same way, from samples rather than from raw wall clock, so a laptop that sleeps for 17 minutes mid-mission is not stopped as a worker that went quiet for 17 minutes. The watchdog is not disarmed by this: the uncharged time extends the window by exactly the suspension, so a worker that is still silent after the wake is stopped one ordinary idle window later, and the bounded tool window and the absolute run ceiling are unchanged. A stop that did span a suspension is reported as a host sleep, which is resumable, rather than as a stall. Raising --idle-timeout is still your choice and was never the fix for this: a wider window only makes the wrong answer rarer.

A sleep is not a verification failure. A killed command reports exit null and a timeout, which used to read as a failing suite and start a repair round against tests that were never broken. A verification interrupted by a suspension now holds the mission as waiting on verification_unavailable (error class host_sleep), with the branch and the objective intact, and says so. You do not have to resume it: after the wake, the recovery sweep re-verifies the same head exactly as it does after a network outage, with the same backoff and the same handoff to you (verification_unavailable_exhausted) if it still gets no result after 10 rounds. See Missions for the schedule.

A held delivery names the right defect. Delivery authority is resolved from the mission's role, so a record that lost its organization mission (recovery and salvage now carry it forward) has no identity to deliver under. That still holds fail-closed, and never falls back to your own credentials, but it now says the attribution cannot be resolved instead of telling you to grant a permission the role already holds.

A network drop is not a refused delivery. Before a role-owned push, Orgabot mints the role's GitHub token. If that mint cannot reach GitHub (fetch failed, a connection reset, a DNS or connect timeout), Orgabot retries it for up to about 30 seconds. It does not report "delivery identity refused". If GitHub is still unreachable after that, a review-remediation push is retried by the review loop with backoff, the same way a dropped push is. A first ship holds with delivery_identity_unavailable. The branch stays committed, the pending record is kept, and nothing is recorded as a denial. An answer from GitHub, such as a 401, a 403, or a 422 because the App installation does not cover the repository, is still a real refusal and holds exactly as before.

A delivery held on an outage ships itself once GitHub is back. You do not need to run orgabot ship for a delivery_identity_unavailable hold. The recovery sweep (the running stack, or orgabot recover sweep from a scheduled job) retries it through the ordinary ship path, so the draft pull request, CI, independent review, and merge gates all still apply. Before each attempt it checks that the branch still exists and that no pull request for it is already open, merged, or closed; if any of that is true, or GitHub cannot be read, it does not ship. Attempts back off per mission (2 minutes, doubling up to 30 minutes) and are rate limited across all missions (3 attempts per 10 minutes). Only outage holds are retried this way: delivery_identity_hold and delivery_repository_access_hold are refusals and still wait for you. Only one retry runs per branch at a time, even when both the running stack and a scheduled orgabot recover sweep are active. Once the pull request is open, the mission's hold changes to "delivery pending" (or "review loop not merged" if the review ran without merging), and the mission completes when the pull request merges. If the new pull request's CI is red (delivery_ci_not_green), or its merge broke the default branch (review_loop_post_merge_broken), the hold says so and waits for you instead. For red CI, see A delivery is held because CI is not green. If a retry is refused for a real reason, or fails without one, the hold changes to say so (delivery_reship_failed for a failure with no named cause) and the automatic retries stop. After 8 attempts with GitHub still unreachable, the hold changes to delivery_reship_exhausted and waits for you. You can still run orgabot ship, or use the dashboard's ship action, at any time.

Salvaged work stays reachable. When your checkout is dirty the mission runs in a temp clone with its own object store, so a branch committed there is invisible to the project checkout. Salvage now mirrors that branch back into the repository, and the message tells you the exact command to continue with. If the mirror fails it says that too, along with the git push that gets the work out.

A worker dies on a large edit

Very large single-line literals (base64 blobs, data URIs) can kill a worker mid-edit. Do that specific swap by hand, then still ship it through the normal draft PR and review loop.

A pull request is open but nothing is advancing it

When a mission holds and its process exits, its pull request has no owner. Orgabot records that in the recovery queue, and the queue can now drive it:

orgabot pr-recovery                       # what is stranded, and what recovery did about it
orgabot pr-recovery resume                # sweep: resume everything that qualifies
orgabot pr-recovery resume acme/web#394   # resume one pull request yourself
orgabot pr-recovery resume --dry-run      # decide without launching anything
orgabot pr-recovery opt-out acme/web#394 --reason "closing by hand"
orgabot pr-recovery opt-in acme/web#394

The same two controls (Resume now, Opt out) sit on every card in the dashboard's recovery queue, alongside what the last attempt did or why it is holding.

A resumption re-enters the ordinary governed path: a mission on the pull request's own branch, composed under the original mission's organization and role, which then goes through Orgabot's usual verify, independent review, and merge gates. Nothing is merged on weaker evidence than a live mission needs.

What it refuses to do:

• Act on a stale record. Every decision re-reads the pull request from GitHub first. An unreadable state holds; it never falls back to the cache.
• Undo your decision. A pull request whose owning mission you cancelled is never resumed by a sweep. Name it explicitly if it should continue.
• Widen authority. No role to inherit means the resumption holds for a role assignment (answer it from orgabot inputs), never the generic worker.
• Double-drive. The claim is taken under a lock, so two passes cannot both resume one pull request, and GitHub reads and launches are rate-limited per time window.
• Touch a pull request Orgabot did not push. A pull request opened from a fork is never resumed, however its branch is named: the head branch of a fork is named by whoever opened it, so orgabot/... there is not evidence that Orgabot created it. If GitHub does not say which repository the head is in, resumption refuses rather than guesses.

The sweep is manual by default. Set ORGABOT_AUTO_RESUME_PRS=1 before starting the dashboard to let it run the same bounded pass on the dashboard's own recovery timer. It never runs as a side effect of loading the dashboard page: a page load is a read, and a read must not be able to start a worker.

CI went red and nothing is fixing it

A failing GitHub Actions run on an open Orgabot pull request is already driven back to green by CI recovery. The gap was everything else, and the commonest case is the worst one: a workflow that fails on the default branch after a merge (a scheduled scan, a post-merge security job). Nobody owned it, so it sat on the Incidents list until a human pressed Fix.

Orgabot now triages that failure itself. It files an issue describing the run, the branch, and the failing job's log, then hands that issue to the ordinary fix-and-ship mission: the same routing, verification, independent review, and merge decision as any other issue.

orgabot ci-triage                                  # what was triaged, and what came of it
orgabot ci-triage run                              # sweep: triage everything that qualifies
orgabot ci-triage run --dry-run                    # decide without filing or launching anything
orgabot ci-triage run acme/web#31601228938         # triage one failing run yourself
orgabot ci-triage opt-out acme/web#31601228938 --reason "known flake, fixing by hand"
orgabot ci-triage opt-in acme/web#31601228938

What it refuses to do:

• Compete with CI recovery. A run on an Orgabot mission branch belongs to the pull request's own recovery loop and is left alone. The orgabot/ prefix alone is not ownership: a mission record for that exact branch must exist.
• Fix a fork's failure. GitHub delivers fork pull-request runs into the base repository, and a fork's head branch is named by whoever opened it. Without positive proof the run's head is in this repository, triage refuses.
• File the same issue twice. The claim is taken under a lock before anything external happens, and a run already reported is never reported again. If the local record is lost, an existing open issue with the same generated title still stops a duplicate.
• Undo your decision. An incident you acknowledged is never triaged, and an opted-out run stays out.
• Report a fix that did not start. If the mission holds (for a role assignment, the token throttle, or a security review), the issue stands as the alert and the hold is reported, never a started mission.
• Chase history. A failure older than 24 hours, or one a later run of the same workflow on the same branch superseded, is skipped. A dashboard restart replays old webhook deliveries, and history is not news.

The sweep is manual by default. Set the ORGABOT_AUTO_FIX_CI opt-in before starting the dashboard to let it run the same bounded pass on the dashboard's own recovery timer. Like every other automatic pass, it never runs as a side effect of loading the dashboard page: a page load is a read, and a read must not be able to file an issue or start a worker.

A delivery is held because CI is not green

When a mission opens its pull request and the CI gate finds the checks failing, or still pending after the wait budget, Orgabot does not review or merge over the red check. The mission holds with delivery_ci_not_green, and the hold names the command that drives the pull request: orgabot review-loop <project> <pr#>. This applies to an ordinary first delivery and to a delivery re-shipped after a GitHub outage.

You usually do not need to run that command. The recovery sweep (the running stack, or orgabot recover sweep from a scheduled job) re-reads the pull request's CI every 10 minutes. Once CI is green and the pull request is still open, it starts orgabot review-loop for you, so the independent review, fix rounds, and merge gates all still apply, and the review loop records the outcome. If the pull request was already merged with green CI, the hold changes to "delivery pending" and the mission completes from that merge. Red, pending, or unreadable CI keeps the hold, and a merge made over red CI does not complete the mission on its own. If the review loop cannot be started, or exits before taking the mission, the sweep starts it again at the next re-check; after 3 such starts it stops and the hold tells you to read the mission log and run the command yourself.

Stray worktrees and temp directories

orgabot clean [--dry-run]     # release what finished missions still hold
orgabot clean --attribute     # report instead: who holds what, and how much
orgabot clean --remove <dir> [--force]   # remove one root clean kept, once you judge it safe
orgabot recover [--dry-run]   # salvage shippable or dead missions

orgabot clean runs one pass over everything a finished mission still holds: its checkout and dependency tree, a worker or mission-host process tree that outlived it, its log (compressed out of hot storage, every byte kept beside it as <id>.log.gz, and still readable in the dashboard and the mission terminal), unused shared dependency-store entries, and orphaned orka-wt-* temp roots. The dashboard runs the same pass on its recovery timer, so this is usually only needed by hand.

Every step refuses on uncertainty: a mission that is not terminal, work that is built but not yet delivered, an unreadable record, a process whose identity no longer matches the pid, or an unreadable process table all leave the resource exactly where it is. That includes the temp-root sweep: a checkout any non-terminal mission still claims is kept and reported as kept, and a mission listing that cannot be read authoritatively keeps every root rather than reading as "nothing owns these". A waiting mission matters most here, because it has no live process and stops heartbeating, so its preserved checkout looks exactly like litter from the outside, and it is the one an operator's answer resumes in place. A process is asked to stop first and only killed on a later pass, once the grace period has actually elapsed.

Isolated clones

When a source checkout is dirty or its branch is already checked out elsewhere, a mission works in an isolated clone (a full checkout with its own .git directory, often about 1 GB) instead of a git worktree. Untracked files inside Orgabot's own .orgabot/ (or .orka/) state directory do not make a checkout dirty for this decision: Orgabot writes them itself (every memorialized instruction under live-context/, pending memory records under memory/), and an untracked file never reaches a linked worktree anyway. Before, those files pushed every mission on a busy repository, open-PR resumptions included, into a clone. orgabot verify lease-run also accepts the requesting mission's own clone when its origin names the same remote as the project's checkout, so a mission that does run in a clone can still take a leased full verification. A clone's object store can be the only copy of committed work, so clean never removes one just because of its shape. Instead it attributes each clone to its mission, by the mission's recorded worktree path, by the branch the clone has checked out, or by the mission id embedded in the directory name (...-mugznmim-f3e8c38c), and then applies the same rule as any other checkout:

• A running, queued, or waiting mission keeps its clone, and the kept line names the mission.
• A finished mission's clone is reclaimed only when its owner has stopped heartbeating, its work is recorded as delivered, and every commit in the clone (each local branch tip and HEAD) is found on a ref in the project's registered source repository. Work that is built but not delivered keeps the clone, and so does any commit that exists only in the clone: a failed mission often has no delivery record, so "delivered" alone never proves the branch lives anywhere else. If the source repository is unknown or the check fails, the clone is kept. A clone created, fetched into, or checked out in the last 15 minutes is also kept, because a new mission reusing a finished mission's branch may still be preparing it before its own record names it.
• A clone no mission can be attributed to is kept (uncertainty keeps), and reported as unattributed with its size, the branch it has checked out, and whether that branch has commits no remote-tracking ref reaches.

orgabot clean --attribute lists every isolated clone with its mission (or "unattributed") and its measured size.

To remove a clone you have judged safe, run orgabot clean --remove <dir> with the orka-wt-* root or any path inside it (add --dry-run to preview). It removes the whole root and records the removal in ~/.orgabot/worktreeRemovals.jsonl. It applies the same attribution to a linked worktree as to a clone. Even with --force it refuses a root any live mission holds (by recorded path, branch, or mission id), a finished mission whose owner heartbeated in the last 15 minutes (it may still be shipping from that checkout), and a root or checkout that could not be inspected; stop or cancel that mission first. Without --force it also refuses a mission whose work is built but not delivered (or whose delivery cannot be read), and a clone with unpushed commits or whose unpushed state cannot be read. The unpushed check is offline and compares against the clone's own remote-tracking refs, so it can report commits that were in fact pushed to an explicit URL; inspect the branch before forcing.

A mission's worktree disappeared

Orgabot records every worktree it removes to ~/.orgabot/worktreeRemovals.jsonl (one JSON line per removal: when, which path, which code removed it, and why it believed it was allowed to). If a checkout vanishes unexpectedly, read that file first: it names the caller, so this is a one-line diagnosis rather than an investigation.

A round whose working directory is removed underneath it is reported as a missing working directory naming the path, not as spawn git ENOENT, and it is retried on a fresh checkout of the mission branch. Every committed change survives, because the branch holds it; only edits that were never committed are lost, and those went with the directory regardless.

"Why is Orgabot using 18 GB?"

orgabot clean --attribute answers by mission, largest first: each checkout and dependency tree with its measured size, and the resident memory of the live process trees that mission still owns. Disk and memory are reported separately and never summed. A resource whose size could not be measured is counted as unmeasured rather than as zero.

--attribute only reports. It returns before any of the reclaim passes run, so nothing is deleted, compressed, or signalled: run orgabot clean without it once you have decided what should go.

Be careful with repo-wide worktree sweeps when several registered projects share one git root: a sweep can reach a sibling project's live checkout.

`npm run orgabot` fails immediately with a missing esbuild binary

The postinstall that fetches esbuild's platform binary was blocked. Fix it with (cd framework && npm rebuild esbuild).