Troubleshooting

Troubleshooting a stuck mission

Start with these three before anything else:

orgabot status              # what is running, held, or waiting right now
orgabot logs <mission-id>   # what that mission actually did
orgabot doctor              # host-level health: stale invocations, wedged state

Between them they cover "what's happening," "why," and "is Orgabot itself healthy." The sections below go deeper on each failure mode.

A mission is held

A hold is Orgabot refusing to guess. Find out which kind:

orgabot status
orgabot logs <mission-id>
orgabot session <mission-id> --json

A held (waiting) mission cannot be cancelled with orgabot cancel; cancel only ever ends a running mission. Aiming it at a held one refuses (exit code 2) and names the verb that actually applies: orgabot stop <mission-id> to end it, or orgabot mission resume <mission-id> to continue it, along with the hold's own code, reason, and remedy so you know which one to pick without a separate orgabot status round trip. orgabot stop on a held mission ends it the same way and prints the same hold detail, since ending it also clears the hold.

A mission also cannot be rescoped. The project is fixed at launch, so a follow-up that says "actually do this on the other repo" will hold forever. Relaunch on the right project instead.

A mission looks alive but is going nowhere

Heartbeat alone is not progress. The real signal is the last worker event time. Two common causes:

• A git hook in the target repo is hanging. A husky or similar pre-commit hook that never returns stalls Orgabot's own git commit forever, while the heartbeat keeps ticking.
• The process is wedged in delivery. A mission stuck in delivering can heartbeat indefinitely and ignore orgabot stop. orgabot cancel <mission-id> is what ends it.

orgabot long-runs lists missions past their alert threshold, and offers exactly three actions: steer (orgabot follow-up), end (orgabot stop), or acknowledge (orgabot ack-long-run). The --timeout value is an alert threshold, not a kill deadline: a healthy but slow worker is never killed.

The dashboard shows stale behavior after a pull

The dashboard is a long-running, non-hot-reloading process. Restart it. Separately, because the CLI runs the TypeScript sources through tsx, a change merged into Orgabot's own framework needs a git pull of your local clone; rebuilding dist/ is neither enough nor required.

You no longer have to notice this yourself: the dashboard checks its own checkout's HEAD on its own timer and raises a "Dashboard code is stale" incident (naming both commits and how many commits behind it is) with a one-click Restart dashboard button once it falls behind - that button drains in-flight requests, tears down the Restate components it owns, and re-execs against the current checkout, so the mission service picks up the fix too. orgabot up --restart does the same thing from a terminal. orgabot up and orgabot review-loop's post-merge step both print the same notice when the dashboard they find is stale, and the System tab always shows the running commit next to Restate readiness.

The launch circuit is open and nothing will launch durably

A MissionDispatcher invocation for a mission that has since ended (or was never recorded) can be left paused/backing-off in Restate, and enough of those hold the durable-launch circuit open. A reaper on the dashboard's own timer kills exactly these - a stuck invocation whose mission is terminal or has no record at all - automatically, and orgabot stop/end/cancel and project retirement's mission deletion now also kill a mission's own invocation as they end it, so this should be rare. If orgabot doctor still reports stale invocations (it lists them separately from genuine LIVE-mission failures, with the manual DELETE /invocations/<id>?mode=Kill command for each), the reaper has not caught up to them yet - it runs on the same clock as the dashboard's other background passes.

Tests fail in a worktree but pass in your checkout

Worktrees start bare. Two classes of problem follow:

• Missing `node_modules`. A fresh worktree has none, and installing may be approval-gated. Symlinking the main checkout's node_modules works; unlink before finishing so it is not committed.
• Stale or partial dependencies. A copy-on-write clone of node_modules can make npm ci skip work as "already present". Presence is not completeness; this is the usual reason a dependency-adding PR fails a pre-merge gate.
• Git LFS assets. Worktrees do not git lfs pull, so LFS-backed files arrive as pointer text and a worker can stall trying to read them.

Several unrelated missions hold on the same failing test

Suspect the base, not the missions. Post-merge verification re-checks the pre-merge baseline when the combined tree is red, so a mission that did not cause the break should complete with a note rather than be held. If many missions still show "post-merge verification needs attention" for the same failure, either the baseline could not be resolved (for example GitHub did not return mergeCommit.oid) or the base is still red for a reason that is being attributed conservatively. Fix the base first.

Related: when you compare two trees, verify the tree you measured. Confident-wrong conclusions come from stale refs; measure both sides in a fresh git worktree add --detach <ref>.

A merge landed but the feature is gone

A clean merge can delete a feature. There is no conflict, but the base branch's rewrite of the same region wins. The tell is a test file failing with zero failing test cases: the assertions vanished along with the code.

GitHub says rate limited

REST and GraphQL have separate limits. Before treating GitHub as unavailable, try the same operation over the other transport: gh pr create is GraphQL, gh api repos/O/R/pulls -X POST is REST: same identity, same authority. On real exhaustion, stop retrying the same command and report the failing path and the reset time.

Also check that the connector can actually see the repository. Authentication is not authorization: a token that can search a repo may still lack pull-request permission, and a token without an App installation or repository grant is not a working fallback at all.

The machine went to sleep while missions were running

Sleep is host-wide, so one suspension kills every running mission at once. Orgabot now handles it as its own case rather than as a mysterious verification failure.

It tries not to happen. Every mission asks macOS to withhold idle sleep for exactly as long as that mission runs. Lid close, low battery, and an explicit Sleep from the Apple menu still apply, deliberately: those are you saying stop. Each mission prints which it got, so an unattended overnight run never looks protected when it is not. ORGABOT_NO_SLEEP_GUARD=1 turns it off.

A suspension no longer eats a command's timeout. Timeouts are charged against time the process could actually run, so a suite that takes 67 seconds is not reported as a 10-minute timeout because the laptop was closed for an hour in the middle of it.

A suspension no longer reads as a hung worker either. The idle watchdog charges its silence budget the same way, from samples rather than from raw wall clock, so a laptop that sleeps for 17 minutes mid-mission is not stopped as a worker that went quiet for 17 minutes. The watchdog is not disarmed by this: the uncharged time extends the window by exactly the suspension, so a worker that is still silent after the wake is stopped one ordinary idle window later, and the bounded tool window and the absolute run ceiling are unchanged. A stop that did span a suspension is reported as a host sleep, which is resumable, rather than as a stall. Raising --idle-timeout is still your choice and was never the fix for this: a wider window only makes the wrong answer rarer.

A sleep is not a verification failure. A killed command reports exit null and a timeout, which used to read as a failing suite and start a repair round against tests that were never broken. A verification interrupted by a suspension now holds the mission as resumable, with the branch and the objective intact, and says so.

A held delivery names the right defect. Delivery authority is resolved from the mission's role, so a record that lost its organization mission (recovery and salvage now carry it forward) has no identity to deliver under. That still holds fail-closed, and never falls back to your own credentials, but it now says the attribution cannot be resolved instead of telling you to grant a permission the role already holds.

Salvaged work stays reachable. When your checkout is dirty the mission runs in a temp clone with its own object store, so a branch committed there is invisible to the project checkout. Salvage now mirrors that branch back into the repository, and the message tells you the exact command to continue with. If the mirror fails it says that too, along with the git push that gets the work out.

A worker dies on a large edit

Very large single-line literals (base64 blobs, data URIs) can kill a worker mid-edit. Do that specific swap by hand, then still ship it through the normal draft PR and review loop.

A pull request is open but nothing is advancing it

When a mission holds and its process exits, its pull request has no owner. Orgabot records that in the recovery queue, and the queue can now drive it:

orgabot pr-recovery                       # what is stranded, and what recovery did about it
orgabot pr-recovery resume                # sweep: resume everything that qualifies
orgabot pr-recovery resume acme/web#394   # resume one pull request yourself
orgabot pr-recovery resume --dry-run      # decide without launching anything
orgabot pr-recovery opt-out acme/web#394 --reason "closing by hand"
orgabot pr-recovery opt-in acme/web#394

The same two controls (Resume now, Opt out) sit on every card in the dashboard's recovery queue, alongside what the last attempt did or why it is holding.

A resumption re-enters the ordinary governed path: a mission on the pull request's own branch, composed under the original mission's organization and role, which then goes through Orgabot's usual verify, independent review, and merge gates. Nothing is merged on weaker evidence than a live mission needs.

What it refuses to do:

• Act on a stale record. Every decision re-reads the pull request from GitHub first. An unreadable state holds; it never falls back to the cache.
• Undo your decision. A pull request whose owning mission you cancelled is never resumed by a sweep. Name it explicitly if it should continue.
• Widen authority. No role to inherit means the resumption holds for a role assignment (answer it from orgabot inputs), never the generic worker.
• Double-drive. The claim is taken under a lock, so two passes cannot both resume one pull request, and GitHub reads and launches are rate-limited per time window.
• Touch a pull request Orgabot did not push. A pull request opened from a fork is never resumed, however its branch is named: the head branch of a fork is named by whoever opened it, so orgabot/... there is not evidence that Orgabot created it. If GitHub does not say which repository the head is in, resumption refuses rather than guesses.

The sweep is manual by default. Set ORGABOT_AUTO_RESUME_PRS=1 before starting the dashboard to let it run the same bounded pass on the dashboard's own recovery timer. It never runs as a side effect of loading the dashboard page: a page load is a read, and a read must not be able to start a worker.

CI went red and nothing is fixing it

A failing GitHub Actions run on an open Orgabot pull request is already driven back to green by CI recovery. The gap was everything else, and the commonest case is the worst one: a workflow that fails on the default branch after a merge (a scheduled scan, a post-merge security job). Nobody owned it, so it sat on the Incidents list until a human pressed Fix.

Orgabot now triages that failure itself. It files an issue describing the run, the branch, and the failing job's log, then hands that issue to the ordinary fix-and-ship mission: the same routing, verification, independent review, and merge decision as any other issue.

orgabot ci-triage                                  # what was triaged, and what came of it
orgabot ci-triage run                              # sweep: triage everything that qualifies
orgabot ci-triage run --dry-run                    # decide without filing or launching anything
orgabot ci-triage run acme/web#31601228938         # triage one failing run yourself
orgabot ci-triage opt-out acme/web#31601228938 --reason "known flake, fixing by hand"
orgabot ci-triage opt-in acme/web#31601228938

What it refuses to do:

• Compete with CI recovery. A run on an Orgabot mission branch belongs to the pull request's own recovery loop and is left alone. The orgabot/ prefix alone is not ownership: a mission record for that exact branch must exist.
• Fix a fork's failure. GitHub delivers fork pull-request runs into the base repository, and a fork's head branch is named by whoever opened it. Without positive proof the run's head is in this repository, triage refuses.
• File the same issue twice. The claim is taken under a lock before anything external happens, and a run already reported is never reported again. If the local record is lost, an existing open issue with the same generated title still stops a duplicate.
• Undo your decision. An incident you acknowledged is never triaged, and an opted-out run stays out.
• Report a fix that did not start. If the mission holds (for a role assignment, the token throttle, or a security review), the issue stands as the alert and the hold is reported, never a started mission.
• Chase history. A failure older than 24 hours, or one a later run of the same workflow on the same branch superseded, is skipped. A dashboard restart replays old webhook deliveries, and history is not news.

The sweep is manual by default. Set the ORGABOT_AUTO_FIX_CI opt-in before starting the dashboard to let it run the same bounded pass on the dashboard's own recovery timer. Like every other automatic pass, it never runs as a side effect of loading the dashboard page: a page load is a read, and a read must not be able to file an issue or start a worker.

Stray worktrees and temp directories

orgabot clean [--dry-run]     # release what finished missions still hold
orgabot clean --attribute     # report instead: who holds what, and how much
orgabot recover [--dry-run]   # salvage shippable or dead missions

orgabot clean runs one pass over everything a finished mission still holds: its checkout and dependency tree, a worker or mission-host process tree that outlived it, its log (compressed out of hot storage, every byte kept beside it as <id>.log.gz, and still readable in the dashboard and the mission terminal), unused shared dependency-store entries, and orphaned orka-wt-* temp roots. The dashboard runs the same pass on its recovery timer, so this is usually only needed by hand.

Every step refuses on uncertainty: a mission that is not terminal, work that is built but not yet delivered, an unreadable record, a process whose identity no longer matches the pid, or an unreadable process table all leave the resource exactly where it is. A process is asked to stop first and only killed on a later pass, once the grace period has actually elapsed.

"Why is Orgabot using 18 GB?"

orgabot clean --attribute answers by mission, largest first: each checkout and dependency tree with its measured size, and the resident memory of the live process trees that mission still owns. Disk and memory are reported separately and never summed. A resource whose size could not be measured is counted as unmeasured rather than as zero.

--attribute only reports. It returns before any of the reclaim passes run, so nothing is deleted, compressed, or signalled: run orgabot clean without it once you have decided what should go.

Be careful with repo-wide worktree sweeps when several registered projects share one git root: a sweep can reach a sibling project's live checkout.

`npm run orgabot` fails immediately with a missing esbuild binary

The postinstall that fetches esbuild's platform binary was blocked. Fix it with (cd framework && npm rebuild esbuild).