Budgets, quota, and analytics for AI workers
USD spend budgets over real accrued cost, quota tracking for subscription executors, the status-bar meter, and the fallback ladder when a pool runs dry.
Last updated
Orgabot tracks two different things that both feel like "running out" and must never be added together:
| Kind | What it measures | Where it shows up |
|---|---|---|
| Spend budget | Accrued real USD cost on API-billed work | orgabot budget status, the Budgets tab |
| Quota | Consumption of a subscription window (a Claude or ChatGPT plan) | The same surfaces, denominated in quota, not dollars |
Counting subscription usage as cash spend produces false "budget exceeded" alarms. The fix is to split budgets by billing class, not to raise the limit.
Spend budgets
orgabot budget set <org> --level org|role|actor [--role slug] [--actor id]
--period daily|weekly|monthly|mission --limit <usd>
[--on-exceed warn|approval|block] [--warn-at f]
orgabot budget list <org> | remove <org> <budget-id> | status <org>Budgets are enforced pre-flight and mid-run, at organization, role, or actor level, over accrued real spend from the analytics journal. Only actual API-key spend accrues; subscription usage never counts toward a spend budget. If Orgabot cannot read a budget's accrued spend, the dashboard shows it as unknown with the reason, never as $0.
Quota-denominated budgets
Subscription usage is tracked off a calibrated QuotaState rather than USD:
- a calibrated-exhausted window routes or holds a launch exactly like an exhausted pool;
- an uncalibrated estimate above the warn threshold only warns, because an estimate must not block real work.
The status-bar quota meter
The dashboard's mission status bar carries a quota meter: a small gauge whose label is the smallest remaining allowance across every configured provider, because that is the one that will stop work first. The label names whose allowance it is (est. 42% left · Claude Max 7d week), so the collapsed bar and the expanded panel can never look like they disagree. A leading est. means the figure is an estimate rather than an observation; it is spelled out because at status-bar size a ~ reads as a minus sign. It stays visible when the bar is collapsed, and it refreshes about once a minute rather than on every dashboard poll.
The gauge itself is a dial: its needle points at that remaining headroom and takes the same colour as the level. When nothing is metered, or the read failed, or no capacity seed can be stood behind, the dial is drawn with no needle at all and the label shows no percentage. An empty dial means "no reading", never "out of quota".
The panel header says how old the reading is (read 2m ago), and says outright when the last refresh failed rather than passing the previous figures off as current. The meter refreshes on its own slow cadence while the dashboard repaints every few seconds, so without that a number that moved between two glances looks like the meter disagreeing with itself.
Clicking it expands a column per configured provider, side by side, each answering the same three questions in as few words as they take:
- a subscription plan meters its plan windows:
est. 85% used,est. 15% remaining, andresets at 3:00when an observed limit event supplied a reset time (a rolling window says it has no fixed reset instead of showing an invented date); - an api key meters the dollar budget that governs it:
45% spent,$90.00 of $200.00,55% remaining ($110.00), and the date the period rolls over. The figure is the spend Orgabot itself journaled against that provider, so it is a floor: a key used outside Orgabot is invisible here, and the column says so; - any individual model pool observed exhausted for that provider, with its reset (window mode) or the fact that it needs credits (entitlement mode).
A provider whose plan has no usage credits at all is called out separately, at the top of its column, as subscription inactive. That is a different condition from being out of quota: a cancelled, lapsed, or unpaid plan is never refilled by a window reset, so the panel tells you to renew it or route the role elsewhere instead of leaving you waiting for a renewal that is not coming.
The budget shown is the tightest active organization budget governing that key. A provider with no plan seed and no budget says it is not metered, and how to meter it, rather than showing an invented figure.
Which usage the meter counts
Orgabot's analytics journal only ever contains Orgabot's own missions. A meter built on it alone measures how much of the plan Orgabot spent, not how much the plan spent, so on a machine where you also use the claude terminal the dashboard and the CLI's own /usage disagree in both directions — and the capacity a limit event measures drifts by the same gap, which is how a weekly window ends up sitting past a denominator that was never the plan's.
So for Claude Code, Orgabot reads the wider record: Claude Code's own session transcripts, which every session on the machine appends to, including interactive terminal work and IDE sessions. Each window row says which record it counted:
counted across every Claude Code session on this machine, not just Orgabot's— the machine-wide read;counted from Orgabot's own journal only— the narrower fallback, used for a provider with no such transcripts, when you have opted out, or while the ledger is still catching up.
Two rules keep it honest. The two records are combined by taking the larger, never their sum: Orgabot's own missions run through the same CLI and are already inside the transcripts, so adding them would double-count. And the machine-wide attribution is claimed only on a complete read — a ledger still catching up raises the floor but is not presented as a measurement.
The ledger is incremental (transcripts are append-only, so only the new tail is ever read), byte-budgeted per refresh, and kept for 15 days. Point it elsewhere with ORGABOT_CLAUDE_SESSION_DIR, or set that variable to the empty string to opt out entirely and keep the journal-only estimate.
When the meter says "unknown"
A percentage is only ever shown when Orgabot has something behind it. Two cases make a plan window read not metered instead of showing a number, and neither is bad news:
- No capacity known. Claude Max ships a built-in token-capacity seed. Codex and Grok do not, because there is no published token figure for a ChatGPT or xAI plan that Orgabot could stand behind, and no limit event can measure one for them either (see below). Their windows still count usage; set your real ceiling with
executorQuotas. - Usage past the capacity. If an uncalibrated estimate has already run past the capacity, that disproves the capacity, not your plan. The window reports unknown rather than claiming "0% left".
The window Orgabot sums over
A plan like Claude Max does not meter a rolling window: it meters a fixed block that empties at a wall-clock time. Summing the last five hours instead is right in the middle of a block and badly wrong just after one resets, because the rolling window still holds the whole previous block. Measured on a live install: 53% of the session used against the CLI's own /usage saying 2%, with the real block only minutes old. Same arithmetic, different hours.
So when Orgabot has ever seen the plan state a reset time, it puts the window on that cadence: the observed reset, stepped forward by the window length, gives the block you are in now. The row then has a real reset to show — resets at 6:50pm · projected from an observed reset — instead of rolling, no fixed reset.
The anchor must be a reset the plan stated, which Orgabot recognises by it landing on a whole minute. The reset times Orgabot derives itself (the observation's own timestamp plus a countdown, or plus the window length) carry seconds and milliseconds; anchoring on one of those would manufacture a boundary out of arithmetic on Orgabot's clock. A window with no such observation stays rolling rather than guessing a phase.
It is still an estimate, and it is marked projected for a reason: it shifts which hours are summed, it never pins a percentage, and an idle stretch longer than the window moves the plan onto a new cadence (the next block starts at your next request). Anchors older than 30 days are dropped for that reason. That horizon is longer than the 14 days a capacity floor gets, because the two observations answer different questions: a reset is a claim about the plan's phase, which a downgrade does not touch, while a floor is a claim about its size, which a downgrade invalidates.
Capacity measured from an observed limit
The seeds are guesses, and a busy operator sits permanently past them: every window then reads "not metered" and the meter shows no figure at all even with plenty of quota left. So a limit event also measures the plan.
When a provider reports a limit, Orgabot sums what had been spent into that window before the banner landed — from the same record the live window uses, so the numerator and the denominator are never measured over different usage. The plan held at least that much, so a seed below that proven floor is a disproved seed and the floor becomes the capacity. The row says capacity measured at an observed limit when that is what you are reading, and the Budgets & Usage tab badges it measured capacity.
Three properties keep it honest:
- it can only ever raise a capacity, never lower one. Even the machine-wide read is a floor rather than a ceiling — usage on the same plan from another machine is not in it — so treating it as exact would meter pessimistically;
- only the machine-wide record may measure a floor. Orgabot's own journal is blind to every token you spend driving the same plan outside Orgabot, so the floor it reports at a limit banner is Orgabot's share of the window, not the plan's capacity — and that share can be arbitrarily small. On a live install it measured a Codex capacity of eight tokens, and dividing by it rendered an exhausted ChatGPT plan as
est. 0% used · 100% remaining · estimated healthy. So a provider with no machine-wide record (today: everything but Claude Code) stays not metered however many limits it hits, rather than reporting a percentage of a number nobody measured. Its limit events still pin the window while they are live; - observations older than 14 days are ignored, because a floor proved on a plan you have since downgraded is no evidence about the plan you have now.
Two more things the meter deliberately does not do. It never reports a plan as out on an estimate alone: seed capacities are operator configuration and any usage record Orgabot holds is a floor, so only an observed limit event may say "out of quota". And it meters plans, not models: a subscription window belongs to the plan, and a provider's models draw on the one window; the per-model dimension that genuinely exists is the exhausted-pool row.
Quota windows and capacities are operator configuration (executorQuotas in ~/.orgabot/config.json); the full picture, including dollar budgets, lives in the Budgets & Usage tab.
When a pool runs dry
orgabot organization role-fallback set <org> <role> "claude:opus,codex"
orgabot organization quota-policy set <org> [<role>] wait-then-escalate|escalate
orgabot pools list | clear <provider> [model]
orgabot pools set <provider> [model] (--reset-at <ISO> | --entitlement)
orgabot pools holds | resume-due | resumeThe fallback chain is an ordered list of engine[:model] alternatives a role uses past an exhausted pool. A mission walks the chain mid-run, so an exhaustion part-way through does not lose the round. When the chain is fully spent, the mission holds and waits for the window to reset, then escalates per the quota policy.
Two rules keep the chain honest:
- The chain is for unavailability only. A healthy executor is never abandoned because a round was slow or a review found something.
- A belief must never shorten the chain. A stale "this pool is exhausted" record must reorder the chain, never delete an available executor from it. A skip at round start is a reordering, not a removal.
- A model belongs to its provider. A mission launched with
--worker claude --model opuskeepsopuson the Claude rung only. When it falls back to Grok or Codex, that rung runs the model its ownengine:modelentry names, or that engine's default. A rung given another vendor's model (for examplegrok:opus) drops it and runs its default instead. - A rung that cannot start is skipped, not failed. When an executor refuses to start on its configuration (an unknown model id, a CLI that is not signed in), the mission moves to the next rung. With no rung left, the round holds and names the reason; it is never recorded as a failure of the work.
orgabot pools clear deletes a belief; orgabot pools set corrects one. When a provider states a reset further out than the stored record (or the record has no horizon at all), write the horizon you know to be true with --reset-at <ISO>; the record counts as observed, so the rung is genuinely skipped until then. --entitlement records a non-resetting credit exhaustion instead (re-checked after 24h). Note that only an observed horizon (an explicit reset time, or an entitlement) skips a rung at round start; an inferred default window is believed but the rung is still tried, and orgabot pools list / orgabot doctor group the two states separately.
Analytics
orgabot analytics backfill # one-time, idempotent projection of existing storesA metrics journal with provider cost accounting feeds the dashboard's Analytics and Budgets tabs. Backfill projects pre-existing stores into the journal without double-counting.
Optimization telemetry
Alongside the metrics journal, Orgabot records optimization signals: one normalized record per worker run, executor call, fallback-chain hop, review verdict, operator correction, governed tool call, GitHub quota observation, CI job and run, capacity sample, and billable cost receipt. They are what optimization rules analyze, so a rule never needs to know how each service works.
- They live in
~/.orka/telemetry/optimization/, one JSON Lines file per month, separate from the metrics journal, so they never change a spend or usage figure. - Every cost and every number carries its basis: measured (Orgabot observed it), reported (the provider said so), or estimated (computed, for example tokens times a price table). Billing says whether a cost is cash, notional subscription value, or free.
- Signals hold identifiers, counts, durations, sizes, outcomes and costs only. Prompts, diffs, tool output, messages and anything shaped like a credential are refused by the schema, not just left out.
- High-frequency sources are sampled: capacity at most once every 5 minutes per provider (plus every pressure change), GitHub quota once a minute per bucket (plus every window rollover).
- To stop recording them, set
ORGABOT_OPTIMIZATION_TELEMETRY=offin the environment Orgabot runs in. The optimization rules and the executor reports below then have nothing new to read.
Optimization rules
orgabot optimize rules # every rule, whether it runs, and its thresholds
orgabot optimize scan # run the enabled rules over the last 7 days of signals
orgabot optimize scan --since 36h --dry-run
orgabot optimize disable redundant_api_calls --reason "too noisy for this org"
orgabot optimize enable redundant_api_callsThe optimization rules read the signals above and point out inefficiencies you can explain without an AI model. Each rule is plain arithmetic over counts, durations, timestamps and identifiers, so the same signals always give the same result.
| Rule | What it looks for |
|---|---|
duplicate_executions | The same CI job passing more than once on the same commit in the same place. |
redundant_api_calls | One tool called 20 or more times within 10 minutes for the same mission. |
polling_despite_webhook | A source polled 3 or more times while it reports that a webhook is available. |
missing_caching_or_batching | CI runs that reuse almost nothing from earlier runs, and bursts of 10 or more back-to-back calls to one tool. |
cloud_after_local | A CI job run remotely on a commit after it already passed on Orgabot's local controller. |
excessive_logging_retention | Retention over 30 days, or more than 5 GiB of logs in the scan window. |
idle_overprovisioned | A remote pool that kept half or more of its workers idle across 3 or more samples with nothing queued. |
unnecessary_remote_inference | 5 or more small model calls (4,000 tokens or fewer) sent to a remote provider while local models were ready. |
repeated_failures | The same operation failing 3 or more times: a worker run, model call, tool call, rate limit, or CI job. |
repeated_identical_requests | The same tool request or model prompt sent again under the same credential while the earlier answer was still fresh, or an unchanged resource downloaded again. |
unbatched_requests | 5 or more different items requested one at a time in a burst, from a connector that declares a batch form. |
missing_dependency_cache | A GitHub Actions job that installed its dependencies on 3 or more successful runs with no cache step. |
The caching and batching rules respect three boundaries before they recommend anything:
- Freshness. A repeat counts only inside the connector's declared cache lifetime (5 minutes when none is declared). If the answer changed between identical requests, the cache would have served stale data, so confidence drops and risk rises. Past half of the repeats, no cache is recommended. A model that answers an identical prompt differently is treated the same way, as not deterministic.
- Authorization. Requests are counted per credential and role. The same request under two credentials is never counted as a saving, and every proposed cache is keyed per credential.
- Correctness. Writes are never cached or batched, and a write to the same resource ends the earlier answer's reuse. A dependency cache is keyed on the lockfile and the install step still runs. If Orgabot could not read the workflow file, the finding asks you to confirm first, because a setup action's
cache:input does not show in step names.
Savings are reported in the service's own unit (the connector's request or quota unit, model tokens, or GitHub Actions minutes) and in normalized units: API calls, calls per day, milliseconds, bytes and US dollars.
orgabot optimize scan records each finding as an optimization opportunity in the operational store, with:
- Evidence written as a sentence, for example issue-sync polled github 12 times between ... (every 60s), spending 36 API calls, while reporting that a webhook is available for this source.
- A confidence (how sure the rule is the inefficiency is real) and a risk (what could go wrong if you act on it, and whether it can be undone). The rule's risk is a default: Orgabot classifies each recommendation itself, as described under Risk classification below.
- Expected effects, current and proposed behavior, and a recommended action.
Recording an opportunity changes nothing. Every opportunity starts as detected, requires approval, and is never applied automatically.
Scanning again does not create duplicates: the same inefficiency in the same scope maps to the same opportunity. Decisions you have made are kept:
- An opportunity you rejected stays rejected.
- A suppressed opportunity stays quiet until its suppression ends. If the rule still sees the problem after that, the opportunity goes back to detected, and its history records why.
- A suppress override for the organization, department, role, mission, or task keeps new findings in that scope from being recorded.
- An opportunity whose suppression ended stays suppressed while a suppress override covers it or the optimization policy blocks it. The next scan after that ends sends it back to detected.
Each rule can be turned off or on by itself with orgabot optimize disable and orgabot optimize enable. Turning one rule off changes nothing about the others. --rule <rule-id> runs only the rules you name, and --dry-run shows what would be found without recording anything.
The Optimization Center
Insights > Optimization Center in the dashboard is where you review and decide on the opportunities that orgabot optimize scan records. orgabot optimization list and orgabot optimization show read the same records from the command line.
The list. By default it shows active recommendations: ones waiting on a decision, accepted ones waiting to be applied, and applied changes that regressed. They are ranked by estimated value, the rule's confidence, and a discount for the classified risk, and a regressed change always comes first. You can filter by:
- Status: needs decision, accepted, needs attention, ignored, being measured, validated, dismissed, or all.
- Area: CI/CD, models and executors, cloud, connectors, local compute, storage, performance, reliability, security, privacy, or cost. A recommendation can be in more than one area. A security tradeoff does not put it under Security; only a security gain does.
- Risk (the classified risk, so a recommendation a factor or policy raised is listed under its raised level) and a text search, sorted by priority, estimated savings, confidence, or newest.
Potential versus realized. The top of the page has two separate blocks:
- Potential impact is estimated. It adds up the estimates of open recommendations that nothing has applied yet: monthly dollars, avoidable cloud minutes, avoidable API requests, avoidable tokens, and the number of local compute opportunities. A rule's window total is projected to a 30-day month using the scan window recorded with that opportunity. An estimate with no recorded window, or priced per call, is left out of the monthly figure and counted in a note.
- Realized impact comes only from measurements recorded after a change was applied and validated, against a credible baseline. An estimate never counts here, and a change that is still being measured counts in neither block. See Projected against realized savings below for how each figure is worked out.
The detail view. Open a recommendation to see what Orgabot observed, why the rule considers it inefficient, the current and proposed behavior (including the exact settings it would change), every estimated effect and how it was worked out, the classified risk with every factor and the reason for each raise, confidence and the governing policy, whether the change may be applied automatically, security, reliability and quality tradeoffs, the supporting evidence with links, and the full history of decisions and changes.
Actions. Each one is recorded in the opportunity's history, with you as the actor:
| Action | What it records |
|---|---|
| Apply | Your acceptance. It does not change any configuration: you make the change through the usual governed path (a mission, a pull request), then use Record as applied, optionally with the PR, commit, or config revision. Not offered when an override denies or suppresses the recommendation, or when the optimization policy in force blocks it. |
| Review details | Opens the detail view. A newly detected recommendation is marked as reviewed (proposed). |
| Ignore | A decision to leave it for now. It leaves the active list until something about it changes, and it is still listed under Ignored. |
| Reject | A decision that you considered the proposed change and declined it. Nothing is changed, it is listed under Dismissed, and later scans keep it rejected. Apply re-proposes it if you change your mind. Offered while it is detected, reviewed, or accepted but not yet applied. |
| Never suggest again | Suppresses it with no end date. Later scans keep it suppressed. Suggest again undoes this, unless a compliance constraint suppressed it. |
| Recommend rollback | For an applied change that regressed: records that rolling it back is recommended. Measurement does this on its own when it finds a regression. |
| Record as rolled back | Records that the regressed change was rolled back, optionally with the revert PR, commit, or restored config revision. Like Apply, it changes nothing itself: you make the rollback through the governed path first. Apply can propose the change again later. |
Measuring applied changes
When you Record as applied, Orgabot starts measuring the change right away. No model is involved: it compares telemetry from before the change with telemetry from after it.
- The window and the baseline. The baseline is the affected workflow's telemetry in the window before the change. The observation window is the same length after it. The length is the recommendation's own detection window when it has one, otherwise 7 days (never less than 1 or more than 30). "The affected workflow" means the recommendation's scope (organization, role, mission, task), the telemetry its rule reads, and the target it changes (a CI workflow or job, a tool, a connector, a worker pool). The detail view lists exactly what was selected.
- What is compared. Every applied change is compared on the same common metrics: dollar cost (cash only), cloud minutes and compute, API requests, quota refusals (rate-limited responses), token spend, latency, failure rate, retries and fallbacks, reviewer approval, rework and human intervention, and local CPU and memory. Each is a figure per unit of work, a mean, or a rate, so a busier week does not look like a change. A rule can add metrics specific to what it changes, such as CI cache hit ratio, polling passes a day, or log volume a day.
- Sample sizes. By default a metric is judged only when both sides have at least 10 observations. It counts as changed only when it moved by at least 10% (and, for a rate, at least 5 percentage points) and the 95% interval of the difference excludes zero, so a noisy metric whose average moved is still read as unchanged. You can change these numbers per scope (see Configuring validation below).
- Other changes are recorded. If another optimization was applied or rolled back over the same scope, an optimization policy was set or revoked over it, or the volume or mix of work shifted, the measurement says so next to the result.
- The verdict. Nothing is concluded while the window is open. When it closes, any guarded metric that got worse moves the recommendation to regressed, whatever happened to cost: "cost fell 42% but rework rose 31%" is a regression. A metric the change was expected to improve that did improve, with nothing worse, moves it to validated, and its measured figures then count in Realized impact. A tradeoff the estimate accepted (for example slower bursts for a smaller pool) only counts as a regression if it is worse by twice the usual margin. Local CPU and memory are shown but never decide the verdict, because everything else on the machine moves them too.
- Against the projection. For cost, cloud minutes, API requests and tokens, the measured saving per day is compared with the saving the estimate projected. If the change improved but realized less than a fifth of what was projected, the result is improved, but well below the projected impact: it stays under measurement for you to decide rather than being validated.
- Against your optimization policy. Each metric's regression threshold is scaled by how much the optimization policy in force at the recommendation's scope weighs that objective. Under Balanced every threshold is the configured one. A profile that weighs an objective at 10 halves its threshold, and one that weighs it at 2.5 or less doubles it. An objective the policy forbids degrading regresses at half the margin, even when the estimate accepted it as a tradeoff. The detail view shows each metric's threshold and why.
- A regression is rolled back, or raised as urgent. When measurement finds a regression, Orgabot restores the previous configuration on its own if automatic rollback is allowed for that change (see Automatic rollback below). Otherwise the recommendation moves on to rollback recommended, marked urgent, with the rollback plan recorded with it, the metrics that got worse, and the reason it was not rolled back automatically. It is listed under Needs attention at high priority and raised in the Mission Inbox. Make the rollback through the governed path, then use Record as rolled back. A recommendation marked regressed any other way gets the same urgent recommendation on the next pass, and is never rolled back automatically, because no measurement shows the regression.
- When the data does not settle it. With too few observations the window keeps extending, up to three times its length. If the expected metrics did not move, the result is no measurable effect. If the change's benefit is not something telemetry measures (a security or privacy gain), the result is not measurable from telemetry. In all three cases the recommendation stays under measurement and you decide; it is never marked validated on thin evidence.
Measurements run on their own every 15 minutes while the dashboard is running, and whenever orgabot optimize scan records. A change recorded as applied any other way enters measurement on the next pass. The baseline and every snapshot are stored, and the detail view's Measured outcome section shows each metric before and after with its sample size, the other changes in the window, and why the verdict is what it is.
orgabot optimize outcomes # every applied change and its measurement verdict
orgabot optimize outcomes <opportunity-id> # every metric before and after, with sample sizes, thresholds, the projection, confounders and any rollback
orgabot optimize outcomes --collect # run a measurement pass nowConfiguring validation
The evaluation window, the minimum sample, and the thresholds can be set at the platform, an organization, a department, a role, or a mission. A narrower scope overrides a broader one field by field, and every verdict records the values it was judged with and the scope that set each one.
| Setting | Flag | Default |
|---|---|---|
| Evaluation window, before and after the change | --window-days <1-30> | The recommendation's own detection window, or 7 days |
| Observations each side needs before a metric is judged | --min-sample <n> | 10 |
| Change that counts as a change | --tolerance <0.01-5> (0.1 is 10%) | 10% |
| Change that counts, for one impact dimension (dollar, resource, latency, quality, reliability, security, privacy) | --threshold <dimension>=<0.01-5>, repeatable | The tolerance |
| Points a rate must also move | --min-rate-delta <0-1> | 0.05 (5 points) |
| How far a thin window may extend | --max-extension <1-10> | 3 times its length |
| Share of the projected impact a validation must realize | --min-projection-share <0-1> | 0.2 |
orgabot optimize validation show --organization <org-id> --role <slug> # the settings in force, where each comes from, and each dimension's regression threshold under the policy in force
orgabot optimize validation set --organization <org-id> --window-days 14 --threshold quality=0.05 --reason "quality matters most here"
orgabot optimize validation set --organization <org-id> --reason "back to the defaults" # no setting: the scope inherits againA new window length applies to changes that start measuring after you set it. Sample sizes and thresholds apply from the next measurement pass.
Applying recommendations automatically
By default Orgabot only recommends. Nothing is applied automatically until you opt in, and you opt in per scope (everything, an organization, a department, a role or a mission) and, if you like, per category. A policy reads like "apply low-risk optimizations with at least 95% confidence and at least $5 a month in projected savings":
orgabot optimize auto-apply set --platform --on --max-risk low --min-confidence 0.95 --min-monthly-savings 5 --max-config-scope platform --reason "apply low-risk savings automatically"Where a change takes effect decides who must opt in. Orgabot holds the opt-in and the configuration scope limit against the scope a change actually configures, not only the scope of the recommendation. Today's supported changes are Orgabot's own optimization rule settings, and a rule setting applies to every organization, so they configure the whole platform even when the recommendation came from one role. Such a change is applied automatically only when automation is on at the platform and the platform's --max-config-scope is platform; an opt-in for one organization or role, however broad, leaves it for you. Every other control is held at both scopes too, and the stricter one wins: if the platform's policy requires approval for cost changes, a role's policy that requires none cannot waive it for a cost change that takes effect platform-wide, and the same goes for the allowed categories, risk, confidence, savings, setting count, change windows and rollback behavior.
The controls. Every one is optional, and a narrower setting wins over a broader one, with a category setting winning over the scope-wide one at the same level.
| Control | Flag | Default once on |
|---|---|---|
| Opt in or out | --on, --off | Off. --off at any scope turns automation off there and everywhere beneath it, whatever a narrower setting says. |
| Allowed categories | --categories cost,latency | Every category. |
| Maximum risk | --max-risk low or medium | Low. High and critical changes are always a person's decision. |
| Minimum confidence | --min-confidence 0.95 | 0.95. |
| Minimum expected benefit | --min-monthly-savings 5 | $0 a month projected. |
| Maximum configuration scope | --max-config-scope role, --max-changes 3 | Organization, any number of settings. A change that configures a wider scope (where the change takes effect, see above), or more settings, waits for you. |
| Approval-required categories | --approval-categories cost | None beyond the approvals your policies already require. |
| Change windows | --window mon-fri@09:00-17:00 --time-zone America/New_York | Any time. A window whose end is before its start runs past midnight. |
| Reversible only | --require-reversible, --allow-irreversible | Reversible only. |
| Rollback behavior | --rollback plan or automatic | plan: Orgabot must be able to undo it without a person. automatic also requires that a regression of it is rolled back automatically (see Automatic rollback below). |
What can never be loosened. Whatever a policy says, a recommendation is left for you when it raises any security, reliability, privacy or compliance concern, degrades any of them, or would break a compliance constraint; when its own policy, an override, or an approval-required category needs a person's approval; when your optimization profile's automation is manual or its risk is above the profile's automation cap; when someone rejected, ignored or rolled it back before; or when Orgabot has no supported way to make and undo the change. Today the supported changes are Orgabot's own optimization rule settings, so recommendations that change a CI workflow, a connector or a cloud resource stay recommendations.
Checking one recommendation. orgabot optimize auto-apply check <opportunity-id> lists every check, marks the hard gates, and says which setting decided each control. The same checks appear in the recommendation's Auto-optimization section in the Optimization Center. The answer is the same every time for the same recommendation, policies and time.
How a change is applied. Orgabot reads every setting first and stops if any no longer holds the value the recommendation saw. It sets each one only if it still holds that value, checked and written in one step, so a change you or another process makes in the meantime is never overwritten. If one fails it puts back any it already set, again only where the setting still holds the value it wrote, so a newer value someone else set is left alone. Then it reads every setting again to confirm. Then the recommendation moves through the same steps as one you approve: proposed, accepted and applied, recorded as applied by the auto-optimization policy, with each setting's old and new value in its history. It starts being measured at once, so it is validated, or regresses into the rollback path, like any other change.
What is recorded. Each automatic application is recorded with the policy as it stood, every check, the values before and after, the projected savings, and the outcome. orgabot optimize auto-apply history lists them with how each is doing now and totals the projected against the realized savings. A change that could not be applied is recorded as failed, nothing is left changed, and Orgabot does not retry it on its own.
orgabot optimize auto-apply show --organization <org-id> # the policy in force and where each control comes from
orgabot optimize auto-apply set --platform --off --reason "change freeze" # off everywhere
orgabot optimize auto-apply set --organization <org-id> --category cost --off --reason "cost changes need a person"
orgabot optimize auto-apply run --dry-run # what the next pass would apply, without applying it
orgabot optimize auto-apply history # every automatic change and how it is doingThe recovery sweep runs the auto-apply pass every 15 minutes, applying at most five changes per pass.
Automatic rollback
When a measurement shows that an applied change regressed, Orgabot can restore the configuration the change replaced without waiting for you. This is separate from applying changes automatically: a change you applied by hand can be rolled back automatically, and a change applied automatically can still need you to roll it back.
A regression is rolled back automatically only when every one of these holds. Otherwise it gets an urgent rollback recommendation instead, which names each one that failed.
| Check | What it requires |
|---|---|
| Measured regression | A measurement concluded the change regressed. A change marked regressed by hand is never rolled back automatically. |
| Significant | The worst metric that got worse moved at least the configured multiple of its regression threshold (1 by default). The thresholds are the ones the measurement used, already scaled by your optimization policy. |
| Turned on here | Automatic rollback is not turned off for the change's scope or category. |
| Policy permits | The recommendation's policy, after any override, permits automatic rollback. Built-in rules do not, so nothing is rolled back until a policy or an override allows it. |
| Reversible | The change is recorded as reversible. |
| Plan runs without a human | Its rollback plan says the platform can run it. |
| Prior values recorded | The application recorded every setting it changed, with the value it replaced. |
| Supported mechanism | Orgabot has a supported way to restore every one of those settings. Today that is Orgabot's own optimization rule settings (turning a rule back on or off). Anything else, such as a CI workflow or a cloud resource, gets an urgent recommendation. |
| Risk below high | The change's classified risk is low or medium. A high or critical change is never rolled back automatically unless every mechanism involved is explicitly safe for high-risk changes. |
How a rollback runs. Orgabot first reads every setting and stops if any no longer holds the value the change set, because someone changed it since and restoring it would overwrite their change. It then restores each setting in order. If one fails, the ones already restored are put back, so the configuration is never left half rolled back. Finally it reads every setting again, and only a read-back equal to the previous value counts. A rollback that does not complete becomes an urgent recommendation.
What is recorded. Every response to a regression, rolled back or not, is recorded with the reason, the measurement and metrics behind it, every check, the recommendation's state before and after, each setting's applied value, previous value and the value it reads now, and the outcome. A rollback also moves the recommendation to rolled back with the restored values in its history, so the audit history and the rebuilt configuration show it.
You are told. An automatic rollback appears in the Mission Inbox and the notification bell as a warning for 14 days, or until you acknowledge it. An urgent recommendation, or a rollback that failed, appears as critical until the change is rolled back or you decide otherwise. The recommendation's Rollback section in the Optimization Center shows every check and every response.
Turning it off. Automatic rollback can be turned off for everything, for one scope (an organization, a department, a role or a mission), or for one category at a scope. A narrower setting wins over a broader one, and a category setting wins over the scope-wide one at the same level, so you can turn it off for an organization and back on for one role. Turning it on never permits what a recommendation's policy does not.
orgabot optimize auto-rollback show --platform # the switch in force and the supported mechanisms
orgabot optimize auto-rollback set --platform --off --reason "change freeze" # off everywhere
orgabot optimize auto-rollback set --organization <org-id> --category cost --off --reason "cost changes need a person"
orgabot optimize auto-rollback set --organization <org-id> --role <slug> --on --reason "this role is fine"
orgabot optimize auto-rollback set --platform --min-severity 2 --reason "only clear regressions"
orgabot optimize auto-rollback check <opportunity-id> # every check for one recommendation, now
orgabot optimize auto-rollback history [<opportunity-id>] # every response to a regressionProjected against realized savings
The Projected against realized section of the Optimization Center, under the two impact blocks, keeps four kinds of value apart and never adds them together:
| Block | What it counts |
|---|---|
| Potential | Estimates for open recommendations that nothing has applied yet. |
| Projected | For every applied change still in effect, the estimate that was in force when it was applied. If the recommendation is revised later, this figure does not move. |
| Realized | What was measured after the change, for validated changes only. This is the only block called a saving. |
| Measured, not counted | Changes measured with enough data that were not validated: they regressed, fell well below the projection, or had no effect. They are shown so nothing is hidden, and they count nowhere else. |
Each block has the same lines, so a gain that is not about dollars sits next to the dollar figures:
- dollars per month and per year
- cloud minutes and compute avoided
- API calls and quota avoided
- tokens avoided
- compute shifted local (the cloud minutes, tokens and dollars a local compute recommendation moved off the cloud)
- storage and log volume per month, and data stored
- latency, quality, reliability, security and privacy, counted as improved or degraded rather than added up, because each is a level in a different workflow
The section also shows what share of the projected dollars was realized, compared only over the changes that were validated, and a table with one row per applied change.
How the figures are worked out.
- Monthly figures use a 30-day month and yearly figures a 365-day year. A projection that a rule stated as a total for its scan window is spread over that window first. An estimate with no recorded window is left out and listed.
- A realized monthly figure is the change measured per unit of work, times the units of work done per day after the change, times 30. A rule metric already measured per day, such as log volume a day, is multiplied by 30. Latency, data stored and rates are reported as the change in their level and never turned into a yearly figure.
- No saving without a credible baseline. A realized figure is shown only for a metric that had the minimum number of observations both before and after the change. A metric that did not move by more than the noise counts as zero, not as its raw difference. If there was no telemetry before the change, too little of it, or the observation window is still open, the change is marked as such and nothing is claimed.
In the detail view, the Projected against realized section lists every line with its projected and realized figure, its status and sample sizes, and how each figure was worked out. It keeps the estimate as first detected next to the one in force when the change was applied, says whether the baseline was credible, lists the other changes seen in the window, and links the evidence: the telemetry the estimate rested on, the PR or commit recorded when it was applied, and the stored measurement.
orgabot optimize savings # potential, projected, realized and measured-but-not-counted, line by line, then every applied change
orgabot optimize savings <opportunity-id> # one change: every line projected and realized, the projection kept, the baseline, methodology and evidence
orgabot optimize savings --json # the same, as JSONOptimization history and audit trail
Every recommendation Orgabot makes stays on record after it stops being active. A rejected, suppressed, validated or rolled-back recommendation keeps every event that led there, so you can answer later what Orgabot recommended, why, on what evidence, who accepted, rejected or suppressed it, what configuration changed, what impact was expected, what was measured, and whether it was validated, regressed or rolled back.
Open it with Audit history above the list of recommendations in Insights > Optimization Center (or go to #insights/optimization/history). It shows:
- Every event, newest first: each lifecycle transition and decision with its time, who made it and through which surface; each configuration change with the setting's old and new value; each estimate revision; and each outcome measurement with its verdict and the before-and-after comparison behind it. A measurement, and the validation or regression it led to, names the stored outcome it came from.
- Filters for scope (a scope includes everything beneath it, so an organization covers its departments, roles and missions), category, actor, status, event type and a date range, plus a search box.
- Recommendations in this history: for each one, what it observed, what it proposed, the evidence, the expected impact, and what was actually measured.
- Configuration reconstructed from the trail: for every setting an optimization changed, its value now, its original value, the change that set it, and the full timeline of applies and rollbacks. If a change did not start from the value the previous one left, the setting is flagged with a gap, because something changed it outside Orgabot in between.
Record as applied records the settings the recommendation changes, with each one's value before and after, and Record as rolled back records the reverse, so the configuration can always be rebuilt from the trail. A recommendation's own detail view has the same correlated history under Lifecycle history, with a link to open it in the audit history.
The history is read from the recommendations and their stored outcome measurements. Nothing is copied into a separate audit store, and no task content is kept for the sake of the audit: an outcome contributes only its aggregate comparisons, never the telemetry behind them.
orgabot optimization history # the newest 200 events across every recommendation, active or not
orgabot optimization history --scope organization:<org-id> --category cost --from 2026-10-01 --to 2026-10-31
orgabot optimization history --actor operator --status rolled_back --kind configuration_change --json
orgabot optimization configuration # every setting optimizations changed: its value now, originally, and its timeline
orgabot optimization configuration --at 2026-10-15T00:00:00Z --target ci_workflow:<owner/repo/file>Optimizer overhead and analysis budgets
Finding savings costs something too, so Orgabot keeps the optimizer from spending more on analysis than the savings are worth. It works in layers, cheapest first:
- Deterministic rules (
orgabot optimize scan): arithmetic over telemetry on this machine. No model, no cash. A scan runs these first. - Statistical analysis: the cohort comparisons some rules run (after the deterministic rules, in the same scan), and the outcome measurement passes. Still no model, no cash.
- AI analysis (
orgabot optimize analyze): a model reviews a recommendation and says whether the inefficiency looks real, what could go wrong, and what to check first. It runs only when every check below passes, and it is advisory: it never changes a recommendation.
Before any model is called, these checks run, in order:
| Check | What it decides |
|---|---|
| AI mode | off, local_only, prefer_local (the default) or any. |
| Still undecided | Only a recommendation awaiting a decision is investigated. |
| Policy | A recommendation the optimization policy blocks, or an override denies or suppresses, is skipped. If the policy or overrides cannot be read, the recommendation is deferred: an unreadable control never allows a model call. |
| Necessity | Deterministic evidence that is already conclusive (90% confidence by default) needs no AI. |
| Expected value | The estimated monthly dollars times 12, weighted by the rule's confidence, must reach the minimum ($100 a year by default). A recommendation with no dollar estimate is worth $0 here. |
| Where it runs | Where the model's inference runs decides this, not how it is billed. A model whose inference runs on this machine is preferred. Claude Code, Codex and Grok are launched on this machine but send the prompt to their vendor, so they count as remote; they cost $0 in cash because your subscription covers them. API providers are remote and billed per token. A local execution rule in the optimization policy, or local_only, never sends a recommendation to a remote model, so with the providers available today it turns AI analysis off. |
| Cadence | One recommendation is investigated at most every 7 days by default. |
| Share of savings | One recommendation's investigations may spend at most 10% of its expected first-year savings by default. |
| Spend budget | AI spend over the last 30 days stays within every budget in the scope chain: the platform's ($5 by default), and the organization's, department's or role's when set. Each investigation reserves its worst-case cost before the model is called, so two analysis runs at the same time cannot both spend the same remaining budget. The worst case is a ceiling: the output limit is sent with every request. The reservation is then replaced by the actual cost. |
A failed check either skips the recommendation or defers it (unreadable policy, cadence and spend budget), and the reason is recorded. With the defaults, a small installation runs every rule and measurement for no cash, and AI analysis only on a subscription CLI or within $5 a month.
The optimizer's own consumption is recorded per run: time, signals read, model, tokens, and incremental cash. The Optimizer overhead section of the Optimization Center shows it per layer for the last 30 days, with the budget left and the overhead as a share of realized and projected savings. It sits beside the savings and is never subtracted from them. A recommendation's detail view shows each AI investigation under AI analysis.
orgabot optimize budget show --platform # the budget in force, where each value comes from, and every spend cap with what is left
orgabot optimize budget set --organization <org-id> --max-spend 2 --ai local_only --reason "small install"
orgabot optimize budget set --platform --max-savings-share 0.1 --min-value 200 --scan-interval-hours 24 --reason "daily scans are enough"
orgabot optimize overhead # what the optimizer consumed, per layer, next to the savings
orgabot optimize analyze --dry-run # every gate decision and why; calls no model
orgabot optimize analyze <opportunity-id> # investigate one recommendation, if the gate allows itWith --scan-interval-hours set, a recording orgabot optimize scan inside the interval is deferred and evaluates nothing; --force scans anyway. A scan covers every scope at once, so the interval can be set only with --platform.
Risk classification
Every recommendation gets a risk level: low, medium, high, or critical. The rule that found it supplies a default, and Orgabot then classifies the recommendation itself by what applying it would do:
| Factor | What raises the risk |
|---|---|
| Security | Degrading security while other safeguards remain is medium. Skipping, reusing, or moving a security or provenance check is medium. Disabling or removing one, or degrading security in a way that cannot be undone, is high. |
| Reliability | Degrading reliability, or moving work between local and remote environments, is medium. Removing redundancy kept for resilience (a fallback, standby, retry, or backup) is high. |
| Privacy | Sending work off this machine is medium. Degrading privacy is medium, or high when it cannot be undone or sends work off the machine. |
| Compliance | Deleting stored data sooner with no way back is medium. A change tagged with a compliance rule is medium, or high when it cannot be undone. Removing or shortening an audit trail is high. |
| Reversibility | A change its rollback plan cannot undo is at least medium. |
| Evidence confidence | No evidence, or a confidence under 50%, is at least medium. |
As a guide: enabling a safe cache, batching requests, or removing identical duplicate work is usually low. Changing a polling interval, moving eligible CI work to this machine, or changing which executor runs first is medium. Disabling security validation or removing a fallback is high.
Risk only goes up. The classified level is the highest of the rule's default, each factor, and any risk floor the optimization policy sets. Nothing lowers it below what the rule said. If a rule later revises its own estimate down on a recommendation that was high risk, or that is a security, privacy, or reliability change, the earlier level stays in force until an operator revises it. An operator's revision is recorded in the history with their reason.
Where you see it. The Optimization Center list shows the level and, when it was raised, the rule's default. The detail view's Risk and confidence section lists every factor, what each found, every raise and its reason, and whether the change may be applied automatically. orgabot optimization show <id> and orgabot optimize policy explain --opportunity <id> print the same.
What it controls.
- Priority. The Center and alerts discount priority by the classified risk, so a recommendation a factor raised ranks lower than the rule's default alone would put it.
- Surfacing. A policy's maximum risk (
--max-risk) is compared with the classified level. - Automation. A policy's automation risk cap (
--max-auto-risk, medium unless a policy sets it) is the highest risk Orgabot may apply on its own. A recommendation above the cap is still listed and you can still accept it; only automatic application is held back. Executor routing counts as a medium-risk change, so a cap of low stops routing from reordering executors. - Risk floors.
--risk-floor <subject>=<level>makes a policy classify matching recommendations at least that high. The subject is a category (cost,security,reliability, and so on),security_check,irreversible, orexecution_shift. Floors add up from the widest scope down: a narrower scope can add or raise a floor, and an attempt to remove or lower one set above it is refused and shown inpolicy explain.
# Nothing riskier than low applies itself in this organization, and no team below may change that.
orgabot optimize policy set --organization org_senternet --max-auto-risk low --lock maximumAutomaticRisk --reason "Humans apply anything with behavior change"
# Treat any change to a security check, or anything that cannot be undone, as high risk.
orgabot optimize policy set --platform --risk-floor security_check=high --risk-floor irreversible=high --reason "Security review for these"Optimization profiles and policies
A profile says what Orgabot should optimize for. You assign one with a policy at the platform, an organization, a department, a role, or a mission. Every recommendation is then held to the policy in force at its scope before it is recorded, listed, alerted on, or applied.
Profiles. orgabot optimize profiles lists them. Each one is a weight from 0 to 10 for seven objectives (cost, latency, quality, reliability, security, privacy, and local-resource preference) plus hard constraints:
| Profile | Leans toward | Hard constraints |
|---|---|---|
| Balanced | every objective about equally | none (the default when no policy is set) |
| Cost-first | cost | never degrade security or privacy; must be a net gain for this profile |
| Quality-first | quality and reliability | never degrade quality, reliability, or security; risk at most high; must be a net gain |
| Latency-first | latency | never degrade latency or security; must be a net gain |
| Local-first | local execution, privacy, cost | never degrade security, privacy, or local-resource preference; must be a net gain |
| Privacy-first | privacy, security, local execution | never degrade privacy or security; no work moves to a remote environment; nothing applies itself |
| Custom | the weights you give it, starting from Balanced | the constraints you set |
Every profile also caps automatic application at medium risk, so a high-risk change is always applied by a person. See Risk classification above.
A policy can also set the constraints directly: objectives that may never be degraded (--forbid-degrading), recommendation categories to ignore (--ignore-category), where work must run (--execution security_checks=trusted_remote, --execution all=local), a minimum confidence, a maximum risk, an automation ceiling, an automation risk cap (--max-auto-risk), risk floors (--risk-floor), and a minimum alignment score. See Risk classification above for the last two.
How a recommendation is judged. Its alignment runs from -1 to 1. Each objective it improves counts for it and each one it degrades counts against it, scaled by the profile's weight. Moving work onto local compute counts as an improvement to local-resource preference, and moving it to a remote environment counts against it. A recommendation is blocked when it falls in an ignored category, degrades a forbidden objective, moves work against an execution rule, is less confident or riskier than the policy allows, or scores below the policy's floor. A blocked recommendation:
- is not recorded by
orgabot optimize scan, which shows why; - is held back in the Optimization Center: it leaves the active list and the potential totals, a note counts it, and the All view lists it with the policy's reasons;
- never alerts, and Apply refuses it.
A policy never deletes a recommendation: revoke or change it and the Center lists the recommendation again, and the next scan records any finding it had held back. While a policy is in force, the Center's ranking also scales each recommendation by how well it fits the profile.
Scopes and overrides. Policies are read from the widest scope to the narrowest: platform, organization, department, role, mission. A narrower scope overrides a broader one, field by field, unless the broader scope locked that field with --lock. A locked field can only be tightened below it (another forbidden objective, a higher floor, a lower ceiling); an attempt to loosen it is refused and shown. A locked profile or locked weights cannot change at all. A role's department comes from the org chart, so a department policy applies to every role in it.
The examples from the original request look like this:
# The organization prefers local execution whenever practical.
orgabot optimize policy set --organization org_senternet --profile local-first --reason "Prefer local execution"
# Security: provenance and security checks always run in a trusted remote environment, and no team below can change that.
orgabot optimize policy set --organization org_senternet --department security \
--execution security_checks=trusted_remote --lock execution --reason "Provenance checks stay remote"
# One mission is quality-first and ignores cost optimizations.
orgabot optimize policy set --mission <mission-id> --profile quality-first --ignore-category cost --reason "Quality first"orgabot optimize policy explain shows how a scope's policy was reached: each scope in order, what it set, tightened, or locked, what was refused, and which scope set each value. With --opportunity <id> it also shows that recommendation's alignment and whether the policy lets it through, and why. The Center's detail view shows the same thing under Optimization policy. orgabot optimize policy list lists the policies in force, and orgabot optimize policy revoke retires one; the record keeps who revoked it and why.
A policy grants nothing. It can only hold a recommendation back, lower how automatically one may be applied, or change the order recommendations are shown in.
Optimization alerts
You don't have to keep checking the Optimization Center. When orgabot optimize scan records a high-value recommendation, it also raises an alert. Alerts appear in the Mission Inbox and the notification bell as Optimization available, for example:
Optimization available: Skip the duplicate security audit. You ran the same security audit locally and in GitHub Actions on 47 of the last 50 merges. Estimated savings: 312 CI/cloud minutes/month (estimate, not measured). Risk: Low, confidence 90%.
Which recommendations alert. Only ones waiting on a decision whose priority (estimated value x confidence, discounted for risk; the same ranking the Center uses) reaches the Center's high band. A recommendation never alerts when:
- its rule is turned off (
orgabot optimize disable <rule-id>); - an override in force denies or suppresses it, whether that override names its scope, its category, or the recommendation itself;
- the optimization policy in force at its scope blocks it;
- a compliance constraint suppressed it, or you chose Ignore or Never suggest again.
When you have several alerts, they are listed highest priority first. An alert disappears as soon as you act on it or something above suppresses it, without waiting for another scan.
No repeated alerts.
- One alert per finding: the same rule against the same workflow, connector, or model in the same scope is one alert, even across scans or under two records.
- Later scans don't raise an alert again. It comes back only if, 14 days or more after it was raised, the estimate has grown by at least half.
- At most two new alerts in any 24 hours. The rest wait for a later scan, highest priority first.
- At most three alerts are shown at once.
Controls on the alert.
| Control | What it does |
|---|---|
| Apply | The Optimization Center's Apply, through the same audited lifecycle: it records your acceptance and changes no configuration. |
| Details | Opens this recommendation's detail view in the Optimization Center, with all of its evidence. |
| Ignore | The Center's Ignore. |
| Never suggest again | The Center's Never suggest again: suppressed with no end date, so no later scan alerts on it. |
| Acknowledge | Hides this alert and records no decision. The recommendation stays in the Center. It is not raised again unless the estimate grows as described above. |
Estimates, not savings. Every figure on an alert is labeled estimated and says it is not measured, and the card states that measured savings appear in the Optimization Center only after a change is applied and validated. Alerts only cover recommendations that nothing has applied, so none has measured savings yet.
orgabot optimize alerts lists the same alerts in the terminal, and orgabot optimize scan reports which alerts it raised and why it held back the others.
Executor outcomes and quality
orgabot analytics executors reads those signals and shows how well each executor and model performs, broken down by role and task type by default:
orgabot analytics executors # by role and task type, all recorded history
orgabot analytics executors --by provider,model --since 30d
orgabot analytics executors --by department,complexity --min-sample 20
orgabot analytics executors --json # the full report, including what each rate counts- Segments.
--bytakes any ofrole,department,task_type,task_class,complexity,source_kind,provider,model,auth_mode,execution_location,effort,organization,project. A run that did not record a value shows asunattributed; Orgabot never guesses one. - What each worker run records. Role and department, task type (implement, verify repair, review fix, and so on), task class (the objective shape), where the mission came from, a size band, executor, model, effort, whether the worker ran on this machine, duration, tokens, cost, retries and fallback-chain steps. The objective's text is never stored. The size band (
small,medium,large) is a stated rule over its word count and checklist items (objective_size_v1), not a judgement of difficulty. - Linking verdicts to the work. Each run, review verdict, operator interrupt and redirect records the mission's lifecycle round. That is how a reviewer's "changes requested" or an operator's correction is counted against the executor that produced the work. Verdicts that cannot be linked are counted and shown as unlinked, never assigned to a nearby run.
- The rates. Completion, retry, review acceptance, first-pass approval, human correction, rework and fallback.
--jsonlists the exact numerator and denominator of each. - Small samples. Every rate carries its counts, a 95% interval and a confidence level (low, moderate or high). Below the minimum sample (10 by default), the report shows the counts and says too few to judge instead of a percentage, so two successes out of two never reads as 100%.
- Cost. Actual and nominal cost are shown separately and never added together, with the share that was estimated rather than reported.
Comparing executors within a workload
orgabot analytics performance answers a narrower question than the report above: for one kind of work, does one executor do better than another? It groups runs into cohorts (role and task type by default), and inside each cohort compares every executor with the one that ran it most (the baseline):
orgabot analytics performance # cohorts by role and task type, executors by provider and model
orgabot analytics performance --cohort department,task_type --since 30d
orgabot analytics performance --executor provider,model,effort --min-sample 20
orgabot analytics performance --json # the full analysis and the evidence behind it- Cohorts, not a ranking. Executors are only compared with other executors that did the same kind of work. There is no single global league table.
--cohorttakes the same keys as--byabove;--executortakes any ofprovider,model,effort,auth_mode,execution_location. - Every tradeoff at once. Each comparison reads four dimensions side by side: cost (mean cost per run), latency (mean run duration), quality (review acceptance, first-pass approval, human correction, rework) and reliability (completion, retry, fallback). Each one says better, worse, no meaningful difference or too few to judge, from the 95% interval of the difference. The overall verdict is candidate better, baseline better, tradeoff (better on one dimension, worse on another), no meaningful difference or inconclusive. A finding reads like: costs 41% less per run, completes 27% faster, with no meaningful change in quality.
- Cost is compared like for like. Two executors are only compared on cost when both were priced the same way (both actual or both nominal). Otherwise cost says not compared and why.
- Small samples. An executor with fewer runs than the minimum (10 by default) is flagged
sparse, and the comparison is inconclusive: no percentages are drawn from it. - Different task mixes. If two executors in a cohort ran noticeably different work (by size band, task class, where the mission came from, department or project; the threshold is
--max-mix-gap, 0.25 by default), the cohort is flaggedtask_mix_imbalance. Orgabot then re-checks the comparison within each part of the mix, weighted to the baseline's mix. If the difference holds, the comparison stands with a note. If it does not, or the executors share too little of the mix to check, the comparison is marked confounded and the finding starts with observed, not causal. - Other flags.
single_executor(nothing to compare with),unattributed_cohort(part of the cohort was not recorded),low_linkage(fewer than half the runs could be linked to review verdicts),mixed_billingandestimated_cost. - Evidence.
--jsonlists, for each executor and comparison, the runs, executions, missions, time range and lifecycle round references it rests on, plus every threshold used. This is the same output optimization recommendations read, so a recommendation can always show the numbers behind it. Nothing in this report changes a role's executor or fallback chain.
Executor chain recommendations
The executor_chain_recommendations rule turns those comparisons into recommendations. When orgabot optimize scan runs, it looks at each role and task type on its own and may recommend putting a different executor first for that cohort, with the current one kept later in the chain as its fallback. For example: role=backend-engineer task_type=implement: run claude-code/sonnet first, keep claude-code/opus as fallback.
- Per cohort, never one global ranking. Each recommendation is scoped to one role (in its organization) and one task type. The same two models can get opposite answers in two cohorts. A cohort whose role or task type was not recorded gets no recommendation, because it mixes unrelated work.
- Only on solid evidence. A comparison counts only when it is not inconclusive (both executors have at least the minimum number of runs) and not confounded (see Different task mixes above), and when the cohort's current executor is not simply better.
- Your optimization policy decides. Each alternative is checked against the optimization policy in force at that role. A hard rule can block it: an ignored category, an objective it may not make worse, or where work must run. The profile's weights then give it a score, and only alternatives that score above zero are recommended, best first. So a Quality-first role is offered an executor that costs more but passes review more often, a Balanced role is not, and a Cost-first role is refused it. A local model that is just as good is offered to a role that values local execution, and is blocked where a policy says work must run in a trusted remote environment.
- Five expected changes. Every recommendation states what it expects to happen to cost, quality, latency, reliability, and local/remote resource use. It gives a figure with its 95% interval where the data shows one, and says no meaningful change or too few observations to judge where it does not.
- Evidence you can check. The evidence lists each executor's runs, executions and missions, the sample size behind every metric, the execution references, any flags, and the policy's ranking with the reason each other alternative was not recommended.
- Nothing changes on its own. A recommendation is an ordinary opportunity in the Optimization Center, with a manual, approval-required policy. Apply records your acceptance; you then change the chain through the usual governed path and Record as applied. Reject declines it, and later scans keep it rejected.
orgabot optimize chains shows the same thing for every cohort without recording anything: the executor each cohort runs today, the sample behind each executor, the policy it was ranked under, each recommended change with its five expected changes and whether it has been recorded, and why every other executor was not recommended.
orgabot optimize chains # every cohort, last 7 days
orgabot optimize chains --since 30d --role backend-engineer --task-type implement
orgabot optimize chains --json # the full assessment, including each recommendation's evidence
orgabot optimize scan --rule executor_chain_recommendationsOptimization-aware executor routing
Recommendations wait for you. Routing acts at launch: when a mission starts, the optimization policy in force at that mission can change the order in which its executor chain is tried. The chain is the role's executor followed by its fallbacks, after any executor the mission may not run has been removed.
- It only reorders. Routing never adds an executor, never removes one, and never brings back one the mission was not eligible for. If the first executor in the new order has run out of quota, the mission still starts on the next one that is available, exactly as before.
- Explicit choices are kept exactly. A worker you pinned with
--worker, or a chain passed in explicitly, runs in the order given. - Hard rules win. If a policy says all work must run locally, local executors are tried first and a remote executor is never moved ahead of a local one. The reverse applies for trusted remote, whatever the profile prefers. An executor is never moved ahead if the evidence shows it is worse on an objective the policy says may not get worse, or if the comparison falls below the policy's confidence floor, risk ceiling, or minimum score. Changing which executor runs first is a medium-risk change, so a policy whose maximum risk or automation risk cap is low keeps the configured order. A policy with automation set to manual (for example Privacy-first) changes nothing by itself.
- Evidence, or the configured order. Moving one executor ahead of another on performance needs a conclusive comparison between the two, from the mission's own role doing the same kind of work (see Comparing executors within a workload above). If there is no policy in force, too few runs, a confounded comparison, or the policy store or telemetry cannot be read, the configured order is used and the reason says so.
How each profile chooses:
| Profile | Prefers |
|---|---|
| Cost-first | The cheaper executor, but only while its review results are not measurably worse than the one it would replace. |
| Local-first | A local executor (a coding CLI on this Mac) before a remote API service. This needs no run history, because where an executor runs is known. |
| Latency-first | The executor that has finished this role's work faster, when its quality holds. |
| Quality-first | The executor that passes review more often, even when it costs more. |
| Balanced or Custom | Whichever executor the profile's weights favor across cost, latency, quality and reliability, when its quality holds. |
Every run records why. The mission log shows the decision in plain words, including each comparison it made and why each executor moved or stayed. The decision is also recorded as a routing_decision signal in the optimization telemetry, and every run's worker_run signal carries the reason code (routing) and the decision it belongs to (routing_decision). The reason codes are optimized, operator_pin, explicit_chain, injected_worker, single_rung, no_policy, policy_manual, policy_unavailable, evidence_unavailable, insufficient_evidence, and configured_order_preferred.
To see how a role's next mission would be routed, without launching anything:
orgabot optimize routing --org acme --role backend-engineer
orgabot optimize routing --org acme --role backend-engineer --json # the full decision, with every comparisonReading the numbers
Every figure on Analytics, Budgets & Usage and the organization home says what it counts, in what unit, and over which range (for example Last 30 days (2026-08-29 to 2026-09-27)), or that it is a current, point-in-time count.
- actual is real money billed to an API key. nominal is subscription usage priced at API-equivalent rates: included in your plan, never billed. estimate marks a figure that was computed rather than reported. The two billing bases are never added together: Analytics shows separate Actual cost and Nominal cost cards, charts and breakdowns, and there is no combined "total spend".
- A figure Orgabot could not read says Unknown (or Unavailable) with the reason. A real zero is shown as zero, so the two never look alike.
- A usage bar appears only when there is a real limit to measure against. When usage goes over its limit the bar stops at the end, and the text keeps the real figure: 134% · $268.00 of $200.00 limit and Over limit by $68.00.
- Every chart names its series in a legend, gives each series its own fill pattern as well as its own colour, writes a one-line summary under it, and has a Chart data as a table disclosure with every value, so nothing depends on colour or on hovering.
- A summary tile that opens another page says Open details and looks like a link; a tile that only reports a number does not.