Code Sandbox

v1 status. Both backends run — e2b (a remote VM per run, on the vendor cloud or a self-hosted cluster) and local (the engine host, as a process tree or in a container) — and the engine-fronted OTLP receiver is wired. providers.sandbox is a catalogue: configure either backend or both, and each seat picks its cell with role.sandbox.run_in.

Crewlet agents author code through the run_sandbox Execute tool: the executor calls it with a concrete code task, the engine provisions an isolated sandbox (a real VM with a shell, a filesystem, and a git checkout), and a coding agent — Claude Code or OpenCode — works on the task autonomously inside it. The call is detached: the Execute tool-loop suspends when the job starts, the engine persists the in-flight conversation, and when the job completes — minutes or hours later — the same loop resumes with the coding agent’s findings spliced in as that tool call’s reply. The executor then reports and acts with its own tools (replying on the originating channel, updating the ticket) in the same turn, with full context.

This is how a Crewlet agent implements a feature, makes tests pass, reproduces a bug, or runs a one-off script — anything that needs a shell and a checkout. The sandbox is the isolation boundary: arbitrary, autonomously generated code runs there, never on the engine host, which is why the coding agent can run fully permissioned (Claude Code --permission-mode bypassPermissions; OpenCode’s permissions rule list allowing every action — {"action": "*", "resource": "*", "effect": "allow"} — with only question denied, since a headless run has nobody to answer it and crewlet-ask asks the seat instead).

The moving parts live in internal/sandbox/ — the provider, the manager, the coding-agent runners under codingagent/, the waiter, the coordinator, the pending store, and launch.go, which is the plumbing between the Execute loop and a detached run.

How a coding task runs

For a role gated with role.sandbox.enabled (and an engine-level providers.sandbox) whose executor is not in agent mode, the turn engine exposes run_sandbox on the executor’s tool surface. The executor calls it with a brief — the concrete task, naming the repository and the exact change or investigation — and reports the result in the same turn. An agent-mode executor is never offered it: it does its code work in its own run, whose box reads the gate’s setup, env and mcp (see Where code work runs).

The details, each load-bearing:

Call → suspend. run_sandbox provisions the box, starts the coding agent as a background command, persists a durable run record in the coordination store (the brief, the task, both conversation values — the conversation identity a later answer is matched on and the resume reports back to, and the inbox partition the kick-off trigger arrived in — the trace context, and whether the turn owes anybody an answer), and returns a tools.Result with Suspend set. The Execute loop leaves that call unanswered and the turn ends. The seat is busy from the write that opens the row — before the box is provisioned, and so before the turn can end and the seat’s consumer move on to its next message. The launch also publishes sandbox_run_started to the seat’s control topic, and processing it recounts the seat from the pending store’s own list of the seat’s active runs, which is what carries a busy seat across a restart or a handoff; but nothing waits for it. The seat used to count as busy only once that event came round, on a subscription the launching turn never waits for, so a message drained beside the launching one — or fetched the moment the turn returned — reached a seat that looked free and ran a turn beside the job. A launch that fails after opening the row (no box, a job that will not start, a conversation the turn cannot write into the row) gives the seat back as it ends the row. Nothing is parked in memory; everything the resume needs is in the row.

The record lands in two writes, and the state says which one it is at. The serialized Execute conversation — the messages, including the dangling run_sandbox call — cannot be written by the tool: the runner holds it only until the turn’s frame unwinds, which happens after the call returns. So the launch opens the row as launching and the unwinding turn writes the conversation, flipping it to running in the same compare-and-swap. In between, the job is already executing and the row has nothing to resume into — so nothing polls or claims a launching run. A completion fired in that window used to be claimed against an empty conversation, and the coordinator, finding nothing to re-enter, failed the run: a coding job that finished before the turn unwound destroyed the agent’s whole turn, permanently and with no retry. The window is milliseconds on an idle host and hundreds of milliseconds on a loaded one, and a trivial run is exactly the one that fits inside it.
A seat busy coding takes no other work. While one of a seat’s coding runs is running (or the seat owes an answer’s resume, or a reply a run let go of), the seat receives nothing at all: not new work, not a colleague’s message, and not a person’s answer to a different coding run of the same seat — whether that answer is a chat reply or one given by naming the run. That is the rule, not a cost of the mechanism: an agent in the middle of a coding job is busy with that job, and it takes up the next thing when this one is done, as a person would; resuming a second run beside the first would put two of the seat’s turns in flight at once, each acting for the seat without seeing the other. The answer waits on the broker with the rest of the mail and is delivered — first — when the busy run settles or parks on its own question, which for a long job can be hours. Nothing is lost while it waits (the reply keeps its place and the question its anchor), and a parked run whose box outlives pause_ttl_seconds in the meantime re-seeds from its branch when the answer arrives.
Busy while it runs. While a run HOLDS the seat, the seat’s inbox is held (the sandbox hold): the coordinator takes it as the launch opens the run’s row — and when a node that takes the seat over recovers a run still going, before the mailbox opens — and lifts it when the last such run settles or parks on a question. Held, nothing is fetched: the mail waits on the broker in order, nothing is held against a broker ack window during an hours-long job, no new turn starts and nothing runs at all until the job is done. The agent runs one coding job at a time: the first call that suspends ends the round — and a person’s answer to another of the seat’s runs waits with the rest of its mail, and is the first thing offered when the job settles or parks. A run parked waiting on a human answer holds nothing — the seat works as usual, and its deliveries are offered to that question before they become turns (see Mid-run clarification).

It used to park that mail instead — requeue each delivery onto the inbox and ack the original — with nothing stopping the consumer, so every message waiting on a busy seat was fetched, screened, looked up as a possible answer, republished and acked again and again for the length of the run. A delivery that reaches a held seat now (it raced the hold, or the queue refused to take it) asks for the hold again and is deferred, keeping its place at the head of the inbox.
Completion → resume. The completion event is claimed at-most-once (an atomic status flip on the row) and only for the job it reports: every launch gets a new id on the row and the completion carries it, so a duplicate that arrives after its job parked on a question, or after the resumed executor started the next job, claims nothing. The result is collected, tokens are post-accounted once per run (see Budgets), the run is published once as a phase of its own (see each run is published as a phase), and the suspended loop is rebuilt from the row: the saved messages, the activated tools and loaded skills replayed, and the coding agent’s findings appended as the run_sandbox call’s reply, framed as “the sandbox did the code work; don’t redo it; report / act now”. The same turn_id continues, so the whole sequence renders as one turn on the dashboard — and so does the round count: the loop re-enters the round the executor parked in, so a suspend and its resume share one turn_engine.max_iterations instead of getting one each. A failed resume dispatch hands the claim back, to the status it was taken from, so the redelivered completion can retry, and the suspended loop is never lost. That includes a resume a drain interrupts: the hand-back does not inherit the drain’s cancellation, so the seat’s next owner resumes the turn rather than reaping it as abandoned. The one exception is a resumed turn that broke after it had already written outside the engine (pushed a branch, commented on an issue): a retry would repeat those writes, so the run is settled instead, its box reclaimed and its record deleted, and the seat is freed. Otherwise it hands back only the claim it took: when the resumed executor has already started another job in the turn, or the seat’s next owner has reaped the run as abandoned, the row is left as it stands, because the relaunch replaced the conversation a retry would re-enter and the reap has already ended the run.
What the resumed half writes under. The resumed turn’s tracker and knowledge-base writes derive their operation ids from the same unit of work as the first half’s, minted at the instant that work began, so a first-half write the second half repeats — the retry an unknown asks for — lands once. The exception is a resume that runs more than twenty-nine days after its work began: the operation ledger keeps each row thirty days, and an operation minted before what a node’s ledger swept, whose row is gone, is answered unknown and never published — so minted at that start, every write the resumed half made would be lost on every node. Such a half is rebased: it mints its ids at the attempt instead and records that instant in the fleet’s coordination store under the unit of work, logged as turn_rebased; the price is that a first-half write it repeats is written a second time. Every attempt is judged against its own clock. A resume that fails is retried — by the completion coming round again, or by the person’s next answer, which can be weeks later — and each retry is judged afresh, never held to a decision an attempt took while the start still looked recent. A later attempt inherits a recorded instant while it is within the same twenty-nine days of it, so a failed resume retried on this node, on the seat’s next owner or after a restart derives the ids the attempt before it derived, and so does the resume of the turn’s next run_sandbox call — whatever the earlier attempt or the earlier half wrote is not written twice. The line is drawn a day short of the thirty so that no sweep passes the instant while one attempt is still working. A turn with no unit of work (a scheduled fire) seeds its ids from its run and mints them at the run’s start, and is rebased by the same rule, under its run. The same rule covers a trigger dispatched long after it arrived — see Turn Engine.
Reuse across calls. The box is a per-Execute-phase resource. On collect it is paused (snapshotted), not destroyed; if the resumed executor calls run_sandbox again — a follow-up fix, a clarification answer to incorporate — the tool reattaches to the same paused box and checkout (clearing the previous run’s result markers, keeping the disk state) and suspends again. When the resumed Execute finishes without another call, the box is torn down. The two are told apart by the launch id, not the row’s status: a new job that finishes, or parks on a question of its own, before the resumed turn has returned keeps its box and its tail.
Publishing a build from the box. While the resumed Execute holds the box, create_artifact and publish_artifact read the run’s output from it: source: {sandbox: "dist"} publishes a directory as an artifact’s whole file set, and from_sandbox one file, each relative to the workspace. The engine reattaches the box the turn’s own run used (a job still working, or another seat’s run, is no box), lists the directory through the box itself with links never followed, holds every listed path to the workspace and the published-path grammar, and streams each file through the artifact upload pipeline, which reserved its bytes first and reads them back verified before the version is decided. The box is never handed an upload URL or a credential. Once the Execute phase ends and the box is torn down, a publish from it is refused not_running.
A settled run has no record. When a run ends, whether its resumed Execute finished or it was lost, its box is reclaimed first and then its record is deleted from the coordination store. The record’s bucket has no age (a parked run can wait days, and an age would reap the only thing that knows a billed box exists), and the completion poll and every seat recovery read all of it, so a finished run kept there would be read on every one of those passes for the life of the deployment. What a run ended as is recorded elsewhere: the resumed turn’s own phase events, or a sandbox_run_failed event naming why it was lost. The box goes first on purpose: a record that outlives its box is reaped by the seat’s next recovery, while a box that outlived its record would be named by nothing.
An ending is decided before it is done, and announced exactly once. Ending a run is several effects in several systems — a person’s reply handed back to the seat’s inbox, the box reclaimed, the loss announced, the record deleted — and a node can stop between any two. So the ending is first decided on the run’s record (ending), in one compare-and-set under the ending’s terms: what it announces, under an identity of its own, and whether a person’s reply goes back. From then on the record takes no other write — no claim, no take of an answer, no hand-back of a claim, no relaunch, no new answer, no park — so nothing can move the run off the ending, and whoever reads the record next (this node’s retry, or the seat’s next holder) finishes that ending from wherever it stopped, never one of its own. The loss is announced (sandbox_run_failed) before the record is deleted, under an id derived from the run and its ending and the instant it was decided, so a repeat after a crash is the same event to the event store — whose key is that (time, id) pair — and to the fleet’s reads, which merge rows on it. The announcement used to be made only after the delete: a delete that landed and reported a failure, or a node that stopped between the two, left the loss announced by nobody, and the seat’s next holder, finding the claim the delete had not taken, announced it as abandoned_tail whatever it had been ended for. A broker that refuses the announcement keeps the ending, and the seat’s mail waits behind it until it is made. The box goes after the decision, never before it. An ending is made by a node that read the run and decided it is over, and that read can be stale: a node that lost the seat while settling a claim it had taken used to kill the box first, off its own read, and only then had its ending refused under the seat’s next holder’s lease — after that holder had given the claim back to its answer, so the run it then resumed had lost its checkout while its record still named the box. The reclaim now rides the decided ending, so the store checks the ending’s lease against the run as it stands before anything is killed, and an ending refused there leaves the box to the run’s holder; whoever finishes the ending — this node or the seat’s next holder — reclaims the box the record names before deleting it. A node that loses the run that way says nothing about it either: what became of the run, and of a reply on it, is announced by whoever holds it now.
Review sees the whole turn. The tool-execution log from the suspended segment (the run_sandbox call and anything before it) is replayed to the front of the resumed loop’s log, so Review’s evidence check sees the code work happened. The clone/test/push commands run inside the box, never as engine-visible tool calls — a successful run_sandbox call is the code work.

There is deliberately no synchronous “block the turn until the sandbox finishes” mode: a turn that held its concurrency slot and its broker message for a real coding job’s runtime would trip the inbox ack window and the shutdown drain. A short job in the detached model simply completes fast.

Enabling it

Two switches, both required: an engine-wide provider and a per-role gate. Without the provider, no role can use the sandbox regardless of gates; without the gate, a role never sees the run_sandbox tool.

Engine provider — `providers.sandbox`

A sibling of providers.llm, live-reloadable, with secrets kept as ${VAR} references and resolved at construction time:

providers:
  sandbox:
    # THE CATALOGUE: which boxes exist. Configure either, or both.
    e2b:
      api_key: "${E2B_API_KEY}"   # required whenever e2b: is present — the API
                                  #   authenticates every call
      domain: ""                  # set → self-hosted / local cluster; empty → the cloud
      template: ""                # empty → the prebuilt template for the coding agent;
                                  #   ALSO how a remote box is sized (below)
    local:
      state_dir: ""               # empty → $CREWLET_SANDBOX_LOCAL_HOME,
                                  #   else ~/.crewlet/sandboxes
      image: ""                   # required only if some seat runs in a container
      runtime: auto               # auto | docker | podman
      run_args: []                # how a container box is SIZED
      runtime_env: {}             # extra environment for the docker/podman CLI
                                  #   itself, never the box (below)

    default_run_in: e2b           # direct | container | e2b — where a seat that
                                  #   names none runs
    default_coding_agent: claude-code   # claude-code | opencode
    default_timeout_seconds: 900  # box TTL / keepalive window — NOT a run cap (below)
    default_pause_ttl_seconds: 1800     # omit for this default; 0 = never pause. how long a blocked run's paused box is
                                        #   held before the reaper takes it
    default_max_turns: 0          # agentic-round cap for coding runs; 0 = uncapped
    setup: []                     # engine-wide provisioning steps (see Setup steps)

The block is a catalogue, not a mode. It says which boxes exist; role.sandbox.run_in says where each seat’s work goes. This is the shape’s whole reason: a company has exactly one providers.sandbox, so a company-wide type: meant choosing the engine host for the seat that needs the operator’s own subscription login and for the seat whose generated code must never touch that host. Those are different decisions about different work. Leave the block out entirely to run with no sandbox at all — there is no type: none any more, because an absent block already says it and a present one that meant “off” read as “on” in every document that carried it.
An empty block is refused. providers.sandbox: {} configures nothing, and without this rule it would apply cleanly: every sandbox-enabled seat plans around a box it never gets, and code work quietly never happens. Give it e2b:, local:, or fake: true.
default_run_in is required unless the catalogue names exactly one cell — or nothing would read it. local: alone is two cells (it serves both direct and container), and there is deliberately no implicit answer: direct runs the coding agent as the engine’s own user, and e2b mints billable VMs, so neither may be chosen for an operator who did not say which they meant. Only an e2b:-only catalogue (and fake: true) resolve on their own. What makes the default required is something that would fall to it: a sandbox-enabled seat whose executor is not in agent mode and that names no run_in, or an agent-mode entry with no cli.run_in — an agent-mode seat’s own gate places nothing, so its run_in never falls to the default. A company whose every such seat and entry names its own cell needs no default at all — that is the shape the catalogue exists for — and validation refuses the silent seat or entry where it is written, offering default_run_in as the other remedy.

The one thing an ambiguous catalogue may not do is reach nothing at all: a default, a seat’s run_in or an agent-mode entry’s cli.run_in has to name some cell. A backend is built only for a cell something reaches, so a catalogue nothing names builds none — a block that reads as configuration and configures nothing, which is the reason an empty block is refused too. Every catalogue that resolves a default reaches it, precisely so a seat added later has somewhere to go.

api_key, domain and template live under e2b:, where they can only mean what they say — a domain: at the top of the block used to read as a cluster address beside type: local and configure nothing. The key is required whenever e2b: is present, including against a self-hosted cluster: domain changes which API is talked to, never whether it authenticates.
local.image, runtime, network, run_args and runtime_env are container-only, and are checked against the cells your seats actually reach: an image is required when some seat runs in a container and refused when none does, because there it looks like configuration and configures nothing. Required unconditionally it would refuse a perfectly good direct-only company; unchecked, the seat’s first coding run fails at container create, minutes into a turn.
default_timeout_seconds is not a run-time limit. It is the box’s TTL / keepalive window: the completion poll refreshes a running box’s TTL to this value on every tick, so the clock never kills a live job. It is effectively the orphan-reclaim grace — how long a box outlives an engine that stopped heart-beating (a crash) before E2B reaps it. A coding job is never force-stopped on a timer.
default_pause_ttl_seconds bounds how long a run blocked on a human answer keeps its paused box. E2B holds a paused sandbox indefinitely — there is no provider-side TTL — and bills for the snapshot, so expiring it is the engine’s job: past this age the completion poll reaps it and the run re-seeds from the pushed branch when the answer arrives. 30 minutes trades a bounded snapshot bill against exact resume for the replies that come back quickly; raise it if your teams answer in hours. 0 means never pause — the box is torn down the moment the run blocks, for zero snapshot cost — and omitting the key is how you take the 1800s default. The two are different settings and the field distinguishes them, which it did not before: read as a plain number, an explicit 0 was indistinguishable from an absent field and was quietly mapped onto the default, so a company that asked for zero snapshot cost was billed for every paused box for half an hour. Negative values are rejected here: an unbounded pause is the leak the knob exists to prevent. A role inherits this default by leaving role.sandbox.pause_ttl_seconds out entirely, and a negative role value is rejected for the same reason. Correctness never depends on the box: the durable state is the pushed branch plus the persisted row.
default_max_turns is the only engine-side bound on a runaway coding agent. A coding job is deliberately never force-stopped on a clock (see default_timeout_seconds above), and the completion poll refreshes a running box’s TTL on every tick precisely so that it cannot be — so nothing else stops an agent that is thrashing rather than working. This caps its agentic rounds instead, which is a measure of work done rather than time spent. 0, the default, is uncapped: the right number depends on the tasks a company gives its agents, and a cap set too low truncates real work mid-task, which is worse than the runaway it was guarding against. There is no companion budget cap — the fleet’s own token meter already post-charges a collected run against the seat’s budget, and a second ceiling denominated in the operator’s dollars would just disagree with it. A role overrides this with role.sandbox.max_turns.
Sizing is template, not a limits knob. There is deliberately no default_limits. A box’s vCPU and RAM are properties of its template, fixed when the template is built; the sandbox-create API accepts no resource arguments at all, and disk is not exposed in either place. An engine-side limits field could only ever be parsed and dropped — an operator setting memory_mb: 16384 would silently get whatever the template had. To give agents bigger boxes, build a template with the resources you want and name it in template:; a container box is sized with local.image and local.run_args instead. There is no per-run or per-seat template field: sizing is a property of the backend, configured once, and the Spec field that used to sit in front of e2b.template was set by nobody while looking exactly like a wired knob.
Hot-reload, in every direction, with no restart. The runtime under the catalogue — the coordinator that holds which seats are mid-run, and the completion poll — is a fact about the process, so it is brought up once, by the first company that reaches a sandbox cell: at boot, or at whichever apply brings one (the apply’s sandbox_runtime stage — see Configuration). That covers a node started with no company, the quickstart’s own route, and a company that adds providers.sandbox later. The same apply prepares the seats the node already holds, which were taken before there was anything to prepare: each attaches its completion topic and recovers the runs recorded on it, exactly as a seat taken afterwards does on its way in, and a seat whose preparation fails is handed back so its next acquisition runs the whole of it. From then on a revision only swaps the manager under the runtime (Coordinator.SetManager), and three rules keep the runs already in flight whole across the swap:
- The poll follows the swap. It asks the coordinator for its current manager on every tick, so a job launched after a reload is polled through the backend that made it, rather than through the catalogue the node first ran.
- A dropped cell stays reachable for what is already on it. A run’s placement is on its row, so a revision that removes a cell (local: taken out, say, while a container job is still running) carries that cell’s backend as retired: a running job is still polled, collected and reclaimed, and a parked one still reaped, through the backend that made it — while no new box is ever provisioned there. A seat still pointing at the removed cell is refused at validation, as it always was.
- Removing the block keeps the last manager. A company with no providers.sandbox validates no seat that launches a run, so it registers no run_sandbox; the runtime and its last manager stay for the runs still in flight, which still have to be finished through it.
A catalogue that cannot be built is refused before anything on the node changes — on a node that runs no sandbox yet as on one that does — so a company whose sandbox-enabled seats would plan around a box nobody can mint is never published.

Per-role gate — `role.sandbox`

Absent → the role never sees the sandbox option. Present with enabled: true → the run_sandbox tool appears in the role’s Execute surface, and the executor is taught when to use it — unless the executor runs in agent mode, which is never offered run_sandbox and is not taught it: there the gate only feeds the executor’s own box.

roles:
  - name: Senior Engineer
    mcp_env:
      github: { Authorization: "Bearer ${GITHUB_TOKEN_SENIOR}" }
    sandbox:
      enabled: true
      run_in: ""                        # direct | container | e2b
                                        #   empty → inherit providers.sandbox.default_run_in
      coding_agent: ""                  # empty → inherit providers.sandbox.default_coding_agent
      # pause_ttl_seconds:              # leave out → provider default; 0 → never hold a paused box
      # max_turns:                      # leave out → provider default; 0 → uncapped
      env:                              # env injected into this seat's runs, ${VAR}-expanded
        GITHUB_TOKEN: "${GITHUB_TOKEN_SENIOR}"   # this seat's own PAT (git-auth recipe + gh)
        NPM_TOKEN: "${NPM_TOKEN_SENIOR}"
        DATABASE_URL: "postgres://${TEST_DB_HOST}/app"
      optional_keys: [DATABASE_URL]     # left out of the run when its value resolves to nothing;
                                        #   every other key's must resolve, or run_sandbox is refused
      mcp:
        servers: [github, atlassian]    # which of the role's MCP servers the coding agent gets
      setup:                            # per-role provisioning steps, applied after the
        - name: node                    #   engine-wide providers.sandbox.setup
          # No `corepack enable`: it writes pnpm's shim beside the Node binary,
          # and a setup command runs as the box's own account — `user` on E2B,
          # the engine's on the host for `direct` — which cannot write there
          # when Node was installed as root. `corepack pnpm …` needs no shim.
          commands: ["command -v corepack >/dev/null || { echo 'no corepack on PATH' >&2; exit 1; }"]
          brief: "Node and corepack are installed; run the repo's pinned package manager through corepack (`corepack pnpm install`), never a bare `pnpm`."

role.sandbox.env is where external-service tokens are declared — the engine names no tool-specific variable of its own (see Credentials and the run environment). By convention a seat points GITHUB_TOKEN / GITLAB_TOKEN at the same ${VAR} as its mcp_env code-host credential, so PRs/MRs land under the agent’s own identity — see GitHub and GitLab.

Where code work runs

role.sandbox.run_in names one cell of the catalogue. The three cells and the two backends behind them:

`run_in`	Backend	Runs as	Isolates state	Isolates the host	Needs
`direct`	`local:`	the engine user, on the host	yes	no	the coding CLI installed on that host
`container`	`local:`	a PID in a Docker/Podman container	yes	yes	`local.image` with the coding CLI
`e2b`	`e2b:`	a fresh remote VM per run	yes	yes	`e2b.api_key`
`self`	none	inside this seat’s own executor run	—	—	an executor in agent mode

self is the one cell with no backend, because it is the executor’s box. A seat whose executor is a coding CLI in agent mode already holds a shell, an editor and a checkout; provisioning a second box beside it would give the seat two filesystems with the work in the one the turn cannot see. So such a seat is never offered run_sandbox: the tool is not on its surface, not in its catalogue and not one activate_tool reaches, and no second box is provisioned. self is what documents that; a seat that leaves run_in empty, or names no gate at all, behaves the same, and the launcher’s own refusal of self is only a guard behind it. self is refused on any other runtime — there the seat has no shell of its own, and self would read as a working choice while quietly turning code work off — and it cannot be a company-wide default_run_in, which would do exactly that to every seat that is not in agent mode.

An agent-mode executor is placed by its own entry’s cli.run_in rather than by the seat’s, because where that runtime runs is a property of the runtime: the CLI’s subscription login is on the engine host, so a local cell reaches it directly while a remote one needs the headless token instead. A seat that names none takes providers.sandbox.default_run_in, resolved at launch rather than at config time, so changing the catalogue reaches every seat that wrote nothing without rewriting their blocks. The entry’s cell is reached exactly as a seat’s is: the backend behind it is built for it, and run_in: container there needs local.image just as it would on a seat. Such a seat’s own gate therefore places nothing: it is never offered run_sandbox, so its role.sandbox.run_in may be self or nothing, and any other cell — which would read as a choice and reach no box — is refused on the write, naming the entry that places the box, as is a coding_agent other than that entry’s own CLI. What the executor’s box does read off the gate — its setup, env and mcp, and its pause_ttl_seconds and max_turns — it reads whether or not the gate says enabled. Beyond refusing what a seat would never read — those two, and a gate that is off with run_in, setup, env or mcp under it on a seat not in agent mode — validation refuses the impossible and nothing else: a run_in whose backend is not configured — on a seat or on an agent-mode entry — a company default naming an unconfigured backend, a sandbox-enabled seat whose executor is not in agent mode, or an agent-mode entry, that names no cell against a catalogue with no default, an agent-mode entry in a company with no catalogue at all, and a seat with an enabled gate in such a company — each of which otherwise reads as working configuration and fails inside a turn.

A run remembers its cell. The placement is written onto the run’s durable row before the box exists, and reconnect, teardown and the pause reaper all read it from there. It is not re-derived: the turn that collects a run may be a different process on a different node, days later, with the company configuration applied again in between — and reconnecting to a remote box through the local backend does not error usefully, it reports a box that has vanished, so a job that is still running would be abandoned as gone. A seat that names no cell is given the company’s default at launch, so every row names one.

Sandbox backends

The provider layer is pluggable behind the sandbox.Provider interface (internal/sandbox/protocol.go). Two backends run code, and one stands in for them:

E2B cloud (e2b:, no domain). Sign up at e2b.dev, export E2B_API_KEY from the dashboard. The engine talks to the documented REST API directly — there is no Go SDK to install, and no optional dependency for any of this — and uses E2B’s prebuilt claude / opencode templates (the coding-agent CLI preinstalled), picked automatically per coding agent when template is empty.
Self-hosted / local E2B (e2b: + domain). E2B’s infrastructure is open source (e2b-dev/infra); set domain to your cluster’s domain and the same code path talks to it — one field is the whole cloud↔self-hosted switch, because the control-plane address and every box’s own hostname are both derived from it. The cluster still issues its own key.
Local (local:). The engine host itself — run_in: direct (a process tree) or run_in: container (Docker/Podman). No remote account and no API key, and the coding agent can use the subscription login crewlet llm login already established. See below.
fake: true — deterministic in-process stubs (internal/sandbox/fake.go): an in-memory filesystem, scripted coding-agent results, no network. The unit-test substrate; it does not run real code. It answers every cell, and is refused beside a real backend — the double replaces the backends rather than joining them, so a real block underneath it would read as configuration and configure nothing.
No sandbox at all — omit providers.sandbox. Sandbox-gated seats fall back to the native executor loop, and run_sandbox never appears. (A seat with an enabled gate in that company is refused, rather than silently offered nothing.)

The closed set run_in is checked against is exactly the set the engine can construct — a test walks it, because a value the config accepts and the engine cannot build fails at the first coding run and nowhere earlier. The engine builds a backend only for the cells your company actually reaches, read from the same computation validation uses: built eagerly instead, it constructed a container backend for a company whose seats all run direct and failed the apply demanding an image the validator had just refused as a field nothing would read.

Two planes, and they behave differently on purpose. The control plane (api.<domain>) mints, kills, pauses and re-times boxes under a bounded timeout, because minting a VM is a real allocation with a real cost when it is abandoned. The in-box plane — running commands, reading and writing files — is reached on each box’s own hostname and carries no overall deadline at all: a coding agent’s foreground command legitimately runs for many minutes, so what bounds it is the gap between output frames rather than the length of the call (ten minutes of silence). A file transfer is bounded the same way and more tightly: envd answers from a file that already exists, so a read or write that moves no byte for sixty seconds — the control plane’s own request budget — is abandoned and says so, rather than holding that box’s poll for as long as the process lives. (The poll of a box is bounded as a whole as well, and no box’s poll waits on another’s — see the completion poll.)

Boxes are created with secured access. A secured box’s in-box agent runs a command or serves a file only for a request carrying that box’s own access token, and the engine sends it (X-Access-Token) on every command and file call. E2B hands the token back on the create, on a reconnect and on a read of the box alike, so the engine never stores it. Every box the engine creates is secured. A template whose in-box agent predates secured access (envd below 0.2.0) is refused at the create, with E2B’s own message to rebuild it.

Local sandboxes

The local: backend runs the coding agent on the machine running crewlet run. It exists for the case the subscription LLM backends create: you already logged a coding CLI in on that host, and you want code work to use that same login — no E2B account, no API key, no token to mint.

providers:
  sandbox:
    local:
      state_dir: ""               # empty → $CREWLET_SANDBOX_LOCAL_HOME,
                                  #   else ~/.crewlet/sandboxes
    default_run_in: direct        # direct | container — no implied answer
    default_coding_agent: claude-code

Both shipped examples pair a local: catalogue with run_in: direct, and the difference between them is worth reading side by side. examples/nimbus.company.yaml runs opencode as the coding agent inside the box and has a code host to push to, so its runs open merge requests. examples/nimbus-claude-cli.company.yaml takes it one step further and has no code host at all: its three engineering seats run a cli-agent entry in agent mode, so the executor itself is the coding CLI’s own agentic loop in a box that reuses the operator’s Claude login, and each seat’s role.sandbox.run_in: self says its code work rides that run rather than starting a second box.

That second shape is worth saying out loud, because a sandbox and a code host are often assumed to come together. A sandbox with nowhere to push is still a place to run code: a spike, a benchmark, a reproduction, an SDK example that has actually executed. Add a code host later and the same runs start opening merge requests; nothing about the sandbox block changes.

local: alone still needs a default_run_in (or a run_in on every sandbox-enabled seat whose executor is not in agent mode and a cli.run_in on every agent-mode entry), because it serves two cells and choosing direct is choosing to run an autonomous coding agent as the engine user. It carries no mode of its own any more: direct and container are a choice about one seat’s work, and a block-level mode could only ever answer for every seat at once.

	`run_in: direct`	`run_in: container`
Runs as	the engine user, on the host	PID inside a Docker/Podman container
Isolates state (per box)	yes	yes
Isolates the host	no	yes
Needs an image	no	yes (`image:`, no default)
A setup file outside the box’s home	refused: it would be the host’s	refused: only the home is mounted
Setup commands run	on the host, as the engine user	in the container: as the engine’s uid on a rootful runtime, as container root on a rootless one
Sizing	the host	`run_args: ["--cpus", "2", "--memory", "4g"]`

Both modes isolate state. Each box gets its own directory under boxes/ with two sides: home/, which is the agent’s — HOME, the XDG variables and TMPDIR are pointed inside it, its working directory is an empty per-box workspace there, and in container mode it is the one directory mounted into the container — and records/, which is the engine’s: the logins the box was seeded with and a digest of each (so a reconnected teardown knows what to write back, where, and against what), the keepalive stamp, a direct job’s process group, and the env file. The two used to be one directory, which let a containerised job rewrite what the engine knew about its own box — name any file the engine’s user could write as the “shared login” to copy a box file over at teardown, or make the env file a link the next exec followed out of the box. A box’s environment is allowlisted (PATH, locale, TLS trust, proxy — never the engine’s own keyring, its Tier A token values, its SLACK_BOT_TOKEN or its collector credential in OTEL_EXPORTER_OTLP_HEADERS). The allowlist is the same one every child the engine starts is handed — a stdio MCP server and a coding CLI get it too. In container mode the run environment is handed to the runtime through an --env-file in the box’s records rather than as -e KEY=value, because a process’s argv is world-readable on the host (/proc/<pid>/cmdline, every ps) and that environment carries the seat’s LLM key and whatever token role.sandbox.env declares; the runtime’s client reads it on the host, so the container has no need of it. A direct job runs as the engine’s own user with no mount, so it can reach its box’s records too — which is what direct is documented to be below.

The container runtime’s own CLI gets an explicit environment too. Every docker/podman invocation — run, exec, pause, rm, ps, info — is started with exactly the environment a stdio MCP server whose command is docker or podman gets, which that table lists name by name: the host allowlist, the engine user’s home, temporary and XDG directories (where the runtime keeps its config, and XDG_RUNTIME_DIR, where a rootless daemon’s socket lives), and the runtime’s own configuration — the DOCKER_* family, Podman’s, and the few settings they read by name. Nothing else of the engine’s reaches it, or any plugin and credential helper it runs.

Whatever else the runtime needs, declare in local.runtime_env. What a runtime’s helpers read is open-ended, so the fixed list cannot carry it: the credential helper that pulls your image from a private registry needs its cloud’s role, profile or key file (docker-credential-ecr-login, docker-credential-gcr), and a runtime plugin may read settings of its own. Name them there, exactly as you would an MCP server’s env: — a ${VAR} resolves from the secret store first and the engine’s own environment behind it, when the backend is built on an apply — and a declared name wins over the fixed list’s value for it:

providers:
  sandbox:
    local:
      image: ghcr.io/acme/crewlet-coding:1
      runtime_env:
        AWS_ROLE_ARN: "${REGISTRY_PULL_ROLE}"         # docker-credential-ecr-login
        AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
        GOOGLE_APPLICATION_CREDENTIALS: "${GCR_KEY_FILE}"   # docker-credential-gcr

It is the runtime’s environment and never the box’s: nothing in it reaches the coding agent except through an argument you add to run_args. That is the one consequence for local.run_args worth knowing — a bare -e NAME copies NAME from the runtime CLI’s environment, so it copies only a name the fixed list carries or runtime_env declares, and anything else arrives in the box unset. crewlet validate warns about each such -e NAME where it is written. Put what a box needs in role.sandbox.env (or a setup step’s env), which reaches the box directly and is what that is for.

Boxes are removed at teardown, and a crash-orphaned one is reaped on the next create, but only when it is genuinely orphaned. Two independent questions are asked. Is anything running in it? A live process group that is still the box’s own job (direct) or an existing container, read from the operating system rather than inferred. Has anything touched it? The completion poll refreshes a keepalive stamp on every tick, which is what Sandbox.SetTimeout means on this backend, exactly as it refreshes an E2B box’s TTL. A box is reaped only when both say no: nothing running, and nothing has touched it for a whole default_timeout_seconds. A run blocked on a human answer is covered by the first (its process group is stopped, not gone) and bounded by pause_ttl_seconds, which is the knob that owns that wait. A reaped box still gives back any login the run refreshed before it was abandoned.

A pid is not an identity. A direct box records its job’s process group as the leader’s pid together with the leader’s start time, read from the kernel’s own process record (/proc/<pid>/stat with the boot id on Linux, sysctl kern.proc.pid on macOS) while the engine still holds the unreaped process. Every later act on that record, from a different process or hours on, first asks whether the group is still that one: the kernel hands a pid out again once its process is gone, and a bare pid read back from disk may by then lead somebody else’s processes. A recorded group that is gone or now belongs to a later process keeps no box alive and receives no signal, so a teardown, a kill or a clarification pause can never reach a stranger’s tree. When the kernel’s record cannot be read, the reaper keeps the directory and a signal is withheld and logged (local_sandbox_signal_withheld), because neither can tell a live job from a recycled pid.

Only container isolates the host. In direct the coding agent runs with its own tools enabled as the engine user, so it can read what that user can read and reach what its credentials reach. That is the normal bargain of a local mode, and it is the right one on a workstation or a dedicated VM — but it is not a security boundary, and Crewlet will not pretend otherwise. On a shared host, or anywhere the work is untrusted, give that seat run_in: container — or point the whole company at it with default_run_in:

providers:
  sandbox:
    local:
      image: ghcr.io/acme/crewlet-coding:1   # must have the coding CLI installed
      runtime: auto                          # auto | docker | podman
      network: ""                            # "none" cuts the box off entirely —
                                             #   including from its own LLM
      run_args: ["--cpus", "2", "--memory", "4g"]
    default_run_in: container

The box directory is bind-mounted at /home/user — the same home E2B uses — so in-box paths, setup steps and briefs are identical across the two backends. The container is started with --init so a finished detached job is reaped rather than left as a zombie (which the completion probe would read as still running).

The container runs as the engine’s own user, and the mount is proved before the box is used. A bind mount is one directory shared by two processes, so both of them have to be able to manage what the other writes. Under a rootful Docker or Podman the container’s root is the host’s root, and every directory the box creates in the mount comes out root-owned — the engine, which is not root, can then neither install the agent’s crewlet-ask shim into it nor reclaim the box’s disk at teardown, so box directories accumulate on the host with no error anywhere. Crewlet therefore passes --user <uid>:<gid> on a rootful runtime and not on a rootless one, where the user namespace already maps container root onto the invoking user and --user would map it into the subuid range instead. The runtime is asked which it is (docker info / podman info); an unanswerable probe is treated as rootful, because that mistake fails loudly at creation while the other is silent.

If your image genuinely needs its own user — one that installs packages at run time, say — put --user 0:0 in run_args. Operator flags are spliced last and both runtimes take the final occurrence of a repeated flag, so yours wins. Be aware you are taking back the problem above.

Every container box is then proved once, at creation, before anything runs in it: the engine writes a token the box reads back, and the box creates a directory the engine writes into and removes. That catches the ownership case and one other that is otherwise invisible — a runtime pointed at a daemon on another host (DOCKER_HOST, a forwarded socket) bind-mounts that machine’s filesystem, so the agent finds no brief, no credentials and no shim. A box that fails the proof is torn down and the error names the cause; use the e2b provider for a genuinely remote runtime.

Setup commands get a provisioning budget, not a control one. Each command may run for timeout_seconds (default 600). Real provisioning — a dependency install, a cold image pull, a large clone — takes minutes, and without its own budget these inherited the backend’s control-plane timeout and were killed, failing the whole acquisition. Raise it for a step you know is slow; lower it for one that should be instant, so a hung command surfaces as a named setup failure rather than eating the turn. A failure names the step and what in it failed — the file it could not write, or the command’s position and either the status it exited with or that it reported none — and never the command text: the engine resolves nothing in a command, which the box’s shell expands from the run environment, but a recipe may still spell a credential out literally, and the failure is logged and handed back to the agent.

Setup steps and the local cells. Neither local cell writes a setup step’s file outside the box’s home. direct has no filesystem virtualisation, so a file anywhere else would be the engine host’s own — its real /usr/local/bin — and is refused with an error naming the mode; and its home is a directory minted for each box, so no absolute path a document can write is inside it. container writes a file on the host side of its one mount, the home at /home/user, and refuses any other path. A relative path is under the home in either cell, and so is a path starting ~/, which names the home in every cell as it does on e2b; a path that climbs out of the home with .., names the home itself, or names another account’s home (~name/…) is refused. So name a file a step needs in the box relative to its home — .npmrc, .local/bin/tool, or ~/.npmrc — which is the one form every cell writes in the same place. File paths in a setup step are not shell-expanded, so $HOME does not work there; a step that needs a file’s absolute path has a commands entry, which does run in a shell, expand "$HOME" — the git-auth recipe writes its helper that way, because git is given the helper’s absolute path.

A step whose file a local box would refuse is refused when the document is written — by crewlet validate and every write path — rather than at the seat’s first box, where it used to surface as a setup failure on every box that seat opened. The check is the boxes’ own rule, the same function both local cells call before they touch a path, applied to every box a seat actually runs in: its run_sandbox box in the cell its gate names (or default_run_in), or, for a seat whose executor runs in agent mode, the one box that executor runs in, placed by its entry’s cli.run_in (or default_run_in). A box in e2b is judged too, by the shapes its in-box agent refuses for certain — another account’s home (~name/…), and the home itself or a directory above it (see files under setup steps) — so an engine-wide step writing /usr/local/bin is still accepted in a company whose seats all run in e2b. The refusal sits at the file in the step that writes it, names the step, the path and the first seat whose box it fails (and how many more), and is an admission rule: a stored revision that keeps such a step still applies, reported as a warning, so a rolling upgrade never takes a node off the fleet’s revision over it. What it cannot see is a link an existing directory on the engine host holds; the box refuses that one itself, when it writes.

A setup command is not confined the way a file is. It runs in the box’s shell, and under direct that shell is the engine host, as the engine’s user: a step that installs a system package installs it on the host, or fails for want of the rights to. Under container it runs inside the container — as the engine’s own uid on a rootful runtime, which an image’s system directories usually refuse too, and as container root on a rootless one. Have such a step check for the tool it needs and name what to install where, and provision a box’s system toolchain deliberately: on the host for direct, in the image for container.

Credentials. When the role’s resolved sandbox LLM provider is a cli-agent entry, the local backend seeds that provider’s credential files — or, for OpenCode, the login document its export printed, which the run imports and exports again (OpenCode) — into the box before the run, and writes one the run refreshed back afterwards — OAuth access tokens expire in hours and most vendors rotate the refresh token with them, so discarding the rewritten file would log the fleet out at the next expiry. The write-back is a compare-and-swap against what the box was seeded with, recorded (as a SHA-256 of the bytes the box was given) in the box’s records at creation: a file is written back only where the box’s copy differs from its seed — the run refreshed it — and the shared login still holds that seed. An idle box therefore writes nothing (an OpenCode box’s export reproduces its import byte for byte, so it would otherwise have put its stale seed back over a login the host had refreshed meanwhile), and a box whose run refreshed while the shared login also moved — another seat’s refresh, a new crewlet llm login — keeps the shared one, which is the newer of the two, and logs local_sandbox_credential_superseded with shared=changed, naming the file and its path, never a value. The comparison and the write are one step under the shared credential directory’s lock (<state_dir>/credentials.lock, beside it), which the CLI backend’s seats and crewlet llm commands take for their own writes; a teardown waits at most 10 s for a writer holding it. A credential the shared directory did not hold when the box was seeded is never created from a box, and nor is one the operator has since removed with crewlet llm logout — a refresh the run made of that one is dropped, and logged as local_sandbox_credential_superseded with shared=removed.

Pause / resume works. A run blocked on a human answer is genuinely suspended: direct SIGSTOPs the job’s process group, container issues docker pause. Both hold memory, which is exactly why the same default_pause_ttl_seconds reaper bounds them as it bounds a billed E2B snapshot.

What local mode does not give you: template-based sizing (use run_args, or the host), provider-enforced network egress policy (use network: plus your own firewall), and a fresh machine per run — a direct box shares the host’s installed toolchain, which is usually the point.

Coding agents

Two runners ship behind the sandbox.Runner interface, selected by role.sandbox.coding_agent (falling back to providers.sandbox.default_coding_agent). Both are handed the same brief and return the same structured result, and on an API entry the engine treats them interchangeably. A cli-agent entry is the exception: it runs only under its own CLI’s runner (see A subscription counts below).

Claude Code

Runs headless — claude -p "<brief>" --output-format stream-json --verbose --permission-mode bypassPermissions — so it prints every message as it happens, one JSON object a line, into its own event stream (stream.jsonl in the box’s work directory). That stream is what a person watching the run sees and what its transcript is built from; --output-format json printed one object when the run ended and nothing before it, and in print mode the CLI writes nothing to stderr either, so a run on it showed nothing while it worked and left no transcript after. (--verbose is not optional: print mode refuses stream-json without it.) The stream’s last line is the result message, which the wrapper copies to result.json after the CLI exits — never through a pipe beside it, whose exit status would be the copy’s rather than the CLI’s — and that line maps directly onto the result (final text, session id, token usage, cost; an error subtype such as error_max_turns and its errors are the failure). A stream that ends on anything else is a run that stopped before it said how it ended, and reads as one rather than as a silent success. A run whose wrapper never made the copy — its process group killed, by an OOM or a host restart, so the poll ended it on the wrapper’s liveness — is read from the stream’s own last line, which is what the copy would have held: the result, where the CLI printed one before the wrapper died, and otherwise what it was saying when it stopped. It used to read as a run that “produced no output” beside a transcript of everything it did. A result file the box refuses (one that is not a regular file, say) is read from that line the same way, and the refusal is still said; a CLI that exited before printing anything at all — not installed, a bad flag — leaves both empty, and reads as what it is.

The run’s input is its whole prompt: the vendor’s input_tokens counts only the uncached remainder, so the engine adds cache_read_input_tokens and cache_creation_input_tokens to it and carries both as the cache’s share, the same way its own Anthropic provider does. Read alone, the uncached figure put a fraction of every coding run’s prompt on the budget counter.

Claude Code speaks the Anthropic API only, so it needs an Anthropic-compatible credential. The sandbox’s LLM derives from the role’s provider chain: role.llm_sandbox → llm → default. Point role.llm_sandbox at an anthropic providers.llm entry: its key becomes ANTHROPIC_API_KEY and a custom base_url on it ANTHROPIC_BASE_URL, so an Anthropic-API gateway works. A seat that signs Claude Code in itself — CLAUDE_CODE_OAUTH_TOKEN (a Pro/Max plan), ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, or a CLAUDE_CODE_USE_BEDROCK / CLAUDE_CODE_USE_VERTEX / CLAUDE_CODE_USE_FOUNDRY toggle in role.sandbox.env — gets neither of the entry’s two variables: Claude Code ranks an API key above a plan token, so the company’s key beside the seat’s plan token would silently move the seat onto metered billing, and the company’s endpoint beside the seat’s own key would send that key to the company’s gateway. A role whose resolved provider is an openai or openai-compatible entry has nothing Claude Code can read — not its key, and not its model, an OpenAI-side id claude --model refuses — so the launch fails with a SandboxModelError (see Failure modes) unless the seat signs Claude Code in itself as above, in which case the run carries none of the entry and runs on Claude Code’s own model choice (ANTHROPIC_MODEL in role.sandbox.env picks one).

A subscription counts — on its own CLI. A cli-agent entry’s model is written in the grammar of the CLI it drives and its sign-in is that CLI’s, so code work on one runs only under that CLI’s runner: a claude-code entry under Claude Code, an opencode entry under OpenCode (role.sandbox.coding_agent: opencode). The model is handed to the run exactly as the entry’s text calls hand it to the CLI. Any other pairing — an opencode text seat that enables a sandbox and leaves the runner at its Claude Code default, or a CLI with no runner at all (codex, gemini-cli, hermes, pi, …) — is refused, on the write that makes it and again at launch with a SandboxModelError; point role.llm_sandbox at an API entry for such a seat. When the pairing matches, the environment that signs that CLI in is exported into the box: the headless subscription token (CLAUDE_CODE_OAUTH_TOKEN, minted by crewlet llm login <key> -capture-token, so Claude Code there bills your Pro/Max plan — no API key anywhere), the key of an auth.mode: api-key entry, the variables inherit-env forwards, and, for opencode, which reads its model provider’s key from its own environment, the credential-named variables of its cli.env. This works on every backend including remote E2B, because each is one scoped, revocable variable. The login itself — Claude Code’s credential files, OpenCode’s exported login — deliberately never leaves the engine host for a remote box: it carries a refresh token whose rotation is shared fleet state. So a claude-code entry with no headless token, or an opencode entry signed in only by crewlet llm login, needs a local cell — run_in: direct or run_in: container — which is seeded with the login and writes a refreshed one back. A codex, gemini-cli, kimi-code, hermes or pi entry cannot back code work in any cell: no runner reads it, so point role.llm_sandbox at an API entry.

OpenCode

Drives OpenCode 2 — the @opencode/cli package, whose opencode --version prints opencode v2.…. 1.x (npm’s opencode-ai) cannot run it, and a box that has 1.x is refused by name before anything runs (this box runs OpenCode 1.18.35; the opencode runner drives OpenCode 2 …) rather than failing on 1.x’s usage text. Install 2 in your box image; E2B does not say which release its prebuilt opencode template carries, so a template that refuses this way needs template: pointed at one built with @opencode/cli. OpenCode is provider-agnostic: it works with any OpenAI-compatible endpoint, so it reuses the LLM provider the role already has — no extra secret. An org whose only provider is OpenAI-compatible should keep default_coding_agent: opencode rather than pay for a parallel Anthropic credential.

A run is a short script around a server of its own, because 2.0 cannot be driven correctly in one command. opencode run alone attaches to a background service — detached from the run, on a port compiled into the binary that every box on a host shares, outliving the run and keeping the environment (so the credentials) of whichever run started it. With --standalone it starts a private server inside the run instead, and sends the brief at once — before the run’s MCP servers have connected, and 2.0 fixes a step’s tools as the step starts, so measured on 2.0.32 the first step of every run with a server declared was offered none of the seat’s bridged tools. So the run:

refuses a 1.x box, and imports the box’s login if it was given one (below);
starts opencode serve on a loopback port of its own and waits — at most 60 s — until it has printed both where it listens and the password it minted, naming whichever of the two it never printed. The server’s password is always its own: opencode serve takes one from OPENCODE_PASSWORD or OPENCODE_SERVER_PASSWORD and then prints none, so the run unsets both before anything starts — a box environment carrying either (a seat’s role.sandbox.env, an image built for OpenCode) used to wait out the whole minute and fail saying the server never listened — and hands the minted one to its own client calls as OPENCODE_PASSWORD, the variable OpenCode’s client reads first. The server is started without it, so nothing the agent runs inherits it;
asks it (opencode api mcp.list) until every MCP server the run declares has left pending, for at most 60 s, twice the 30 s OpenCode gives one server to list its tools — past that it starts without the straggler and says so on stderr. Each server is looked for by its name exactly as OpenCode’s answer spells it, &, <, >, U+2028 and U+2029 literal, so a server named R&D tools is seen the moment it settles;
runs the brief against that server, on stdin — OpenCode re-quotes an argv prompt, so a brief passed as an argument reached the model wrapped in quotes with every quote inside it escaped — under a session it names, with a title, which stops 2.0 making a second billed model call to write one;
prints the session’s usage report as the stream’s last line, writes the box’s refreshed login back, and stops the server. The run’s exit status stays the CLI’s own.

The script is plain POSIX sh, written for every sh a box image is likely to have — dash (Debian, Ubuntu), bash in its POSIX mode (Fedora, RHEL, Arch) and busybox ash (Alpine, which OpenCode 2’s musl builds are for).

How the run’s --model is named depends on the entry. An opencode cli-agent entry’s model is passed exactly as written — the same <provider>/<model> (anthropic/claude-sonnet-5, openrouter/anthropic/claude-sonnet-5) every text call on that entry passes — and is signed in by that provider’s own key from the entry’s cli.env or the seat’s role.sandbox.env, or by the entry’s login in a local cell. An API entry’s model is addressed as anthropic/<model> for an anthropic entry and openai/<model> for an openai one, OpenCode’s built-in providers reading ANTHROPIC_API_KEY or OPENAI_API_KEY from the run environment, or as crewlet/<model> when the entry sets a base_url (below). A key the seat names under that same variable in role.sandbox.env replaces the entry’s and is used at the entry’s endpoint.

A login follows a run into a local cell as OpenCode’s own export. OpenCode 2 keeps its logins in the database it keeps its sessions in, so the entry’s login travels as the document opencode auth export printed (Subscription LLM Backends): a direct or container box is seeded with it at ~/.crewlet/exported-credentials, and the run imports it before its server starts. A token OpenCode refreshes mid-run then lives only in that database, and every teardown of a run still in flight — a seat retired mid-run, a launch settled as failed after its job started, the pause reaper — kills the job’s group outright (SIGKILL, or docker rm -f for a container box), so nothing in the box runs at its end. So the run exports the database back over the document while it works, through its own server every 30 s, as well as once more when it ends; a SIGTERM, SIGINT or SIGHUP to the run exports once more before it goes, exiting 143, 130 or 129. Thirty seconds is the window a kill can still lose a refresh in: an OAuth access token lives an hour or more, so under one refresh in a hundred falls inside it, and an export through the server costs about 0.14 s (measured on 2.0.32). An export replaces the document only where it differs and holds a login — a whole, non-empty JSON array, never the [] a database that dropped its credential prints — and nothing is exported where the box was given none. The box’s teardown then writes the document back to the shared login under the rule every local box’s login follows (Local sandboxes). A remote box is never given it.

The runner writes OpenCode 2’s own configuration into the checkout (opencode.json; OpenCode takes no config flag, so the file’s location is the wiring). When the role’s provider pins a custom base_url it declares a custom provider (crewlet) with OpenCode’s native package for the entry’s wire (@opencode/ai/providers/openai-compatible, or …/anthropic-compatible), an explicit settings.baseURL and the exact model, addressed as crewlet/<model>. The baseURL is the entry’s base_url as that package expects one: an openai or openai-compatible entry’s as written, and an anthropic entry’s with /v1 added — the engine and Claude Code read an Anthropic base_url as the host root and add /v1/messages themselves, while OpenCode’s package posts to <baseURL>/messages, so the root handed over as written reached a path no gateway serves. Declaring the provider is what bypasses OpenCode’s catalog and the vendor default endpoint, both of which would otherwise break a custom gateway. The provider names the variable its key is read from (env: ["ANTHROPIC_API_KEY"]), so the secret rides the sandbox env and is never written into the file. The same file sets permissions to allow every action and deny only question (headless runs have no human to approve a prompt — a gate left at ask is auto-rejected — and the one tool that asks a person is replaced by crewlet-ask), share: disabled, update: disable (a CLI that updates itself mid-run swaps the tool under the brief while the seat’s turn is suspended waiting on it) and snapshots: false (a git snapshot of the checkout at every step, for an undo a headless run never offers). Each MCP server is declared under mcp.servers with codemode: false, so its tools are offered to the model directly as <server>_<tool> rather than only through Code Mode’s JavaScript execute tool — which a model may never take to, and which the transcript would record as execute whatever it reached — and a remote server with oauth: false, since the bridge authenticates with the run’s token and a headless run cannot complete an OAuth flow. That file is written on every run, including one that names no provider and no server: a run that wrote none would not fall back to the vendor’s defaults, it would inherit whatever the previous run left on a reused box, MCP block and dead per-run bridge URL included.

Its tokens are the session’s own totals. run --format json drops the final step’s counts, so the run asks its session (opencode api session.get) once it ends and the collection reads that: the whole prompt, the cache’s two shares included, and the output with the reasoning OpenCode reports beside it added back, as every provider bills it — plus OpenCode’s price, as it reported it (zero where it has no pricing for the provider, which the record reads as no price). A run whose report is missing reports zero rather than an estimate.

Runs are uncapped; each run is published as a phase

A detached coding job is never force-stopped on a wall-clock timer. The waiter refreshes the box’s TTL every poll tick (keepalive), so a legitimately long run is free to finish; the TTL only reclaims a box the engine has stopped heart-beating for. Completion is detected by tracking the job itself — see Durability and completion.

Because the brief instructs the coding agent that it is not the last step, it writes a structured findings report to a known file before stopping; collect prefers that file for the result text (it survives a run whose process group died before it printed its final message, and a tool-only run that streamed no final message). The report names what the run delivered, a line of its own for each branch it pushed and each pull or merge request it opened — Delivered: fix/retry-backoff, Delivered: https://git.example.com/acme/api/pull/42 — and those lines are the run’s delivered refs, in the report’s order: a branch as well as a pull request, on any host and any scheme. The engine keeps no list of code hosts to match against, because which hosts a box can push to is set by the seat’s own environment and setup steps, which it deliberately never names, and a seat can push to a host no integration block mentions. A line counts when what follows the label is a URL carrying no credentials, or a name git accepts for a branch, alone on the line — a list marker, emphasis or backticks around it are fine, and Delivered: the fix for the flake is a sentence, not a ref. A report that names none is read for pull and merge request URLs by their shape, on any host and port (/pull/N, /pulls/N, /-/merge_requests/N, /pull-requests/N, /pullrequest/N) — a guess, since a pull request the run only read looks exactly like one it opened, which is why a named ref always wins over a scraped one, and the report’s over the CLI’s final message. The refs ride the run’s phase record and the resumed executor’s text; nothing judges the turn on them. The runner also reconstructs an activity transcript from each CLI’s event stream — OpenCode’s line-flushed --format json events, Claude Code’s stream-json messages — in one shape: what the agent said, one line per tool call naming the tool and the first line of its subject ([tool] Bash: go test ./..., [tool] Read: <file>), a failed call marked with the first line of its error, and how a run that did not succeed ended. Never a tool’s output: a successful result’s body is a whole file read or everything a command printed, and it is what the transcript summarises. Nor the model’s thinking, nor a message type the build does not know (a CLI release adds them). The error stream is the transcript only where the event stream says nothing — a CLI that failed before it could stream says why there.

Each collected run is a phase of its turn. When the completion is claimed and the result read back, the coordinator publishes one agent_phase_completed with phase: sandbox and backend: sandbox — the run’s own record, beside the executor’s:

Field	What it holds
`turn_id`, `iteration`, `launch_id`	The identity. `iteration` is the executor iteration that launched the run, written onto the run’s record with the suspended conversation; a turn can launch twice in one iteration, so `launch_id` tells the runs apart
`started_at`, `duration_ms`	From the launch, dated by the store when it named the job, to the collection on the node that read the result — so the completion poll’s interval is inside it
`input_tokens`, `output_tokens`, `total_tokens`, `cache_read_tokens`, `cache_write_tokens`	What the run spent, as its CLI reported it (for OpenCode, its session’s totals — see above)
`cost_usd`	What the run’s CLI said it billed, where it says so. On this record only — it is the run’s fact, and it used to ride the resumed executor’s record, which a run that parked on a question never gets
`model`, `coding_agent`, `sandbox_id`	The model the launch pointed the agent at (empty when it named none), the CLI and the box
`response`, `delivered_refs`, `activity_transcript`	The findings report, what it delivered, and the transcript
`delivered_refs_elided`, `activity_transcript_elided_lines`, `activity_transcript_elided_bytes`	What the record’s bounds left out: refs it does not list, and the lines and bytes of the transcript’s middle. Absent when nothing was left out
`failed`, `error`	Set when the run did not succeed. A run that stopped to ask a question is not failed; its `notes` carry the question
`work_item`, `work_key`, `conversation_key`	As on every phase record

It is published once per run, however often the tail is retried: a resume that fails hands the completion back and the retry collects the same job again, so the run’s coordination record says the phase already went out, in the same write that hands the claim back — exactly as the charge is recorded. A publish the broker refused is not recorded and is offered again by a retry, if one comes; telemetry never fails the resume. Because every spend reader folds these records — the spend rollup, each node’s daily usage and the turn list — a coding run’s tokens are counted where every other phase’s are; before this record existed they reached the budget counter and nothing a person reads.

The whole record fits its event, because the broker refuses one over 8 MiB whole and this record is the run’s only spend record — refused, it would take the run’s tokens, cost, report and refs with it. Every piece of text a box wrote that rides it has a bound, in bytes: the report, the failure’s detail and the transcript 256 KiB each (sandbox.MaxRunTextBytes), the question in its notes 16 KiB (sandbox.MaxQuestionBytes) and the delivered refs 16 KiB (sandbox.MaxDeliveredRefBytes). JSON can grow text six-fold — a control byte becomes \u00XX, < and its kin \u003c, an invalid byte \ufffd — so the worst case is (3 × 256 + 16 + 16) KiB × 6, about 4.7 MiB, inside the ceiling with room to spare. What happens past a bound depends on who reads the piece:

The report and the failure are what the resumed executor acts on, so one past the bound is condensed by the seat’s auxiliary model (llm_auxiliary, falling back to llm) and labelled as condensed — never cut, because a report’s opening read as the whole report is a run that seems to have said less than it did. The failure’s detail is any piece the engine could not read, the exit status, the coding CLI’s own error and the end of its stderr, in that order — the stderr read from its end, with its unread start said by size (see What is read whole, and what is read as a stream below). The whole of it is held to what one condensation reads (sandbox.MaxCondenseBytes, 2 MiB) less 1 KiB the engine keeps for a sentence of its own it may put in front: the engine’s sentences are taken as they are, and the stderr gets what is left after them. A failure still past what a condensation reads — a CLI error as long as the file it came from — keeps whole lines from its start and its end up to that, with the lines between counted where they were, and that is what the model is shown. Where no model can condense one, it keeps whole lines up to the bound and says how many it left out: a report from its start (it is read from the top), and a failure from both ends — its first 32 KiB, an eighth of the bound, where the engine states the exit status, the CLI’s own error and any piece it could not read, and its end for the rest, because the line naming what broke is the last one printed. Kept from its end alone, a failure with a long stderr lost every one of the engine’s own sentences.
The question a run parked on is read by the person it is put to — on the run’s banner, through answer_run — and by the resumed executor beside the answer, and it rides the run’s coordination row, which every status flip and every listing of active runs reads whole for as long as it waits. So one past 16 KiB, a page of prose, is condensed the same way, as a question: the question itself, every option and what was ruled out. Where no model can condense it, it keeps whole lines from its start and says how many it left out; one whose first line alone is past the bound is not asked — whole lines leave nobody anything to answer, and a fragment would be a question the agent never asked — and the run resumes as not succeeded, saying why.
The delivered refs are read from the whole report with no count to them — every Delivered: line it writes, or every pull-request URL its prose holds where it names none — so they are redacted, deduplicated and listed up to 16 KiB — more than a hundred and sixty pull-request URLs of the usual length — with the rest counted (delivered_refs_elided) and said in the resumed executor’s text, never dropped unsaid.
The transcript is read by a person — no model and no prompt reads it — so it is not condensed: a long one keeps whole lines from its start, up to 64 KiB, and from its end, up to 192 KiB, with one note line where its middle was, naming how many lines and KiB were left out. The end gets the larger share because it is where the run’s conclusion and what broke sit; the start is kept because it is the run’s plan and its first look at the code — some 370 tool lines at their longest, since a tool line is the tool’s name and at most 160 bytes of what it ran, never its output — which a log cut to its end loses entirely. A single line longer than its half keeps its own start (or end), on a character, marked …. The counts also ride the record as activity_transcript_elided_lines and activity_transcript_elided_bytes, so the dashboard’s Activity fold says the record is not the whole log. The bound is applied once, by the coordinator, after it redacts the whole transcript again — never by a runner, so a runner that forgot it cannot publish an unbounded one.

Everything read out of the box is secret-redacted whole, before anything is bounded — a secret straddling a cut would otherwise survive as a fragment the pattern no longer recognises — and again at the publish. That includes the preview a transcript line shows of what a tool call ran, why it failed and how a run ended: each is redacted whole before it is cut to its 160 bytes, because a token whose start fell just before the cut kept most of its body, too little for any later pass to recognise and enough to identify it. A private key — PEM’s armour, or OpenPGP’s -----BEGIN PGP PRIVATE KEY BLOCK----- — is the one shape that spans lines, and it is redacted as a block, read by its structure whether or not its END line was written: its headers (RFC 1421’s Proc-Type:/DEK-Info:, OpenPGP’s Version:/Comment:), its base64 lines at the encoder’s width and one shorter last line — then its END, when it has one — with any number of blank lines among them (a key printed double-spaced is one key) and at most one other line between two of its own (a log line another process wrote in the middle of it). A key is read forward from its BEGIN, and back from an END that no BEGIN’s block reached: a body cut off from its BEGIN by two such lines in a row, or whose BEGIN is not in the text at all — a stream’s end read from inside a key. Read back, a full line also has to read like a key’s, carrying upper and lower case letters as base64 of random bytes does, digit or none: a line of random base64 lacks a digit about once in fifty thousand, and lines a key’s encoding fixes in part (an EC key’s second line, a last line of forty-odd characters) far more often — so a few lines of CamelCase identifiers or mixed-case paths directly above a bare END line are read as its body and redacted with it. The END’s own line is read both ways where what comes before its armour ends in base64 — a diff’s + is a prefix and a line of base64 both — and the block reaches as far back as either reading does. Two lines in a row end a block, and an END after them reads back across them to a line of the key exactly as wide as an encoder writes one — 64 characters (PEM as OpenSSL and Go write it, PKCS#8, OpenPGP), 70 (OpenSSH) or 76 (MIME’s base64, which Java’s, Python’s and .NET’s encoders write), one more under a diff’s + — from a full line of the same width, from the body’s last line or from the END itself, with the body’s last line and one more line another process wrote above the two; OpenPGP’s checksum line (= and a CRC-24, after the body) is read back from as its END is. A password prompt — a line ending in a password’s key and its separator (Password:, hunter2 Retype password:, pwd=x password:), that key neither the value of a key before it (as in pwd= password:) nor part of a token run into it (sk-…Pwd:) — is not one of the two: the read back passes over it, so another process asking for a password three times between a key’s lines and its END no longer publishes the key above, and the prompt is redacted with the key. So a key read from inside, its BEGIN not in the text, is redacted whole when another process broke it with two lines, wherever they fell, and with two and one more. What no read reaches is published: where three lines in a row, two whose nearest full line of the key above them is at no encoder’s width (a key wrapped at some other width), or two and two more with only the body’s short last line between them part a key, the part above them is published when its BEGIN is not in the text, and the part below them when it holds no full line — the short last line alone, BEGIN or none. Read back, a blank line under a wrapping that marks every line — a line number, a grep’s context mark — counts as a line between two of the key’s, so a key printed double-spaced under one and broken by one line as well is broken by three. Where the key’s BEGIN is in the text, the block read forward from it and the one read back across the two are one, and so is their marker: the two lines are redacted with the key, as a single line another process wrote into it is. A line that is not a key’s straight after a BEGIN ends its block before it has a body, so an END after it closes nothing: a BEGIN line with no key under it, such as a grep’s match for the armour line, is left as it is and takes nothing after it, even when a later grep matches the END — and read back from that END, the same prose is where the read stops. Every wrapping a key reaches a transcript in is read the same way: a JSON string with its line breaks escaped (once or twice), a line number, timestamp, grep prefix or diff + on every line (shaped like the armour line’s own), a string concatenated across lines, a line echoed at a time, or the whole key on one line, its encrypted headers too — and two whole keys flattened onto one line (echo $A $B, an env dump with the breaks stripped) are two keys, each closed at its own END rather than the first abandoned the moment the second’s BEGIN appears. Read forward, the body’s short last line that ends in the text around it — the quote closing the string the key was written into — ends the block at its base64 unless it reads like a piece of a key (below), and what follows a full line on its line ends nothing, since only the last line is short. A cut is read wherever it lands. A tool that bounds what it shows keeps a key’s line up to a point (MIIEow…[...truncated], or MIIEow… before a bare ellipsis), from a point ([… 812 bytes truncated]q3O4Ou…, a window’s first line) or both, with its mark between them (MIIEow…[… 2,345 characters truncated …]…Xyz) — on one line, on every line, or every line cut short with no mark at all. What is read is the piece of the key such a line still holds where a key’s line would stand — the run of base64 that opens the line after its prefix, or the one that ends it before its suffix, eight characters or more in upper and lower case with a digit — and never the mark, which no two tools spell alike; once a piece has said a cut wrote into the line, the run at its other end is the key’s too when it reads like one, digit or none, since the first sixteen characters of an RSA key’s line (MIIEogIBAAKCAQEA) hold none. Read forward, such a line is a line of the body, its key material running to that run or to its last piece, and the block goes on from it; so is a short line that is a piece straight after the BEGIN or with text after it (the head a cut kept and a short last line a quote closes read alike), and a short piece with another below it, since a body has one last line and two in a row are a key cut short on every line; and so is the tail a cut kept between a BEGIN’s armour and its line break. Read back, every line a cut wrote into is a line of the body, however narrow and whichever end of it the cut kept, and so is a short line that is a piece — so a key cut on every line, from its start or its end, behind a mark or with none, is read back from its END whole — and since the cut hid the width the line was at, a read across two lines another process wrote lands on it from a line at an encoder’s width below, and goes on from it to one above. And a key written on one line is read back from its END on that line as it is read forward from its BEGIN, across one cut in it — once, as a block carries one line that is not the key’s between two that are — to the key’s runs beyond the mark, when they hold one at a full line’s width, or, where nothing written up to the END reads as the key’s, one exactly as wide as an encoder writes or wider than any, since a path in mixed case before a grep’s match for the END is forty characters too. Nothing another rule redacts is a piece of a key: a password’s key, its separator and its value, and a token, are left to their own rules, so no cut’s reading takes part of one into the key’s redaction and publishes the rest. What a cut leaves that reads like no key’s is published: a piece shorter than eight characters, or one with no digit or in one case alone, where it is all a line holds of the key — and that line is one that is not the key’s, so straight after the BEGIN it ends the block before it has a body, and two of them together end a block anywhere, the key’s lines below published unless an END reads back to them; a piece between two marks, at neither end of its line; on a key written on one line, what lies across a second cut, or across a cut beyond which no run of the key is that wide; with no END below, the tail a mark on a line of its own kept after a head cut short, and the key’s lines after it; and, read from inside, the lines above two another process wrote into a key cut short on every line. A line of prose beside a key that opens or ends in such a run — === RUN TestSHA256Sum above an END — is read as one a cut wrote into and redacted with the key, and lines of prose that each open or end in one, straight above an END, are read as a key cut on every line, as far up as they go; a line holding such a run anywhere else is not, except an END’s own line, where such a run before the armour, up to one shorter run after it, is read as the body a window kept, key or none (=== RUN TestSHA256Sum x -----END PRIVATE KEY-----, a wrapping’s gutter [Worker1A] before the END). And after a key whose END was never written, a short line after a line a cut wrote into is read as the body’s last line, as it is after any full line, so a word the tool printed next is redacted with its mark. A line another rule names is no line of the key — a token, or a password’s key and its separator: under a key whose END was never written, Password:, or a pwd= whose = ends its line, is a line interrupting the key rather than its last, so the value on the next line is redacted with the key, or by the password’s rule when it cannot be the key’s last line — never published with the separator left behind. What that costs is a key’s last line that ends in a password’s key and =, about one key in fifty thousand, which is published when no END follows it. Another block’s armour, a certificate’s, ends a key’s block in either direction, on a line of its own or on the key’s, and so does a key already redacted, whose marker stands where its armour stood — which is what makes text redacted twice read as it did once: a one-line key whose END was lost is redacted up to the certificate or the next key written after it, and so is one whose END comes after a word. A password’s key is read as the text writes it: closed by the quote or the markup it is written in ({"password": "x"}, **Password**: x, `password`: x) — a quote only where it closes the key’s own string, a name such as "db_password", so a longer string ending in the word ("POST /auth/password": () =>) is no key — and separated from its value by :, = or the operator a language assigns or compares with (:=, ==, =>) — read with a bare : or = straight after the key, a JSON object’s password was no key at all, and an operator’s second character was taken for the value. The markup that closes what the key’s line opened before its label is the key’s too (**Root password:** x, | Password: | x |, | **Password:** | x |), and the value is read after it — on its line, or, where nothing follows it there, on a line below as after a separator (**Password:** over the value, - **Password:** over the value indented under it); a quote is no such markup, since a logged string closes with one and what follows it is the next field. A password’s value is read where its wrapping puts it: when what follows password: on its line is only the punctuation the wrapping writes after every line — the quote closing an echo’s string and its redirection, the " + of a string concatenated across lines, a table’s | — or nothing, the value is on the first line after it that is not blank, read past the gutter that line opens with — the start of what the key’s line has before the key (a reader’s line number, a grep’s context mark, a diff’s +, a log’s timestamp, the quote opening a concatenated string), shaped so that a number moving on from line to line, or a timestamp’s fraction written on one line and not the next, is still the gutter, and ending where a word of both lines does — and ended before the same punctuation, or before what a list’s or a concatenation’s last line closes with instead, or before the >> that appends to a file the key’s line opened with >. The wrapping’s blank lines between — an empty numbered line, echo "" >> key.pem — are passed over; a line holding a password’s key where the value would be ends the read, since its key names what is there, while a key further along the line is the line’s own and the value before it is still read; a line that does not end in that punctuation says it was no wrapping’s, so the punctuation is the value, as *** is; and a word in any script, or an empty quoted string, is a value, never punctuation. The value alone is replaced, so the gutter, the punctuation and the blank lines stay. Taking the first word after the separator wherever it stood, the rule redacted the wrapping’s own syntax — an echo’s closing quote, the quote that closes a concatenated string’s piece, the number of a numbered line a tab follows or of a blank one — and published the value on the line after. What is still published: a value under a quoted key whose name is more than letters, digits, _, - and . ({"Admin Password": "x"}); a value printf '%s\n' writes a line at a time (the escaped break is inside every line’s own prefix); a blank numbered line that lost the tab after its number, or a log’s blank line that lost the space after its timestamp, which is read as the value; a value shaped like the word the key’s line opens with, a space after it on both (pass2 next under pass1 Retype password:), which is read as the gutter as one timestamp is under another; a value on a line that opens with the punctuation the key’s line has straight before its key, after words that line does not repeat — a list’s dash after a logged message (msg: - Password: over - hunter2x9), a string’s first line ({"text": "> Password:\n> hunter2x9"}) — which is read as the value, since the gutter is read from the line’s start and ends where the two lines’ words part; where the markup that closes the key ends its piece of a string ({"text": "| Password: |\n| hunter2x9 |"}), no value is read at all; a value written over several lines — a YAML block scalar (password: |, password: >-), a YAML sequence under the key (- hunter2), a heredoc — of which the first word at most is read; a diff whose key is on a context line and whose value changed, of which the removed line’s first word is read and the added value published (where the values are indented, that word is the - alone, and both are published); a value under a gutter that holds an armour, or under a blank line of the wrapping whose gutter holds a password’s key; under a wrapping’s suffix, a value on a line that holds a password’s key of its own (echo "hunter2 password:" >> f under echo "Password:" >> f), the key’s line taking its own punctuation for its value instead, or the closing of the markup that alone ends it (| Password: | over | > Retype password: |), since redacting the value would change how that line ends once the second key’s value is replaced, and a second pass would read the first key’s value elsewhere; the part of a value after an escaped break inside it when that part is punctuation alone (pwd=ab\n!!), which reads as a string’s closing quote; and a key’s piece on the line of a value read below a password’s key that a key’s END line goes on to (-----END RSA PRIVATE KEY-----|passwd:): the keys are read with that value claimed, in the text as written, so the piece is no line of a key, while the rule reads the value in the text the keys’ blocks leave, where the END’s line is a marker that can join the value’s line to it or take the gutter the END’s line opened with, and reads another value, or none — and a second pass, the block gone, can read the piece as a key’s and redact it. What it costs is a line after the key that ends as the key’s did and holds no value: a prompt logged as msg="password:" over msg="connected" has connected redacted, where the rule used to replace the quote; a key a quote closes whose separator ends its line — a Go case "password": — has the next line’s first word redacted, and so does a JSON key whose value is on the next line, as it should; a code span in prose, `Password:` and a value, has the word after it redacted; and a name in a string or a code span that is no key where it stands is read as one, so a choice between two strings that opens with one (masked ? "password" : "text") has the other redacted, and the shell’s `pwd` before a colon (-v `pwd`:/app) the path after it. What the other rules write once the keys are read leaves a second pass nothing new to read either: a password’s rule replaces its value alone, so password: and the line break before a value on the next line stay where they were, and an escaped line break inside a value stays too, the value redacted on either side of it (pwd=ab\ncd becomes pwd=[REDACTED:password]\n[REDACTED:password], since whether the \n is a break or two characters of the password the text does not say); a value or a token stops at an armour or a marker; and an armour line that holds a marker — before its armour or after it, which is what a pass leaves beside an armour it read no key from — holds no key read across its lines, and a key written on it is read back from its END no further than the marker. So the coordinator, redacting again a transcript the runner redacted, changes nothing — save the key’s piece beside an END line that goes on to a password’s key, above. A block is never read more than 64 KiB from its armour (redact.MaxKeyBlockBytes, the largest key there is more than twice over in any wrapping), so a BEGIN on a line that runs on past that — a minified bundle naming the armour — holds back nothing after that line. And the pass stays linear on a line of many armours: each one’s scan stops at the armour next to it.

What is read whole, and what is read as a stream. A file meant to be read whole — the findings report, the question the ask shim recorded, the result line, the exit status — is read whole, and one past 32 MiB (sandbox.MaxFileBytes) is refused rather than read in part, on the engine host exactly as on a remote box, because a clipped report reads as a finished one. A refusal degrades only that piece: it is described by its size, the run reads as not succeeded with that as the reason, and its tokens, delivered refs and transcript are still collected, charged and published. A run’s machine streams are never refused for their size, because the engine keeps a bounded share of each anyway: the event stream is decoded a line at a time in bounded memory (a single line past 32 MiB is skipped and counted where it was), and the error stream is read from its end — at most what one condensation takes beside the engine’s own sentences in front of it (2 MiB less 1 KiB, and less those sentences), with its unread start said by size, because a process’s conclusion is the last thing it prints. It used to be read to the whole 2 MiB and carried behind those sentences, which put every failure with a long stderr a few hundred bytes past what a condensation reads, so none of them was ever condensed. Reading them whole used to refuse a run for the size of its own log: a poll that could not read a 32 MiB stdout never reached its liveness probe and held a finished-but-hung or dead job’s seat for good, and a collection that could not read it lost a result its report file held. On the engine host a box’s file is read only if it is a regular file where it lies: the other side of a container box’s mount is the agent’s, and a named pipe made at the done marker used to block the completion poll for good (an open waits for a writer and cannot be cancelled), so anything that is not a regular file — a symbolic link at the path included, whether it leads nowhere or was put there after the path was checked — is refused, naming the path and what it is, and never read as an empty file. The poll reports it; a collection degrades only that piece, exactly as for one past the size cap — described for what it is, the run not succeeded with that as the reason, and the rest collected, charged and published (a refused result file is read from the event stream’s last line, which is what the wrapper copies into it) — because what a path names is the box’s own doing, and a collection retried over it would be refused again until the run was settled as lost. A write takes the same rule: what the engine writes into a box on its host — a setup step’s files, the ask shim, a coding CLI’s configuration, each written after something may already have run there — lands only on a regular file where it lies, or where nothing is, so a link at the path is refused rather than followed (one that led nowhere used to have its target created on the engine host) and a named pipe is refused rather than waited on, and the launch fails naming the path and what it is. The engine’s own records about a local box are kept beside its home rather than in it (see Local sandboxes) and read the same way, so a record that is not a regular file answers at once rather than blocking the poll, a collection or a reaper: an unusable record of the box’s logins is logged (local_sandbox_credential_map_unreadable) and costs only the write-back of a login the run refreshed, and an unusable job record reads as not knowing whether the job runs, which keeps the box.

The dashboard renders the run as its own card after the executor that launched it: the report, and the transcript as a closed Activity fold. A turn with a coding run therefore reads as the executor (its reasoning, the launch and — once resumed — what it did with the result) followed by one card per run. Where the record’s bounds left something out, the screens say so rather than showing the part as the whole: the Activity fold counts kept of all lines (412 of 4,000 lines, or the bytes left out of a single over-long line), and a run’s Delivered — on its page and on its span in the turn trace — names how many more refs its report holds than the record lists (… (+41 more its report names)).

Setup steps — provisioning the box

A coding agent’s box needs environment wiring beyond the CLI itself: git auth, registry credentials, toolchains. Provisioning is a declarative setup-step framework (internal/sandbox/setup.go); each sandbox.SetupStep contributes:

files — content written into the box (helper scripts, config files), at paths that are not shell-expanded but are cleaned, in every cell, so a trailing / or a . segment names the same file. A relative path is under the box’s home, and so is one starting ~/, which names the home in every cell; that is how to name a file a step needs (.npmrc, ~/.config/tool.toml), since it lands in the home whichever cell the box runs in. A local box writes one only inside its home — a direct box’s is minted per box, so no absolute path names it, and a container box’s is its mount at /home/user — so a step that must put a file anywhere else has a command do it, with whatever rights that box’s commands run with, and a document whose step writes a file a seat’s box would refuse is refused when it is written, naming the step, the path and the seat (see Local sandboxes). An e2b box writes a file at any other path — its in-box agent creates it where the path says, missing directories included, and gives it to the account every template runs as (user), whose home a relative path lands in — but refuses another account’s home (~name/…) and a path naming a directory: the home itself and the directories above it, which every template has, are refused on the write too, while a directory only the template made (/tmp) is refused by the box at setup. A setup command there runs as that same account, not as root;
commands — shell commands run after the files land; a non-zero exit fails the acquisition and tears the box down, so a half-provisioned box never receives a brief promising an environment it doesn’t have;
env — variables merged into the coding agent’s run env (${VAR}-resolved with the rest);
optional_keys — keys of the step’s env, and paths of its files, whose value may resolve to nothing: such a variable is left out of the run and such a file is not written, where any other reference that resolves to nothing refuses the run (see Credentials and the run environment);
brief — a paragraph folded into the coding agent’s ”## Your environment” block, telling it what the step made true (an agent that wastes rounds rediscovering how to authenticate is slower than one that knows).

Steps come entirely from company config — the engine ships none of its own, git auth included — applied in order: providers.sandbox.setup (engine-wide, every sandbox role) then role.sandbox.setup (per-role extras). Setup commands execute with the run env, so a recipe can read the engine’s per-launch identity facts ($CREWLET_AGENT_HANDLE, $CREWLET_AGENT_EMAIL) and its own configured tokens at provisioning time. A reused box skips re-apply — its provisioning survives with its disk state.

Every step has a name, unique within its list: it is what a setup failure points at, and because a step’s env and files are credentials, it is also what a config read masks them under and what sending the read back restores them by. A document with two steps of one name in the same list is refused on every write; the same name in providers.sandbox.setup and in a seat’s own sandbox.setup is fine. A config read shows a step’s env and files values only when each is exactly one ${VAR} reference.

A file is content, and it reaches the box as written. A step’s env is a setting: a ${VAR} anywhere in it is the engine’s, expanded at launch with the rest of the run environment. A file’s body is written into the box, so a ${…} inside it belongs to whatever reads the file there — the script’s shell, npm reading .npmrc — and the engine never expands one. The single exception is a body that is exactly one ${VAR} and nothing else: that names the whole file, and at launch the box receives the value the variable holds (the secret store first, then the engine’s environment), byte for byte. So //registry.npmjs.org/:_authToken=${NPM_TOKEN} in an .npmrc is left for npm to expand from the NPM_TOKEN the step’s env puts in the box, while .npmrc: "${NPMRC}" writes the whole stored file, into the box’s home in every cell. A whole reference that resolves to nothing — nothing answers for it, or what answers is empty — is never written as an empty file: it refuses the run exactly as an env value that does not resolve does (below), and where the step lists the path under its optional_keys the file is simply not written. The same rule holds for a file on a seat’s own sandbox.setup.

The git-auth recipe

Injecting a code-host token is necessary but not sufficient: a headless git clone https://github.com/... has no way to supply it and dies with could not read Username. The recommended wiring is a config recipe — a credential helper plus rewrites, shipped as an ordinary engine-wide step:

providers:
  sandbox:
    setup:
      - name: git-auth
        commands:
          - mkdir -p "$HOME/.local/bin"
          - |
            cat > "$HOME/.local/bin/git-credential-crewlet" <<'SH'
            #!/bin/sh
            [ "$1" = "get" ] || exit 0
            [ -n "$GITHUB_TOKEN" ] || exit 0
            ok=""
            while IFS= read -r line; do
                [ -z "$line" ] && break
                [ "$line" = "host=github.com" ] && ok=1
            done
            [ -n "$ok" ] || exit 0
            echo "username=x-access-token"
            echo "password=$GITHUB_TOKEN"
            SH
          - chmod +x "$HOME/.local/bin/git-credential-crewlet"
          - 'git config --global credential."https://github.com".helper "!''$HOME/.local/bin/git-credential-crewlet''"'
          - 'git config --global --add url."https://github.com/".insteadOf "[email protected]:"'
          - 'git config --global --add url."https://github.com/".insteadOf "ssh://[email protected]/"'
          - 'git config --global user.name "$CREWLET_AGENT_HANDLE"'
          - 'git config --global user.email "$CREWLET_AGENT_EMAIL"'
        env:
          GIT_TERMINAL_PROMPT: "0"
        brief: >-
          Use $GITHUB_TOKEN (already in your environment) for all GitHub work:
          git authenticates to github.com with it automatically — clone/fetch/push
          over plain HTTPS; SSH remotes are rewritten; never embed the token in a
          URL. Your git commit identity is preconfigured.

Each piece is load-bearing: the helper is scoped to the code host twice (the credential config key and the host= check in the script), so the PAT can never be offered to a foreign host such as a malicious submodule URL; it reads the token from the env at git-runtime (never persisted to disk) and stays silent without one, so public clones fall through to anonymous; the insteadOf rewrites use --add because the key is multi-valued (without it the second value replaces the first and scp-style git@…: remotes stay SSH, failing on host-key verification in a keyless box); and commit identity comes from the engine’s generic $CREWLET_AGENT_* facts, so commits attribute to the seat without the recipe hardcoding a name. And the helper lives in the box’s home, the one place every cell lets a setup step write it (see Local sandboxes), and is written by a command rather than a files: entry, though a relative files: path would land in the same place: git has to be given the helper’s absolute path, which only the box’s shell knows ("$HOME", expanded), and the helper has to be made executable, both of which are commands. It is registered as a shell snippet whose path is quoted, "!'$HOME/…'", because git runs a helper’s value through the shell and a box’s home may hold a space: a direct box’s sits under local.state_dir — by default ~/.crewlet/sandboxes, and a relative one, like the examples’ ./nimbus-data/sandboxes, is resolved against wherever the engine was started. Written this way, the one step works in every cell.

The GitLab form of this recipe — same shape, gitlab.com scoping, oauth2 username, MRs opened via git push -o merge_request.create push options — is in GitLab Integration; the block above is itself the GitHub form, and GitHub Integration covers the rest of a GitHub seat’s code authoring. The repo to work in is not role config — it is task context the executor names in the brief, and the coding agent clones it inside the box with the seat’s injected token.

Credentials and the run environment

The run env injected into each job is assembled by the engine at launch (internal/engine/sandbox.go), and internal/sandbox/launch.go adds the telemetry variables. The engine contributes only tool-agnostic facts:

LLM credentials, derived from the role’s resolved providers.llm entry (the chain above), for the runner that reads them: the entry’s key as ANTHROPIC_API_KEY for an anthropic entry and OPENAI_API_KEY for an openai or openai-compatible one, and, under Claude Code alone, an anthropic entry’s base_url as ANTHROPIC_BASE_URL — only when the entry sets one. OpenCode is handed its endpoint in opencode.json’s declared provider (above), the one place its run takes the entry’s endpoint from, so no base-URL variable is exported for it. They are the values the entry’s own calls use: its ${VAR} references resolved through the secret store and the environment when the configuration was applied, and the first key of its api_keys, because a box cannot rotate through the pool. Under Claude Code the entry’s key and endpoint are one unit: a seat that signs Claude Code in through role.sandbox.env gets neither (see Claude Code). A metered key now sits in every box run on its entry, remote cells included, readable by the coding agent, which runs with every permission. What the run reports back is passed through the engine’s redaction, but that matches known credential shapes (the vendors’ sk-… prefixes and the like), not values: a gateway key of no recognisable shape that the agent prints into its report or transcript is carried into the phase record as printed. Give a box a key scoped to the work — a gateway key of its own, or a provider project with a spend limit — rather than the company’s broadest one.
Agent identity — CREWLET_AGENT_HANDLE and CREWLET_AGENT_EMAIL, the per-launch facts static config cannot know, which setup recipes map into tool shape (git commit identity above).
A subscription sign-in, when the resolved provider is a cli-agent entry: whatever authenticates that CLI in its environment — the headless token in the profile’s own variable (CLAUDE_CODE_OAUTH_TOKEN) under subscription, the api_key_env value under api-key, the variables inherit-env forwards, and, for opencode, the credential-named cli.env keys of the provider its model names. Only an entry whose own CLI is the seat’s runner gets this far — any other is refused (see SandboxModelError). Its login — credential files, or OpenCode’s exported login — travels only to a local box, never to a remote one.

Everything tool-specific comes from config: external-service tokens (GITHUB_TOKEN, GITLAB_TOKEN, registry tokens, a test DATABASE_URL, …) are declared in role.sandbox.env or a setup step’s env and merely ${VAR}-resolved — the engine never extracts, names, or special-cases them. Precedence, later wins: derived LLM creds → agent identity → setup-step env → role.sandbox.env → non-secret OTel values — except that, under Claude Code, an entry’s key and endpoint are not added at all beside a sign-in the seat’s own environment carries.

Every reference resolves, or its key is declared optional. A ${VAR} reference that resolves to nothing — whole ("${TOKEN}") or embedded ("Bearer ${TOKEN}"), unset or set to an empty value — is never handed to the box as an empty string, by the rule every tool server is held to (Tools & MCP): a run given an empty code-host token clones nothing and pushes nothing, after spending its box, its install and its model’s tokens. A run_sandbox call whose run environment lacks one is refused before any box starts, as the call’s own result — classed unavailable, so it costs the turn one round rather than the turn, and naming the variable and its remedy — and the launch logs sandbox_env_unresolved naming the keys, the files and the variables (never a value or a body). Both say why the value is missing: at info, telling the seat to try again, while a provisioning pass is minting it for the seat — the same ${VAR} an mcp_env code-host credential points at, for example — and as a warning, with the remedy, where nothing will mint it or that pass cannot until somebody acts. An agent-mode executor’s run is not refused: it is the turn itself rather than a tool call the turn chose, and refusing it would fail every message the seat could answer over its tool bridge for a token a pass may mint in a minute. It starts without what is missing and logs the same line. The engine’s own identity facts never refuse a run: each is contributed when it has a value, and a seat email: naming a variable nothing sets is left out of CREWLET_AGENT_EMAIL and said on the same line, so a setup step’s git config user.email reading it empty has a reason beside it.

A setting a run can go without — a test DATABASE_URL, a proxy — is declared by its key under optional_keys, beside the env that writes it: on role.sandbox, or on a setup step, where a file’s path may be listed too. Such a key whose value does not resolve is left out of the run environment whole, never handed on half-substituted, and the run goes ahead:

roles:
  - name: Backend
    sandbox:
      enabled: true
      env:
        GITLAB_TOKEN: "${SWE_GITLAB_TOKEN}"            # required: run_sandbox is refused without it
        DATABASE_URL: "postgres://${TEST_DB_HOST}/app"
      optional_keys: [DATABASE_URL]                    # left out when TEST_DB_HOST resolves to nothing

The declaration that counts is the one beside the value that won: a key in role.sandbox.env replacing a setup step’s is judged by role.sandbox.optional_keys, never by the step’s. A value a provisioning pass mints for the seat is waited for whatever the declaration says, as a tool server’s is: the run is refused, telling the seat to try again, rather than started without its code-host identity. An entry that leaves nothing out — a key or path its block does not write, a variable’s name where its key belongs, or a key whose values reference nothing — is refused on the write that adds it (an admission rule), because the key it was meant for stays required. Resolved values are injected only into that sandbox’s process env and die with it; the DB stores only the ${VAR} references. Shared infrastructure secrets — above all the OTLP backend’s ingest token — never enter the sandbox at all (see below).

MCP inside the sandbox

The coding agent is itself an MCP client, so the engine renders the role’s servers into the box (internal/sandbox/mcprender.go → Claude Code’s .mcp.json via --mcp-config --strict-mcp-config, or OpenCode’s opencode.json). Scoping is at the server level only: role.sandbox.mcp.servers names which of the role’s mcp_servers to expose, and the coding agent gets every tool those servers expose. There is no per-tool allowlist — OpenCode has no allowlist flag and Claude Code runs bypassPermissions headless, so a curated list couldn’t be enforced uniformly and would give a false sense of restriction. A role that shouldn’t reach a surface simply doesn’t list that server.

Credentials from role.mcp_env are resolved into the rendered specs (HTTP servers get them as headers, stdio servers as env), so the in-box server instances authenticate as the seat. So is every ${VAR} in the server’s own env, headers and url: the engine resolves each through the secret store and its own environment, exactly as it does for the server’s child on the engine host, and never writes a reference into the box for the coding CLI to expand — OpenCode does not read that syntax, and Claude Code would read it from the run environment, which holds a company secret only where the seat’s sandbox.env or a setup step repeats it. So a server’s reference means in the box what it means on the engine host: one the secret store and the engine’s environment do not answer is not answered by a value the run environment happens to carry either. The connected server names are also listed in the coding agent’s environment brief. A server any of whose values does not resolve — the seat’s or its own, under a key the server does not declare optional — is not rendered at all, by the rule the server’s own child follows (Tools & MCP): handed an empty token, an in-box server refuses to start or acts as nobody, and the agent would spend a round finding that out. A key it does declare optional is left out of what the box is given. The run logs sandbox_mcp_server_withheld naming the variables: at info while the integration that mints them is minting them — a run launched once the seal reaches the node is given the server — and as a warning where they are unset, or where that integration cannot mint them until somebody acts.

A named server the company’s mcp_servers does not carry is skipped, with a warning, rather than rendered empty — the agent must never be offered a server it cannot reach, because it spends a round discovering that. The same holds for a server whose transport is neither stdio nor http. One skipped entry does not cost the seat the rest of its surface.

Two practical caveats:

A cloud sandbox cannot reach your laptop. An HTTP MCP server (or a git host) on localhost / a private LAN address is unreachable from an E2B cloud box. For laptop-local development stacks, use a self-hosted E2B domain on the same network, a tunnel, or scope those servers out of the sandbox.
stdio servers must exist in the box. A stdio spec is spawned inside the sandbox, so its command must be present in the template (E2B’s prebuilt templates cover common cases; anything else belongs in a custom template or a setup step).

The tool bridge — a seat’s own tools, from inside a box

A coding agent doing code work has the servers above rendered into its box. A coding agent running its seat’s executor — see Subscription LLM backends — needs something different: the seat’s whole tool surface, including the engine’s own builtins and the delivery tools the turn is judged on. Those cannot be shipped in. Most are MCP children holding the seat’s credentials, several are engine control, and handing either to a box running generated code is the failure the sandbox exists to prevent.

So the box gets one MCP server, on the engine, and every call comes back out:

box ──MCP/streamable-HTTP──▶  POST /mcp/{token}  ──▶  the seat's live tools.Surface
                                                        (grant · skill guard · recording)

The dispatch is the seat’s own. A bridged call goes through the same tools.Surface.Execute a native tool loop calls, so the grant, the required-skill guard, the failure shape and the recording are all the ones an operator already knows. internal/api/mcpbridge never grows a second idea of whether a tool is allowed — a second copy would drift, and it would drift on the security half.

The credential is per-run and expires with it. The box holds no API token; what admits a request is a signed token in the endpoint’s own path, the same shape the OTLP receiver uses (internal/runtoken). Two gates stand behind it: the signature says this fleet minted it, and the session map says the run it names is still going, so a box that outlived its run keeps no working key. Set CREWLET_MCP_BRIDGE_URL to this node’s own HTTP listener, as the sandbox reaches it; unset, no bridge is mounted and agent mode is refused for the seat rather than starting a run that cannot call anything. A node that sets it and binds no listener (api.port: 0) refuses agent mode too, naming api.port.

Every bridged call acts as the run’s seat. Because the token is the credential, the caller is the seat the run belongs to, and the bridge states that on every call — the same identity a native turn carries, with the same four grants — so the authority table decides a bridged list_work_items exactly as it decides the native one. It is set by the bridge itself rather than read off the request, because /mcp/ is exempt from the credential guard: the guard resolves a box that presents nothing as anonymous, and a development build started with -dev-principal resolves it as that principal. Neither may decide a seat’s tool call, so the bridge replaces the request’s answer with the seat’s on this route and nowhere else. A session whose tool surface is bound to no seat is refused at Open rather than served.

Every call is written down where a resume can read it. A native tool loop keeps its calls in memory and the turn writes them at the end; a bridged run’s calls are made by a process outside the engine, minutes apart, possibly across a restart. Each one is appended to the run’s own coordination row, bounded, dropping from the middle rather than the start — how a run began and how it ended are what explain it. Without that, the reviewer of a resumed run judges a turn whose entire tool log is gone, which the delivery check reads as a turn that acted on nothing.

The node that runs the seat serves its bridge. A session is a live tool surface in the process that opened it (the seat’s MCP children, its skill guard, its recording), so no other process can answer a box, and CREWLET_MCP_BRIDGE_URL must address that node itself: not a load balancer in front of several, and not a peer that runs only the ingress role. A node whose roles leave out ingress still binds api.port for this one route when it runs seats and has a bridge URL set (see Running a Fleet). With Tier A api.public.port set, the bridge is served on that public listener instead, on every node and never on api.port, so the bridge URL names the public port (Exposing webhooks without the admin API). The token is signed from Tier A secrets.keys, under a domain that separates bridge tokens from telemetry tokens, so that a peer a misrouted call reaches can log that the URL addresses the wrong node (mcp_bridge_unresolved) instead of treating the call as forged. Every node holds that keyring — a node without one does not start — so there is no per-process key to fall back to: a keyring that cannot sign for the fleet is refused by name when the bridge is built.

Mid-run clarification (`crewlet-ask`)

Once it is in the code, the coding agent may hit something only a person can answer — an ambiguous spec, a design decision above its remit. Headless coding agents can’t pause to ask interactively, so every box gets a shim, crewlet-ask (internal/sandbox/codingagent/ask.go), and the brief instructs: don’t guess — commit and push your WIP branch, run crewlet-ask "<question>" --to <audience> --branch <the branch you pushed> from inside the repository, and stop. The audience is requester, team, manager, or a named teammate — the brief carries the seat’s unit roster and its manager, by handle, so the agent can name a real person and the park can resolve the name exactly. (That instruction had no caller until the answer-by-turn change: the shim was installed in every box and no brief mentioned it, so a run blocked on a decision had no way to park on a question at all.)

The shim is signal-only: it records the question, its audience and the branch the work was pushed to to a file, and never posts anything itself. The branch is what a run re-seeded on a fresh machine is told to check out (below), so it is recorded as its own fact at the one moment anything knows it: the name --branch gives, or — where the agent names none and the image has git — what git says where the agent asked, the name the checkout’s upstream has on its remote (the name it was pushed under), else the branch checked out there. A detached HEAD, a directory outside any repository or an image with no git records no branch; git is never required. The engine keeps the branch only if it is a name git accepts for a branch and at most 1 KiB (sandbox.MaxBranchBytes); otherwise the run parks with no branch, and its re-seed is told to check out the branch it pushed in general terms rather than a name that does not exist. It is never inferred from the refs the run delivered, which name a pull request’s URL as often as a branch. When the run completes with a pending question:

The engine (never the sandbox) announces the question as sandbox_clarification_requested on the event stream, attributed to the seat whose run asked and carrying the question, the audience as the agent wrote it, the run’s conversation and its work item, and resolves that audience into the people it is put to (below), keeping identity attribution and capability guards on the engine. That is how it reaches them: the run becomes a row in each of those people’s own decisions on the dashboard (Home and the Inbox’s Needs you, and Asked of me on My work, all served by GET /work/decisions), a banner on the run’s page and its rail in Live › Coding runs, a row of Live › Now running’s Waiting on a person, and an entry on the board an operator reads (GET /sandbox-runs?audience=<handle>, “what is waiting on me”). Nothing is posted to a chat surface: the engine sends no message into the run’s conversation or to anybody’s direct messages, so a person who works only in chat learns of the question from somebody who has seen it there. It is answered from that row, by turn, or by a reply on the run’s conversation (step 3).
The run is parked (awaiting_clarification) with both conversation values; the suspended Execute state is preserved untouched; the box is left paused so the checkout survives. Unlike a running job, the agent goes free — a human reply can take days, so its inbox flows and it can do other work. The question is announced first and recorded second, and the agent is freed only once the record lands: a question recorded but never announced would wait for an answer nobody was asked for, and a seat freed ahead of the record reads as idle with nothing open, which is the state in which a person’s reply is run as an unrelated turn.

A park that could not be written is not a park. The run reaches this step holding the claim a completion took, so a step that fails leaves the claim to give back: handed back, the completion poll fires again, the paused box is collected a second time and the park is retried. If the claim cannot be handed back either, nothing retries at all — a row left in the claim is read by no poll, re-claimed by no redelivery, matched by no answer and expired by no reaper — so the run is ended like any other lost turn (record deleted, box reclaimed, sandbox_run_failed with reason claim_unreverted) rather than left to strand a billed box until the seat changes hands. It is ended only while it is still the claim, though — its status, its job and its lease, which the store checks as it decides the ending: a hand-back that reported a failure may still have landed, and a run it did reach is back where the claim found it and owed its retry, so it is left to that retry rather than killed under it — and an ending that could not reach the store either, kept and finished later, that finds the run handed back after all resumes it with its reply instead of settling into nothing. The ending is decided first and the box reclaimed second for the same reason, since the decision is what establishes the run was still the claim’s to end.
The first reply posted in that conversation after the question was asked is the answer. The row records the instant the question was put (asked_at, taken before the question is announced, so nobody can have read it earlier), and a delivery answers only if it was posted at or after that instant — its envelope’s own timestamp, which for a chat message is the vendor’s post time. Without that anchor the rule was purely positional (“the next thing to arrive here”), and two things went wrong with it: a message written while the job was still running was taken as the answer the moment the run parked, and a reply that answered an earlier question, held back and redelivered, was spliced in as the answer to whatever the run asked next. The conversation is matched on the conversation identity the row was written with, which lets the answer arrive wherever the person actually replies. It has to be the identity rather than the inbox partition: a run launched from a top-level direct message parks under the bare DM channel, and the engine’s own chat prompt tells the seat to reply as a thread — so the person’s answer arrives partitioned on that thread, and comparing partitions meant the clarification was never delivered and the box waited out its pause TTL. A direct message is one conversation however it is threaded, which is exactly what the identity says. A row with no identity — a run launched by a schedule tick or an A2A wake — is answered by no chat reply at all. Every question is parked with its anchor — a park without one is refused — and a delivery that cannot show it was posted after the question answers nothing: it goes its ordinary way.

The answer is recorded before anything is done with it. The first qualifying reply is written onto the run by a compare-and-set — the run moves to answered, carrying the reply (answer) — and only then is the delivery acknowledged. From that moment the answer is durable and nothing about it depends on the inbox again: a second reply racing it, on this node or across a seat handoff, loses the compare-and-set and finds the question answered. The coordinator then claims the answered run (at-most-once) and resumes the same suspended Execute loop, splicing in the question, the answer, and the WIP branch as the run_sandbox call’s reply, with instructions to call run_sandbox again incorporating the answer, which reattaches to the paused box and checkout. No fresh turn and no re-investigation.

Recording and resuming were split because the inbox’s retry order is the wrong order for an answer. The answer used to be held by its delivery until a resume succeeded, and a resume that failed handed the message back to the broker with a NAK. On JetStream a NAK’d message comes back after its redelivery backoff, behind the conversation’s newer mail (event system): so the person’s next message reached the still-waiting run first and was spliced in as its answer, and the real answer came round afterwards to answer whatever the run asked next — or ran as a turn with no context. The in-memory twin replayed a NAK at the head, so every engine test certified the order production never gave. A resume that fails now is the coordinator’s to retry, on its own schedule, and the person’s message is never handed back for it.

The seat’s later mail waits behind the answer. While a seat owes any of its runs an answer’s resume, the coordinator holds the seat’s inbox (the sandbox hold a busy seat is held under too) and lifts the hold once the resume has returned, so the person’s follow-up is worked as an ordinary turn — with the run’s own replies already in the thread it reads — rather than racing the resume. A hold rather than a park: the mail waits on the broker with its order and its delivery count intact. A later reply is an ordinary message, not a note to the resumed run: a steer note reaches a turn that is running, and while the answer’s resume is owed no turn is.

Every delivery to a seat with a question open is offered to that match first, and that follows from step 2 rather than adding to it: a parked run frees its seat, so the answer arrives on the ordinary path looking like any other message — nothing about a chat reply says which coding run it ends. So the dispatcher offers each delivery to the parked run before consuming it, and only what no parked run claims becomes a turn. It is offered after the completion ledger has had its say, so a redelivered trigger that already produced a turn is never spliced in as an answer as well. A held seat offers nothing — its inbox is held, so a reply to a second run of the same seat, waiting while the first is still going, waits with the seat’s mail and is offered when the first stops holding the seat. A delivery that becomes an answer is recorded in that same ledger, so a copy of it — a redelivery whose acknowledgement was lost — is dropped rather than run as a turn, on whichever node it reaches; while the run still holds the answer, a copy is recognised on the run itself and spent as the answer it is.

The offer answers what to DO with the delivery, in three words, and an error is never one of them. Consumed — the answer is recorded on the run (whatever its resume then does), or the delivery is a copy of one already recorded — so it is acked and no turn runs. Deferred — a run is waiting and the answer could not be recorded, because the coordination store failed or could not say whether the write landed — so the delivery is handed back by deferral and offered again: it keeps its place at the head of the seat’s inbox (a NAK would put it behind the person’s next message), and the seat’s inbox stops until the seat host’s next renew. Not mine — nothing was awaiting this conversation, or the delivery was written before the question — so it is worked as the ordinary message it looks like. A delivery whose answer was recorded is consumed even when the run’s first resume ends it for good — no conversation to re-enter, a claim that could not be given back, a resume that broke before its turn: the ending hands the reply back through the run’s row as below, with the seat’s inbox held until the copy is out, so it still reaches the seat once and in order. The inline attempt used to hand the delivery it held on as the ordinary message instead, which needed the store to delete a run still holding the reply on that caller’s word — and an ending that could not land then left the reply both on the row and handed on. The ambiguous cases resolve towards the run, because an answer that arrives twice is recoverable and an answer that is spent is not.

A lookup that cannot be made is decided by what this node already knows. It is the ambiguous case in its purest form — the store could not say which of the seat’s runs is awaiting a reply — so it is not answered by the store at all, but by the seat’s own awaiting count, which the node holds in memory (seeded from the store when it claimed the seat, moved since by its own transitions) and can therefore read when the store is exactly what is failing. No awaiting run on the seat: not mine, because nothing is owed the delivery and an unreadable store must not swallow an ordinary message — on a company with no parked runs that is every message a seat receives. An awaiting run: deferred, because something here is waiting for somebody’s reply and only which row could not be read. Its hand-backs are bounded by the same count as any other, under a key naming the seat; past it the message is let go to the ordinary route and sandbox_answer_lookup_exhausted says so.

Both series are bounded, and the bounds are the old ones. The resume of a recorded answer is retried by the coordinator a second after it fails, doubling to a 30-second ceiling — the queue’s own failure backoff, deliberately — for at most 10 attempts and at most the run’s own pause_ttl_seconds, measured from the first failure: about two and a half minutes, long enough for the failures that actually reach it to clear (a seat lease moving to a node that can resume the run, 45 s; a config apply that brings the missing runner; a store blip). A retry the seat’s own conditions refuse charges nothing, and it waits for what refused it rather than re-checking on a timer: a person’s pause is waited out until the pause watch hears them resume the seat, a company with no model until the apply that brings one, a spent budget window until the instant it turns over (or an apply that moves the ceiling), a node whose posture refuses new work until the reconcile tick that finds it admitting again, and a seat whose ownership cannot be proven until it can: the moment the seat’s acquisition is established — which is how an answer the seat inherits from its last holder resumes as soon as the new holder can run it, rather than a heartbeat later — or the renew that proves it again after a store blip, with one heartbeat on the clock behind both for a renew that was merely late. A signal that lands while a retry is being refused is not slept through: the wait that follows it re-checks at once. Each of those is a discrete event the engine already observes, and it re-checks exactly the answers waiting on it, at once; nothing runs while the condition holds. A five-minute backstop re-checks a wait whose signal never came, so a missed signal costs a late resume — well inside the 30-minute default pause_ttl_seconds — and never a stranded answer. Past the bound the answer is let go: the run goes back to waiting on its question, the reply is handed to the seat’s inbox as the ordinary message it is (under a new id derived from the original, because the original is spent in the completion ledger), and that reply is declined for this question (declined_answers) so it never answers it again — sandbox_answer_requeue_exhausted says so. Letting go is exactly once, across a crash at any step: one write clears the answer, declines the reply and its copy, and records the copy on the run as owed to the seat (hand_back); the copy is published off that record and cleared from it afterwards. A node that stops before the write leaves the answer recorded and nothing published, and the seat’s next holder drives it again; one that stops after the write leaves the copy owed, and whoever reads the run next — this node’s retry, the seat’s next holder before it opens the mailbox, or the run’s own ending — publishes it; one that stops after the publish leaves the copy published and owed, and the repeat is the same message under the same id, which the inbox’s same-id dedupe and the completion ledger collapse. It used to publish first and write second, so a crash between the two delivered the reply twice: as an ordinary message and as the answer a later resume used. The run is not ended, because every failure that gets there is a statement about this node — no resumer, a suspended conversation this build cannot decode, a seat not in this node’s company — and the seat’s next holder, or this node after a restart, takes over an answered run from its record and drives the resume itself. A run that ends still owing a copy is not deleted until the copy is out: the store refuses to delete a row that owes the seat anything, so the ending publishes the copy, clears it, and only then deletes the row — a copy published off the deleted record, as it used to be, was lost to a crash between the delete and the publish. A publish that fails keeps the row, holds the seat’s inbox behind the copy, and the ending is retried with the hand-back. What a run owes is bounded by one decline: a second answer is let go of only once the first one’s copies are cleared, so the row never carries more than one delivery batch of copies (20 events by default, the size of the answer they were made from); past it nothing is lost — the second answer stays recorded and owed, its seat stays held, and the retry publishes the copies already owed before it decides again. And each answer gets its own attempts: one recorded while an earlier answer’s copy is still owed starts a fresh count, rather than being let go of on the strength of the earlier answer’s spent one.

A reply that leaves its run without a turn taking it goes back to the seat, once — whoever ends the run. The run can end under its own resume, on the inline attempt or a retry — it has no suspended conversation to re-enter, its claim cannot be given back, or the resume broke before its turn began — or under a seat that left the company, or a reap, and the delivery that carried the reply was spent when the answer was recorded, so nothing else would ever bring the reply to the seat. The ending is decided on the run’s record first, and the decision settles whether the reply goes back: a turn cannot take the answer once the ending is decided, so one no turn took by then goes back to the seat, and one a turn took has been used and goes with the run. The ending then lets the answer go in one write that takes it off the run and records its copy as owed (hand_back), and hands the copy back before it deletes the row — the store will not delete a run still holding a reply the ending owes the seat, so an ending can never take the reply with the record. An ending is decided only under its own terms: a hand-back that landed and reported a failure, or an old holder’s hand-back landing under a reap, puts a run back to answered with the reply on it, and a claim’s own ending is licensed for the claim alone, so it decides nothing about that run, which is left to its retry, reply and all. The crash guarantees are a decline’s: stopped before the let-go’s write, the reply is still on the row and whoever reads it next lets it go; after it, the next reader publishes the copy; after the publish, the repeat is the same message; after the clear, nothing is published again. An ending that cannot be finished — the let-go not recorded, the copy not published, the announcement refused, or the delete itself not taken — keeps the row and its terms: the seat’s inbox is held behind it, and this node finishes it, once the store takes it, exactly as decided — the box included, where the ending reclaims that only after its decision. A delete the store did not take used to be left for the seat’s next recovery pass, which comes only when the seat moves, and the record it left was a claim, which holds the seat: the seat took no mail at all until it changed hands. And a kept ending used to be finished without the reclaim it was decided with, so the only record of a paused box was deleted and the box billed, named by nothing, until its provider’s TTL. A reply used to be published straight off the retry after the ending had deleted the row, and later straight off a refused record, so a crash between the delete and the publish, or a broker refusing that publish too, lost it.

A claim whose node stopped before its turn took the answer is revived, and the run resumed with it. The coordinator claims an answered run for its resume, and between that claim and the resumed turn lie steps that can stop the process — a crash, a deploy, a seat that moves. The row reads resumed from the claim until it is settled either way, so the seat’s next holder, finding the claim as it takes the seat, could not tell a reply nobody had acted on from one a turn had used, and read every one of them as spent: the person’s answer was lost without a trace. The resume now records, in one compare-and-set at the last moment before the turn runs, that the turn took the answer (answer.taken_at); a resume that cannot make that write does not run, and a release that gives the claim back for a retry clears it. A claim whose answer no turn took never used it — and never touched the box, because the take is the first thing a resumed turn does — so the seat’s next holder gives the claim back to the answer (sandbox_answer_revived): in one compare-and-set the run is answered again, fenced to the holder’s lease, with the lost claim counted on the answer (answer.lost_claims, answer.first_lost_at), and the holder drives its resume exactly as it drives any answer its last holder recorded and never resumed with — the seat’s mail held behind it, and the box still paused or, past pause_ttl_seconds, reclaimed and the run re-seeded from its branch, as any answered run’s is. It used to be reaped, the run ended and the reply handed to the seat as an ordinary message, so the answer reached the seat but never the run it answered, which was lost. A revival the store does not confirm — a write that failed, or a claim that moved under it and could not be read again — is retried by the holder on the hand-back’s spacing with the seat’s mail held behind it (sandbox_answer_revive_owed), never read as a reason to end the run; finding it landed after all, or the claim given back, the holder resumes the run it finds. A claim a node made itself and lost track of — the claim’s write reported a failure and landed — is held by no resume either, but it is not a lost claim: no node stopped. The answer’s next attempt on that node finds it and gives it back to the answer under the lease it was taken under (sandbox_answer_claim_given_back), uncounted, and goes on with the answer’s own attempts — resuming, or declining the answer once those are spent, never ending the run. That holds on the answer’s last attempt too, and when its window lapses under a claim: the attempt that spends the budget gives the claim back first and then hands the reply back once, rather than declining a run that is not answered any more — which the store refused, leaving the run in a claim nothing drove, the person’s answer on it and the seat held behind it. The run used to sit in that claim, the seat busy behind it, until the seat happened to move; and once that was fixed, a few of those claims in a row were counted as dead ones and ended the run as an abandoned_tail on a node that had never stopped. Revival is bounded, by two counts kept on the row, because a node that stops resets every count it kept: after 3 revivals of one answer (sandbox.MaxAnswerRevivals), or once the run’s own pause_ttl_seconds has passed since the first claim of it died, the next claim that dies is reaped instead. Three claims dying in a row is not bad luck but a resume that takes its node down before its turn begins, and reviving it again would hand it to the next holder to fall over the same way, the seat’s mail held behind it each time. The store will not delete that run with the reply on it, so the reap lets the answer go as above — the copy handed back, the run ended, and its loss announced once as abandoned_tail, saying why it was not resumed again and that the reply went back to the seat (sandbox_answer_not_revived). A claim whose turn took the answer has used it, and a copy or a second resume now would answer the person twice: it is spent, the run reaped, and the announcement says nothing went back. The take, the revival and the ending are exclusive in the store’s own compare-and-set: a revival and an ending both refuse an answer a turn took, a revived run has no claim left to take it under, and a decided ending takes no take — so whichever lands first wins, whatever any lease says, and a node still on its way to its turn when the seat’s next holder revives or ends the run cannot also answer the person with it. (Only the claim’s own ending lets go of an answer its turn took: a turn that gave its claim back as a retry has said nothing it did reached anybody, and once the hand-back cannot land, no retry will hand the reply to a turn again.) The reap fences the row first, moving the run to its own, newer lease, so a node that lost the seat but is still on its way to that turn has its take refused and the person is answered once, through the seat’s next holder, rather than also by a turn the seat’s lease will stop. That fence holds because every claim carries the claimant’s own lease: the claim stamps it on the row and every write the claim makes after it — the take, the hand-back, the ending — carries it back off the row; a launch stamps the lease that launched it the same way, and the seat’s new holder stamps its own on every run it recovers, parked ones included. And a take, a claim and the record of an answer are refused under no lease at all on a run any lease owns, which other writes let through so that a recovery holding no lease yet can still write: a run launched under no lease, or one nothing re-stamped, sat at the zero epoch, and a stalled claimant that held no lease took the answer on a row its successor had fenced — the person was answered by its turn and by the copy. A node that knows it lost the seat writes nothing on it. Its seat host no longer holds the lease, and every write the coordinator would make on the seat’s behalf — recording an answer, claiming one or a completion, giving a claim back, launching a run — is refused before it reaches the store (sandbox_seat_not_held): a reply or an answer by turn goes back to reach the seat’s holder, a completion is handed back to be fired again for the holder, and a retry this node was making is dropped for the holder’s recovery pass to drive. It used to write under the zero fence, which constrains nothing, so a node that had noticed it lost the seat recorded and claimed a person’s answer past its successor’s fence, took it and ran the turn on a seat it no longer held; one that has not noticed yet writes under its old lease, which the successor’s fence outranks — an answer it records is refused rather than left on a run its successor recovered as waiting, which nothing would then drive. The reap reads the row again after fencing it, so a claim its old holder gave back first is taken over as the run it is, owed its retry, rather than reaped. Where the reap can neither fence nor read the row, it is revived or reaped as listed: the revival is a compare-and-set that stamps the holder’s lease itself and refuses an answer a turn took, and past the bounds the store’s refusal to delete a reply no turn took is what hands the reply back. A resume that broke before its turn began — a panic re-entering the conversation — is settled without a retry like every abandoned resume, but nothing ran, so its reply is handed back the same way rather than read as spent, and the run is announced lost once, as resume_broken: no turn ran to publish a completion of its own, and the run used to end with a guard breach and nothing else, so the turn parked on it read as parked for good.

A reply’s original delivery is spent the moment the reply leaves the run. When a turn takes the answer — on the inline attempt and on a retry alike, at the same commit that records the take — and before the write that lets an answer go, by a decline, an ending or a reap, the delivery that carried it is recorded as worked in the completion ledger; a reap that finds an answer a turn took records it again, for a node that stopped between the take and its spend. A node that recorded the answer — or took it inline, holding the delivery unacknowledged through the whole resumed turn — and stopped before acknowledging the delivery leaves it to come round again, and by then the run may have used the reply (its turn may already have answered the person), handed a copy back, relaunched, or be gone, so nothing on the run would recognise it; the ledger drops it instead of running it as a second message — or, for an answer by turn, instead of asking the run about it again and announcing it gone or not_awaiting after it resumed the run. That holds for an answer by turn because its route reads the ledger before it reaches the run, and because its delivery is recorded whatever its type: the ledger otherwise keeps only the types that run a turn, which an answer by turn never does, so its spend used to record nothing at all. It used to be spent only once a retried resume’s turn had returned, so a node that stopped mid-turn left the original to run as a second turn on whichever node took the seat. A reply spent and still on its row is not lost: the row, never the delivery, is what resumes it or hands it back, and a copy handed back goes under an id of its own.

A delivery whose answer could not be recorded is handed back by deferral, and that series has three clauses. The first is the one the message carries: every hand-back spends one of the message’s deliveries, so the offer stops while 5 deliveries are still left and those are kept for the ordinary route. That reserve is read on each message’s own count, never on the partition’s: a delivery is a whole partition, its messages sit at different counts as a matter of course, and gated on the partition’s number a person’s clarification reply on its first delivery would be refused the answer route whenever an older message on the same conversation is near its own budget — and refused for good. One message with the deliveries to spare is enough to keep the offer open; sandbox_answer_offered_over_a_spent_sibling says when the hand-back may spend the last delivery of a co-partitioned one, and sandbox_answer_headroom_reserved when the route lets go. The second is 10 attempts on one message inside one process, and the third the run’s own pause_ttl_seconds (sandbox_answer_record_exhausted). A second reply gets its own count, and one run keeps at most four of them at a time.

Two questions open on one direct message are told apart by the thread each was asked in: the identity admits both — that is what makes a DM one conversation — and the run whose own partition matches the reply’s wins, with the most recently parked question answering a top-level reply that matches neither.

Who a question is put to

The audience a coding agent names is its own words, so the park resolves it — once, against the org chart, in the same write as the question — into the seats the question waits on (audience_handles on the run’s record and on GET /sandbox-runs):

Audience	Put to
`requester`	The seat whose message, notice or ask woke the turn that launched the run. The launch records it on the run (`requester`), because the park runs when the job finishes — days later, perhaps on another node — and sees no trigger.
`manager`	The seat’s manager.
`team`	The lead of the seat’s unit and the people in it; the unit’s other agents are left out, since the seat could ask them itself.
a name	The one colleague it names exactly — a handle, an id on a chat surface, or a role name, case aside. A partial or approximate match is not a person: a guessed colleague is somebody else’s question in their inbox.

When the audience names nobody the chart has — no requester was recorded, the seat has no manager, the name matches nobody — the question is put to the seat’s lead chain instead (its managers, nearest first, or, for a seat nobody manages, the leads of the units that hold it) and audience_fallback says so. A question is never parked waiting on nobody while somebody is answerable for the seat, and a person can tell a question put to them from one that reached them because nobody else could be named. ?audience=<handle> on the board is “what is waiting on me”.

Answering a parked run

A reply on the question’s conversation (step 3 above) is one way to answer. The other is to name the run by its turn, with the operator tool answer_run — from your own assistant over /operator/mcp, or from the dashboard. It is the only way for a run started by a schedule, a task assignment or a colleague’s ask: those turns have no conversation, so the run’s record names none and nothing a person says in chat can reach it. Until this existed such a run’s question waited out its pause TTL however promptly somebody knew the answer.

In the dashboard it is the Answer on a parked run’s banner — on the run’s page and its rail in Live › Coding runs, on Live › Now running’s “Waiting on a person”, in the Inbox and on Home. The banner names who the question is put to (the audience resolved when the run parked), how long the box is held, and whether a reply on the turn’s conversation would reach it too.

Who may answer is the run’s requester — the person whose message, notice or ask woke the turn that launched it — or whoever leads the run’s seat, or an administrator (admin), and never a seat: the person who asked for the work answers what it asks them, and nobody who merely leads that person does (see the authority table).

answer_run{turn_id, answer} checks the run is waiting (not_running if it is not) and puts a sandbox_answer_given event on the seat’s own inbox, answering pending. The node holding the seat takes it off the inbox before its inbox screening — it is addressed to the run, so the screening’s offer to a parked question and its turn are not for it — and hands it to the coordinator, which records it on the run exactly as it records a chat reply — the answer, the credential and the person it names, and the event it arrived in — and resumes the same suspended Execute loop with it, the answer attributed to the person who gave it. From the record on, an answer by turn is a recorded answer like any other: the resume’s first attempt runs inline, its retries are the coordinator’s with the seat’s mail held behind them, a resumed turn takes it at the last moment before it runs, and a claim whose node stops before its turn took it is revived by the seat’s next holder, which resumes the run with it (see Mid-run clarification). It used to be resumed with directly, held only by the claim the resume took: a node that stopped between that claim and the turn left a claim its successor reaped as an abandoned tail, and the answer’s delivery, coming round to the new holder, found the run gone. It is never a turn: whatever it finds, the delivery is spent there, and only an answer the run is still owed but could not be recorded against (the store could not be read, or the record could not be confirmed) is handed back to the broker for its spaced retry — a redelivery that finds the record landed after all is spent as the answer it already is. Once it has become something — recorded, declined, gone, not_awaiting — it is recorded as worked in the completion ledger, and so is an answer its run spends (taken by a resumed turn, let go of, reaped); the route reads the ledger first, so a redelivery after either is acknowledged without reaching the run and without a second announcement (sandbox_answer_by_turn_already_spent). That hand-back is a NAK, and on this route it is safe in a way it is not for a chat reply: the answer names its run, so coming back behind the seat’s newer mail lets no later message take its place, and it cannot be taken as the answer to anything else. An answer already recorded against the question wins: the run is answered, no longer waiting, and the answer by turn reports not_awaiting. An answer by turn the run could not be resumed with is let go of like a chat reply, after the same attempts: the run waits on its question again, and the copy handed back to the seat’s inbox — having no ordinary form to become — is spent there and announced declined, once, so the person knows to answer again; it is never recorded against that question again. One let go of because its run ended is announced gone the same way. Because it travels on the inbox, an answer given while the seat is paused waits for the resume, one given while the seat is busy with another coding run waits until that run settles or parks and is then the first thing offered, and one given while the holder restarts is still there afterwards. It answers the question it was given against, and no other. Any of those waits can outlast the question: somebody else’s answer resumes the run meanwhile, its turn calls run_sandbox again, and the new job parks on a question of its own — and an answer that named only its run claimed whatever the run waited on by then, resuming it a second time with the first question’s answer presented as the second’s. So the event names the question too — the job the run held when it asked (launch_id), off the row answer_run checked — and the run is resumed with it only while it still waits on that question; once it has moved on, the answer is spent as not_awaiting. Every new question comes with a new job, so the job identifies it. The question is required: an event naming none is spent without resuming the run, with the reason logged (sandbox_answer_names_no_question) — it cannot be told from an answer to a question the run has moved past — and answer_run never delivers one. Two answers to one question — a chat reply and an answer by turn, or two people, or a retried answer_run — contend for the one record, and the run is resumed once, however far apart they arrive.

Both routes leave a record. sandbox_run_answered names the run, the route (chat or operator), who answered (the chat sender, or answered_by and answered_by_kind as iam.ActorFor names the person — the seat they hold, which every person does — with the credential they came through in operator_id) and what the answer became: resumed, not_awaiting (another answer had it first, or the run is running a job), gone (the run had ended) or declined (an answer by turn every attempt failed to resume the run with, let go of — the run waits on its question again). The chat route used to leave nothing at all. A chat message that matched no parked run was never an answer and records nothing; an answer recorded on its run whose resume is still being retried has not become anything yet and records nothing until it has. A chat reply let go of becomes the ordinary message it is, and records nothing either.

The durable state is always git plus the persisted row — the WIP branch is pushed before the agent asks, so the paused box is a fast-resume optimization, never the store. Clarification rounds chain within the same turn, bounded by the turn’s existing iteration cap and the budget cascade.

The pause reaper. A human reply can take days, and E2B holds a paused box — billing for the snapshot — until something kills it. Nothing else would, so the completion poll enforces pause_ttl_seconds on the same tick it uses for running jobs. The box’s age is durable (paused_at on the run record, stamped when the box is paused), so the deadline survives an engine restart — and a seat handoff — instead of resetting with the process. Past the TTL, a parked run gets:

the box killed by id with sandbox.Provider.Kill, never Connect, which auto-resumes a paused sandbox and would boot the VM back up purely to shut it down;
the box released on the row (sandbox_id cleared, paused_at cleared) — which is also what makes the next run_sandbox provision a fresh box rather than reattach to a dead id;
the run flipped to reseed — still claimable, still matched by conversation, because the run is not over. The answer can arrive days later and still resumes the same suspended Execute loop; only the spliced reply differs, telling the executor the machine is gone and the brief must re-check-out the pushed branch.

A pause that was never recorded is still a pause. Stamping paused_at is a second store write, made once the box is already paused, and it is warn-only: the job is over by then, and refusing a collected result over a timestamp would throw the whole run away. So a run can park holding a snapshot its row does not date — and reading that as “no snapshot” skipped exactly the box that needed reaping, on a remote provider, for ever. A parked row naming a box therefore is a held box: with no stamp, the deadline runs from the row’s own last write, which on a parked run is the park itself (a park that does not land is not a park — see step 2 above), and which can only be later than the pause it failed to record, never earlier. The same reading draws the dashboard’s paused_at, so the screen and the reclaim cannot disagree about which boxes are being paid for. A pause held for any other reason takes no such fallback: every one of those lasts a single dispatch and is settled by the tail that made it, so there a missing stamp means the box is live.

An answer still waiting on its resume is still a clarification wait. A recorded answer (answered) is resumed by whichever node holds the seat, when the seat’s own conditions allow — and a seat that no node holds, or one that stays paused, can leave that resume waiting as long as the question waited. Its box is held for the whole of that wait, so the reaper treats it exactly as it treats a parked run: it kills the box and clears it from the row (with the same last-write fallback when no pause instant was recorded). It does not change the status. The answer stays recorded and is still owed its resume, which re-seeds from the pushed branch just as a reseed run’s would. If the answer is later let go, the question reopens as reseed rather than awaiting_clarification, because the box it was parked in no longer exists. A resume that has already claimed the run moved it out of answered, so the reaper’s flip loses, exactly as it does against an answer’s claim.

pause_ttl_seconds: 0 takes the same path with no snapshot at all: the box is torn down the moment the run blocks and the run parks straight into reseed.

The reaper is scoped to clarification waits on purpose. The lifecycle pauses a box in one other place — between collecting a completion and resuming the Execute loop — but that pause lasts a single dispatch and is settled by the tail that made it, so expiring it would only break in-turn reuse. The one paused box no tail will ever settle is a run whose engine died mid-resume; that one is reaped by the coordinator’s recovery pass when the seat is next acquired, the single moment the row is unambiguously abandoned.

Durability, completion, and observability

Pending runs survive restarts, and seat handoffs. Every detached run is a record in the fleet’s coordination store, not in the node’s own database, and the difference is the whole reason it is durable at all: a run outlives its turn, its process and sometimes its node, and the node that owns the seat next is the one whose recovery pass has to find it. Per-node, that pass listed nothing, so the suspended conversation was unreachable and a billed box was neither resumed nor reaped. On acquiring a seat (boot included), and for every seat a node already holds when an apply brings its sandbox up, the coordinator’s recovery pass (Coordinator.RecoverSeat) takes ownership of the seat’s running jobs and marks the seat busy again; the waiter then drives those jobs to completion. Recovery also takes over a run left in resumed: that state can only mean the previous engine died between claiming the run’s tail — a completion, or a person’s answer — and settling it, so nothing will ever pick it up again (the at-most-once claim already flipped) and its paused box would leak. A completion’s claim is reaped. A claim taken for a person’s recorded answer that no turn took yet is revived instead — given back to the answer, and the run resumed with it — and reaped, its answer handed back to the seat, only past the revival’s bounds; one whose resumed turn had already taken the answer is reaped as spent (see Mid-run clarification). The store revives and lets go of only an answer no turn took, and will not delete a run still holding one, so neither can happen beside the take. The reap fences the row to the new holder’s lease first, so the node that stopped cannot take the answer after all, and reads it again, so a claim that node gave back first is taken over as the run it is rather than reaped. Every run the seat holds is fenced the same way, a parked one included, so a node that lost the seat cannot claim the answer that arrives for it either. A run left in launching is reaped for the same reason one step earlier: that engine died between starting the job and writing the conversation a resume would re-enter, so there is nothing to re-enter and never will be. Both are only safe to reap here: taking the seat’s lease is what proves no live process is still driving the row, which for a launching row is the whole argument, since one on a seat this node already owns is a launch happening right now. A restart never orphans a job, because E2B boxes run server-side, independent of the engine, and any process can reconnect by sandbox_id.

The completion poll is THE completion signal. The sandbox.Waiter reconnects to each running box every 15 s (DefaultPollInterval) and asks the runner whether the job finished. There is deliberately no push callback from inside the sandbox: only a poll can see a job that died before its last step, and the tick has to run anyway because it doubles as the box keepalive (the TTL refresh that makes runs uncapped).

No box waits on another. Each tick claims the waiter duty, lists the runs and hands every box that needs work — a poll, or the reclaim of a pause past its TTL — to a task of its own, at most 16 at once across every tick in flight (sandbox.MaxConcurrentPolls, half the shared transport’s warm connections to one host, so the launches and collections beside the poll keep the other half). The tick does not wait for its tasks: the next one comes on schedule, renewing the duty and polling every box whose last poll has finished, and passes over a box whose poll is still running, so one box is never polled by two polls at once. A box’s completion is announced the moment its own poll finds it. Free slots go first to the boxes whose last poll ended longest ago, so a box that held one goes to the back of the line after its turn. Each poll — reaching the box, its keepalive and asking its runner — is bounded by the duty claim that authorised it: the claim’s TTL less one interval, 30 s at the default cadence, or 60 s (the control plane’s whole-request budget) where that comes first. A peer can take the duty only once it lapses, so a poll never overlaps a peer’s on the same box; a node told the duty is a peer’s stops its polls at once. A box that took the whole bound is not answering, and it costs that box’s poll (counted against reaching it, where it was reaching it) and a slot, never anybody’s tick: with N boxes held at the bound, a healthy box waits at most N/16 bounds, rounded up, for a slot — against a fifteen-minute TTL. The tick used to poll its boxes one after another, so a box whose in-box agent held a command open for its ten-minute silence bound held every box behind it, their completions unannounced and their keepalives unsent toward a fifteen-minute TTL.

Two signals cover every way a job can end:

The done-marker — the wrapper shell wrote the exit code after the agent process returned (clean finish or crash; the marker is non-empty by construction so it can’t be confused with “not yet”).
Process liveness: the box is asked whether the detached wrapper is still running. A read that fails never skips this probe: the probe still runs, a dead wrapper is still a finished job, and the read’s error is reported beside a “not done”. A remote or container box answers with kill -0 on the wrapper’s pid inside its own process namespace; a direct box answers from the job’s recorded identity (the pid and its start time, see Local sandboxes), because on the engine host a pid the kernel has handed to another process would otherwise read as the job and hold a dead run open. A dead wrapper with no marker means the whole process group was killed (SIGKILL / OOM); the run completes and collect surfaces the partial output and exit code instead of stalling forever.

Both coding agents this build drives exit when their work is done, so nothing is read off the end of a stream to decide it. OpenCode 1.x did not — it finished and held its process open — and was once polled for a terminal event at its stream’s end; OpenCode 2 exits when its session does, and its run’s script exits after it.

A reader never wakes a box. The poll and the live view reach a box with Provider.Attach, which never resumes it: on E2B it reads the box (GET /sandboxes/{id}) and refuses a paused one, rather than POST /connect, which wakes a paused box and resets its timer; a paused container is refused too, since its liveness probe is an exec it cannot take; a direct box is read as it is, stopped or not, since its files and its probe are the host’s. Only the collection and a follow-up run reconnect with Connect, which resumes. This closes a race: the poll’s list of running runs is a moment old, and a collection claims a run, reads its box and pauses it — a poll reaching the box after that pause used to wake it, nothing renewed it again, and on E2B it was killed at its timer, taking the parked run’s snapshot with it. A paused box sends the poll to the run’s record instead, and only a record still running the same job in the same box has it woken: that is a collection that paused the box and then handed its claim back to be tried again, and the box is the run’s again.

A vanished sandbox can’t orphan a run. If reconnecting fails on several consecutive ticks (~1 min at defaults), the waiter declares the box gone and fires completion anyway: E2B reclaimed it, a network partition, or the engine was down past the keepalive grace. The coordinator’s collect then cannot reconnect either, so it frees the agent, ends the run and announces it as sandbox_run_failed with reason collect_unreachable, rather than polling a dead box forever — at once when the provider says the box is gone, and after the collection’s own minute of retries (see Failure modes) when the box is only unreachable, which a partition that heals in that minute survives.

Per-run OTel, without handing the sandbox a secret. The coding agent’s telemetry exports to an engine-fronted OTLP receiver, enabled by setting CREWLET_SANDBOX_OTEL_RECEIVER_URL to the engine API base the sandbox can reach — the public listener’s address when Tier A sets api.public.port, since /otlp/ is served there and nowhere else. Each run gets a per-run, trace-scoped, expiring token embedded in the endpoint path (POST /otlp/{token}/v1/{signal}), so OTEL_EXPORTER_OTLP_HEADERS — the backend’s ingest credential — never enters the box; the receiver validates the token and forwards upstream, adding the real auth outside the sandbox. That variable is set empty in the box rather than left unset, because an exporter reads it from its ambient environment otherwise — and while a box inherits nothing of the engine’s own environment, its image or E2B template may export a value of its own, and so may a -e in local.run_args. The empty value is what makes the endpoint’s token the only credential the run’s exporter presents.

The token is signed and self-describing rather than a key into a store, because minting and verifying can happen on different nodes: the node running the seat mints when it launches a run, and whichever node the box exports to verifies. Every node derives its signing keys from the Tier A keyring, so a fleet agrees without sharing state. There is no per-process fallback: every node holds a keyring (a node without one does not start), and a keyring that cannot sign for the fleet is refused by name when the receiver is built, because a key of one process’s own would validate at the node that minted and nowhere else. Expiry rides in the token, so nothing is reaped and a restart does not invalidate a live run’s endpoint. The token also names the key that signed it, so a keyring rotation does not invalidate a run already in flight: the key stays on the ring until the operator drops it, and a token naming a key a node no longer holds is refused rather than retried against the active one. The runbook is in Secret store.

The receiver accepts and drops when no upstream is configured: the engine’s own per-turn span still carries the trace, and a deployment with no backend should not have every export fail. A failing backend is never reported to the box either — an exporter that gets a 5xx retries, and retries against a backend that is down turn one outage into two. The run env also carries non-secret resource attributes (crewlet.turn_id, crewlet.agent_handle) and a TRACEPARENT, so Claude Code’s spans/metrics nest under the turn (its CLAUDE_CODE_* telemetry toggles are injected only for that runner). OpenCode emits no OTLP today — its observability is the published transcript plus the engine-side lifecycle events.

A run keeps the work item its turn is on. The launch writes the turn’s work_item onto the run’s record and onto sandbox_run_started, and a question the run parks on carries it on sandbox_clarification_requested. The record is where the resumed turn reads its item from — the event that resumes it, a box’s completion or a person’s chat answer, names none — so the second half of the turn is charged to the same work as the first, under the basis resume (see Which work a turn is on). What the turn had written before it detached travels on its suspended conversation for the same reason. A record keeps every key it cannot read: each status flip, claim and release rewrites the whole record, and a node running an older build writes back the keys a newer one added rather than dropping them, so a rolling upgrade cannot strip the item, or anything added after it, off a parked run.

Dashboard. The live-state projection keeps the set of runs in flight, and it builds that set from two sources, because neither is enough alone.

The lifecycle events keep the set current. sandbox_run_started adds a run. sandbox_clarification_requested moves the run to awaiting_clarification. sandbox_run_completed or sandbox_run_failed removes it. A failure used to be read by nothing, so a lost run stayed on the panel as running.
The durable run record decides which runs exist. The record is read once when the node starts, before the node serves (before its listener is handed the API surface), and then every 30 seconds (livestate.ReconcileInterval). What the record holds replaces what the events built. A process that starts while runs are in flight never saw their starts, and a completion that never arrived would otherwise leave a finished run on the panel with nothing to remove it. Two cases keep the event’s version. A run whose event arrived after the read began is newer than the read. A run whose end the events already reported is not put back, because the record lags the end by the collection the end starts. A completion names the job it finished, and a failure is compared with the record’s last write.

This replaced a twelve-hour age-out, which was wrong in both directions. A finished run whose completion was lost stayed on the panel as running for twelve hours. A run parked on a question for longer than twelve hours disappeared from the panel, and a run waiting on a person is the one the panel most needs to show.

Each entry is {turn_id, role, agent_handle, agent_id, coding_agent, sandbox_id, task, status, started_at, question, audience, work_item, owner, paused_at}.

status uses the record’s own vocabulary: launching, running, awaiting_clarification, reseed or answered (a reply is recorded and the engine is resuming the run with it). A resumed run is over, because its turn has already taken back the result, so it is not listed. The panel used to write awaiting_input, a status the record cannot hold, so the dashboard never saw a live run as waiting on a person.
owner is the node driving the run.
paused_at is the pause reaper’s own reading of when the box began being held.

Every node reads the same fleet-wide record, so every node’s panel shows the fleet’s runs. The dashboard’s Coding runs screen reads the record directly (sandbox_runs). A run whose box has already been reclaimed is therefore still listed there, with the row’s own facts that the panel does not carry.

Watching a run live

Nothing a running job writes leaves its box until the run is collected, when its transcript — held to the record’s bound, its start and its end — rides the run’s agent_phase_completed{phase: sandbox} record. While it runs, its live output is a question, asked only while somebody looks: sandbox_tail{turn_id, launch_id} (GET /sandbox-runs/{turn_id}/tail?launch_id=…). There is no event and no row behind it.

It names the job, not only the run. A turn can launch more than one job, so sandbox_run_started carries the job’s launch_id (the one the run’s record holds and its phase record repeats) and the instant the store recorded the launch (started_at). A request naming only the turn would show whichever job the record holds now.
The node that owns the run answers it, from one reading per job. Every node serves crewlet.observe.sandbox_tail; the asker reads the run’s record, and the incarnation it names as the owner (PendingRun.Owner) re-reads the record and answers from its reading of the job (Runner.Follow): the transcript the CLI’s decoder builds from the event stream, or the error stream while the event stream has said nothing yet (a CLI that failed to start), read through a box attached without waking it (Provider.Attach). A reading begins at the start of each stream’s last mebibyte, on a line, and every read after takes only what was written since — a ranged read of the stream’s end sized to what it grew. When that mebibyte holds no line break at all — a run writing a line longer than the window, \r-only progress or a minified blob — the reading has no line to begin after, so it DROPS the rest of that one line, up to the first break that lands after it, the way a partial first line always is, and begins showing from there — rather than show it as a whole line redacted with none of the text before it, which could have put the tail of a credential straddling the window’s edge on screen. Those bytes are not counted as held, since nothing will ever show them; the reading says instead that it began after the stream’s start (front), which is what that line is. Nothing in the box is written, signalled or cleared.
What it shows is settled. A line is shown only once nothing written after it can change how it redacts (redact.Settled): a line still being written waits for its line break, a private key’s BEGIN and the lines under it wait while what comes next could still be the key’s body or its END — so a mention of the armour line waits only for the next line, which is not a key’s — a line that reads like a key’s body — base64 as wide as a key’s line, in upper and lower case both — waits until two lines in a row that cannot be one follow it with two more lines after them (four such lines in a row, most often, and five for a run exactly as wide as a key’s line, 64, 70 or 76 characters, which an END can read back to across two lines — counted again above a line that could be OpenPGP’s checksum, which an END reads back from as from itself), since an END written next could read it back — and a password key waits for the line that decides its value: the value’s own, past the wrapping’s blank lines, or the next one holding a key or not ending as the key’s line did. Not where that line only confirms what the key’s line gave on its own, so a run of password:" lines is shown as it is written rather than held back whole. A key followed on its line by punctuation and then more — msg="Enter password:" level=info, whose quote may be a wrapping’s — waits for the next line too, and so does a key a quote closes whose separator ends its line (case "password":): one line. Over the 53 files of Go’s source tree and x/crypto that hold a password’s key, read as written, numbered or as a diff, 14 or 15 lines wait one line longer than they did before a value was read on a later line, by at most 84 bytes. A hash in hex or a path in lower case is in one case and is shown at once; a Go test’s name, a CamelCase identifier or a path in mixed case as wide as a key’s line is in both, with or without a digit, and waits like a key’s line, so a verbose test run’s output can arrive in bursts, held as far back as 64 KiB until lines that are no key’s release it. A line holding a run that reads like a piece of a key anywhere on it — eight characters or more in both cases with a digit, which an END written next could read back to as a line a cut wrote into — waits too, until four lines that hold none follow it. Fed go test -v output of Crewlet’s own packages a line at a time, a few hundred bytes wait on average and at most about 4 KiB. So what a viewer sees, read by read, is exactly the whole text redacted — never a key’s body that its END line, a moment later, would have hidden. What is held back is counted (held), so the screen can say why a line has not appeared.
One read of the box answers everybody watching. The owner reads a job’s box at most once every two seconds (sandbox.LiveReuse), however many people have the run open, and requests that arrive while a read is in flight wait for that one. Each answer waits for the box at most three quarters of its asker’s budget (the request carries it), so a request the asker gave up on frees its answer slot instead of holding it for as long as a slow box takes; the read itself goes on, under its own budget of one completion-poll interval, and the next request is answered from what it found — or, if the box is still being read, from the last read there was, stamped with when it was taken. A reading nobody has asked about for 30 seconds is dropped by the owner’s own sweep, every ten seconds, whether or not anything asks it anything again — which in a fleet nothing does once a run stops: the node serving the dashboard reads the run’s record, answers not_running itself and never asks the owner again, and a viewer who closes the page asks nobody.
A viewer is sent only what it lacks. Every request says what the asker holds — the reading’s epoch, the byte offset it holds through (after) and the digest the owner gave it there, all three absent on a first request, which holds nothing — and is answered with the text after that offset, its start and end offsets beside it — always both, 0 included, since a run that has settled nothing yet is answered at 0 and followed from it. Anything else is a reset carrying the last 256 KiB in whole lines (the most a viewer holds, the record’s own bound): an epoch that names another reading (the owner restarted, the reading began again further on, the account switched from the error stream to the transcript), a digest that does not match the window the asker holds (a node on a build that derives the stream differently), or a viewer more than 256 KiB behind. A run whose owner moves to a node on the same build continues: the new owner’s reading of the same stream from the same place derives the same text, and the digest says so. A local box cannot be reached from another host at all, so a run moved off its host answers an error naming why.
Five outcomes, never an empty answer standing in for another. tail is the output (possibly empty: a run that has written nothing yet says so). launching is a job whose box is still being made and provisioned — not a run that stopped, and a screen keeps asking. not_running names the record’s own status — parked on a question, replaced by a later job on the same run, or no record at all because the run was collected. owner_silent{node} is an owner that did not answer inside the two-second fleet read budget — an owner that is draining or whose heartbeat lapsed is asked like any other, and its silence is this — or a run no node holds right now, and it names the node. box_paused is a running record whose box is paused where its backend cannot read it without waking it — the moment between a collection pausing the box and the record moving on — and the box is left paused. A box the owner could not read is an error carrying the owner’s own sentence.
The dashboard asks every three seconds, and only while somebody is looking: while a running run’s span is open on the turn trace, or while the run’s own page (Live › Coding runs, #/live/runs/{turn_id}) is open on a running job. Each request is chained after the last answer, so one open view has one request in flight; closing it or the run ending stops the poll. The run’s page asks by the launch_id the run board names (sandbox_runs serves it), so it shows the job the row holds now, and when a later job replaces it the page reads the board again and follows the new one.
After collection the record is the turn’s. A collected run’s row is deleted, and its page reads the turn instead: every sandbox phase the turn published — what the job reported, what it delivered, its tokens and its transcript — under the task its sandbox_run_started carried. A link to a run that has finished still lands on what it did.

Budgets

The coding agent calls the LLM itself, from inside the sandbox, so the engine cannot gate it mid-stream like the native loop. The model is cap + post-account:

Pre-flight floor. A launch is refused when the agent’s remaining token budget is below turn_engine.sandbox_min_budget_tokens (default 2000); the run_sandbox call returns a normal failure the executor can act on, without suspending.
Self-limits. role.sandbox.max_turns becomes the coding agent’s own cap — Claude Code’s --max-turns — so the run self-limits. And a pre-flight floor: a seat whose remaining token budget is below turn_engine.sandbox_min_budget_tokens (default 2000) is refused the launch outright, because a coding run costs a box, a clone and a toolchain install before it produces a token. A budget read that FAILED is refused too — launching on an unknown budget is how a company discovers its ceiling by spending past it.
A spent seat is not resumed into a refusal. On a seat no run holds, a person’s reply to a run’s clarification question is offered to the run only after the budget park has asked whether the seat has room, because resuming the run charges tokens exactly as a turn does. A seat whose window is refusing holds the reply on its inbox until the window turns over, then offers it. A collected run that takes a window past its ceiling parks the seat’s next delivery the same way.
Post-accounting. On collect, reported usage is charged to the budget cascade, and the phase event carries the real usage and cost so the Tokens view rolls sandbox spend up alongside native phases. Mid-run enforcement is best-effort by design, the price of the capability. OpenCode’s tokens and price come from its session’s own report (OpenCode), and a run whose report is missing reports zero rather than an estimate. What a run reports is never a refund: the usage is parsed from what the coding CLI printed inside a box the run’s own agent can run any command in, so a negative token count or cost is read as nothing spent (sandbox_usage_negative) rather than subtracted from the seat’s counter and from every rollup — every runner hands such a figure on as reported, so the floor, and the warning, apply to all of them alike.
Charged once per run. A run’s tokens reach the seat’s counter and the company’s once, however many times its completion is delivered. A resume that fails (the seat moved between the publish and the receive, say) sends the completion back, and the retry collects the same finished job again; the run’s coordination record says its spend is already counted, so that retry charges nothing, whether it lands on the same node, on the seat’s next owner, or after a restart. The record is written by the same compare-and-swap that hands the completion back, so no retry can find the run reopened without it. A second run_sandbox call in the same turn is a second job and is charged on its own. The charge is a post-charge: the tokens are already spent, so no cap can refuse them, and both counters record the run in full — in the day, week and month it is collected in, on the company’s clock — whatever the ceilings say. A run that takes a counter past a cap is logged as sandbox_spend_over_budget, the turn continues regardless, and the next round the seat or the company attempts is refused against the recorded figure — the resumed turn reads the counter once before it calls anything, so its own next round is refused before it is sent rather than billed and then refused. An over-cap charge is recorded like any other, so a retry does not offer it again; only a charge the counter never answered (sandbox_accounting_failed) is, and if that one had in fact landed, the retry counts it twice, which trips a cap early rather than late. A charge that reached the company’s counter and failed on the seat’s (sandbox_accounting_partial) keeps the company’s share — nothing a counter recorded is taken back — and the run’s record says so, so the retry records the seat’s share alone rather than counting the company twice. A resume that succeeds after such a charge brings no retry, so that one seat’s own counter stays short of the run until its window turns over; the company’s stays exact.
Charged to the task too. A turn on a native task charges that task for the runs it detached: the turn segment that resumes from a collected run adds the run’s tokens to the task’s own spend, beside its own phases, along with the condensing of its report, failure or question where one was past what the run’s record carries (see Spend is on the task). A run that stopped to ask a question has no such segment until a person answers, so its tokens and that condensing are recorded with the question on the run’s row, and the segment the answer resumes is the one that pays them — whichever node it runs on, however long the question waited. Each segment is recorded once, under the run and launch it resumed, so a retried resume adds nothing.

Security model

The sandbox is the isolation boundary. Arbitrary, autonomously generated code runs there, never on the engine host. That is the whole reason running the coding agent with permissions bypassed is acceptable: the blast radius is one ephemeral, provider-isolated VM that is torn down at phase end. Human gating, where wanted, happens at the merge request — not mid-run.
run_in: direct moves that boundary, deliberately. There the coding agent runs as the engine user: per-box state isolation and an allowlisted environment still hold, but the host does not. Pick it for a workstation or a dedicated VM you are willing to hand to an autonomous agent; pick container (or e2b) anywhere else. This is the one cell in Crewlet where the sandbox is not a security boundary, and it is opt-in twice over — a local: block plus an explicit run_in naming it, on the seat or as the company default. Because the choice is now per seat, a company can run the seat that needs the host login direct and every other seat somewhere the host is not reachable, which the block-wide mode this replaced could not express.
Only the seat’s own identity enters the box. The injected env carries credentials the agent legitimately acts as — its LLM key and the external-service tokens its config declares (role.sandbox.env). Leaking those from inside the box exposes only what the agent already is. Because the code-host token is the seat’s own PAT, an MR/PR the coding agent opens is attributed to the agent, not to a shared bot identity.
Shared infrastructure secrets never enter the box. The OTLP backend’s ingest token is the canonical example: the sandbox sees only an engine-fronted, path-token-scoped OTLP endpoint (CREWLET_SANDBOX_OTEL_RECEIVER_URL); the engine attaches the real backend credential upstream. Apply the same rule to any credential you add — if the agent doesn’t act as it, don’t inject it.
Injected values are ephemeral. ${VAR} references are resolved at acquire time into that sandbox’s process env only; the database stores references, never values, and the values die with the box.
Workers cannot launch sandboxes. run_sandbox is on the worker control denylist, alongside delegate and the discovery meta-tools, denied by name whether a task names it explicitly or the worker finds it through discovery. The role.sandbox.enabled gate does not cover this on its own: it is a property of the seat, not of a phase, so on a sandbox-enabled seat the tool would otherwise be reachable from a worker’s grant too. The reason it must be denied is ownership rather than capability: a detached run is keyed to the parent turn (the pending row carries the parent’s turn_id, and a completion resumes the parent’s suspended executor), and a worker’s loop cannot suspend to wait for it, so the parent turn would finish without persisting a suspended conversation and the run would come back with nothing to resume into.
Network egress is the provider’s policy. The engine does not enforce egress rules; use your E2B template’s network policy for that, and prefer an allowlist (the LLM endpoint, the git host, the engine’s OTLP receiver) in templates for sandbox-enabled production roles.
Budgets still bound the run (previous section), and the collected artifact passes through the same result validators as a native Execute before reaching Review.

Failure modes

What an operator should recognize:

SandboxCredentialError at launch — the seat runs code in a remote cell on a cli-agent provider whose login cannot follow it there. The login stays on the engine host (it carries a refresh token whose rotation is fleet state), so unless something in the run environment authenticates, the coding agent would fail at its first model call with the vendor’s own “not authenticated” — minutes in, naming nothing you could act on. Only a claude-code or opencode entry reaches this check (any other is a SandboxModelError first). The run never starts, and the error carries your remedy: Claude Code, which mints a headless token, says “run crewlet llm login <key> -capture-token”; OpenCode, which reads its provider’s key from its environment, names the provider its model names (openrouter for openrouter/anthropic/claude-sonnet-5) and says to set that provider’s key in the entry’s cli.env or the seat’s role.sandbox.env, both exported into the box — for OpenCode any credential-named variable in the run environment satisfies the check, since which variable it reads depends on that provider; and a profile that does neither — a custom one, or a built-in one whose cli.overrides removed its token or its environment sign-in — says to give the seat run_in: direct or run_in: container. A credential you declared yourself in role.sandbox.env counts — the check asks the CLI’s profile which variable names authenticate and looks for those in the merged run environment, rather than trying to recognise a credential by inspection. That is what lets it be a refusal instead of a warning. An unresolved ${VAR} never counts: a run_sandbox run is refused for it before this check, and a key left out — declared optional, or in an agent-mode run — is not in the run environment at all.
SandboxModelError at launch — the seat’s coding agent cannot use the entry its code work runs on (role.llm_sandbox, else llm, else the company default). Either the entry is a cli-agent entry driving another CLI — its model is written for that CLI and its sign-in is that CLI’s, so a claude-code entry under OpenCode or an opencode one under Claude Code would start a box that fails at its first model call — or Claude Code was given an openai / openai-compatible entry and the seat signs Claude Code in with nothing of its own. The run never starts, and the error names the entry and the remedies for its case. For a cli-agent entry: point role.llm_sandbox at an API entry or at a cli-agent entry driving the runner’s own CLI, or — when the entry’s CLI has a runner of its own, which only claude-code and opencode do — set role.sandbox.coding_agent to it. For Claude Code on an OpenAI-wire entry: point role.llm_sandbox at an anthropic entry, set role.sandbox.coding_agent: opencode, or give the seat a Claude Code sign-in of its own in role.sandbox.env. The cli-agent pairing is also refused on the write that creates it (an admission rule, warned about on a revision being applied), reported at the seat’s llm_sandbox.
setup step "…" writes …, which a direct box refuses (or a container box, or an e2b box), from crewlet validate or a write: a setup step writes a file outside the home of a local box some seat runs in, or one an e2b box refuses — another account’s ~name/…, or the home itself or a directory above it — so every box of that seat would fail at setup. The problem sits at the file’s entry in the step, and names the first seat whose box it fails and how many more. Name the file relative to the box’s home instead — .npmrc, .local/bin/tool, or ~/.npmrc — which every cell writes under its home, or have a command of the step write it under "$HOME": a direct box’s home is minted per box, so no absolute path a file entry holds is inside it. Moving the seat to the other local cell helps only one way round: an absolute path under /home/user, which a direct box refuses, is taken by a container box, while a path outside the home is refused by both. Moving it to e2b helps for any absolute path the local cells refuse, /home/user/… and /usr/local/bin/tool alike, since that box writes both; it refuses another account’s ~name/… and the home itself or a directory above it, as both local cells do.
local sandbox (run_in "direct") refuses to touch … or … is not a file under the sandbox home mount at /home/user (a sandbox.LocalError): a path a box was asked to write or read is outside its home, which neither local cell writes outside of — direct because the path would be the engine host itself, container because only the home is mounted. A setup file that names such a path is refused on the write (above), so at launch this is a stored revision that keeps such a step — an apply warns about it rather than refusing it — or a link an existing directory on the engine host holds, which only the box can see. For a setup file the remedy is the one above: name it relative to the box’s home, or have a setup command write it under "$HOME".
… neither docker nor podman is on the engine host's PATH (a sandbox.LocalError): run_in: container with no container runtime on the engine host’s PATH. Install one, name it in local.runtime, or use direct.
cannot see its own box directory — the runtime is driving a daemon on another host, so the bind mount is that machine’s filesystem rather than this one’s. Point local.runtime at a local daemon, or use run_in: e2b for a remote runtime.
"container" needs the local: backend configured under providers.sandbox — a seat names a cell the catalogue does not hold. Add the block the cell needs, or point the seat at one that is configured; the message lists what this catalogue has.
the catalogue offers more than one place to do it — a sandbox-enabled seat (or an agent-mode entry) names no cell and the catalogue names no default_run_in. Name the cell where the message points, or name the default. A catalogue is not refused for being ambiguous on its own — only a seat or entry that would actually fall to the missing default is.
nothing names one, so no backend is built — an ambiguous catalogue that nothing reaches: no default_run_in, no seat’s run_in, no agent-mode cli.run_in. Nothing would be built and no seat could run code, so the block reads as configuration and configures nothing. Name the default, give a seat a run_in, or remove the block.
writes its box directory as a user this engine cannot manage — the container is running as someone the engine cannot follow into the mount, which happens when run_args overrides the --user the backend chose. Drop that override, or run the engine as the same user the container does.
sandbox_env_unresolved, and a run_sandbox result saying this seat's run environment is not complete, so no box was started — a ${VAR} in the run environment, or a setup file that is one whole ${VAR}, resolves to nothing and its key is not declared optional. The line names the keys and the <step>:<path> files, never a value or a body; an agent-mode executor started without them instead of being refused. At info the value is being minted for the seat by a provisioning pass, and a run launched after its seal reaches this node is given it. As a warning, set the variable the line names (crewlet secrets set, or the integration’s provision command), act on the integration’s finding where its pass is blocked, or list the key under optional_keys where the run can go without it.
Setup-step failure (sandbox_setup_failed) — a step’s provisioning did not complete, and the box is torn down before any brief runs. The message names the step and says which of three things failed: could not write <path>, a file the box refused (in a local box, with the reason above), command #N failed (exit S), a command that exited non-zero (in every cell, 124 is its timeout_seconds running out, with the timeout named before what the command printed, and a status above 128 is a signal that ended it — 128 plus the signal’s number, 137 for a SIGKILL such as an out-of-memory kill — as a shell reports one; where the signal ended the box’s own shell in a direct or e2b box, rather than a program that shell waited on, the signal is also named after what the command printed, while a container box’s command reads as docker exec reports it, the status alone), or command #N reported no exit status, one the box could not start or whose box went away before it finished, with the box’s own account after it — an e2b box’s in-box agent closing the command’s stream with an error of its own names that error there. This is an operator config problem (a broken recipe), logged distinctly from a coding-agent install failure.
Sandbox lost mid-run: the waiter’s gone-detection above. The run is ended and announced as sandbox_run_failed (reason collect_unreachable); look for sandbox_connect_failed streaks preceding it. If the engine was down longer than default_timeout_seconds, E2B reaped the box as an orphan, and that TTL is the knob to raise for deployments with long maintenance windows.
Collect failure: the box could not be read back. It is asked again first: nothing the completion writes — the charge, the run’s phase record, the park, the resume — happens before a collection succeeds, so handing the claim back leaves the run exactly as the completion found it (running, its box kept alive, its turn suspended), and the completion comes back on the queue’s backoff while the poll fires a fresh one on its next tick (sandbox_collect_retried). The allowance is the poll’s own for an unreachable box — at least two consecutive failed attempts spanning a minute, counted on the run’s record so another node or a restart does not start a fresh one, and ended by a collection that did read the box (one whose park or resume then failed and handed the claim back), so a box that answered is never given up on the strength of failures from before it answered — and past it the box is torn down, the run ended and announced (collect_unreachable), and the agent freed. A box its provider says is gone is settled at once, since no attempt can read it. A collection this node stopped — a drain, a restart, the seat moving to another node — is handed back without being counted (sandbox_collect_interrupted), because it says nothing about the box; a collection that ran out of its delivery’s time is counted, or a box whose read always outlasts it would be retried for ever. A file merely too large to read whole is never a collect failure: that piece is described by its size and the run is collected without it.
sandbox_suspension_* errors (sandbox_run_failed with reason suspension_unrecorded): the turn suspended but the conversation a resume re-enters never reached the run’s record, because the runner recorded none, it would not serialize, the coordination store refused the write, or the run was no longer launching. The coding job was already executing, so a run still launching has its box reclaimed at once and is ended, rather than left to run to its TTL with nothing able to collect it. A run that is no longer launching is left alone and not announced: a write reported as refused can have landed, which makes it an ordinary suspended run the completion poll resumes, and a run already ended was announced by whoever ended it. A store refusal points at the coordination store; the other three are a defect in the engine worth reporting.
sandbox_run_failed with reason claim_unreverted: the coordination store refused two writes on one row in a row. A completion claims a run’s tail before collecting it, and every failure after that claim is recoverable only because the claim can be given back — so when the park a run asked for, or the revert a failed resume makes, cannot be written and the claim cannot be returned either, the run is ended here instead. The row would otherwise sit in resumed, which no completion poll reads, no redelivery can re-claim, no answer matches and no pause reaper expires: an unrecoverable turn plus a paused box billed until the seat changes hands. Look at the coordination store — this reason means it refused or could not be reached twice within one delivery.
turn_rebased (a log line, not an announcement): a turn’s unit of work began more than twenty-nine days before this attempt — a run resumed that long after its work began, or a trigger dispatched that long after it arrived — so its operation ids are minted at this attempt rather than at that start, and the instant is recorded so a later attempt at the same work inherits it (see How a coding task runs). seed is the unit of work, or the run where there is none; began is the start, inherited the instant an earlier attempt had recorded (the start where none had) and rebased_to the new one. Nothing is wrong and nothing needs doing: it says that a tracker or knowledge-base write an earlier attempt made and this one repeats is a second write rather than a retry of the first — minted at the start, every node’s operation ledger would have answered every new write of this attempt unknown instead. It is logged once per rebase; an attempt that inherits a recorded instant logs nothing above debug.
sandbox_answer_headroom_reserved (a log line, not an announcement): every message of this delivery has spent nearly all of its deliveries, so the answer route stopped offering it and the 5 that are left are kept for the ordinary route, which is being run on it now. It is what stops a reply being handed back until the broker dead-letters it, and it is the expected end of a resume that fails on every node rather than on this one — the per-process attempt count below cannot see that, because a seat handoff resets it and the broker’s count is on the message. The line names the seat, what the partition had left and the reserve. A parked run is unaffected: it keeps waiting, bounded by pause_ttl_seconds.
sandbox_answer_offered_over_a_spent_sibling (a log line): the opposite decision, and the one whose cost is paid elsewhere. A message of this delivery still has the deliveries to spare, so a parked run is being offered it — while another message in the same partition is inside the reserve, and a hand-back returns the whole partition. It is the advance notice for a dead_lettered line on that sibling, and it is the expected shape of a conversation whose earlier message has been handed back many times and which is still collecting fresh replies.
sandbox_answer_resume_failed (a log line, at warning): the answer recorded on a parked run could not be resumed with — the error says why — and the coordinator will try again after the spacing the line names. The seat’s later mail is held behind it meanwhile. Nothing is asked of the person: the answer is on the run.
sandbox_answer_requeue_exhausted (a log line, not an announcement): this node could not resume one parked run with the answer recorded for it in every attempt its budget allowed — ten of them, spaced a second apart doubling to thirty, or as many as the run’s pause_ttl_seconds left room for — so the answer was let go: the reply was handed to the seat’s inbox as the ordinary message it is and is being worked as the turn it looks like, and the run waits on its question again with that reply declined for it. Its box is still bounded by pause_ttl_seconds. The line names the run and the window; the error beside it is the one to read, and it is nearly always this node rather than the run — no resumer on this node, a suspended conversation a peer’s build wrote, or a seat that is not in this node’s company. The seat’s next holder, or this node after a restart, starts a clean set of attempts on the next reply.
sandbox_answer_record_exhausted (a log line, not an announcement): this node could not RECORD one message as the answer a parked run is waiting on — the coordination store refused the write — in every attempt its budget allowed, so the message was let go to the ordinary route.
sandbox_answer_lookup_exhausted (a log line, not an announcement): this node could not read which of a seat’s runs is awaiting a reply — the coordination store refused the lookup — for every attempt the message’s budget allowed, while the seat’s own count said one of its runs is waiting for one. The message has been let go to the ordinary route and is being worked as the turn it looks like. The line names the seat and the delivery; the error beside it is the store’s. sandbox_answer_lookup_failed is the same failure before the budget is spent, at warn, once per attempt.
sandbox_claimed_settled_unread (a log line, not an announcement): the resumed turn came back — cleanly or having broken after writing outside the engine — and the run’s record could not be read to find out which box to reclaim. The settle does not go quiet over that: leaving the row in resumed is the same unrecoverable state claim_unreverted exists to avoid. Instead it asks the coordination store to end the run while it is still the claim it took, which is what keeps it off a job the resumed turn relaunched — a relaunch moves the row back to launching and reuses the same box, so the ending declines it (sandbox_claimed_settle_declined) and that run is ended by its own turn or by the seat’s next recovery pass. A store that refuses this too logs sandbox_claimed_settle_kept: the ending is kept, and this node finishes it — box included — once the store answers, or the seat’s next recovery pass reaps the run if the seat moves first; both lines point at the coordination store.
sandbox_run_failed with reason abandoned_tail: the node that held this seat stopped — a crash, a deploy, a lease that lapsed — between starting a run and writing the conversation it re-enters, or between claiming its tail and settling it, and the seat’s new holder ended the run as it took the seat (sandbox_abandoned_tail_reaped). Nothing retries it: the turn that suspended into the run cannot be continued. A claim for a person’s answer that no turn had taken yet is not ended this way but revived and resumed with the answer (sandbox_answer_revived, below) — unless 3 claims of that answer have already died the same way, or the run’s pause_ttl_seconds has passed since the first did: then the detail says which, and the answer has gone back to the seat as the ordinary message it is, once (sandbox_answer_not_revived names the bound with its reason and lost_claims; sandbox_abandoned_tail_reaped carries answer_handed_back=true, and the announcement’s detail ends “the answer a person gave it goes back to the seat”). answer_handed_back is the field to search: the line’s answer_held and answer_taken say only what the row held, and a held answer a turn took is held too. A line with ending_decided=false was written before the store confirmed the ending; its retry decides it, and the announcement it makes says whether the answer went back. An answer a turn had already taken was used by it and is not handed back (answer_handed_back=false). Look at why the previous holder stopped — and, past the revivals, at what stops a node between a resume’s claim and its turn: three in a row is a resume that takes its node down, not a coincidence of deploys.
sandbox_answer_revived (a log line, at info): the seat’s new holder found a person’s answer claimed for a resume by a node that stopped before its turn took it, and gave it back to the run, which it now resumes. lost_claims counts the claims of this answer that have died so far and first_lost_at dates the first; nothing is asked of the person. A sandbox_answer_revive_owed warning is the same holder unable to confirm the revival with the coordination store: it keeps the claim, holds the seat’s mail and tries again every 30 seconds.
sandbox_run_failed with reason resume_broken: the turn a run belongs to could not be re-entered — resuming it broke before the turn began, typically a panic decoding or re-entering the suspended conversation (sandbox_resume_panicked and a turn.guard_breach on the log). It is not retried, because every attempt re-enters the same bytes and reaches the same defect; a person’s reply that drove the resume goes back to the seat as the ordinary message it is, and the detail says so. Look at the panic. A resume that broke after its turn began is not announced here: that turn published its own failed completion.
sandbox_run_failed with reason seat_removed: the seat left the company’s agent seats — it was removed from the chart, or made a person’s and not made an agent’s again within the 24-hour retirement grace — so its runs were ended when its mailbox was retired. See Seat Ownership § The removed seat.
A run that “finishes” with no report — the coding agent neither wrote its findings file nor streamed a final message. collect surfaces the exit status, the CLI’s own account of how it ended (an error subtype, or that its stream stopped before its result) and the stderr as the error; the resumed executor sees a failed run rather than an empty success.

Testing

providers.sandbox.fake: true wires the in-process fakes (sandbox.FakeProvider, sandbox.FakeRunner): scripted results, an in-memory filesystem, no network, per the project rule that tests never touch real services. The runner tests pin the exact CLI invocations (claude -p … --output-format stream-json --verbose, opencode run … --format json) and output parsers — Claude Code’s against fixtures written from the vendor’s documented stream-json message schema, never a captured run, and once through a real shell with a stand-in CLI so the wrapper’s copy of the result line and the CLI’s own exit status are exercised as they run — and internal/config/examples_test.go parses the git-auth recipe out of this page and holds it to its security properties (host scoping at both layers, --add rewrites, identity from the engine’s agent facts) so an edit cannot quietly widen them. This is the copy readers take, so this is the copy under test.

Part of Crewlet. Generated from crewlet/crewlet main at ee70177.