Secret Store

The secret store is the company’s encrypted credential store: one sealed value per ${VAR} name, held on the coordination KV where every node reads the same rows, and consulted ahead of the process environment. It gives Crewlet a place to keep a secret that the engine can read back — so a provisioner that mints a credential can hand it straight to the engine instead of writing a file a human must source, and a rotation reaches the whole fleet rather than the one node an operator pointed a command at.

It is optional and inert until used. With nothing stored, ${VAR} resolution is byte-for-byte what it has always been: the process environment.

Related: Configuration § Secrets covers whole-config encryption at rest — a different mechanism with the same keyring. See Which one do I want? below.

The problem

${VAR} substitution has exactly one choke point in the engine, and its only source used to be the process environment. The stored config keeps the reference ("${GITLAB_TOKEN_SWE}"); only os.Getenv could answer it.

That is why the provisioning CLIs write .env files: the environment was the only address the reader knew. It leaves an awkward seam in an otherwise automated flow —

crewlet gitlab provision company.yaml    # mints PATs → .env.gitlab
source .env.gitlab                       # ← a human, in the right shell
crewlet run                              # ← restarted, in that same shell

— and every step in the middle is a way to get it wrong. The failure is silent: a ${VAR} nothing answers resolves to "", which produces an empty Bearer token that looks live to every layer until the provider rejects it.

The design

One record per environment-variable name, each value sealed with the Tier A keyring before it leaves the process:

_secrets (a coordination KV bucket, no TTL)
  name        the key            -- "GITLAB_TOKEN_SWE"
  value       "enc:v1:<key_id>:<base64>"
  key_id      which keyring entry sealed it
  updated_at
  updated_by       the author: the seat's handle for a person (every person
                   holds one) or a credential bound to a seat, the login of
                   a credential bound to none — `token:<id>` for a Tier A
                   token — or the engine's own name for a key or a pass it
                   wrote itself
  updated_by_kind  agent | human | operator | system
  operator_id      the credential it was stored through — `pat:<id>`,
                   `session:<lineage>`, a Tier A token's login — and empty
                   for a write no credential made
  source      the write path's own word, never the caller's:
              "api" | "setup" | "provision" | "migrated" on the fleet,
              "iam" on the engine's own keys, and "cli" | "provision" |
              "llm-login" on a stopped node's own table

Who stored a value is three facts, and a rekey keeps all of them. The author is whose authority the write exercised and the credential is what it was exercised through; a row that kept only one used to keep the credential, so a value stored through a person’s machine token named pat:<id> — and that token’s own row, the only thing saying whose it was, is swept a week after it lapses. A rekey re-seals a value it did not choose, so it leaves every provenance field as it found it: it used to stamp its own caller and rekey over each row, and after a rotation every credential in the company read as set by whoever rotated the keyring.

source is provenance, never location: every record is sealed in this bucket whichever path wrote it. It is the write path’s own word, and no request names it. PUT /secrets/{name} stamps api whoever calls it — the dashboard, crewlet secrets set and every provisioning command run against a running node, any other client — and refuses a write that carries a query string at all (400 invalid_query). An integration’s setup in the dashboard stamps setup, an integration’s provisioning pass provision, the engine’s mint of one of its own keys iam, and the boot-time move off a node’s own table migrated. A stopped node’s own table records the command’s word instead — cli for crewlet secrets set, provision for a vendor’s provision -secret-store, llm-login for the headless token crewlet llm login stores and the bundle crewlet llm export -secret-store packs — and the move onto the fleet replaces it, so the fleet’s listing shows only api, setup, provision and migrated. Each write replaces the word, so it names the LAST writer — and a rekey, which writes no value of its own, leaves it as it found it, like the three facts above.

Why the request cannot choose it. ?source= and -source once stored whatever word they were sent, so anybody holding the write could make a row claim any path, and a client that recognised its own rows by their word took a row somebody else had written over, under that word, for its own. Who wrote a row is what the author and the credential say, and both come from the credential the write authenticated with; the source only says which path carried it, so each path stamps its own and the stores refuse any other word.

The name is the reference grammar’s, and a write that is not one is refused. A record is keyed by the name a ${VAR} resolves through, so the two rules are one rule: letters, digits and underscores, starting with a letter or an underscore. A store that accepted gitlab-token would seal the value, list it, report the write as done, and resolve it from nowhere: the operator’s only evidence a provider failing to authenticate hours later, far from the name they chose. Every write path checks: PUT /secrets/{name} answers 400 invalid_name, and crewlet secrets set refuses against a running node and a stopped one alike. Reading and removing take the name as given, so nothing becomes unremovable.

Coordination owns the bytes; the engine owns the key. The bucket holds an envelope whose key it does not have, which is what makes a shared store safe to put credentials in: a peer that can read the bucket learns which names exist and when they changed, not what they are. It is the same cipher, the same keyring and the same bucket family the company config already travels through.

The bucket has no TTL, unlike the delivery dedupe and the notification valve beside it. Retention elsewhere in coordination is a bucket’s age; a credential is not short-horizon state, and an expiring secret is an outage on a timer.

At boot the engine loads every record into a process-local snapshot and installs it as the secret source. From then on ${VAR} resolution asks the store first and falls back to the process environment:

Because substitution has a single choke point, that one change covers everything downstream: LLM API keys, per-role mcp_env, sandbox env, webhook secrets, contact identities, knowledge-search tokens.

Store wins; the environment is the fallback

Deliberately, and not negotiable: env-first would let a stale .env shadow a freshly rotated secret, which is exactly the failure this removes. A rotation would appear to succeed and then quietly not take effect, surfacing days later as an auth error far from its cause.

When a name exists in both with different values, boot logs secret_shadowed_env at WARNING with the names (never the values). That is a cue to delete the stale export, not a problem in itself.

A keyring is required

The secret store has no plaintext mode, and neither does anything else any more: every node’s Tier A carries a keyring (crewlet validate refuses a file without one, and a node refuses to start), and a company document stored unsealed is refused by every reader. A store whose whole purpose is holding secrets should not be able to hold them in the clear. The one place a command runs with no keyring is a provisioning run given no Tier A file at all, which opens no store and resolves from the environment alone — and says so.

Each value is sealed with AES-256-GCM and the record’s own name bound in as associated data, so a ciphertext moved to another name fails to decrypt rather than silently impersonating a different secret. That is also what makes a read fail closed: a snapshot that skipped a record it could not open would let the environment answer for it, which is exactly the stale-.env shadowing the store exists to prevent — so an unopenable record refuses the whole snapshot, loudly, and the previous one keeps serving.

Two things that can never live in the store

The store’s own address and the keyring itself. Tier A carries exactly what is needed to reach and decrypt the store, so it cannot source values from it. That boundary is explicit in the code — Tier A resolves through config.EnvOnly() — not an accident of boot ordering.

# crewlet.yaml (Tier A) — always env/file-sourced, never from the store
store:
  path: /var/lib/crewlet/index.db
secrets:
  active_key_id: "2026-01"
  keys:
    - id: "2026-01"
      material: "${CREWLET_SECRET_KEY_2026_01}"

The same bucket holds key material the engine keeps for itself: the identity directory’s blind-index key, which every stored address is matched under. It lives here because every node must derive under the same key and this is the one store every node reads — and it is not your secret:

Name	What it is
`iam/blind-index-key`	The key an address is blinded under in the identity directory

A person’s own values — their name, their address, a second factor’s seed — are not keyed here at all: they are sealed under the keyring itself, in the identity estate’s rows (see Identity and Access).

The namespace has one owner, the identity directory (iam/); an owner is reserved before it writes its first key, so no engine key is ever briefly an ordinary operator row. A name with a / in it is one no ${VAR} can spell, so none of them can be resolved into a provider, an mcp_env or a child process by any document. And every operator surface refuses them by name, whatever the caller holds: GET, PUT and DELETE /secrets/{name} — a reveal included — answer 403 reserved_name, and crewlet secrets get, set and unset refuse before they open anything. The listing does not name them; it counts them, per keyring key, as engine_keys, because a rotation has to move them too — and a rekey does, reporting them as a count (engine_keys_moved) beside the names of yours. The listing counts the identity estate’s sealed values per key the same way (identity_values), and the rekey moves those too (see Rotating the keyring). A dry run counts both still under an old key, so an operator is never told “nothing to move” while the address index’s key or every person’s name is still sealed under the key they are about to retire.

The blind-index key is never minted over one that was deleted — it comes back with the coordination store it lived in, from the backup that holds it (Backups § Restoring).

Using it

From the CLI

crewlet secrets keygen                           # a fresh key for the keyring
echo "$TOKEN" | crewlet secrets set GITLAB_TOKEN_SWE
crewlet secrets set SLACK_BOT_TOKEN_CEO          # reads stdin
crewlet secrets list                             # names + metadata, never values
crewlet secrets unset STALE_TOKEN
crewlet secrets get TOKEN -reveal                # break-glass; logged
crewlet secrets rekey                            # after a keyring rotation: every row, every person's values

The keyring is not optional, and it does three jobs

A deployment has one keyring, and everything that has to be authenticated across a fleet derives from it. It seals the company config and the secret store’s rows — and a sealed company document is an authenticated one, so a node refuses one its peers published unsealed. It signs the two per-run tokens a sandbox carries. And it signs every record on every state log — the tracker’s, the knowledge base’s, and any domain a later build registers.

That third job is why a node with no keyring now refuses to start. The embedded broker has no auth of its own; on a fleet it clusters its members over a port, and a record on a domain’s log is not a message but an instruction — the next node applies it and derives rows from it. Unsigned, such a record is whatever anyone who can reach that port writes. Each of the three derives a different key from the same material, each binding its own label in, so a record signature and a token signature over one key are never the same bytes.

A record’s frame names the key that signed it, and a verifier looks that id up. A key it does not hold means “not yet” — the record is retained and applied once the key arrives. A key it does hold whose signature fails means “not this fleet” — the applier stops, because continuing past it would be applying whatever it was written to precede.

Rotating the keyring, with nothing in flight lost

The keyring seals the company’s secrets and also signs the two per-run tokens a sandbox carries — the OTLP receiver’s and the MCP bridge’s — and every session cookie a signed-in browser holds. The tokens travel in a URL path, are minted on the node that starts a run and verified on whichever node the box can reach, and a detached coding run holds one for as long as the run lasts, which outlives a rollout. A cookie is the same shape of problem with a longer life and a person on the other end of it.

So a token names the key that signed it, the same way a sealed value’s envelope carries the id of the key that sealed it, and a verifier looks that id up. A key it does not hold is refused rather than retried against the active key, which is what makes dropping a key mean something.

That makes the rotation a runbook with no window in it:

Add the new key to secrets.keys, leaving active_key_id where it is.
Restart every node. Nothing has changed about what is minted; every node now holds both keys.
Flip active_key_id to the new key and restart every node again. New tokens name the new key; outstanding ones still name the old one, which is still on the ring.
Re-seal both halves under the new active key. The keyring seals two stores, and a key dropped while either still holds something under it leaves that thing unreadable on every node:
- The secret store and the identity estate — run crewlet secrets rekey, once for the fleet, through a running node, because it moves the identity estate’s values too: every person’s name, address and second factor is sealed under the keyring in rows derived from the identity log, which only a record can change, so the node publishes one per person still under an old key and reports how many (identity in the answer — every data node holds the identity estate from boot, company or none). An outstanding invitation’s address is counted and left — only a re-issue could carry it again — so the rekey says how many are still under an old key.
- The company document — POST /config/reload to a node that is up stores the active document again, sealed under that node’s active key, and activates it. On a stopped node, crewlet config rekey does the same offline, and the node publishes the result at its next start. Without this half the document stays sealed under the old key, and dropping that key makes the company configuration unreadable at the next apply.
Drop the old key once both halves of step 4 have run, the longest token lifetime has passed, every state log has trimmed past the records signed under it, the absolute session lifetime (api.auth.session.absolute) has elapsed since step 3, and crewlet secrets rekey -dry-run reports no outstanding invitation under it — each is redeemed or lapses within a week of being issued. From that moment a token, a record or a cookie signed under it is refused, which is the point. Dropping it while a log still holds such records leaves those records unappliable on a node replaying from the floor, which reports itself as retained records rather than as data loss — add the key back and they apply. The same holds for a person’s value a rekey did not move: it renders as sealed until the key is back on the ring.

Dropping it early is what signs people out. A session cookie names the key it was signed under, so every browser still holding one minted before step 3 is refused the moment that key leaves the ring, and its owner sees a sign-in screen. Sessions move onto the new key by themselves — a cookie is re-issued under the active key the first time it is used after step 3 — so waiting out the absolute lifetime is waiting for the sessions nobody has touched since the flip.

Nodes may restart in any order at every step, because a key is either on a node’s ring or it is not, and tokens name which one they need. There is no state in which two nodes disagree about a token they both hold the key for.

The value is read from stdin by default rather than from -value, because an argv value is visible in ps output and lands in shell history. Exactly one trailing newline is stripped, so echo "$TOKEN" | ... does the right thing — and no more than one, because a secret may legitimately end in whitespace and altering it silently is a failure nobody can see.

Reading a value back is break-glass on both sides. crewlet secrets get refuses without -reveal, and the route behind it (GET /secrets/{name}) serves only metadata unless the request carries an explicit ?reveal=true — a spelling a crawl or a link cannot reach by accident. Both log the access by name, against the operator the API guard authenticated; the answer is marked Cache-Control: no-store so it cannot sit in a shared proxy. The refusal points at list, because the common need is “is X set and when did it last change” — which the listing answers without putting a credential into a terminal, a scrollback buffer and a screen-share.

Which store the CLI writes

crewlet secrets reaches the fleet’s rows through a running node’s authenticated API, because on the default topology the coordination KV lives inside the engine’s own process and listens on no socket. There is nothing to guess about which route it takes: the engine holds an exclusive lock on its database for as long as it runs, so a locked store means “the engine is up, write through it” with a pid attached, and an unlocked one means it is not.

The engine on this node is	`crewlet secrets` writes	Reaches
running	its `/secrets` API	every node, immediately
stopped	this node’s own `secret_values` table	this node, until it starts

A write made while the node was stopped is not stranded: at its next start the engine migrates those rows onto the fleet and removes them locally, preserving the original author and the instant it was written, and recording the source migrated in place of the stopped node’s own word. That is the bootstrap path — the equivalent of crewlet config import against a stopped node — and the CLI says which of the two it used after every write.

-api URL writes through a named node instead, which is how the command works from a machine that is not the node at all. It authenticates with CREWLET_API_TOKEN, and with nothing else: api.auth.tokens is what a node accepts, not a wallet the CLI helps itself from, and a write whose author was whichever credential happened to be listed first landed under a name the operator had not chosen. Export the variable — one of the Tier A values, or a machine token minted with crewlet iam token. The credential’s own identity is recorded as the author of the write.

crewlet secrets keygen needs no config and no store: it is what an operator runs before either exists, and it prints the base64 form the keyring’s material field takes.

Every other command reads the Tier A config for its keyring, never the company document. The store holds only ciphertext; the key material lives in the bootstrap file — on disk or in the environment, never in the database it opens.

From the dashboard

Settings › Secrets (#/settings/secrets) lists what the fleet holds and can store, rotate and remove a row, over the same routes the CLI uses — the listing and every write on admin. It never shows a value: the one route that returns one is break-glass on both sides, and putting that behind a click in a page anyone who may list the credentials can open is not a trade worth making. crewlet secrets get -reveal stays the deliberate path, and a rotation asks for the new credential rather than editing the old one.

Before a removal the screen says which config fields point at the row, read from GET /config/references and listed by path. That is the failure this confirmation exists to prevent: the config keeps ${VAR} pointers, so removing a row a seat’s bot_token still names leaves that pointer resolving to "" at the next activation, and the webhook route or transport holding it starts refusing deliveries with nothing naming the row that went away. Removing a referenced row takes an explicit acknowledgement, and so does removing one when the check itself did not answer, “the configuration could not be read” is never rendered as “nothing points at this”, because the second is a reassurance the screen has not earned. A rotation is safe by construction, since the pointer keeps naming a row that still exists.

The API’s own bearer tokens are NOT in this store: they are Tier A (api.auth.tokens), the root of trust the store’s keyring comes from. Settings › API tokens (#/settings/tokens) lists the answering node’s labels, never a value — each beside the directory row holding its token:<id> login, that row’s stage and the seat it binds the token to, read from GET /iam/node-tokens on admin. A label nobody’s row holds is a token acting as itself, under its own login, which is what that listing exists to show.

A model’s keys are ${VAR} pointers into this store too. Settings › Models & keys (#/settings/models) lists each model’s keys by the variable they name and says, per key, whether it resolves on the node that answered (Not set when neither this store nor the environment holds it), whether a vendor is refusing it and until when, read from GET /credential-pool — never a value. A model that names no api_keys reads its vendor’s conventional variable (ANTHROPIC_API_KEY, OPENAI_API_KEY) from this store first; one whose named keys resolve to nothing runs on no key rather than borrowing it.

From a provisioner

Every provisioning CLI takes -secret-store in place of -env-file:

GITLAB_ADMIN_TOKEN="$GITLAB_ADMIN_TOKEN" crewlet gitlab provision company.yaml \
  -public-url https://engine.example.com \
  -secret-store

Minted PATs and the generated webhook signing secret go straight into the encrypted table under the same ${VAR} names the config already references. The three-step dance collapses to one command — no file to source, no shell to be in.

-secret-store follows the same routing as crewlet secrets: against a running node it writes through that node’s API and the minted credential is on every node at once, and it takes the same -api URL flag. Against a stopped one it writes the local table, which the engine migrates at its next start. The row’s source follows the route too: the node’s api, or provision in the local table until the move onto the fleet makes it migrated. Who ran the command is what the row’s author and credential say, either way.

crewlet slack provision -secret-store and crewlet mattermost provision -secret-store work identically. Slack’s own app-configuration token pair is the one credential that does not go here: it is the operator’s, not the company’s, and it lives in the app ledger beside the company file.

Where it pays off most is a credential the third-party app shows once. Every third-party app here returns a token’s value once and never again — so the recorded copy is the credential. A file someone has to remember to source is a worse home for that than an encrypted table the engine reads back itself.

Propagation

When	What picks up a new value
`crewlet run` boot (every node role)	Reads the whole store before resolving any Tier B `${VAR}`
A config revision activates (`PUT /config`, `crewlet config import`)	Every node re-reads the fleet’s store as it converges on the new activation epoch — engine and API halves alike
A provisioning pass seals a credential	The node that ran it refreshes its snapshot and re-activates the current revision, so every node rebuilds against the minted value — coalesced into one apply per 15 seconds (below)
Otherwise	The running process keeps its snapshot

A pass that mints a credential does the re-activation itself, and the row above is the whole of why. Sealing makes a value resolvable; it does not touch anything that already resolved the old one, and everything a company actually routes through — seat identities, parsers, transports, provider clients, MCP children — was built at the last apply. So the provisioning pass performs the same gesture an operator would: it re-activates the current revision unchanged. Before it did, a pass could create an agent’s Jira account, seal its token, report the integration ready, and leave the running engine routing every issue for that agent to a project lead, for ever. A node with no config surface (a worker-only one) seals and logs the outstanding rebuild instead, naming POST /config/reload.

The re-activations are coalesced: the first in a quiet period runs at once, and any that arrive within 15 seconds of it fold into a single apply at the end of that window rather than each costing a whole-company rebuild and a permanent revision. Connecting one integration seals several values within a second or two, so without it a single button press wrote several. It is also a bound, and one the mechanism needs rather than merely benefits from: an apply wakes the reconcile pass that caused it, so a vendor pass that can never converge — one whose own minted credential is refused — seals on every tick, and inline that ran as fast as an apply could complete. See Integration Reconcile.

A write reaches every node, and the value is a value rather than a file to copy. crewlet secrets set against a running node puts one sealed record on the coordination KV; every peer reads that same record at its next boot or activation. Nothing has to be run once per node, and nothing has to be copied to a node that scales up at 3am.

The node-local table is still there, and it is the bootstrap path. The engine takes an exclusive lock on its database for as long as it runs — the store is one file, one process, and the driver does not reliably refuse a second writer, so before the lock existed a crewlet secrets against a live node corrupted the database silently. That lock is now also the routing signal: a locked store sends the command to the node’s API, and an unlocked one means the engine is stopped and the local table is the only place a value can go until it starts.

The lock is released by the kernel when the engine exits, however it exits — a crash, a kill -9 and an OOM all free it, and there is no stale lock to clear by hand.

So crewlet secrets set against a running node takes effect fleet-wide at the next config activation or restart — the CLI says so after each write, naming POST /config/reload. Re-activating the current revision is a valid way to ask a running engine to pick up a rotated credential, and on a running fleet the reload is that gesture; the apply re-reads the store before it builds anything precisely so that gesture works, and the pointer’s own KV sequence is the epoch — the store assigns a new one on every write — precisely so a re-activation still moves what every node is watching (see Control Plane).

What “picks up” means. A rotated value is not useful until the things that captured it are rebuilt — an MCP child baked the resolved value into its spawn environment, an LLM provider holds it inside a constructed client, a transport holds it in a header. So re-activating an unchanged revision has to rebuild them, even though its payload is byte-identical.

It reaches most of them, and the reason is the shape of the control plane rather than a comparison. The pointer’s KV sequence is the epoch and the store assigns a new one on every write, so re-activating a revision mints a new epoch; a node’s reconciler skips on the epoch it has applied, never on the payload, so a re-activation always applies. ${VAR} references stay verbatim in the stored revision and are resolved where a provider is constructed — and the secret snapshot is re-read immediately before that. There is no payload-equality shortcut for a rotation to slip through.

The MCP children are reached too, per-role ones included. A per-role child belongs to its seat’s lease rather than to the epoch, but the node holding the seat reconciles its children on every apply: one whose resolved credential moved is restarted with the new value, and every other is left running. The chat transports are reached as well: the Slack transport is rebuilt on every apply, and the Mattermost transport whenever a value it is built from moved, a rotated bot token included. Control Plane § Rotation has the full table of what a re-activation does and does not replace.

One more surface, and it is out of the engine’s hands entirely rather than merely off the apply path: a running code sandbox received its credentials in the box’s environment at launch, and no engine-side refresh reaches a live box. There the effective bound is the run’s duration plus any clarification pause, not seconds. Tear the run down if a rotation is a revocation.

Which one do I want?

Two mechanisms share the Tier A keyring and are easy to confuse:

	Whole-config encryption	Secret store (this page)
What is encrypted	The entire `company_config` payload as one blob	One value per row
Keyed by	Revision	Environment-variable name
Where the secret lives	Inline in the config document	In the `_secrets` coordination bucket, referenced by `${VAR}`
Rotation	New revision (a full immutable copy)	`UPDATE` of one row
Written by	`PUT /config`, `crewlet config import`	`PUT /secrets/{name}`, `crewlet secrets set`, provisioners

They compose: encrypt the config document and keep credentials in the store. That is the recommended shape for any provisioned deployment, on one node or twenty — both travel the same replicated path, sealed with the same keyring.

Why credentials belong in the store rather than inlined as literals in the config, even though the config is itself encrypted:

Rotation would archive the old secret forever. Every revision is an immutable full copy and revisions are never scrubbed, so each rotation leaves the superseded credential readable in history.
One credential, several pointers. role.integrations.slack.bot_token and role.mcp_env.slack.SLACK_MCP_XOXB_TOKEN reference the same variable — one credential with two readers. Inlining literals duplicates it across pointers that must then update atomically or the identity split-brains. Keying by variable name keeps it one row.
Blast radius. Losing the keyring with inlined literals loses your credentials, not just your config.

How a node gets its secrets in the first place

The store is an overlay, not the source of truth, and knowing which is which answers most operational questions about it.

Resolution order on any node, for any ${VAR}:

Tier A (crewlet.yaml) resolves from the process environment alone. It has to: this file holds the keyring that opens the store, so a resolver reaching into the store would have Tier A reading from the thing it describes.
The engine connects to coordination and takes a snapshot of the whole secret store. (It also migrates any rows left in its own table first — see Which store the CLI writes.)
Tier B (the company document) resolves store first, environment second.

So a node with an empty store resolves everything from the environment and runs normally. That is not a degraded mode. A brand-new node always starts from the environment. Nothing has to be copied to it, and there is no first-boot step that reads a peer. (What a node cannot do is start with no keyring at all: every node needs one — see the three jobs above.)

Which one to use

	Where credentials live	Rotation	New node starts
One node	Store (env as the fallback)	`crewlet secrets set`, then re-activate	From the env, then the store overlays it
A fleet	Store (env as the fallback)	`crewlet secrets set` against any node, then re-activate	From the env, then the same store every peer reads

The two columns are the same, and that is the point. A credential is company-wide state, so it lives where the company config lives: one sealed copy on the coordination KV, written through any node’s authenticated API, read by all of them.

The environment stays the bootstrap path, and it is a perfectly good place to keep credentials if your platform already does — a Kubernetes Secret projected as env, systemd’s EnvironmentFile=, Compose’s env_file:. A node whose store is empty resolves everything from the environment and runs normally. What is no longer true is that a fleet has to work that way.

Provisioner-minted credentials work fleet-wide too. crewlet gitlab provision, crewlet slack provision and the rest MINT credentials, and -secret-store against a running node records them where every node reads them. Run it against a stopped node and the credential is that node’s alone until it starts and migrates — which is fine for a first provisioning run and wrong for a rotation on a live fleet. -env-file PATH and -print remain for feeding a secret manager instead.

What still has to be in the environment

Most ${VAR} resolution funnels through one function, so the store covers it. A handful of places read a variable by name instead, and each was decided deliberately:

Site	Source	Why
`providers.llm.*` / embeddings conventional-key fallback (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)	Store, then env	Otherwise `crewlet secrets set OPENAI_API_KEY` would work through a config reference but not through the fallback
Everything a provisioning command reads out of the company document (`integrations.*.url`, `.workspace`, `.token`, `.signing_secret`)	Store, then env	The same chain the engine resolves through. A command that saw only the environment read an empty string for every value already rotated into the store, and for the GitLab signing secret, empty is the signal to mint, so a re-run replaced a working webhook secret at the third-party app. Each command takes `-config` for this
Sandbox launch credential check	Store, then env	A seat whose token lives only in the store must not read as unresolved
Tier A bootstrap (`providers.database.dsn`, `secrets.keys[].material`)	Env/file only	Root of trust — this is what opens and decrypts the store
Operator provisioning credentials (`GITLAB_ADMIN_TOKEN`, `MATTERMOST_ADMIN_TOKEN`)	Env only	Human operator credentials, never persisted by Crewlet and never read back from the store: a GitLab admin PAT carries `api` scope over the whole group, and the store is read by every node holding the keyring. Reading one from it would imply it may be kept there
OTLP endpoint / protocol / headers, `CREWLET_SANDBOX_OTEL_RECEIVER_URL`	Env only	Deployment-environment settings that belong to the host, not the company; several are read before the store loads
`CREWLET_TOOL_SKILLS_SPACE`	Env only	Not secrets — flag defaults for the import/resync commands, read by nothing else
MCP stdio subprocess environment	Declared only, over an allowlist	A child is handed the host allowlist (`PATH`, locale, TLS trust, proxy), the engine user’s `HOME`, `TMPDIR` and XDG directories, and exactly what its `env:` and the seat’s `mcp_env` declare, already resolved — never the engine’s own environment, which is where Tier A’s keyring, token values and client secret resolve from. A server that reads an undeclared variable names it in `env:` with a `${VAR}` reference, which resolves through the store and then the environment like any other. Store values are not poured in either: injecting the whole store would hand every seat’s token to every subprocess. See What a stdio server’s environment is

Nothing writes a minted value back into the process environment. The sink is the only durability path, and a value minted this run is read back through the sink, which is why a run must name one before it touches the third-party app, and why -print reports itself as holding nothing rather than pretending otherwise.

A -print run that ROLLS BACK emits unset VAR for every value it printed, followed by the comment saying why. The stream is meant to be sourced — that is the whole reason it emits export lines — and a comment is a no-op to a shell, so an operator who piped the output into source and then hit a rollback would keep a revoked token exported in their session. Sourcing the stream to its end now leaves the environment as it started.

No sink feeds the engine directly. -env-file writes a file to source before the engine starts; the engine itself reads ${VAR} from this store and then from the process environment, and from nowhere else. -secret-store is the one that needs no file and no restart — see crewlet config activate.

Operational notes

A missing value still resolves to "". The store removes the most common cause, not the failure mode itself. A tool server and a coding run are held to theirs: a tool server, or a run_sandbox coding run, whose reference resolves to nothing is not started — and says which variable, by name — unless the key holding it is listed under optional_keys (Tools & MCP, Code sandbox); an agent-mode executor starts without it and logs sandbox_env_unresolved (names only); and the sandbox credential check refuses to launch a coding agent on an empty credential. A cli-agent entry’s cli.env is not held: it is resolved when the company’s providers are built, where a refusal would take the whole company down rather than one entry, so an unset reference there reaches the CLI empty, with a cli_agent_env_unresolved warning naming it.
Backups. The bucket holds only ciphertext; the keyring is the sole root of trust and lives in Tier A. Back them up separately — a coordination backup alone is unrecoverable, which is the point. With the keyring, though, a backup opens everything it held when it was taken — the bucket’s values, and the identity estate’s sealed names and addresses in its rows and its log, those of every person removed since included. Removing somebody cannot reach a backup that already exists, so an erasure is finished only once every backup taken before the identity log’s trim passed their records is gone (what a removal reaches).
Key rotation. Follow the rotation runbook: add the new key on every node and restart, flip active_key_id and restart again, re-seal both halves (crewlet secrets rekey, and the company document with POST /config/reload or, on a stopped node, crewlet config rekey), and only then drop the old key. Flipping active_key_id in the same restart that adds the key is safe on a single node and not on a fleet: a node that flips first signs records, cookies and requests under a key its peers do not hold yet. Each record’s envelope names the key that sealed it, so mixed-key states are readable throughout. The store is shared, so crewlet secrets rekey is run once for the fleet, not once per node — and it refuses if the node it reaches seals under a different active_key_id than the config it was given, because a silent success there would report a rotation that did not happen. It aborts rather than half-completing if any record cannot be opened with the keyring in hand. Each record is re-sealed only at the version the pass read, so a rotation or a deletion that lands while it runs is never undone — a credential an operator deleted in particular stays deleted rather than being written back as the value the pass read before. The same command moves every person’s sealed values in the identity estate, one record per person, and a run that cannot confirm one says so and is run again.
The _secrets bucket is created on demand by the first node to open coordination, and it is durable: on the embedded topology it lives under stream.store_dir like every other bucket, so a restart does not lose it. Back that directory up alongside the keyring.
Rows left in a node’s own secret_values table — written by crewlet secrets set while its engine was stopped — are migrated onto the fleet at that node’s next start and removed locally. The pass copies before it deletes, and the later write wins: a name the fleet already holds is replaced when the local row was written after the fleet’s value (an offline rotation, logged in secrets_migrated_onto_the_fleet’s replaced), and kept when the fleet’s value is the later one — somebody rotated it through a running node after this one stopped — which logs secrets_offline_writes_superseded naming it. A migrated record keeps the instant its operator wrote it, never the instant the node booted, so two stopped nodes holding writes to one name are ordered by when the writes were made whichever node starts first. The two instants are the writing nodes’ own clocks, so writes closer together than those clocks agree are ordered by the skew. A failure leaves the local rows in place and logs secret_migration_incomplete; the node keeps serving from them and retries at the next start.