Replication

Crewlet’s own tracker, knowledge base, identity directory, artifacts and artifact datastores are replicated state machines: seven domains on one framework, the embeddings and each node’s usage history riding it as compacted domains. One ordered stream per domain is the write-ahead log; every data node applies it into its own SQL database; the checkpoint commits in the same transaction as the rows it covers. That last clause is the whole design: a node’s position and its rows can never disagree, because they are written together or not at all.

This page is for operators. It covers what a write means, what an acknowledgement promises, how far behind a node can be, and what the design does not promise.

Three numbers that are not the same number

Confusing these is the single most common misreading of this system, so they are stated first.

Number	Where it lives	What it is for
the arbitration anchor	`statelog_anchor`	The position of the last record this node consumed on a subject, whatever that record then did. It is what the broker arbitrates a new write against.
`version`	a column on every object row	The object’s accepted state. It is what a caller’s `if_match` compares against.
the barrier comparison	computed	`MAX(version, scoped_through)` — what a read barrier compares, because a record can change an object without being about it.

The anchor and the version are equal only while every accepted record produces rows, and an apply gate is by definition the rule that makes them differ. An evicted node’s append is accepted by the broker at sequence 102 and dropped by every applier: the object’s version stays at 100 on every node while the broker’s last message on that subject is 102. A writer that formed its expectation from the version would be refused, re-read 100, and burn its whole round budget — for at least the trim’s age floor, and unbounded above while any retention term blocks.

scoped_through is the third. A record bumps version only on its own subject; every other row it writes carries scoped_through instead. A barrier that compared only version would let a read past a record that had already changed the row it was about to return.

A write has three outcomes

Not two. applied, pending, unknown — and a caller does something different with each.

applied — the record is durable at its position and this node has applied it. Read it back and you will see it.
pending — the record is durable at its position and this node has not applied it yet. The write succeeded. Do not retry it; retrying publishes a second record. What is not yet true is that you can read it back here.
unknown — nothing can be established about the record. It may be on the log and it may not. This is the only outcome a retry is correct for, and the retry carries the same operation id so the ledger collapses a duplicate. It is also what a write answers when the broker says the record may yet land: on a clustered stream, a second append under an operation id whose first record is proposed and not yet applied (duplicate message id is in process), or a leader whose store closed under a record raft had already committed (store is closed). Neither is a refusal — the record usually applies a moment later — so the retry under the same id is answered by the broker’s duplicate acknowledgement once it lands.

A clustered leader has a third answer to a conditional append that is not a refusal either: wrong last sequence with no sequence in it, which means another write to the same subject is still in flight at the leader — often this node’s own previous write, already acknowledged, whose bookkeeping the leader clears only after the acknowledgement has gone out. It compared nothing and stored nothing of the append, so the append is simply asked again — a millisecond later, doubling to 64 ms, within the append’s own deadline — until the leader answers with something it decided: the record lands, or wrong last sequence: N says who wrote first. Read as that second answer, it sent a write whose expectation was perfectly current round the lost-race path, re-decided against the same anchor and refused again, until the round budget reported a conflict nobody was causing. One still in flight at the append’s deadline is treated as an append nobody answered: the log is asked whether this operation’s record is on it, and finding none, the write decides again. The coordination store’s compare-and-sets take the same rule — see Coordination § A compare-and-set waits for the leader to decide.

Two pendings, and only one says something about the node

There are two ways to get pending, and they carry opposite information about this node:

It waited and the record did not arrive. This node is behind. It is the common one, and it is what the rest of this section is about.
Nothing waited. A caller asked for the answer the broker’s acknowledgement already establishes, because it has no use for the row.

The answer says which: a pending that waited reports waited: true. Reported as one they would be indistinguishable, and a deliberate skip would read to an operator as a node falling behind under load — a health signal they would go and act on.

Exactly one write in this engine takes the second path: a sign-in’s session record. The cookie minted from it carries the record’s own position, every node that serves the API — a data node — validates a bearer against its own applier, and a node below that position serves reads on the signature and the epoch alone — so the row this node would be waiting for is a row nothing in the answer reads. What the wait costs there is a quarter to half a second of parked browser fetch against a 50 ms credential verify, on the one request a person judges the whole product by.

Nothing else may take it. A directory write answers 200 only once this node’s rows carry it, because the caller’s next read goes to this node and would not see what it just wrote.

A lost acknowledgement is resolved from the log before it is ever called unknown: the write reads the newest record on its subject, and a record that carries its own operation id is its answer — applied where this node has applied it and no gate dropped it, even where this node’s ledger has lost the operation’s row, and refused under the gate that dropped it otherwise — the domain’s own, or one of the log’s: a rule a reanchor placed on the checkpoint (abandoned, overtaken). A gate that drops a record by who wrote it (an eviction) is asked about the node the record names, which is not always the one asking: an append the broker collapses onto another node’s copy of the same operation is answered by that copy.

A gesture made of several records in order — a cross-project move, a merge of duplicates, a dependency with its mirror, a subtree’s removal or restore — treats a step answered unknown as the end of the walk, not as a step that landed. Nothing after it is written: no descendant follows a root whose own move or removal is unknown, a mid-move or mid-merge mark is not taken down, and a dependency’s mirror is not written over an authored edge nobody can vouch for (a mirror whose own outcome is unknown is reported one-sided, the state the tracker duty repairs). The gesture fails naming the operation id, and running it again under that id answers each step that landed from the ledger and finishes the rest. Every step is named by the task it writes rather than by its place in the walk, because a re-run reads its list afresh — a restore’s is what is still in the trash — and a step named by position would carry another task’s operation id.

What “under that id” means depends on who is asking. A seat derives its ids from its turn, the call’s arguments and how many different calls to the same tool it made first, so it runs a gesture again by repeating the call with exactly the same arguments — before calling that tool with anything else. The operator’s MCP has no turn, so every write’s answer carries its op_id and a call that brings it back is that operation again (/operator/mcp) — and only that one: the id names the call’s tool and a digest of its arguments, and brought back with any other it is refused before anything is written, rather than half-answered from the first call. The dashboard’s act route and the /work and /pages routes take the key in an Idempotency-Key header instead — the act route refuses a write without one — and answer the op_id they wrote under. The purge and node-gate routes take theirs as ?op_id=. A caller that sends neither starts a new operation, which finishes nothing: a create repeated that way files a second item. A create whose item landed and whose dependencies did not says so — the item is named, and its dependencies are finished on it with update_work_item rather than by filing it again.

What a retry is judged by: the instant its operation was minted

Every data node keeps an operation ledger — which operations its own applier has applied, and where — and it is the ledger, not the broker, that makes a retry safe. The broker also collapses a repeated operation id, but only inside its two-minute duplicate window, and a turn re-run after a crash or a caller repeating an unknown routinely comes later than that.

A retry of an operation the ledger records is answered, not decided again: it comes back applied, at the position the first copy landed, however long afterwards it arrives. Deciding it again would mean deciding against rows that already hold it — a retried purge would find its task gone and refuse, an update conditioned on a version would find that version moved by its own first copy and refuse as stale. A retried create comes back as the task the first copy filed, under the key it took — the new item’s id is derived from the operation, so a turn re-run after a crash that makes the same call files one work item, not two. Only a create that took its key number and never filed its task is finished on a fresh number, which leaves a gap in the project’s numbering (ENG-8 is skipped) rather than two items sharing a key.

That holds for a re-run’s calls only where they are the same calls. A turn’s operation ids are derived from its unit of work, the verb, the object, the call’s arguments and how many different calls to the same tool the run made before it (see the turn engine), and the model behind a re-run is sampled again: a title or a comment it words differently is a different operation, so that re-run files a second item or posts a second comment. The engine cannot tell a reworded call from a new one, and a crashed run leaves no record of its calls to compare against.

The same holds for a write whose acknowledgement was lost and which the ledger then answers: unless the broker acknowledged this very attempt at the position the ledger names, the record there may be an earlier copy’s — decided on another node, against other rows — so the write is answered as that copy, and anything the attempt computed in its own decision is not used. A key is then read from the item’s row, or minted again on a fresh number. A retried create on a node that was behind used to be answered as its own stale decision, and filed its item under the number another item already held.

An operation id names one write, and a ledger row answers only for a write to the object it landed on. The same id sent with a write to a different object is refused op_reused, naming where it landed — whether the ledger finds it before anything is decided or the broker’s duplicate window collapses the second append onto the first record — rather than answered applied at a position on an object the second write never touched. A different write takes a fresh id.

The ledger travels inside a snapshot. A node that adopted one holds a row for every operation its donor applied, so a retry of one is answered from it exactly as the donor would have answered it, and an operation neither holds never applied — so a turn woken by work that was queued before the join files its writes like any other, however old the work is.

What the ledger can do is lose rows, and it keeps a watermark saying how far back it may have. Each node deletes its rows thirty days after applying them, and records how far back each pass deleted. An operation minted before the watermark whose row is gone is answered unknown — before anything is published — rather than decided a second time, so a retry a month on is never applied twice. It is answered unknown rather than refused, too: a retry’s rows already hold what its first copy did, which is exactly what makes a re-run refuse (a move finds its task already in the target project, a create finds its object already there), so on such an operation “no” would be the ledger’s silence read as an answer. Only a refusal about the node itself — it holds a record it cannot read, or the object is deleted for good — stands either way. A create can say more than unknown, because its item’s id is derived from the operation: one the ledger cannot vouch for is answered from that item’s own row — the item the first copy filed, under the key it took, where this node holds it, and unknown with no number minted where it does not, since the first copy may be on the log beyond what this node has applied. A gesture that writes several records in order and meets such a step stops there and says so: re-running it under the same id on this node stops at the same step every time, because the row that step needs is the one the loss took, and it is a node whose ledger lost nothing that far back that can finish it. The thirty days are sized for the slowest real retrier, a seat that only runs on a schedule carrying an operation id across a long weekend; a write that is retried later than that is told unknown, and the operation’s own record, if it landed, is on the log. The watermark travels with the ledger, so a node that adopts inherits its donor’s.

One ledger forgets one kind of operation sooner. The identity estate keeps a session’s operations for an hour rather than the month — a sign-in is answered inside its own request and never asked again, a sign-out decides again rather than retrying, and those rows are the busiest thing the domain writes — and that sweep records a watermark for the session kind alone, beside the ledger’s own. What a retry is weighed against is the later of the two for its own kind: a session operation minted more than an hour ago is answered unknown, while every other identity operation — an invitation’s redemption, a create retried under its key, a token’s mint — is still vouched for over the whole thirty days.

What decides it is the instant the operation was minted, and that instant travels inside the operation id itself: every id the engine mints is a time-ordered one whose leading bits are its mint time, followed by a name saying what the operation is (01a0…-7…-….update-<task>). A retry reuses the id, so it reuses the instant — including a turn re-run on another node, whose writes derive their ids from the unit of work and the instant it began rather than from the run. The exception is a unit of work older than the ledger — a trigger dispatched, or a turn resumed from a coding run, more than twenty-nine days after the work began: an id minted at that start would be one no node’s ledger can vouch for any more, so that attempt mints at its own instant and records it in the coordination store, and a later attempt at the same work inherits it while it is recent enough to (see Turn Engine). An operation id the engine did not mint carries no instant and is read as older than everything the ledger ever lost: on a node whose ledger ever lost a row, a write under it answers unknown unless that node’s ledger holds its row — which is why the routes that accept an operation id from outside, a purge’s and a node gate’s retry, refuse one the engine did not mint.

The instant is compared with the watermark, which another node’s clock may have set — the sweeping node’s, which a joining node inherits with its donor’s ledger — so the fleet’s clocks are assumed to agree to within the margin the sweep leaves, its thirty days. It is the same kind of assumption the trim’s age term rests on. Keep them synchronised.

pending is the outcome an ordinary busy fleet produces most often under load: the applier is 16 seconds into a bulk apply and a small write’s five-second wait for its own record expires. Five seconds is the applier’s own stall grace divided by twelve — many times an ordinary batch commit and its linger, and short enough that a caller holding a request open learns “durable, unresolved here” rather than waiting out a node that has stopped applying. The record is safe. A caller that treated pending as a failure would double every write it made during a burst.

`applied` is not permanent

Two gates can drop a record after a node has applied it, and both are deliberate:

The eviction gate. A record written by a node the fleet evicted before the record’s own position is dropped everywhere. A node that applied it before learning it was evicted will drop it on replay. Every record names the node that published it and the generation it was decided in; the write path is handed both and refuses to append a record that does not carry them, because a record naming nobody is one this gate can never drop.
The deletion gate. A record about a task a purge destroyed — a turn’s record of what it spent on the task included — or about a purged page applies nowhere, for ever. This is what stops a redelivery months later resurrecting rows an operator deliberately removed.

Neither is a bug being worked around: they are what make an eviction and a purge mean something on a system where the log outlives the decision.

Every node that can write is one the trim waits for

No check on a write decides whether its node may make it, because every node that can is already one the trim waits for. Only a data node runs the logs and writes to them — a node without the data role holds no estate and reaches it through a data node (how a request reaches the estate) — and every live data node is in every log’s counted set from the moment it starts, at position zero until its first heartbeat reports one. So the trim never removes a record that a node which may still write has not applied, and the one way out of that set while a node is still running — an eviction — is the eviction gate’s.

Who a refusal names

A write refused evicted that names a position is on the log, and it holds its operation id for the log’s duplicate window (two minutes for every shipped log): the broker collapses the same id, sent again inside it by any node, onto that record, and the answer is evicted again — the resolution judges the record by the node that wrote it, never by the node asking. So another data node takes the write under a fresh operation id, or under the same one once the window has passed; neither can apply twice, because the record in the way applies nowhere. An evicted refusal with no position was made before anything was appended, and another node takes the write under the same id at once.

When the record in the way is another node’s — the asking node’s append was collapsed onto a copy that node wrote — the refusal names that node as the copy’s writer, because the reason is then its standing and not the asking node’s. The asking node passed its own checks before it appended, so it is the one that takes the write, once the window has passed; a node gesture refused this way says so in its hint and offers the same operation id again rather than another node. That holds for every gate that drops a record for what its writer was or did — evicted, abandoned and overtaken — and for none other: a copy dropped because its task or page was purged is refused deleted and names no writer, because the purge refuses the asking node’s own write exactly as it did the copy, now and on every retry.

A record whose scope meets a deferred scope is deferred too

A record this build cannot decode is retained, not dropped: the bytes are the only copy, and a rolling upgrade puts records on the wire the older half has never heard of. What follows is the rule that keeps that safe:

If a record’s declared scope meets a deferred record’s scope, it is deferred too.

So an object’s rows on any node are always a prefix of that object’s own record sequence. There is no state in which record 5 has applied and record 3 has not. That is what lets a reader say “this node is behind” rather than “this node has a hole in the middle of this object’s history”.

What an acknowledgement promises

stream.sync decides it, and the default is the strong value.

Setting	An acknowledged write has reached
`always` (default)	the disk, on every replica that acknowledged.
a duration, e.g. `30s`	the replicas’ memory, and their disks within that window.

The five failure classes, and which the design survives at each topology:

Failure	Single node	Clustered, one host	Clustered across hosts	External NATS
F1 the process is killed	survived	survived	survived	survived
F2 the host loses power	survived only with `sync: always`	survived for any one host	survived for any one host	whatever that cluster promises
F3 the disk is lost	not survived	survived	survived	whatever that cluster promises
F4 the filesystem corrupts	not survived	survived for any one host’s disk	survived for any one host’s disk	whatever that cluster promises
F5 the site is lost	not survived	not survived	only if the hosts are in different sites, which the engine does not know and does not claim	ask your NATS operator

On stream.type: nats the engine refuses stream.sync rather than warning: it is a server option on a server the engine does not run, and a field that named a mechanism it cannot reach would let an operator hold a durability belief nothing delivers. Ask the NATS operator for sync_interval: always instead; crewlet validate says it cannot check it.

A declined stream.sync is refused on a leaf (stream.leaf.urls) for the same reason: a leaf’s broker runs no JetStream and has no file store at all — the members it joins decide what an acknowledged write has reached, in their own stream.sync. A satellite’s durability is the members’ row of the table above, never its own.

Above the floor and below it — two different regimes

This is the sentence the backup schedule hangs on.

Above the trim floor, the log holds every record on R replicas and every node holds the applied rows. Losing a node loses nothing.

Below the trim floor the log may already hold nothing — the trim has licensed deleting it — and each node’s own database file is the only copy of that history — N of them, independent, none replicated. Losing history below the floor takes all N disks, and it is covered only by the backup gate.

That is why the trim refuses to advance past a floor no backup has reached. The backup schedule is a correctness input, not hygiene. See Retention.

Adopting a peer’s copy

A node below the floor has nothing left to replay and no amount of waiting produces it. What it does instead is ask the fleet for a snapshot — one member’s replicated database file — and install it wholesale before anything reads from it. That happens at boot, and again without one: a node that falls below the floor while running pauses its appliers, joins the same way, and starts them again over the installed file. The transfer is verified against a checksum and against the positions the file itself keeps, because a checkpoint commits in the same transaction as the rows and a manifest claiming a position the file does not hold is describing a different artefact.

Every data node runs every state-log domain whatever its other roles, so a donor’s artefact covers everything the recipient applies and is installed whole:

An artefact that names no position for a domain the recipient runs — a donor on an older build, mid-upgrade — is refused outright, from the manifest, before anything is transferred; see Running a fleet.

A node resumes from its rows, never from its reader

Each node reads each log through a durable consumer of its own on the broker, named after the node. That consumer’s position is a second, weaker number than the checkpoint: it moves after the commit, and the broker keeps it when the node’s database does not. The checkpoint is what the node resumes from — but resuming can only drop what arrives below it. Nothing on the node’s side can make the broker hand over again a record it has already delivered and been told was applied.

So at every boot the node compares the two, and a consumer that disagrees with the rows is deleted and created again at the checkpoint. Each rebuild is logged as jetstream_domain_consumer_rebuilt, naming the drift:

`drift`	What it means	What it would have cost
`acknowledged_past_checkpoint` (a `WARN`)	The broker was told records past the checkpoint were applied, so this node’s replicated database is older than its reader. The database was deleted, or restored from a backup.	The node never hydrates, and every write on the missing objects refuses `behind`. On the vector log, where nothing checks for gaps, the skipped records’ rows are just missing.
`delivered_past_checkpoint`	Records past the checkpoint were handed to a process that has since stopped.	The node stays behind until the 30-second ack window returns them.
`in_flight_for_a_gone_reader`	Deliveries at or below the checkpoint are held for a process that has since stopped.	Each one holds a slot of the in-flight ceiling for 30 seconds. When they fill it, nothing new arrives.
`behind_checkpoint`	The rows moved without the reader, which is what adopting a snapshot at boot does.	Every record in between is delivered only to be dropped.

A clean restart logs none of these: a node that applied and acknowledged everything it was handed keeps its consumer as it is.

A node below the trim floor is not ahead of its rows either, although its reader reports records past the checkpoint as read: the broker moves a reader over the records the trim removed, to the first one the log still holds. The node reads the log’s first sequence to tell the two apart. If it cannot read it, it judges the raw positions and rebuilds, and the acknowledged_past_checkpoint warning then says the log’s first sequence was unreadable, so it may be either case.

A rebuild is a delete and a create on the broker, and on a fleet whose broker is not answering, either one can fail. What happens then depends on the drift:

behind_checkpoint and in_flight_for_a_gone_reader: the node asks the broker which reader it now holds, keeps that one, and logs jetstream_domain_consumer_rebuild_failed as a WARN. Neither drift has handed over a record past the checkpoint, so keeping the reader costs what the table says and loses nothing. The next boot rebuilds it.
acknowledged_past_checkpoint and delivered_past_checkpoint: the boot fails and names the domain. A node left on that reader could wait for ever.
Any drift where the broker holds no reader after the failure, or cannot say which it holds: the boot fails too. A reader the node guessed at could be one that no longer exists, and every read through it would fail while the node reported itself up.

A rebuild touches only this node. No peer reads through its consumer, and no retention term reads it either: the trim works from the positions each node publishes from its own rows. What the rebuild makes safe is losing the replicated database while the broker keeps its estate. A node started on an empty crewlet-replicated.db replays each log from the broker, or adopts a peer’s snapshot where the trim has already removed the start of one. It no longer waits on records its old reader already acknowledged.

Replication lag is two positions

Not one number. Every data node publishes both:

seq — the record it has consumed: its checkpoint, the contiguous prefix it has moved over.
applied_through — the prefix it has actually applied, which is lower whenever a record was retained rather than applied.

Beside them rides stream_created_at, the creation instant of the stream the node’s rows are keyed to. A generation cannot say which stream a sequence is on — a stream deleted and remade keeps the generation and counts from 1 again — so this is what lets a reanchor tell a peer further along the lost stream from one that came up on the new one. And checkpoint_stored_at names the record the checkpoint stands on — the broker’s instant for it — which a sequence alone cannot: after a broker restored from an older copy is written past a node’s rows, the log holds another record at the same sequence. It is what lets a reanchor tell a peer whose history is the log’s from one holding history the log lost.

Folding them into one would make a node that is applying nothing while its position advances look identical to one that is fully caught up. crewlet retention status prints seq and applied_through per node and per domain, beside the generation the position is in (GEN) — a position from a generation the log has left prints left gen N in place of a lag, since its sequence compares with nothing the log holds — and marks a node that reported log_diverged. The two instants are on the retention answer’s JSON (stream_created_at, checkpoint_stored_at) rather than in the table.

Lag does not move a node’s seats, at any size. A node that is behind keeps every seat it holds and claims no new ones until it is level — see a copy that is behind, and a copy that is wrong for the six states that take a copy out of service, none of which is a distance — and none of which moves a seat either: the node’s seats read the estate from the other data nodes. What lag does bound is how fresh an answer a read can ask for (Read Consistency).

The bulk-apply degradation, priced

One applier per domain, one goroutine, one totally ordered log. The measured drain is at least 2 000 rows/s and the steady-state load of the reference company is 0.244 rows/s — 0.012 %. What binds is not the average; it is the burst.

That floor is deliberately conservative, and it is the number every figure below is derived from. An applier writes a record’s child rows — tags, watchers, relations, dependency mirrors, the inverted index’s postings — as multi-row inserts chunked to the engine’s probed bind-parameter limit, and to at most 1 000 rows a statement, rather than one statement per row. The pinned driver accepts more parameters than the probe’s own 32 766 ceiling, so the 1 000-row cap is what binds: a seven-column row and a three-column row both batch 1 000 to a statement, and an 8 000-row apply is 8 statements rather than 8 000. Measured unloaded that shape drains about four times faster than one statement per row; measured on a loaded CI runner under the race detector, closer to 1.5 times. The floor above is the loaded, contended figure, so a fleet sized against it has margin rather than a number it has to hope for.

One maximal bulk update is about 31 800 rows, which is about 16 seconds of applier occupancy on every peer. For those 16 seconds two things degrade fleet-wide without tripping any alarm:

an unrelated single-object write’s five-second wait expires and returns pending with its position and its lag — correct, and designed for;
a barriered read answers behind with a computed retry_after of about 16.

Both are the system working. crewlet retention status publishes the longest apply transaction and the longest batch actually observed, so this is a measured property of your fleet rather than a surprise.

The occupancy is per domain, not per node. Every domain’s applier writes the same replicated database, and a node hands that database’s write lock to its writers in the order they asked for it. A bulk update commits one transaction at a time — at most 4 000 rows, about two seconds at the measured drain — and a waiting writer takes the lock as soon as the one in front of it commits. So a bulk update in the work tracker delays the knowledge base’s apply by the transactions already queued ahead of it, at most one per other writer on that node, never by the whole 16 seconds.

A writer that does not reach the front within store.busy_timeout_seconds fails retryably and rejoins the line, logged as store_tx_retry naming the knob. A writer that still has not reached the front after a second wait gives up with the store’s busy mark, and a tool or a person’s surface answers that unavailable with a Retry-After, never as a fault: the writers ahead of it commit, and the same write made again lands.

The default does not scale with the number of domains, on purpose. Five seconds is half the dashboard’s 10 s query timeout, and that is the whole derivation: a busy wait longer than the timeout above it turns lock contention into a request that fails with nothing to show. What a waiter can legitimately queue behind, on the other hand, is roughly one bounded transaction per other applier on the node — about two seconds each — so with every state-log domain applying and a bulk update in flight, the legitimate wait is already past five seconds and grows with each domain the engine gains. (The usage domain’s transactions are a node-day’s handful of rows, and weigh nothing here outside a replay.)

That is why store_tx_retry on a node doing bulk work is a signal to raise the knob, not a fault: the writer rejoins the line and the work lands, one round later. Raise it toward — never past — the query timeout above it, and raise it before you widen anything else: a longer per-waiter bound lets one stuck holder block the line for longer, and this way the line still drains in order.

No apply transaction is ever aborted by a commit elsewhere in the database, and none is ever re-run because of one. Every write transaction takes the lock when it begins rather than at its first write, so there is no window between an applier’s read and its write for another commit to land in. The crewlet.statelog.apply.tx.aborts counter in Metrics reads zero on a healthy node for that reason, and a non-zero count is the retry budget being spent on something else.

What a rolling upgrade blocks

A record at a version this build cannot decode is retained, and everything whose scope meets it is retained too. That is bounded rather than open-ended: the deferral grace is 30 minutes, past which deferred_old fires — naming the log, the position of the oldest record held and the record version it was written at against the one this build reads. Where the node’s copy is judged by that log (the tracker’s and the knowledge base’s are; the identity estate’s, the vectors’ and the usage log’s are not) the node takes its copy out of service at the same moment — its seats stay, and read the estate from the other data nodes — because both are measured from one instant — the position heartbeat’s first sighting of the record — and the alarm says which of the two it is. It is one alarm however many logs hold such a record, and it lists each of them in its domains with that log’s own detail, so an upgrade that moved two logs’ record versions names both, each saying what its own grace did: a detail naming only the node’s oldest record would have said the identity estate’s took nothing out of service while the tracker’s had. Its log lines are written per log, so the second log’s record passing the grace, and the first log’s clearing while the second’s stands, are each said.

So a rolling upgrade should finish inside that window. An upgrade that stalls half-done leaves the old nodes holding records they cannot apply and refusing reads about the objects those records touched, and writes to them — with deferred naming exactly what to do, which is finish the upgrade. What a record touched includes where it sits: a record about a knowledge space’s settings covers every page in that space, a page purge covers the whole space it re-files the page’s children in, and a task purge covers every task it rewrites — its dependents, the blockers it waits on, the tasks related to it or referencing it, and its subtree — each by name, or by its project once there are too many to name. It covers every key the task held, current and former, too, because the purge hands each one on to another task holding it, and which task that is depends on the order the key’s claims were applied in. So every record that can change who holds a key names that key: a task’s create, a move that re-keys it, and its purge. A node holding back a newer peer’s create of a task under a key therefore holds back a later create of the same key and the purge of its claimant with it, and hands the key to the same task as every other node.

A record is written at the lowest version that can apply it whole, never at the newest the build knows, so what an older node holds back is exactly the objects whose shape changed — the next section lists them. Every barrier and every generation record stays at version 1 for good: an older node retaining those would hold a deferral for every linearizable read, or never make the transition a reanchor announced. So does every record that installs a gate, and for a harder reason: a node that deferred an eviction would go on applying everything the evicted node appends, so a gate record no build can decode stops that build’s applier instead, and its shape may only ever grow by addition.

The upgraded node applies what it retained at its next boot, before its applier consumes anything new: every retained record it can now read, in log order, each released in the transaction that applied it. A record whose scope met a retained one is applied after it, which is what makes the objects’ rows a prefix of their history again rather than a hole. A record still above the new build’s version stays retained, with everything it covers, until a build that reads it boots.

Which records an upgrade holds back

Only the ones that need the newer build. A record is stamped with the lowest version that can read it: 1 when it carries nothing a later build added, and the version of the newest field it carries otherwise. So during a rolling upgrade an old node applies everything the new nodes write except the records that use something new, and it retains those whole rather than apply them with the new part dropped — which is what would leave its copy of that object different from its peers’ for good. An upgrade that adds no record field holds nothing back at all.

Every strict domain states that as a table, and the table is the rule rather than a description of it. Each field a record has gained since the base format is one row of the domain’s RecordFields: the version it arrived at, the operation it rides (or every one) and its path from the record’s root. A writer stamps the highest version among the rows its record carries — present with any value but null — and a build reads exactly up to its table’s highest row, so it can never accept a version no field introduced and then drop the field it was minted for. A version, once a build has read it, means one thing for ever: a version whose field is gone is retired and stays empty rather than being given to the next field. The conformance suite every domain passes holds each table to the records its encoder actually writes.

Every domain’s table is empty today: version 1 is everything this build writes, so every record is stamped 1. The first release that adds a field to a record gives it version 2, and from then on an older node holds back exactly the records carrying it — except a gate record, which an older node stops on rather than retains, so a row that adds a field to one is marked as a gate’s and its writer waits for the fleet’s census of the versions its nodes read before publishing it.

Values the engine computes are recomputed once

Some columns are not copied out of any record but computed from the rows a node already holds: in the tracker, how often a task was reopened (reopens), how much of a project’s open work has been started (task_counts.active), the hand-off count on each history row, and when a task last changed (updated_at); in the identity estate, the seat an invitation holds (iam_invites.seat_id, read out of the invitation’s own document so a seat hold is one indexed lookup). The apply maintains each one as it goes. When the rules that compute them differ from the ones a node’s rows were derived under — a build that adds such a column or changes how one is computed, a node that just adopted a snapshot from a peer on a different build, or a checkpoint a reanchor created — the node’s next boot recomputes them from the rows it holds, in the same transaction that records which rules the rows now follow, before it applies anything new. It happens once per change, on every data node; the statelog_rederived log line names the domain, the rule versions it moved between and how many rows it wrote. Both domains are at rule set 1, the one this build applies as it goes.

Two compacted domains: the embeddings and each node’s day

Two domains do not keep a history at all. Their streams keep one message per subject — a keyed table rather than a log — so a node that joins replays the current value of each object and nothing before it, a gap is a coverage number rather than a fault, and neither gates a node’s seats or claims that two nodes hold identical rows.

Domain	Stream	One subject per	Written by	Kept for
vectors	`CREWLET_TRACKER_VECTORS`	embedded source	the fleet’s one embedding duty	90 days per message; the rows until the source is forgotten
usage	`CREWLET_USAGE_LOG`	(node, company day, seat, schedule or person)	every data node, for its own days only	181 days

usage is what makes history fleet-wide. Each node derives its own day — the spend by phase, worker, model and provider slot, the turns that ended and how (failed, reviewed, first pass, sent back, a duration histogram), the pages its seats read, the schedules it fired — from its own event log, every 15 seconds while the day moves, and publishes each object’s whole cumulative value. A node without data keeps no event log of its own: what its seats did is handed to the data node that keeps its events, and is counted in that node’s day. Every data node applies every node’s days, so any data node answers for the fleet and a node that leaves takes nothing with it: its days stay in every peer’s rows, and a node that joins later replays them from the stream. Because the node is part of the subject, no two nodes ever write one object and nothing is arbitrated; an apply replaces the object’s rows.

A day is cut on the company’s clock (timezone). A node re-derives today and yesterday on every tick, so the turn that ended a second before midnight reaches the fleet. What it has not yet published whole — the day before the newest it fingerprinted, or an older day a tick could not settle — it keeps as a watermark in its own store (usage_watermark), and every start re-derives each day from that one. So the tail of the day a node stopped in, written after its last tick, reaches the fleet when it comes back, if that day began less than the event log’s 30-day read floor before the node derives it again (a day that began more than 30 days back keeps the value it was last published at, even when the node was down for less than 30 days); a day more than 7 days back — the dispatch ledger’s retention — is derived without its schedules, which the ledger may already have swept. An orderly stop also flushes today and yesterday once more on its way out, after every writer of a usage row on the node has stopped, so a node stopped in an orderly way and never started again leaves its last day whole. A node that crashes and never comes back loses up to one 15-second tick of it.

A node’s day counts the rows it keeps. A custody batch a data node has written but not yet settled may be another data node’s to keep, so no day counts it until it settles — counted on both, every reader summing the fleet would bill a stateless node’s spend twice. Every count of a day is read off rows inside it — a turn’s reviews, send-backs and duration off the one record that ends the turn, which states the whole turn however its other records were spread over the data nodes (a turn whose end states no tally — one a build before that record ended, or one resumed from a park such a build made — is read off its records, as that build read it) — so once a batch settles as kept its rows count on the days they fall on and on no other, which can be days back after a long partition. The settle records the oldest of them (the node’s own usage_rederive rows) and the next tick derives every day from it again. Never a day older than the event log’s 30-day read floor: its rows may already be swept, and a short derivation would replace the whole day on every node with less, so such a day keeps the value it was last published at and the node logs usage_rederive_past_floor once.

A day derived again is cut on the instants it was first cut on (the node’s usage_day_windows), so moving the company’s timezone never moves rows between days already published: the day the clock moves on is longer or shorter by the move instead. For the hours between the old clock’s midnight and the new one’s, a node’s usage day label is not the company’s local date. Moved west, those hours belong to the first day cut on the new clock, which starts where the old clock’s last day ended, so the label runs one ahead. A move of more than a day makes it two ahead for its first hours, and makes that first day more than two days long. Moved east, those hours stay on the old clock’s last day, so the label runs one behind. A move of more than a day leaves the date in between with no instants at all: the node publishes no record for it, and the label runs two behind until the old clock’s day ends. A chart that buckets by label shows the node’s spend for those hours on the neighbouring date.

And no day — today and yesterday included — is ever published below what the fleet already holds of it. Inside the read floor a day only grows, so a derivation that comes out smaller in any figure — fewer turns, failed turns, reviews, send-backs or long turns, fewer tokens, calls, page reads or fires — means the node’s own event log lost rows that record counted: a store restored beside a usage log copied later, one started empty, or a turn whose end states no tally and whose earliest records the event log’s sweep removed (a parked turn’s records reach back to wherever it started). So what a node publishes is its derivation joined with the fleet’s record: every figure at the larger of the two. On an ordinary day that is the derivation itself; after a loss the figures the fleet held more of stay at the fleet’s while every other keeps moving, and the node logs usage_rederive_short once. A seat’s handle and role are the derivation’s wherever it states them and the fleet’s where it does not, so a seat since removed from the org chart keeps its name in the history even where this node’s rows of the day — spend alone, which carries no handle — name nobody. A handle never changes, so the fleet’s is never another seat’s, and a label carried this way is not a short derivation. A join cannot recover what was lost: a figure the fleet held more of counts the node’s new rows only once its own count passes the fleet’s. The node derives no day at all until its usage applier has replayed the log to its end, so the rows it joins with are the fleet’s and not a restore’s.

History older than 181 days leaves in the same transaction that applies the day that makes it old, on every data node alike — there is no sweep. The decision is ADR-0020.

The six capacity ceilings

The broker’s storage limit — stream.store_max_bytes. Every ceiling below is a reservation checked against this one number, so it is the ceiling above the ceilings. Unset, the embedded broker takes three quarters of the free space on stream.store_dir when its JetStream comes up, measured once. Divide it when more than one engine shares a filesystem: free space bounds their sum, not each of them.
Each log’s byte ceiling: stream.tracker_log_max_bytes, stream.tracker_vectors_max_bytes, stream.pages_log_max_bytes, stream.artifacts_log_max_bytes, stream.artifact_data_log_max_bytes, stream.iam_log_max_bytes and stream.usage_log_max_bytes, sized together as below. The last two are not derived from the disk — the identity estate grows with sign-ins and the usage log with the node-days it holds, and a volume has an opinion about neither — so each takes a flat default and its own floor. A full log refuses appends rather than shedding old records; see Retention. On every log that claims identity — the tracker’s, the knowledge base’s, the identity estate’s, the artifacts log’s and the artifact datastore’s, ordinary writes are refused short of it, the top being kept for gate records so that an eviction can still unpin a full log.
The trim floor — how far back the log can be replayed from, which is what bounds how long a node may be away.
The store’s own size — every data node is a full replica, so the corpus is held once per data node.
Applier occupancy — the 16 seconds above.
The snapshot repository — one artefact per node, sized in Retention. An artefact’s manifest names a position per domain, and a joiner refuses one that names no position for a domain it runs, because a checkpoint for a domain the file has no rows for is worse than no artefact at all. That is why adding a domain is a coordinated upgrade — see Running a fleet, and above for what happens in the other direction.

How the byte ceilings are sized

A byte ceiling is a reservation. The broker grants it in full when it creates the stream, before a single record is written, and refuses to create a stream whose ceiling it could not honour. So the state logs compete for one number, and a node sizes them together, once, when it creates their streams — all seven this build registers (the tracker’s, the vectors’, the knowledge base’s, the identity estate’s, the usage log, the artifacts log and the artifact datastore’s log). Every data node runs every one whatever its other roles, and a stream keeps whatever ceiling the node that created it gave it, so whichever node creates them first sets the ceilings the fleet keeps:

Step	What happens
What the broker can grant	Read from the broker itself. An embedded broker’s limit is `stream.store_max_bytes` where you set one, and otherwise three quarters of the free space on the volume holding `stream.store_dir`, counting what its own streams already hold there; an external one’s is the NATS account’s JetStream limit. What counts against it is the ceilings already granted, not the bytes stored.
The logs’ share	Half of that, with the ceilings the logs’ own streams already hold counted as theirs, so a restart divides the same half the first boot did. Where the broker states no limit, the share is half of the stream volume’s free space less what the broker has already reserved for other streams (the `nats` object bucket’s reservation, say), and nothing is added for the logs: a reservation never spends free space, so that figure already contains what the logs hold. Where the broker’s budget cannot be read at all, the share is half of the free space. The other half is for everything that reserves nothing: every mailbox, every coordination bucket and the snapshot a joining node reads.
Each log’s ask	Its Tier A field when you set one. Unset, the mutation log asks for a quarter of the stream volume’s free space (4..64 GiB); the knowledge base’s log for a quarter of the mutation log’s ceiling (the one you set, or the one it derived) because both hold a trailing window of records and the knowledge base’s grows at about a quarter of the rate (1..16 GiB from a volume alone); the artifacts log (`stream.artifacts_log_max_bytes`, 1..256 GiB) for the same quarter of the mutation log’s ceiling, for the same reason: a publish is a human-scale event whose records are a version’s manifest, not its bytes, which the object store keeps; the artifact datastore’s log (`stream.artifact_data_log_max_bytes`, 1..512 GiB) for the same quarter (1..16 GiB from a volume alone), the largest share the pool can give it beside a vector changelog that holds a model change, although its records carry the rows themselves, every cell an artifact page or a seat writes, and a busy sheet writes at the tracker’s rate or above it: a company whose pages write heavily sets the field, which accepts twice the artifacts log’s bound for that reason; the vector changelog for the same quarter of the volume the mutation log derives, whatever the mutation log is set to, because its peak is the whole corpus rather than a window of it; the identity estate’s log a flat 512 MiB (`stream.iam_log_max_bytes`, 64 MiB..16 GiB) and the usage log a flat 1 GiB (`stream.usage_log_max_bytes`, 64 MiB..64 GiB): neither of those two grows with anything the volume has a say in: the identity log grows with headcount and sign-ins, and the usage log is a count of node-days, which more disk does not grow.
The fit	A log whose stream already exists takes the ceiling it holds off the logs’ half first, whatever its field says now. A ceiling you set for a log being created comes off next, and is never scaled. The unset ones being created share what is left in proportion to what each asked for, none goes below 1 GiB, and none is created above what it would get if no log existed yet — the figure every later boot reports its stream against. The 1 GiB is a floor on scaling DOWN, never a raise: an unset ask already below it, the identity log’s, is held at exactly what it asked for, whatever the pool.

A stream that already exists keeps its ceiling. Sizing decides what a missing stream is created with and nothing else: a booting node never rewrites a running stream’s configuration, and the broker never re-checks a reservation it has already granted. A log created larger than today’s sizing would make it boots as it is, and the node logs jetstream_stream_capacity_differs with both numbers — its ceiling, and what this sizing would create it with if no log existed yet. Its reservation was granted when it was made, so this is harmless. To reclaim it (or to raise any log), use crewlet retention set-capacity.

And it counts at that ceiling when another log is created beside it. A log a new version adds, or one whose stream was deleted, is sized from what the existing logs leave of the half, not from what they would ask for today — so a log being created fits inside what the existing logs leave of their share, past it only by the 1 GiB floor and a ceiling you set for it. What the existing logs already hold is not reduced: when they hold more than the share (a ceiling set and later unset, a log created while the volume had more room, one raised with crewlet retention set-capacity), the logs reserve that much past it too, and a log created beside them gets the floor. When a log is created below what it would have had on an empty broker, the node says so with statelog_ceiling_short_of_fit, naming each log it created short, the ceiling it got (created) and the one it would have had (fit), and what each existing log holds (held); once the node is up, crewlet retention set-capacity gives an existing log’s reservation back and raises the new one. It is never created above that fit, even where the existing logs leave more: every later boot reports its stream against that figure, and a log created past it would be reported as a capacity difference on every one of them.

A ceiling the broker would not report is counted as absent. Before it sizes anything a node reads every existing log’s ceiling from the broker — all four at once, each asked again every two seconds for up to thirty while nobody answers. A read still unanswered after that, or one the broker refused, counts that log as one this boot would create. A log that does exist keeps the ceiling it holds, but every log this boot creates is sized without knowing what it holds, so it can come out larger or smaller than a boot that read it would make, and keeps that ceiling. The node logs statelog_ceiling_unread naming the stream; once it is up, crewlet retention set-capacity changes either log.

A boot that still cannot reserve a log says why. When even the floors, or a ceiling you set, do not fit, the node refuses to boot with an error naming the stream, the bytes it needed, the bytes the broker had left and what sets that limit, and the Tier A field the ceiling came from:

engine: the broker refused to reserve the pages log's ceiling:
CREWLET_PAGES_LOG needed 1073741824 bytes and the broker had 536870912 bytes
left to reserve (… of its …-byte limit already reserved), and that limit is
stream.store_max_bytes where you set one, and otherwise three quarters of the
free space on the volume holding stream.store_dir (/var/lib/crewlet/stream),
counting what the broker's streams already hold there.
stream.pages_log_max_bytes is unset, so the ceiling was derived and scaled
into what the state logs that already exist leave of their share of the
broker, and it goes no lower than 1073741824 bytes. Give the broker more room;
a fresh node of this configuration reserves at least 9305762474 bytes of the
broker: 6979321856 for the state logs at their floors and 2326440618 for the
nats object bucket, a quarter of the broker's limit unless
store.objects.nats_max_bytes sets it; the state logs that already exist keep
the ceilings they were created with, and no Tier A setting changes them: …

The sentence about the Tier A field says which of three things the ceiling was: set (… sets the ceiling, with what would have fitted), derived from the volume and scaled, as above, or the log’s own fixed default — the identity log’s and the usage log’s, which do not follow the volume at all.

The remedies are the ones it lists. Give the broker more room: raise stream.store_max_bytes where you set one, or, where you did not, free space on that volume, which the broker measures again when the node next starts. The refusal quotes how much a fresh node of its configuration reserves, and Deployment breaks that figure down and names the settings that lower it on a small host (store.objects.nats_max_bytes above all). Or, when the refused log’s ceiling is above the smallest value its own field accepts, set that field to a smaller ceiling — the refusal offers it only then, and names that floor: 1 GiB for the tracker’s, the vectors’, the knowledge base’s, the artifacts log’s and the artifact datastore’s fields, 64 MiB for stream.iam_log_max_bytes and stream.usage_log_max_bytes. So a refused identity log at its 512 MiB default, or a refused usage log at its 1 GiB, is offered a smaller one.

A log that already exists cannot be shrunk to make room from here. Its ceiling changes only through crewlet retention set-capacity, which runs on a node whose state logs are up, and every mode starts them, maintenance and seal included: a node refused here cannot run it.

Reading a node’s state-log lines

Every line the state log writes about its own work carries component=statelog, so filtering on it narrows a node’s log to its replication:

Line	Level	What it says
`statelog_apply_retrying`	`WARN`	The applier hit a failure it retries in place. Written once, when the run of failures starts.
`statelog_apply_faulted`	`ERROR`	The same failure has outlived the retry budget (30 seconds): this node’s rows have stopped moving, its reads refuse and it stops serving the estate — its seats read it from the other data nodes — until a retry succeeds. Written once per run of failures, when it crosses the budget — not on every retry. While it lasts, the node’s status and every refused read name the current error, and `crewlet.statelog.apply.retries` counts the attempts.
`statelog_apply_recovered`	`INFO`	A retry succeeded and the run of failures is over, with how long it lasted (`after`) and the last error it saw. A failure after it starts a new run, written again from `statelog_apply_retrying`.
`statelog_applier_stopped`	`ERROR`	The applier stopped for good — a gate this build cannot read, a hole that will not close, a recreated stream, a record written in a generation this node never entered — naming the stream, the position its rows froze at and why. Every read of that domain refuses from then on, and for the tracker or the knowledge base the node also stops serving the estate: the seats it holds stay and read it from the other data nodes, and a new one is admitted only on a data node whose copy is sound. What resumes it is a build that can read what this one could not, at its next boot; for a recreated stream, `crewlet retention reanchor` of that one stream, which resumes it in place with no restart; and for a log a peer re-anchored, the snapshot this node adopts on its own. Written once per stop.
`statelog_adopted`	`INFO`	The node replaced its replicated database with a peer’s snapshot, naming the donor, the artefact’s `sha256` (the donor’s `statelog_snapshot_sent` carries the same one) and when it was taken (`taken_at`), which is how old the history it installed is.
`statelog_record_gated`	`WARN`	A durable record this node’s applier dropped: it applies on no node, and this line is its only witness. Each node’s applier writes it once per record, when the transaction that drops it commits. It names the `domain`, the record’s `position` and `kind`, the node that published it (`writer`, empty for a record that names none) and the `gate` that dropped it. The domain’s own gates: `evicted` — the `writer` was evicted below this position and not readmitted; `deleted` — the record is about a task (its turns’ records included) or a page a purge destroyed, whose marker holds every writer’s record on it for ever. The framework’s: `abandoned` for a record written in a generation a reanchor skipped because only an evicted peer held it (retention), and `overtaken` for one a node wrote in the old generation after a restored reanchor’s own record, before it learned of the move (retention). A record more than one gate holds is logged under the first the applier asks — the framework’s two, then the writer’s eviction, then the object’s marker — so a write refused over the same record can name another (`statelog_write_gated` names the one that holds it now). Counted by `crewlet.statelog.records_gated` under the same `gate`, which the `records_gated` alarm reads.
`statelog_write_gated`	`WARN`	A write refused because what it appended applies nowhere, naming the `domain`, `subject` and `op_id`, the `gate` it was refused under, a `position` and a `writer`. When `writer` names a node other than the one whose log this is, the record at `position` is that node’s copy of the operation, which this write’s append was collapsed onto — not a record this node published — and under every gate but `deleted` the refusal is about that node and names it (above). When `writer` is this node, the record at `position` is its own, or another operation’s — the newest on the subject — at which the gate holds this node, so whatever it appended under the operation applies nowhere. The gate is the one that holds the record now, which can differ from the one the applier logged: once a task or page is purged, a record an eviction dropped on it is refused `deleted`. It is a refusal, counted under `crewlet.statelog.publish.refusals` by its reason, and not a second drop: the drop is the `statelog_record_gated` line each node’s applier writes once, which `crewlet.statelog.records_gated` and the `records_gated` alarm count.
`statelog_publish_unknown`	`WARN`	A write could not tell whether its record landed. The operation id is in the line; retry under that id, never a fresh one.
`statelog_write_unvouched`	`WARN`	A write was answered `unknown` rather than published or refused (a `refusal` field says what the decision refused), because its operation was minted (`minted_at`) before this node’s operation ledger may have lost rows to the ledger’s thirty-day sweep, and the ledger holds no row to say whether it already landed. Retrying on this node answers the same; a node whose ledger lost nothing that far back can answer it, and the operation’s own record — if it landed — is on the log.
`statelog_reanchor_started`, `statelog_reanchored`	`WARN`	A generation transition of ONE domain. Both name the domain (`domain`), the one stream it moved (`stream`), the new generation, the stream’s live creation instant (`stream_created_at`), the case (`case`: `recreated`, followed from its first surviving record; `restored`, followed from its end; or `abandoned`, followed from this node’s own checkpoint with the records of the generation an evicted peer held void) and the new checkpoint (`cursor`); the start also names the instant the rows were keyed to before (`keyed_to`), this node’s checkpoint (`position`) and where the log ends (`last_seq`) and the generation its rows stood at (`from_generation` — every generation strictly between it and the new one is abandoned), and the completion gives the stream’s high-water mark before the reanchor (`prev_last_seq_seen`). A restored reanchor the operator ran with `-discard` names, on the start as `discarding` and on the completion as `discarded`, the sequence of the newest record written after the restore that it applied on no node (0 when it discarded none). No other domain’s checkpoint moves, and the domain’s applier resumes without a restart.

The snapshotter, the donor and the adopter write under the same component. The lines the engine writes around those loops — statelog_stream_recreated and statelog_below_the_floor among them — carry component=engine, because the component names the code that wrote a line rather than what it is about. Four of them are about a peer’s reanchor: statelog_generation_passed (WARN) is the heartbeat finding a log re-anchored past this node’s generation, naming both (generation, fleet_generation), from which point the domain refuses; statelog_behind_a_reanchor (WARN) is the join asking the fleet for a snapshot at the new generation, naming the domains and the generations it asks at; statelog_generation_holder_unread (WARN) is a beat that could not read whether the node that opened that generation is evicted, which decides whether the refusal names the adoption or a reanchor, and leaves it naming what it did; and statelog_generations_unread (WARN) is a beat that read the positions register but could not establish which generation the fleet is on — the trim floors unread, or an eviction record unreadable. Such a beat judges neither that nor the truncation below, and leaves both verdicts where they were: the peer whose rows hold what a log lost is usually the peer that then re-anchors past it, so judging the truncation alone would lift that fence on the very beat that could not yet say the passed verdict replaces it. See a node a peer re-anchored past. Two are about a broker restored from an older copy: statelog_log_diverged (ERROR) is the log holding, at this node’s checkpoint, another record than the one it consumed there — the restored log was written past this node’s rows — written once, at boot or by the heartbeat, from which point the domain applies nothing past its checkpoint and refuses until it is re-anchored or its rows are replaced; and statelog_checkpoint_unverified (WARN) is a heartbeat that could not read the record at the checkpoint to compare it, which the applier asks again before it applies past it. And three are the same restore seen from a node whose rows are the copy’s age: statelog_log_truncated (ERROR) is the heartbeat finding a peer whose rows hold records the log lost — past its end, or reporting log_diverged — naming it (peer, peer_seq, last_seq, peer_diverged), from which point the node’s writes of that domain refuse log_truncated while its reads go on; statelog_log_truncated_cleared (WARN) is that ending, once the peer has re-anchored, been rebuilt or been evicted; and statelog_truncation_unread (WARN) is a beat that could not establish it either way, which leaves the verdict where it was — as does a statelog_generations_unread beat, which does not ask.

Three things CI cannot prove

Stated here because the alternative is implying a guarantee nobody measured:

Durability under real power loss. The fsync counting and the fault-injection suites test the code path; a real machine losing power at a real disk’s worst moment is not reproducible in CI.
Behaviour of an external NATS cluster. The engine does not run it and cannot assert its settings.
Search capacity under production load. The scan benchmark has a concurrency axis, but the supported corpus at your concurrency is a projection until it runs on your hardware.

What the design does not promise

N nodes applying one ordered log deterministically produce N identical copies including of an apply bug. A replica set protects against machine loss and against nothing else.

The residual is real and is not engineered away: a bug that corrupts silently and is noticed after the backup window has rolled costs history. That is why the trim refuses to advance past a floor no backup has reached — the backup gate is not ceremony — and why the restore test has a cadence rather than being a thing somebody did once.