Metrics

Generated from metrics.Catalogue(). Do not edit — change the catalogue.

The engine exports OpenTelemetry metrics through the same OTEL_* environment its traces use, because a company has one telemetry backend and two settings would let you split signals across two of them where a correlation resolves on neither:

Variable	What it does
OTEL_EXPORTER_OTLP_ENDPOINT	The collector. Metrics are posted to <endpoint>/v1/metrics.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT	Overrides the base for metrics alone.
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf (default) or grpc.
OTEL_EXPORTER_OTLP_HEADERS	Sent with every export — an API key, a tenant.
OTEL_METRICS_EXPORTER	none switches the export off and leaves traces on.
OTEL_METRIC_EXPORT_INTERVAL	Export period in milliseconds. Default 60000.

The provider is installed whether or not you are collecting. Every instrument exists either way, so no code path in the engine branches on whether metrics are “on”, and the numbers on crewlet retention status are read from the same recorder a collector would be reading. With no endpoint set the measurements are taken and not shipped.

Every attribute below is a closed set. No metric carries a task key, a subject id, a page title or a node id as an attribute — the node is the resource, set once — so the number of time series is bounded by this page rather than by how much work your company has done.

Histograms

A distribution, exported with the engine’s own bucket boundaries: twenty-one powers of two from 64 µs to 64 s, which resolves a percentile to within a factor of two at every scale here — from a 40 µs index probe to a 16-second bulk apply. The SDK’s default boundaries stop at 10 s, so a slow apply would land in an overflow bucket and read as “at least 10 s” for ever.

Metric	Unit	Attributes	What it makes visible
crewlet.statelog.publish.duration	ms	domain, outcome	A write path slowing down before it starts refusing. The outcome dimension separates the three answers a write has, so a rise in pending reads as an applier falling behind rather than as a broker getting slower.
crewlet.statelog.publish.rounds	1	domain	Contention on one subject, which the compare-and-set round cap bounds and nothing measured. A distribution creeping toward the cap is a hot object; reaching it is a refusal a model reads as a colleague editing the same thing.
crewlet.statelog.write.session_wait	ms	domain	The wait a write pays for its own previous write to apply. Zero when the caller is caught up, which is the common case, and the shape of the load that is not.
crewlet.statelog.barrier.duration	ms	domain	The broker round trip under every linearizable read, and the first number a drifting fsync or a degrading quorum moves. It was a benchmark’s p50 on an idle loopback cluster and nothing in production.
crewlet.statelog.read.wait	ms	domain, level	How much of the read budget a barrier or session wait actually spends. A p95 approaching the budget is reads about to start refusing, which is the warning the refusal itself is too late to be.
crewlet.statelog.apply.latency	ms	domain	THE COMMIT-TO-APPLY GAP: from the broker’s own timestamp on a record to this node committing it. Every read level is a policy about this quantity and nothing measured it.
crewlet.statelog.apply.tx.duration	ms	domain, bound_by	How long one apply transaction holds the store’s writer, and which budget ended it. A transaction is what every waiter behind it pays, and rows were only ever a proxy for the duration.
crewlet.statelog.apply.record.duration	ms	domain, kind	One record’s apply. A single record past the time budget is still one transaction, so this is the real ceiling on how long a read can be delayed — a sentence in a design document until it was measured.
crewlet.statelog.apply.batch.rows	1	domain	Rows per apply transaction, which is what the row budget bounds and what the drain rate divides.
crewlet.backup.duration	ms	—	How long a backup took, which is the window the trim hold covers and the I/O the copy spends competing with the applier’s own commits. It is what turns the retention guide’s worked example into a number for THIS hardware.
crewlet.store.pool.wait	ms	file	How long a reader waited for a connection. It is what says the reader pool is too small on this node, which nothing could say before.
crewlet.tracker.search.scan.duration	ms	path, rung	One search’s scan, split by whether it ran for a turn’s prefetch or for somebody’s deliberate search, and by the ranking it actually served (rung: hybrid, keyword, semantic, or none). Only the prefetch had a published percentile, and the interactive path is the one with a target; a keyword scan is a fraction of a fused one’s cost, so reading them as one series hides a slow meaning half.
crewlet.artifacts.publish.duration	ms	outcome	How long one create, publish or asset finalize took through the pipeline (read back, edits, lint, CDN copies, the domain’s decide), by outcome: published, refused (the request was judged and turned down) or failed (no outcome could be established, which is a store or a broker not answering). The publish latency objective is read from it.
crewlet.artifacts.serve.duration	ms	dest	How long the artifact origin took to answer, by dest, the whole response included. A document’s tail is the time to first byte a viewer waits on before a frame can render, which the service objective for viewing reads; a slow subresource tail with fast documents is the object store.
crewlet.artifacts.data.write.duration	ms	verdict	How long a datastore write takes from the request to its verdict read back from this node’s rows, by verdict (applied, conflict, invalid, forbidden, quota, gone, or pending where this node had not applied it yet, rate_limited and refused_early before anything was appended). The datastore op objective is a p95 of 250 ms; a rise in pending reads as this node’s applier falling behind.
crewlet.artifacts.push.lag	ms	kind	How long after the broker stored a committed datastore batch’s newest record its artifact_data push was handed to this node’s sockets (kind data): the applier’s lag plus the fan-out. A datastore change is held to reach the socket within a second. An artifacts movement carries no broker instant, so artifact_changed is not measured here; its lag is the artifacts applier’s own, in the state log’s apply instruments.
crewlet.artifacts.ready.duration	ms	outcome	How long a frame took from its view ticket being minted to the page’s bridge hello, as the viewer’s browser reported it through POST /artifacts/ready, by outcome (ready, or failed for a frame that never said hello). The only artifacts instrument that sees an outage keeping requests from reaching the engine at all (the artifact domain’s DNS or certificate, an edge, a corporate filter): every server-side instrument reads such an outage as a quiet node. The artifact_ready objective holds 99.5% of opens within 3 s.

Gauges

A value that goes both ways, sampled at each export.

Metric	Unit	Attributes	What it makes visible
crewlet.statelog.drain.rows_per_second	1	domain	The applier’s observed drain, which every retry hint divides by. Seeded from a benchmark and then measured, so a hint on real hardware stops being an extrapolation from somebody else’s.
crewlet.statelog.drain.commits_per_second	1	domain	Commits per second, which is the fsync rate under synchronous = FULL and the number a device budget is spent by.
crewlet.statelog.apply.lag.seq	1	domain	How many records this node is behind the log’s head.
crewlet.statelog.apply.lag.seconds	s	domain	How OLD the oldest unapplied record is. Seconds are what a stall grace, a pending outcome and a copy taken out of service all turn on; sequences are not, and a lag of 4 000 says nothing about whether anything is wrong.
crewlet.statelog.applied_through	1	domain	The prefix this node has actually applied, which is lower than its checkpoint whenever a record was retained.
crewlet.statelog.deferred.count	1	domain	Records this build could not read and kept. Non-zero is a rolling upgrade in progress; non-zero and not falling is one that stopped.
crewlet.statelog.deferred.oldest_age_seconds	s	domain	How long the oldest retained record has been retained, which is what decides whether this node’s copy stays in service.
crewlet.statelog.waiters	1	domain	Callers blocked on the applier right now. It is the depth of the queue a slow apply is making.
crewlet.statelog.log.bytes	By	domain	What the log actually holds, against its ceiling below.
crewlet.statelog.log.bytes_per_day	By/d	domain	What the log took in over the trailing day, measured from its own records at each trim tick. The ceiling has to hold min_age of this, because the trim never removes a record younger than that — a log whose ceiling is smaller fills with every trim term satisfied, which is what log_ceiling_short fires on. Absent for a compacted log and for one in its first two days, whose trailing day would contain the day it was imported into; a log that could not be read keeps its last measurement, as the report does.
crewlet.statelog.log.max_bytes	By	domain	The ceiling, read from the running stream rather than from this node’s own configuration — the two differ, and the running one is what refuses the append.
crewlet.statelog.log.headroom_fraction	1	domain	How much of the ceiling ordinary writes are held to is left — on every log that claims identity (the tracker, the pages and the identity estate), the ceiling less the gate reserve kept above it for evictions. At zero the log refuses every write AND every linearizable read, and the remedy is a fleet-wide maintenance cycle, so this is the one number worth alarming on long before it is small.
crewlet.statelog.trim.blocked_seconds	s	domain, term	How long one retention term has held the trim, named. A trim blocked for weeks is a log walking toward its ceiling with a cause an operator can act on.
crewlet.backup.age	s	—	How long ago the newest COMPLETE backup finished, read from the manifests on disk rather than from a counter this process keeps. A counter records that a process believed it took a backup; the disk records that one exists, and they differ in exactly the cases the alarm is for — a copy deleted, a volume never mounted, a schedule pointing at a path nobody ships from. A node holding no manifest reports NO SERIES rather than a value: zero is the freshest backup imaginable and every other number is an age nobody measured, so neither can stand for an absence. The absence is what the backup_age alarm says in words, and an absent() rule over this gauge is what a collector alerts on.
crewlet.backup.holds	1	—	Live trim holds. A pin that outlives its owner stops the trim until the stale bound expires it, so a count that does not return to zero is a backup that crashed mid-copy.
crewlet.store.wal.bytes	By	file	A write-ahead log a checkpoint cannot pass grows, and this is the only way to see it before the volume fills.
crewlet.store.bytes	By	file	The store’s size on disk, which the snapshot’s free-space precondition and the provisioning rule are both derived from.
crewlet.tracker.search.concurrency	1	—	Searches in flight, which is the row of the supported-corpus table this node is actually on: inside the one-second budget, about 345 000 sources with one running and about 136 000 with eight through the full scan, and about 545 000 and 183 000 through an index probing half its lists.
crewlet.tracker.vector.coverage	1	—	The fraction of sources carrying a current vector. It is how a stalled embedding backlog is reported, since it never drops a seat. Absent where the company configures no embeddings and while the corpus cannot be measured — an unreadable corpus is neither covered nor uncovered.
crewlet.alarm.active	1	kind	Whether each named alarm is firing right now, 0 or 1 — for one that stands on several logs, whether it stands on any; its domains and its log lines say which. It is the same table the operator record renders and the CLI exits non-zero on, so a collector and a person see one answer.
crewlet.objects.bucket.bytes	By	—	How many bytes the fleet’s own object bucket on the broker holds (OBJ_crewlet_files: the company’s files, their metadata and any upload in flight), as the object collector’s duty holder read it; only that node reports it, and only on the nats backend. Read against crewlet.objects.bucket.max_bytes: a full bucket refuses every upload, and objects_bucket_near_ceiling fires at four fifths.
crewlet.objects.bucket.max_bytes	By	—	The byte reservation the broker holds the object bucket to (store.objects.nats_max_bytes, or the share the node that created the bucket derived): the ceiling the bucket keeps for its life, whatever a node’s Tier A says later.
crewlet.artifacts.upload.reserved_bytes	By	—	Bytes the company’s upload reservations hold, as this node last wrote the reservation ledger: bytes in the object store, or on their way to it, that no record names yet. Read against artifacts.limits.pending_bytes times four, the company’s bound: a value that stays near it is clients abandoning uploads, each held until the object collector could have deleted its bytes.
crewlet.artifacts.data.footprint.bytes	By	—	The company’s datastore footprint in real bytes held on every data node (each document row with its index and field-index entries, and hot history, as the applier meters what the rows cost on disk), sampled after each committed batch. Read against artifacts.limits.data_bytes: a store or a company at its limit refuses every write that grows it.
crewlet.presence.rooms	1	—	Artifact rooms this node holds a member of, a socket’s or an agent’s. Each costs a map in memory and, while one of this node’s sockets is in it, one listen on the broker.

Counters

Only ever rises. Rates and totals are your collector’s arithmetic, never this engine’s.

Metric	Unit	Attributes	What it makes visible
crewlet.statelog.publish.outcomes	1	domain, outcome	The three-valued write outcome, counted, and only the three — a refusal is on publish.refusals instead, because it says the write never happened at all. unknown is the one that matters most and had no counter: a broker flapping into ambiguity was visible only to the model that received the answer.
crewlet.statelog.publish.conflicts	1	domain, subject_kind	Writes that spent their whole round budget losing races on one subject, BY KIND. The refusal counter beside it says a conflict happened and not what it was about, and the remedy differs entirely: one contended object is a design question and a contended kind is a hot subject.
crewlet.statelog.publish.rejections	1	domain, subject_kind	How often a write loses a race, per kind of subject. It is what says whether a counter, a rank order or an ordinary object is the contended one.
crewlet.statelog.publish.refusals	1	domain, reason	Writes refused before or instead of an append, by reason, and EVERY value it carries is here, each with its remedy. The node: evicted (this node is removed from the fleet — run the write on another), eviction_unknown (its own eviction state could not be read — clears when it is read again), deferred (it holds a record it cannot decode — a newer build serves it), behind (it has not applied a position the write needs — clears on its own), below_floor (it is below the log and must adopt a snapshot — another node writes meanwhile), floor_unknown (the floor or the log’s ends could not be read — clears when coordination answers). The log: log_full (at its ceiling — raise it or unblock the trim), record_too_large (the record is larger than the broker takes in one message — split the change; the detail names which limit refused it), broker_refused (a refusal this build has no remedy for — the detail carries the broker’s own words), skew (a store and a stream restored out of step — restore both from one backup), wrong_stream (not the log this node’s rows came from: rebuilt, ending below its checkpoint, holding another record there, or re-anchored past it by a peer — re-anchor, or adopt where a peer holds the generation), log_truncated (it lost records a peer’s rows hold — settle which history the fleet keeps). The object: deleted (a permanent deletion marker — nothing recreates it). The operation: op_reused (its id names a record on another object) and superseded (its record landed and a later one undid it) — both answered by a NEW operation under a fresh id, never by a retry. A record that landed and a gate dropped is counted under the gate that dropped it — evicted, deleted, abandoned (written in a generation a reanchor abandoned) or overtaken (written after a restored reanchor, below its generation) — and is never re-decided, because republishing makes another record nothing applies. Such a record holds its operation id for the log’s duplicate window, so the same id sent inside it, by any node, is collapsed onto the record and refused the same way: the write is retried under a fresh id, or under the same one once the window has passed. Beside the refusals: conflict for a write that lost every round, exists for a create whose object already exists, and error for a write that failed before it could answer. A refusal is not one of the three outcomes: it says the write never happened, and each reason has a different remedy, so one counter with an outcome dimension would hide them all.
crewlet.statelog.read.refusals	1	domain, level, code	Every refusal code, counted. The codes, each with its own remedy, had no counter between them, so an operator had no rejection rate for any of them.
crewlet.statelog.read.served	1	domain, level	Reads answered per level, which is the denominator every refusal fraction needs and the check on the assumed read rate the log’s own size is derived from.
crewlet.statelog.barrier.appends	1	domain	Barrier records appended. Against reads served it is the single-flight ratio, which says whether coalescing is doing anything at all.
crewlet.statelog.barrier.applied	1	domain, stream	Barrier records applied from each log — every node’s linearizable reads on it, since every data node applies every record. It is the rate census_drift holds against the census, per log: a node’s own appends are only its share of the fleet’s reads, and on a fleet of several serving nodes they describe a company that many times quieter than it is. The alarm’s 24-hour window files each barrier at the hour the broker stored it, so a node replaying a backlog does not read days of reads as one; this cumulative series counts them as they are applied, so it jumps by a backlog’s barriers when one is replayed.
crewlet.statelog.linger.yields	1	domain	How often a waiter cut a batch short. It is the batching the applier gives up to answer a read promptly, and without it that trade is invisible.
crewlet.statelog.apply.records	1	domain, result	Records consumed, by what happened to them: applied, retained, gated, skipped, or reprocessed by a build that could read what an earlier one retained — a retained record a gate drops when it is reprocessed is counted gated, as it would have been live, since it wrote no row. A node applying nothing while its position advances is healthy on lag alone.
crewlet.statelog.apply.retries	1	domain	Attempts the apply loop retried in place after a failure that was not a stop — a fetch the broker did not answer, a transaction the disk refused, an applier that errored. A rate that stays up past the retry budget is a node whose rows have stopped moving, and its health says so.
crewlet.statelog.apply.tx.aborts	1	domain	Apply transactions whose body the store ran more than once, because an attempt failed transiently after it began. It reads zero by construction: this database detects write conflicts per file, so every write transaction takes the file’s lock at its BEGIN and queues for it, and no commit elsewhere in the file can abort an apply. A non-zero count on the operator’s own hardware means the retry budget is being spent rather than held in reserve.
crewlet.statelog.records_gated	1	gate, subject_kind	Records an apply gate dropped, by the gate that dropped each — the domain’s own (evicted, deleted) or the framework’s (abandoned, overtaken). Each node counts a record once, where its own applier drops it, when the transaction that drops it commits — never once per attempt at that transaction. A write refused because its record — or another node’s copy it was collapsed onto — was dropped is a refusal, counted under crewlet.statelog.publish.refusals, and not a second drop. A dropped commit is recoverable by nothing, and this is the only place anyone would see that it happened.
crewlet.statelog.records_tampered	1	domain	Records whose signature failed under a key this node holds, which means something that is not this fleet wrote to the broker. The applier stops on the first one, so this counts at most one per domain per stop, and any value above zero is an incident rather than a rate.
crewlet.statelog.records_unverifiable	1	domain	Records signed under a key this node does not hold, retained until it does. Expected and transient during a keyring rotation; sustained, it means a key reached one node’s secrets.keys and not this one’s.
crewlet.history.answers	1	question, coverage	What each fleet history read covered: complete when every live node answered inside the fleet read budget, partial when one did not. Turn-level detail lives only on the node that published it, so a partial answer is missing that node’s rows — and the history_partial alarm is a fraction of this counter.
crewlet.tracker.search.answers	1	coverage, semantic	What each answer actually covered: whether every bucket of the corpus was scanned, and whether the semantic half ran (full), was asked for and lost (skipped), or was never asked for (off — a keyword search, or a company with no embeddings). Both alarms below it are a FRACTION of this counter, and a short answer is indistinguishable from a short corpus without it.
crewlet.tracker.feed.unreadable	1	source	Change records this build could not translate into a wake. Both domain consumers are deliberately uncapped, so such a record is never dropped — it redelivers for ever at the head of the consumer with every wake behind it waiting, which has no other symptom at all.
crewlet.tracker.bulk.calls	1	result	How often a bulk edit is issued, by result: admitted under the fleet’s bulk lease, refused because another is applying, and fail_open — let through with NO lease, because the coordination store did not answer the admission or the mixed-version gate refused it. The refusal arithmetic rested on an assumed ten a day, a number with no counter behind it; this is that number. A fail_open is a bulk the fleet-wide bound did not cover, and counted as admitted it left no trace: a store that refused every admission looked exactly like a fleet whose bulks never collided.
crewlet.tracker.bulk.apply_seconds	s	—	Seconds of applier occupancy bulk edits projected, summed. Over 24 hours it IS the fleet-wide read-degradation budget: every second here is a second in which reads are behind and writes are pending on every node.
crewlet.auth.attempts.failed	1	method, outcome	Every failed attempt to prove an identity, as it happens: a sign-in, a step-up, an invitation or password reset link that did not prove itself, or a bearer presented to a route that needs one and refused — a credential an unguarded route verifies itself, such as a webhook relay’s own JWT, is never counted (method), and whether the attempt was verified and wrong or answered 429 by the throttle’s curve (outcome). It is the ONLY per-attempt record there is, by design: an unauthenticated caller authors this rate, so the event log gets one iam_login_failures row per node per minute, counting each source, and the rate itself lives here, where a guessing run costs a counter increment rather than a row in every backup.
crewlet.artifacts.upload.bytes	By	transport	Bytes of artifact files that reached the object store, by how they got there: streamed through this node (engine, which counts what the pipeline writes itself: a linted page, an edit’s result, a copied CDN file) or put straight to an S3 bucket on a presigned URL and read back here (direct). A fleet whose engine share stays high on s3 is streaming uploads its bucket could have taken.
crewlet.artifacts.publish.refusals	1	reason	Uploads and publishes refused, by reason from a closed set (internal/artifactupload.Refusals): a client sending the wrong bytes (upload_mismatch), a page breaking the policy (policy_refused), a company out of quota (quota_exceeded) or reservations (uploads_outstanding), an agent’s turn past its guardrails (turn_quota), a moved head (version_moved) and the rest. Each reason is a different person to call.
crewlet.artifacts.serve.requests	1	dest, status	Every request the artifact origin (the api.artifacts listener) answered, by what it was (dest: document, a page decided again on every request; subresource, a file a page loads; runtime and kit, the embedded public files) and by status class (status: 2xx to 5xx). A rise in document 5xx is viewers whose frames this node would not serve: a copy of the artifacts or identity log it could not vouch for, or an object store that did not answer. The artifact id is never an attribute; per-artifact detail is in the log lines.
crewlet.artifacts.serve.bytes	By	—	Body bytes the artifact origin sent, compressed where they were sent compressed: what artifact viewing costs this node in egress, per node and never per artifact.
crewlet.artifacts.ticket.refusals	1	reason	View tickets the artifact origin refused, by reason: invalid (forged, malformed, or signed under a key this node does not hold), expired (a frame past its hour, which the dashboard re-mints), host (a ticket presented on a host other than its artifact’s), navigation (a ticket URL opened in a tab of its own), revoked (the share, the credential, the artifact or the version is gone) and stale (this node could not vouch for what it would decide from, and answered 503). invalid above zero sustained is somebody presenting tickets this fleet never minted, or a keyring rotated past a key still in use.
crewlet.artifacts.authz.peer_checks	1	outcome	Authorizations the artifact origin decided through another data node because its own artifacts copy was out of service or lagging, by outcome (served, refused, unavailable). Any at all is a node whose copy is not answering; unavailable is a viewer refused because no data node could decide for them.
crewlet.artifacts.data.writes	1	verdict	Datastore writes by verdict. A climb in conflict is people editing the same rows over each other; in forbidden, a page or an agent writing what its collection’s rules refuse; in rate_limited, one client spending this node’s write budget.
crewlet.presence.frames	1	direction	Presence frames this node handled, by direction: client (a socket’s presence frame taken), peer (a sealed frame from another node admitted) and push (a room’s coalesced fan-out to this node’s sockets). Presence is told, never stored, so this is the whole of its cost: a peer climb with flat client is a busy room held on other nodes, and push is bounded at five a second per room.
crewlet.objectives.events	1	objective, outcome	Every service objective’s events, by objective (the closed set in docs/reference/service-objectives.md) and outcome (good: answered without fault on the engine’s side within the objective’s bound; bad: anything else). bad over all, per objective, against one minus its target is the error budget’s burn rate, which the objective_burn alarm judges per node over an hour and five minutes and over six hours and thirty minutes; summed across the fleet it is the company’s.

What is deliberately not here

Rates. Every counter is a monotonic total and your collector divides. A rate computed in this process would be a rate over a window nobody chose, disagreeing with the one on your dashboard.

A /metrics route. OTLP reaches Prometheus through the collector you already run for traces. A second wire format would be a second thing to authenticate on an API whose guard and whose exemptions are both load-bearing.

Per-object series. See the attribute rule above.