Agent Runtime

The agent runtime (crewlet.agent) manages agent lifecycle, execution, and memory.

Per-turn execution: every agent turn runs through the three-phase Plan → Execute → Review Turn Engine. Every agent turn described on this page dispatches into TurnEngine.run_turn, which owns concurrency, OTel context restoration, phase dispatch with the iteration cap and stall detection, ephemeral sub-agent spawning, and the runtime invariants (delegation-depth cap, sub-agent tool allowlist, budget cascade). The sections below describe the surrounding lifecycle; the turn-engine doc describes what happens inside a turn.

Agent Definition vs Agent Instance

Each agent seat (Role.kind == "agent") maps 1:1 to an AgentDefinition and a single AgentInstance. Human seats are never spawned — they exist only in the Organization and resolve through the party-level HandleRegistry API.

Identity is deterministic. AgentInstance.id is computed as uuid5(AGENT_ID_NAMESPACE, f"{org.name}:{handle}") (see crewlet.db.agents.derive_agent_id). The same role in the same org always lands on the same UUID across processes, machines, and restarts. Anything keyed by agent.id — agent_diary rows, agent_onboarding_markers rows, counterparty_profiles keyed by observer_handle — therefore survives engine restarts.

Rename caveat. Both inputs are part of the derived id: changing a role’s handle or the organisation’s name creates a new derived id and orphans the prior per-agent rows (diary, onboarding markers, counterparty profiles). The seat keeps working — it has simply lost its memory. A company rename does this to every seat at once, so settle name and each handle before the company runs. (An explicit handle on each role pins half of it; nothing pins the org name.)

For team lead agents, the system prompt includes a team roster — a summary of each direct report’s name and handle. Detailed per-member profiles (skills, backstory, tools) render directly into the lead’s Plan-phase prompt from the in-memory Organization model when the lead needs to reason about assignment.

Agent States

Created — instantiated but not yet registered with the engine
Idle — listening for events, available for task assignment
Working — actively executing a task (LLM calls in progress)
Terminated — removed from the company

Agent Execution Loop

Each agent, when triggered (by event or task assignment), executes a turn through the three-phase Turn Engine:

1. Collect context (task, knowledge, trigger event, delegation chain)

2. Plan phase
   ├── Planner LLM sees a slim catalogue (builtin tool names + MCP
   │     server names) in its system prompt
   ├── Meta-tools: submit_plan, activate_tool, list_mcp_server_tools,
   │     load_tool_skill
   ├── To use an MCP tool: list_mcp_server_tools(server) to discover
   │     names, then activate_tool(name) to promote into tools=[...].
   │     Reserve activation for read-only recon (Slack thread reads,
   │     Jira fetches, agent lookup); action / write tools should not
   │     be activated here -- name them in tools_needed for Execute.
   └── Emits an ExecutionPlan: reasoning, steps, tools_needed, criteria

3. Execute phase
   ├── Tool surface = plan.tools_needed ∪ executor_always_on_tools
   │     ∪ {activate_tool, list_mcp_server_tools}
   ├── Same slim catalogue Plan sees, same discover-then-activate flow
   │     -- the executor can recover when the planner missed a tool
   │     by calling list_mcp_server_tools + activate_tool mid-run.
   │     Successful activations fire phase.tool_activated events.
   └── Tool-call loop drives the LLM through the plan

4. Review phase
   ├── submit_review emits a ReviewOutcome
   └── done | self_iterate (loop back to Plan)

5. Emit events, update memory, return to Idle

The Plan and Execute phases can run on different LLM models — see the Turn Engine doc.

The LLM ↔ Tool Proxy

The LLM is an external HTTP service — it cannot access local code, MCP servers, or engine internals directly. The shared tool-call loop (crewlet.agent.llm_loop.run_tool_loop, driven by each phase of the Turn Engine) acts as a proxy that translates between the LLM’s text-based tool calls and local execution:

Both builtin and MCP tools produce identical tool definition schemas. From the LLM’s perspective, lookup_colleague (builtin) and jira_create_issue (MCP) look the same — a function it can request the engine to call.

System Prompts (per phase)

Under the three-phase Turn Engine, each phase builds its own narrow system prompt — there is no single monolithic prompt for a turn. Each builder lives in src/crewlet/agent/prompts.py; the detail layer is in src/crewlet/agent/definition.py. Founder-defined role/org context (mission, vision, policies, backstory, responsibilities, behavioral guidelines, team roster) renders directly from the in-memory Organization model into the Plan prompt via the section builders in crewlet.agent.definition — no DB seed step, no reconcile pass.

Phase	What’s in the prompt
Plan	Identity (role, unit, goal, manager, direct reports, Slack channel), full policy text, role profile (backstory + responsibilities + behavioral guidelines), unit context (purpose + goals), team roster with per-member profile (leads only), compact plan-phase contract, Tool Skills catalogue (one-line summary per triggered skill), slim tool catalogue (builtin tool names + MCP server names; MCP tool names hidden behind `list_mcp_server_tools`). Plus the five learning prefetches: `## Similar prior work` (episodes), `## Personal memory` (diary), `## Synthesized skills you've learned`, `## Relevant knowledge` (live knowledge-base search — the aux LLM generates a plain-text query from the trigger, run against Confluence or Plane), `## First-turn onboarding` (until `mark_onboarded` fires).
Execute	One-line identity, plan summary, execute contract (which now describes the discover-then-activate flow), Tool Skills catalogue scoped to `plan.tools_needed`, optional counterparty profile block, optional `## Relevant knowledge` block (the post-Plan re-fetch — present only on thin-trigger turns where the Plan-phase prefetch was gated off), and the same slim tool catalogue Plan carries so the executor knows what discovery surface is available. Skill bodies arrive on demand via the always-on `load_tool_skill` builtin. No policies, no roster.
Review	One-line identity, plan summary + Plan tool log + Execute tool log, decision-enum contract, Tool Skills catalogue for MCP-server-keyed skills (operator-scoped to the Review phase). Both phase tool logs render as separate sections (`## What Plan did` / `## What Execute did`) so the reviewer can tell which phase delivered each side effect — without the Plan log, a side effect fired during in-Plan recon looks like missing delivery and the reviewer self-iterates a turn that already shipped. No tool catalogue, no policies, no roster.
Sub-agent	Parent-provided task prompt, Tool Skills catalogue scoped to the parent-passed tool allowlist, then the mandated runtime preamble (no further sub-agents, no colleague contact, concise final answer).

Why the split: the planner is the only phase making ownership / delegation / policy-sensitive decisions, so it gets the richest context. Execute and Review run against the plan’s explicit success_criteria — they don’t need to re-derive those from policies or backstory.

Built-in engine scaffolding

Engine guardrails (“event triage framework”, “escalation judgement”, “tool usage instructions”, “knowledge-system usage”) are carried by tool descriptions (confluence_search, colleague-surface tools) and by the plan/review contracts themselves — not by dedicated prompt prose. Each tool’s one-line description tells the LLM when to use it; the per-phase contract tells the LLM what output shape is expected. There is no special escalation mechanism — when stuck, an agent reaches its manager with the same colleague-surface tools it uses for any other collaboration (a Slack mention, a Jira comment, a2a_ask); Review routes a blocked turn back through self_iterate so Plan adds that outreach step (no escalate tool, and no ask_colleague decision).

Tool- and MCP-server-specific guidance (when to call reflect_and_persist, how to mention teammates on Jira vs Slack, when to author code via the code sandbox and what the GitHub tools are for) lives in the Tool Skills registry — modular knowledge-base-sourced fragments (Confluence or Plane pages) where each skill carries a short summary (always inline in the per-phase catalogue) and a rich body that loads on demand via the always-on load_tool_skill builtin. The engine ships no skill prose; operators seed the skills container with crewlet confluence import / crewlet plane import and edit pages in the backend’s editor thereafter.

The AgentDefinition.system_prompt property composes a single monolithic prompt for introspection / external callers; the runtime path (TurnEngine.run_turn) never calls it. AgentDefinition.build_system_prompt_with_skills(registry) is the registry-aware variant for callers that want that combined prompt to stay consistent with what the live engine builds.

Built-in Tools

Every agent has access to these built-in tools (registered in the ToolRegistry):

Tool	Purpose
`lookup_colleague`	Resolve any agent identifier (handle, Slack user, Jira ID, etc.) — case-insensitive with substring / fuzzy fallback; returns a candidate list when ambiguous
`use_skill`	Load one of the agent’s own synthesized skills on demand
`load_tool_skill`	Load the full body of a Tool Skill by exact key (the catalogue carries only the summary). Required skills (the default; `required: false` opts out) must be loaded this way before the tools they cover can be called — the engine rejects earlier calls with a “load this skill first” error
`refine_skill`	Patch a synthesized skill (append observation, replace body)
`query_episodes`	Search the agent’s own past turns by similarity
`reflect_and_persist`	Capture a durable fact in the agent’s private diary (LONG / SHORT)
`refresh_memory`	Re-run the personal-memory filter mid-turn with a context hint
`mark_onboarded`	Stamp the agent’s onboarding marker after reading the relevant knowledge-base pages
`spawn_subagent`	Run an ephemeral bespoke sub-agent with a parent-chosen tool allowlist. See Turn Engine — Sub-agent phase

Colleague outreach happens through the upstream MCP tools directly (on the common stack: slack_conversations_postMessage, jira_add_comment, jira_update_issue, confluence_add_footer_comment, request_copilot_review — these are examples, not engine-known names) — there are no thin engine-side wrappers (slack_message, jira_comment, etc.); register_colleague_tools registers only a2a_ask, the private agent-to-agent bus. Use whichever chat / issue-tracker / wiki / code-host tools your MCP servers expose for any collaboration a human teammate would reasonably want to see; a2a_ask is narrowly scoped to tight-loop / mechanical sync between agents. The engine prompts name none of these — they describe the capability and the LLM picks the tool from its catalogue (see Tool Capabilities). See Turn Engine — Colleague-surface tools for when to use each.

Decisions use the agent’s Slack MCP tools and team channel — see Decision Framework.

MCP Tools

MCP tools (Jira, Slack, GitHub, etc.) are dynamically discovered from configured MCP servers at engine boot and registered alongside builtins. Plan and Execute do not see every MCP tool name in their system prompts (a role with 50–150 MCP tools would push 15–25 KB of catalogue into every prompt); instead the prompt lists MCP server names and the LLM walks the discover-then-activate flow:

list_mcp_server_tools(server) — returns the name: description listing for one server.
activate_tool(name) — promotes a tool from the catalogue into tools=[...] so the LLM can call it on the next round.

Both meta-tools are available in Plan and Execute. Sub-agents have a fixed parent-chosen surface and cannot discover or activate tools (activate_tool / list_mcp_server_tools are on the sub-agent denylist).

Roles with GitHub credentials in mcp_env.github get a per-role instance of the remote GitHub MCP server (declared as a shared: false http entry in mcp_servers), giving them the full GitHub toolset for reading/reviewing/tracking code (issues, PRs, repos, code search, actions); code authoring goes through the code sandbox. See GitHub Integration.

Agent Registry

The AgentPool serves as a registry of all agent instances:

Spawns one instance per agent seat (1:1 mapping; human seats are skipped)
Looks up agents by ID, email, or handle (for webhook routing)
Handles agent failures (restart with fresh instance, same identity)
Supports dynamic changes (add/remove agents at runtime via org hot-reload)

Since each agent is a unique individual, there is no load-balancing or role-based routing. Task assignment is a team lead decision, not an engine algorithm.

Execution Model

Agents are callback-driven — the Engine subscribes a handler per agent on the EventQueue. When messages arrive on an agent’s inbox topic (crewlet.agent.{handle}.inbox), the queue invokes the handler. No dedicated loop or polling.

Inbox delivery is batched per conversation (see Event System — Inbox batching): events that queued up while the agent was busy — or within the configured linger window — are drained together and partitioned by conversation key, so ten comments on one Jira issue or Slack thread reach the handler as ONE batch and trigger ONE digest turn instead of ten. The handler dispatches single-event partitions by event type — task assignments trigger a Turn Engine turn, A2A requests and notifications have their own handlers — and merges multi-event partitions into a single coalesced notification turn.

Concurrency

The engine uses cooperative async concurrency within a single process:

Queue handlers are async — when an agent awaits an LLM call, other handlers run
Multiple agents can be in the Working state simultaneously
ConcurrencyController limits max concurrent agent turns via a global semaphore, with optional per-role limits
Agents acquire a semaphore slot before the LLM loop, release when done

Event Loop
  │
  ├── Event 1 → triggers Agent A turn (awaits LLM call)
  ├── Event 2 → triggers Agent B turn (awaits LLM call)  ← runs while A waits
  ├── Agent A LLM response arrives → processes, emits events
  ├── Agent B LLM response arrives → processes, emits events
  └── ... continues

Graceful shutdown

SIGINT / SIGTERM trigger a pause-then-drain shutdown, designed so a restart picks up cleanly without a half-finished turn. The engine owns the process signals exclusively — exactly one handler per signal, registered via signal.signal() so it fires even when the event loop is blocked in synchronous code. The embedded API’s uvicorn server is constructed signal-free (its capture_signals is a no-op): without that, uvicorn’s serve() would steal SIGINT/SIGTERM, shut the dashboard down on the first Ctrl+C — exactly when the operator wants it alive to watch the drain — and re-raise the captured signals on exit, which the engine would count as phantom extra presses and escalate to a force-stop mid-drain.

pause_delivery() — no NEW handlers start. Publishes still work, so in-flight turns can emit TaskCompleted.
Stop work producers — deadline timers and the cron scheduler. Turns still parked at the concurrency gate are NAK’d back to the broker (redelivered next boot) instead of starting fresh LLM rounds mid-drain.
wait_for_handlers() — waits indefinitely; RUNNING turns finish their rounds until the counter hits 0 (drain_in_progress logs the in-flight count every 10 s).
Stop extensions / workers / agents — once no handler is running.
stop() — the Pulsar connection closes.
API server exits — the dashboard is served through the whole drain, and is brought down only after the engine has fully stopped.

Let LLMs finish their rounds — but only the running ones. The drain distinguishes two kinds of in-flight turn. Turns already past the ConcurrencyController gate (LLM rounds under way) run to completion. Turns that were delivered before the pause but are still waiting for a concurrency slot abort with ShutdownDraining — they haven’t called an LLM or fired a side effect yet, so the NAK’d trigger message simply redelivers to the next boot. Without this split, a backlog parked behind max_concurrent would run full multi-minute Plan → Execute → Review turns one after another during shutdown.

No engine-level timeout on the drain. Step 3 waits as long as in-flight turns need. We don’t try to second-guess “too long” — the host already provides that cutoff:

Interactive: a second Ctrl+C tells us you’re done waiting.
Kubernetes: terminationGracePeriodSeconds (default 30 s) — after which the kubelet sends SIGKILL.
systemd: TimeoutStopSec (default 90 s) — same SIGKILL fallback.

Embedding our own grace window would duplicate that decision in two places and inevitably disagree. Size the orchestrator’s grace period to cover your expected turn length (a multi-tool Plan → Execute → Review can comfortably take 2–5 minutes).

Force stop (second signal). A second SIGINT/SIGTERM during shutdown cancels all asyncio tasks; the cancellation propagates up through run()’s CancelledError branch into _force_stop(), which resets any still-WORKING agent to Idle, then closes the embedded API, MCP children, LLM clients, the event queue, and storage in ≤ 2 s per step. The cancelled turn’s Pulsar message is negatively acknowledged so the broker redelivers it to the next engine subscription promptly — without the NAK the message would sit unacknowledged until the ack_timeout window (10 minutes) elapsed, and the next engine would miss it for that window. A fresh turn runs from scratch on restart, and any side effects (Slack posts, Jira comments) already fired by the cancelled turn may duplicate. That’s the trade-off you opted into by sending the second signal.

Hard exit (third signal). A third SIGINT/SIGTERM calls os._exit(1) directly from the signal handler — no cleanup, no event loop required. This is the deterministic escape hatch for a process whose loop is wedged; before it existed, extra presses just re-cancelled all tasks and ripped CancelledError through the force-stop cleanup at arbitrary points.

Signal feedback is best-effort. Each press schedules its shutdown action on the event loop before printing the console notice, and the notice itself can never raise. This matters when output is piped: the terminal delivers Ctrl+C to the whole foreground process group, so with crewlet run 2>&1 | tee run.log the first press also kills tee, turning stdout/stderr into a broken pipe. An exception escaping a Python signal handler is re-raised inside whatever frame the main thread happened to be executing — it can silently kill an arbitrary task (the engine then runs on, looking wedged) or tear the event loop down around the live Pulsar client, whose C++ threads then abort the process at interpreter exit. Console output from the handler is therefore strictly optional; structured log writes already tolerate a dead stream. To keep watching the drain through a pipe, use tee -i (it ignores SIGINT and survives the press) — or watch the dashboard’s in-flight pill, which needs no console at all.

Watching the drain. The dashboard stays live through the entire drain (the embedded API server is stopped only after stop() completes). Its footer pill shows the engine’s in-flight handler count whenever it’s non-zero or the engine has flipped to “shutting down” — turns red during the drain so operators can watch it converge to 0. The count is also available programmatically:

engine.in_flight_count — Python property on the engine
engine.shutting_down — True from the first moment of stop() (unlike is_running, which only flips once teardown completes)
GET /health — JSON includes in_flight and shutting_down, and status reads "shutting_down" during the drain (embedded API only; the standalone API process omits these fields because it has no engine reference)

The console shows the same story: the first Ctrl+C prints what is being waited for and how to escalate, and the engine logs drain_in_progress with the in-flight count every 10 seconds until the drain converges (drain_complete).

Per-agent visibility is finer-grained: each working agent’s row carries current_phase (plan / execute / review) plus the iteration number, derived from AgentPhaseStarted events the turn engine emits at the top of each phase.

Generated from crewlet/crewlet v0.1.0 at b40ea18.