docs: state control boundaries and add a self-hosting page (#579)

Reword the local-model claim in providers.md to the models behind runnable examples, add framework-versus-integrator boundaries to durable-approvals.md, external-agents.md, and run-journal.md, and add docs/self-hosting.md covering runtime footprint, network scope, state locations, redaction, and air-gapped deployment.

AGENTS.md

@@ -104,6 +104,7 @@ These constraints span multiple files and can cause behavioral or compatibility
 | Evaluation, scorers, stores, reports, sampling, and gates | [docs/evaluation.md](docs/evaluation.md) |
 | CLI commands and JSON schemas | [docs/cli.md](docs/cli.md) |
 | Process and ACP backends | [docs/external-agents.md](docs/external-agents.md) |
+| Runtime footprint, network scope, state locations, and air-gapped deployment | [docs/self-hosting.md](docs/self-hosting.md) |
 | Routing, scheduling, consensus, recovery, and replay | [model-routing](docs/model-routing.md), [execution-routing](docs/execution-routing.md), [task-scheduling](docs/task-scheduling.md), [consensus](docs/consensus.md), [adaptive-recovery](docs/adaptive-recovery.md), [plan-replay](docs/plan-replay.md) |
 
 ## Adding an LLM adapter

docs/durable-approvals.md

@@ -161,6 +161,48 @@ exact. Use access controls and encryption appropriate for that data.
 is lossy and would change the content hash; it remains available for ordinary
 checkpoint and shared-memory redaction.
 
+## What the framework provides, and what you build
+
+The framework's side of an approval is the execution mechanism and the durable
+record: the gates that can return `{ action: 'suspend' }`, the primary approval
+row keyed under `__oma_approval__/<requestId>`, the exact-content binding, the
+`getApprovalRecord` / `decideApproval` / `DurableApprovalLedger` reviewer
+helpers, and `OpenMultiAgent.restore()`. That is the whole surface.
+
+There is deliberately no reviewer product on top of it:
+
+- **No approval UI.** Nothing in the package renders a request or collects a
+  decision.
+- **No CLI command.** The `oma` CLI is a non-interactive wrapper. Its commands
+  are `run`, `task`, `dashboard`, `eval run`, `eval gate`, `provider`, and
+  `help`; there is no `approve` or `reject`.
+- **No transport.** OMA never notifies a reviewer, routes a request to a queue,
+  or exposes an HTTP endpoint. A suspended run returns `pendingApprovals` to
+  the caller and stops.
+- **The dashboard is read-only about approvals.** The Run Viewer maps the
+  `oma.approval.approved` and `oma.plan.approved` trace facts to display labels
+  and renders them alongside the rest of the run. It is a post-run HTML
+  artifact with no write path back into the ledger.
+
+An integrator therefore supplies three things:
+
+1. **A store.** A `MemoryStore` whose `compareAndSet` is atomic across every
+   writer that will decide, and that is reachable from both the suspended
+   process and the reviewer. Suspension fails closed with
+   `APPROVAL_ATOMIC_STORE_REQUIRED` when the store cannot decide atomically.
+   See [Store requirements](#store-requirements) for what the bundled stores
+   do and do not give you.
+2. **A review surface.** Something that shows the reviewer `request.content`
+   and `request.requestHash`, and collects back a decision plus a reviewer
+   identity. The hash is the part that matters: a reviewer who submits a hash
+   they did not inspect defeats the exact-content binding, and OMA cannot tell
+   the difference. Authentication, authorization, and the audit trail of who
+   was shown what are entirely yours.
+3. **A resume path.** A process that calls `decideApproval(store, { requestId,
+   requestHash, decision, reviewer })` and then rebuilds the team wiring and
+   calls `restore(team, { checkpoint: { store } })`. The two can be the same
+   process or different ones, subject to the store's cross-writer guarantees.
+
 ## Explicit limits
 
 - Tool-call suspension is supported only inside checkpointed built-in LLM

docs/external-agents.md

@@ -205,6 +205,49 @@ at the run boundary, so an ACP usage delta can exhaust the remaining run budget.
 The process backend continues to contribute `{0, 0}` because it has no token
 signal.
 
+## Control boundary
+
+An external agent is a first-class team member at the orchestration layer and
+an opaque subprocess below it. The split is structural rather than a policy
+choice: when `AgentConfig.backend` is set, the agent resolves to the external
+backend and never constructs an `AgentRunner`, and the runner is where the tool
+registry, the tool executor, the filesystem sandbox, the context strategy, and
+the per-turn journal emission all live. Everything the orchestrator does around
+a task still happens; everything the runner does inside one does not.
+
+**Still applies.** These are enforced above the backend and are
+backend-agnostic:
+
+| Control | Where it acts |
+|---|---|
+| Task DAG placement, dependency ordering, and failure cascade to dependents | Task queue and scheduler; a failed external task skips its dependents like any other |
+| `onPlanReady`, `onApproval`, and `onTaskDispatch` gates | Orchestration level, so the assignee's backend kind is irrelevant to them. `onTaskDispatch` in particular runs immediately before dispatch and before the backend is resolved, so rejecting it means the subprocess is never spawned |
+| Token budget | The backend's reported usage aggregates into the run and is checked against `maxTokenBudget`. Read the [accounting caveat](#acp-token-accounting-caveat) first: an ACP agent that emits no `usage_update`, and every process-backend agent, contributes `{0, 0}` and is therefore not budget-gated in practice |
+| Shared memory | The orchestrator writes the completed task's output to shared memory under `task:<id>:result` after the task finishes |
+| Run and task journal events | `run/start`, `run/end`, `plan/set`, `task/status`, `memory/set`, `approval/request`, `approval/decision`, and `checkpoint/saved` are emitted by the orchestrator and task-execution layers |
+| Abort propagation | `RunOptions.abortSignal` reaches both backends. The process backend kills the child's process tree. The ACP backend sends `session/cancel` and relies on the agent to stop, so cancellation there is cooperative and the subprocess stays alive for the next turn |
+| Process stderr redaction | A non-zero exit or signal builds its error message through the shared credential redactor, so stderr is scrubbed before it reaches the task result |
+
+**Does not apply.** None of these reach an external backend, and no
+configuration makes them:
+
+| Control | Why not |
+|---|---|
+| `onToolCall` per-call gate | The gate is evaluated by the tool executor inside `AgentRunner`. An external agent's tool calls never pass through it. ACP `tool_call` updates still populate `result.toolCalls`, but that is reporting, not a gate |
+| Filesystem sandbox | `AgentConfig.cwd` scopes the built-in filesystem tools through that same runner. A backend's `cwd` is a plain working directory the subprocess reads and writes directly, with your process's permissions |
+| `egressPolicy` | Scoped to framework-owned LLM requests. External backends and tool code are outside it by design, and the child owns its own network behavior. See the [egress enforcement matrix](egress-policy.md#enforcement-matrix) |
+| Tool-level journal events | `turn/start`, `turn/end`, `user/message`, `assistant/message`, `llm/request`, `tool/call`, `tool/result`, and `context/replace` are emitted only by the runner. Neither backend reads the journal recorder it is handed, so a journaled run records that an external task ran, not what happened inside it |
+| Mid-task checkpoints | Neither backend calls the runner checkpoint hook, so an external task checkpoints at task boundaries only. A `suspend` tool decision fails closed there; see [durable approvals](durable-approvals.md#explicit-limits) |
+
+**ACP permissions default to auto-approve.** `permission` defaults to
+`'auto-approve'`, so unless the application sets it, every permission prompt
+the agent raises is answered yes. OMA picks the least-privilege option offered
+(`allow_once` before `allow_always`), which bounds the blast radius of one
+decision but does not change the default answer. `'reject'` or a callback is
+the only way to make a permission prompt a real gate. The process backend has
+no protocol-level permission prompts at all, so the configured `command`,
+`args`, `env`, and `cwd` are the entire control surface.
+
 ## Programmatic API
 
 Most users only touch `backend`. To construct a backend directly, import from the

docs/providers.md

@@ -173,7 +173,9 @@ Reasoning is streamed as `reasoning` events. Preserving reasoning across a provi
 
 The framework supports tool-calling with local models served by Ollama, vLLM, LM Studio, or llama.cpp. Tool-calling is handled natively through the OpenAI-compatible API.
 
-Verified local models include Gemma 4, Llama 3.1, Qwen 3, Mistral, and Phi-4. Ollama publishes its tool-capable models at [ollama.com/search?c=tools](https://ollama.com/search?c=tools).
+The local models exercised so far are the ones behind the runnable examples in this repository: Gemma 4 on Ollama ([`providers/gemma4-local`](../packages/core/examples/providers/gemma4-local.ts)), Llama 3.1 on Ollama ([`providers/ollama`](../packages/core/examples/providers/ollama.ts)), and a quantized Qwen2.5 on vLLM or llama-server ([`providers/local-quantized`](../packages/core/examples/providers/local-quantized.ts)). Those examples are run by hand against a local server you start yourself.
+
+The automated coverage underneath them is narrower: `packages/core/tests/text-tool-extractor.test.ts` unit-tests the text tool-call extractor against the formats it recognizes, including bare JSON, fenced code blocks, and Hermes-style `<tool_call>` tags. It pins those output shapes, not any particular model. **No local model is covered by the end-to-end suite.** `packages/core/tests/e2e/` contains only hosted-provider cases; the whole directory is excluded from `npm test` unless `RUN_E2E` is set, and each suite then skips itself without its provider's credentials. Treat any other local model, and any other quantization or serving stack, as unverified here and check it against your own workload. Ollama publishes its tool-capable models at [ollama.com/search?c=tools](https://ollama.com/search?c=tools).
 
 If a local model returns tool calls as text instead of the `tool_calls` wire format, the framework automatically extracts them from the text output. This helps with thinking models or misconfigured local servers.
 

docs/run-journal.md

@@ -254,6 +254,27 @@ new OpenMultiAgent({
 
 This holds at approval boundaries too, where ordinary checkpoint saves escalate. Durability there is the [durable approval ledger](durable-approvals.md)'s job; the journal only records that the boundary existed. Losing the audit trail must never roll back a run that actually happened.
 
+## Identity and integrity
+
+Two properties an audit reader tends to assume are not there, and both are worth stating before someone builds a compliance story on this file.
+
+**Journal events identify agents, not people.** Every event carries `agentName` where one applies, alongside `runId`, `attempt`, and `taskId`. There is no operator, user, actor, or principal field on any event type, and nothing populates one. The journal answers which agent produced a block, never which person authorized it.
+
+Human identity enters the record in exactly two places, both downstream of a [durable approval](durable-approvals.md):
+
+- **`approval/decision` events**, which carry the `ApprovalDecisionRecord` verbatim, including its `reviewer.id` and optional `reviewer.displayName`. That identity is whatever the application passed to `decideApproval`; OMA validates that `id` is a non-empty string and does not authenticate it.
+- **The execution receipt**, whose approval summary copies the same values as `reviewerId` and `reviewerDisplayName`.
+
+An unapproved run therefore has no human name in it at all, which is the correct reading: nobody was asked.
+
+**`verifyRun()` detects drift, not tampering.** It checks three things: sequence monotonicity, referential integrity of `sourceEventSeqs`, and per-block content hashes against the events a request names. What it does not check is the file's provenance:
+
+- **No hash chain.** No event carries a digest of its predecessor. Hashes cover one content block each and are recorded on the citing `llm/request`, so rewriting an event in place, or dropping one and renumbering what follows, leaves nothing structurally inconsistent to find; only a raw reorder or a duplicated `seq` trips the sequence check.
+- **No signature.** Nothing in the journal is signed. The subsystem uses SHA-256 for content identity only and holds no key material.
+- **No WORM storage.** `JsonlRunJournal` appends to an ordinary file with no lock, which anything with write access can rewrite afterwards. `InMemoryRunJournal` is a ring buffer in process memory and is not durable at all. Neither backend makes a written event immutable.
+
+A writer that can edit the journal can edit an event and recompute the `contentHash` that cites it in the same pass, and `verifyRun()` will return `ok`. This is the same trust boundary [durable approvals](durable-approvals.md#exact-content-binding) draws: integrity checking across OMA's own records, not a cryptographic claim against whoever controls the storage. If you need tamper-evidence, put the journal on storage that provides it, and treat `verifyRun()` as a check on what OMA wrote rather than on what is on disk now.
+
 ## Journal versus telemetry
 
 [Trace records](observability.md) and journal events describe the same run and deliberately do not depend on each other. Traces are **telemetry**: losing them must never roll back durable state, and they may be sampled, batched, exported, or dropped. Journal events are **execution state**: they record what the run did and what the model saw. The `journal/` module does not import from `observability/`, so trace loss cannot imply journal loss and neither can the reverse. Events carry `traceId`/`spanId` when a trace runtime is active, which is enough to join the two streams without coupling them.

docs/self-hosting.md

@@ -0,0 +1,204 @@
+# Self-hosting and data residency
+
+`@open-multi-agent/core` is a library, not a service. There is no OMA backend,
+no account, and no hosted control plane: the framework runs entirely inside
+your Node.js process, and everything it persists is written through a store you
+supply. This page collects the facts a review needs in one place. Each claim is
+about the framework itself; where your own configuration decides the answer,
+that is said explicitly.
+
+## Runtime footprint
+
+Core's runtime dependencies are:
+
+| Package | Purpose |
+|---|---|
+| `@anthropic-ai/sdk` | Anthropic provider adapter |
+| `openai` | OpenAI adapter and every OpenAI-compatible provider, including local servers |
+| `zod` | Tool input schemas and structured output validation |
+
+Everything else is a peer dependency and optional: the ACP SDK, the MCP SDK,
+the Google GenAI SDK, the AWS Bedrock client, and the Vercel AI SDK. Each is
+loaded through a dynamic `import()` at the moment a feature needs it, so an
+install that never uses ACP never resolves the ACP SDK. Provider adapter
+modules are dynamically imported too, so the two bundled provider SDKs load
+only when an adapter that needs them is constructed.
+
+The supported runtime is Node.js 20 or newer. OpenTelemetry lives in the
+separate `@open-multi-agent/otel` package and is never pulled in by the core
+import.
+
+## What connects to the network
+
+The framework opens outbound HTTP for one purpose: talking to the LLM endpoint
+your configuration names. `AgentConfig.baseURL` and the provider environment
+variables decide where that is. Point them at a local server and the framework
+has no remaining reason to leave the machine. The one provider that reaches
+past its own model API is GitHub Copilot, whose adapter also calls
+`api.github.com` for a token and, for interactive device login, `github.com`.
+
+There is **no telemetry, no analytics, no license check, no update check, and
+no phone-home of any kind** in the package. "Telemetry" inside this codebase
+means the local [observability](observability.md) subsystem: trace records that
+go to a sink you construct and a store you own. Nothing is transmitted unless
+you wire an exporter yourself, and the OpenTelemetry adapter that would do so
+is a separate optional package with its own lifecycle.
+
+Four things start subprocesses rather than connections, and each is something
+you configured:
+
+- **MCP servers** connect over stdio only. The MCP client uses
+  `StdioClientTransport`, so a server is a child process on the same machine.
+  There is no HTTP or SSE MCP transport in core.
+- **`process` and `acp` backends** spawn the command you name. See
+  [external agents](external-agents.md).
+- **The `bash` tool** runs commands through a local shell.
+- **Process-tree cleanup** shells out to `taskkill` on Windows to terminate a
+  child's descendants.
+
+Each of those children has your process's permissions and its own network
+access. The framework starts them and exchanges bytes; it does not contain
+them.
+
+## Bounding LLM egress
+
+`egressPolicy` narrows where a framework-owned LLM request may go. Two modes:
+
+- `offline` permits only loopback origins: `localhost`, `*.localhost`,
+  `127.0.0.0/8`, and `[::1]`.
+- `allowlist` permits only the exact HTTP(S) origins you list.
+
+Policies configured at the orchestrator, run, and agent levels are
+intersected, so a narrower scope can tighten but never widen a broader one.
+Enforcement means the origin is checked before the provider SDK loads, a
+guarded fetch re-checks every request, and redirects are rejected. An adapter
+OMA cannot enforce fails closed rather than proceeding unguarded.
+
+**Read the scope statement carefully.** The policy covers LLM requests the
+framework itself issues. It does not cover tool code, the `bash` tool, MCP
+server internals, process and ACP children, application callbacks, or your own
+trace exporters. `offline` is a statement about OMA's provider calls, not
+evidence that the Node.js process and its descendants are offline. The full
+per-surface table is the
+[egress enforcement matrix](egress-policy.md#enforcement-matrix).
+
+## Where state lives
+
+Nothing is persisted anywhere you did not choose. Shared memory, checkpoints,
+and durable approvals all go through the `MemoryStore` interface, and core
+ships three implementations:
+
+| Store | Durability | Notes |
+|---|---|---|
+| `InMemoryStore` | None; dies with the process | Process-local `compareAndSet` |
+| `FileStore` | One JSON file, rewritten atomically per mutation | Single Node process at a time; no cross-process file lock |
+| `RedactingStore` | Delegates to whatever it wraps | Scrubs values on write; deliberately exposes no `compareAndSet`, so it cannot back durable approvals. See [redaction](#redaction) |
+
+**There is no bundled database store.** Redis, Postgres, and SQLite appear in
+the source only as examples of what an application might implement behind
+`MemoryStore`. Writing one is the supported path when you outgrow a file.
+
+**Cross-process atomicity is not provided.** `FileStore.compareAndSet` is
+atomic among callers sharing one `FileStore` instance, and that is the whole
+guarantee. A cross-process reviewer should therefore decide only after the
+suspended runner has exited, or the deployment needs a database-backed store
+whose `compareAndSet` is atomic across all writers. This matters most for
+[durable approvals](durable-approvals.md#store-requirements), where the
+decision write is the correctness boundary.
+
+Trace records and evaluation records have their own stores and the same shape
+of choice: `InMemoryTraceStore` or `FileTraceStore`, `InMemoryEvalStore` or
+`FileEvalStore`. Both file-backed variants are append-only local files.
+
+## Filesystem scope
+
+Built-in filesystem tools resolve every path, symlinks included, inside a
+sandbox root. Unset, that root is `<process.cwd()>/.agent-workspace`, created
+on first write. `OrchestratorConfig.defaultCwd` or `AgentConfig.cwd` moves it,
+`process.cwd()` widens it to the whole project, and `null` disables it.
+
+**The `bash` tool is not sandboxed.** It takes a `cwd` argument straight from
+the model and runs the command through a shell with your process's
+permissions. The sandbox root does not constrain it, and neither does the
+per-call gate unless you configure one. Granting `bash` is granting shell
+access to the host; see [tool configuration](tool-configuration.md) for the
+grant model and the per-call `onToolCall` gate.
+
+## Redaction
+
+A shared credential redactor runs at several surfaces before content is
+persisted or exported:
+
+- trace attributes and the tool input/output an agent run records
+- observability status messages and record processors
+- task metadata
+- `bash` tool output
+- `process` backend stderr on a failed exit
+- evaluation sampling and report payloads
+- `RedactingStore` values, and `JsonlRunJournal` events when `redact` is set
+
+Its built-in patterns are credential-shaped: key/secret/token/password-style
+field names, `Authorization` and `Bearer` headers, PEM private key blocks, and
+literal token formats such as `sk-`, `ghp_`, `AKIA`, and `xox`-prefixed keys.
+
+**PII is not covered by default.** Emails, national IDs, account numbers, and
+anything else domain-specific pass through unchanged unless you supply the
+patterns yourself. The two surfaces that accept them are `RedactingStore` and
+`JsonlRunJournal`, both through the `patterns` option (the underlying helper
+calls the same list `extraPatterns`); the other surfaces above run the built-in
+patterns only and take no caller additions. Redaction is also best-effort by
+construction: it is a set of regular expressions over text, not a classifier.
+Do not rely on it as the only control over what an agent may emit.
+
+One boundary is easy to miss: telemetry redaction does not reach persisted run
+state. Shared-memory writes and checkpoint saves store agent output verbatim
+unless the store itself is wrapped in `RedactingStore`. See
+[shared memory](shared-memory.md#redacting-persisted-secrets).
+
+## Air-gapped deployment
+
+Two pieces of configuration cover the framework's own behavior:
+
+1. **An OpenAI-compatible endpoint.** Configure it as `provider: 'openai'` with
+   a `baseURL` pointing at the server, plus a non-empty placeholder `apiKey`
+   because the OpenAI SDK validates that the field is set even when the server
+   ignores it. See
+   [local model tool-calling](providers.md#local-model-tool-calling).
+2. **An egress policy**, so a stale `baseURL` or an inherited environment
+   variable fails closed instead of reaching a hosted provider. Use
+   `{ mode: 'offline' }` when the endpoint is on the same host: it permits
+   loopback only, and deliberately rejects private LAN addresses. A server
+   elsewhere on the network needs `{ mode: 'allowlist', allowedOrigins:
+   ['http://<host>:<port>'] }` instead, because `offline` would block it.
+
+Everything else in the boundary is outside what the framework can enforce: an
+MCP server that calls out, a `bash`-granted agent, an external backend whose
+child has network access, or a trace exporter pointed off the machine. The
+framework will not open a connection on its own, but it also cannot contain the
+code it starts on your behalf. A network namespace, an egress proxy, or a host
+firewall is the control that actually does.
+
+## What is not provided
+
+So that nothing here is read as more than it is, the framework does **not**
+include:
+
+- **Multi-tenancy.** There is no tenant concept. Isolating one customer's runs,
+  stores, and workspaces from another's is an application concern.
+- **RBAC or any authorization model.** No roles, no permissions, no policy
+  engine. The tool grant model and `onToolCall` gate are per-agent
+  configuration, not an identity system.
+- **SSO or authentication.** OMA never authenticates anyone. A durable approval
+  records the `reviewer.id` string you pass it and validates only that it is
+  non-empty.
+- **An approval UI or workflow.** The suspend/decide/resume API and the durable
+  record exist; the reviewer-facing product does not. See
+  [durable approvals](durable-approvals.md#what-the-framework-provides-and-what-you-build).
+- **Tamper-evident audit.** The run journal has no hash chain, no signature,
+  and no WORM storage. See
+  [identity and integrity](run-journal.md#identity-and-integrity).
+- **A database-backed store.** Only in-memory and single-file implementations
+  ship. Anything with cross-process guarantees is yours to write.
+- **Encryption at rest.** File-backed stores write plain JSON and
+  newline-delimited JSON. Use disk or filesystem encryption appropriate to the
+  data those files will hold.