mirror of
https://github.com/JackChen-me/open-multi-agent.git
synced 2026-10-19 01:29:57 -05:00
Jack Chen
e36cd1f4c0
A checkpoint records what a run can resume from, not who is allowed to advance it. Two processes can restore the same snapshot and both continue it, and a worker that stalled past its own liveness can wake up and overwrite the state a replacement has since written. Add an opt-in `runStore`: one authoritative, versioned record per logical run holding its lifecycle status, execution lease, and a monotonic fencing token. - `RunRecord` keeps `version` for optimistic concurrency and `fencingToken` for ownership, so a worker holding an old token stays detectable across any number of unrelated writes. Legal transitions are enumerated and terminal statuses have no outgoing edge. - `RunStore` is three methods plus an explicit `atomicity` declaration. Lease expiry, fencing, transition legality, and command idempotency live above the seam in `RunLedger`, so a Postgres or Redis backend reproduces storage semantics rather than a state machine. - `MemoryStoreRunStore` adapts any `MemoryStore` with compare-and-set and defaults to `atomicity: 'process'`. `FileStore` remains the sequential restart reference store and is not presented as a lease backend. - The orchestrator acquires the lease before it dispatches, fences every checkpoint write with its token, stops the run at the dispatch gate when the lease is lost, and records the terminal transition. A worker that lost its lease writes no terminal status and never reports success. - `restore()` is the resume command: it makes a suspended record eligible and takes ownership before reconciling the approval ledger. Ownership and lifecycle writes fail closed rather than inheriting the best-effort semantics of checkpoint and telemetry writes. With no run store configured, existing single-process checkpoint and restore behavior is unchanged. Ships a reusable `RunStore` conformance suite covering create races, version-matched compare-and-set, lease expiry, takeover, stale-writer rejection, and terminal closure, so an out-of-repository backend can prove the same contract. Closes #481
Internal engineering records
This directory holds engineering records kept for auditability: release-readiness reviews, benchmark snapshots, and similar evidence gathered at a specific point in time on specific hardware.
These are not user documentation. They describe one measurement or one review, against the commit and machine named inside them, and they are not maintained as the code moves on. Read them as history, not as current behavior or as a performance promise.
For current behavior, use the user documentation in ../, starting with
observability.md.