Files
roshan 92d5149cb5 fix(craft): preserve runtime files after cleanup checkpoint failures (#16398)
## Description

Idle cleanup preserves recoverable files after checkpoint or workspace
discovery failures, including on unhealthy endpoints. Kubernetes
requires complete workspace-list output. Docker rejects failed or
incomplete snapshot probes.

Migration `45662b7b63a2` adds nullable `sandbox.cleanup_checkpoint`
JSONB storage. Cleanup commits proof after snapshots and final checks,
before deletion. The proof update requires the current provisioning
attempt and observed heartbeat.

Retry requires a confirmed absent runtime and matching attempt,
workspace storage identity, and heartbeat. Heartbeats and file mutations
invalidate old proof before files change. Successful sleep and new
provisioning clear the proof.

## How Has This Been Tested?

614 sandbox tests passed with two pytest workers. Tests cover partial
Docker deletion across workers, changed storage, stale attempts, resumed
activity, and failed checkpoint commits. Migration upgrade and downgrade
SQL generation passed. Pre-commit checks passed.

PostgreSQL persistence and heartbeat-race tests were added. Their local
run was blocked because PostgreSQL was unavailable. Live Docker services
were also unavailable locally.

## Known CI blocker

Preview deployment failed with an unauthorized deploy-token error after
one infrastructure retry. See the [failed preview
run](https://github.com/onyx-dot-app/onyx/actions/runs/39660940780).

Code, Craft, database, and type checks passed. Other CI jobs are still
running.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Idle sandboxes remain running when workspace discovery or snapshot
checks fail, preserving their files, runtime, and ports.
* Interrupted cleanup can resume when the sandbox state is unchanged.
Existing workspace storage is preserved when cleanup cannot be verified.
* Uploads and file deletions are coordinated with session activity,
preventing cleanup from interfering with active file changes.
* Cleanup checkpoints are cleared when sandbox activity or lifecycle
changes make them outdated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-22 23:02:37 +00:00
..