Files
onyx/tools/ods/internal/devstack/project_down_test.go
1720ffc203 feat(ods): list dev stacks machine-wide, stop orphans by project, and release unused port leases (#16285)
## Description

Adds `ods stack ls` to list every Onyx dev stack on the machine from any
directory, and lets `stop`/`down --project NAME` act on a stack whose
worktree is gone. The PR also releases port leases when a stack stops
owning them.

**`ods stack ls [--json] [--all] [--orphaned]`**
(`internal/devstack/inventory.go`, `cmd/stack_ls.go`)
- It merges four sources, keyed by Compose project: `docker ps -a` +
`docker inspect` (labels, state, health), `projects/*/owner.json`, every
reachable `state.json` (owner state dirs, plus
`<git-common-dir>/{ods-stack,ods-stack-shared,ods-stack-frontend,worktrees/*/ods-stack,worktrees/*/ods-stack-frontend}`
from owner, lease, Docker and state roots), and `Registry.Leases()`. It
does not call `devstack.Open`, and it takes no locks.
- An Onyx Compose project is one with `relational_db` + `cache`
services, or one that a state or owner record claims.
- Statuses: `active` (an app process is alive, checked with
`groupIsOurs`, so a reused PID does not count), `app-stopped` (infra
runs, phase `stopped`), `infra-only` (infra runs, no app process: a
Compose-only project or dead processes), `stopped`, `orphaned`, and
`stale-lease` (`--all` only).
- Orphan rules: the Compose working dir is gone and no state claims the
project; a state's root is gone; or the root exists but is no longer a
git worktree (state rows, or Docker rows with an owner record whose
state is gone). A row counts as orphaned only when something is left to
clean up: processes, containers, leases or an owner record.
- A missing or corrupt state degrades its row (`errors`). When Docker is
down, `ls` prints a warning and shows rows from saved state only. Exit
codes: 0; 3 when `--orphaned` finds an orphan; 1 when no source can be
read.

**`down`/`stop --project NAME` and `prune`**
(`internal/devstack/project_down.go`)
- The command resolves the stack through `owner.json` → `state_dir`,
then any state that claims the project, then Docker labels. It runs from
any directory.
- It takes the state lock (only if the state dir still exists, so git
metadata is never recreated) and the project lock. It stops recorded
groups with `stopServices`/`stopRecords` and runs `docker compose -p
NAME down` from a neutral dir. Volumes are kept.
- After a successful down it releases the stack's leases and removes
`owner.json` for an orphan.
- It refuses a stack with a live service under an existing git worktree
unless you pass `--force`. If `--project` already matches the current
checkout's stack, the existing checkout path runs.
- `ods stack prune` runs this for every orphaned row after a
confirmation (`--yes` skips it). `--volumes` has its own confirmation,
and `--yes` does not skip it.

**Port lease fixes** (`ports.go`, `lifecycle.go`, `legacy.go`)
- `Stack.Stop` releases app leases (everything except `*_HOST_PORT`).
`Stack.Down` releases app leases after `stopServices`, and every lease
after Compose down. New `Registry.ReleaseNames`.
- Shared adoption (`releaseLegacyLeases`, runs after the legacy guard
and before `leasePorts`): each stopped legacy candidate without live
process groups loses all its leases, or only its app leases while its
containers run.
- Reclaim: `Lease.StateDir` is now recorded (`LeaseFor`). Another
owner's lease is reclaimable when its state is `stopped` or missing,
none of its recorded groups is alive, and, for `*_HOST_PORT` only, its
Compose project has no running container. That last check is lazy, so
app-port leasing stays fast. Leases without `StateDir` are reclaimable
only for shared/frontend keys, which are their own state dirs.
- Drift down: `choosePort` moves a lease to the lowest free port from
`Preferred` when the service is not running. I added
`PortRequest.Running` because `Fixed` alone is not set for a running
service that does not hold its port yet, and moving such a service
changed `web dev`'s full-stack owner
(`TestWebRuntimeUsesTheFullStackWebOwner` caught this).

Docs: `ods stack` help, `tools/ods/README.md`,
`tools/ods/integration/README.md`, and the onyx-dev-stack skill (a new
"Find and stop old stacks" step, and the updated port bullet).

Known limit: `git worktree remove` deletes
`<common>/worktrees/<name>/ods-stack`, so the recorded PIDs are lost.
`down --project` still removes the containers, leases and owner record,
but it cannot find app processes that outlived that removal.

**Discovery-first CLI.** Bare `ods stack` prints a short intro, then
groups its subcommands with the same `AddGroup` pattern as the root
command: Find stacks (`status`, `ls`, `logs`, `plan`), Run this
worktree's stack (`up`, `start`, `restart`, `stop`, `down`), and Clean
up (`prune`). Each `ls` row carries `next`, the exact command that stops
it, and the text output logs it with `log.Info`. Empty results and
refusals name the command to run.

Example on the VM (an isolated stack, then its worktree deleted with `rm
-rf`):
```
$ ods stack ls
PROJECT           STATUS       APP  INFRA  WEB  AGE  ROOT
demo-wt-fd863404  app-stopped  -    7/7    -    1m   ~/demo-wt
$ rm -rf ~/demo-wt && ods stack ls --orphaned   # exit 3
PROJECT           STATUS    APP  INFRA  WEB  AGE  ROOT
demo-wt-fd863404  orphaned  -    7/7    -    1m   ~/demo-wt (missing)
level=info msg="To stop demo-wt-fd863404 (orphaned), run: ods stack down --project demo-wt-fd863404"
```
After `stack stop`, only the `*_HOST_PORT` leases remained. After `down
--project`, no containers or leases were left.

## How Has This Been Tested?

- Unit tests for the inventory merge (fake containers, owners, states,
leases, liveness): live, app-stopped, infra-only, Compose-only, orphan
with a missing root, a removed worktree whose directory remains, a stale
lease, Docker unavailable, unreadable sources, and sort order.
- Real-process tests for `DownProject`/`StopProject`: an orphan in a
deleted linked worktree is stopped, its leases are released and its
owner record is removed; a live worktree is refused without `--force`.
- Lease tests: reclaim of a stopped owner, live process group skipped,
non-stopped owner skipped, `*_HOST_PORT` with and without running
containers, drift down vs. running, `ReleaseNames`, stop/down releases,
adoption with and without a live legacy group, and the frontend handoff
releasing preview leases.
- Coverage gates: `cmd` is at 90.6% against a 90.4% floor, and
`internal/docker` at 99.3% against 99.2%.
- `go test ./internal/... ./cmd/` passes, except `TestRunCI_failures`,
which also fails on a clean main in this environment. golangci-lint
passes.
- New integration test `TestStackLiveOrphanedWorktree` (`-tags
integration`) passed on the VM (246 s). It starts an isolated stack in a
disposable worktree, deletes the worktree, asserts that `ls --orphaned`
lists the stack with exit 3, then that `down --project` stops its
processes, removes its containers and releases its leases.

## Additional Options

- [ ] [Optional] Please cherry-pick this PR to the latest release
version.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added `ods stack ls` to view active, stopped, orphaned, and
stale-lease stacks with service, health, port, age, and location
details.
* Added JSON output, filtering, orphan detection, and project-based
shutdown support.
* Added `ods stack prune` to safely clean up orphaned stacks, with
optional volume removal.
* **Bug Fixes**
  * Improved port lease reclamation and reassignment for stopped stacks.
* `stop` now releases application ports while preserving infrastructure
ports; `down` releases all ports.
* **Documentation**
  * Expanded lifecycle, recovery, listing, and cleanup guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: roshan <rohod04@gmail.com>
Co-authored-by: roshan <38771624+rohoswagger@users.noreply.github.com>
2026-10-20 00:22:08 +00:00

195 lines
7.4 KiB
Go

//go:build !windows
package devstack
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/onyx-dot-app/onyx/tools/ods/internal/docker"
)
type fakeCompose struct {
projects []string
}
func (c *fakeCompose) down(_ context.Context, project string, _ bool) error {
c.projects = append(c.projects, project)
return nil
}
func discoverWith(containers []docker.ComposeContainer) DiscoverOptions {
return DiscoverOptions{ListContainers: func() ([]docker.ComposeContainer, error) { return containers, nil }}
}
// isolatedLinkedStack starts an isolated stack in a linked worktree.
func isolatedLinkedStack(t *testing.T, project string) (*Stack, ServiceRecord) {
t.Helper()
a, b := sharedPair(t)
stack, err := Open(b.Root, project, Isolated)
if err != nil {
t.Fatal(err)
}
stack.Registry = a.Registry
t.Cleanup(func() { _ = stack.stopServices(testOptions().StopTimeout) })
mustUp(t, stack, &fakeProvider{services: []string{"api"}})
return stack, servicePID(t, stack, "api")
}
func TestDownProjectStopsOrphanedStackAndReleasesLeases(t *testing.T) {
stack, api := isolatedLinkedStack(t, "orphan-data")
containers := onyxContainers(stack.Project, stack.Root, true, time.Now())
if err := os.RemoveAll(stack.Root); err != nil {
t.Fatal(err)
}
inv := Discover(stack.Registry, discoverWith(containers))
row := rowFor(t, inv, stack.Project)
if row.Status != StatusOrphaned || row.StateDir != stack.StateDir || row.LeaseRoot != stack.Root {
t.Fatalf("deleted worktree is not an orphan: %+v", row)
}
compose := &fakeCompose{}
result, err := DownProject(context.Background(), stack.Registry, stack.Project, compose.down,
ProjectOptions{StopTimeout: testOptions().StopTimeout, LockWait: time.Second, Discover: discoverWith(containers)})
if err != nil {
t.Fatal(err)
}
if isCurrent(api) || result.StoppedServices != 1 {
t.Fatalf("orphaned service still runs: %+v", result)
}
if len(compose.projects) != 1 || compose.projects[0] != stack.Project {
t.Fatalf("compose down calls = %v", compose.projects)
}
if got := leaseNames(t, stack.Registry, stack.Root); len(got) != 0 || result.ReleasedLeases == 0 {
t.Fatalf("orphan leases were not released: %v %+v", got, result)
}
if _, err := os.Stat(ownerPath(stack.Registry, stack.Project)); !os.IsNotExist(err) || !result.RemovedOwner {
t.Fatalf("orphan owner record remains: %v", err)
}
}
func TestDownProjectRefusesLiveWorktreeWithoutForce(t *testing.T) {
stack, api := isolatedLinkedStack(t, "live-data")
compose := &fakeCompose{}
opts := ProjectOptions{StopTimeout: testOptions().StopTimeout, LockWait: time.Second, Discover: discoverWith(nil)}
if _, err := DownProject(context.Background(), stack.Registry, stack.Project, compose.down, opts); err == nil || !strings.Contains(err.Error(), "--force") {
t.Fatalf("down --project stopped a live worktree's stack: %v", err)
}
if !isCurrent(api) || len(compose.projects) != 0 {
t.Fatal("refused down changed the stack")
}
opts.Force = true
if _, err := DownProject(context.Background(), stack.Registry, stack.Project, compose.down, opts); err != nil {
t.Fatal(err)
}
if isCurrent(api) || len(compose.projects) != 1 {
t.Fatal("forced down did not stop the stack")
}
if _, err := os.Stat(ownerPath(stack.Registry, stack.Project)); err != nil {
t.Fatalf("down of a live worktree's stack removed its owner record: %v", err)
}
}
func TestStopProjectKeepsInfrastructure(t *testing.T) {
stack, api := isolatedLinkedStack(t, "stop-data")
if err := os.RemoveAll(stack.Root); err != nil {
t.Fatal(err)
}
if _, err := StopProject(context.Background(), stack.Registry, stack.Project, ProjectOptions{StopTimeout: testOptions().StopTimeout, LockWait: time.Second, Discover: discoverWith(nil)}); err != nil {
t.Fatal(err)
}
if isCurrent(api) {
t.Fatal("stop --project left the service running")
}
if _, err := os.Stat(filepath.Join(stack.StateDir, stateFileName)); err != nil {
t.Fatal(err)
}
}
func TestFindProjectRejectsUnknownProject(t *testing.T) {
if _, err := FindProject(&Registry{Dir: t.TempDir()}, "nothing-here", discoverWith(nil)); err == nil {
t.Fatal("found a stack that does not exist")
}
}
func TestCleanupRowKeepsContainersOfAnotherOwner(t *testing.T) {
stack, _ := isolatedLinkedStack(t, "shared-data")
secondary := StackRow{Project: stack.Project, Mode: Isolated, StateDir: t.TempDir(), Status: StatusOrphaned}
compose := &fakeCompose{}
result, err := CleanupRow(context.Background(), stack.Registry, secondary, compose.down, ProjectOptions{LockWait: time.Second})
if err != nil {
t.Fatal(err)
}
if len(compose.projects) != 0 || result.ComposeDown {
t.Fatal("a secondary state stopped the owner's containers")
}
if _, err := os.Stat(ownerPath(stack.Registry, stack.Project)); err != nil {
t.Fatalf("a secondary state removed the owner record: %v", err)
}
}
func TestStopProjectReleasesApplicationLeases(t *testing.T) {
stack, _ := isolatedLinkedStack(t, "stop-leases")
if err := os.RemoveAll(stack.Root); err != nil {
t.Fatal(err)
}
if _, err := StopProject(context.Background(), stack.Registry, stack.Project, ProjectOptions{StopTimeout: testOptions().StopTimeout, LockWait: time.Second, Discover: discoverWith(nil)}); err != nil {
t.Fatal(err)
}
if got := leaseNames(t, stack.Registry, stack.Root); got["api"] != 0 {
t.Fatalf("stop --project kept application leases: %v", got)
}
}
func TestDownProjectRemovesOwnerRecordWithoutState(t *testing.T) {
registry := &Registry{Dir: t.TempDir()}
path := ownerPath(registry, "old-data")
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(`{"state_dir":"/missing/ods-stack"}`), 0o644); err != nil {
t.Fatal(err)
}
compose := &fakeCompose{}
if _, err := DownProject(context.Background(), registry, "old-data", compose.down, ProjectOptions{LockWait: time.Second, Discover: discoverWith(nil)}); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Fatalf("owner record remains: %v", err)
}
}
func TestCleanupRowRefusesWhenTheOwnerStateIsUnreadable(t *testing.T) {
stack, _ := isolatedLinkedStack(t, "corrupt-owner")
if err := os.WriteFile(filepath.Join(stack.StateDir, stateFileName), []byte("not json"), 0o644); err != nil {
t.Fatal(err)
}
secondary := StackRow{Project: stack.Project, Mode: Isolated, StateDir: t.TempDir(), Status: StatusOrphaned}
compose := &fakeCompose{}
if _, err := CleanupRow(context.Background(), stack.Registry, secondary, compose.down, ProjectOptions{LockWait: time.Second}); err == nil || !strings.Contains(err.Error(), "cannot be read") {
t.Fatalf("err = %v", err)
}
if len(compose.projects) != 0 {
t.Fatal("cleanup stopped containers of an unreadable owner")
}
if _, err := os.Stat(ownerPath(stack.Registry, stack.Project)); err != nil {
t.Fatalf("owner record removed: %v", err)
}
}
func TestCleanupRowRefusesWhenTheOwnerRecordIsUnreadable(t *testing.T) {
stack, _ := isolatedLinkedStack(t, "corrupt-record")
if err := os.WriteFile(ownerPath(stack.Registry, stack.Project), []byte("not json"), 0o644); err != nil {
t.Fatal(err)
}
secondary := StackRow{Project: stack.Project, Mode: Isolated, StateDir: t.TempDir(), Status: StatusOrphaned}
compose := &fakeCompose{}
if _, err := CleanupRow(context.Background(), stack.Registry, secondary, compose.down, ProjectOptions{LockWait: time.Second}); err == nil || len(compose.projects) != 0 {
t.Fatalf("err = %v, compose = %v", err, compose.projects)
}
}