Files
onyx/tools/ods/cmd/stack_ls.go
T
1720ffc203 feat(ods): list dev stacks machine-wide, stop orphans by project, and release unused port leases (#16285)
## Description

Adds `ods stack ls` to list every Onyx dev stack on the machine from any
directory, and lets `stop`/`down --project NAME` act on a stack whose
worktree is gone. The PR also releases port leases when a stack stops
owning them.

**`ods stack ls [--json] [--all] [--orphaned]`**
(`internal/devstack/inventory.go`, `cmd/stack_ls.go`)
- It merges four sources, keyed by Compose project: `docker ps -a` +
`docker inspect` (labels, state, health), `projects/*/owner.json`, every
reachable `state.json` (owner state dirs, plus
`<git-common-dir>/{ods-stack,ods-stack-shared,ods-stack-frontend,worktrees/*/ods-stack,worktrees/*/ods-stack-frontend}`
from owner, lease, Docker and state roots), and `Registry.Leases()`. It
does not call `devstack.Open`, and it takes no locks.
- An Onyx Compose project is one with `relational_db` + `cache`
services, or one that a state or owner record claims.
- Statuses: `active` (an app process is alive, checked with
`groupIsOurs`, so a reused PID does not count), `app-stopped` (infra
runs, phase `stopped`), `infra-only` (infra runs, no app process: a
Compose-only project or dead processes), `stopped`, `orphaned`, and
`stale-lease` (`--all` only).
- Orphan rules: the Compose working dir is gone and no state claims the
project; a state's root is gone; or the root exists but is no longer a
git worktree (state rows, or Docker rows with an owner record whose
state is gone). A row counts as orphaned only when something is left to
clean up: processes, containers, leases or an owner record.
- A missing or corrupt state degrades its row (`errors`). When Docker is
down, `ls` prints a warning and shows rows from saved state only. Exit
codes: 0; 3 when `--orphaned` finds an orphan; 1 when no source can be
read.

**`down`/`stop --project NAME` and `prune`**
(`internal/devstack/project_down.go`)
- The command resolves the stack through `owner.json` → `state_dir`,
then any state that claims the project, then Docker labels. It runs from
any directory.
- It takes the state lock (only if the state dir still exists, so git
metadata is never recreated) and the project lock. It stops recorded
groups with `stopServices`/`stopRecords` and runs `docker compose -p
NAME down` from a neutral dir. Volumes are kept.
- After a successful down it releases the stack's leases and removes
`owner.json` for an orphan.
- It refuses a stack with a live service under an existing git worktree
unless you pass `--force`. If `--project` already matches the current
checkout's stack, the existing checkout path runs.
- `ods stack prune` runs this for every orphaned row after a
confirmation (`--yes` skips it). `--volumes` has its own confirmation,
and `--yes` does not skip it.

**Port lease fixes** (`ports.go`, `lifecycle.go`, `legacy.go`)
- `Stack.Stop` releases app leases (everything except `*_HOST_PORT`).
`Stack.Down` releases app leases after `stopServices`, and every lease
after Compose down. New `Registry.ReleaseNames`.
- Shared adoption (`releaseLegacyLeases`, runs after the legacy guard
and before `leasePorts`): each stopped legacy candidate without live
process groups loses all its leases, or only its app leases while its
containers run.
- Reclaim: `Lease.StateDir` is now recorded (`LeaseFor`). Another
owner's lease is reclaimable when its state is `stopped` or missing,
none of its recorded groups is alive, and, for `*_HOST_PORT` only, its
Compose project has no running container. That last check is lazy, so
app-port leasing stays fast. Leases without `StateDir` are reclaimable
only for shared/frontend keys, which are their own state dirs.
- Drift down: `choosePort` moves a lease to the lowest free port from
`Preferred` when the service is not running. I added
`PortRequest.Running` because `Fixed` alone is not set for a running
service that does not hold its port yet, and moving such a service
changed `web dev`'s full-stack owner
(`TestWebRuntimeUsesTheFullStackWebOwner` caught this).

Docs: `ods stack` help, `tools/ods/README.md`,
`tools/ods/integration/README.md`, and the onyx-dev-stack skill (a new
"Find and stop old stacks" step, and the updated port bullet).

Known limit: `git worktree remove` deletes
`<common>/worktrees/<name>/ods-stack`, so the recorded PIDs are lost.
`down --project` still removes the containers, leases and owner record,
but it cannot find app processes that outlived that removal.

**Discovery-first CLI.** Bare `ods stack` prints a short intro, then
groups its subcommands with the same `AddGroup` pattern as the root
command: Find stacks (`status`, `ls`, `logs`, `plan`), Run this
worktree's stack (`up`, `start`, `restart`, `stop`, `down`), and Clean
up (`prune`). Each `ls` row carries `next`, the exact command that stops
it, and the text output logs it with `log.Info`. Empty results and
refusals name the command to run.

Example on the VM (an isolated stack, then its worktree deleted with `rm
-rf`):
```
$ ods stack ls
PROJECT           STATUS       APP  INFRA  WEB  AGE  ROOT
demo-wt-fd863404  app-stopped  -    7/7    -    1m   ~/demo-wt
$ rm -rf ~/demo-wt && ods stack ls --orphaned   # exit 3
PROJECT           STATUS    APP  INFRA  WEB  AGE  ROOT
demo-wt-fd863404  orphaned  -    7/7    -    1m   ~/demo-wt (missing)
level=info msg="To stop demo-wt-fd863404 (orphaned), run: ods stack down --project demo-wt-fd863404"
```
After `stack stop`, only the `*_HOST_PORT` leases remained. After `down
--project`, no containers or leases were left.

## How Has This Been Tested?

- Unit tests for the inventory merge (fake containers, owners, states,
leases, liveness): live, app-stopped, infra-only, Compose-only, orphan
with a missing root, a removed worktree whose directory remains, a stale
lease, Docker unavailable, unreadable sources, and sort order.
- Real-process tests for `DownProject`/`StopProject`: an orphan in a
deleted linked worktree is stopped, its leases are released and its
owner record is removed; a live worktree is refused without `--force`.
- Lease tests: reclaim of a stopped owner, live process group skipped,
non-stopped owner skipped, `*_HOST_PORT` with and without running
containers, drift down vs. running, `ReleaseNames`, stop/down releases,
adoption with and without a live legacy group, and the frontend handoff
releasing preview leases.
- Coverage gates: `cmd` is at 90.6% against a 90.4% floor, and
`internal/docker` at 99.3% against 99.2%.
- `go test ./internal/... ./cmd/` passes, except `TestRunCI_failures`,
which also fails on a clean main in this environment. golangci-lint
passes.
- New integration test `TestStackLiveOrphanedWorktree` (`-tags
integration`) passed on the VM (246 s). It starts an isolated stack in a
disposable worktree, deletes the worktree, asserts that `ls --orphaned`
lists the stack with exit 3, then that `down --project` stops its
processes, removes its containers and releases its leases.

## Additional Options

- [ ] [Optional] Please cherry-pick this PR to the latest release
version.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added `ods stack ls` to view active, stopped, orphaned, and
stale-lease stacks with service, health, port, age, and location
details.
* Added JSON output, filtering, orphan detection, and project-based
shutdown support.
* Added `ods stack prune` to safely clean up orphaned stacks, with
optional volume removal.
* **Bug Fixes**
  * Improved port lease reclamation and reassignment for stopped stacks.
* `stop` now releases application ports while preserving infrastructure
ports; `down` releases all ports.
* **Documentation**
  * Expanded lifecycle, recovery, listing, and cleanup guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: roshan <rohod04@gmail.com>
Co-authored-by: roshan <38771624+rohoswagger@users.noreply.github.com>
2026-10-20 00:22:08 +00:00

346 lines
10 KiB
Go

package cmd
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"text/tabwriter"
"time"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"github.com/onyx-dot-app/onyx/tools/ods/internal/devstack"
"github.com/onyx-dot-app/onyx/tools/ods/internal/docker"
"github.com/onyx-dot-app/onyx/tools/ods/internal/prompt"
)
func newStackLsCommand() *cobra.Command {
var asJSON, all, orphaned bool
cmd := &cobra.Command{
Use: "ls",
Short: "List every dev stack on this machine, with the command that stops each",
Long: `List every Onyx dev stack on this machine, from any directory. ls reads
Docker, the project owner records, the stack state files and the port
leases. It takes no locks and changes nothing.
Statuses:
orphaned no worktree owns the stack any more: stop it with
'ods stack down --project NAME' or 'ods stack prune'
active application processes run
app-stopped infrastructure runs, 'ods stack stop' stopped the app
infra-only infrastructure runs, no application process does
stopped nothing runs (shown with --all)
stale-lease port leases that no stack uses (shown with --all)
Exit codes: 0 success, 3 when --orphaned finds an orphan, 1 when no
source could be read.`,
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
inv := discoverStacks()
for _, warning := range inv.Warnings {
log.Warn(warning)
}
if inv.Unreadable {
log.Error("no stack source could be read")
exitHook(1)
return
}
rows := filterStackRows(inv.Rows, all, orphaned)
if len(rows) == 0 && !asJSON {
switch {
case orphaned:
log.Info("No orphaned dev stacks found")
case all:
log.Info("No dev stacks found on this machine; run 'ods stack up' in a worktree to start one")
default:
log.Info("No running dev stacks found; run 'ods stack ls --all' to include stopped stacks")
}
return
}
if asJSON {
printJSON(rows)
} else {
printStackRows(os.Stdout, rows, time.Now())
if hidden := len(filterStackRows(inv.Rows, true, orphaned)) - len(rows); hidden > 0 {
log.Infof("%d stopped stack(s) or stale lease(s) hidden; run 'ods stack ls --all' to show them", hidden)
}
}
if orphaned && len(rows) > 0 {
exitHook(3)
}
},
}
cmd.Flags().BoolVar(&asJSON, "json", false, "Print the stacks as a JSON array")
cmd.Flags().BoolVar(&all, "all", false, "Also show stopped stacks and stale port leases")
cmd.Flags().BoolVar(&orphaned, "orphaned", false, "Show only orphaned stacks")
return cmd
}
func newStackPruneCommand() *cobra.Command {
var yes, volumes bool
var stopTimeout, lockWait time.Duration
cmd := &cobra.Command{
Use: "prune",
Short: "Run 'down --project' for every orphaned stack (volumes are kept)",
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
inv := discoverStacks()
for _, warning := range inv.Warnings {
log.Warn(warning)
}
rows := filterStackRows(inv.Rows, true, true)
if len(rows) == 0 {
log.Info("No orphaned stacks")
return
}
printStackRows(os.Stdout, rows, time.Now())
if !yes && !prompt.Confirm(fmt.Sprintf("Stop %d orphaned stack(s) and their containers? Volumes are kept. (yes/no): ", len(rows))) {
log.Fatal("Aborted")
}
if volumes && !prompt.Confirm(fmt.Sprintf("Also DELETE the Docker volumes (database, index, files) of %s? (yes/no): ", stackRowNames(rows))) {
log.Fatal("Aborted")
}
registry := runtimeRegistry()
failed := 0
for _, row := range rows {
result, err := devstack.CleanupRow(context.Background(), registry, row, composeDownProject,
devstack.ProjectOptions{StopTimeout: stopTimeout, LockWait: lockWait, Volumes: volumes})
if err != nil {
log.Errorf("%s: %v", stackRowName(row), err)
failed++
continue
}
logProjectResult(result)
}
if failed > 0 {
exitHook(1)
}
},
}
cmd.Flags().BoolVar(&yes, "yes", false, "Do not ask for confirmation (the --volumes confirmation is still asked)")
cmd.Flags().BoolVar(&volumes, "volumes", false, "Also delete the Docker volumes of the orphaned stacks")
cmd.Flags().DurationVar(&stopTimeout, "timeout", 20*time.Second, "Grace period before a service is killed")
cmd.Flags().DurationVar(&lockWait, "lock-wait", 10*time.Minute, "How long to wait for another lifecycle command on a stack")
return cmd
}
func discoverStacks() devstack.Inventory {
var dirs []string
if cwd, err := os.Getwd(); err == nil {
dirs = append(dirs, cwd)
}
return devstack.Discover(runtimeRegistry(), devstack.DiscoverOptions{Dirs: dirs})
}
func filterStackRows(rows []devstack.StackRow, all, orphaned bool) []devstack.StackRow {
kept := []devstack.StackRow{}
for _, row := range rows {
switch {
case orphaned:
if row.Status != devstack.StatusOrphaned {
continue
}
case !all:
if !row.Running() {
continue
}
}
kept = append(kept, row)
}
return kept
}
// runProjectLifecycle stops (or downs) a stack selected by project from any
// directory, for a stack whose checkout is gone or is elsewhere.
func runProjectLifecycle(name, project string, opts devstack.ProjectOptions) {
registry := runtimeRegistry()
var result *devstack.ProjectResult
var err error
if name == "down" {
result, err = devstack.DownProject(context.Background(), registry, project, composeDownProject, opts)
} else {
result, err = devstack.StopProject(context.Background(), registry, project, opts)
}
if result != nil {
logProjectResult(result)
}
if err != nil {
log.Fatal(err)
}
}
// projectInCheckout reports whether the stack of the current checkout in
// the selected mode already records project, so the checkout path applies.
func projectInCheckout(project string) bool {
cwd, err := os.Getwd()
if err != nil {
return false
}
mode := devstack.Shared
if rootIsolated {
mode = devstack.Isolated
}
stack, err := devstack.Open(cwd, project, mode)
if err != nil {
return false
}
st, err := stack.ReadState()
if err != nil || st == nil || st.Project != docker.NormalizeProjectName(project) {
return false
}
return mode != devstack.Shared || st.Root == stack.Root
}
func logProjectResult(result *devstack.ProjectResult) {
parts := []string{fmt.Sprintf("%d service(s) stopped", result.StoppedServices)}
if result.ComposeDown {
parts = append(parts, "containers removed")
}
if result.ReleasedLeases > 0 {
parts = append(parts, fmt.Sprintf("%d port lease(s) released", result.ReleasedLeases))
}
if result.RemovedOwner {
parts = append(parts, "owner record removed")
}
log.Infof("%s: %s", stackRowName(result.Row), strings.Join(parts, ", "))
}
// composeDownProject runs Compose down by project name alone, so it works
// when the checkout that started the project is gone.
func composeDownProject(ctx context.Context, project string, volumes bool) error {
args := []string{"compose", "-p", project, "down"}
if volumes {
args = append(args, "--volumes")
}
cmd := exec.CommandContext(ctx, "docker", args...)
// A neutral directory keeps Compose from loading an unrelated compose file.
cmd.Dir = os.TempDir()
cmd.Stdout, cmd.Stderr = os.Stderr, os.Stderr
if err := cmd.Run(); err != nil {
return fmt.Errorf("docker %s: %w", strings.Join(args, " "), err)
}
return nil
}
func printJSON(value any) {
data, err := json.MarshalIndent(value, "", " ")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))
}
func printStackRows(out io.Writer, rows []devstack.StackRow, now time.Time) {
if len(rows) == 0 {
return
}
w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
_, _ = fmt.Fprintln(w, "PROJECT\tSTATUS\tAPP\tINFRA\tWEB\tAGE\tROOT")
statuses := map[string]bool{}
for _, row := range rows {
statuses[row.Status] = true
app, infra, web := "-", "-", "-"
if row.App != nil && row.App.Total > 0 {
app = fmt.Sprintf("%d/%d", row.App.Running, row.App.Total)
}
if row.Infra != nil {
infra = fmt.Sprintf("%d/%d", row.Infra.Running, row.Infra.Total)
if row.Infra.Health != "" && row.Infra.Health != "healthy" {
infra += " " + row.Infra.Health
}
}
if row.Web != "" {
web = row.Web
}
_, _ = fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", stackRowName(row), row.Status, app, infra, web, formatAge(row.Since, now), displayRoot(row))
for _, e := range row.Errors {
_, _ = fmt.Fprintf(w, " error: %s\t\t\t\t\t\t\n", e)
}
}
_ = w.Flush()
for _, row := range rows {
if row.Next != "" {
log.Infof("To stop %s (%s), run: %s", stackRowName(row), row.Status, row.Next)
}
}
if statuses[devstack.StatusOrphaned] {
log.Info("Orphaned stacks have no worktree; 'ods stack prune' stops all of them and keeps their volumes")
}
}
func stackRowName(row devstack.StackRow) string {
name := row.Project
if name == "" {
name = "-"
}
if row.Mode == devstack.Frontend {
name += " (frontend)"
}
return name
}
func stackRowNames(rows []devstack.StackRow) string {
names := make([]string, 0, len(rows))
for _, row := range rows {
names = append(names, stackRowName(row))
}
return strings.Join(names, ", ")
}
func formatAge(since, now time.Time) string {
if since.IsZero() {
return "-"
}
d := now.Sub(since)
switch {
case d < time.Minute:
return fmt.Sprintf("%ds", max(0, int(d.Seconds())))
case d < time.Hour:
return fmt.Sprintf("%dm", int(d.Minutes()))
case d < 24*time.Hour:
return fmt.Sprintf("%dh", int(d.Hours()))
}
return fmt.Sprintf("%dd", int(d.Hours()/24))
}
func displayRoot(row devstack.StackRow) string {
if row.Root == "" {
return "-"
}
root := shortenPath(row.Root)
switch {
case !row.RootExists:
root += " (missing)"
case !row.Worktree && row.StateDir != "":
root += " (not a worktree)"
case row.Branch != "":
root += " [" + row.Branch + "]"
}
return root
}
// shortenPath abbreviates the home directory and keeps the last two
// elements of a deep path.
func shortenPath(path string) string {
prefix := ""
rest := path
if home, err := os.UserHomeDir(); err == nil && home != "" && (path == home || strings.HasPrefix(path, home+string(filepath.Separator))) {
prefix, rest = "~", strings.TrimPrefix(path, home)
}
parts := strings.Split(strings.Trim(rest, string(filepath.Separator)), string(filepath.Separator))
if len(parts) <= 3 {
if prefix == "" {
return path
}
return prefix + rest
}
sep := string(filepath.Separator)
return prefix + sep + "…" + sep + strings.Join(parts[len(parts)-2:], sep)
}