mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-23 20:02:57 -05:00
## Description
Adds `ods stack ls` to list every Onyx dev stack on the machine from any
directory, and lets `stop`/`down --project NAME` act on a stack whose
worktree is gone. The PR also releases port leases when a stack stops
owning them.
**`ods stack ls [--json] [--all] [--orphaned]`**
(`internal/devstack/inventory.go`, `cmd/stack_ls.go`)
- It merges four sources, keyed by Compose project: `docker ps -a` +
`docker inspect` (labels, state, health), `projects/*/owner.json`, every
reachable `state.json` (owner state dirs, plus
`<git-common-dir>/{ods-stack,ods-stack-shared,ods-stack-frontend,worktrees/*/ods-stack,worktrees/*/ods-stack-frontend}`
from owner, lease, Docker and state roots), and `Registry.Leases()`. It
does not call `devstack.Open`, and it takes no locks.
- An Onyx Compose project is one with `relational_db` + `cache`
services, or one that a state or owner record claims.
- Statuses: `active` (an app process is alive, checked with
`groupIsOurs`, so a reused PID does not count), `app-stopped` (infra
runs, phase `stopped`), `infra-only` (infra runs, no app process: a
Compose-only project or dead processes), `stopped`, `orphaned`, and
`stale-lease` (`--all` only).
- Orphan rules: the Compose working dir is gone and no state claims the
project; a state's root is gone; or the root exists but is no longer a
git worktree (state rows, or Docker rows with an owner record whose
state is gone). A row counts as orphaned only when something is left to
clean up: processes, containers, leases or an owner record.
- A missing or corrupt state degrades its row (`errors`). When Docker is
down, `ls` prints a warning and shows rows from saved state only. Exit
codes: 0; 3 when `--orphaned` finds an orphan; 1 when no source can be
read.
**`down`/`stop --project NAME` and `prune`**
(`internal/devstack/project_down.go`)
- The command resolves the stack through `owner.json` → `state_dir`,
then any state that claims the project, then Docker labels. It runs from
any directory.
- It takes the state lock (only if the state dir still exists, so git
metadata is never recreated) and the project lock. It stops recorded
groups with `stopServices`/`stopRecords` and runs `docker compose -p
NAME down` from a neutral dir. Volumes are kept.
- After a successful down it releases the stack's leases and removes
`owner.json` for an orphan.
- It refuses a stack with a live service under an existing git worktree
unless you pass `--force`. If `--project` already matches the current
checkout's stack, the existing checkout path runs.
- `ods stack prune` runs this for every orphaned row after a
confirmation (`--yes` skips it). `--volumes` has its own confirmation,
and `--yes` does not skip it.
**Port lease fixes** (`ports.go`, `lifecycle.go`, `legacy.go`)
- `Stack.Stop` releases app leases (everything except `*_HOST_PORT`).
`Stack.Down` releases app leases after `stopServices`, and every lease
after Compose down. New `Registry.ReleaseNames`.
- Shared adoption (`releaseLegacyLeases`, runs after the legacy guard
and before `leasePorts`): each stopped legacy candidate without live
process groups loses all its leases, or only its app leases while its
containers run.
- Reclaim: `Lease.StateDir` is now recorded (`LeaseFor`). Another
owner's lease is reclaimable when its state is `stopped` or missing,
none of its recorded groups is alive, and, for `*_HOST_PORT` only, its
Compose project has no running container. That last check is lazy, so
app-port leasing stays fast. Leases without `StateDir` are reclaimable
only for shared/frontend keys, which are their own state dirs.
- Drift down: `choosePort` moves a lease to the lowest free port from
`Preferred` when the service is not running. I added
`PortRequest.Running` because `Fixed` alone is not set for a running
service that does not hold its port yet, and moving such a service
changed `web dev`'s full-stack owner
(`TestWebRuntimeUsesTheFullStackWebOwner` caught this).
Docs: `ods stack` help, `tools/ods/README.md`,
`tools/ods/integration/README.md`, and the onyx-dev-stack skill (a new
"Find and stop old stacks" step, and the updated port bullet).
Known limit: `git worktree remove` deletes
`<common>/worktrees/<name>/ods-stack`, so the recorded PIDs are lost.
`down --project` still removes the containers, leases and owner record,
but it cannot find app processes that outlived that removal.
**Discovery-first CLI.** Bare `ods stack` prints a short intro, then
groups its subcommands with the same `AddGroup` pattern as the root
command: Find stacks (`status`, `ls`, `logs`, `plan`), Run this
worktree's stack (`up`, `start`, `restart`, `stop`, `down`), and Clean
up (`prune`). Each `ls` row carries `next`, the exact command that stops
it, and the text output logs it with `log.Info`. Empty results and
refusals name the command to run.
Example on the VM (an isolated stack, then its worktree deleted with `rm
-rf`):
```
$ ods stack ls
PROJECT STATUS APP INFRA WEB AGE ROOT
demo-wt-fd863404 app-stopped - 7/7 - 1m ~/demo-wt
$ rm -rf ~/demo-wt && ods stack ls --orphaned # exit 3
PROJECT STATUS APP INFRA WEB AGE ROOT
demo-wt-fd863404 orphaned - 7/7 - 1m ~/demo-wt (missing)
level=info msg="To stop demo-wt-fd863404 (orphaned), run: ods stack down --project demo-wt-fd863404"
```
After `stack stop`, only the `*_HOST_PORT` leases remained. After `down
--project`, no containers or leases were left.
## How Has This Been Tested?
- Unit tests for the inventory merge (fake containers, owners, states,
leases, liveness): live, app-stopped, infra-only, Compose-only, orphan
with a missing root, a removed worktree whose directory remains, a stale
lease, Docker unavailable, unreadable sources, and sort order.
- Real-process tests for `DownProject`/`StopProject`: an orphan in a
deleted linked worktree is stopped, its leases are released and its
owner record is removed; a live worktree is refused without `--force`.
- Lease tests: reclaim of a stopped owner, live process group skipped,
non-stopped owner skipped, `*_HOST_PORT` with and without running
containers, drift down vs. running, `ReleaseNames`, stop/down releases,
adoption with and without a live legacy group, and the frontend handoff
releasing preview leases.
- Coverage gates: `cmd` is at 90.6% against a 90.4% floor, and
`internal/docker` at 99.3% against 99.2%.
- `go test ./internal/... ./cmd/` passes, except `TestRunCI_failures`,
which also fails on a clean main in this environment. golangci-lint
passes.
- New integration test `TestStackLiveOrphanedWorktree` (`-tags
integration`) passed on the VM (246 s). It starts an isolated stack in a
disposable worktree, deletes the worktree, asserts that `ls --orphaned`
lists the stack with exit 3, then that `down --project` stops its
processes, removes its containers and releases its leases.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `ods stack ls` to view active, stopped, orphaned, and
stale-lease stacks with service, health, port, age, and location
details.
* Added JSON output, filtering, orphan detection, and project-based
shutdown support.
* Added `ods stack prune` to safely clean up orphaned stacks, with
optional volume removal.
* **Bug Fixes**
* Improved port lease reclamation and reassignment for stopped stacks.
* `stop` now releases application ports while preserving infrastructure
ports; `down` releases all ports.
* **Documentation**
* Expanded lifecycle, recovery, listing, and cleanup guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: roshan <rohod04@gmail.com>
Co-authored-by: roshan <38771624+rohoswagger@users.noreply.github.com>
346 lines
10 KiB
Go
346 lines
10 KiB
Go
package cmd
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"io"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strings"
|
|
"text/tabwriter"
|
|
"time"
|
|
|
|
log "github.com/sirupsen/logrus"
|
|
"github.com/spf13/cobra"
|
|
|
|
"github.com/onyx-dot-app/onyx/tools/ods/internal/devstack"
|
|
"github.com/onyx-dot-app/onyx/tools/ods/internal/docker"
|
|
"github.com/onyx-dot-app/onyx/tools/ods/internal/prompt"
|
|
)
|
|
|
|
func newStackLsCommand() *cobra.Command {
|
|
var asJSON, all, orphaned bool
|
|
cmd := &cobra.Command{
|
|
Use: "ls",
|
|
Short: "List every dev stack on this machine, with the command that stops each",
|
|
Long: `List every Onyx dev stack on this machine, from any directory. ls reads
|
|
Docker, the project owner records, the stack state files and the port
|
|
leases. It takes no locks and changes nothing.
|
|
|
|
Statuses:
|
|
orphaned no worktree owns the stack any more: stop it with
|
|
'ods stack down --project NAME' or 'ods stack prune'
|
|
active application processes run
|
|
app-stopped infrastructure runs, 'ods stack stop' stopped the app
|
|
infra-only infrastructure runs, no application process does
|
|
stopped nothing runs (shown with --all)
|
|
stale-lease port leases that no stack uses (shown with --all)
|
|
|
|
Exit codes: 0 success, 3 when --orphaned finds an orphan, 1 when no
|
|
source could be read.`,
|
|
Args: cobra.NoArgs,
|
|
Run: func(cmd *cobra.Command, args []string) {
|
|
inv := discoverStacks()
|
|
for _, warning := range inv.Warnings {
|
|
log.Warn(warning)
|
|
}
|
|
if inv.Unreadable {
|
|
log.Error("no stack source could be read")
|
|
exitHook(1)
|
|
return
|
|
}
|
|
rows := filterStackRows(inv.Rows, all, orphaned)
|
|
if len(rows) == 0 && !asJSON {
|
|
switch {
|
|
case orphaned:
|
|
log.Info("No orphaned dev stacks found")
|
|
case all:
|
|
log.Info("No dev stacks found on this machine; run 'ods stack up' in a worktree to start one")
|
|
default:
|
|
log.Info("No running dev stacks found; run 'ods stack ls --all' to include stopped stacks")
|
|
}
|
|
return
|
|
}
|
|
if asJSON {
|
|
printJSON(rows)
|
|
} else {
|
|
printStackRows(os.Stdout, rows, time.Now())
|
|
if hidden := len(filterStackRows(inv.Rows, true, orphaned)) - len(rows); hidden > 0 {
|
|
log.Infof("%d stopped stack(s) or stale lease(s) hidden; run 'ods stack ls --all' to show them", hidden)
|
|
}
|
|
}
|
|
if orphaned && len(rows) > 0 {
|
|
exitHook(3)
|
|
}
|
|
},
|
|
}
|
|
cmd.Flags().BoolVar(&asJSON, "json", false, "Print the stacks as a JSON array")
|
|
cmd.Flags().BoolVar(&all, "all", false, "Also show stopped stacks and stale port leases")
|
|
cmd.Flags().BoolVar(&orphaned, "orphaned", false, "Show only orphaned stacks")
|
|
return cmd
|
|
}
|
|
|
|
func newStackPruneCommand() *cobra.Command {
|
|
var yes, volumes bool
|
|
var stopTimeout, lockWait time.Duration
|
|
cmd := &cobra.Command{
|
|
Use: "prune",
|
|
Short: "Run 'down --project' for every orphaned stack (volumes are kept)",
|
|
Args: cobra.NoArgs,
|
|
Run: func(cmd *cobra.Command, args []string) {
|
|
inv := discoverStacks()
|
|
for _, warning := range inv.Warnings {
|
|
log.Warn(warning)
|
|
}
|
|
rows := filterStackRows(inv.Rows, true, true)
|
|
if len(rows) == 0 {
|
|
log.Info("No orphaned stacks")
|
|
return
|
|
}
|
|
printStackRows(os.Stdout, rows, time.Now())
|
|
if !yes && !prompt.Confirm(fmt.Sprintf("Stop %d orphaned stack(s) and their containers? Volumes are kept. (yes/no): ", len(rows))) {
|
|
log.Fatal("Aborted")
|
|
}
|
|
if volumes && !prompt.Confirm(fmt.Sprintf("Also DELETE the Docker volumes (database, index, files) of %s? (yes/no): ", stackRowNames(rows))) {
|
|
log.Fatal("Aborted")
|
|
}
|
|
registry := runtimeRegistry()
|
|
failed := 0
|
|
for _, row := range rows {
|
|
result, err := devstack.CleanupRow(context.Background(), registry, row, composeDownProject,
|
|
devstack.ProjectOptions{StopTimeout: stopTimeout, LockWait: lockWait, Volumes: volumes})
|
|
if err != nil {
|
|
log.Errorf("%s: %v", stackRowName(row), err)
|
|
failed++
|
|
continue
|
|
}
|
|
logProjectResult(result)
|
|
}
|
|
if failed > 0 {
|
|
exitHook(1)
|
|
}
|
|
},
|
|
}
|
|
cmd.Flags().BoolVar(&yes, "yes", false, "Do not ask for confirmation (the --volumes confirmation is still asked)")
|
|
cmd.Flags().BoolVar(&volumes, "volumes", false, "Also delete the Docker volumes of the orphaned stacks")
|
|
cmd.Flags().DurationVar(&stopTimeout, "timeout", 20*time.Second, "Grace period before a service is killed")
|
|
cmd.Flags().DurationVar(&lockWait, "lock-wait", 10*time.Minute, "How long to wait for another lifecycle command on a stack")
|
|
return cmd
|
|
}
|
|
|
|
func discoverStacks() devstack.Inventory {
|
|
var dirs []string
|
|
if cwd, err := os.Getwd(); err == nil {
|
|
dirs = append(dirs, cwd)
|
|
}
|
|
return devstack.Discover(runtimeRegistry(), devstack.DiscoverOptions{Dirs: dirs})
|
|
}
|
|
|
|
func filterStackRows(rows []devstack.StackRow, all, orphaned bool) []devstack.StackRow {
|
|
kept := []devstack.StackRow{}
|
|
for _, row := range rows {
|
|
switch {
|
|
case orphaned:
|
|
if row.Status != devstack.StatusOrphaned {
|
|
continue
|
|
}
|
|
case !all:
|
|
if !row.Running() {
|
|
continue
|
|
}
|
|
}
|
|
kept = append(kept, row)
|
|
}
|
|
return kept
|
|
}
|
|
|
|
// runProjectLifecycle stops (or downs) a stack selected by project from any
|
|
// directory, for a stack whose checkout is gone or is elsewhere.
|
|
func runProjectLifecycle(name, project string, opts devstack.ProjectOptions) {
|
|
registry := runtimeRegistry()
|
|
var result *devstack.ProjectResult
|
|
var err error
|
|
if name == "down" {
|
|
result, err = devstack.DownProject(context.Background(), registry, project, composeDownProject, opts)
|
|
} else {
|
|
result, err = devstack.StopProject(context.Background(), registry, project, opts)
|
|
}
|
|
if result != nil {
|
|
logProjectResult(result)
|
|
}
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
}
|
|
|
|
// projectInCheckout reports whether the stack of the current checkout in
|
|
// the selected mode already records project, so the checkout path applies.
|
|
func projectInCheckout(project string) bool {
|
|
cwd, err := os.Getwd()
|
|
if err != nil {
|
|
return false
|
|
}
|
|
mode := devstack.Shared
|
|
if rootIsolated {
|
|
mode = devstack.Isolated
|
|
}
|
|
stack, err := devstack.Open(cwd, project, mode)
|
|
if err != nil {
|
|
return false
|
|
}
|
|
st, err := stack.ReadState()
|
|
if err != nil || st == nil || st.Project != docker.NormalizeProjectName(project) {
|
|
return false
|
|
}
|
|
return mode != devstack.Shared || st.Root == stack.Root
|
|
}
|
|
|
|
func logProjectResult(result *devstack.ProjectResult) {
|
|
parts := []string{fmt.Sprintf("%d service(s) stopped", result.StoppedServices)}
|
|
if result.ComposeDown {
|
|
parts = append(parts, "containers removed")
|
|
}
|
|
if result.ReleasedLeases > 0 {
|
|
parts = append(parts, fmt.Sprintf("%d port lease(s) released", result.ReleasedLeases))
|
|
}
|
|
if result.RemovedOwner {
|
|
parts = append(parts, "owner record removed")
|
|
}
|
|
log.Infof("%s: %s", stackRowName(result.Row), strings.Join(parts, ", "))
|
|
}
|
|
|
|
// composeDownProject runs Compose down by project name alone, so it works
|
|
// when the checkout that started the project is gone.
|
|
func composeDownProject(ctx context.Context, project string, volumes bool) error {
|
|
args := []string{"compose", "-p", project, "down"}
|
|
if volumes {
|
|
args = append(args, "--volumes")
|
|
}
|
|
cmd := exec.CommandContext(ctx, "docker", args...)
|
|
// A neutral directory keeps Compose from loading an unrelated compose file.
|
|
cmd.Dir = os.TempDir()
|
|
cmd.Stdout, cmd.Stderr = os.Stderr, os.Stderr
|
|
if err := cmd.Run(); err != nil {
|
|
return fmt.Errorf("docker %s: %w", strings.Join(args, " "), err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func printJSON(value any) {
|
|
data, err := json.MarshalIndent(value, "", " ")
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
fmt.Println(string(data))
|
|
}
|
|
|
|
func printStackRows(out io.Writer, rows []devstack.StackRow, now time.Time) {
|
|
if len(rows) == 0 {
|
|
return
|
|
}
|
|
w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
|
|
_, _ = fmt.Fprintln(w, "PROJECT\tSTATUS\tAPP\tINFRA\tWEB\tAGE\tROOT")
|
|
statuses := map[string]bool{}
|
|
for _, row := range rows {
|
|
statuses[row.Status] = true
|
|
app, infra, web := "-", "-", "-"
|
|
if row.App != nil && row.App.Total > 0 {
|
|
app = fmt.Sprintf("%d/%d", row.App.Running, row.App.Total)
|
|
}
|
|
if row.Infra != nil {
|
|
infra = fmt.Sprintf("%d/%d", row.Infra.Running, row.Infra.Total)
|
|
if row.Infra.Health != "" && row.Infra.Health != "healthy" {
|
|
infra += " " + row.Infra.Health
|
|
}
|
|
}
|
|
if row.Web != "" {
|
|
web = row.Web
|
|
}
|
|
_, _ = fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", stackRowName(row), row.Status, app, infra, web, formatAge(row.Since, now), displayRoot(row))
|
|
for _, e := range row.Errors {
|
|
_, _ = fmt.Fprintf(w, " error: %s\t\t\t\t\t\t\n", e)
|
|
}
|
|
}
|
|
_ = w.Flush()
|
|
for _, row := range rows {
|
|
if row.Next != "" {
|
|
log.Infof("To stop %s (%s), run: %s", stackRowName(row), row.Status, row.Next)
|
|
}
|
|
}
|
|
if statuses[devstack.StatusOrphaned] {
|
|
log.Info("Orphaned stacks have no worktree; 'ods stack prune' stops all of them and keeps their volumes")
|
|
}
|
|
}
|
|
|
|
func stackRowName(row devstack.StackRow) string {
|
|
name := row.Project
|
|
if name == "" {
|
|
name = "-"
|
|
}
|
|
if row.Mode == devstack.Frontend {
|
|
name += " (frontend)"
|
|
}
|
|
return name
|
|
}
|
|
|
|
func stackRowNames(rows []devstack.StackRow) string {
|
|
names := make([]string, 0, len(rows))
|
|
for _, row := range rows {
|
|
names = append(names, stackRowName(row))
|
|
}
|
|
return strings.Join(names, ", ")
|
|
}
|
|
|
|
func formatAge(since, now time.Time) string {
|
|
if since.IsZero() {
|
|
return "-"
|
|
}
|
|
d := now.Sub(since)
|
|
switch {
|
|
case d < time.Minute:
|
|
return fmt.Sprintf("%ds", max(0, int(d.Seconds())))
|
|
case d < time.Hour:
|
|
return fmt.Sprintf("%dm", int(d.Minutes()))
|
|
case d < 24*time.Hour:
|
|
return fmt.Sprintf("%dh", int(d.Hours()))
|
|
}
|
|
return fmt.Sprintf("%dd", int(d.Hours()/24))
|
|
}
|
|
|
|
func displayRoot(row devstack.StackRow) string {
|
|
if row.Root == "" {
|
|
return "-"
|
|
}
|
|
root := shortenPath(row.Root)
|
|
switch {
|
|
case !row.RootExists:
|
|
root += " (missing)"
|
|
case !row.Worktree && row.StateDir != "":
|
|
root += " (not a worktree)"
|
|
case row.Branch != "":
|
|
root += " [" + row.Branch + "]"
|
|
}
|
|
return root
|
|
}
|
|
|
|
// shortenPath abbreviates the home directory and keeps the last two
|
|
// elements of a deep path.
|
|
func shortenPath(path string) string {
|
|
prefix := ""
|
|
rest := path
|
|
if home, err := os.UserHomeDir(); err == nil && home != "" && (path == home || strings.HasPrefix(path, home+string(filepath.Separator))) {
|
|
prefix, rest = "~", strings.TrimPrefix(path, home)
|
|
}
|
|
parts := strings.Split(strings.Trim(rest, string(filepath.Separator)), string(filepath.Separator))
|
|
if len(parts) <= 3 {
|
|
if prefix == "" {
|
|
return path
|
|
}
|
|
return prefix + rest
|
|
}
|
|
sep := string(filepath.Separator)
|
|
return prefix + sep + "…" + sep + strings.Join(parts[len(parts)-2:], sep)
|
|
}
|