mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-23 11:53:08 -05:00
## Description
Adds `ods stack ls` to list every Onyx dev stack on the machine from any
directory, and lets `stop`/`down --project NAME` act on a stack whose
worktree is gone. The PR also releases port leases when a stack stops
owning them.
**`ods stack ls [--json] [--all] [--orphaned]`**
(`internal/devstack/inventory.go`, `cmd/stack_ls.go`)
- It merges four sources, keyed by Compose project: `docker ps -a` +
`docker inspect` (labels, state, health), `projects/*/owner.json`, every
reachable `state.json` (owner state dirs, plus
`<git-common-dir>/{ods-stack,ods-stack-shared,ods-stack-frontend,worktrees/*/ods-stack,worktrees/*/ods-stack-frontend}`
from owner, lease, Docker and state roots), and `Registry.Leases()`. It
does not call `devstack.Open`, and it takes no locks.
- An Onyx Compose project is one with `relational_db` + `cache`
services, or one that a state or owner record claims.
- Statuses: `active` (an app process is alive, checked with
`groupIsOurs`, so a reused PID does not count), `app-stopped` (infra
runs, phase `stopped`), `infra-only` (infra runs, no app process: a
Compose-only project or dead processes), `stopped`, `orphaned`, and
`stale-lease` (`--all` only).
- Orphan rules: the Compose working dir is gone and no state claims the
project; a state's root is gone; or the root exists but is no longer a
git worktree (state rows, or Docker rows with an owner record whose
state is gone). A row counts as orphaned only when something is left to
clean up: processes, containers, leases or an owner record.
- A missing or corrupt state degrades its row (`errors`). When Docker is
down, `ls` prints a warning and shows rows from saved state only. Exit
codes: 0; 3 when `--orphaned` finds an orphan; 1 when no source can be
read.
**`down`/`stop --project NAME` and `prune`**
(`internal/devstack/project_down.go`)
- The command resolves the stack through `owner.json` → `state_dir`,
then any state that claims the project, then Docker labels. It runs from
any directory.
- It takes the state lock (only if the state dir still exists, so git
metadata is never recreated) and the project lock. It stops recorded
groups with `stopServices`/`stopRecords` and runs `docker compose -p
NAME down` from a neutral dir. Volumes are kept.
- After a successful down it releases the stack's leases and removes
`owner.json` for an orphan.
- It refuses a stack with a live service under an existing git worktree
unless you pass `--force`. If `--project` already matches the current
checkout's stack, the existing checkout path runs.
- `ods stack prune` runs this for every orphaned row after a
confirmation (`--yes` skips it). `--volumes` has its own confirmation,
and `--yes` does not skip it.
**Port lease fixes** (`ports.go`, `lifecycle.go`, `legacy.go`)
- `Stack.Stop` releases app leases (everything except `*_HOST_PORT`).
`Stack.Down` releases app leases after `stopServices`, and every lease
after Compose down. New `Registry.ReleaseNames`.
- Shared adoption (`releaseLegacyLeases`, runs after the legacy guard
and before `leasePorts`): each stopped legacy candidate without live
process groups loses all its leases, or only its app leases while its
containers run.
- Reclaim: `Lease.StateDir` is now recorded (`LeaseFor`). Another
owner's lease is reclaimable when its state is `stopped` or missing,
none of its recorded groups is alive, and, for `*_HOST_PORT` only, its
Compose project has no running container. That last check is lazy, so
app-port leasing stays fast. Leases without `StateDir` are reclaimable
only for shared/frontend keys, which are their own state dirs.
- Drift down: `choosePort` moves a lease to the lowest free port from
`Preferred` when the service is not running. I added
`PortRequest.Running` because `Fixed` alone is not set for a running
service that does not hold its port yet, and moving such a service
changed `web dev`'s full-stack owner
(`TestWebRuntimeUsesTheFullStackWebOwner` caught this).
Docs: `ods stack` help, `tools/ods/README.md`,
`tools/ods/integration/README.md`, and the onyx-dev-stack skill (a new
"Find and stop old stacks" step, and the updated port bullet).
Known limit: `git worktree remove` deletes
`<common>/worktrees/<name>/ods-stack`, so the recorded PIDs are lost.
`down --project` still removes the containers, leases and owner record,
but it cannot find app processes that outlived that removal.
**Discovery-first CLI.** Bare `ods stack` prints a short intro, then
groups its subcommands with the same `AddGroup` pattern as the root
command: Find stacks (`status`, `ls`, `logs`, `plan`), Run this
worktree's stack (`up`, `start`, `restart`, `stop`, `down`), and Clean
up (`prune`). Each `ls` row carries `next`, the exact command that stops
it, and the text output logs it with `log.Info`. Empty results and
refusals name the command to run.
Example on the VM (an isolated stack, then its worktree deleted with `rm
-rf`):
```
$ ods stack ls
PROJECT STATUS APP INFRA WEB AGE ROOT
demo-wt-fd863404 app-stopped - 7/7 - 1m ~/demo-wt
$ rm -rf ~/demo-wt && ods stack ls --orphaned # exit 3
PROJECT STATUS APP INFRA WEB AGE ROOT
demo-wt-fd863404 orphaned - 7/7 - 1m ~/demo-wt (missing)
level=info msg="To stop demo-wt-fd863404 (orphaned), run: ods stack down --project demo-wt-fd863404"
```
After `stack stop`, only the `*_HOST_PORT` leases remained. After `down
--project`, no containers or leases were left.
## How Has This Been Tested?
- Unit tests for the inventory merge (fake containers, owners, states,
leases, liveness): live, app-stopped, infra-only, Compose-only, orphan
with a missing root, a removed worktree whose directory remains, a stale
lease, Docker unavailable, unreadable sources, and sort order.
- Real-process tests for `DownProject`/`StopProject`: an orphan in a
deleted linked worktree is stopped, its leases are released and its
owner record is removed; a live worktree is refused without `--force`.
- Lease tests: reclaim of a stopped owner, live process group skipped,
non-stopped owner skipped, `*_HOST_PORT` with and without running
containers, drift down vs. running, `ReleaseNames`, stop/down releases,
adoption with and without a live legacy group, and the frontend handoff
releasing preview leases.
- Coverage gates: `cmd` is at 90.6% against a 90.4% floor, and
`internal/docker` at 99.3% against 99.2%.
- `go test ./internal/... ./cmd/` passes, except `TestRunCI_failures`,
which also fails on a clean main in this environment. golangci-lint
passes.
- New integration test `TestStackLiveOrphanedWorktree` (`-tags
integration`) passed on the VM (246 s). It starts an isolated stack in a
disposable worktree, deletes the worktree, asserts that `ls --orphaned`
lists the stack with exit 3, then that `down --project` stops its
processes, removes its containers and releases its leases.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `ods stack ls` to view active, stopped, orphaned, and
stale-lease stacks with service, health, port, age, and location
details.
* Added JSON output, filtering, orphan detection, and project-based
shutdown support.
* Added `ods stack prune` to safely clean up orphaned stacks, with
optional volume removal.
* **Bug Fixes**
* Improved port lease reclamation and reassignment for stopped stacks.
* `stop` now releases application ports while preserving infrastructure
ports; `down` releases all ports.
* **Documentation**
* Expanded lifecycle, recovery, listing, and cleanup guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: roshan <rohod04@gmail.com>
Co-authored-by: roshan <38771624+rohoswagger@users.noreply.github.com>
262 lines
8.3 KiB
Go
262 lines
8.3 KiB
Go
package devstack
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"os"
|
|
"path/filepath"
|
|
"time"
|
|
|
|
"github.com/onyx-dot-app/onyx/tools/ods/internal/docker"
|
|
)
|
|
|
|
// ComposeDownFunc runs `docker compose -p project down` for a project.
|
|
type ComposeDownFunc func(ctx context.Context, project string, volumes bool) error
|
|
|
|
// ProjectOptions tune StopProject, DownProject and CleanupRow.
|
|
type ProjectOptions struct {
|
|
StopTimeout time.Duration
|
|
LockWait time.Duration
|
|
// Force stops a stack that still runs in an existing worktree.
|
|
Force bool
|
|
// Volumes also removes the Compose volumes.
|
|
Volumes bool
|
|
Discover DiscoverOptions
|
|
}
|
|
|
|
// ProjectResult reports what a project stop or down changed.
|
|
type ProjectResult struct {
|
|
Row StackRow
|
|
StoppedServices int
|
|
ComposeDown bool
|
|
ReleasedLeases int
|
|
RemovedOwner bool
|
|
}
|
|
|
|
// StopProject stops the application processes recorded for project,
|
|
// wherever its checkout is. The infrastructure keeps running.
|
|
func StopProject(ctx context.Context, registry *Registry, project string, opts ProjectOptions) (*ProjectResult, error) {
|
|
row, err := FindProject(registry, project, opts.Discover)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return CleanupRow(ctx, registry, row, nil, opts)
|
|
}
|
|
|
|
// DownProject stops the application processes and the Compose
|
|
// infrastructure of project, wherever its checkout is, then releases the
|
|
// stack's port leases. Volumes are kept unless opts.Volumes is set.
|
|
func DownProject(ctx context.Context, registry *Registry, project string, compose ComposeDownFunc, opts ProjectOptions) (*ProjectResult, error) {
|
|
if compose == nil {
|
|
return nil, errors.New("no Compose runner")
|
|
}
|
|
row, err := FindProject(registry, project, opts.Discover)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return CleanupRow(ctx, registry, row, compose, opts)
|
|
}
|
|
|
|
// FindProject resolves the stack that owns a Compose project: the state
|
|
// named by its owner record, else any state that claims it, else its
|
|
// Docker containers.
|
|
func FindProject(registry *Registry, project string, opts DiscoverOptions) (StackRow, error) {
|
|
project = docker.NormalizeProjectName(project)
|
|
inv := Discover(registry, opts)
|
|
ownerDir := ""
|
|
if registry != nil {
|
|
if owner := readOwner(ownerPath(registry, project)); owner.Err == nil {
|
|
ownerDir = owner.StateDir
|
|
}
|
|
}
|
|
var fallback *StackRow
|
|
for i := range inv.Rows {
|
|
row := &inv.Rows[i]
|
|
if row.Project != project || row.Mode == Frontend || row.Status == StatusStaleLease {
|
|
continue
|
|
}
|
|
if ownerDir != "" && row.StateDir == ownerDir {
|
|
return *row, nil
|
|
}
|
|
if fallback == nil || (fallback.Infra == nil && row.Infra != nil) {
|
|
fallback = row
|
|
}
|
|
}
|
|
if fallback != nil {
|
|
return *fallback, nil
|
|
}
|
|
if registry != nil && pathExists(ownerPath(registry, project)) {
|
|
return StackRow{Project: project, OwnerRecord: ownerPath(registry, project), StateDir: ownerDir, Status: StatusOrphaned, Errors: []string{}}, nil
|
|
}
|
|
return StackRow{}, fmt.Errorf("no dev stack found for project %s; list stacks with 'ods stack ls --all'", project)
|
|
}
|
|
|
|
func ownerPath(registry *Registry, project string) string {
|
|
return filepath.Join(registry.Dir, projectsDirName, ProjectName(project), ownerFileName)
|
|
}
|
|
|
|
// CleanupRow stops the live processes recorded for row and, with compose,
|
|
// runs Compose down for its project, releases its port leases, and removes
|
|
// the owner record of an orphan. It refuses a stack that still runs in an
|
|
// existing worktree unless opts.Force is set.
|
|
func CleanupRow(ctx context.Context, registry *Registry, row StackRow, compose ComposeDownFunc, opts ProjectOptions) (*ProjectResult, error) {
|
|
result := &ProjectResult{Row: row}
|
|
var stack *Stack
|
|
if row.StateDir != "" && pathExists(row.StateDir) {
|
|
stack = &Stack{StateDir: row.StateDir, Project: row.Project, Mode: row.Mode, Root: row.Root}
|
|
release, err := stack.Lock(ctx, "stack down --project", opts.LockWait)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer release()
|
|
}
|
|
infra := compose != nil && row.Mode != Frontend && row.Project != ""
|
|
if lockDir := projectLockDir(registry, row, infra); lockDir != "" {
|
|
lock := &Stack{StateDir: lockDir}
|
|
release, err := lock.Lock(ctx, "stack down --project", opts.LockWait)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer release()
|
|
}
|
|
|
|
if stack != nil {
|
|
st, err := stack.ReadState()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if st != nil && st.Project != row.Project {
|
|
return nil, fmt.Errorf("the state in %s now records project %s; run 'ods stack ls' again", row.StateDir, st.Project)
|
|
}
|
|
if st != nil {
|
|
if !opts.Force && liveInWorktree(st) {
|
|
flag := ""
|
|
if row.Mode == Isolated {
|
|
flag = "--isolated "
|
|
}
|
|
return nil, fmt.Errorf("project %s still runs in the worktree %s; run 'cd %s && ods %sstack down', or add --force to stop it from here", row.Project, st.Root, shellQuote(st.Root), flag)
|
|
}
|
|
for _, rec := range st.Services {
|
|
if rec.Skipped == "" && groupIsOurs(rec) {
|
|
result.StoppedServices++
|
|
}
|
|
}
|
|
if err := stack.stopServices(opts.StopTimeout); err != nil {
|
|
return result, err
|
|
}
|
|
if compose == nil && registry != nil && row.LeaseRoot != "" {
|
|
if err := registry.ReleaseNames(row.LeaseRoot, appLeaseNames(st)); err != nil {
|
|
return result, err
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if compose == nil {
|
|
return result, nil
|
|
}
|
|
elsewhere := false
|
|
if infra {
|
|
var err error
|
|
if elsewhere, err = ownedElsewhere(registry, row); err != nil {
|
|
return result, err
|
|
}
|
|
}
|
|
if infra && !elsewhere {
|
|
if err := compose(ctx, row.Project, opts.Volumes); err != nil {
|
|
return result, err
|
|
}
|
|
result.ComposeDown = true
|
|
}
|
|
if registry == nil {
|
|
return result, nil
|
|
}
|
|
if row.LeaseRoot != "" {
|
|
if err := registry.Release(row.LeaseRoot); err != nil {
|
|
return result, err
|
|
}
|
|
result.ReleasedLeases = len(row.Leases)
|
|
}
|
|
if row.Status == StatusOrphaned && (row.Project != "" || row.OwnerRecord != "") {
|
|
removed, err := removeOrphanOwner(registry, row)
|
|
if err != nil {
|
|
return result, err
|
|
}
|
|
result.RemovedOwner = removed
|
|
}
|
|
return result, nil
|
|
}
|
|
|
|
// liveInWorktree reports whether st has a live service under a checkout
|
|
// that still exists as a git worktree.
|
|
func liveInWorktree(st *State) bool {
|
|
if st.Root == "" || !pathExists(st.Root) || !inspectCheckout(st.Root).Worktree {
|
|
return false
|
|
}
|
|
for _, rec := range st.Services {
|
|
if rec.Skipped == "" && groupIsOurs(rec) {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
func removeOrphanOwner(registry *Registry, row StackRow) (bool, error) {
|
|
path := row.OwnerRecord
|
|
if path == "" {
|
|
path = ownerPath(registry, row.Project)
|
|
}
|
|
owner := readOwner(path)
|
|
if errors.Is(owner.Err, os.ErrNotExist) {
|
|
return false, nil
|
|
}
|
|
if owner.Err == nil && owner.StateDir != row.StateDir && pathExists(filepath.Join(owner.StateDir, stateFileName)) {
|
|
return false, nil
|
|
}
|
|
if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
|
|
return false, err
|
|
}
|
|
return true, nil
|
|
}
|
|
|
|
// projectLockDir is the project directory that guards row's owner record.
|
|
func projectLockDir(registry *Registry, row StackRow, infra bool) string {
|
|
switch {
|
|
case registry == nil:
|
|
return ""
|
|
case infra:
|
|
return filepath.Join(registry.Dir, projectsDirName, ProjectName(row.Project))
|
|
case row.Project == "" && row.OwnerRecord != "":
|
|
return filepath.Dir(row.OwnerRecord)
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// ownedElsewhere reports whether the project's owner record names another
|
|
// state that still claims the project, so row must not stop its containers.
|
|
func ownedElsewhere(registry *Registry, row StackRow) (bool, error) {
|
|
if registry == nil || row.Project == "" {
|
|
return false, nil
|
|
}
|
|
owner := readOwner(ownerPath(registry, row.Project))
|
|
if errors.Is(owner.Err, os.ErrNotExist) {
|
|
return false, nil
|
|
}
|
|
if owner.Err != nil {
|
|
return false, fmt.Errorf("project %s: the owner record cannot be read (%w); repair or remove it, then run 'ods stack ls'", row.Project, owner.Err)
|
|
}
|
|
if owner.StateDir == row.StateDir {
|
|
return false, nil
|
|
}
|
|
if _, err := os.Stat(filepath.Join(owner.StateDir, stateFileName)); errors.Is(err, os.ErrNotExist) {
|
|
return false, nil
|
|
} else if err != nil {
|
|
return false, fmt.Errorf("project %s: cannot check the owner state in %s: %w", row.Project, owner.StateDir, err)
|
|
}
|
|
st, err := (&Stack{StateDir: owner.StateDir}).ReadState()
|
|
if err != nil {
|
|
return false, fmt.Errorf("project %s: the owner state in %s cannot be read (%w); repair or remove it, then run 'ods stack ls'", row.Project, owner.StateDir, err)
|
|
}
|
|
return st != nil && st.Project == row.Project, nil
|
|
}
|