mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-23 11:53:08 -05:00
## Description
Adds `ods stack ls` to list every Onyx dev stack on the machine from any
directory, and lets `stop`/`down --project NAME` act on a stack whose
worktree is gone. The PR also releases port leases when a stack stops
owning them.
**`ods stack ls [--json] [--all] [--orphaned]`**
(`internal/devstack/inventory.go`, `cmd/stack_ls.go`)
- It merges four sources, keyed by Compose project: `docker ps -a` +
`docker inspect` (labels, state, health), `projects/*/owner.json`, every
reachable `state.json` (owner state dirs, plus
`<git-common-dir>/{ods-stack,ods-stack-shared,ods-stack-frontend,worktrees/*/ods-stack,worktrees/*/ods-stack-frontend}`
from owner, lease, Docker and state roots), and `Registry.Leases()`. It
does not call `devstack.Open`, and it takes no locks.
- An Onyx Compose project is one with `relational_db` + `cache`
services, or one that a state or owner record claims.
- Statuses: `active` (an app process is alive, checked with
`groupIsOurs`, so a reused PID does not count), `app-stopped` (infra
runs, phase `stopped`), `infra-only` (infra runs, no app process: a
Compose-only project or dead processes), `stopped`, `orphaned`, and
`stale-lease` (`--all` only).
- Orphan rules: the Compose working dir is gone and no state claims the
project; a state's root is gone; or the root exists but is no longer a
git worktree (state rows, or Docker rows with an owner record whose
state is gone). A row counts as orphaned only when something is left to
clean up: processes, containers, leases or an owner record.
- A missing or corrupt state degrades its row (`errors`). When Docker is
down, `ls` prints a warning and shows rows from saved state only. Exit
codes: 0; 3 when `--orphaned` finds an orphan; 1 when no source can be
read.
**`down`/`stop --project NAME` and `prune`**
(`internal/devstack/project_down.go`)
- The command resolves the stack through `owner.json` → `state_dir`,
then any state that claims the project, then Docker labels. It runs from
any directory.
- It takes the state lock (only if the state dir still exists, so git
metadata is never recreated) and the project lock. It stops recorded
groups with `stopServices`/`stopRecords` and runs `docker compose -p
NAME down` from a neutral dir. Volumes are kept.
- After a successful down it releases the stack's leases and removes
`owner.json` for an orphan.
- It refuses a stack with a live service under an existing git worktree
unless you pass `--force`. If `--project` already matches the current
checkout's stack, the existing checkout path runs.
- `ods stack prune` runs this for every orphaned row after a
confirmation (`--yes` skips it). `--volumes` has its own confirmation,
and `--yes` does not skip it.
**Port lease fixes** (`ports.go`, `lifecycle.go`, `legacy.go`)
- `Stack.Stop` releases app leases (everything except `*_HOST_PORT`).
`Stack.Down` releases app leases after `stopServices`, and every lease
after Compose down. New `Registry.ReleaseNames`.
- Shared adoption (`releaseLegacyLeases`, runs after the legacy guard
and before `leasePorts`): each stopped legacy candidate without live
process groups loses all its leases, or only its app leases while its
containers run.
- Reclaim: `Lease.StateDir` is now recorded (`LeaseFor`). Another
owner's lease is reclaimable when its state is `stopped` or missing,
none of its recorded groups is alive, and, for `*_HOST_PORT` only, its
Compose project has no running container. That last check is lazy, so
app-port leasing stays fast. Leases without `StateDir` are reclaimable
only for shared/frontend keys, which are their own state dirs.
- Drift down: `choosePort` moves a lease to the lowest free port from
`Preferred` when the service is not running. I added
`PortRequest.Running` because `Fixed` alone is not set for a running
service that does not hold its port yet, and moving such a service
changed `web dev`'s full-stack owner
(`TestWebRuntimeUsesTheFullStackWebOwner` caught this).
Docs: `ods stack` help, `tools/ods/README.md`,
`tools/ods/integration/README.md`, and the onyx-dev-stack skill (a new
"Find and stop old stacks" step, and the updated port bullet).
Known limit: `git worktree remove` deletes
`<common>/worktrees/<name>/ods-stack`, so the recorded PIDs are lost.
`down --project` still removes the containers, leases and owner record,
but it cannot find app processes that outlived that removal.
**Discovery-first CLI.** Bare `ods stack` prints a short intro, then
groups its subcommands with the same `AddGroup` pattern as the root
command: Find stacks (`status`, `ls`, `logs`, `plan`), Run this
worktree's stack (`up`, `start`, `restart`, `stop`, `down`), and Clean
up (`prune`). Each `ls` row carries `next`, the exact command that stops
it, and the text output logs it with `log.Info`. Empty results and
refusals name the command to run.
Example on the VM (an isolated stack, then its worktree deleted with `rm
-rf`):
```
$ ods stack ls
PROJECT STATUS APP INFRA WEB AGE ROOT
demo-wt-fd863404 app-stopped - 7/7 - 1m ~/demo-wt
$ rm -rf ~/demo-wt && ods stack ls --orphaned # exit 3
PROJECT STATUS APP INFRA WEB AGE ROOT
demo-wt-fd863404 orphaned - 7/7 - 1m ~/demo-wt (missing)
level=info msg="To stop demo-wt-fd863404 (orphaned), run: ods stack down --project demo-wt-fd863404"
```
After `stack stop`, only the `*_HOST_PORT` leases remained. After `down
--project`, no containers or leases were left.
## How Has This Been Tested?
- Unit tests for the inventory merge (fake containers, owners, states,
leases, liveness): live, app-stopped, infra-only, Compose-only, orphan
with a missing root, a removed worktree whose directory remains, a stale
lease, Docker unavailable, unreadable sources, and sort order.
- Real-process tests for `DownProject`/`StopProject`: an orphan in a
deleted linked worktree is stopped, its leases are released and its
owner record is removed; a live worktree is refused without `--force`.
- Lease tests: reclaim of a stopped owner, live process group skipped,
non-stopped owner skipped, `*_HOST_PORT` with and without running
containers, drift down vs. running, `ReleaseNames`, stop/down releases,
adoption with and without a live legacy group, and the frontend handoff
releasing preview leases.
- Coverage gates: `cmd` is at 90.6% against a 90.4% floor, and
`internal/docker` at 99.3% against 99.2%.
- `go test ./internal/... ./cmd/` passes, except `TestRunCI_failures`,
which also fails on a clean main in this environment. golangci-lint
passes.
- New integration test `TestStackLiveOrphanedWorktree` (`-tags
integration`) passed on the VM (246 s). It starts an isolated stack in a
disposable worktree, deletes the worktree, asserts that `ls --orphaned`
lists the stack with exit 3, then that `down --project` stops its
processes, removes its containers and releases its leases.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `ods stack ls` to view active, stopped, orphaned, and
stale-lease stacks with service, health, port, age, and location
details.
* Added JSON output, filtering, orphan detection, and project-based
shutdown support.
* Added `ods stack prune` to safely clean up orphaned stacks, with
optional volume removal.
* **Bug Fixes**
* Improved port lease reclamation and reassignment for stopped stacks.
* `stop` now releases application ports while preserving infrastructure
ports; `down` releases all ports.
* **Documentation**
* Expanded lifecycle, recovery, listing, and cleanup guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: roshan <rohod04@gmail.com>
Co-authored-by: roshan <38771624+rohoswagger@users.noreply.github.com>
382 lines
12 KiB
Go
382 lines
12 KiB
Go
//go:build !windows
|
|
|
|
package devstack
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"os"
|
|
"path/filepath"
|
|
"reflect"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
|
|
"github.com/onyx-dot-app/onyx/tools/ods/internal/gittest"
|
|
)
|
|
|
|
func linkedWorktree(t *testing.T, root string) string {
|
|
t.Helper()
|
|
gittest.Git(t, root, "-c", "user.name=Test", "-c", "user.email=test@example.com", "-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "commit", "--allow-empty", "-m", "fixture")
|
|
dir := filepath.Join(t.TempDir(), "linked")
|
|
gittest.Git(t, root, "worktree", "add", "--detach", dir)
|
|
return dir
|
|
}
|
|
|
|
func sharedPair(t *testing.T) (*Stack, *Stack) {
|
|
t.Helper()
|
|
registry := t.TempDir()
|
|
fixture := newTestStack(t, registry)
|
|
linked := linkedWorktree(t, fixture.Root)
|
|
if err := os.MkdirAll(filepath.Join(linked, ".vscode"), 0o755); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
for _, name := range []string{"env_template.txt", "env.web_template.txt"} {
|
|
data, err := os.ReadFile(filepath.Join(fixture.Root, ".vscode", name))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if err := os.WriteFile(filepath.Join(linked, ".vscode", name), data, 0o644); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
first, err := Open(fixture.Root, "", Shared)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
second, err := Open(linked, "", Shared)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
first.Registry = &Registry{Dir: registry}
|
|
second.Registry = first.Registry
|
|
t.Cleanup(func() {
|
|
if err := first.stopServices(testOptions().StopTimeout); err != nil {
|
|
t.Error(err)
|
|
}
|
|
})
|
|
return first, second
|
|
}
|
|
|
|
func TestIsolatedWorktreesKeepTheirOwnDataAndState(t *testing.T) {
|
|
a, b := sharedPair(t)
|
|
first, err := Open(a.Root, "", Isolated)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
second, err := Open(b.Root, "", Isolated)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if first.Project == second.Project || first.StateDir == second.StateDir || first.StateDir == a.StateDir {
|
|
t.Fatal("isolated stacks share data or lifecycle state")
|
|
}
|
|
}
|
|
|
|
type handoffProvider struct {
|
|
*fakeProvider
|
|
previous ServiceRecord
|
|
}
|
|
|
|
func (p *handoffProvider) Migrate(ctx context.Context, env []string) error {
|
|
if isCurrent(p.previous) {
|
|
return errors.New("previous checkout is still running during migration")
|
|
}
|
|
return p.fakeProvider.Migrate(ctx, env)
|
|
}
|
|
|
|
func TestSharedUpTransfersCheckoutBeforeMigrationsAndRetainsPorts(t *testing.T) {
|
|
a, b := sharedPair(t)
|
|
provider := &fakeProvider{services: []string{"web", "api"}, infra: []PortRequest{{Name: "POSTGRES_HOST_PORT", Preferred: testBasePort + 600}}}
|
|
initial, err := a.Up(context.Background(), provider, testOptions())
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
st, err := a.ReadState()
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
previous := st.Services[0]
|
|
next, err := b.Up(context.Background(), &handoffProvider{fakeProvider: provider, previous: previous}, testOptions())
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if !next.Ready || next.Root != b.Root || !reflect.DeepEqual(initial.Ports, next.Ports) {
|
|
t.Fatalf("handoff lost ownership or ports: %+v", next)
|
|
}
|
|
fromOld, err := a.Status()
|
|
if err != nil || fromOld.Root != b.Root {
|
|
t.Fatalf("status hid active checkout: %+v %v", fromOld, err)
|
|
}
|
|
again, err := b.Up(context.Background(), provider, testOptions())
|
|
if err != nil || again.Services[0].PID != next.Services[0].PID {
|
|
t.Fatalf("same owner restarted unchanged services: %+v %v", again, err)
|
|
}
|
|
}
|
|
|
|
func TestForeignLifecycleCommandsCannotStopSharedOwner(t *testing.T) {
|
|
for _, action := range []string{"start", "restart", "stop", "down"} {
|
|
t.Run(action, func(t *testing.T) {
|
|
a, b := sharedPair(t)
|
|
p := &fakeProvider{services: []string{"web"}}
|
|
initial, err := a.Up(context.Background(), p, testOptions())
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
switch action {
|
|
case "start":
|
|
_, err = b.Start(context.Background(), p, testOptions())
|
|
case "restart":
|
|
_, err = b.Restart(context.Background(), p, testOptions())
|
|
case "stop":
|
|
err = b.Stop(context.Background(), testOptions())
|
|
case "down":
|
|
err = b.Down(context.Background(), p, testOptions())
|
|
}
|
|
if err == nil || !strings.Contains(err.Error(), a.Root) {
|
|
t.Fatalf("foreign %s was not rejected: %v", action, err)
|
|
}
|
|
after, err := a.Status()
|
|
if err != nil || !after.Ready || after.Services[0].PID != initial.Services[0].PID {
|
|
t.Fatalf("foreign command changed owner: %+v %v", after, err)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestSharedProjectSelectionPersists(t *testing.T) {
|
|
a, b := sharedPair(t)
|
|
a.Project = "existing-dev-data"
|
|
a.ProjectExplicit = true
|
|
p := &fakeProvider{services: []string{"web"}}
|
|
if _, err := a.Up(context.Background(), p, testOptions()); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
reopened, err := Open(b.Root, "", Shared)
|
|
if err != nil || reopened.Project != a.Project {
|
|
t.Fatalf("explicit project not retained: %+v %v", reopened, err)
|
|
}
|
|
if _, err := b.Up(context.Background(), p, testOptions()); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if b.Project != a.Project {
|
|
t.Fatal("concurrent stale opener changed project binding")
|
|
}
|
|
}
|
|
|
|
func TestFailedHandoffRetainsOwnerAndSkipsMigration(t *testing.T) {
|
|
a, b := sharedPair(t)
|
|
p := &fakeProvider{services: []string{"web"}}
|
|
if _, err := a.Up(context.Background(), p, testOptions()); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
st, err := a.ReadState()
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
actual := st.Services[0]
|
|
defer func() {
|
|
st.Services = []ServiceRecord{actual}
|
|
if err := a.WriteState(st); err != nil {
|
|
t.Error(err)
|
|
}
|
|
}()
|
|
st.Services[0].PID = -1
|
|
st.Services[0].StartMarker = ""
|
|
st.Services[0].Members = nil
|
|
if err := a.WriteState(st); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
next := &fakeProvider{services: []string{"web"}}
|
|
if _, err := b.Up(context.Background(), next, testOptions()); err == nil {
|
|
t.Fatal("handoff forgot an unverified live process group")
|
|
}
|
|
after, err := a.ReadState()
|
|
if err != nil || after.Root != a.Root || len(after.Services) != 1 || len(next.migrated) != 0 {
|
|
t.Fatalf("failed handoff lost owner or migrated: %+v %v", after, err)
|
|
}
|
|
}
|
|
|
|
func TestConcurrentSharedStartupConvergesToOneOwner(t *testing.T) {
|
|
a, b := sharedPair(t)
|
|
var wg sync.WaitGroup
|
|
errs := make(chan error, 2)
|
|
for _, stack := range []*Stack{a, b} {
|
|
wg.Add(1)
|
|
go func() {
|
|
defer wg.Done()
|
|
_, err := stack.Up(context.Background(), &fakeProvider{services: []string{"web"}}, testOptions())
|
|
errs <- err
|
|
}()
|
|
}
|
|
wg.Wait()
|
|
close(errs)
|
|
for err := range errs {
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
st, err := a.ReadState()
|
|
if err != nil || len(st.Services) != 1 || !isCurrent(st.Services[0]) || (st.Root != a.Root && st.Root != b.Root) {
|
|
t.Fatalf("shared startups did not converge: %+v %v", st, err)
|
|
}
|
|
}
|
|
|
|
func TestSharedBootstrapReusesSingleStoppedLegacyProject(t *testing.T) {
|
|
legacy := newTestStack(t, t.TempDir())
|
|
legacy.Project = "existing-data"
|
|
if err := legacy.WriteState(&State{Phase: PhaseStopped}); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
shared, err := Open(legacy.Root, "", Shared)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
shared.Registry = legacy.Registry
|
|
t.Cleanup(func() { _ = shared.Stop(context.Background(), testOptions()) })
|
|
if _, err := shared.Up(context.Background(), &fakeProvider{services: []string{"web"}}, testOptions()); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if shared.Project != legacy.Project {
|
|
t.Fatal("shared bootstrap created a fresh dataset")
|
|
}
|
|
if err := legacy.Down(context.Background(), &fakeProvider{}, testOptions()); err == nil {
|
|
t.Fatal("legacy shutdown could stop adopted shared infrastructure")
|
|
}
|
|
}
|
|
|
|
func TestSharedBootstrapRequiresChoiceBetweenLegacyDatasets(t *testing.T) {
|
|
a, b := sharedPair(t)
|
|
for _, root := range []string{a.Root, b.Root} {
|
|
legacy, err := Open(root, "", Isolated)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if err := legacy.WriteState(&State{Phase: PhaseStopped}); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
p := &fakeProvider{services: []string{"web"}}
|
|
if _, err := a.Up(context.Background(), p, testOptions()); err == nil || !strings.Contains(err.Error(), "several existing dev datasets") {
|
|
t.Fatalf("ambiguous data was selected: %v", err)
|
|
}
|
|
if p.composeUp != 0 || len(p.migrated) != 0 {
|
|
t.Fatal("ambiguous bootstrap changed infrastructure")
|
|
}
|
|
}
|
|
|
|
func TestStartCannotChangeStoppedProjectWithoutCompose(t *testing.T) {
|
|
a, _ := sharedPair(t)
|
|
if err := a.WriteState(&State{Phase: PhaseStopped}); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
a.Project = "different-data"
|
|
a.ProjectExplicit = true
|
|
p := &fakeProvider{services: []string{"web"}}
|
|
if _, err := a.Start(context.Background(), p, testOptions()); err == nil {
|
|
t.Fatal("start changed project while retaining old infrastructure ports")
|
|
}
|
|
}
|
|
|
|
func TestSharedWorktreesSelectSameInfrastructure(t *testing.T) {
|
|
a := newTestStack(t, t.TempDir())
|
|
bRoot := linkedWorktree(t, a.Root)
|
|
first, err := Open(a.Root, "", Shared)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
second, err := Open(bRoot, "", Shared)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if first.Project != second.Project || first.StateDir != second.StateDir {
|
|
t.Fatalf("worktrees must share project and lifecycle state: %+v %+v", first, second)
|
|
}
|
|
}
|
|
|
|
func TestProjectAliasesUseOneInfrastructureIdentity(t *testing.T) {
|
|
base := newTestStack(t, t.TempDir())
|
|
upper, err := Open(base.Root, "MY-DATA", Isolated)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
lower, err := Open(base.Root, "my-data", Isolated)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if upper.Project != lower.Project {
|
|
t.Fatal("Docker-equivalent names bypass infrastructure identity")
|
|
}
|
|
}
|
|
|
|
// stoppedLegacyStack starts an isolated stack in root and stops it the way
|
|
// an older ODS did: the processes end and the leases stay.
|
|
func stoppedLegacyStack(t *testing.T, root, project string, registry *Registry, services []ServiceRecord) (*Stack, map[string]int) {
|
|
t.Helper()
|
|
legacy, err := Open(root, project, Isolated)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
legacy.Registry = registry
|
|
status := mustUp(t, legacy, &fakeProvider{services: []string{"web", "api"}})
|
|
if err := legacy.stopServices(testOptions().StopTimeout); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
st, err := legacy.ReadState()
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
st.Services = services
|
|
if err := legacy.WriteState(st); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
return legacy, status.Ports
|
|
}
|
|
|
|
func TestSharedAdoptionReleasesStoppedLegacyLeases(t *testing.T) {
|
|
fakeContainersRunning(t, false)
|
|
a, b := sharedPair(t)
|
|
legacy, ports := stoppedLegacyStack(t, b.Root, "legacy-data", a.Registry, nil)
|
|
shared, err := Open(a.Root, "", Shared)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
shared.Registry = a.Registry
|
|
status := mustUp(t, shared, &fakeProvider{services: []string{"web", "api"}})
|
|
if shared.Project != legacy.Project {
|
|
t.Fatalf("shared stack did not adopt %s", legacy.Project)
|
|
}
|
|
if status.Ports["web"] != ports["web"] || status.Ports["api"] != ports["api"] {
|
|
t.Fatalf("shared stack got %v, want the adopted ports %v", status.Ports, ports)
|
|
}
|
|
if got := leaseNames(t, a.Registry, legacy.Root); len(got) != 0 {
|
|
t.Fatalf("adopted legacy stack kept leases: %v", got)
|
|
}
|
|
}
|
|
|
|
func TestSharedAdoptionKeepsLeasesOfLiveLegacyProcesses(t *testing.T) {
|
|
fakeContainersRunning(t, false)
|
|
a, b := sharedPair(t)
|
|
pid := livePGroup(t)
|
|
live, ports := stoppedLegacyStack(t, b.Root, "live-data", a.Registry, []ServiceRecord{{Name: "worker", Slug: "worker", PID: pid, PGID: pid}})
|
|
adopted, err := Open(a.Root, "legacy-data", Isolated)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if err := adopted.WriteState(&State{Phase: PhaseStopped}); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
shared, err := Open(a.Root, "legacy-data", Shared)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
shared.Registry = a.Registry
|
|
status := mustUp(t, shared, &fakeProvider{services: []string{"web", "api"}})
|
|
if status.Ports["web"] == ports["web"] || status.Ports["api"] == ports["api"] {
|
|
t.Fatalf("shared stack took leases of a legacy stack with live processes: %v", status.Ports)
|
|
}
|
|
if got := leaseNames(t, a.Registry, live.Root); got["web"] != ports["web"] || got["api"] != ports["api"] {
|
|
t.Fatalf("live legacy stack lost leases: %v", got)
|
|
}
|
|
}
|