diff --git a/.agents/feature-map/components/craft-sessions.md b/.agents/feature-map/components/craft-sessions.md index 2b9816c015..cccbe069e0 100644 --- a/.agents/feature-map/components/craft-sessions.md +++ b/.agents/feature-map/components/craft-sessions.md @@ -692,6 +692,10 @@ DNS/VPN into the cluster, and the vscode debugger attached to instead only when the change is in the docker sandbox backend or `sandbox-proxy` specifically (`SANDBOX_BACKEND=docker`); it is slower for general Craft work. +`deployment/helm/dev/k8s-up.sh` removes Kindnet's CPU limit and waits for its +rollout on both new and existing clusters. CPU requests and memory settings stay +unchanged. This prevents the container's CPU quota from throttling network-policy +processing and blocking sandbox egress during OpenCode startup. ### Manual reproduction diff --git a/deployment/helm/dev/k8s-up.sh b/deployment/helm/dev/k8s-up.sh index 982ae6eab6..d29e11dbe0 100755 --- a/deployment/helm/dev/k8s-up.sh +++ b/deployment/helm/dev/k8s-up.sh @@ -85,6 +85,13 @@ if [[ "$CURRENT_CTX" != "$EXPECTED_CTX" ]]; then exit 1 fi +# Kindnet handles network-policy packets; a CPU cap can stall sandbox egress. +echo "removing the kindnet CPU limit ..." +kubectl --context "$EXPECTED_CTX" -n kube-system patch daemonset kindnet \ + --type=strategic \ + -p '{"spec":{"template":{"spec":{"containers":[{"name":"kindnet-cni","resources":{"limits":{"cpu":null}}}]}}}}' +kubectl --context "$EXPECTED_CTX" -n kube-system rollout status daemonset/kindnet --timeout=120s + # ---- 2. helm install / upgrade ---- if [[ "$SKIP_HELM" -eq 1 ]]; then diff --git a/docs/craft/dev/local-kubernetes.md b/docs/craft/dev/local-kubernetes.md index a36d008619..4afdc070a9 100644 --- a/docs/craft/dev/local-kubernetes.md +++ b/docs/craft/dev/local-kubernetes.md @@ -119,6 +119,9 @@ script is idempotent and refuses to run unless your kubectl context is per cluster. New clusters use the `kindest/node:v1.33.1` node image so Craft's native init sidecar pod shape is supported. Existing clusters are not recreated; set `KIND_NODE_IMAGE` to override the default for a newly created cluster. +The script removes Kindnet's CPU limit so network-policy processing can use available CPU. +It keeps the CPU request and memory settings, then waits for the rollout. +This applies to existing clusters too, including runs with `--skip-helm`. Watch pods (opensearch and CNPG-postgres take a minute or two on first boot): @@ -388,6 +391,27 @@ before launching the api_server. ## Troubleshooting +### Sandbox egress stalls with Kindnet CPU throttling + +Kindnet processes network-policy packets. Its default `100m` CPU limit can delay +connections to the sandbox proxy and Kubernetes resource watches. OpenCode startup +can then time out while installing plugins, before sending an LLM request. + +Apply the local networking configuration without reinstalling Onyx: + +```bash +deployment/helm/dev/k8s-up.sh --skip-cluster-create --skip-helm +``` + +Check the active CPU quota and throttling counters: + +```bash +kubectl --context kind-onyx-dev -n kube-system exec daemonset/kindnet -- \ + cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat +``` + +`cpu.max` starts with `max` when the container has no CPU limit. + ### VPN or proxy certificate errors For `x509: certificate signed by unknown authority` or `CERTIFICATE_VERIFY_FAILED`,