mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-19 01:50:51 -05:00
fix(craft): remove kindnet CPU throttling in local clusters (#15724)
This commit is contained in:
1 parent
9cbbc5a475
commit
939aa526f8
3 files changed
+35
No files matched your search
@@ -692,6 +692,10 @@ DNS/VPN into the cluster, and the vscode debugger attached to
|
||||
instead only when the change is in the docker sandbox backend or
|
||||
`sandbox-proxy` specifically (`SANDBOX_BACKEND=docker`); it is slower for
|
||||
general Craft work.
|
||||
`deployment/helm/dev/k8s-up.sh` removes Kindnet's CPU limit and waits for its
|
||||
rollout on both new and existing clusters. CPU requests and memory settings stay
|
||||
unchanged. This prevents the container's CPU quota from throttling network-policy
|
||||
processing and blocking sandbox egress during OpenCode startup.
|
||||
|
||||
### Manual reproduction
|
||||
|
||||
|
||||
@@ -85,6 +85,13 @@ if [[ "$CURRENT_CTX" != "$EXPECTED_CTX" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Kindnet handles network-policy packets; a CPU cap can stall sandbox egress.
|
||||
echo "removing the kindnet CPU limit ..."
|
||||
kubectl --context "$EXPECTED_CTX" -n kube-system patch daemonset kindnet \
|
||||
--type=strategic \
|
||||
-p '{"spec":{"template":{"spec":{"containers":[{"name":"kindnet-cni","resources":{"limits":{"cpu":null}}}]}}}}'
|
||||
kubectl --context "$EXPECTED_CTX" -n kube-system rollout status daemonset/kindnet --timeout=120s
|
||||
|
||||
# ---- 2. helm install / upgrade ----
|
||||
|
||||
if [[ "$SKIP_HELM" -eq 1 ]]; then
|
||||
|
||||
@@ -119,6 +119,9 @@ script is idempotent and refuses to run unless your kubectl context is
|
||||
per cluster. New clusters use the `kindest/node:v1.33.1` node image so Craft's
|
||||
native init sidecar pod shape is supported. Existing clusters are not recreated;
|
||||
set `KIND_NODE_IMAGE` to override the default for a newly created cluster.
|
||||
The script removes Kindnet's CPU limit so network-policy processing can use available CPU.
|
||||
It keeps the CPU request and memory settings, then waits for the rollout.
|
||||
This applies to existing clusters too, including runs with `--skip-helm`.
|
||||
|
||||
Watch pods (opensearch and CNPG-postgres take a minute or two on first boot):
|
||||
|
||||
@@ -388,6 +391,27 @@ before launching the api_server.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Sandbox egress stalls with Kindnet CPU throttling
|
||||
|
||||
Kindnet processes network-policy packets. Its default `100m` CPU limit can delay
|
||||
connections to the sandbox proxy and Kubernetes resource watches. OpenCode startup
|
||||
can then time out while installing plugins, before sending an LLM request.
|
||||
|
||||
Apply the local networking configuration without reinstalling Onyx:
|
||||
|
||||
```bash
|
||||
deployment/helm/dev/k8s-up.sh --skip-cluster-create --skip-helm
|
||||
```
|
||||
|
||||
Check the active CPU quota and throttling counters:
|
||||
|
||||
```bash
|
||||
kubectl --context kind-onyx-dev -n kube-system exec daemonset/kindnet -- \
|
||||
cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat
|
||||
```
|
||||
|
||||
`cpu.max` starts with `max` when the container has no CPU limit.
|
||||
|
||||
### VPN or proxy certificate errors
|
||||
|
||||
For `x509: certificate signed by unknown authority` or `CERTIFICATE_VERIFY_FAILED`,
|
||||
|
||||
Reference in new issue
Block a user