## Description
This PR removes two sources of CI infrastructure flakes.
**Terraform provider acceptance tests.** The `acceptance` job installed
Go but not Terraform. It did not set `TF_ACC_TERRAFORM_PATH` or
`TF_ACC_TERRAFORM_VERSION`. So terraform-plugin-testing (v1.16.0,
`internal/plugintest/config.go`) looked up the latest CLI on
checkpoint-api.hashicorp.com and downloaded it from
releases.hashicorp.com on every `resource.Test`. One bad response (502,
500, wrong zip content type, timeout) exited the test binary with
`failed to find or install Terraform CLI`. This caused about 24% of the
failures.
- Add the pinned `hashicorp/setup-terraform` step (same SHA as
`docs-check`) with an exact `terraform_version: "1.16.6"` and
`terraform_wrapper: false`. 1.11 is the floor for the write-only tests.
- Export `TF_ACC_TERRAFORM_PATH` from `command -v terraform`. With this
variable set, the library uses only that binary and makes no network
calls.
**dhi.io login.** `.github/actions/dhi-base-images` logged in with
`docker/login-action` once, with no fallback. A dhi.io outage (`503` on
`/v2/`) failed every image build.
- Replace the login with a bash step: `docker login dhi.io
--password-stdin`, 4 attempts, backoff of 15s, 30s, 45s. The token goes
in on stdin and never in an argument.
- Export the DHI build args only when the login succeeds.
- Add a `required` input (default `"true"`). When it is `"true"`, a
failed login fails the step, as before. When it is `"false"`, the step
logs a `::warning::` and the build uses the public base images, as fork
PRs already do (#13915).
- `build-model-server-image` gets a `dhi-required` input (default
`"true"`) and passes it through.
- Only `pr-playwright-tests.yml` (web image and model server image) and
`pr-integration-tests.yml` (model server image) set it to `"false"`.
`deployment.yml`, `cve-alerts.yml` and the nightly workflows stay
strict.
- The export step keeps its exact format, because
`.github/scripts/base-image-ref.sh` parses it.
## How Has This Been Tested?
- `pre-commit run` on the touched files: zizmor, actionlint, check-yaml
and ripsecrets pass.
- `shellcheck` on the new login script passes.
- Ran the login script locally with a fake `docker` that returns 503.
With `required=false` it retries 4 times, warns, writes
`logged-in=false` and exits 0. With `required=true` it exits 1. An
invalid `required` value exits 1.
- `.github/scripts/base-image-ref.sh web` still reads the digest from
the action file.
- This PR's CI runs the changed jobs: Terraform provider `acceptance
(bootstrap)` and `acceptance (api-key)`, and the Playwright and
integration image builds. The `Point the acceptance tests at the
installed Terraform CLI` step prints the Terraform version.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes two sources of CI flakiness: the Terraform provider acceptance job
no longer downloads the latest Terraform CLI on every test (a bad
response from HashiCorp's APIs failed the whole run), and the dhi.io
login now retries instead of failing every image build during an outage.
- The acceptance job installs Terraform 1.16.6 once with
`hashicorp/setup-terraform` and exports `TF_ACC_TERRAFORM_PATH`, so
`terraform-plugin-testing` uses that binary only and makes no network
calls.
- The `dhi-base-images` action retries `docker login dhi.io` 4 times
with backoff. A new `required` input (default `"true"`) keeps release
and CVE workflows strict; `pr-playwright-tests.yml` and
`pr-integration-tests.yml` set it to `"false"` so PR test images fall
back to the public base images with a warning instead of failing.
<sup>Written for commit 51501f59c5.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16332?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16332?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16332&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
- **Chores**
- Builds can optionally continue with public base images when private
base-image services are unavailable, with warnings instead of immediate
failure.
- Private base-image authentication now retries before applying the
configured fallback or failing the build.
- Terraform acceptance tests use a fixed CLI version and configured
executable path for more consistent results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Description
**Bug:** when a GitHub stack merges, each upper PR's merge event carries
its parent branch as `base.ref`. The `base.ref == 'main'` gate skips
them, so only the bottom PR gets a cherry-pick.
**Fix:**
- Drop the `base.ref` gate. Check the merge commits are on main through
the compare API.
- Read the PR's stack through GraphQL. Layers merged within 30s of each
other form one batch. Only the run for the top PR of a batch proceeds,
after waiting out the window.
- One `ods cherry-pick` call gets every SHA in the batch in stack order,
so the batch opens one PR. A conflict fails the whole batch and alerts
#cherry-pick-prs as today.
- A partially merged stack cherry-picks only the layers in that merge
batch.
- One checked layer picks the whole batch.
- Slack messages and the cherry-pick PR body show the stack.
- `workflow_dispatch` with only `pr_number` resolves the batch the same
way.
- Gate command failures now send the Slack failure alert.
## How Has This Been Tested?
- Ran the gate script locally against real PRs: Roshan's 6-layer stack
(#16366 to #16371), the lower layers of that stack, a partially merged
stack (#16322 to #16326), plain checked and unchecked PRs, a dispatch by
SHA, a disallowed merger, and a GraphQL error.
- Dispatched this branch's workflow with `pr_number=16371`. The gate
resolved all 6 PRs in order with `GITHUB_TOKEN`:
https://github.com/onyx-dot-app/onyx/actions/runs/39697078780. The
cherry-pick step failed because all 6 were already on release/v5.0 by
hand. `ods` fails on two empty cherry-picks in a row, which is a
separate `ods` bug.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
## Description
Craft development required repeated image builds, copied cluster
settings, and editor-specific Telepresence commands. Startup could
report success before services were usable.
- Share configuration resolution and launch commands between Make and VS
Code. Read deployed ConfigMaps and Secrets in memory, with personal
`.env.k8s` overrides.
- Add separate infrastructure and application readiness checks,
`craft-doctor`, and shared Telepresence connection/intercept handling.
- Reuse images, detect source changes, and provide checkout-scoped
cleanup previews. Add pause/resume and migration commands with
database-target checks.
- Default Kubernetes sandbox callbacks to the internal API Service.
Preserve explicit `ONYX_SERVER_URL` overrides and bump the Helm chart to
0.9.20.
- Update the local Kubernetes guide, Craft development skill, and
feature map.
- Give Helm chart CI a 100 GB runner disk. The previous runner exhausted
disk while unpacking OpenSearch.
- Upgrade the Storybook package family to 10.6.2 to fix the critical
audit finding GHSA-f2ph-cmg5-wxc8.
Existing `.env.k8s` files remain untouched; old service entries continue
to override chart settings. Image refresh updates the PodTemplate for
future sandboxes; existing sandbox pods retain their images.
## Review guide
[Read or download the Markdown review
guide](https://github.com/onyx-dot-app/onyx/blob/d3cbc42d89bfb150d1cfc815638645043c34b69e/docs/craft/dev/kubernetes-review-guide.md).
## How Has This Been Tested?
- Live kind and Telepresence: startup, intercept reuse, doctor, API and
worker hot reload, readiness success/failure, and migration
status/upgrade.
- Three real Craft smoke tests: sandbox provisioning, file
upload/readback, and web app preview through the frontend. No real LLM
calls were exercised.
- 76 Craft Helm tests and 27 development-tool regression tests passed.
- Pre-commit checks, explicit development-tool type checking, frontend
type coverage, and skill validation passed.
- The local web audit passes its critical-severity gate without
suppressions. The Storybook production build passes.
- Removed the disposable test cluster, local processes, and temporary
files.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Harden local Craft Kubernetes development by sharing configuration and
launch logic between Make and VS Code, and by checking service readiness
before startup reports success.
**What’s included**
- Resolve deployed ConfigMaps, Secrets, and container env in memory
through a shared launcher; personal `.env.k8s` values take precedence,
and resolved secrets never touch disk or terminal output.
- Add shared lifecycle commands (`craft-doctor`, staged
infrastructure/application readiness, Telepresence connect and
intercept, pause, resume, migration, cleanup previews) used by both Make
and VS Code.
- Reuse built images when source is unchanged, force a rebuild with
`--rebuild-sandbox-image`, and keep probe and command invocations
bounded while preserving actionable errors.
- Default sandbox callbacks to the internal API Service, preserving
explicit `ONYX_SERVER_URL` overrides; drop the now-unneeded validation
and bump the Helm chart to 0.9.20.
- Give the Helm CI runner a 100 GB disk, since Docker and kind each
store images and OpenSearch exhausted the default disk. Bump Storybook
tooling to 10.6.2 to patch a development server vulnerability.
- Update the local Kubernetes guide, Craft development skill, feature
map, and add a downloadable PR review guide.
**Validation**
- Passed 76 Craft Helm tests, 27 development-tool regression tests, and
new unit tests for the shared runtime and lifecycle CLI.
- Tested startup, readiness, Telepresence, hot reload, migrations, and
three Craft smoke tests against kind; no real LLM calls were exercised.
<sup>Written for commit d3cbc42d89.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16376?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16376?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16376&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
- **New Features**
- Added a unified local Kubernetes workflow for Craft, including API and
worker launching, connectivity, readiness checks, migrations,
pause/resume, diagnostics, cleanup, and disk management.
- Added sandbox image reuse, rebuilding, loading, and configurable image
selection.
- Sandbox callbacks now default to the internal API service URL, with
support for explicit overrides.
- **Documentation**
- Added guidance for local Kubernetes setup, configuration, lifecycle
management, troubleshooting, and verification.
- **Bug Fixes**
- Improved readiness validation and stopped sensitive generated
credentials from being printed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
The merge queue on `main` now runs only the jobs behind its 7 required
checks. Integration, Playwright and the Golang coverage snapshot move to
a post-merge run on `main`.
**Why**
- The queue merges as soon as the 7 required checks pass:
`quality-checks`, `playwright-required`, `database-tests`,
`backend-check`, `mypy-check`, `Jest Tests`, `required`. `required` and
`playwright-required` come from `merge-group.yml`, which reuses the PR's
verdict (#7080, tested-base #14682).
- 12 more workflows also ran on `merge_group`, and nothing waited for
them. They are ~27% of runs-on compute, and each queued PR started a
full CI run (~470 vCPU per group, 5 at a time). Those bursts fill the
vCPU quota and the spot pools.
- Last 14 days (640 merge groups, from ci.onyx.app): in the queue,
Integration failed 91 times, Ext-dep 81, Playwright 40, TF provider 29
and MCP 28. The PRs merged anyway, and none of it left `main` broken.
Push-to-main failures were only Connector Tests, Golang (3) and the Helm
release (1). So removing these runs does not lower protection; they
never gated a merge.
- Precedent: #11673 already moved Connector Tests out of the queue to PR
+ push-to-main, for the same reasons.
**Kept on `merge_group`** (required checks): `merge-group.yml`,
`pr-quality-checks`, `pr-database-tests`, `pr-python-tests`,
`pr-python-checks`, `pr-jest-tests`. The release-branch queue requires
only quality / database / backend / mypy, and release branches use their
own workflow files.
**Removed from `merge_group`:** mcp-compatibility, craft compose
integration, craft compose tests, craft k8s, desktop build,
external-dependency unit tests, golang, helm chart testing, storybook,
terraform provider, integration, playwright. Their PR triggers do not
change.
**Moved to push to `main`**
- **Integration:** full EE + MIT matrix. This keeps the MIT coverage
that only the queue ran (#5413). `reindex_port` is not run on main; PRs
(related changes) and the daily run still cover it.
- **Playwright:** full suite, the S3 visual-baseline update (only from a
fully green run, #8410), and the Okta job (informational on push).
- **Golang:** the coverage snapshot for `main` is published by the push
run instead of the queue run.
- **Concurrency:** pushes to `main` share one group and queue
(`cancel-in-progress: false` for push), with one run in progress and the
newest pending. A burst of merges gives about 2 runs, and every run
finishes. PRs still cancel older runs.
**Trade-off:** a conflict between two PRs that only integration or
Playwright catches shows up on `main` right after the merge, not before
it. That was already the case: nothing in the queue waited for these
jobs.
## How Has This Been Tested?
- `pre-commit` on the changed files: actionlint, zizmor and check-yaml
pass. All workflows parse. `merge_group` stays only in the 6 kept files.
- After merge, on ci.onyx.app: merge_group compute (Merge queue
dashboard) drops; queue time to merge is unchanged; post-merge
integration and Playwright runs appear on `main` (Workflows dashboard);
baselines and the golang snapshot keep updating.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
The merge queue now runs only its required checks. Twelve workflows
previously ran on `merge_group` without gating the merge and consumed
~27% of CI compute, so they are removed from the queue; Integration,
Playwright, and the Golang coverage snapshot move to post-merge runs on
`main`.
- Kept on `merge_group`: `merge-group.yml`, `pr-quality-checks`,
`pr-database-tests`, `pr-python-tests`, `pr-python-checks`, and
`pr-jest-tests`; all other workflows keep their PR triggers unchanged.
- Integration and Playwright now run the full suite on push to `main`;
Playwright also updates S3 baselines (only from a fully green run,
#8410) and runs the informational real-Okta job.
- The Golang coverage snapshot for `main` is published by the push run
instead of the queue run.
- Push-to-`main` runs share a concurrency group that queues instead of
cancelling, so a burst of merges still produces finished runs.
- Trade-off: conflicts only Integration or Playwright catch now surface
on `main` after the merge, not before — nothing in the queue waited on
those jobs either way.
<sup>Written for commit 43eef98437.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16430?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16430?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16430&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Chores**
* Updated automated validation workflows to run on pull requests,
scheduled jobs, manual runs, tagged releases, and selected branch pushes
instead of merge-queue events.
* Standardized change detection so path-based filtering applies to pull
requests, while scheduled, manual, and release runs continue with their
intended defaults.
* Improved push handling for the main branch, including integration and
browser test execution, baseline updates, coverage publishing, and run
cancellation behavior.
* Updated required-check messaging and workflow documentation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Description
Integration test shards spend ~23% of their time in "Install Python
deps" (`uv sync` inside the devcontainer, with no cache). Across all
shards that is ~14% of runs-on vCPU-minutes per day (~61k vCPU-min,
~$1.2k/mo). This PR caches the uv download cache for those shards.
- `install-python-deps` mounts a host uv cache dir into the devcontainer
(`UV_CACHE_DIR=/uv-cache`, `UV_LINK_MODE=copy`, since uv cannot hardlink
across bind mounts).
- The cache is stored in the RunsOn S3 cache (`runs-on/cache/restore` /
`save`), keyed on OS + arch + `uv.lock`, with a prefix restore key.
- Every caller restores it. Only `integration-tests` shard 0 saves, and
only on a cache miss (new `save-cache` input). So a `uv.lock` change
uploads once, not once per shard.
- Both steps skip on GitHub-hosted runners, so the 10 GB GitHub cache is
not used.
Scope: this PR was first a wider change that also moved small jobs to
GitHub-hosted runners. That part is dropped. It undid #15792 (Docker Hub
rate limits) and the ECR pull-through cache. Only the cache change is
left.
## How Has This Been Tested?
- `pre-commit` on the changed files: actionlint, zizmor and check-yaml
pass.
- CI on this PR: the first run misses the cache and shard 0 saves it.
Rerun the integration workflow and compare the "Install Python deps"
step time on a warm cache against main.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Description
Add 11 selectable embedding models and an upgrade-only model picker.
Fresh self-hosted installs use Granite. Existing deployments keep their
stored model until an admin starts a re-index. On startup, the API
server waits until the model server can embed with the self-hosted model
in use. If that model is not in its cache, the model server downloads it
first.
| Type | Model | Output dimensions |
| --- | --- | --- |
| Self-hosted | `ibm-granite/granite-embedding-97m-multilingual-r2`
(default, in the image) | 384 |
| Self-hosted | `voyageai/voyage-4-nano` | 2048 |
| Self-hosted | `nvidia/Nemotron-3-Embed-1B-BF16` | 2048 |
| Cohere | `embed-v5.0-pro`, `embed-v5.0-fast` | 2048 |
| Google | `gemini-embedding-2` | 3072 |
| OpenAI | `text-embedding-3-large`, `text-embedding-3-small` | 3072 /
1536 |
| Voyage | `voyage-4-large`, `voyage-4`, `voyage-4-lite` (added on main)
| 1024 |
The 16 older models are now **legacy**: nomic, E5, GTE-small, Cohere
v3/v4, Voyage v2 and older Gemini/Vertex models. Existing indexes on
them keep working. The API refuses a legacy model as a new target, but a
re-index with the current model is still allowed.
## Size of this PR
After review, I removed the parts that the feature does not need. The
behavior stays the same. The diff against `main` went from **80 files,
+9,855 / −504** to **69 files, +4,740 / −467**.
| Area | Files | Added | Removed |
| --- | --- | --- | --- |
| Backend source | 16 | 1,004 | 96 |
| Backend unit tests and test managers | 16 | 1,568 | 32 |
| Backend manual suites (opt-in, not in CI) | 7 | 617 | 0 |
| Web source | 6 | 617 | 255 |
| Web tests | 5 | 630 | 1 |
| Web messages (9 locales) | 9 | 225 | 72 |
| Deployment, CI and feature map | 10 | 79 | 11 |
The feature map (`.agents/feature-map/`) now describes the new selection
rules.
Removed:
- **The branch's own cloud probe.** All selectable cloud models now use
main's probe (`probe_embedding_dimension`), which Bifrost already used.
It now also sends the stored Vertex configuration, the Azure API version
and the deployment name. Before this change, the branch's probe ignored
the Vertex configuration, so a Google provider that uses Workload
Identity could not switch to `gemini-embedding-2`.
- **The Vertex fan-out.** Main already sends `gemini-embedding-2` texts
one at a time. The Vertex code is now the same as on main.
- **The `expected_dim` path** in the model-server loader, and the
startup warning for custom models with registry names. These protected
only one rare case; see **Custom models with a registry name** in [Known
limits](#known-limits).
- **The UI fallback** that tested a new key with a default model when
the key failed with the current model. The credential test now embeds
with the staged or current model of that provider. Only a provider with
neither uses its first selectable model.
- **Picker cards for current and staged bring-your-own models.** Main
does not show them either. The current model shows in the summary at the
top of the page.
- **Shared gates and fixtures for the manual suites.** Each suite has
one gate env var.
- **The screenshots in `docs/assets`.** The images below come from an
earlier commit of this PR.
- **Tests for removed code, and repeated test cases.** The unit tests
went from 3,842 to 2,667 lines.
### CPU speed and memory cards
All nine locales show these planning estimates:
| Model | Card speed | Recommended CPU RAM |
| --- | --- | --- |
| Granite 97M | Up to **200 docs / min / CPU core** | **4 GB** per model
server |
| Voyage 4 Nano | Up to **50 docs / min / CPU core** | **4 GB** per
model server |
| Nemotron 3 Embed 1B BF16 | Up to **5 docs / min / CPU core** | **8
GB** per model server |
Nemotron says **GPU recommended**. It runs on CPU too, but slowly. The
cards tell the admin that speed changes with the hardware and the
document size. The RAM value is for one loaded model. A re-index and the
rest of Onyx need more memory.
These values are planning estimates, not guarantees. The Nemotron
per-core value is an extrapolation from an earlier multi-thread run.
Document counts depend on chunking, and throughput does not always scale
linearly with cores. The RAM values are recommendations, not measured
minimum or peak values.
## Implementation
- **Registry.** `shared_configs/embedding_models.py` defines the model
names, dimensions, prefixes, normalization, status and pinned Hugging
Face revisions. A parity test compares it with the web registry.
- **Upgrade-only guard.** `_check_embedding_model_target` runs in
`set-new-search-settings`:
- The current model can always re-index, legacy models too. The
dimension must stay the same. In Onyx Cloud, `reduced_dimension` must
also stay the same, because tenants share the index of a model.
- A selectable model must use the registry settings (dimension,
normalization, prefixes).
- Other OpenAI, Cohere, Google and Voyage models, and legacy self-hosted
models, get `INVALID_INPUT`. The error lists the selectable models.
- Custom Hugging Face models, LiteLLM, Azure and Bifrost stay open.
- A self-hosted model gets `INVALID_INPUT` for a new
`reduced_dimension`, because the model server ignores it. A re-index
keeps a stored value.
- **Cloud probe.** Before a re-index to a selectable cloud model, the
API embeds a test text with the stored credentials and checks the
dimension. A bad key, region or model ID fails before indexing starts.
The probe runs before the PRESENT row lock, with no open transaction, as
main's Bifrost probe does.
- **Self-hosted loads.** New models load at a pinned revision with
`trust_remote_code=False`, a dimension check and a finite-vector check.
CPU-only hosts load FP32. CUDA and MPS keep the checkpoint dtype. Legacy
and custom models keep main's `SentenceTransformer` call, so their
vectors do not change.
- **Voyage 4 Nano.** An Onyx loader adds bidirectional attention and the
learned 1024-to-2048 projection. The generic loader with remote code
disabled does not apply them.
- **Model server.** All models load from the cache first, then from
Hugging Face. Loads run off the event loop on two load workers, so at
most two sets of weights (PRESENT and FUTURE) load at the same time.
Registry models get 32 tokens of headroom for the prefix. The RoPE
pre-warm now runs again only for a longer context. On main, it ran for
every change of context length.
- **Image.** The model-server image bundles pinned Granite. It no longer
bundles nomic: nomic downloads on first use. The startup cache merge
keeps the models that a volume already has.
- **Picker.** Self-hosted and cloud groups, a Legacy label on the
current legacy model, and an **Advanced** section for custom Hugging
Face models, LiteLLM, Azure and Bifrost. A staged custom model keeps the
custom provider, also when its name is the same as a registry model.
- **Error detail.** `EmbeddingModel` now keeps the model server's error
detail. Before, every HTTP failure showed "HTTP error occurred -
response is None".
### Startup wait for self-hosted models
On startup, the API server waits until the inference model server can
embed with the current self-hosted model. The model server loads the
model on the first request, and downloads it first if it is not cached.
Example: a deployment that upgrades while it uses nomic, which the new
image does not bundle.
- The API server logs the wait, each failed attempt with its cause, and
the time when the model is ready. The model server logs a download.
- `LOCAL_EMBEDDING_MODEL_STARTUP_TIMEOUT` sets the wait (default 1800
seconds). When the wait runs out, startup fails. The error names the
model server, the model, the last error and the fix. `0` disables the
wait.
- The check request runs in a daemon thread. A request that still waits
at the deadline (for example, during a slow download) therefore does not
keep the failed process alive. Before this fix, the process stayed alive
until the model server read timeout (600 s by default).
- There is no wait in Onyx Lite (`DISABLE_VECTOR_DB`), for cloud models,
in multi-tenant deployments, in integration tests, or with
`DISABLE_MODEL_SERVER`.
- The wait uses the existing embed endpoint, so it also works with an
older model server during a rolling upgrade.
- This replaces the startup call to `warm_up_bi_encoder`. `SKIP_WARM_UP`
(default `true`) skipped that call, so it did not run by default.
- Only the inference model server is checked. The indexing model server
still loads a model on first use. In Compose, the API server does not
know the host of the indexing model server.
- `env.template` and Helm `values.yaml` document the setting. The Helm
chart is now 0.9.19.
Live test at commit `0eb5881da9`, with an existing nomic deployment on a
local stack. The size reduction did not change the wait:
| Case | Result |
| --- | --- |
| nomic not cached, model server online | The model server downloaded
nomic (523 MB). Startup continued after 19 s. |
| nomic cached | Ready in 3 s, with no download. |
| Model server starts after the API server | A warning every 10 s, then
ready at 33 s. |
| nomic not cached, no internet access | Each warning showed the cause
("not in the model server's Hugging Face cache and could not be
downloaded"). Startup failed at the timeout, and the process exited. |
| Onyx Lite, no model server | No wait. |
| Timeout `0` | No wait. The model loads on first use. |
## Upgrade notes
- **Offline deployments that use nomic.** The new image does not bundle
nomic. A deployment with no internet access and no kept model cache
cannot download it. The API server then fails at startup with a clear
error, instead of starting with a search that cannot embed queries.
Before you upgrade such a deployment, add the current model to the model
server's Hugging Face cache, or keep the cache on a volume. Helm has no
cache volume by default. This PR has no schema migration, so a return to
the previous image works while the current model stays the same.
- **Helm.** The chart has no Hugging Face cache volume for the model
servers, so a model that the image does not bundle (nomic, Voyage 4
Nano, Nemotron) downloads again on every pod start. `values.yaml`
documents an opt-in volume with the existing
`inferenceCapability.volumes` and `volumeMounts` values. A default
volume is not added: a ReadWriteOnce claim hangs rolling updates across
nodes, breaks `replicaCount` above 1, and needs a StorageClass. The
bundled Granite works with or without a volume.
- **API probes.** `/health` answers only after the startup wait. A
liveness probe needs a startup probe that allows the migrations plus
`LOCAL_EMBEDDING_MODEL_STARTUP_TIMEOUT`. The api probe example in
`values.yaml` shows one.
- **Multi-tenant template snapshots.** Migration `dbaa756c2ccf` seeds a
search settings row with `DOCUMENT_ENCODER_MODEL`. When that variable is
empty, the seed is now Granite, not nomic. If a deploy runs
`alembic/check_tenant_snapshot.py`, it reports a difference in
`search_settings` until the template snapshots are built again
(`--snapshot-template`). To keep the seed the same as on main, set
`DOCUMENT_ENCODER_MODEL=nomic-ai/nomic-embed-text-v1`.
- **No return to a legacy model.** After a switch, the old index is
cleaned up and the upgrade-only rule blocks a return to a legacy model.
Keep a compatible backup before you switch.
## Audit: retries and failures
| Layer | Behavior | Compared with main |
| --- | --- | --- |
| Local encode | Catches `RuntimeError`, waits 0.1 seconds, then tries
again: **up to four encode attempts**. It also retries errors that do
not go away, such as OOM. | Unchanged. |
| API to model server | Passage requests retry request failures three
times with five-second waits. HTTP 429 allows ten attempts with
ten-second waits. Connect/read timeouts are 30/600 seconds. Query
requests do not get these retries. | Unchanged. |
| Document indexing | A failed batch falls back to one document at a
time after a two-second delay. Failures are recorded, and the other
documents continue. | Unchanged. |
| Cloud embedding | Authentication errors are separate. Errors mapped to
`RuntimeError` get exponential backoff with jitter: eight attempts in
indexing-only mode, else two. | Unchanged. `gemini-embedding-2` sends
one text per request, in sequence, as on main. |
| Model loading | Cache first, then online. A partial cache falls back
online. A failed load is not cached, so a later request can try again.
Requests for the same model and context length share one load job. |
Cache-first lookup is new for all models. Registry models add pinned
revisions and checks. |
Model downloads run off the event loop. When a waiting request is
cancelled, the shared load continues. A load job has no deadline, so it
can continue after an HTTP timeout.
One gap from main remains: the response JSON is parsed outside the retry
wrapper. See [local
retries](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/model_server/encoders.py#L157),
[request
retries](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/natural_language_processing/search_nlp_models.py#L1322),
and [cloud
retries](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/natural_language_processing/search_nlp_models.py#L942).
## Audit: migration and cancellation
There is **no schema migration, and no automatic change to existing
embedding settings**. Granite is the fresh-install default. Cloud tenant
provisioning still uses Cohere v3. Existing stored prefixes and
dimensions stay in use.
1. The API checks the target with the upgrade-only guard.
2. A new cloud target embeds a test text with the stored credentials
before the migration. The probe checks the dimension. It runs before the
PRESENT row lock.
3. The API locks PRESENT and refuses overlapping migrations. It also
refuses an index name that is live or still waiting for cleanup. FUTURE,
the first work and the cleanup consent commit together.
4. The existing port flow reads stored documents from the current index,
embeds them again and writes the new index. Connectors do not fetch
again. All-connectors mode includes paused connectors. Active-only mode
leaves them out and needs consent for the data that it leaves out. User
files are included.
5. Normal promotion waits for successful required ports, no open writers
or sync work, and a sample check in the new index. A failed or paused
required port blocks promotion. The sample check does not check the full
corpus.
6. Port batches retry up to five times. The cursor moves after a
successful batch. The first watchdog retry is immediate. Repeated
failures at the same cursor use exponential backoff from 30 seconds, up
to one hour. Five failures in sequence at that cursor pause the unit. An
operator can resume it.
7. A cancel of FUTURE keeps PRESENT active, stops port work at a batch
boundary, and removes the partial index after writers stop. **Instant
switch promotes first**, so search is not complete during backfill, and
this cancel path is not available.
See [migration
endpoint](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/server/manage/search_settings.py#L140),
[upgrade-only
guard](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/server/manage/search_settings.py#L456),
[swap
gates](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/db/swap_index.py#L367),
and
[cancellation](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/server/manage/search_settings.py#L668).
## Known limits
- **Model memory.** Two load workers limit how many models load at the
same time. Loaded models are not evicted (as on main), so the old and
new models stay in memory during a switch. Helm's default memory limit
for the indexing model server is **6 GiB**, below the **8 GB** that the
Nemotron card recommends. See [load
executor](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/model_server/encoders.py#L35)
and [Helm
limits](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/deployment/helm/charts/onyx/values.yaml#L309).
- **Legacy models are not selectable.** This is by design. The current
legacy model stays visible and can re-index. After a switch, the
standard flow cannot select it again.
- **Bring-your-own options are under Advanced.** Unknown model names for
the built-in OpenAI, Cohere, Google and Voyage providers get
`INVALID_INPUT`. LiteLLM, Azure, Bifrost and custom models stay open.
- **The cloud probe reads the provider settings once.** If an admin
changes the key or region while a re-index starts, the probe can use the
old values. A key can also change at any time during a re-index, so no
preflight check can guarantee the key that the workers use. Indexing
failures then show in the re-index status, and the admin can cancel.
- **Custom models with a registry name.** Before this PR, an admin could
add a registry model name, such as `voyageai/voyage-4-nano`, as a custom
model. The model server now loads that name as the registry model. For
`voyage-4-nano`, the vectors then have 2048 dimensions, not 1024, so a
1024-dimension index fails with an error. Re-index to the registry model
to fix it. For the other registry models, a generic load gives the same
dimension.
- **Shared `max_seq_length`.** The model server changes `max_seq_length`
on the shared model for each request. Two requests with different
context lengths can overlap. This is the same on main. This PR does not
change it.
- **Voyage 4 Nano depends on the Transformers and SentenceTransformers
layout.** Run the real-weight suite when those dependencies change.
- **No reduced dimensions for Cohere v5 and the self-hosted models.**
Cloud provisioning stays on Cohere v3. These are follow-ups.
- **The cancel dialog has two buttons labeled Cancel.** One closes the
dialog, the other cancels the re-index. This is the same on main.
- The branch is rebased onto `main` at `ad9c0030e9`. This PR does not
claim better retrieval quality.
## Validation for this update
- **Backend: 15,343 unit tests passed** before the review fixes. I left
out the EE license tests, because they write a key to a live Redis. `ty`
and pre-commit passed. After the review fixes and the merge of `main`,
the PR's backend unit tests passed again (199 tests). Pre-commit passed
on all 69 changed files.
- **Startup exit check:** a fake model server that never answers, with a
3 s wait and a 20 s read timeout. The wait raised at 3.3 s. The process
exited at 5 s with the fix, and at 22 s without it.
- **Web: type check, lint and 3,236 Jest tests (307 suites) passed.**
- **Self-hosted real-weight suite: 6 of 6 passed.** Golden scores and
the bidirectional-attention check for Granite, Voyage 4 Nano and
Nemotron.
- **Cloud live suite: `gemini-embedding-2` passed.**
- **End to end on a local stack with branch code: 11 of 11 passed.** The
API refused all 6 legacy self-hosted targets and accepted a same-model
re-index. Then the stack switched to Granite, Voyage 4 Nano, Nemotron
and `gemini-embedding-2`, in sequence. After each switch, a pure vector
search found each of 5 seeded facts first.
- **Not run for this update: Cohere, OpenAI and Voyage cloud tests.** A
TLS proxy on the test network blocked `api.cohere.com` and
`api.openai.com`. The probe then refused the switch with `400`, as
designed. These tests passed on an earlier commit of this PR, and the
probe that they use is main's probe. I had no Voyage key.
- **CI:** at `9321cb51db`, every check passed except the Dependency
Audit: Playwright, integration, acceptance, database,
external-dependency unit, Jest and quality checks. The audit fails on
open Dependabot alerts for Pillow and certifi in other manifests
(`loadtest/uv.lock` and the Craft sandbox requirements). Other open PRs
fail the same way at the moment.
- **Not run for this update:** speed benchmarks.
<details>
<summary>Earlier validation in this PR, not run again for this
update</summary>
- 28 real-weight tests and 24 cloud tests passed.
- End-to-end switches to all eight selectable models of that time
passed. The API refused all 16 legacy targets.
- Legacy-upgrade equivalence, Docker cache keeping and multi-tenant
provisioning checks passed.
- 11,521 backend unit tests passed. 10 EE license failures also occurred
on main.
</details>
## Voyage retrieval-quality check — October 14, 2026
The branch's Voyage loader passed a local EnterpriseRAG-Bench loading
check. **Vector-only mean document Recall@10 was 91.85%**, above the
proposed 50% threshold. This supports a correct load. It does not prove
numerical equality with the upstream implementation.
The corpus had **5,000 documents**: all **722 labeled supporting
documents** and **4,278 random distractors**, sampled with seed
`20261013`. All **470 questions with document labels** were scored. The
30 questions with no labels were left out. No question or answer text
went into the corpus.
| Retrieval mode | Mean Recall@1 | Mean Recall@5 | Mean Recall@10 | Mean
Recall@20 |
| --- | --- | --- | --- | --- |
| Onyx content-vector-only retrieval | 66.13% | 87.37% | **91.85%** |
94.16% |
| Onyx hybrid search API | 75.85% | 93.32% | **96.37%** | 97.70% |
Recall is the fraction of labeled documents found for each question,
averaged over the questions. Rankings use the first hit of each
document. Vector-only hit rate@10 (at least one relevant document) was
94.26%. That is a different metric from recall.
| Question category | Questions | Vector-only Recall@10 | Hybrid
Recall@10 |
| --- | --- | --- | --- |
| Basic | 175 | 97.71% | 99.43% |
| Semantic | 125 | 84.80% | 92.80% |
| Intra-document reasoning | 40 | 97.50% | 100.00% |
| Project related | 40 | 86.43% | 95.35% |
| Constrained | 30 | 98.33% | 98.33% |
| Conflicting information | 20 | 90.00% | 97.50% |
| Completeness | 20 | 68.03% | 79.03% |
| Miscellaneous | 20 | 100.00% | 100.00% |
The check used CPU FP32, 2048-dimensional normalized vectors, and the
registry's query and document prefixes. MPS and CUDA were off. There was
no query expansion, reranking, or LLM document selection. All 5,000
document IDs matched the saved manifest, with 15,612 indexed chunks. All
query vectors were finite and had unit length. The last indexing attempt
had no document errors. Two earlier attempts stopped after
stale-heartbeat timeouts and continued from saved progress.
**Search path:** on this OpenSearch branch, `hybrid_alpha=1` still runs
hybrid retrieval, so the API result includes keyword scores. The
vector-only result calls `OpenSearchDocumentIndex.semantic_retrieval`
directly, with content vectors only. Keyword matches therefore cannot
hide a broken embedding loader in that result.
**Scope:** this is an easier 5,000-document subset that contains all
gold documents, not the full 511,958-document benchmark. Do not compare
these values with full-corpus leaderboard scores. No loader or ranking
code changed to get these results.
Revisions: Onyx `913bfa8d7a32c9cda340df92f42d0f706fd424c2`;
EnterpriseRAG-Bench `d36685e273713975ee20299bbf1ab64165575b3c`; Voyage
`67fabc9bef010dabc5f6024aa1b1b6b93410426f`. Runtime: PyTorch 2.9.1,
Transformers 5.14.1, SentenceTransformers 5.4.1.
## UI screenshots
From the local development stack at commit `98bfc70080`, before the size
reduction. These show the real flow, not mockups.
<details open>
<summary>Self-hosted cards and current model</summary>
</details>
<details>
<summary>Cloud model catalog</summary>
</details>
<details>
<summary>Staged Granite change and migration strategy</summary>
</details>
<details>
<summary>Re-index in progress</summary>
</details>
<details>
<summary>Cancel confirmation</summary>
</details>
<details>
<summary>Advanced custom model setup</summary>
</details>
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [ ] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Description
**Evidence.** `test_slack_permission_sync` fails in CI in two ways:
- `assert "Sara's favorite number ..." not in [...]`: user_1 still sees
the private message after the kick and perm sync.
- `TimeoutError: Permission sync was not completed within 300 seconds`:
about 9 stack PRs ran the Slack job in the same minute on one Slack app.
Every Slack call got a 429 with retry-after=30.
Also, a failed first attempt can never get a full second attempt. The
test step retries 3 times with 20 minutes each, but the job stops after
25 minutes.
**Cause.**
- `CCPairManager.wait_for_sync` waited until *any* N docs had
`last_synced >= last_modified >= after`. The public doc could fill that
count while the private doc's index update was still waiting for the
next doc-sync beat. The search then saw the old ACL.
- The Slack channel pool allowed 20 slots, so it never limited
concurrency.
- The job `timeout-minutes` did not fit the retry budget.
**Fix.**
- `wait_for_sync` takes an optional `doc_ids`. The index wait now also
waits until each listed doc is synced. The timeout error names the
missing docs. Other callers do not pass it, so their behavior does not
change.
- `SlackManager.add_message_to_channel` returns the Onyx doc id of the
posted message (built with the connector's `_build_doc_id`). The test
passes the private message's doc id to the post-kick wait. The earlier
"user_1 can see it" check does not have this problem: indexing writes
the ACL, and the test waits for indexing to complete.
- Slack channel pool: `_MAX_POOL_SLOTS` 20 → 3, so extra runs queue for
a slot. A slot is held for one test (usually 1-3 minutes), so the
10-minute claim timeout queues a run behind several full turns of the
pool. The busy-slot poll goes from 15 s to 30 s, because each poll reads
the lock channel with `conversations.history`, which the running tests
also use. Slots 3-19 stay unused. The sweep does not touch pool
channels.
- Workflow: the `integration-tests` matrix job timeout goes from 25 to
70 minutes (3 × 20 min attempts, plus setup of about 3-6 min and log
upload). This keeps the 20-minute attempt: the slowest suites take about
12 minutes, and the Slack suite can now wait for a pool slot. One job
definition serves the full matrix, so this also covers `reindex_port`
and the other connector suites. The `onyx-lite-tests` job had the same
mismatch (3 × 20 min in a 10-minute job, for a 2-minute suite). It now
uses 3 × 8 min attempts in a 30-minute job.
The workflow file and the Slack test dir are in the `slack` paths
filter, so this PR runs the Slack job.
## How Has This Been Tested?
- A local check with mocked API calls ran `wait_for_sync`. With
`doc_ids`, it waited until the private doc was synced. Without
`doc_ids`, it returned as soon as the public doc was synced, which is
the old flaw. On timeout, the error lists the missing doc id.
- pre-commit passed on the changed files (ruff, ty, actionlint, zizmor).
- I did not run the live Slack test locally: it resets the shared local
stack. CI runs the Slack job on this PR.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the flaky Slack permission sync test that failed when the public
doc synced before the private doc's ACL change, and when concurrent
Slack runs hit API rate limits.
- `wait_for_sync` now accepts `doc_ids` to wait until each specified doc
is synced; the timeout error lists missing docs.
- `add_message_to_channel` returns the Onyx doc id of the posted message
via the now-public `build_slack_doc_id`, so the test can wait on the
private doc.
- Slack channel pool slots dropped from 20 to 3 to queue concurrent
runs, and claims are released on any failure so slots free before the
TTL.
- Integration job timeouts now fit all retry attempts (70 min for the
main matrix, 30 min for onyx-lite); the lite suite's retry timeout drops
from 20 to 8 min.
<sup>Written for commit 5dcb65da6d.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16388?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16388?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16388&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
- **Tests**
- Improved integration test validation to confirm specific Slack
documents are indexed before completion.
- Enhanced coverage for Slack permission synchronization in private
channels.
- Updated test timing and concurrency settings to better accommodate
longer-running integrations and service rate limits.
- **Chores**
- Adjusted integration test timeouts and retry behavior to reduce
premature failures during longer-running test suites.
- Improved cleanup handling when Slack synchronization attempts are
interrupted or encounter errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
<!-- claude-slack-attribution -->
_Requested by **Nikolas Garza** · [Slack
thread](https://onyx-company.slack.com/archives/C0C5SMAM4TT/p1792538621965719?thread_ts=1792538621.965719&cid=C0C5SMAM4TT)_
Before: The CVE Alerts fix bot could edit only `package.json` and
`pyproject.toml` (plus regenerated lockfiles). It could not fix a pin in
`initial-requirements.txt`, so the audit kept failing.
After: The bot can also edit compiled requirements files
(`*requirements*.txt`, today only `initial-requirements.txt`). It still
rejects source code, workflows, and all other files.
This lets the bot fix a pin in any manifest file that the audit can fail
on and that the workflow can safely validate.
How: PR #16337 missed a certifi pin in `initial-requirements.txt`, and
#16361 fixed it by hand. The change adds `**/*requirements*.txt` to the
editable file list and to the patch allowlist in `cve-alerts.yml`.
`cve-check-manifests.ts` now checks `.txt` edits: only the package's
`==` line may change, and only to the fixed version. `cve-fix.md` tells
the bot to edit that line and keep the header and `# via` comments.
`backend/requirements/*.txt` stay workflow-regenerated and off-limits to
the bot. Go and Rust alerts stay out of scope, because the bot has no
regeneration step for them.
Linear: none
---
_Generated by [Claude
Code](https://claude.ai/code/session_01E2K27DFf4J3iAGXYpe4xtR)_
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Lets the CVE Alerts fix bot edit compiled requirements files so it can
fix a pin in `initial-requirements.txt` instead of leaving the audit
failing.
- Requirements files are validated line by line: only the package's
plain `==` pin may change to the fixed version, byte-identical
elsewhere.
- `backend/requirements/*.txt` stay workflow-regenerated and off-limits;
source code, workflows, and other files are still rejected.
<sup>Written for commit 17a29722b7.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16362?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16362?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16362&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
- **Bug Fixes**
- CVE checks now include supported Python `requirements*.txt` files
alongside existing dependency manifests.
- Dependency updates are validated line by line to require exact fixed
versions while preserving file structure and metadata.
- Unsupported requirement syntax and unsafe version changes are rejected
during validation.
- **Documentation**
- Dependency-fix guidance now covers compiled requirements files and
preserves generated metadata during updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
<!-- claude-slack-attribution -->
_Requested by **Nikolas Garza** · [Slack
thread](https://onyx-company.slack.com/archives/C0C5SMAM4TT/p1792538621965719?thread_ts=1792538621.965719&cid=C0C5SMAM4TT)_
## Description
Before: When the only fix for an advisory was a new major version, the
CVE Alerts workflow changed nothing. It left the issue open for a manual
bump.
After: In that case the workflow bumps to the lowest fixed release of
the lowest major that has one, and starts the note with "MAJOR bump".
The reviewer then checks the PR for breaking changes.
This change lets the CVE Alerts workflow open a fix PR for a major
version bump when no fix exists on the current major line.
How: The rule lived only in the triage prompt,
`.github/prompts/cve-fix.md`, so only that file changes. The workflow
and `cve-check-manifests.ts` do not check major versions. The PR body
already includes Claude's note, so the "MAJOR bump" flag shows in the
PR. The motivating case is pillow 12.3.0 for GHSA-8g5g-8vmr-9782. The
only fix is 13.0.0, so the workflow changed nothing and left issue
#16335 (see PR #16342).
## How Has This Been Tested?
I checked that no other workflow or script code enforces the old rule. I
did not run the workflow. The `pre-commit` hooks could not run in my
environment because the `shellcheck_py` build failed.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01E2K27DFf4J3iAGXYpe4xtR)_
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Allows the CVE Alerts workflow to open a fix PR for a major version bump
when no fix exists on the current major line.
Previously, when the only fix for an advisory was a new major version,
the workflow changed nothing and left the issue open for a manual bump.
Now it bumps to the lowest fixed release of the lowest major that has
one and starts the note with "MAJOR bump" so the reviewer checks the PR
for breaking changes. This change only touches the triage prompt in
`.github/prompts/cve-fix.md`; no workflow or script code enforces the
old rule. The motivating case is pillow 12.3.0 for GHSA-8g5g-8vmr-9782,
whose only fix is 13.0.0.
<sup>Written for commit 4cffabf4c7.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16343?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16343?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16343&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Documentation**
* Updated vulnerability remediation guidance to prioritize fixed
versions within the newest pinned major release.
* Added guidance for selecting and documenting the lowest available
fixed release when a major-version change is required.
* Expanded unchanged-result reporting to cover cases where no fix is
available or a major upgrade is necessary.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Claude <noreply@anthropic.com>
## Description
The merge-queue check `check-tested-base.sh` follows the chain of open
PRs from a stacked PR's tested parent down to the target branch. It
stopped after 10 PRs. The connector-checks stack (#16156 → #16200) is 12
PRs deep, so the `required` and `playwright-required` jobs failed in the
merge queue although the chain reaches `main` ([run
39354693529](https://github.com/onyx-dot-app/onyx/actions/runs/39354693529/job/118406689215)).
This PR raises the limit to 30 and names it `MAX_CHAIN_DEPTH`. Each
level is one `gh pr list` call, so the cost stays small.
## How Has This Been Tested?
- Ran the edited `reaches_target` read-only against the live stack from
`feat/checks-linear` with `TARGET=main`. It follows all 11 links and
reaches `main`. With the old limit it stops at
`feat/checks-incomplete-status`, as in the failed run.
- `bash -n` and pre-commit (shellcheck) pass.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the merge-queue tested-base check so it follows chains of open PRs
up to 30 deep instead of stopping at 10, which caused 12-PR stacks to
fail even when their chain reached `main`. Raising the limit from 10 to
30 costs one `gh pr list` call per level, so it stays cheap.
<sup>Written for commit 43ca73ea21.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16303?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16303?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16303&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Increased the validation limit for resolving linked pull request
chains, allowing checks to process deeper chains.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
<!-- claude-slack-attribution -->
_Requested by **roshan desai** · [Slack
thread](https://onyx-company.slack.com/archives/C07V90R4GTX/p1792009768004209?thread_ts=1792009768.004209&cid=C07V90R4GTX)_
Before: an agent could start work on a stale branch or an old `main`, so
it read an out-of-date feature map (`.agents/feature-map/`) and missed
recent changes.
After: at session start, Claude Code and Codex fetch `origin/main` and
tell the agent to rebase on it only when the branch has merge conflicts
with `origin/main` or its merge base with `origin/main` is more than 3
days old. Otherwise the hook is silent. Keeping a branch current avoids
a much more involved conflict resolution later.
This adds a SessionStart hook that, when it fires, reports how far HEAD
is behind `origin/main`, why (conflicts, stale, or both) and which
`.agents/` files changed on main since the branch diverged. A branch
that already contains `origin/main`, or is only a few recent commits
behind without conflicts, gets no message.
How: `.agents/hooks/ensure_latest_main.py` runs `git fetch origin main`
(20 second limit, no credential prompts) and checks for conflicts with
`git merge-tree --write-tree HEAD origin/main` (exit 1) and for
staleness (merge-base commit date older than `_STALE_AFTER_DAYS = 3`),
and prints a message that the agent reads as context only if either
applies. It never rebases, pulls or touches the working tree, and it
always exits 0, so an offline or failed fetch only adds a note. It is
wired for the `startup` and `resume` sources in `.claude/settings.json`
and `.codex/hooks.json`. Other agents (for example Cursor, which has no
hook config in this repo) get the same rule as a line in `AGENTS.md`.
Set `ONYX_SKIP_MAIN_SYNC_CHECK=1` (or `true`, `yes`, `on`) to turn it
off; `0` or `false` leaves it on. It also repeats the check, fetching at
most once an hour, on later prompts.
## How Has This Been Tested?
External dependency tests
(`backend/tests/external_dependency_unit/scripts/test_ensure_latest_main_repos.py`,
run by the `scripts` group of the PR workflow) run the script against
throwaway git repos: up to date, fresh and clean (silent), merge base
older than 3 days (warns), conflicting (warns), a dirty or
untracked-only tree, an offline remote, malformed payloads,
skip-variable values, fetch throttling, and a directory that is not a
repo. All exit 0 and leave the working tree unchanged. A stubbed unit
test (`backend/tests/unit/scripts/test_ensure_latest_main.py`) covers
the ssh command handling. `ruff` and `mypy --strict` pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01TYt2dm3XFyAdzZygsG6CN7
---
_Generated by [Claude
Code](https://claude.ai/code/session_01TYt2dm3XFyAdzZygsG6CN7)_
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Agents now check at session start and on later prompts whether their
checkout conflicts with `origin/main` or is stale, so they stop working
from an outdated `.agents/feature-map/` and missing recent changes.
- Claude Code and Codex run a new hook that fetches `origin/main` and
tells the agent how to update, without ever pulling, rebasing, or
touching the working tree.
- The hook speaks only when the branch has merge conflicts with
`origin/main` (via `git merge-tree --write-tree`) or its merge base is
more than 3 days old; a branch that is merely a few recent commits
behind stays silent. It always exits 0, so offline or hanging fetches
degrade gracefully, and on prompts it fetches at most once an hour. Set
`ONYX_SKIP_MAIN_SYNC_CHECK` to a truthy value to disable it. When a
merge or rebase is already in progress, it tells the agent to finish or
abort it instead of starting another update.
- Other agents without hook config get the same rule in `AGENTS.md`, and
the suggested update command uses an explicit refspec so it works in
single-branch clones.
- Git-repo integration tests run under the external dependency test
group; the unit file keeps the one fully stubbed test.
<sup>Written for commit ad0de70fed.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16043?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16043?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16043&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added checks at session start and during later prompts to flag
branches that may conflict with `main` or have an outdated merge base,
with guidance for updating. Checks use the last known remote state if
fetching fails and do not change the working tree.
* Prompt-triggered fetches are limited to once per hour.
* **Documentation**
* Added guidance for keeping work up to date while preserving
uncommitted changes.
* **Tests**
* Added coverage for up-to-date, stale, conflicting, offline, and other
branch scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
