## Description
A fresh hand-run Compose install no longer starts MinIO or
`object-store-copy`. Fixes
[ENG-4429](https://linear.app/onyx-app/issue/ENG-4429/a-fresh-manual-compose-install-still-starts-minio).
**Why**
- With the v5 compose file and `env.template` defaults, MinIO started on
a fresh install. Only onyx-cli set `MINIO_REPLICAS=0`.
- Even with it set, `object-store-copy` started and idled forever.
**What**
- `env.template` sets `MINIO_REPLICAS=0`, since a fresh install never
held files in MinIO.
- `object-store-copy` follows `MINIO_REPLICAS` like `minio` does.
- The compose default stays `1`, so an earlier release's `.env`, which
lacks the variable, keeps MinIO and the copy running.
- `env.prod.template` names the object store, the S3 key pair and
`MINIO_REPLICAS=0`. A fresh prod `.env` defaulted the app to MinIO
before.
- The retire step stops MinIO with `docker compose up -d --no-deps minio
object-store-copy`, since a plain `up -d` recreates every service that
loads `.env`.
- Not changed: Compose still creates the empty `minio_data` volume for a
zero-replica service. Only a profile split across the CLI, ods, CI and
installers would avoid it.
Docs: https://github.com/onyx-dot-app/documentation/pull/521
## How Has This Been Tested?
Live on scratch Compose projects with the v5.0.0-beta.2 images. A `.env`
copied from the new `env.template` ran only `object-store` and
`relational_db`, with no `minio` or `object-store-copy` container. An
upgraded `.env` without `MINIO_REPLICAS` ran `minio` and
`object-store-copy`, and the copy started. `docker compose config` shows
replicas 0 and 1 for the two cases. With `docker-compose.prod.yml` and a
fresh `env.prod.template` plus keys, `api_server` points at
`object-store` and `minio` and `object-store-copy` are at 0, and empty
keys still refuse to start. `go test ./...` in `cli` passes. In
`tools/ods`, `TestDevcontainerCommands` and `TestRunDevRebuild` fail
locally because `SSH_AUTH_SOCK` adds a mount, which is unrelated.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
## Description
Add 11 selectable embedding models and an upgrade-only model picker.
Fresh self-hosted installs use Granite. Existing deployments keep their
stored model until an admin starts a re-index. On startup, the API
server waits until the model server can embed with the self-hosted model
in use. If that model is not in its cache, the model server downloads it
first.
| Type | Model | Output dimensions |
| --- | --- | --- |
| Self-hosted | `ibm-granite/granite-embedding-97m-multilingual-r2`
(default, in the image) | 384 |
| Self-hosted | `voyageai/voyage-4-nano` | 2048 |
| Self-hosted | `nvidia/Nemotron-3-Embed-1B-BF16` | 2048 |
| Cohere | `embed-v5.0-pro`, `embed-v5.0-fast` | 2048 |
| Google | `gemini-embedding-2` | 3072 |
| OpenAI | `text-embedding-3-large`, `text-embedding-3-small` | 3072 /
1536 |
| Voyage | `voyage-4-large`, `voyage-4`, `voyage-4-lite` (added on main)
| 1024 |
The 16 older models are now **legacy**: nomic, E5, GTE-small, Cohere
v3/v4, Voyage v2 and older Gemini/Vertex models. Existing indexes on
them keep working. The API refuses a legacy model as a new target, but a
re-index with the current model is still allowed.
## Size of this PR
After review, I removed the parts that the feature does not need. The
behavior stays the same. The diff against `main` went from **80 files,
+9,855 / −504** to **69 files, +4,740 / −467**.
| Area | Files | Added | Removed |
| --- | --- | --- | --- |
| Backend source | 16 | 1,004 | 96 |
| Backend unit tests and test managers | 16 | 1,568 | 32 |
| Backend manual suites (opt-in, not in CI) | 7 | 617 | 0 |
| Web source | 6 | 617 | 255 |
| Web tests | 5 | 630 | 1 |
| Web messages (9 locales) | 9 | 225 | 72 |
| Deployment, CI and feature map | 10 | 79 | 11 |
The feature map (`.agents/feature-map/`) now describes the new selection
rules.
Removed:
- **The branch's own cloud probe.** All selectable cloud models now use
main's probe (`probe_embedding_dimension`), which Bifrost already used.
It now also sends the stored Vertex configuration, the Azure API version
and the deployment name. Before this change, the branch's probe ignored
the Vertex configuration, so a Google provider that uses Workload
Identity could not switch to `gemini-embedding-2`.
- **The Vertex fan-out.** Main already sends `gemini-embedding-2` texts
one at a time. The Vertex code is now the same as on main.
- **The `expected_dim` path** in the model-server loader, and the
startup warning for custom models with registry names. These protected
only one rare case; see **Custom models with a registry name** in [Known
limits](#known-limits).
- **The UI fallback** that tested a new key with a default model when
the key failed with the current model. The credential test now embeds
with the staged or current model of that provider. Only a provider with
neither uses its first selectable model.
- **Picker cards for current and staged bring-your-own models.** Main
does not show them either. The current model shows in the summary at the
top of the page.
- **Shared gates and fixtures for the manual suites.** Each suite has
one gate env var.
- **The screenshots in `docs/assets`.** The images below come from an
earlier commit of this PR.
- **Tests for removed code, and repeated test cases.** The unit tests
went from 3,842 to 2,667 lines.
### CPU speed and memory cards
All nine locales show these planning estimates:
| Model | Card speed | Recommended CPU RAM |
| --- | --- | --- |
| Granite 97M | Up to **200 docs / min / CPU core** | **4 GB** per model
server |
| Voyage 4 Nano | Up to **50 docs / min / CPU core** | **4 GB** per
model server |
| Nemotron 3 Embed 1B BF16 | Up to **5 docs / min / CPU core** | **8
GB** per model server |
Nemotron says **GPU recommended**. It runs on CPU too, but slowly. The
cards tell the admin that speed changes with the hardware and the
document size. The RAM value is for one loaded model. A re-index and the
rest of Onyx need more memory.
These values are planning estimates, not guarantees. The Nemotron
per-core value is an extrapolation from an earlier multi-thread run.
Document counts depend on chunking, and throughput does not always scale
linearly with cores. The RAM values are recommendations, not measured
minimum or peak values.
## Implementation
- **Registry.** `shared_configs/embedding_models.py` defines the model
names, dimensions, prefixes, normalization, status and pinned Hugging
Face revisions. A parity test compares it with the web registry.
- **Upgrade-only guard.** `_check_embedding_model_target` runs in
`set-new-search-settings`:
- The current model can always re-index, legacy models too. The
dimension must stay the same. In Onyx Cloud, `reduced_dimension` must
also stay the same, because tenants share the index of a model.
- A selectable model must use the registry settings (dimension,
normalization, prefixes).
- Other OpenAI, Cohere, Google and Voyage models, and legacy self-hosted
models, get `INVALID_INPUT`. The error lists the selectable models.
- Custom Hugging Face models, LiteLLM, Azure and Bifrost stay open.
- A self-hosted model gets `INVALID_INPUT` for a new
`reduced_dimension`, because the model server ignores it. A re-index
keeps a stored value.
- **Cloud probe.** Before a re-index to a selectable cloud model, the
API embeds a test text with the stored credentials and checks the
dimension. A bad key, region or model ID fails before indexing starts.
The probe runs before the PRESENT row lock, with no open transaction, as
main's Bifrost probe does.
- **Self-hosted loads.** New models load at a pinned revision with
`trust_remote_code=False`, a dimension check and a finite-vector check.
CPU-only hosts load FP32. CUDA and MPS keep the checkpoint dtype. Legacy
and custom models keep main's `SentenceTransformer` call, so their
vectors do not change.
- **Voyage 4 Nano.** An Onyx loader adds bidirectional attention and the
learned 1024-to-2048 projection. The generic loader with remote code
disabled does not apply them.
- **Model server.** All models load from the cache first, then from
Hugging Face. Loads run off the event loop on two load workers, so at
most two sets of weights (PRESENT and FUTURE) load at the same time.
Registry models get 32 tokens of headroom for the prefix. The RoPE
pre-warm now runs again only for a longer context. On main, it ran for
every change of context length.
- **Image.** The model-server image bundles pinned Granite. It no longer
bundles nomic: nomic downloads on first use. The startup cache merge
keeps the models that a volume already has.
- **Picker.** Self-hosted and cloud groups, a Legacy label on the
current legacy model, and an **Advanced** section for custom Hugging
Face models, LiteLLM, Azure and Bifrost. A staged custom model keeps the
custom provider, also when its name is the same as a registry model.
- **Error detail.** `EmbeddingModel` now keeps the model server's error
detail. Before, every HTTP failure showed "HTTP error occurred -
response is None".
### Startup wait for self-hosted models
On startup, the API server waits until the inference model server can
embed with the current self-hosted model. The model server loads the
model on the first request, and downloads it first if it is not cached.
Example: a deployment that upgrades while it uses nomic, which the new
image does not bundle.
- The API server logs the wait, each failed attempt with its cause, and
the time when the model is ready. The model server logs a download.
- `LOCAL_EMBEDDING_MODEL_STARTUP_TIMEOUT` sets the wait (default 1800
seconds). When the wait runs out, startup fails. The error names the
model server, the model, the last error and the fix. `0` disables the
wait.
- The check request runs in a daemon thread. A request that still waits
at the deadline (for example, during a slow download) therefore does not
keep the failed process alive. Before this fix, the process stayed alive
until the model server read timeout (600 s by default).
- There is no wait in Onyx Lite (`DISABLE_VECTOR_DB`), for cloud models,
in multi-tenant deployments, in integration tests, or with
`DISABLE_MODEL_SERVER`.
- The wait uses the existing embed endpoint, so it also works with an
older model server during a rolling upgrade.
- This replaces the startup call to `warm_up_bi_encoder`. `SKIP_WARM_UP`
(default `true`) skipped that call, so it did not run by default.
- Only the inference model server is checked. The indexing model server
still loads a model on first use. In Compose, the API server does not
know the host of the indexing model server.
- `env.template` and Helm `values.yaml` document the setting. The Helm
chart is now 0.9.19.
Live test at commit `0eb5881da9`, with an existing nomic deployment on a
local stack. The size reduction did not change the wait:
| Case | Result |
| --- | --- |
| nomic not cached, model server online | The model server downloaded
nomic (523 MB). Startup continued after 19 s. |
| nomic cached | Ready in 3 s, with no download. |
| Model server starts after the API server | A warning every 10 s, then
ready at 33 s. |
| nomic not cached, no internet access | Each warning showed the cause
("not in the model server's Hugging Face cache and could not be
downloaded"). Startup failed at the timeout, and the process exited. |
| Onyx Lite, no model server | No wait. |
| Timeout `0` | No wait. The model loads on first use. |
## Upgrade notes
- **Offline deployments that use nomic.** The new image does not bundle
nomic. A deployment with no internet access and no kept model cache
cannot download it. The API server then fails at startup with a clear
error, instead of starting with a search that cannot embed queries.
Before you upgrade such a deployment, add the current model to the model
server's Hugging Face cache, or keep the cache on a volume. Helm has no
cache volume by default. This PR has no schema migration, so a return to
the previous image works while the current model stays the same.
- **Helm.** The chart has no Hugging Face cache volume for the model
servers, so a model that the image does not bundle (nomic, Voyage 4
Nano, Nemotron) downloads again on every pod start. `values.yaml`
documents an opt-in volume with the existing
`inferenceCapability.volumes` and `volumeMounts` values. A default
volume is not added: a ReadWriteOnce claim hangs rolling updates across
nodes, breaks `replicaCount` above 1, and needs a StorageClass. The
bundled Granite works with or without a volume.
- **API probes.** `/health` answers only after the startup wait. A
liveness probe needs a startup probe that allows the migrations plus
`LOCAL_EMBEDDING_MODEL_STARTUP_TIMEOUT`. The api probe example in
`values.yaml` shows one.
- **Multi-tenant template snapshots.** Migration `dbaa756c2ccf` seeds a
search settings row with `DOCUMENT_ENCODER_MODEL`. When that variable is
empty, the seed is now Granite, not nomic. If a deploy runs
`alembic/check_tenant_snapshot.py`, it reports a difference in
`search_settings` until the template snapshots are built again
(`--snapshot-template`). To keep the seed the same as on main, set
`DOCUMENT_ENCODER_MODEL=nomic-ai/nomic-embed-text-v1`.
- **No return to a legacy model.** After a switch, the old index is
cleaned up and the upgrade-only rule blocks a return to a legacy model.
Keep a compatible backup before you switch.
## Audit: retries and failures
| Layer | Behavior | Compared with main |
| --- | --- | --- |
| Local encode | Catches `RuntimeError`, waits 0.1 seconds, then tries
again: **up to four encode attempts**. It also retries errors that do
not go away, such as OOM. | Unchanged. |
| API to model server | Passage requests retry request failures three
times with five-second waits. HTTP 429 allows ten attempts with
ten-second waits. Connect/read timeouts are 30/600 seconds. Query
requests do not get these retries. | Unchanged. |
| Document indexing | A failed batch falls back to one document at a
time after a two-second delay. Failures are recorded, and the other
documents continue. | Unchanged. |
| Cloud embedding | Authentication errors are separate. Errors mapped to
`RuntimeError` get exponential backoff with jitter: eight attempts in
indexing-only mode, else two. | Unchanged. `gemini-embedding-2` sends
one text per request, in sequence, as on main. |
| Model loading | Cache first, then online. A partial cache falls back
online. A failed load is not cached, so a later request can try again.
Requests for the same model and context length share one load job. |
Cache-first lookup is new for all models. Registry models add pinned
revisions and checks. |
Model downloads run off the event loop. When a waiting request is
cancelled, the shared load continues. A load job has no deadline, so it
can continue after an HTTP timeout.
One gap from main remains: the response JSON is parsed outside the retry
wrapper. See [local
retries](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/model_server/encoders.py#L157),
[request
retries](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/natural_language_processing/search_nlp_models.py#L1322),
and [cloud
retries](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/natural_language_processing/search_nlp_models.py#L942).
## Audit: migration and cancellation
There is **no schema migration, and no automatic change to existing
embedding settings**. Granite is the fresh-install default. Cloud tenant
provisioning still uses Cohere v3. Existing stored prefixes and
dimensions stay in use.
1. The API checks the target with the upgrade-only guard.
2. A new cloud target embeds a test text with the stored credentials
before the migration. The probe checks the dimension. It runs before the
PRESENT row lock.
3. The API locks PRESENT and refuses overlapping migrations. It also
refuses an index name that is live or still waiting for cleanup. FUTURE,
the first work and the cleanup consent commit together.
4. The existing port flow reads stored documents from the current index,
embeds them again and writes the new index. Connectors do not fetch
again. All-connectors mode includes paused connectors. Active-only mode
leaves them out and needs consent for the data that it leaves out. User
files are included.
5. Normal promotion waits for successful required ports, no open writers
or sync work, and a sample check in the new index. A failed or paused
required port blocks promotion. The sample check does not check the full
corpus.
6. Port batches retry up to five times. The cursor moves after a
successful batch. The first watchdog retry is immediate. Repeated
failures at the same cursor use exponential backoff from 30 seconds, up
to one hour. Five failures in sequence at that cursor pause the unit. An
operator can resume it.
7. A cancel of FUTURE keeps PRESENT active, stops port work at a batch
boundary, and removes the partial index after writers stop. **Instant
switch promotes first**, so search is not complete during backfill, and
this cancel path is not available.
See [migration
endpoint](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/server/manage/search_settings.py#L140),
[upgrade-only
guard](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/server/manage/search_settings.py#L456),
[swap
gates](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/db/swap_index.py#L367),
and
[cancellation](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/onyx/server/manage/search_settings.py#L668).
## Known limits
- **Model memory.** Two load workers limit how many models load at the
same time. Loaded models are not evicted (as on main), so the old and
new models stay in memory during a switch. Helm's default memory limit
for the indexing model server is **6 GiB**, below the **8 GB** that the
Nemotron card recommends. See [load
executor](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/backend/model_server/encoders.py#L35)
and [Helm
limits](https://github.com/onyx-dot-app/onyx/blob/4784da70697fb59261f43088383d85671544ec98/deployment/helm/charts/onyx/values.yaml#L309).
- **Legacy models are not selectable.** This is by design. The current
legacy model stays visible and can re-index. After a switch, the
standard flow cannot select it again.
- **Bring-your-own options are under Advanced.** Unknown model names for
the built-in OpenAI, Cohere, Google and Voyage providers get
`INVALID_INPUT`. LiteLLM, Azure, Bifrost and custom models stay open.
- **The cloud probe reads the provider settings once.** If an admin
changes the key or region while a re-index starts, the probe can use the
old values. A key can also change at any time during a re-index, so no
preflight check can guarantee the key that the workers use. Indexing
failures then show in the re-index status, and the admin can cancel.
- **Custom models with a registry name.** Before this PR, an admin could
add a registry model name, such as `voyageai/voyage-4-nano`, as a custom
model. The model server now loads that name as the registry model. For
`voyage-4-nano`, the vectors then have 2048 dimensions, not 1024, so a
1024-dimension index fails with an error. Re-index to the registry model
to fix it. For the other registry models, a generic load gives the same
dimension.
- **Shared `max_seq_length`.** The model server changes `max_seq_length`
on the shared model for each request. Two requests with different
context lengths can overlap. This is the same on main. This PR does not
change it.
- **Voyage 4 Nano depends on the Transformers and SentenceTransformers
layout.** Run the real-weight suite when those dependencies change.
- **No reduced dimensions for Cohere v5 and the self-hosted models.**
Cloud provisioning stays on Cohere v3. These are follow-ups.
- **The cancel dialog has two buttons labeled Cancel.** One closes the
dialog, the other cancels the re-index. This is the same on main.
- The branch is rebased onto `main` at `ad9c0030e9`. This PR does not
claim better retrieval quality.
## Validation for this update
- **Backend: 15,343 unit tests passed** before the review fixes. I left
out the EE license tests, because they write a key to a live Redis. `ty`
and pre-commit passed. After the review fixes and the merge of `main`,
the PR's backend unit tests passed again (199 tests). Pre-commit passed
on all 69 changed files.
- **Startup exit check:** a fake model server that never answers, with a
3 s wait and a 20 s read timeout. The wait raised at 3.3 s. The process
exited at 5 s with the fix, and at 22 s without it.
- **Web: type check, lint and 3,236 Jest tests (307 suites) passed.**
- **Self-hosted real-weight suite: 6 of 6 passed.** Golden scores and
the bidirectional-attention check for Granite, Voyage 4 Nano and
Nemotron.
- **Cloud live suite: `gemini-embedding-2` passed.**
- **End to end on a local stack with branch code: 11 of 11 passed.** The
API refused all 6 legacy self-hosted targets and accepted a same-model
re-index. Then the stack switched to Granite, Voyage 4 Nano, Nemotron
and `gemini-embedding-2`, in sequence. After each switch, a pure vector
search found each of 5 seeded facts first.
- **Not run for this update: Cohere, OpenAI and Voyage cloud tests.** A
TLS proxy on the test network blocked `api.cohere.com` and
`api.openai.com`. The probe then refused the switch with `400`, as
designed. These tests passed on an earlier commit of this PR, and the
probe that they use is main's probe. I had no Voyage key.
- **CI:** at `9321cb51db`, every check passed except the Dependency
Audit: Playwright, integration, acceptance, database,
external-dependency unit, Jest and quality checks. The audit fails on
open Dependabot alerts for Pillow and certifi in other manifests
(`loadtest/uv.lock` and the Craft sandbox requirements). Other open PRs
fail the same way at the moment.
- **Not run for this update:** speed benchmarks.
<details>
<summary>Earlier validation in this PR, not run again for this
update</summary>
- 28 real-weight tests and 24 cloud tests passed.
- End-to-end switches to all eight selectable models of that time
passed. The API refused all 16 legacy targets.
- Legacy-upgrade equivalence, Docker cache keeping and multi-tenant
provisioning checks passed.
- 11,521 backend unit tests passed. 10 EE license failures also occurred
on main.
</details>
## Voyage retrieval-quality check — October 14, 2026
The branch's Voyage loader passed a local EnterpriseRAG-Bench loading
check. **Vector-only mean document Recall@10 was 91.85%**, above the
proposed 50% threshold. This supports a correct load. It does not prove
numerical equality with the upstream implementation.
The corpus had **5,000 documents**: all **722 labeled supporting
documents** and **4,278 random distractors**, sampled with seed
`20261013`. All **470 questions with document labels** were scored. The
30 questions with no labels were left out. No question or answer text
went into the corpus.
| Retrieval mode | Mean Recall@1 | Mean Recall@5 | Mean Recall@10 | Mean
Recall@20 |
| --- | --- | --- | --- | --- |
| Onyx content-vector-only retrieval | 66.13% | 87.37% | **91.85%** |
94.16% |
| Onyx hybrid search API | 75.85% | 93.32% | **96.37%** | 97.70% |
Recall is the fraction of labeled documents found for each question,
averaged over the questions. Rankings use the first hit of each
document. Vector-only hit rate@10 (at least one relevant document) was
94.26%. That is a different metric from recall.
| Question category | Questions | Vector-only Recall@10 | Hybrid
Recall@10 |
| --- | --- | --- | --- |
| Basic | 175 | 97.71% | 99.43% |
| Semantic | 125 | 84.80% | 92.80% |
| Intra-document reasoning | 40 | 97.50% | 100.00% |
| Project related | 40 | 86.43% | 95.35% |
| Constrained | 30 | 98.33% | 98.33% |
| Conflicting information | 20 | 90.00% | 97.50% |
| Completeness | 20 | 68.03% | 79.03% |
| Miscellaneous | 20 | 100.00% | 100.00% |
The check used CPU FP32, 2048-dimensional normalized vectors, and the
registry's query and document prefixes. MPS and CUDA were off. There was
no query expansion, reranking, or LLM document selection. All 5,000
document IDs matched the saved manifest, with 15,612 indexed chunks. All
query vectors were finite and had unit length. The last indexing attempt
had no document errors. Two earlier attempts stopped after
stale-heartbeat timeouts and continued from saved progress.
**Search path:** on this OpenSearch branch, `hybrid_alpha=1` still runs
hybrid retrieval, so the API result includes keyword scores. The
vector-only result calls `OpenSearchDocumentIndex.semantic_retrieval`
directly, with content vectors only. Keyword matches therefore cannot
hide a broken embedding loader in that result.
**Scope:** this is an easier 5,000-document subset that contains all
gold documents, not the full 511,958-document benchmark. Do not compare
these values with full-corpus leaderboard scores. No loader or ranking
code changed to get these results.
Revisions: Onyx `913bfa8d7a32c9cda340df92f42d0f706fd424c2`;
EnterpriseRAG-Bench `d36685e273713975ee20299bbf1ab64165575b3c`; Voyage
`67fabc9bef010dabc5f6024aa1b1b6b93410426f`. Runtime: PyTorch 2.9.1,
Transformers 5.14.1, SentenceTransformers 5.4.1.
## UI screenshots
From the local development stack at commit `98bfc70080`, before the size
reduction. These show the real flow, not mockups.
<details open>
<summary>Self-hosted cards and current model</summary>
</details>
<details>
<summary>Cloud model catalog</summary>
</details>
<details>
<summary>Staged Granite change and migration strategy</summary>
</details>
<details>
<summary>Re-index in progress</summary>
</details>
<details>
<summary>Cancel confirmation</summary>
</details>
<details>
<summary>Advanced custom model setup</summary>
</details>
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [ ] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Description
Documents the schema downgrade that a rollback from v5 needs, in the
Helm migration guide and the Compose README. Fixes
[ENG-4431](https://linear.app/onyx-app/issue/ENG-4431/rollback-to-v49-needs-a-downgrade-to-the-v492-alembic-head-and-nothing).
**Why**
- A rollback does not change the schema. v4.9.2 then fails `alembic
upgrade head` with `Can't locate revision`, and only the v5 image can
downgrade. Evan hit it on Compose and Justin on Helm.
- On Helm the stuck rollout leaves the 0.9.x pod serving without the
object-store Service, so file reads fail. MIGRATION.md said only `helm
rollback`.
**What**
- `deployment/helm/MIGRATION.md` Rollback: ordered steps (replay missed
MinIO writes, `alembic downgrade 25053020dd5a` from a 0.9.x pod, `helm
rollback` immediately) plus recovery if you rolled back first.
- `deployment/docker_compose/README.md`: new "Rolling back the
deployment" section, version-generic, with `docker run ... alembic
heads` to find the target and `onyx-cli deploy upgrade
--allow-downgrade`.
- The embedded CLI copy of the README is synced by the pre-commit hook.
- Both name the data the downgrade deletes (child chat sessions, OAuth
grants and tokens, personal tool preferences).
Docs site: https://github.com/onyx-dot-app/documentation/pull/520
## How Has This Been Tested?
Live on a scratch Compose Postgres with the real images. v4.9.2 on the
v5.0.0-beta.2 schema exits 255 with `Can't locate revision`. The
documented `IMAGE_TAG=v5.0.0-beta.2 docker compose run --rm --no-deps
api_server alembic downgrade 25053020dd5a` took 5 s, after which v4.9.2
migrated cleanly to `25053020dd5a (head)` and v5 upgraded again.
`alembic heads` was run in the v4.6.5 image with no network. The Helm
path was not run here. Justin ran the same downgrade on kind with chart
0.8.40 and 0.9.17, and the claim that `kubectl exec deploy/...` reaches
the ready 0.9.x pod comes from kubectl's pod sort
(`sort.Reverse(ActivePods)`).
## Additional Options
- [x] [Optional] Please cherry-pick this PR to the latest release
version.
## Description
Fixes
[ENG-4428](https://linear.app/onyx-app/issue/ENG-4428/a-manual-compose-upgrade-silently-stays-on-minio).
**Bug:** a hand-run Compose upgrade (`docker compose down`, new
`IMAGE_TAG`, `up`) keeps `S3_ENDPOINT_URL=http://minio:9000` from the
old `.env`, so the app keeps writing files to MinIO while SeaweedFS and
`object-store-copy` run empty. Nothing says so, and a later release
removes MinIO.
**Why:** only `onyx-cli deploy upgrade` rewrites the endpoints. A
compose default cannot help because the old `.env` wins, and defaulting
the legacy endpoint would make external-S3 installs dual-write into
MinIO.
**Fix:**
- `app_configs` logs a startup warning when the s3 backend still points
at the bundled MinIO, naming the two `.env` settings to change (or
`onyx-cli deploy upgrade`).
- `object-store-copy` logs the same hint instead of the neutral "No
legacy MinIO store to copy from."
- Compose README gets the manual-upgrade steps, scoped to installs on
the bundled MinIO. Docs: onyx-dot-app/documentation#519.
## How Has This Been Tested?
Unit test for the predicate (bundled MinIO endpoint with and without a
trailing slash flags; object store, external S3, no endpoint and the
postgres backend do not). Exercised the real code paths locally by
importing `app_configs` and calling `copy_legacy_objects()` with the env
set three ways: MinIO endpoint logs the warning, object-store endpoint
plus legacy endpoint logs nothing, no endpoint logs the unchanged info
line. Not run: a full Compose upgrade from v4.9.2, which Evan's bug bash
already reproduced.
## Additional Options
- [x] [Optional] Please cherry-pick this PR to the latest release
version.
Documents indexed before the bounded-sketch chunker (#16115) still hold
hundreds of row chunks each. `TABULAR_RECHUNK_ENABLED` arms a beat task
that re-chunks them in place, from the raw workbook Onyx already stores,
without re-crawling the source.
An operator turns it on by deploying with the variable set and nothing
else.
## Why a normal re-index cannot do this
`Document.content_hash()` walks Text and Image sections only, so a
tabular document's hash never moves however much the workbook changed.
The content-hash gate therefore skips every spreadsheet on any re-index,
and `force_update` — the only bypass — was wired solely into the
per-document retry path. This exposes it on `run_indexing_pipeline`;
that is the one change to shared indexing code, and it is a no-op unless
passed.
## How it picks its work
The scan walks `document` by primary key and takes only tabular
documents holding more chunks than the current chunker could produce
(`TABULAR_MAX_CHUNKS_PER_WORKBOOK` plus the large chunks multipass
derives from them — 30 by default). That bound means it skips its own
output and anything an in-flight crawl writes, so it can run during an
initial index without fighting it. Scanning by primary key keeps every
page an index scan: one pass of the table over the migration's life, not
a sort per run.
A KV record holds the cursor and a completed flag. Once the scan ends,
each later tick is one cached read, so leaving the variable set across
restarts re-runs nothing.
## Preserving what the upsert would overwrite
Re-indexing an unchanged document means `_upsert_documents_in_db` writes
the rebuilt document over a row that was already correct, so anything
the rebuild drops is destroyed corpus-wide:
- `metadata` and `title` are not kept in Postgres. They are read back
from the document's own stored chunk, as the reindex port does for
documents it cannot re-fetch.
- `doc_metadata`, owners, hierarchy and `file_id` come from the row.
Owners round-trip through their stored string form.
- `external_access` is left unset, so `includes_permissions` stays false
and the upsert leaves perm-synced ACL columns alone.
Unit tests pin each of these, plus the content-hash blindness that makes
`force_update` necessary.
## Guards
Pauses while a FUTURE index exists (the pipeline would write there
instead) or a port backfill is pending (the port copies chunks boundary
for boundary). Skips entirely under `SKIP_SPREADSHEET_CONTENT` or
`DISABLE_VECTOR_DB`. Documents whose raw file was never staged are left
out — only a crawl can rebuild those.
## Not yet verified
**This has never been run end to end.** The unit tests cover the
document rebuild and the gate reasoning; the query plan, the chunk-head
read and the delete-then-write replacement are unexercised. An
external-dependency-unit test over one real oversized document is the
obvious next step, and I'd want that before this points at a large
deployment.
Known side effects: it takes a docprocessing thread for up to 4 minutes
per 5-minute tick; every re-chunked document becomes eligible for
metadata sync; and OpenSearch reclaims disk on segment merge, so disk
lags chunk count.
## Reverting
No migration, no schema change, no data model change. Removing it is
deleting three files and reverting small hunks in thirteen; the two that
touch shared paths (`force_update`, `get_chunk_heads`) are inert on
their own.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds an opt-in beat task, armed by `TABULAR_RECHUNK_ENABLED`, that
re-chunks tabular documents indexed before the bounded-sketch chunker,
in place from the workbook already in the file store, without
re-crawling the source. A KV record stops it once done, so leaving the
variable set across restarts re-runs nothing.
- The scan walks the `document` table by primary key, resolving the
cc-pair with a LATERAL so every page stays an index scan, and touches
only documents holding more chunks than the current chunker could
produce. Hidden documents are passed over, since the chunk-head read
never returns one.
- The beat entry is registered only when the variable is set, and the
task checks the completed flag before taking the lock, so a finished
migration costs one cached read per tick. Ticks run at HIGH priority
with a 4-minute expiry on the docprocessing queue.
- The re-index passes `force_update` and `skip_document_row_writes`, so
only the chunks and `chunk_count` change. Re-indexing no longer resets
admin-set `boost`/`hidden`, and ancestors resolve from
`parent_hierarchy_node_id`.
- Title and metadata are read back from the document's own stored chunk,
fetched by deterministic id; owners are restored from the row because
the chunk writer reads them off the document; Drive sheets are matched
and dispatched by stored MIME type.
- The migration's adapter re-reads `file_id` and `chunk_count` under the
document lock and aborts the batch if a crawl rewrote a document, so an
older workbook never replaces newer chunks.
- A re-chunk that fails after chunking no longer commits the smaller
count, so the document stays oversized and the next pass finds it by
rescanning; retries are bounded at three passes.
- Skips under `SKIP_SPREADSHEET_CONTENT` or `DISABLE_VECTOR_DB`; pauses
while a FUTURE index exists, while an INSTANT-swap port is backfilling,
or if the live index changes—re-checked under the document lock too,
since a swap landing mid-batch would write into the retired index. The
lock lease is extended per batch so a throttled batch can't let a second
tick start from the same cursor.
**Not yet verified**
- The migration has never completed end to end; it ran once against a
live index, which surfaced the port guard bug now fixed.
<sup>Written for commit 8cdc3ffac0.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16153?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16153?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16153&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added an opt-in, one-time migration to re-chunk oversized tabular
documents using existing indexed content, without requiring a source
re-crawl.
- Migration progress is saved and resumes safely across restarts, with
retries for eligible failures.
- Documents that are hidden, unavailable, or being actively changed are
skipped or retried safely.
- **Documentation**
- Added configuration guidance for enabling the tabular re-chunk
migration.
- Clarified migration behavior, completion tracking, scheduling, and
vector database requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nikolas Garza
