DanelegendandClaude Opus 5
e6772d6f41
Documents indexed before the bounded-sketch chunker (#16115) still hold
hundreds of row chunks each. `TABULAR_RECHUNK_ENABLED` arms a beat task
that re-chunks them in place, from the raw workbook Onyx already stores,
without re-crawling the source.
An operator turns it on by deploying with the variable set and nothing
else.
## Why a normal re-index cannot do this
`Document.content_hash()` walks Text and Image sections only, so a
tabular document's hash never moves however much the workbook changed.
The content-hash gate therefore skips every spreadsheet on any re-index,
and `force_update` — the only bypass — was wired solely into the
per-document retry path. This exposes it on `run_indexing_pipeline`;
that is the one change to shared indexing code, and it is a no-op unless
passed.
## How it picks its work
The scan walks `document` by primary key and takes only tabular
documents holding more chunks than the current chunker could produce
(`TABULAR_MAX_CHUNKS_PER_WORKBOOK` plus the large chunks multipass
derives from them — 30 by default). That bound means it skips its own
output and anything an in-flight crawl writes, so it can run during an
initial index without fighting it. Scanning by primary key keeps every
page an index scan: one pass of the table over the migration's life, not
a sort per run.
A KV record holds the cursor and a completed flag. Once the scan ends,
each later tick is one cached read, so leaving the variable set across
restarts re-runs nothing.
## Preserving what the upsert would overwrite
Re-indexing an unchanged document means `_upsert_documents_in_db` writes
the rebuilt document over a row that was already correct, so anything
the rebuild drops is destroyed corpus-wide:
- `metadata` and `title` are not kept in Postgres. They are read back
from the document's own stored chunk, as the reindex port does for
documents it cannot re-fetch.
- `doc_metadata`, owners, hierarchy and `file_id` come from the row.
Owners round-trip through their stored string form.
- `external_access` is left unset, so `includes_permissions` stays false
and the upsert leaves perm-synced ACL columns alone.
Unit tests pin each of these, plus the content-hash blindness that makes
`force_update` necessary.
## Guards
Pauses while a FUTURE index exists (the pipeline would write there
instead) or a port backfill is pending (the port copies chunks boundary
for boundary). Skips entirely under `SKIP_SPREADSHEET_CONTENT` or
`DISABLE_VECTOR_DB`. Documents whose raw file was never staged are left
out — only a crawl can rebuild those.
## Not yet verified
**This has never been run end to end.** The unit tests cover the
document rebuild and the gate reasoning; the query plan, the chunk-head
read and the delete-then-write replacement are unexercised. An
external-dependency-unit test over one real oversized document is the
obvious next step, and I'd want that before this points at a large
deployment.
Known side effects: it takes a docprocessing thread for up to 4 minutes
per 5-minute tick; every re-chunked document becomes eligible for
metadata sync; and OpenSearch reclaims disk on segment merge, so disk
lags chunk count.
## Reverting
No migration, no schema change, no data model change. Removing it is
deleting three files and reverting small hunks in thirteen; the two that
touch shared paths (`force_update`, `get_chunk_heads`) are inert on
their own.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds an opt-in beat task, armed by `TABULAR_RECHUNK_ENABLED`, that
re-chunks tabular documents indexed before the bounded-sketch chunker,
in place from the workbook already in the file store, without
re-crawling the source. A KV record stops it once done, so leaving the
variable set across restarts re-runs nothing.
- The scan walks the `document` table by primary key, resolving the
cc-pair with a LATERAL so every page stays an index scan, and touches
only documents holding more chunks than the current chunker could
produce. Hidden documents are passed over, since the chunk-head read
never returns one.
- The beat entry is registered only when the variable is set, and the
task checks the completed flag before taking the lock, so a finished
migration costs one cached read per tick. Ticks run at HIGH priority
with a 4-minute expiry on the docprocessing queue.
- The re-index passes `force_update` and `skip_document_row_writes`, so
only the chunks and `chunk_count` change. Re-indexing no longer resets
admin-set `boost`/`hidden`, and ancestors resolve from
`parent_hierarchy_node_id`.
- Title and metadata are read back from the document's own stored chunk,
fetched by deterministic id; owners are restored from the row because
the chunk writer reads them off the document; Drive sheets are matched
and dispatched by stored MIME type.
- The migration's adapter re-reads `file_id` and `chunk_count` under the
document lock and aborts the batch if a crawl rewrote a document, so an
older workbook never replaces newer chunks.
- A re-chunk that fails after chunking no longer commits the smaller
count, so the document stays oversized and the next pass finds it by
rescanning; retries are bounded at three passes.
- Skips under `SKIP_SPREADSHEET_CONTENT` or `DISABLE_VECTOR_DB`; pauses
while a FUTURE index exists, while an INSTANT-swap port is backfilling,
or if the live index changes—re-checked under the document lock too,
since a swap landing mid-batch would write into the retired index. The
lock lease is extended per batch so a throttled batch can't let a second
tick start from the same cursor.
**Not yet verified**
- The migration has never completed end to end; it ran once against a
live index, which surfaced the port guard bug now fixed.
<sup>Written for commit 8cdc3ffac0.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16153?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16153?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16153&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added an opt-in, one-time migration to re-chunk oversized tabular
documents using existing indexed content, without requiring a source
re-crawl.
- Migration progress is saved and resumes safely across restarts, with
retries for eligible failures.
- Documents that are hidden, unavailable, or being actively changed are
skipped or retried safely.
- **Documentation**
- Added configuration guidance for enabling the tabular re-chunk
migration.
- Clarified migration behavior, completion tracking, scheduling, and
vector database requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Welcome to Onyx
To set up Onyx there are several options, Onyx supports the following for deployment:
- Quick guided install via the install.sh script
- Pulling the repo and running
docker compose up -dfrom the deployment/docker_compose directory
- Note, it is recommended to copy over the env.template file to .env and edit the necessary values
- For large scale deployments leveraging Kubernetes, there are two options, Helm or Terraform.
This README focuses on the easiest guided deployment which is via install.sh.
For more detailed guides, please refer to the documentation: https://docs.onyx.app/deployment/overview
install.sh script
curl -fsSL https://raw.githubusercontent.com/onyx-dot-app/onyx/main/deployment/docker_compose/install.sh > install.sh && chmod +x install.sh && ./install.sh
The script installs the Onyx CLI (onyx-cli) and hands over to onyx-cli deploy install, which is
where the guided installation lives. Any flags you pass are forwarded to it. If you already have the
CLI (pip install onyx-cli), skip the script and run onyx-cli deploy install directly.
This provides a guided installation of Onyx via Docker Compose. It will deploy the latest version of Onyx and set up the volumes to ensure data is persisted across deployments or upgrades.
The deployment files are stored in ~/.config/onyx (an existing onyx_data directory from an older
install is detected and kept in place; --dir targets another location). Note that no application
critical data is stored in that directory so even if you delete it, the data needed to restore the app
will not be destroyed.
The data about chats, users, etc. are instead stored as named Docker Volumes. This is managed by Docker
and where it is stored will depend on your Docker setup. You can always delete these as well by running
onyx-cli deploy uninstall.
To shut down the deployment without deleting, use onyx-cli deploy stop.
Managing the deployment
Beyond installing, the CLI covers the rest of the lifecycle:
| Command | What it does |
|---|---|
onyx-cli deploy status |
Installed version, containers, and health (--json for scripts) |
onyx-cli deploy logs [service...] |
Logs of the deployment's containers |
onyx-cli deploy stop |
Stop the containers, keep the data |
onyx-cli deploy upgrade [--tag vX.Y.Z] |
Upgrade in place (see below) |
onyx-cli deploy uninstall |
Remove the containers, volumes, and deployment directory |
Upgrading the deployment
Onyx maintains backwards compatibility across all minor versions following SemVer, so upgrading is
onyx-cli deploy upgrade (add --tag vX.Y.Z to pin a version). It rewrites only IMAGE_TAG, preserves
your .env edits, and backs up hand-edited files.
If you are more comfortable running docker compose commands, you can also run commands directly from
the directory with the docker-compose.yml file. First bring the containers down (docker compose down),
verify the version you want in the environment file (see below), (if using latest tag, be sure to run
docker compose pull) and run docker compose up to restart the services on the latest version
Customizing the compose files
The CLI owns the compose files it writes. Each onyx-cli deploy upgrade refreshes them to match the
new version. If you edit one, the CLI asks before it replaces the file, and it keeps a backup — but
your edits are not applied again.
To customize the deployment, put your changes in a file next to docker-compose.yml, named
compose.override.yml, compose.override.yaml, docker-compose.override.yml, or
docker-compose.override.yaml. These are the same names docker compose finds on its own; if more than
one is present, the CLI picks the same one Compose would, in that order. The CLI applies that file last,
after its own files, so your changes win. The CLI does not manage the override: it never writes,
replaces, or backs it up, and an upgrade does not touch it. install, upgrade, stop, logs, and
uninstall all apply it.
For example, to put Onyx behind your own reverse proxy, stop nginx from publishing a host port and attach it to your proxy's network:
# deployment/docker-compose.override.yml
services:
nginx:
# !reset replaces the list. Without it, Compose adds to the list.
ports: !reset []
networks: [default, proxy]
networks:
proxy:
external: true
Use docker compose -f docker-compose.yml -f docker-compose.override.yml config in the deployment
directory to see the merged result. Add each file the CLI applies (docker-compose.onyx-lite.yml,
docker-compose.prod.yml, and so on) in the same order the CLI does.
Environment variables
The Docker Compose files try to look for a .env file in the same directory. The installer sets it up
from a file called env.template. Feel free to edit the .env file to customize your deployment. The most
important / common changed values are located near the top of the file. Later onyx-cli deploy runs
keep your edits.
IMAGE_TAG is the version of Onyx to run. It is recommended to leave it as latest to get all updates with each redeployment.
Every image publishes a -dev twin for each of its tags (e.g. latest-dev, v1.2.3-dev), so a single
IMAGE_TAG=latest-dev selects the dev variant of the whole deployment. Today only the backend image actually differs:
its -dev twin adds interactive debugging tools (vim, nano, curl, ps, psql) that the default image leaves out to stay
minimal. The web-server, model-server, and sandbox -dev tags are identical to their plain counterparts and exist so
that one version string covers every image.