Documents indexed before the bounded-sketch chunker (#16115) still hold
hundreds of row chunks each. `TABULAR_RECHUNK_ENABLED` arms a beat task
that re-chunks them in place, from the raw workbook Onyx already stores,
without re-crawling the source.
An operator turns it on by deploying with the variable set and nothing
else.
## Why a normal re-index cannot do this
`Document.content_hash()` walks Text and Image sections only, so a
tabular document's hash never moves however much the workbook changed.
The content-hash gate therefore skips every spreadsheet on any re-index,
and `force_update` — the only bypass — was wired solely into the
per-document retry path. This exposes it on `run_indexing_pipeline`;
that is the one change to shared indexing code, and it is a no-op unless
passed.
## How it picks its work
The scan walks `document` by primary key and takes only tabular
documents holding more chunks than the current chunker could produce
(`TABULAR_MAX_CHUNKS_PER_WORKBOOK` plus the large chunks multipass
derives from them — 30 by default). That bound means it skips its own
output and anything an in-flight crawl writes, so it can run during an
initial index without fighting it. Scanning by primary key keeps every
page an index scan: one pass of the table over the migration's life, not
a sort per run.
A KV record holds the cursor and a completed flag. Once the scan ends,
each later tick is one cached read, so leaving the variable set across
restarts re-runs nothing.
## Preserving what the upsert would overwrite
Re-indexing an unchanged document means `_upsert_documents_in_db` writes
the rebuilt document over a row that was already correct, so anything
the rebuild drops is destroyed corpus-wide:
- `metadata` and `title` are not kept in Postgres. They are read back
from the document's own stored chunk, as the reindex port does for
documents it cannot re-fetch.
- `doc_metadata`, owners, hierarchy and `file_id` come from the row.
Owners round-trip through their stored string form.
- `external_access` is left unset, so `includes_permissions` stays false
and the upsert leaves perm-synced ACL columns alone.
Unit tests pin each of these, plus the content-hash blindness that makes
`force_update` necessary.
## Guards
Pauses while a FUTURE index exists (the pipeline would write there
instead) or a port backfill is pending (the port copies chunks boundary
for boundary). Skips entirely under `SKIP_SPREADSHEET_CONTENT` or
`DISABLE_VECTOR_DB`. Documents whose raw file was never staged are left
out — only a crawl can rebuild those.
## Not yet verified
**This has never been run end to end.** The unit tests cover the
document rebuild and the gate reasoning; the query plan, the chunk-head
read and the delete-then-write replacement are unexercised. An
external-dependency-unit test over one real oversized document is the
obvious next step, and I'd want that before this points at a large
deployment.
Known side effects: it takes a docprocessing thread for up to 4 minutes
per 5-minute tick; every re-chunked document becomes eligible for
metadata sync; and OpenSearch reclaims disk on segment merge, so disk
lags chunk count.
## Reverting
No migration, no schema change, no data model change. Removing it is
deleting three files and reverting small hunks in thirteen; the two that
touch shared paths (`force_update`, `get_chunk_heads`) are inert on
their own.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds an opt-in beat task, armed by `TABULAR_RECHUNK_ENABLED`, that
re-chunks tabular documents indexed before the bounded-sketch chunker,
in place from the workbook already in the file store, without
re-crawling the source. A KV record stops it once done, so leaving the
variable set across restarts re-runs nothing.
- The scan walks the `document` table by primary key, resolving the
cc-pair with a LATERAL so every page stays an index scan, and touches
only documents holding more chunks than the current chunker could
produce. Hidden documents are passed over, since the chunk-head read
never returns one.
- The beat entry is registered only when the variable is set, and the
task checks the completed flag before taking the lock, so a finished
migration costs one cached read per tick. Ticks run at HIGH priority
with a 4-minute expiry on the docprocessing queue.
- The re-index passes `force_update` and `skip_document_row_writes`, so
only the chunks and `chunk_count` change. Re-indexing no longer resets
admin-set `boost`/`hidden`, and ancestors resolve from
`parent_hierarchy_node_id`.
- Title and metadata are read back from the document's own stored chunk,
fetched by deterministic id; owners are restored from the row because
the chunk writer reads them off the document; Drive sheets are matched
and dispatched by stored MIME type.
- The migration's adapter re-reads `file_id` and `chunk_count` under the
document lock and aborts the batch if a crawl rewrote a document, so an
older workbook never replaces newer chunks.
- A re-chunk that fails after chunking no longer commits the smaller
count, so the document stays oversized and the next pass finds it by
rescanning; retries are bounded at three passes.
- Skips under `SKIP_SPREADSHEET_CONTENT` or `DISABLE_VECTOR_DB`; pauses
while a FUTURE index exists, while an INSTANT-swap port is backfilling,
or if the live index changes—re-checked under the document lock too,
since a swap landing mid-batch would write into the retired index. The
lock lease is extended per batch so a throttled batch can't let a second
tick start from the same cursor.
**Not yet verified**
- The migration has never completed end to end; it ran once against a
live index, which surfaced the port guard bug now fixed.
<sup>Written for commit 8cdc3ffac0.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16153?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16153?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16153&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added an opt-in, one-time migration to re-chunk oversized tabular
documents using existing indexed content, without requiring a source
re-crawl.
- Migration progress is saved and resumes safely across restarts, with
retries for eligible failures.
- Documents that are hidden, unavailable, or being actively changed are
skipped or retried safely.
- **Documentation**
- Added configuration guidance for enabling the tabular re-chunk
migration.
- Clarified migration behavior, completion tracking, scheduling, and
vector database requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Onyx CLI
A CLI for querying enterprise knowledge from Onyx. Includes an interactive chat TUI for humans and non-interactive commands for AI agents and scripts.
Installation
pip install onyx-cli
Or with uv:
uv pip install onyx-cli
Standalone binaries for Linux, macOS, and Windows (amd64/arm64) are also attached
to each cli/v* GitHub release.
Setup
Run the interactive chat TUI — on first launch it will guide you through setup:
onyx-cli chat
This prompts for your Onyx server URL and personal access token (PAT), tests the connection, and saves config to ~/.config/onyx-cli/config.json (or $XDG_CONFIG_HOME/onyx-cli/config.json if set). To reconfigure later, use the /configure command inside the TUI.
Environment variables override config file values:
| Variable | Required | Description |
|---|---|---|
ONYX_SERVER_URL |
No | Server origin or already-prefixed API base (default: https://cloud.onyx.app) |
ONYX_API_PREFIX |
No | API path prefix (default: /api); set to empty for direct backend access |
ONYX_PAT |
No | Personal access token for authentication (required if no config file) |
ONYX_PERSONA_ID |
No | Default agent/persona ID |
ONYX_STREAM_MARKDOWN |
No | Enable/disable progressive markdown rendering (true/false) |
ONYX_SSH_HOST_KEY |
No | Path to SSH host key for serve command |
Usage
Interactive chat
onyx-cli chat
onyx-cli chat --no-stream-markdown
onyx-cli chat --agent-name "Support Agent"
| Flag | Description |
|---|---|
--no-stream-markdown |
Disable progressive markdown rendering during streaming |
--agent-id <int> |
Agent ID to use for this session |
--agent-name <string> |
Agent name to use for this session (exact or unique substring; mutually exclusive with --agent-id) |
One-shot question
onyx-cli ask "What is our company's PTO policy?"
onyx-cli ask --agent-id 5 "Summarize this topic"
onyx-cli ask --agent-name "Support Agent" "hello"
onyx-cli ask --json "Hello"
| Flag | Description |
|---|---|
--agent-id <int> |
Agent ID to use (overrides default) |
--agent-name <string> |
Agent name to use (exact or unique substring; mutually exclusive with --agent-id) |
--json |
Output NDJSON stream events instead of plain text |
--prompt <string> |
Question text (use with piped stdin context) |
--quiet |
Buffer output and print once at end |
--max-output <int> |
Max bytes before truncating (0 to disable) |
List agents
onyx-cli agents
onyx-cli agents --json
Serve over SSH
# Start a public SSH endpoint for the CLI TUI
onyx-cli serve --host 0.0.0.0 --port 2222
# Connect as a client
ssh your-host -p 2222
Clients can either:
- paste a personal access token (PAT) at the login prompt, or
- skip the prompt by sending
ONYX_PATover SSH:
export ONYX_PAT=your-pat
ssh -o SendEnv=ONYX_PAT your-host -p 2222
Useful hardening flags:
--host-key(default~/.config/onyx-cli/host_ed25519)--idle-timeout(default15m)--max-session-timeout(default8h)--rate-limit-per-minute(default20)--rate-limit-burst(default40)--rate-limit-cache(default4096)
Commands
| Command | Mode | Description |
|---|---|---|
chat |
Interactive | Launch the interactive chat TUI (requires terminal) |
ask |
Agent / Script | Ask a question and print the answer to stdout |
agents |
Agent / Script | List available agents (ID, name, description) |
validate-config |
Agent / Script | Check CLI configuration and server connectivity |
install-skill |
Agent / Script | Install the Onyx CLI agent skill file |
experiments |
Agent / Script | List experimental features and their status |
serve |
Interactive | Serve the Onyx TUI over SSH |
Global Flags
| Flag | Description |
|---|---|
--version, -v |
Print client and server version information |
--debug |
Run in debug mode (verbose logging) |
Agent / Non-Interactive Use
When called without a TTY (e.g., by an AI agent or piped into another command), onyx-cli adjusts its behavior:
- No subcommand: prints help and exits 0 (instead of launching the TUI)
- Results to stdout, progress/errors to stderr
- No ANSI codes or interactive prompts
askoutput truncated to 50000 bytes by default; full response saved to a temp file. Use--max-output 0to disable.searchstdout stays valid JSON: over the limit, whole results are dropped and atruncationobject carries metadata plus the temp file path of the full response.
Configuration
If a human has already run onyx-cli chat (which includes first-time setup), the CLI works out of the box — no additional setup needed. Environment variables can override the config file or serve as an alternative when no config file exists:
export ONYX_SERVER_URL="https://your-onyx-server.com"
export ONYX_PAT="your-pat"
Exit Codes
| Code | Name | When |
|---|---|---|
| 0 | Success | Command completed |
| 1 | General | Unknown error |
| 2 | BadRequest | Invalid arguments |
| 3 | NotConfigured | Missing config/PAT |
| 4 | AuthFailure | Invalid PAT (401/403) |
| 5 | Unreachable | Server unreachable |
| 6 | RateLimited | Server returned 429 |
| 7 | Timeout | Request timed out |
| 8 | ServerError | Server returned 5xx |
| 9 | NotAvailable | Feature/endpoint doesn't exist |
Skill File
Install the bundled SKILL.md so AI coding agents can discover the CLI:
onyx-cli install-skill
onyx-cli install-skill --global
onyx-cli install-skill --copy
onyx-cli install-skill --agent claude-code
| Flag | Description |
|---|---|
--global, -g |
Install to home directory instead of project |
--copy |
Copy files instead of symlinking |
--agent, -a |
Target specific agents (e.g. claude-code; can be repeated) |
Slash Commands (in TUI)
| Command | Description |
|---|---|
/help |
Show help message |
/clear |
Clear chat and start a new session |
/agent |
List and switch agents (by ID or name) |
/attach <path> |
Attach a file to next message |
/sessions |
List recent chat sessions |
/configure |
Re-run connection setup |
/connectors |
Open connectors in browser |
/settings |
Open settings in browser |
/quit |
Exit Onyx CLI |
Keyboard Shortcuts
| Key | Action |
|---|---|
Enter |
Send message |
Escape |
Cancel current generation |
Ctrl+O |
Toggle source citations |
Ctrl+D |
Quit (press twice) |
Scroll / Shift+Up/Down |
Scroll chat history |
Page Up / Page Down |
Scroll half page |
Building from Source
Requires Go 1.24+.
cd cli
go build -o onyx-cli .
Development
# Run tests
go test ./...
# Build
go build -o onyx-cli .
# Lint
golangci-lint run ./...
Publishing to PyPI
The CLI is distributed as a Python package via PyPI. The build system uses hatchling with manygo to cross-compile Go binaries into platform-specific wheels.
CI release (recommended)
Tag a release and push — the release-cli.yml workflow builds wheels for all platforms and publishes to PyPI automatically. ods release cli calculates and pushes the next tag for you:
ods release cli # bumps the patch version
ods release cli --bump minor
ods release cli --version 1.5.0
ods release cli --dry-run # computes the version, tags nothing
To do this manually:
git tag cli/v0.1.0
git push origin cli/v0.1.0
Important
ods releasetagsHEAD, notmain, and does not check which branch you are on. Comparegit rev-parse HEADwithgit rev-parse origin/mainfirst. Tagging from a stale branch publishes that branch under the new version.
The workflow builds wheels for:
- linux/amd64 manylinux
- linux/amd64 musllinux
- linux/arm64 manylinux
- linux/arm64 musllinux
- darwin/amd64
- darwin/arm64
- windows/amd64
- windows/arm64
Manual release
Build a wheel locally with uv. Set GOOS and GOARCH to cross-compile for other platforms (Go handles this natively — no cross-compiler needed):
# Build for current platform
uv build --wheel
# Cross-compile for a different platform
GOOS=linux GOARCH=amd64 uv build --wheel
# Build a musllinux-tagged Linux wheel for Alpine/musl environments
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 \
ONYX_CLI_WHEEL_PLATFORM_TAG=musllinux_1_2_x86_64 \
uv build --wheel
# Upload to PyPI
uv publish
Build constraints
The release workflows build every wheel against
tools/requirements/build-constraints.txt, shared by cli, tools/ods and
tools/ods-audit. They pass it as UV_BUILD_CONSTRAINT with
UV_REQUIRE_HASHES, so the build environment (hatchling, go-bin, manygo and
their dependencies) is pinned and verified.
A plain local uv build resolves those dependencies freely. Set the same
variables to build against the pinned closure:
UV_BUILD_CONSTRAINT="$(git rev-parse --show-toplevel)/tools/requirements/build-constraints.txt" \
UV_REQUIRE_HASHES=true GOTOOLCHAIN=local \
uv build --wheel
tools/requirements/build-constraints.txt is compiled from its .in by the
pip-compile pre-commit hook, so a plain pre-commit run pip-compile refreshes
it. To upgrade a pinned build dependency, edit build-constraints.in and the
matching [build-system] requires in all three pyproject.toml files; the
build-constraints-drift hook fails if they disagree.
go-bin ships the Go toolchain used to compile the binary. Bumping Go means
moving it everywhere at once: the go directive in go.mod, cli/Dockerfile,
the setup-go and GO_VERSION pins across .github/workflows/,
.devcontainer/Dockerfile, the go-bin pin in all three pyproject.toml files
and in tools/requirements/build-constraints.in, and the compiled build-constraints.txt.
Versioning
Versions are derived from git tags with the cli/ prefix (e.g. cli/v0.1.0). The tag is parsed by internal/_version.py and injected into the Go binary via -ldflags at build time.