Files
onyx/cli
DanelegendandClaude Opus 5 e6772d6f41 feat(indexing): re-chunk tabular documents indexed by the old chunker (#16153)
Documents indexed before the bounded-sketch chunker (#16115) still hold
hundreds of row chunks each. `TABULAR_RECHUNK_ENABLED` arms a beat task
that re-chunks them in place, from the raw workbook Onyx already stores,
without re-crawling the source.

An operator turns it on by deploying with the variable set and nothing
else.

## Why a normal re-index cannot do this

`Document.content_hash()` walks Text and Image sections only, so a
tabular document's hash never moves however much the workbook changed.
The content-hash gate therefore skips every spreadsheet on any re-index,
and `force_update` — the only bypass — was wired solely into the
per-document retry path. This exposes it on `run_indexing_pipeline`;
that is the one change to shared indexing code, and it is a no-op unless
passed.

## How it picks its work

The scan walks `document` by primary key and takes only tabular
documents holding more chunks than the current chunker could produce
(`TABULAR_MAX_CHUNKS_PER_WORKBOOK` plus the large chunks multipass
derives from them — 30 by default). That bound means it skips its own
output and anything an in-flight crawl writes, so it can run during an
initial index without fighting it. Scanning by primary key keeps every
page an index scan: one pass of the table over the migration's life, not
a sort per run.

A KV record holds the cursor and a completed flag. Once the scan ends,
each later tick is one cached read, so leaving the variable set across
restarts re-runs nothing.

## Preserving what the upsert would overwrite

Re-indexing an unchanged document means `_upsert_documents_in_db` writes
the rebuilt document over a row that was already correct, so anything
the rebuild drops is destroyed corpus-wide:

- `metadata` and `title` are not kept in Postgres. They are read back
from the document's own stored chunk, as the reindex port does for
documents it cannot re-fetch.
- `doc_metadata`, owners, hierarchy and `file_id` come from the row.
Owners round-trip through their stored string form.
- `external_access` is left unset, so `includes_permissions` stays false
and the upsert leaves perm-synced ACL columns alone.

Unit tests pin each of these, plus the content-hash blindness that makes
`force_update` necessary.

## Guards

Pauses while a FUTURE index exists (the pipeline would write there
instead) or a port backfill is pending (the port copies chunks boundary
for boundary). Skips entirely under `SKIP_SPREADSHEET_CONTENT` or
`DISABLE_VECTOR_DB`. Documents whose raw file was never staged are left
out — only a crawl can rebuild those.

## Not yet verified

**This has never been run end to end.** The unit tests cover the
document rebuild and the gate reasoning; the query plan, the chunk-head
read and the delete-then-write replacement are unexercised. An
external-dependency-unit test over one real oversized document is the
obvious next step, and I'd want that before this points at a large
deployment.

Known side effects: it takes a docprocessing thread for up to 4 minutes
per 5-minute tick; every re-chunked document becomes eligible for
metadata sync; and OpenSearch reclaims disk on segment merge, so disk
lags chunk count.

## Reverting

No migration, no schema change, no data model change. Removing it is
deleting three files and reverting small hunks in thirteen; the two that
touch shared paths (`force_update`, `get_chunk_heads`) are inert on
their own.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds an opt-in beat task, armed by `TABULAR_RECHUNK_ENABLED`, that
re-chunks tabular documents indexed before the bounded-sketch chunker,
in place from the workbook already in the file store, without
re-crawling the source. A KV record stops it once done, so leaving the
variable set across restarts re-runs nothing.

- The scan walks the `document` table by primary key, resolving the
cc-pair with a LATERAL so every page stays an index scan, and touches
only documents holding more chunks than the current chunker could
produce. Hidden documents are passed over, since the chunk-head read
never returns one.
- The beat entry is registered only when the variable is set, and the
task checks the completed flag before taking the lock, so a finished
migration costs one cached read per tick. Ticks run at HIGH priority
with a 4-minute expiry on the docprocessing queue.
- The re-index passes `force_update` and `skip_document_row_writes`, so
only the chunks and `chunk_count` change. Re-indexing no longer resets
admin-set `boost`/`hidden`, and ancestors resolve from
`parent_hierarchy_node_id`.
- Title and metadata are read back from the document's own stored chunk,
fetched by deterministic id; owners are restored from the row because
the chunk writer reads them off the document; Drive sheets are matched
and dispatched by stored MIME type.
- The migration's adapter re-reads `file_id` and `chunk_count` under the
document lock and aborts the batch if a crawl rewrote a document, so an
older workbook never replaces newer chunks.
- A re-chunk that fails after chunking no longer commits the smaller
count, so the document stays oversized and the next pass finds it by
rescanning; retries are bounded at three passes.
- Skips under `SKIP_SPREADSHEET_CONTENT` or `DISABLE_VECTOR_DB`; pauses
while a FUTURE index exists, while an INSTANT-swap port is backfilling,
or if the live index changes—re-checked under the document lock too,
since a swap landing mid-batch would write into the retired index. The
lock lease is extended per batch so a throttled batch can't let a second
tick start from the same cursor.

**Not yet verified**

- The migration has never completed end to end; it ran once against a
live index, which surfaced the port guard bug now fixed.

<sup>Written for commit 8cdc3ffac0.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16153?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16153?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16153&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added an opt-in, one-time migration to re-chunk oversized tabular
documents using existing indexed content, without requiring a source
re-crawl.
- Migration progress is saved and resumes safely across restarts, with
retries for eligible failures.
- Documents that are hidden, unavailable, or being actively changed are
skipped or retried safely.

- **Documentation**
- Added configuration guidance for enabling the tabular re-chunk
migration.
- Clarified migration behavior, completion tracking, scheduling, and
vector database requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-20 00:33:40 +00:00
..

Onyx CLI

Release CLI PyPI

A CLI for querying enterprise knowledge from Onyx. Includes an interactive chat TUI for humans and non-interactive commands for AI agents and scripts.

Installation

pip install onyx-cli

Or with uv:

uv pip install onyx-cli

Standalone binaries for Linux, macOS, and Windows (amd64/arm64) are also attached to each cli/v* GitHub release.

Setup

Run the interactive chat TUI — on first launch it will guide you through setup:

onyx-cli chat

This prompts for your Onyx server URL and personal access token (PAT), tests the connection, and saves config to ~/.config/onyx-cli/config.json (or $XDG_CONFIG_HOME/onyx-cli/config.json if set). To reconfigure later, use the /configure command inside the TUI.

Environment variables override config file values:

Variable Required Description
ONYX_SERVER_URL No Server origin or already-prefixed API base (default: https://cloud.onyx.app)
ONYX_API_PREFIX No API path prefix (default: /api); set to empty for direct backend access
ONYX_PAT No Personal access token for authentication (required if no config file)
ONYX_PERSONA_ID No Default agent/persona ID
ONYX_STREAM_MARKDOWN No Enable/disable progressive markdown rendering (true/false)
ONYX_SSH_HOST_KEY No Path to SSH host key for serve command

Usage

Interactive chat

onyx-cli chat
onyx-cli chat --no-stream-markdown
onyx-cli chat --agent-name "Support Agent"
Flag Description
--no-stream-markdown Disable progressive markdown rendering during streaming
--agent-id <int> Agent ID to use for this session
--agent-name <string> Agent name to use for this session (exact or unique substring; mutually exclusive with --agent-id)

One-shot question

onyx-cli ask "What is our company's PTO policy?"
onyx-cli ask --agent-id 5 "Summarize this topic"
onyx-cli ask --agent-name "Support Agent" "hello"
onyx-cli ask --json "Hello"
Flag Description
--agent-id <int> Agent ID to use (overrides default)
--agent-name <string> Agent name to use (exact or unique substring; mutually exclusive with --agent-id)
--json Output NDJSON stream events instead of plain text
--prompt <string> Question text (use with piped stdin context)
--quiet Buffer output and print once at end
--max-output <int> Max bytes before truncating (0 to disable)

List agents

onyx-cli agents
onyx-cli agents --json

Serve over SSH

# Start a public SSH endpoint for the CLI TUI
onyx-cli serve --host 0.0.0.0 --port 2222

# Connect as a client
ssh your-host -p 2222

Clients can either:

  • paste a personal access token (PAT) at the login prompt, or
  • skip the prompt by sending ONYX_PAT over SSH:
export ONYX_PAT=your-pat
ssh -o SendEnv=ONYX_PAT your-host -p 2222

Useful hardening flags:

  • --host-key (default ~/.config/onyx-cli/host_ed25519)
  • --idle-timeout (default 15m)
  • --max-session-timeout (default 8h)
  • --rate-limit-per-minute (default 20)
  • --rate-limit-burst (default 40)
  • --rate-limit-cache (default 4096)

Commands

Command Mode Description
chat Interactive Launch the interactive chat TUI (requires terminal)
ask Agent / Script Ask a question and print the answer to stdout
agents Agent / Script List available agents (ID, name, description)
validate-config Agent / Script Check CLI configuration and server connectivity
install-skill Agent / Script Install the Onyx CLI agent skill file
experiments Agent / Script List experimental features and their status
serve Interactive Serve the Onyx TUI over SSH

Global Flags

Flag Description
--version, -v Print client and server version information
--debug Run in debug mode (verbose logging)

Agent / Non-Interactive Use

When called without a TTY (e.g., by an AI agent or piped into another command), onyx-cli adjusts its behavior:

  • No subcommand: prints help and exits 0 (instead of launching the TUI)
  • Results to stdout, progress/errors to stderr
  • No ANSI codes or interactive prompts
  • ask output truncated to 50000 bytes by default; full response saved to a temp file. Use --max-output 0 to disable.
  • search stdout stays valid JSON: over the limit, whole results are dropped and a truncation object carries metadata plus the temp file path of the full response.

Configuration

If a human has already run onyx-cli chat (which includes first-time setup), the CLI works out of the box — no additional setup needed. Environment variables can override the config file or serve as an alternative when no config file exists:

export ONYX_SERVER_URL="https://your-onyx-server.com"
export ONYX_PAT="your-pat"

Exit Codes

Code Name When
0 Success Command completed
1 General Unknown error
2 BadRequest Invalid arguments
3 NotConfigured Missing config/PAT
4 AuthFailure Invalid PAT (401/403)
5 Unreachable Server unreachable
6 RateLimited Server returned 429
7 Timeout Request timed out
8 ServerError Server returned 5xx
9 NotAvailable Feature/endpoint doesn't exist

Skill File

Install the bundled SKILL.md so AI coding agents can discover the CLI:

onyx-cli install-skill
onyx-cli install-skill --global
onyx-cli install-skill --copy
onyx-cli install-skill --agent claude-code
Flag Description
--global, -g Install to home directory instead of project
--copy Copy files instead of symlinking
--agent, -a Target specific agents (e.g. claude-code; can be repeated)

Slash Commands (in TUI)

Command Description
/help Show help message
/clear Clear chat and start a new session
/agent List and switch agents (by ID or name)
/attach <path> Attach a file to next message
/sessions List recent chat sessions
/configure Re-run connection setup
/connectors Open connectors in browser
/settings Open settings in browser
/quit Exit Onyx CLI

Keyboard Shortcuts

Key Action
Enter Send message
Escape Cancel current generation
Ctrl+O Toggle source citations
Ctrl+D Quit (press twice)
Scroll / Shift+Up/Down Scroll chat history
Page Up / Page Down Scroll half page

Building from Source

Requires Go 1.24+.

cd cli
go build -o onyx-cli .

Development

# Run tests
go test ./...

# Build
go build -o onyx-cli .

# Lint
golangci-lint run ./...

Publishing to PyPI

The CLI is distributed as a Python package via PyPI. The build system uses hatchling with manygo to cross-compile Go binaries into platform-specific wheels.

Tag a release and push — the release-cli.yml workflow builds wheels for all platforms and publishes to PyPI automatically. ods release cli calculates and pushes the next tag for you:

ods release cli              # bumps the patch version
ods release cli --bump minor
ods release cli --version 1.5.0
ods release cli --dry-run    # computes the version, tags nothing

To do this manually:

git tag cli/v0.1.0
git push origin cli/v0.1.0

Important

ods release tags HEAD, not main, and does not check which branch you are on. Compare git rev-parse HEAD with git rev-parse origin/main first. Tagging from a stale branch publishes that branch under the new version.

The workflow builds wheels for:

  • linux/amd64 manylinux
  • linux/amd64 musllinux
  • linux/arm64 manylinux
  • linux/arm64 musllinux
  • darwin/amd64
  • darwin/arm64
  • windows/amd64
  • windows/arm64

Manual release

Build a wheel locally with uv. Set GOOS and GOARCH to cross-compile for other platforms (Go handles this natively — no cross-compiler needed):

# Build for current platform
uv build --wheel

# Cross-compile for a different platform
GOOS=linux GOARCH=amd64 uv build --wheel

# Build a musllinux-tagged Linux wheel for Alpine/musl environments
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 \
  ONYX_CLI_WHEEL_PLATFORM_TAG=musllinux_1_2_x86_64 \
  uv build --wheel

# Upload to PyPI
uv publish

Build constraints

The release workflows build every wheel against tools/requirements/build-constraints.txt, shared by cli, tools/ods and tools/ods-audit. They pass it as UV_BUILD_CONSTRAINT with UV_REQUIRE_HASHES, so the build environment (hatchling, go-bin, manygo and their dependencies) is pinned and verified.

A plain local uv build resolves those dependencies freely. Set the same variables to build against the pinned closure:

UV_BUILD_CONSTRAINT="$(git rev-parse --show-toplevel)/tools/requirements/build-constraints.txt" \
  UV_REQUIRE_HASHES=true GOTOOLCHAIN=local \
  uv build --wheel

tools/requirements/build-constraints.txt is compiled from its .in by the pip-compile pre-commit hook, so a plain pre-commit run pip-compile refreshes it. To upgrade a pinned build dependency, edit build-constraints.in and the matching [build-system] requires in all three pyproject.toml files; the build-constraints-drift hook fails if they disagree.

go-bin ships the Go toolchain used to compile the binary. Bumping Go means moving it everywhere at once: the go directive in go.mod, cli/Dockerfile, the setup-go and GO_VERSION pins across .github/workflows/, .devcontainer/Dockerfile, the go-bin pin in all three pyproject.toml files and in tools/requirements/build-constraints.in, and the compiled build-constraints.txt.

Versioning

Versions are derived from git tags with the cli/ prefix (e.g. cli/v0.1.0). The tag is parsed by internal/_version.py and injected into the Go binary via -ldflags at build time.