DanelegendandClaude Opus 5 769ea3f7af feat(indexing): re-chunk tabular documents indexed by the old chunker (#16153) to release v4.8 (#16310) ...

Cherry-pick of #16153 to release v4.8 — the migration that re-chunks
tabular documents already indexed by the old chunker.

Upstream commit: e6772d6f41

**Conflict resolution** — mostly neighbouring main-only entries this
branch does not have:

- `app_base.py`, `constants.py`, `beat_schedule.py`: kept this branch's
entries and added only the `tabular_rechunk` ones. Upstream's
`cc_pair_ids_backfill` task, KV key and lock are not on this branch and
were not carried over.
- env templates and `values.yaml`: took the `TABULAR_RECHUNK_ENABLED`
entries, dropped the `SKIP_SPREADSHEET_CONTENT` ones (not a setting on
this branch).
- `Chart.yaml`: bumped this branch's version 0.8.28 → 0.8.29 and
replaced the change list, per the convention on this branch.
- `db/document.py`: kept both sides of the import block
(`KGEntity`/`KGRelationship` plus the new `FileRecord`), and added
`true` to the SQLAlchemy imports — `get_oversized_tabular_documents`
joins with it, and nothing on this branch imported it yet. The
cherry-pick also pulled in `PortedScope` and
`sample_ported_document_ids` as context; those belong to #14463, which
is not on this branch, so they were removed.

**Adapted to this branch's document index**, which still has both Vespa
and OpenSearch:

- `run_tabular_rechunk.py` imports `DocumentSectionRequest` from
`interfaces_new` (upstream's `interfaces` is the post-#15336 name).
- It resolves the index with `get_default_document_index(...)` instead
of `build_opensearch_document_index(...)`, typed `DocumentIndex`.
Hardcoding OpenSearch would read chunk heads from an index a
Vespa-serving deployment never populates, and every document would be
skipped as "not re-chunkable". `id_based_retrieval` has the same
signature on both backends. `test_run_tabular_rechunk.py` patches the
new name.
- `tabular_rechunk/tasks.py`: dropped the `SKIP_SPREADSHEET_CONTENT`
gate from `_blocked_reason`, since that setting does not exist here. The
other gates (disabled vector DB, reindex in flight, port backfilling)
are unchanged.

Verified: `tests/unit/onyx/background` and `tests/unit/onyx/indexing`
pass (303 passed, 6 skipped); `ruff check` clean on every file this
touches. `test_oversized_tabular_scan.py` came across unchanged and was
not run here (it needs Postgres).

**Stacked backport, merge last.** Order: #16296 → #16298 → #16308 →
#16309 → this PR.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Backports #16153, the one-time migration that re-chunks tabular
documents indexed by the old chunker, to v4.8. With
`TABULAR_RECHUNK_ENABLED` set, a beat task re-chunks oversized tabular
documents in place from the stored workbook, without re-crawling the
source; a KV record stores the cursor and completion state, so it
resumes across ticks and never re-runs once finished.

**Backport adaptations**

- Left out upstream content not on this branch: the
`cc_pair_ids_backfill` task, KV key, lock, and the `PortedScope` pieces
from #14463.
- The run imports `DocumentSectionRequest` from `interfaces_new`
(upstream's `interfaces` is the post-#15336 name), and resolves the
index with `get_default_document_index(...)` typed `DocumentIndex`
instead of `build_opensearch_document_index(...)`: this branch serves
both Vespa and OpenSearch, and hardcoding OpenSearch would skip every
document under a Vespa-serving deployment.
- The `SKIP_SPREADSHEET_CONTENT` gate was dropped from `_blocked_reason`
because that setting does not exist here; the remaining gates (disabled
vector DB, reindex in flight, port backfilling) are unchanged.
- `db/document.py` keeps both import blocks and adds `true` to the
SQLAlchemy imports; `miscellaneous_utils.py` gains
`experts_from_stores_representations` for the owner round-trip.

Verified: `tests/unit/onyx/background` and `tests/unit/onyx/indexing`
pass (303 passed, 6 skipped), and `ruff check` is clean on every touched
file. `test_oversized_tabular_scan.py` needs Postgres and was not run.

<sup>Written for commit 8e1c69b270.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16310?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16310?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16310&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

- [x] [Optional] Override Linear Check

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-10-21 02:18:17 +00:00

..

data/nginx

feat(cli): teach onyx-cli deploy to manage prod compose deployments (#13594)

2026-07-29 22:48:54 +00:00

docker_compose

feat(indexing): re-chunk tabular documents indexed by the old chunker (#16153) to release v4.8 (#16310)

2026-10-21 02:18:17 +00:00