mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-22 19:32:55 -05:00
DanelegendandClaude Opus 5
769ea3f7af
Cherry-pick of #16153 to release v4.8 — the migration that re-chunks tabular documents already indexed by the old chunker. Upstream commit:e6772d6f41**Conflict resolution** — mostly neighbouring main-only entries this branch does not have: - `app_base.py`, `constants.py`, `beat_schedule.py`: kept this branch's entries and added only the `tabular_rechunk` ones. Upstream's `cc_pair_ids_backfill` task, KV key and lock are not on this branch and were not carried over. - env templates and `values.yaml`: took the `TABULAR_RECHUNK_ENABLED` entries, dropped the `SKIP_SPREADSHEET_CONTENT` ones (not a setting on this branch). - `Chart.yaml`: bumped this branch's version 0.8.28 → 0.8.29 and replaced the change list, per the convention on this branch. - `db/document.py`: kept both sides of the import block (`KGEntity`/`KGRelationship` plus the new `FileRecord`), and added `true` to the SQLAlchemy imports — `get_oversized_tabular_documents` joins with it, and nothing on this branch imported it yet. The cherry-pick also pulled in `PortedScope` and `sample_ported_document_ids` as context; those belong to #14463, which is not on this branch, so they were removed. **Adapted to this branch's document index**, which still has both Vespa and OpenSearch: - `run_tabular_rechunk.py` imports `DocumentSectionRequest` from `interfaces_new` (upstream's `interfaces` is the post-#15336 name). - It resolves the index with `get_default_document_index(...)` instead of `build_opensearch_document_index(...)`, typed `DocumentIndex`. Hardcoding OpenSearch would read chunk heads from an index a Vespa-serving deployment never populates, and every document would be skipped as "not re-chunkable". `id_based_retrieval` has the same signature on both backends. `test_run_tabular_rechunk.py` patches the new name. - `tabular_rechunk/tasks.py`: dropped the `SKIP_SPREADSHEET_CONTENT` gate from `_blocked_reason`, since that setting does not exist here. The other gates (disabled vector DB, reindex in flight, port backfilling) are unchanged. Verified: `tests/unit/onyx/background` and `tests/unit/onyx/indexing` pass (303 passed, 6 skipped); `ruff check` clean on every file this touches. `test_oversized_tabular_scan.py` came across unchanged and was not run here (it needs Postgres). **Stacked backport, merge last.** Order: #16296 → #16298 → #16308 → #16309 → this PR. <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Backports #16153, the one-time migration that re-chunks tabular documents indexed by the old chunker, to v4.8. With `TABULAR_RECHUNK_ENABLED` set, a beat task re-chunks oversized tabular documents in place from the stored workbook, without re-crawling the source; a KV record stores the cursor and completion state, so it resumes across ticks and never re-runs once finished. **Backport adaptations** - Left out upstream content not on this branch: the `cc_pair_ids_backfill` task, KV key, lock, and the `PortedScope` pieces from #14463. - The run imports `DocumentSectionRequest` from `interfaces_new` (upstream's `interfaces` is the post-#15336 name), and resolves the index with `get_default_document_index(...)` typed `DocumentIndex` instead of `build_opensearch_document_index(...)`: this branch serves both Vespa and OpenSearch, and hardcoding OpenSearch would skip every document under a Vespa-serving deployment. - The `SKIP_SPREADSHEET_CONTENT` gate was dropped from `_blocked_reason` because that setting does not exist here; the remaining gates (disabled vector DB, reindex in flight, port backfilling) are unchanged. - `db/document.py` keeps both import blocks and adds `true` to the SQLAlchemy imports; `miscellaneous_utils.py` gains `experts_from_stores_representations` for the owner round-trip. Verified: `tests/unit/onyx/background` and `tests/unit/onyx/indexing` pass (303 passed, 6 skipped), and `ruff check` is clean on every touched file. `test_oversized_tabular_scan.py` needs Postgres and was not run. <sup>Written for commit8e1c69b270. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/onyx-dot-app/onyx/pull/16310?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="View guided diff" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <a href="https://www.cubic.dev/action/auto-fix/pr/onyx-dot-app/onyx/16310?returnTo=https%3A%2F%2Fgithub.com%2Fonyx-dot-app%2Fonyx%2Fpull%2F16310&source=description" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img alt="Turn on auto-fix" src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a> <!-- End of auto-generated description by cubic. --> - [x] [Optional] Override Linear Check Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
