* indexing status optimization first draft
* refactor: update pagination logic and enhance UI for indexing status table
* add index attempt pruning job and display federated connectors in index status page
* update celery worker command to include index_attempt_cleanup queue
* refactor: enhance indexing status table and remove deprecated components
* mypy fix
* address review comments
* fix pagination reset issue
* add TODO for optimizing connector materialization and performance in future deployments
* enhance connector indexing status retrieval by adding 'get_all_connectors' option and updating pagination logic
* refactor: transition to paginated connector indexing status retrieval and update related components
* fix: initialize latest_index_attempt_docs_indexed to 0 in CCPairIndexingStatusTable component
* feat: add mock connector file support for indexing status retrieval and update indexing_statuses type to Sequence
* mypy fix
* refactor: rename indexing status endpoint to simplify API and update related components
* WIP
* renamed and moved tasks (WIP)
* minio migration
* bug fixes and finally add document batch storage
* WIP: can suceed but status is error
* WIP
* import fixes
* working v1 of decoupled
* catastrophe handling
* refactor
* remove unused db session in prep for new approach
* renaming and docstrings (untested)
* renames
* WIP with no more indexing fences
* robustness improvements
* clean up rebase
* migration and salesforce rate limits
* minor tweaks
* test fix
* connector pausing behavior
* correct checkpoint resumption logic
* cleanups in docfetching
* add heartbeat file
* update template jsonc
* deployment fixes
* fix vespa httpx pool
* error handling
* cosmetic fixes
* dumb
* logging improvements and non checkpointed connector fixes
* didnt save
* misc fixes
* fix import
* fix deletion of old files
* add in attempt prefix
* fix attempt prefix
* tiny log improvement
* minor changes
* fixed resumption behavior
* passing int tests
* fix unit test
* fixed unit tests
* trying timeout bump to see if int tests pass
* trying timeout bump to see if int tests pass
* fix autodiscovery
* helm chart fixes
* helm and logging
* Split up engine file
* Switch to schema_translate_map
* Fix mass serach/replace
* Remove unused
* Fix mypy
* Fix
* Add back __init__.py
* kg fix for new session management
Adding "<tenant_id>" in front of all views.
* additional kg fix
* better handling
* improve naming
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
Jamison Lahman