* indexing status optimization first draft
* refactor: update pagination logic and enhance UI for indexing status table
* add index attempt pruning job and display federated connectors in index status page
* update celery worker command to include index_attempt_cleanup queue
* refactor: enhance indexing status table and remove deprecated components
* mypy fix
* address review comments
* fix pagination reset issue
* add TODO for optimizing connector materialization and performance in future deployments
* enhance connector indexing status retrieval by adding 'get_all_connectors' option and updating pagination logic
* refactor: transition to paginated connector indexing status retrieval and update related components
* fix: initialize latest_index_attempt_docs_indexed to 0 in CCPairIndexingStatusTable component
* feat: add mock connector file support for indexing status retrieval and update indexing_statuses type to Sequence
* mypy fix
* refactor: rename indexing status endpoint to simplify API and update related components
* WIP
* renamed and moved tasks (WIP)
* minio migration
* bug fixes and finally add document batch storage
* WIP: can suceed but status is error
* WIP
* import fixes
* working v1 of decoupled
* catastrophe handling
* refactor
* remove unused db session in prep for new approach
* renaming and docstrings (untested)
* renames
* WIP with no more indexing fences
* robustness improvements
* clean up rebase
* migration and salesforce rate limits
* minor tweaks
* test fix
* connector pausing behavior
* correct checkpoint resumption logic
* cleanups in docfetching
* add heartbeat file
* update template jsonc
* deployment fixes
* fix vespa httpx pool
* error handling
* cosmetic fixes
* dumb
* logging improvements and non checkpointed connector fixes
* didnt save
* misc fixes
* fix import
* fix deletion of old files
* add in attempt prefix
* fix attempt prefix
* tiny log improvement
* minor changes
* fixed resumption behavior
* passing int tests
* fix unit test
* fixed unit tests
* trying timeout bump to see if int tests pass
* trying timeout bump to see if int tests pass
* fix autodiscovery
* helm chart fixes
* helm and logging
* Split up engine file
* Switch to schema_translate_map
* Fix mass serach/replace
* Remove unused
* Fix mypy
* Fix
* Add back __init__.py
* kg fix for new session management
Adding "<tenant_id>" in front of all views.
* additional kg fix
* better handling
* improve naming
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* db setup
* transfer 1 - incomplete
* more adjustments
* relationship table + query update
* temp view creation
* restructuring
* nits
* updates
* separate read_only engine
* extraction revamp
* focus on metadata relatonships 1
* dev
* migration downgrade fix
* rebase migration change
* a3+
* progress
* base
* new extraction
* progress
* fixed KG extraction
* nits
* updates
* simplifications & cleanup
* fixes
* updates
* more feature flag checks
* fixes
* extraction process fix
* read-only user creation as part of setup
* fix for missing entity attributes
* kg read-only user creation as part of migration
* typo
* EL initial comments
* initial Account/SF Connector chnges
* SF Connector update
- include account information
* base w/ salesforce
* evan updates + quite a bit more
* kg-filtered search
* EL changes pt 2
* migrations and env vars
* quick migration fix
* migration update
* post_rebase fixes
* mypy fixes
* test fixes
* test fix
* test fix
* read_only pool + misc
* nf
* env vars
* test improvements
* salesforce fix
* test update
* small changes
* small adjustments
* SF Connector fix & kg_stage removal for one table
* mypy fix
* small fixes
* EL + RK (pt 1) comments
* nit
* setting updated
* Salesforce test update
* EL comments
* read-only user replacement & cleanup
* SQL View fix
* converting entity type-name separators
* sql view group ownership
* view fix
* SQL tweak
* dealing with docs that were skipped by indexing
* increased error handling
* more error handling
* Output formatting fix
* kg-incremental-reindexing
* 0-doc found improvement
* celery
* migration correction
* timeout adjustments
* nit
* Updated migration
* Entity Normalization for KG Dev 1 (#4746)
* feat: trigrams column
* fix: reranking and db
* feat: v1
* fix: convert to orm
* feat: parallel
* fix: default to id_name
* fix: renamed semantic_id and semantic_id_trigrams
* fix: scalar subquery
* fix: tuning + redundancy
* fix: threshold
* fix: typo
* fix: shorten names
* wip
* fix: reverted
* feat: config
* feat: works but it was dumb
* feat: clustering works
* fix: mypy
* normalization <-> language awareness for SQL generation
* small type fixes
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* mypy
* typo and dead code
* kg_time_fencing
* feat: remove temp views on migration downgrade
* remove functions and triggers for now
* rebase adjustments
* EL code review results
* quick fix + trigger/funcs for single tenant
* fix: typo, mypy, dead code
* fix: autoflake
* small updatesd
* nit
* fix: typo
* early + faster view creation
* Extension creation in MT migration
* nit changes to default ETs
* Incremental Clustering and KG Refactor V1 (#4784)
Optimized/restructured incremental clustering. New pipeline actually that moves vespa updates to clustering.
Also, celery configuration has been updated.
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* Move file
* Fix all prior imports
* Clean sidebar items logic; add kg page
* Add kg_processing celery background task
* prompt tweak & ET extraction reset
* more general hierarchical structure
* feat: better vespa reset logic
* Add basic knowledge graph configuration
* Add configurations for KG entity-type
* prompt optimization and entity replacemants
* small prompt changes
* Implement backend APIs
* KG Refactor V2 (#4814)
Clustering & Extraction improvements & various nits
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* add connector-level coverage days
* Update APIs to be more frontend ergonomic
* Add simple test
* Make config optional in test
* fix: nit
* initial EL responses
* refactor: helper functions for formatting
* fix: more helper fns & comments
* fix: comment code that's been implemented elsewhere
* Add entity-types APIs
* Hook up frontend to backend
* Finish hookup up entity-types to backend
* Update ordering of entity-types and fix form submitting
* Add backend API to get kg-exposed
* Add kg-exposed to sidebar
* Fix path
* Use existing values, even if kg-enabled is false
* Update what initial values are used
* Add skeleton for kg resetting
* Add return type
* Add default entity-type population when fetching entity-types
* Remove circular deps
* Minor fixes to logic
* Edit logic for default entity-types population
* Add re-index API + skeleton
* Update verbiage for KG
* Remove templatization in favour of function
* Address comments on PR
* Pull call out into its own binding
* Remove re-index API and revert implement of reset back to stub
* Fix circular import error
* Remove 'reindex' button
* Edit how the empty vendor name list is handled
* Edit how exposed is processed
* Redirect if navigated to `/admin/kg` and kg is not exposed
* Address comments on PR
* reset + entity type table display & updating updates
* Update fetching entity-types
* Make KG entity types refresh when reset
* Edit verbiage of reset button
* Update package-lock.json file
* Protect against overflowing
* Re-implement refreshing table after reset
* Edit message when nothing is shown.
* UI enhancements
* small fixes
* remove form validation?
* fix
* nit
* nit
* nit
* nit
* fix configure max coverage days
* EL comments for JR
* refactor: moved functions where they belong to fix circular import
* feat: intuitive coverage days
* feat: intuitive coverage days
* fix: safe date picker
* fix: startdate
* evan fixes
* fix: evan comment on enable/disable
* fix: style
* fix: ui issues
* fix: ui issues for reset too
* fix: tests
* fix: kg entity is not enabled
* fix: entity type reload on enable
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
Co-authored-by: Rei Meguro <36625832+Orbital-Web@users.noreply.github.com>
* Add error clarity to restart containers script
* erroneous cleanup on exit
* fix when starting containers for the first time
---------
Co-authored-by: Wenxi Onyx <wenxi-onyx@Wenxis-MacBook-Pro.local>
* add more info
* fix headers
* add filename as param (merge)
* db manager entry in launch template
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* add utility function
* add utility functions to DocExternalAccess
* refactor db access out of individual celery tasks and put it directly into the heavy task
* code review and remove leftovers
* fix circular imports
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* Update mode to be a default parameter in `FileStore.read`
* Move query history exporting process to be a background job instead
* Move hardcoded report-file-naming to a common utility function
* Add type annotations
* Update download component
* Implement button to re-ping and download CSV file; fix up some backend file-checking logic
* De-indent logic (w/ early return)
* Return different error codes dependings on the type of task status
* Add more resistant failure retrying mechanisms
* Remove default parameter in helper function
* Use popup for error messaging
* Update return code
* Update web/src/app/ee/admin/performance/query-history/DownloadAsCSV.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Add type to useState call
* Update backend/ee/onyx/server/query_history/api.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/onyx/file_store/file_store.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update backend/ee/onyx/background/celery/apps/primary.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Move rerender call to after check
* Run formatter
* Add type conversions back (smh greptile)
* Remove duplicated call to save_file
* Move non-fallible logic out of try-except block
* Pass date-ranges into API call
* Convert to ISO strings before passing it into the API call
* Add API to list all tasks
* Create new pydantic model to represent tasks to return instead
* Change helper to only fetch query-history tasks
* Use `shared_tasks` instead of old method
* Address more comments from PR; consolidate how task name is generated
* Mark task as failed if any exception is raised
* Change the task object which is returned back to the FE
* Add a table to display previously generated query-history-csv's
* Add timestamps to task; delete tasks as soon as file finishes processing
* Raise exception if start_time is not present
* Convert hard-coded string to constant
* Add "Generated At" field to table
* Return task list in sorted order (based off of start-time)
* Implement pagination
* Remove unused props and cleanup tailwind classes
* Change the name of kickoff button
* Redesign how previous query exports are viewed
* Make button a constant width even when contents change
* Remove timezone information before comparing
* Decrease interval time for re-pinging API
* Add timezone to start-time creation
* Add a refreshInterval for getting updated task status
* Add new background queue
* Edit small verbiage and remove error popup when max-retries is hit
* Change up heavy worker to recognize new task in new module
* Ensure `celery_app` is imported
* Change how `celery_app` is imported and defined
* Update comment on why `celery_app` must be imported
* Add basic skeleton for new beat task to cleanup any dead / failed query-history-export tasks
* Move cleanup task to different worker / queue
* Implement cleanup task
* Add return type
* Address comment on PR
* Remove delimiter from prefix
* Change name of function to be more descriptive
* Remove delimiter from prefix constant
* Move function invocation closer to usage location
* Move imports to top of file
* Move variable up a scope due to undefined error
* Remove dangling if-statement
* Make function more pure-functional
* Remove redefinition
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* debug script + slight refactor of db class
* better comments
* move setup logger
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* tool to generate vespa schema variations for our cloud
* extraneous assign
* use a real templating system instead of search/replace
* fix float
* maybe this should be double
* remove redundant var
* template the other files
* try a spawned process
* move the wrapper
* fix args
* increase timeout
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* tool to generate vespa schema variations for our cloud
* extraneous assign
* float, not double
* back to double
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* initial working version
* ranking profile
* modification for keyword/instruction retrieval
* mypy fixes
* EL comments
* added env var (True for now)
* flipped default to False
* mypy & final EL/CW comments + import issue
* bump fastapi and starlette
* bumping llama index and nltk and associated deps
* bump to fix python-multipart
* bump aiohttp
* update package lock for examples/widget
* bump black
* sentencesplitter has changed namespaces
* fix reorder import check, fix missing passlib
* update package-lock.json
* black formatter updated
* reformatted again
* change to black compatible reorder
* change to black compatible reorder-python-imports fork
* fix pytest dependency
* black format again
* we don't need cdk.txt. update packages to be consistent across all packages
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* fix acl prefixing
* increase timeout a tad
* block access to init'ing DocumentAccess directly, fix test to work with ee/MIT
* fix env var checks
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
* remove title for slack
* initial working code
* simplification
* improvements
* name change to information_content_model
* avoid boost_score > 1.0
* nit
* EL comments and improvements
Improvements:
- proper import of information content model from cache or HF
- warm up for information content model
Other:
- EL PR review comments
* nit
* requirements version update
* fixed docker file
* new home for model_server configs
* default off
* small updates
* YS comments - pt 1
* renaming to chunk_boost & chunk table def
* saving and deleting chunk stats in new table
* saving and updating chunk stats
* improved dict score update
* create columns for individual boost factors
* RK comments
* Update migration
* manual import reordering
* early work in progress
* rename utility script
* move actual data seeding to a shareable function
* add test
* make the test pass with the fix
* fix comment
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* wip checkpointing/continue on failure
more stuff for checkpointing
Basic implementation
FE stuff
More checkpointing/failure handling
rebase
rebase
initial scaffolding for IT
IT to test checkpointing
Cleanup
cleanup
Fix it
Rebase
Add todo
Fix actions IT
Test more
Pagination + fixes + cleanup
Fix IT networking
fix it
* rebase
* Address misc comments
* Address comments
* Remove unused router
* rebase
* Fix mypy
* Fixes
* fix it
* Fix tests
* Add drop index
* Add retries
* reset lock timeout
* Try hard drop of schema
* Add timeout/retries to downgrade
* rebase
* test
* test
* test
* Close all connections
* test closing idle only
* Fix it
* fix
* try using null pool
* Test
* fix
* rebase
* log
* Fix
* apply null pool
* Fix other test
* Fix quality checks
* Test not using the fixture
* Fix ordering
* fix test
* Change pooling behavior
