Weves
d9adee168b
Add simple job client to try and get rid of some of the flakiness / weirdness that we are seeing with Dask
2023-11-01 17:43:58 -07:00
Weves
9cd0c197e7
Fix frozen jobs
2023-11-01 14:30:51 -07:00
Yuhong Sun
61c9343a7e
Clean Up Duplicate Code ( #670 )
2023-10-31 23:25:26 -07:00
Yuhong Sun
0618b59de6
Fix Indexing Frozen ( #660 )
2023-10-30 20:49:39 -07:00
Weves
517a539d7e
Make indexing jobs use more cores again
2023-10-30 18:50:40 -07:00
Weves
815c30c9d0
Fix jobs erroring while waiting on queue
2023-10-29 01:35:04 -07:00
Yuhong Sun
57ecab0098
Fix Json Output Issue and Fix miscount of new docs per Index Attempt ( #641 )
2023-10-29 00:34:28 -07:00
Weves
bfa338e142
Adjust time_updated assignment + increase frozen timeout to 3hrs
2023-10-28 22:27:18 -07:00
Yuhong Sun
fe117513b0
Reorganize and Cleanup for Hybrid Search ( #643 )
2023-10-28 14:24:28 -07:00
Yuhong Sun
604e511c09
Alternative solution to up the number of threads for torch ( #632 )
2023-10-25 22:30:57 -07:00
Chris Weaver
89807c8c05
Fix deletion status display + add celery util + fix seg faults ( #615 )
2023-10-22 19:41:29 -07:00
Yuhong Sun
e279918f95
Introduce Time Filters ( #610 )
2023-10-22 15:06:52 -07:00
Chris Weaver
e519dfc849
Torch more cpus ( #583 )
2023-10-17 09:53:20 -07:00
Yuhong Sun
b5982c10c3
Celery Beat ( #575 )
2023-10-16 14:59:42 -07:00
Yuhong Sun
dbf59d2acc
Dockerfile to build smaller Images ( #567 )
2023-10-12 13:08:47 -07:00
Weves
876c6fdaa6
Address bug with automatic document set cleanup on connector deletion
2023-10-11 14:42:45 -07:00
Weves
fa460f4da1
Small tweak to the connector deletion flow
2023-10-10 16:38:36 -07:00
Weves
091c2c8a80
Automatically delete document set relationships when deleting a ConnectorCredentialPair
2023-10-10 15:32:18 -07:00
Chris Weaver
9c89ae78ba
Move is_public from Credential to ConnectorCredentialPair ( #523 )
2023-10-05 20:55:41 -07:00
Weves
dbe33959c0
Move auto-ACL update to background job
2023-10-02 00:37:51 -07:00
Chris Weaver
0c58c8d6cb
Adding Document Sets ( #477 )
...
Adds:
- name for connector credential pairs + frontend changes to start populating this field
- document set table migration
- during indexing, document sets are now checked and inserted into Vespa
- background job to check if document sets need to be synced
- document set management APIs
- document set management dashboard in the UI
2023-09-26 12:53:19 -07:00
Chris Weaver
8594bac30b
Transition to using access_control_list to manage access in Vespa ( #450 )
2023-09-26 12:26:39 -07:00
Yuhong Sun
8b95e2631d
Make Cross Encoders Optional ( #476 )
2023-09-23 17:17:54 -07:00
Weves
c4e0face9b
Move connector / credential pair deletion to celery
2023-09-14 16:23:13 -07:00
Weves
e72f26ef53
Fix indexing job cleanup
2023-09-10 11:01:34 -07:00
Weves
58b75122f1
Fix deletion for overlapping connectors
2023-09-04 21:12:08 -07:00
Yuhong Sun
43efa9da94
Mark incomplete Index Attempts as Failed on job restart ( #371 )
2023-08-31 17:43:03 -07:00
Yuhong Sun
ac2a4f9051
Ruff Styling ( #368 )
2023-08-31 15:55:01 -07:00
Yuhong Sun
c1727e63ad
bug squashed
2023-08-31 14:32:13 -07:00
Chris Weaver
038f646c09
Fe for feedback ( #346 )
2023-08-30 12:52:24 -07:00
Yuhong Sun and Weves
b2a51283d1
Learn from feedback backend ( #343 )
...
---------
Co-authored-by: Weves <chrisweaver101@gmail.com >
2023-08-28 13:29:29 -07:00
Yuhong Sun
a2ec1e2cda
Vespa Deployment ( #330 )
...
Large Change!
2023-08-26 15:35:19 -07:00
Yuhong Sun
384bf1befe
Warm up models before first document indexed ( #333 )
2023-08-24 20:01:50 -07:00
Yuhong Sun
8159fdcdce
Add Vespa and rework Document Indices ( #317 )
2023-08-24 08:46:28 -07:00
Chris Weaver
e307275774
Add support for multiple indexing workers ( #322 )
2023-08-22 18:11:31 -07:00
Chris Weaver
067503bc84
Background logs to stdout ( #315 )
2023-08-18 19:02:32 -07:00
Weves
820f8b7b48
Add document-level logging for each batch of indexed documents
2023-08-15 18:09:39 -07:00
Weves
ecb26ddaf7
Add polling range for updates
2023-08-12 18:04:48 -07:00
Chris Weaver
d5bb10b61f
Improve indexing status display ( #278 )
...
Adds:
- actual error message in UI for indexing failure
- if a connector is disabled, stops indexing immediately (after the current batch of documents) to allow for deletion
- adds num docs indexed for the current run + a speed
2023-08-12 14:49:04 -07:00
Chris Weaver
89f71ac335
Support deletion of documents when a connector is deleted ( #271 )
2023-08-09 00:53:42 -07:00
Chris Weaver
132a9f750d
Add Github Action to run mypy / reorder-python-imports / black on all PRs ( #251 )
...
Also fixes import ordering (previously, local imports weren't grouped together as they should have been)
2023-07-29 16:53:38 -07:00
Yuhong Sun
87fe6f7575
Add ingestion metrics ( #256 )
2023-07-29 16:37:22 -07:00
Chris Weaver
3e8f5fa47e
Fix a few bugs with Google Drive polling ( #250 )
...
- Adds some offset to the `start` for the Google Drive connector to give time for `modifiedTime` to propagate so we don't miss updates
- Moves fetching folders into a separate call since folder `modifiedTime` doesn't get updated when a file in the folder is updated
- Uses `connector_credential_pair.last_successful_index_time` instead of `updated_at` to determine the `start` for poll connectors
2023-07-28 18:27:32 -07:00
Yuhong Sun
e019db0bc7
Indexing Job has timezone discrepancy with DB making Poll timeframes incorrect ( #231 )
2023-07-23 21:59:00 -07:00
Yuhong Sun
d6ca865034
Support GPT4All in memory ( #230 )
2023-07-23 12:26:14 -07:00
Yuhong Sun
0708002953
Check for Credential delete before running queued index attempt ( #221 )
2023-07-19 23:52:48 -07:00
Jan Kantert
7290f1893d
catch crash when deleting a datasource
...
Danswer background crashes when the index task for a deleted source is still in the task queue. Without this is won't recover without manual database cleanup.
2023-07-18 13:42:16 +02:00
Yuhong Sun
4b699fdab3
Better Logging ( #184 )
2023-07-16 01:41:48 -07:00
Weves
620579cbec
Fix connection pooling
2023-06-19 14:45:26 -06:00
Weves
490d39f081
Removing deprecated field
2023-06-19 14:45:26 -06:00
Weves