Commit Graph
90 Commits
Author SHA1 Message Date
trial-dansweranddanswer-trial 8b2ecb4eab EE movement followup for Standard Answers (#2467)
* Move StandardAnswer to EE section of danswer/db/models

* Move StandardAnswer DB layer to EE

* Add EERequiredError for distinct error handling here

* Handle EE fallback for slack bot config

* Migrate all standard answer models to ee

* Flagging categories for removal

* Add missing versioned impl for update_slack_bot_config

---------

Co-authored-by: danswer-trial <danswer-trial@danswer-trials-MacBook-Pro.local>
2024-09-16 22:05:53 +00:00
pablodanswer fc98c560a4 Add fix for logging (#2431) 2024-09-13 11:27:20 -07:00
rkuo-danswer da8e68b320 reformat celery logging to match danswer style logging across services (#2409)
* reformat celery logging to match danswer style logging across services

* mypy fixes

* handle logfile argument
2024-09-13 01:51:51 +00:00
pablodanswer 0d749ebd46 add ccpair id to logging (#2391) 2024-09-11 01:27:03 +00:00
rkuo-danswerandRichard Kuo f1c5e80f17 Feature/background processing (#2275)
* first cut at redis

* some new helper functions for the db

* ignore kombu tables in alembic migrations (used by celery)

* multiline commands for readability, add vespa_metadata_sync queue to worker

* typo fix

* fix returning tuple fields

* add constants

* fix _get_access_for_document

* docstrings!

* fix double function declaration and typing

* fix type hinting

* add a global redis pool

* Add get_document function

* use task_logger in various celery tasks

* add celeryconfig.py to simplify configuration. Will be used in a subsequent commit

* Add celery redis helper. used in a subsequent PR

* kombu warning getting spammy since celery is not self managing its queue in Postgres any more

* add last_modified and last_synced to documents

* fix task naming convention

* use celeryconfig.py

* the big one. adds queues and tasks, updates functions to use the queues with priorities, etc

* change vespa index log line to debug

* mypy fixes

* update alembic migration

* fix fence ordering, rename to "monitor", fix fetch_versioned_implementation call

* mypy

* switch to monotonic time

* fix startup dependencies on redis

* rebase alembic migration

* kombu cleanup - fail silently

* mypy

* add redis_host environment override

* update REDIS_HOST env var in docker-compose.dev.yml

* update the rest of the docker files

* harden indexing-status endpoint against db changes happening in the background.  Needs further improvement but OK for now.

* allow no task syncs to run because we create certain objects with no entries but initially marked as out of date

* add back writing to vespa on indexing

* update contributing guide

* backporting fixes from background_deletion

* renaming cache to cache_volume

* add redis password to various deployments

* try setting up pr testing for helm

* fix indent

* hopefully this release version actually exists

* fix command line option to --chart-dirs

* fetch-depth 0

* edit values.yaml

* try setting ct working directory

* bypass testing only on change for now

* move files and lint them

* update helm testing

* some issues suggest using --config works

* add vespa repo

* add postgresql repo

* increase timeout

* try amd64 runner

* fix redis password reference

* add comment to helm chart testing workflow

* rename helm testing workflow to disable it

* adding clarifying comments

* address code review

* missed a file

* remove commented warning ... just not needed

---------

Co-authored-by: Richard Kuo <rkuo@rkuo.com>
2024-09-10 16:28:19 +00:00
pablodanswer ebe3674ca7 update for edge case (#2336) 2024-09-05 17:58:49 +00:00
hagen-danswer c358c91e4c Added instance domain to telemetry (#2310) 2024-09-03 21:04:40 -07:00
pablodanswer 76db4b765a Detect GPU on startup for default multi-pass indexing value (#2242) 2024-08-30 17:38:31 -07:00
Yuhong Sun 1e67332078 Remove warning on user signup (#2252) 2024-08-27 18:49:05 -07:00
Weves c44c22a009 Fix model server 2024-08-19 07:23:24 -07:00
Yuhong Sun 5ab4d94d94 Logging Level Update (#2165) 2024-08-18 21:53:40 -07:00
Yuhong Sun 119aefba88 Add log files to containers (#2164) 2024-08-18 19:18:28 -07:00
Yuhong Sun 8a7bc4e411 Log Level Default (#2163) 2024-08-18 14:35:32 -07:00
rkuo-danswer 492797c9f3 Feature/indexing errors (#2148)
* backend changes to handle partial completion of index attempts

* typo fix

* Display partial success in UI

* make log timing more readable by limiting printed precision to milliseconds

* forgot alembic

* initial cut at "completed with errors" indexing

* remove and reorganize unused imports

* show view errors while indexing is in progress

* code review fixes
2024-08-18 19:14:32 +00:00
Yuhong Sun 739058aacc Logging updates (#2159) 2024-08-17 22:05:09 -07:00
Nathan SchwerdfegerandWeves c7e5b11c63 EE Connector Deletion Bugfix + Refactor (#2042)
---------

Co-authored-by: Weves <chrisweaver101@gmail.com>
2024-08-11 20:33:07 -07:00
Yuhong Sun c8ead6a0dc Need Reindexing Flag Setup (#2102) 2024-08-09 17:44:57 -07:00
Weves 51731ad0dd Fix issue where large docs/batches break openai embedding 2024-08-02 01:07:09 -07:00
Yuhong Sun f2f60c9cc0 Fix EE Import backoff Logic (#1959) 2024-07-27 11:06:11 -07:00
pablodanswer 87fadb07ea COMPLETE USER EXPERIENCE OVERHAUL (#1822) 2024-07-17 19:44:21 -07:00
Yuhong Sun 56b175f597 Fix Sitemap Robo (#1826) 2024-07-14 13:29:26 -07:00
Zoltan Szabo 1b311d092e Try to find the sitemap for a given site (#1538) 2024-07-14 13:24:10 -07:00
Weves 1ee8ee9e8b Prepare EE to merge with MIT 2024-06-25 15:07:56 -07:00
Weves 7cc51376f2 Allow basic seeding of Danswer via env variable 2024-06-25 15:07:56 -07:00
Chris Weaver db8ce61ff4 Fix group prefix (#11) 2024-06-25 15:07:56 -07:00
Weves eef54c8a86 Add non-ee fallback to fetch_versioned_implementation 2024-05-08 16:26:45 -07:00
Yuhong Sun 01476a37c3 Encrypted Sensitive Fields (#1423) 2024-05-06 18:02:42 -07:00
Yuhong Sun b59912884b Fix Model Server (#1320) 2024-04-10 23:13:22 -07:00
Yuhong Sun 2db906b7a2 Always Use Model Server (#1306) 2024-04-07 21:25:06 -07:00
Yuhong Sun c28a95e367 Port File Store from Volume to PG (#1241) 2024-03-21 20:10:08 -07:00
Yuhong Sun 3a6d32da7c Port KV Store to Postgres (#1227) 2024-03-19 16:21:22 -07:00
Chris WeaverandKevin Shi 2ace03081c More fetch_versioned_implementation logging (#1187)
---------

Co-authored-by: Kevin Shi <kevinshisvf@gmail.com>
2024-03-05 09:46:02 -08:00
Kevin Shi 0b0665044f @lru_cache on fetch_versioned_implementation (#1178) 2024-03-04 14:50:51 -08:00
Yuhong Sun c7d228e292 Trim Chunks if LLM tokenizer differs from Embedding tokenizer (#1143) 2024-02-28 13:01:32 -08:00
Yuhong Sun b3b88f05d3 Anonymous User Telem (#1041) 2024-02-04 13:41:19 -08:00
Yuhong Sun cf4ede2130 Embedding Models Table (#1006) 2024-01-26 18:40:53 -08:00
Yuhong Sun d0fa02c8dc Multiple Indices in Vespa (#1000) 2024-01-25 13:56:29 -08:00
Yuhong Sun 1e84b0daa4 Fix escape character handling in DanswerBot (#880) 2023-12-25 12:28:35 -08:00
Yuhong Sun c3cf9134bb Telemetry Revision (#868) 2023-12-24 17:39:37 -08:00
Yuhong Sun 65fde8f1b3 Chat Backend (#801) 2023-12-14 22:14:37 -08:00
Yuhong Sun fda89ac810 Expert Recommendation Heuristic Only (#791) 2023-11-29 15:53:57 -08:00
Chris Weaver 78d1ae0379 Customizable personas (#772)
Also includes a small fix to LLM filtering when combined with reranking
2023-11-28 00:57:48 -08:00
Yuhong Sun 39d09a162a Danswer APIs Document Ingestion Endpoint (#716) 2023-11-26 19:09:22 -08:00
Yuhong Sun fa0d19cc8c LLM Chunk Filtering (#735) 2023-11-18 17:12:24 -08:00
Yuhong Sun be5ef77896 Optional Anonymous Telemetry (#727) 2023-11-16 09:22:36 -08:00
Yuhong Sun 0125d8a0f6 Source Filter Extraction (#708) 2023-11-07 14:21:04 -08:00
Yuhong Sun 61c9343a7e Clean Up Duplicate Code (#670) 2023-10-31 23:25:26 -07:00
Yuhong Sun fe117513b0 Reorganize and Cleanup for Hybrid Search (#643) 2023-10-28 14:24:28 -07:00
Yuhong Sun e279918f95 Introduce Time Filters (#610) 2023-10-22 15:06:52 -07:00
Yuhong Sun d9076a6ff6 Use strict=False for Parsing LLM Jsons (#594) 2023-10-19 12:28:16 -07:00