starred/onyx
1
0 Fork 0
mirror of https://github.com/onyx-dot-app/onyx.git synced 2026-10-23 03:42:59 -05:00
Code Issues Packages Projects Releases 10 Wiki Activity
4,135 Commits 2,107 Branches 2,544 Tags
391e710b6e73af9b87fb86e8500a203f00a60e8d
Commit Graph
47 Commits
Author SHA1 Message Date
Arun Philip e74a0398dc Update Docker Compose restart policy to unless-stopped 
Changed the restart policy to unless-stopped to ensure containers
automatically restart after failures or reboots but allow manual stop
without immediate restart.

This is preferable over always because it prevents containers from
restarting automatically after a manual stop, enabling controlled
shutdowns and maintenance without unintended restarts.
 2025-06-24 13:27:50 -07:00
joachim-danswer 3272050975 docker dev and prod template (#4936) 
* docker dev and prod template

* more dev files
 2025-06-23 21:43:42 +00:00
Chris Weaver 6f8176092e S3 like file store (#4897) 
* Move to an S3-like file store

* Add non-mocked test

* Add S3 tests

* Improve migration / add auto-running tests

* Refactor

* Fix mypy

* Small fixes

* Improve migration to handle downgrades

* fix file store tests

* Fix file store tests again

* Fix file store tests again

* Fix mypy

* Fix default values

* Add MinIO to other compose files

* Working helm w/ minio

* Fix test

* Address greptile comments

* Harden migration

* Fix README

* Fix it

* Address more greptile comments

* Fix it

* Rebase

* Handle multi-tenant case

* Fix mypy

* Fix test

* fix test

* Improve migration

* Fix test
 2025-06-20 14:22:05 -07:00
joachim-danswerandRei Meguro c5adbe4180 Knowledge Graph v1 (#4626) 
* db setup

* transfer 1 - incomplete

* more adjustments

* relationship table + query update

* temp view creation

* restructuring

* nits

* updates

* separate read_only engine

* extraction revamp

* focus on metadata relatonships 1

* dev

* migration downgrade fix

* rebase migration change

* a3+

* progress

* base

* new extraction

* progress

* fixed KG extraction

* nits

* updates

* simplifications & cleanup

* fixes

* updates

* more feature flag checks

* fixes

* extraction process fix

* read-only user creation as part of setup

* fix for missing entity attributes

* kg read-only user creation as part of migration

* typo

* EL initial comments

* initial Account/SF Connector chnges

* SF Connector update

 - include account information

* base w/ salesforce

* evan updates + quite a bit more

* kg-filtered search

* EL changes pt 2

* migrations and env vars

* quick migration fix

* migration update

* post_rebase fixes

* mypy fixes

* test fixes

* test fix

* test fix

* read_only pool + misc

* nf

* env vars

* test improvements

* salesforce fix

* test update

* small changes

* small adjustments

* SF Connector fix & kg_stage removal for one table

* mypy fix

* small fixes

* EL + RK (pt 1) comments

* nit

* setting updated

* Salesforce test update

* EL comments

* read-only user replacement & cleanup

* SQL View fix

* converting entity type-name separators

* sql view group ownership

* view fix

* SQL tweak

* dealing with docs that were skipped by indexing

* increased error handling

* more error handling

* Output formatting fix

* kg-incremental-reindexing

* 0-doc found improvement

* celery

* migration correction

* timeout adjustments

* nit

* Updated migration

* Entity Normalization for KG Dev 1 (#4746)

* feat: trigrams column

* fix: reranking and db

* feat: v1

* fix: convert to orm

* feat: parallel

* fix: default to id_name

* fix: renamed semantic_id and semantic_id_trigrams

* fix: scalar subquery

* fix: tuning + redundancy

* fix: threshold

* fix: typo

* fix: shorten names

* wip

* fix: reverted

* feat: config

* feat: works but it was dumb

* feat: clustering works

* fix: mypy

* normalization <-> language awareness for SQL generation

* small type fixes

---------

Co-authored-by: joachim-danswer <joachim@danswer.ai>

* mypy

* typo and dead code

* kg_time_fencing

* feat: remove temp views on migration downgrade

* remove functions and triggers for now

* rebase adjustments

* EL code review results

* quick fix + trigger/funcs for single tenant

* fix: typo, mypy, dead code

* fix: autoflake

* small updatesd

* nit

* fix: typo

* early + faster view creation

* Extension creation in MT migration

* nit changes to default ETs

* Incremental Clustering and KG Refactor V1 (#4784)

Optimized/restructured incremental clustering. New pipeline actually that moves vespa updates to clustering.
Also, celery configuration has been updated.
---------

Co-authored-by: joachim-danswer <joachim@danswer.ai>

* prompt tweak & ET extraction reset

* more general hierarchical structure

* feat: better vespa reset logic

* prompt optimization and entity replacemants

* small prompt changes

* KG Refactor V2 (#4814)

Clustering & Extraction improvements & various nits 

Co-authored-by: joachim-danswer <joachim@danswer.ai>

* add connector-level coverage days

* fix: nit

* initial  EL responses

* refactor: helper functions for formatting

* fix: more helper fns & comments

* fix: comment code that's been implemented elsewhere

* fix: tenant_id missing arg

* fix: removed debugging stuff

* fix: moved kg_interactions db query to helper fn

* fix: tenant_id

* fix: tenant_id & removed outdated helper fn

* fix always set entity class

* fix: typo

* fix alembic heads

* fix: celery logging

* fix: migrations fix

* fix: multi tenant permissions

* fix: temp connector fix

* fix: downgrade

* Fix upgrade migration

* fix: tenant for normalization

* added additional acl

* stray EL comments

* fix: connector test

* fix mypy

* fix: temporary connector test fix

* fix: jira connector test

* nit

* small nits

* fix: black

* fix: mypy

* fix: mypy

---------

Co-authored-by: Rei Meguro <36625832+Orbital-Web@users.noreply.github.com>
 2025-06-07 23:14:20 +00:00
rkuo-danswerandRichard Kuo 51be9000bb Feature/vespa bump (#4819) 
* bump cloudformation

* update kubernetes

* bump helm chart

* bump docker compose

* update chart.lock

* ai accident!

* bump vespa helm chart for fix

* increase timeout

---------

Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
 2025-06-04 00:03:01 +00:00
Evan Lohn 4c71a5f5ff drive perm sync logs + misc deployment improvements (#4788) 
* some logs

* give postgress more memory

* give postgress more memory

* give postgress more memory

* revert

* give postgress more memory

* bump external access limit

* vespa timeout

* deployment consistency

* bump vespa version

* skip upgrade check

* retry permission by ids

* logs

* fix temp docx file issue

* fix drive file deduping

* RK comments

* mypy

* aggregate logs
 2025-06-01 23:36:57 +00:00
Chris Weaver e3aab8e85e Improve index attempt display (#4511) 2025-04-13 15:57:47 -07:00
Chris Weaver 7648627503 Save all logs + add log persistence to most Onyx-owned containers (#4368) 
* Save all logs + add log persistence to most Onyx-owned containers

* Separate volumes for each container

* Small fixes
 2025-03-26 22:25:39 -07:00
Chris Weaver 0d3c72acbf Add basic memory logging (#4234) 
* Add basic memory logging

* Small tweaks

* Switch to monotonic
 2025-03-08 03:49:47 +00:00
Chris Weaver f731beca1f Add ONYX_QUERY_HISTORY_TYPE to the dev compose files (#4196) 2025-03-05 17:34:55 +00:00
pablonyx e2700b2bbd Remove left over yaml errors (#3527) 
* remove left over vim command

* additional misconfigurations

* ensure all regions updated
 2024-12-29 01:45:07 +00:00
Chris Weaver 23ecf654ed Add support for custom LLM error messages (#3501) 
* Add support for custom LLM error messages

* Fix mypy
 2024-12-17 22:58:17 -08:00
pablonyx 8db6d49fe5 IAM Auth for RDS (#3479) 
* k

* functional iam auth

* k

* k

* improve typing

* add deployment options

* cleanup

* quick clean up

* minor cleanup

* additional clarity for db session operations

* nit

* k

* k

* update configs

* docker compose spacing
 2024-12-17 22:02:37 +00:00
pablodanswer 21ec5ed795 welcome to onyx 2024-12-13 09:56:10 -08:00
hagen-danswer 9209fc804b multiple slackbot support (#3077) 
* multiple slackbot support

* app_id + tenant_id key

* removed kv store stuff

* fixed up mypy and migration

* got frontend working for multiple slack bots

* some frontend stuff

* alembic fix

* might be valid

* refactor dun

* alembic stuff

* temp frontend stuff

* alembic stuff

* maybe fixed alembic

* maybe dis fix

* im getting mad

* api names changed

* tested

* almost done

* done

* routing nonsense

* done!

* done!!

* fr done

* doneski

* fix alembic migration

* getting mad again

* PLEASE IM BEGGING YOU
 2024-11-20 01:49:43 +00:00
rkuo-danswer 6913efef90 fresh indexing feature branch (#2790) 
* fresh indexing feature branch

* cherry pick test

* Revert "cherry pick test"

This reverts commit 2a62422068.

* set multitenant so that vespa fields match when indexing

* cleanup pass

* mypy

* pass through env var to control celery indexing concurrency

* comments on task kickoff and some logging improvements

* use get_session_with_tenant

* comment out all of update.py

* rename to RedisConnectorIndexingFenceData

* first check num_indexing_workers

* refactor RedisConnectorIndexingFenceData

* comment out on_worker_process_init

* fix where num_indexing_workers falls back

* remove extra brace
 2024-10-18 22:40:05 +00:00
pablodanswer d5b9a6e552 add vespa + embedding timeout env variables (#2689) 
* add vespa + embedding timeout env variables

* nit: integration test

* add dangerous override

* k

* add additional clarity

* nit

* nit
 2024-10-09 03:20:28 +00:00
rkuo-danswer 3755e575a5 harden connections to redis (#2677) 
* set broker_connection_retry_on_startup to silence deprecation warning (we're OK with retrying on startup)

* env var for CELERY_BROKER_POOL_LIMIT

* add redis retry on timeout and health check interval

* set socket_keepalive = True

* remove shadow declaration of REDIS_HEALTH_CHECK_INTERVAL, add socket_keepalive_options where possible

* fix mypy complaint

* pass through vars in docker compose

* remove extra '='

* wrap in a try
 2024-10-04 16:00:48 +00:00
Chris WeaverandRichard Kuo 1362d4b583 Allow config of background concurrency (#2648) 
* Allow config of background concurrency

* Add comment

* Fix light worker

* use backslashes to continue lines in supervisord with bash

---------

Co-authored-by: Richard Kuo (Danswer) <rkuo@danswer.ai>
 2024-10-04 00:55:28 +00:00
pablodanswer df464fc54b Allow for CORS Origin Setting (#2449) 
* allow setting of CORS origin

* simplify

* add environment variable + rename

* slightly more efficient

* simplify so mypy doens't complain

* temp

* go back to my preferred formatting
 2024-09-16 18:54:36 +00:00
rkuo-danswer 140c34e59e ephemeral behavior for redis (#2373) 
* ephemeral behavior for redis

* notes for redis command line consistency
 2024-09-13 04:48:50 +00:00
pablodanswer f69922fff7 Add environment variable for setting vespa search threads (#2400) 2024-09-11 22:37:38 -07:00
pablodanswer aee5fcd4e0 Add env variables for overriding embedding batch size (#2395) 
* add env variabels for overriding

* proper ports

* proper overrides
 2024-09-12 00:51:45 +00:00
hagen-danswer 74a2271422 Added HARD_DELETE_CHATS to environment variables (#2397) 2024-09-11 18:08:29 +00:00
rkuo-danswer c967f53c02 docker versions have been deprecated for a while, so fixing the annoying warning (#2372) 2024-09-09 18:26:12 +00:00
rkuo-danswerandRichard Kuo 2933c3598b first cut at redis (#2226) 
* first cut at redis

* fix startup dependencies on redis

* kombu cleanup - fail silently

* mypy

* add redis_host environment override

* update REDIS_HOST env var in docker-compose.dev.yml

* update the rest of the docker files

* update contributing guide

* renaming cache to cache_volume

* add redis password to various deployments

* try setting up pr testing for helm

* fix indent

* hopefully this release version actually exists

* fix command line option to --chart-dirs

* fetch-depth 0

* edit values.yaml

* try setting ct working directory

* bypass testing only on change for now

* move files and lint them

* update helm testing

* some issues suggest using --config works

* add vespa repo

* add postgresql repo

* increase timeout

* try amd64 runner

* fix redis password reference

* add comment to helm chart testing workflow

* rename helm testing workflow to disable it

---------

Co-authored-by: Richard Kuo <rkuo@rkuo.com>
 2024-09-06 19:21:29 +00:00
Chris Weaver 033ec0b6b1 Remove unused env variables (#2299) 2024-09-02 20:29:14 +00:00
Chris Weaver c0e1a02e8e Add it on merge queue (#2112) 
* Github action to run integration tests

* Improve

* Fix build

* Add pull

* Fix readiness script

* Add IT runner

* Add IT runner

* Add logs

* update

* Fix

* Fix path

* file path

* test

* fix

* fix

* fix

* test

* network

* fix

* cleanup

* fix

* test

* Fix downgrade

* Add OpenAI API key

* Add VESPA_HOST

* test pulling first

* Add API server host

* Cache tweak

* Fix pull/push settings:

* Stop pushing to latest tag

* test cache change

* test

* test

* test

* remove cache temporarily

* Fix

* Enable EE

* test

* Remove duplicate funcs

* add back build

* Update all

* Fix stop cmd

* Add to merge queue

* Cleanup image tag
 2024-08-26 07:20:28 +00:00
Chris Weaver 730a757090 Disable oidc_expiry by default (#2182) 2024-08-20 13:24:58 -07:00
Yuhong Sun 386b229ed3 Cohere Rerank (#2109) 2024-08-11 14:22:42 -07:00
pablodanswer 6d67d472cd Add answers to search (#2020) 2024-08-04 23:02:55 -07:00
rkuo-danswer 288e6fa606 Bugfix/pg connections (#2002) 
* increase max_connections to 150 in all docker files

* lower celery worker concurrency to 6
 2024-07-31 19:49:20 +00:00
pablodanswer 807bef8ada Add environment variable for defaulted sidebar toggling (#1923) 
* add env variable for defaulted sidebar toggling

* formatting

* update naming
 2024-07-24 21:23:37 +00:00
Weves 2344edf158 Change default login time to 7 days 2024-07-19 13:58:50 -07:00
hagen-danswer 25b3dacaba Seperated model caching volumes (#1845) 2024-07-15 15:32:04 -07:00
hagen-danswer 47a550221f slackbot doesnt respond without citations/quotes (#1798) 
* slackbot doesnt respond without citations/quotes

fixed logical issues

fixed dict logic

* added slackbot shim for the llm source/time feature

* mypy fixes

* slackbot doesnt respond without citations/quotes

fixed logical issues

fixed dict logic

* Update handle_regular_answer.py

* added bypass_filter check

* final fixes
 2024-07-11 00:18:26 +00:00
Yuhong Sun 7c8f8dba17 Break the Danswer LLM logging from LiteLLM Verbose (#1795) 2024-07-09 18:18:29 -07:00
Yuhong Sun 8e8f262ed3 Docker Compose Eval Pipeline Cleanup (#1780) 2024-07-06 12:04:57 -07:00
Yuhong Sun 8be42a5f98 Touchup for Multilingual Users (#1725) 2024-06-26 22:44:06 -07:00
Weves 5d552705aa Change EE environment variable name 2024-06-25 16:59:09 -07:00
Weves 1ee8ee9e8b Prepare EE to merge with MIT 2024-06-25 15:07:56 -07:00
hagen-danswer 54c2547d89 Add connector document pruning task (#1652) 2024-06-18 14:26:12 -07:00
Bijay Regmi 260149b35a Feature/support all gpus (#1515) 
* add `count` explicitely to ensure backwards compat

* fix main document URL
 2024-06-08 17:45:26 -07:00
Weves 81aada7c0f Add option to hide logout 2024-06-03 12:13:40 -07:00
Yuhong Sun 546815dc8c Consolidate File Processing (#1449) 2024-05-11 23:11:22 -07:00
Weves 7f1ffa3921 Add predefined feedback option 2024-05-09 19:11:17 -07:00
Bijay Regmi 436806f2e3 add gpu support and README for documentation (#1398) 2024-05-09 10:51:37 -07:00