* squash: combine all DR commits into one
Co-authored-by: Joachim Rahmfeld <joachim@onyx.app>
Co-authored-by: Rei Meguro <rmeguro@umich.edu>
* Fixes
* show KG in Assistant only if available
* KG only usable for KG Beta (for now)
* base file upload
* raise error if uploaded context is too long
* improvements
* More improvements
* Fix citations
* better decision making
* improved decision-making in Orchestrator
* generic_internal tools
* Small tweak
* tool use improvements
* add on
* More image gen stuff
* fixes
* Small color improvements
* Markdown utils
* fixed end conditions (incl early exit for image generation)
* remove agent search + image fixes
* Okta tool support for reload
* Some cleanup
* Stream back search tool results as they come
* tool forcing
* fixed no-Tool-Assistant
* Support anthropic tool calling
* Support anthropic models better
* More stuff
* prompt fixes and search step numbers
* Fix hook ordering issue
* internal search fix
* Improve citation look
* Small UI improvements
* Improvements
* Improve dot
* Small chat fixes
* Small UI tweaks
* Small improvements
* Remove un-used code
* Fix
* Remove test_answer.py for now
* Fix
* improvements
* Add foreign keys
* early forcing
* Fix tests
* Fix tests
---------
Co-authored-by: Joachim Rahmfeld <joachim@onyx.app>
Co-authored-by: Rei Meguro <rmeguro@umich.edu>
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* db setup
* transfer 1 - incomplete
* more adjustments
* relationship table + query update
* temp view creation
* restructuring
* nits
* updates
* separate read_only engine
* extraction revamp
* focus on metadata relatonships 1
* dev
* migration downgrade fix
* rebase migration change
* a3+
* progress
* base
* new extraction
* progress
* fixed KG extraction
* nits
* updates
* simplifications & cleanup
* fixes
* updates
* more feature flag checks
* fixes
* extraction process fix
* read-only user creation as part of setup
* fix for missing entity attributes
* kg read-only user creation as part of migration
* typo
* EL initial comments
* initial Account/SF Connector chnges
* SF Connector update
- include account information
* base w/ salesforce
* evan updates + quite a bit more
* kg-filtered search
* EL changes pt 2
* migrations and env vars
* quick migration fix
* migration update
* post_rebase fixes
* mypy fixes
* test fixes
* test fix
* test fix
* read_only pool + misc
* nf
* env vars
* test improvements
* salesforce fix
* test update
* small changes
* small adjustments
* SF Connector fix & kg_stage removal for one table
* mypy fix
* small fixes
* EL + RK (pt 1) comments
* nit
* setting updated
* Salesforce test update
* EL comments
* read-only user replacement & cleanup
* SQL View fix
* converting entity type-name separators
* sql view group ownership
* view fix
* SQL tweak
* dealing with docs that were skipped by indexing
* increased error handling
* more error handling
* Output formatting fix
* kg-incremental-reindexing
* 0-doc found improvement
* celery
* migration correction
* timeout adjustments
* nit
* Updated migration
* Entity Normalization for KG Dev 1 (#4746)
* feat: trigrams column
* fix: reranking and db
* feat: v1
* fix: convert to orm
* feat: parallel
* fix: default to id_name
* fix: renamed semantic_id and semantic_id_trigrams
* fix: scalar subquery
* fix: tuning + redundancy
* fix: threshold
* fix: typo
* fix: shorten names
* wip
* fix: reverted
* feat: config
* feat: works but it was dumb
* feat: clustering works
* fix: mypy
* normalization <-> language awareness for SQL generation
* small type fixes
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* mypy
* typo and dead code
* kg_time_fencing
* feat: remove temp views on migration downgrade
* remove functions and triggers for now
* rebase adjustments
* EL code review results
* quick fix + trigger/funcs for single tenant
* fix: typo, mypy, dead code
* fix: autoflake
* small updatesd
* nit
* fix: typo
* early + faster view creation
* Extension creation in MT migration
* nit changes to default ETs
* Incremental Clustering and KG Refactor V1 (#4784)
Optimized/restructured incremental clustering. New pipeline actually that moves vespa updates to clustering.
Also, celery configuration has been updated.
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* Move file
* Fix all prior imports
* Clean sidebar items logic; add kg page
* Add kg_processing celery background task
* prompt tweak & ET extraction reset
* more general hierarchical structure
* feat: better vespa reset logic
* Add basic knowledge graph configuration
* Add configurations for KG entity-type
* prompt optimization and entity replacemants
* small prompt changes
* Implement backend APIs
* KG Refactor V2 (#4814)
Clustering & Extraction improvements & various nits
Co-authored-by: joachim-danswer <joachim@danswer.ai>
* add connector-level coverage days
* Update APIs to be more frontend ergonomic
* Add simple test
* Make config optional in test
* fix: nit
* initial EL responses
* refactor: helper functions for formatting
* fix: more helper fns & comments
* fix: comment code that's been implemented elsewhere
* Add entity-types APIs
* Hook up frontend to backend
* Finish hookup up entity-types to backend
* Update ordering of entity-types and fix form submitting
* Add backend API to get kg-exposed
* Add kg-exposed to sidebar
* Fix path
* Use existing values, even if kg-enabled is false
* Update what initial values are used
* Add skeleton for kg resetting
* Add return type
* Add default entity-type population when fetching entity-types
* Remove circular deps
* Minor fixes to logic
* Edit logic for default entity-types population
* Add re-index API + skeleton
* Update verbiage for KG
* Remove templatization in favour of function
* Address comments on PR
* Pull call out into its own binding
* Remove re-index API and revert implement of reset back to stub
* Fix circular import error
* Remove 'reindex' button
* Edit how the empty vendor name list is handled
* Edit how exposed is processed
* Redirect if navigated to `/admin/kg` and kg is not exposed
* Address comments on PR
* reset + entity type table display & updating updates
* Update fetching entity-types
* Make KG entity types refresh when reset
* Edit verbiage of reset button
* Update package-lock.json file
* Protect against overflowing
* Re-implement refreshing table after reset
* Edit message when nothing is shown.
* UI enhancements
* small fixes
* remove form validation?
* fix
* nit
* nit
* nit
* nit
* fix configure max coverage days
* EL comments for JR
* refactor: moved functions where they belong to fix circular import
* feat: intuitive coverage days
* feat: intuitive coverage days
* fix: safe date picker
* fix: startdate
* evan fixes
* fix: evan comment on enable/disable
* fix: style
* fix: ui issues
* fix: ui issues for reset too
* fix: tests
* fix: kg entity is not enabled
* fix: entity type reload on enable
---------
Co-authored-by: joachim-danswer <joachim@danswer.ai>
Co-authored-by: Rei Meguro <36625832+Orbital-Web@users.noreply.github.com>
* bump fastapi and starlette
* bumping llama index and nltk and associated deps
* bump to fix python-multipart
* bump aiohttp
* update package lock for examples/widget
* bump black
* sentencesplitter has changed namespaces
* fix reorder import check, fix missing passlib
* update package-lock.json
* black formatter updated
* reformatted again
* change to black compatible reorder
* change to black compatible reorder-python-imports fork
* fix pytest dependency
* black format again
* we don't need cdk.txt. update packages to be consistent across all packages
---------
Co-authored-by: Richard Kuo (Onyx) <rkuo@onyx.app>
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
* try adding back some params
* raise timeout
* update chromatic version
* fix typo
* use chromatic imports
* update gitignore
* slim down the config file
* update readme
---------
Co-authored-by: Richard Kuo (Danswer) <rkuo@onyx.app>
* Made copy button and cmd+c work for cmd+v and cmd+shift+v
* made sub selections work as well
* ok it works
* fixed npm run build
* im not from earth
* added logging
* more logging
* bye logs
* should work now
* whoops
* added stuff
* made it robust
* ctrl shift v behavior
* initial PoC
* preliminary working config
* first cut at chromatic tests
* first cut at chromatic tests
* fix yaml
* fix yaml again
* use workingDir
* adapt playwright example
* remove env
* fix working directory
* fix more paths
* fix dir
* add playwright setup
* accidentally deleted a step
* update test
* think we don't need home.png right now
* remove unused home.png
---------
Co-authored-by: Richard Kuo <rkuo@rkuo.com>
Raunak Bhagat