mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-19 01:50:51 -05:00
docs(agents): refresh AGENTS.md, split backend standards into backend/AGENTS.md (#13301)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jamison LahmanandClaude Fable 5 authored and GitHub committed
1 parent
e31a3336f6
commit
84cc779e44
9 files changed
+296
-334
No files matched your search
@@ -6,9 +6,9 @@ Running **inside the Onyx dev container**. These notes are additive to the root
|
||||
## No Docker daemon in here
|
||||
|
||||
Don't use `docker` / `docker exec` / `docker compose`. Onyx services run as sibling
|
||||
containers on the `onyx_default` network, reachable directly by hostname — so the root
|
||||
guide's `docker exec -it onyx-relational_db-1 psql ...` won't work. Use
|
||||
`psql -h relational_db -U postgres -c "<SQL>"` instead.
|
||||
containers on the `onyx_default` network, reachable directly by hostname — the root
|
||||
guide's `psql` command works as-is here (the env vars below plus `POSTGRES_PASSWORD`
|
||||
are exported); its `docker exec` fallback won't.
|
||||
|
||||
## Service hostnames (`onyx_default` network)
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
"ONYX_DEVCONTAINER_FIREWALL": "${localEnv:ONYX_DEVCONTAINER_FIREWALL}",
|
||||
"OPENSEARCH_HOST": "opensearch",
|
||||
"POSTGRES_HOST": "relational_db",
|
||||
"POSTGRES_PASSWORD": "${localEnv:POSTGRES_PASSWORD:password}",
|
||||
"REDIS_HOST": "cache",
|
||||
"S3_ENDPOINT_URL": "http://minio:9000",
|
||||
"SSH_AUTH_SOCK": "/tmp/ssh-agent.sock",
|
||||
|
||||
@@ -4,131 +4,61 @@ This file provides guidance to AI agents when working with code in this reposito
|
||||
|
||||
## KEY NOTES
|
||||
|
||||
- Python deps live in a `uv`-managed virtualenv at `.venv` (repo root). If it doesn't exist yet, create it \
|
||||
- Python deps live in a `uv`-managed virtualenv at `.venv` (repo root). If it doesn't exist yet, create it
|
||||
with `uv sync --frozen`, then `source .venv/bin/activate`.
|
||||
- To make tests work, check the `.env` file at the root of the project to find an OpenAI key.
|
||||
- Test secrets (API keys etc.) are resolved by `backend/tests/utils/aws_secrets.py`, in order: process
|
||||
env vars → the gitignored `.vscode/.env` (also used by the pytest commands in `backend/AGENTS.md`;
|
||||
create it by copying `.vscode/env_template.txt`) → AWS Secrets Manager (requires `aws sso login`).
|
||||
Tests declare what they need via `@pytest.mark.secrets(TestSecret.X)`. If a key you need still can't
|
||||
be resolved, ask the user rather than skipping tests.
|
||||
- If using `playwright` to explore the frontend, log in with username `admin_user@example.com` and password
|
||||
`TestPassword123!` (the admin user created by the playwright global setup — see
|
||||
`web/tests/e2e/constants.ts`). If it doesn't exist yet, register it via the signup page; the first user
|
||||
registered automatically becomes admin. The app can be accessed at `http://localhost:3000`.
|
||||
- You should assume that all Onyx services are running. To verify, you can check the `backend/log` directory to
|
||||
make sure we see logs coming out from the relevant service.
|
||||
- To connect to the Postgres database, use: `docker exec -it onyx-relational_db-1 psql -U postgres -c "<SQL>"`
|
||||
- To connect to the Postgres database, use:
|
||||
`PGPASSWORD="${POSTGRES_PASSWORD:-password}" psql -h "${POSTGRES_HOST:-localhost}" -U postgres -c "<SQL>"`.
|
||||
This works on a host checkout and inside the devcontainer. If no `psql` client is available, fall back to
|
||||
`docker exec onyx-relational_db-1 psql -U postgres -c "<SQL>"` (no `-it` — agent shells have no TTY).
|
||||
- When making calls to the backend, always go through the frontend. E.g. make a call to `http://localhost:3000/api/persona` not `http://localhost:8080/api/persona`
|
||||
- Put ALL db operations under the `backend/onyx/db` / `backend/ee/onyx/db` directories. Don't run queries
|
||||
outside of those directories.
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Onyx** (formerly Danswer) is an open-source Gen-AI and Enterprise Search platform that connects to company documents, apps, and people. It features a modular architecture with both Community Edition (MIT licensed) and Enterprise Edition offerings.
|
||||
|
||||
### Background Workers (Celery)
|
||||
### Technology Stack
|
||||
|
||||
Onyx uses Celery for asynchronous task processing with multiple specialized workers:
|
||||
- **Backend**: Python 3.13, FastAPI, SQLAlchemy, Alembic, Celery
|
||||
- **Frontend**: Next.js 16, React 19, TypeScript, Tailwind CSS
|
||||
- **Database**: PostgreSQL with Redis caching
|
||||
- **Search**: OpenSearch-backed keyword and vector document index
|
||||
- **Auth**: OAuth2, SAML, multi-provider support
|
||||
- **AI/ML**: LangChain, LiteLLM, multiple embedding models
|
||||
|
||||
#### Worker Types
|
||||
### Repository Layout & Sub-project Guides
|
||||
|
||||
1. **Primary Worker** (`celery_app.py`)
|
||||
Each sub-project has its own agents file with the standards for that area — read it before working
|
||||
there:
|
||||
|
||||
- Coordinates core background tasks and system-wide operations
|
||||
- Handles connector management, document sync, pruning, and periodic checks
|
||||
- Runs with 4 threads concurrency
|
||||
- Tasks: connector deletion, document-index sync, pruning, LLM model updates, user file sync
|
||||
- `backend/` — FastAPI app + Celery workers. `onyx/` is the Community Edition core, `ee/` mirrors its
|
||||
layout for Enterprise features, `alembic/` holds migrations, `tests/` the test suites. Standards
|
||||
(Celery, migrations, testing, error handling, LLM tracing): `backend/AGENTS.md`.
|
||||
- `web/` — Next.js frontend. Standards (also cover `desktop/`, the Tauri shell): `web/AGENTS.md`.
|
||||
- `mobile/` — React Native + Expo app. Standards: `mobile/AGENTS.md`. Mobile differs from web on
|
||||
several points (no DOM, NativeWind, expo-router), so do **not** assume the web rules apply there.
|
||||
|
||||
2. **Docfetching Worker** (`docfetching`)
|
||||
Explore the tree with `ls` rather than relying on docs for the full package list.
|
||||
|
||||
- Fetches documents from external data sources (connectors)
|
||||
- Spawns docprocessing tasks for each document batch
|
||||
- Implements watchdog monitoring for stuck connectors
|
||||
- Configurable concurrency (default from env)
|
||||
|
||||
3. **Docprocessing Worker** (`docprocessing`)
|
||||
|
||||
- Processes fetched documents through the indexing pipeline:
|
||||
- Upserts documents to PostgreSQL
|
||||
- Chunks documents and adds contextual information
|
||||
- Embeds chunks via model server
|
||||
- Writes chunks to the OpenSearch-backed document index
|
||||
- Updates document metadata
|
||||
- Configurable concurrency (default from env)
|
||||
|
||||
4. **Light Worker** (`light`)
|
||||
|
||||
- Handles lightweight, fast operations
|
||||
- Tasks: document-index metadata sync, connector deletion, doc permissions upsert, checkpoint cleanup, index attempt cleanup
|
||||
- Higher concurrency for quick tasks
|
||||
|
||||
5. **Heavy Worker** (`heavy`)
|
||||
|
||||
- Handles resource-intensive operations
|
||||
- Tasks: connector pruning, document permissions sync, external group sync, CSV generation
|
||||
- Runs with 4 threads concurrency
|
||||
|
||||
6. **Monitoring Worker** (`monitoring`)
|
||||
|
||||
- System health monitoring and metrics collection
|
||||
- Monitors Celery queues, process memory, and system status
|
||||
- Single thread (monitoring doesn't need parallelism)
|
||||
- Cloud-specific monitoring tasks
|
||||
|
||||
7. **User File Processing Worker** (`user_file_processing`)
|
||||
|
||||
- Processes user-uploaded files
|
||||
- Handles user file indexing and project synchronization
|
||||
- Configurable concurrency
|
||||
|
||||
8. **Beat Worker** (`beat`)
|
||||
- Celery's scheduler for periodic tasks
|
||||
- Uses DynamicTenantScheduler for multi-tenant support
|
||||
- Schedules tasks like:
|
||||
- Indexing checks (every 15 seconds)
|
||||
- Connector deletion checks (every 20 seconds)
|
||||
- Document-index metadata sync checks (every 20 seconds)
|
||||
- Pruning checks (every 20 seconds)
|
||||
- Monitoring tasks (every 5 minutes)
|
||||
- Cleanup tasks (hourly)
|
||||
|
||||
#### Key Features
|
||||
|
||||
- **Thread-based Workers**: All workers use thread pools (not processes) for stability
|
||||
- **Tenant Awareness**: Multi-tenant support with per-tenant task isolation. There is a
|
||||
middleware layer that automatically finds the appropriate tenant ID when sending tasks
|
||||
via Celery Beat.
|
||||
- **Task Prioritization**: High, Medium, Low priority queues
|
||||
- **Monitoring**: Built-in heartbeat and liveness checking
|
||||
- **Failure Handling**: Automatic retry and failure recovery mechanisms
|
||||
- **Redis Coordination**: Inter-process communication via Redis
|
||||
- **PostgreSQL State**: Task state and metadata stored in PostgreSQL
|
||||
|
||||
#### Important Notes
|
||||
|
||||
**Defining Tasks**:
|
||||
|
||||
- Always use `@shared_task` rather than `@celery_app`
|
||||
- Put tasks under `background/celery/tasks/` or `ee/background/celery/tasks`
|
||||
- Never enqueue a task without an expiration. Always supply `expires=` when
|
||||
sending tasks, either from the beat schedule or directly from another task. It
|
||||
should never be acceptable to submit code which enqueues tasks without an
|
||||
expiration, as doing so can lead to unbounded task queue growth.
|
||||
|
||||
**Defining APIs**:
|
||||
When creating new FastAPI APIs, do NOT use the `response_model` field. Instead, just type the
|
||||
function.
|
||||
|
||||
**Testing Updates**:
|
||||
If you make any updates to a celery worker and you want to test these changes, you will need
|
||||
to ask me to restart the celery worker. There is no auto-restart on code-change mechanism.
|
||||
|
||||
**Task Time Limits**:
|
||||
Since all tasks are executed in thread pools, the time limit features of Celery are silently
|
||||
disabled and won't work. Timeout logic must be implemented within the task itself.
|
||||
|
||||
### Code Quality
|
||||
## Code Quality
|
||||
|
||||
```bash
|
||||
# Install and run pre-commit hooks
|
||||
pre-commit install
|
||||
pre-commit run --all-files
|
||||
|
||||
# Faster: run only on the files you touched
|
||||
pre-commit run --files <path> [<path> ...]
|
||||
```
|
||||
|
||||
NOTE: Always make sure everything is strictly typed (both in Python and Typescript).
|
||||
@@ -136,161 +66,11 @@ NOTE: Always make sure everything is strictly typed (both in Python and Typescri
|
||||
NOTE: Keep comments brief and focused on information that stays relevant long-term. Don't write
|
||||
comments that only describe the instantaneous change (e.g. what was just added/removed/refactored).
|
||||
|
||||
## Architecture Overview
|
||||
## Testing
|
||||
|
||||
### Technology Stack
|
||||
|
||||
- **Backend**: Python 3.13, FastAPI, SQLAlchemy, Alembic, Celery
|
||||
- **Frontend**: Next.js 15+, React 18, TypeScript, Tailwind CSS
|
||||
- **Database**: PostgreSQL with Redis caching
|
||||
- **Search**: OpenSearch-backed keyword and vector document index
|
||||
- **Auth**: OAuth2, SAML, multi-provider support
|
||||
- **AI/ML**: LangChain, LiteLLM, multiple embedding models
|
||||
|
||||
OpenSearch is the current document index backend for search and indexing. Some
|
||||
legacy modules, Celery task names, and migration helpers still mention Vespa; treat
|
||||
those as compatibility or migration artifacts unless the active `DocumentIndex`
|
||||
factory/config path explicitly uses them.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
backend/
|
||||
├── onyx/
|
||||
│ ├── auth/ # Authentication & authorization
|
||||
│ ├── chat/ # Chat functionality & LLM interactions
|
||||
│ ├── connectors/ # Data source connectors
|
||||
│ ├── db/ # Database models & operations
|
||||
│ ├── document_index/ # OpenSearch-backed DocumentIndex integration
|
||||
│ ├── federated_connectors/ # External search connectors
|
||||
│ ├── llm/ # LLM provider integrations
|
||||
│ └── server/ # API endpoints & routers
|
||||
├── ee/ # Enterprise Edition features
|
||||
├── alembic/ # Database migrations
|
||||
└── tests/ # Test suites
|
||||
|
||||
web/
|
||||
├── src/app/ # Next.js app router pages
|
||||
├── src/components/ # Reusable React components
|
||||
└── src/lib/ # Utilities & business logic
|
||||
```
|
||||
|
||||
## Frontend Standards
|
||||
|
||||
Frontend standards for the `web/` and `desktop/` projects live in `web/AGENTS.md`.
|
||||
|
||||
Standards for the **mobile** app (React Native + Expo) live in `mobile/AGENTS.md`. Mobile differs
|
||||
from web on several points (no DOM, NativeWind instead of web Tailwind — e.g. spacing classes are
|
||||
pixel-valued, not web's rem step scale — expo-router, RN primitives), so do **not** assume the web
|
||||
rules apply when working in `mobile/`.
|
||||
|
||||
## Database & Migrations
|
||||
|
||||
### Running Migrations
|
||||
|
||||
```bash
|
||||
# Standard migrations
|
||||
alembic upgrade head
|
||||
|
||||
# Multi-tenant (Enterprise)
|
||||
alembic -n schema_private upgrade head
|
||||
```
|
||||
|
||||
### Creating Migrations
|
||||
|
||||
```bash
|
||||
# Create migration
|
||||
alembic revision -m "description"
|
||||
|
||||
# Multi-tenant migration
|
||||
alembic -n schema_private revision -m "description"
|
||||
```
|
||||
|
||||
Write the migration manually and place it in the file that alembic creates when running the above command.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
First, activate the virtualenv: `source .venv/bin/activate`. If `.venv` doesn't exist yet, create it first with `uv sync --frozen`.
|
||||
|
||||
There are 4 main types of tests within Onyx:
|
||||
|
||||
### Model choice for tests that make real LLM calls
|
||||
|
||||
When a test makes a real LLM call (e.g. External Dependency Unit / integration tests
|
||||
that hit a live provider), use the cheap-and-fast tier for each provider:
|
||||
|
||||
- **OpenAI**: `gpt-5-mini` (never `gpt-4o` / `gpt-4o-mini`)
|
||||
- **Anthropic**: `claude-haiku-4-5`
|
||||
|
||||
### Unit Tests
|
||||
|
||||
These should not assume any Onyx/external services are available to be called.
|
||||
Interactions with the outside world should be mocked using `unittest.mock`. Generally, only
|
||||
write these for complex, isolated modules e.g. `citation_processing.py`.
|
||||
|
||||
To run them:
|
||||
|
||||
```bash
|
||||
pytest -xv backend/tests/unit
|
||||
```
|
||||
|
||||
### External Dependency Unit Tests
|
||||
|
||||
These tests assume that all external dependencies of Onyx are available and callable (e.g. Postgres, Redis,
|
||||
MinIO/S3, OpenSearch are running + OpenAI can be called + any request to the internet is fine + etc.).
|
||||
|
||||
However, the actual Onyx containers are not running and with these tests we call the function to test directly.
|
||||
We can also mock components/calls at will.
|
||||
|
||||
The goal with these tests are to minimize mocking while giving some flexibility to mock things that are flakey,
|
||||
need strictly controlled behavior, or need to have their internal behavior validated (e.g. verify a function is called
|
||||
with certain args, something that would be impossible with proper integration tests).
|
||||
|
||||
A great example of this type of test is `backend/tests/external_dependency_unit/connectors/confluence/test_confluence_group_sync.py`.
|
||||
|
||||
To run them:
|
||||
|
||||
```bash
|
||||
python -m dotenv -f .vscode/.env run -- pytest backend/tests/external_dependency_unit
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
Standard integration tests. Every test in `backend/tests/integration` runs against a real Onyx deployment. We cannot
|
||||
mock anything in these tests. Prefer writing integration tests (or External Dependency Unit Tests if mocking/internal
|
||||
verification is necessary) over any other type of test.
|
||||
|
||||
Tests are parallelized at a directory level.
|
||||
|
||||
When writing integration tests, make sure to check the root `conftest.py` for useful fixtures + the `backend/tests/integration/common_utils` directory for utilities. Prefer (if one exists), calling the appropriate Manager
|
||||
class in the utils over directly calling the APIs with a library like `requests`. Prefer using fixtures rather than
|
||||
calling the utilities directly (e.g. do NOT create admin users with
|
||||
`admin_user = UserManager.create(name="admin_user")`, instead use the `admin_user` fixture).
|
||||
|
||||
A great example of this type of test is `backend/tests/integration/tests/streaming_endpoints/test_chat_stream.py`.
|
||||
|
||||
To run them:
|
||||
|
||||
```bash
|
||||
python -m dotenv -f .vscode/.env run -- pytest backend/tests/integration
|
||||
```
|
||||
|
||||
### Playwright (E2E) Tests
|
||||
|
||||
These tests are an even more complete version of the Integration Tests mentioned above. Has all services of Onyx
|
||||
running, _including_ the Web Server.
|
||||
|
||||
Use these tests for anything that requires significant frontend <-> backend coordination.
|
||||
|
||||
Tests are located at `web/tests/e2e`. Tests are written in TypeScript.
|
||||
|
||||
To run them:
|
||||
|
||||
```bash
|
||||
bunx playwright test <TEST_NAME>
|
||||
```
|
||||
|
||||
For shared fixtures, best practices, and detailed guidance, see `backend/tests/README.md`.
|
||||
There are 4 main types of tests: unit, external dependency unit, integration, and playwright e2e
|
||||
(`web/tests/e2e`). Commands and guidance for all four live in `backend/AGENTS.md`; shared fixtures
|
||||
and deeper detail in `backend/tests/README.md`. Prefer integration tests over the other types.
|
||||
|
||||
## Logs
|
||||
|
||||
@@ -300,36 +80,14 @@ will be tailing their logs to this file.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Never commit API keys or secrets to repository
|
||||
- Use encrypted credential storage for connector credentials
|
||||
- Follow RBAC patterns for new features
|
||||
- Implement proper input validation with Pydantic models
|
||||
- Use parameterized queries to prevent SQL injection
|
||||
|
||||
## AI/LLM Integration
|
||||
|
||||
- Multiple LLM providers supported via LiteLLM
|
||||
- Configurable models per feature (chat, search, embeddings)
|
||||
- Streaming support for real-time responses
|
||||
- Token management and rate limiting
|
||||
- Custom prompts and agent actions
|
||||
|
||||
### Tracing — every LLM invocation must be tagged
|
||||
|
||||
Every LLM, embedding, rerank, image-generation, voice (STT/TTS), and intent-classification call must open a generation span tagged with a value from the `LLMFlow` registry in `backend/onyx/tracing/flows.py`. Use one of:
|
||||
|
||||
- `llm_generation_span(llm=..., flow=LLMFlow.X, input_messages=...)` for calls going through an `LLM` subclass.
|
||||
- `traced_llm_call(flow=LLMFlow.X, model=..., provider=..., input_messages=...)` for direct provider SDK / `litellm` / model_server HTTP calls that bypass the `LLM` abstraction.
|
||||
|
||||
Rules:
|
||||
|
||||
1. Add a new `LLMFlow` enum value before instrumenting a new operation. Don't pass raw strings.
|
||||
2. Flow tags name the **operation** (e.g. `IMAGE_EDIT`, `RERANK`) — not the provider. Provider lives in `model_config["model_provider"]`.
|
||||
3. The auto-wrap fallback in `onyx/llm/tracing_wrap.py` emits `LLMFlow.UNTAGGED_INVOKE` / `UNTAGGED_STREAM` for calls that reach `LLM.invoke` / `LLM.stream` without an explicit span. These sentinels are visible in dashboards and indicate missing instrumentation — fix the call site, don't rely on the fallback.
|
||||
- Never commit API keys or secrets to the repository
|
||||
- Use the encrypted credential storage for connector credentials
|
||||
- Follow existing RBAC patterns for new features
|
||||
|
||||
## Creating a Plan
|
||||
|
||||
When creating a plan in the `plans` directory, make sure to include at least these elements:
|
||||
When creating a plan in the `plans` directory (gitignored — create it if it doesn't exist), make sure to
|
||||
include at least these elements:
|
||||
|
||||
**Issues to Address**
|
||||
What the change is meant to do.
|
||||
@@ -351,45 +109,6 @@ Keep it high level. You can reference certain files or functions though.
|
||||
|
||||
Before writing your plan, make sure to do research. Explore the relevant sections in the codebase.
|
||||
|
||||
## Error Handling
|
||||
|
||||
**Always raise `OnyxError` from `onyx.error_handling.exceptions` instead of `HTTPException`.
|
||||
Never hardcode status codes or use `starlette.status` / `fastapi.status` constants directly.**
|
||||
|
||||
A global FastAPI exception handler converts `OnyxError` into a JSON response with the standard
|
||||
`{"error_code": "...", "detail": "..."}` shape. This eliminates boilerplate and keeps error
|
||||
handling consistent across the entire backend.
|
||||
|
||||
```python
|
||||
from onyx.error_handling.error_codes import OnyxErrorCode
|
||||
from onyx.error_handling.exceptions import OnyxError
|
||||
|
||||
# ✅ Good
|
||||
raise OnyxError(OnyxErrorCode.NOT_FOUND, "Session not found")
|
||||
|
||||
# ✅ Good — no extra message needed
|
||||
raise OnyxError(OnyxErrorCode.UNAUTHENTICATED)
|
||||
|
||||
# ✅ Good — upstream service with dynamic status code
|
||||
raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=upstream_status)
|
||||
|
||||
# ❌ Bad — using HTTPException directly
|
||||
raise HTTPException(status_code=404, detail="Session not found")
|
||||
|
||||
# ❌ Bad — starlette constant
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied")
|
||||
```
|
||||
|
||||
Available error codes are defined in `backend/onyx/error_handling/error_codes.py`. If a new error
|
||||
category is needed, add it there first — do not invent ad-hoc codes.
|
||||
|
||||
**Upstream service errors:** When forwarding errors from an upstream service where the HTTP
|
||||
status code is dynamic (comes from the upstream response), use `status_code_override`:
|
||||
|
||||
```python
|
||||
raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=e.response.status_code)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
In addition to the other content in this file, best practices for contributing
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
# BACKEND STANDARDS
|
||||
|
||||
Guidance for the Python backend (`backend/`): the FastAPI app, Celery workers, database, and
|
||||
tests. Additive to the root `AGENTS.md`.
|
||||
|
||||
## Key Rules
|
||||
|
||||
- Put ALL db operations under the `backend/onyx/db` / `backend/ee/onyx/db` directories. Don't run
|
||||
queries outside of those directories.
|
||||
- When creating new FastAPI APIs, do NOT use the `response_model` field. Instead, just type the
|
||||
function.
|
||||
- OpenSearch is the current document index backend for search and indexing. Some legacy modules,
|
||||
Celery task names, and migration helpers still mention Vespa; treat those as compatibility or
|
||||
migration artifacts unless the active `DocumentIndex` factory/config path explicitly uses them.
|
||||
|
||||
## Background Workers (Celery)
|
||||
|
||||
Onyx uses Celery for asynchronous task processing. Worker apps live in
|
||||
`backend/onyx/background/celery/apps/`; the periodic schedule is defined in
|
||||
`backend/onyx/background/celery/tasks/beat_schedule.py`.
|
||||
|
||||
| Worker | Role |
|
||||
| --- | --- |
|
||||
| `primary` | Coordinates core background tasks: connector management/deletion, document-index sync, pruning checks, LLM model updates, user file sync |
|
||||
| `docfetching` | Fetches documents from connectors, spawns docprocessing tasks; watchdog for stuck connectors |
|
||||
| `docprocessing` | Indexing pipeline: upsert docs to Postgres, chunk, embed via model server, write chunks to the document index, update metadata |
|
||||
| `light` | Fast lightweight ops: metadata sync, permissions upsert, checkpoint / index-attempt cleanup |
|
||||
| `heavy` | Resource-intensive ops: pruning, document permissions sync, external group sync, CSV generation |
|
||||
| `monitoring` | System health monitoring & metrics collection |
|
||||
| `user_file_processing` | User-uploaded file indexing & project synchronization |
|
||||
| `scheduled_tasks` | Executes user-scheduled (Craft) task runs |
|
||||
| `beat` | Scheduler for periodic tasks; uses `DynamicTenantScheduler` for multi-tenant support |
|
||||
|
||||
Key facts:
|
||||
|
||||
- All workers use thread pools (not processes) — this is why Celery time limits don't work (see below).
|
||||
- Multi-tenant: `DynamicTenantScheduler` explicitly adds `tenant_id` to each Celery Beat task's
|
||||
kwargs; direct task sends must propagate it themselves (`TenantAwareTask` silently falls back to
|
||||
the default schema when it's absent).
|
||||
- Tasks route to named queues and carry a separate High/Medium/Low priority (`OnyxCeleryQueues` /
|
||||
`OnyxCeleryPriority` in `backend/onyx/configs/constants.py`); Redis coordinates inter-process
|
||||
communication; task state and metadata live in PostgreSQL.
|
||||
|
||||
### Defining Tasks
|
||||
|
||||
- Always use `@shared_task` rather than `@celery_app`
|
||||
- Put tasks under `background/celery/tasks/` or `ee/background/celery/tasks`
|
||||
- Never enqueue a task without an expiration. Always supply `expires=` when
|
||||
sending tasks, either from the beat schedule or directly from another task. It
|
||||
should never be acceptable to submit code which enqueues tasks without an
|
||||
expiration, as doing so can lead to unbounded task queue growth.
|
||||
|
||||
### Task Time Limits
|
||||
|
||||
Since all tasks are executed in thread pools, the time limit features of Celery are silently
|
||||
disabled and won't work. Timeout logic must be implemented within the task itself.
|
||||
|
||||
### Testing Worker Changes
|
||||
|
||||
If you make any updates to a celery worker and you want to test these changes, you will need
|
||||
to ask the user to restart the celery worker. There is no auto-restart on code-change mechanism.
|
||||
|
||||
## Database & Migrations
|
||||
|
||||
Run all `alembic` commands from `backend/` (where `alembic.ini` lives) through `uv run`.
|
||||
|
||||
### Running Migrations
|
||||
|
||||
```bash
|
||||
# Standard migrations
|
||||
uv run alembic upgrade head
|
||||
|
||||
# Multi-tenant (Enterprise)
|
||||
uv run alembic -n schema_private upgrade head
|
||||
```
|
||||
|
||||
### Creating Migrations
|
||||
|
||||
```bash
|
||||
# Create migration
|
||||
uv run alembic revision -m "description"
|
||||
|
||||
# Multi-tenant migration
|
||||
uv run alembic -n schema_private revision -m "description"
|
||||
```
|
||||
|
||||
Write the migration manually and place it in the file that alembic creates when running the above command.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Run pytest through `uv run` from the repo root — no venv activation needed (`uv run` uses the
|
||||
lockfile-pinned environment and creates/syncs `.venv` as needed).
|
||||
|
||||
There are 4 main types of tests within Onyx:
|
||||
|
||||
### Model choice for tests that make real LLM calls
|
||||
|
||||
When a test makes a real LLM call (e.g. External Dependency Unit / integration tests
|
||||
that hit a live provider), use the cheap-and-fast tier for each provider:
|
||||
|
||||
- **OpenAI**: `gpt-5-mini` (never `gpt-4o` / `gpt-4o-mini`)
|
||||
- **Anthropic**: `claude-haiku-4-5`
|
||||
|
||||
### Unit Tests
|
||||
|
||||
These should not assume any Onyx/external services are available to be called.
|
||||
Interactions with the outside world should be mocked using `unittest.mock`. Generally, only
|
||||
write these for complex, isolated modules e.g. `citation_processing.py`.
|
||||
|
||||
To run them:
|
||||
|
||||
```bash
|
||||
uv run pytest -xv backend/tests/unit
|
||||
```
|
||||
|
||||
### External Dependency Unit Tests
|
||||
|
||||
These tests assume that all external dependencies of Onyx are available and callable (e.g. Postgres, Redis,
|
||||
MinIO/S3, OpenSearch are running + OpenAI can be called + any request to the internet is fine + etc.).
|
||||
|
||||
However, the actual Onyx containers are not running and with these tests we call the function to test directly.
|
||||
We can also mock components/calls at will.
|
||||
|
||||
The goal with these tests are to minimize mocking while giving some flexibility to mock things that are flakey,
|
||||
need strictly controlled behavior, or need to have their internal behavior validated (e.g. verify a function is called
|
||||
with certain args, something that would be impossible with proper integration tests).
|
||||
|
||||
A great example of this type of test is `backend/tests/external_dependency_unit/connectors/confluence/test_confluence_group_sync.py`.
|
||||
|
||||
To run them:
|
||||
|
||||
```bash
|
||||
uv run --env-file .vscode/.env pytest backend/tests/external_dependency_unit
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
Standard integration tests. Every test in `backend/tests/integration` runs against a real Onyx deployment. We cannot
|
||||
mock anything in these tests. Prefer writing integration tests (or External Dependency Unit Tests if mocking/internal
|
||||
verification is necessary) over any other type of test.
|
||||
|
||||
Tests are parallelized at a directory level.
|
||||
|
||||
When writing integration tests, make sure to check the root `conftest.py` for useful fixtures + the `backend/tests/integration/common_utils` directory for utilities. Prefer (if one exists), calling the appropriate Manager
|
||||
class in the utils over directly calling the APIs with a library like `requests`. Prefer using fixtures rather than
|
||||
calling the utilities directly (e.g. do NOT create admin users with
|
||||
`admin_user = UserManager.create(name="admin_user")`, instead use the `admin_user` fixture).
|
||||
|
||||
A great example of this type of test is `backend/tests/integration/tests/streaming_endpoints/test_chat_stream.py`.
|
||||
|
||||
To run them:
|
||||
|
||||
```bash
|
||||
uv run --env-file .vscode/.env pytest backend/tests/integration
|
||||
```
|
||||
|
||||
### Playwright (E2E) Tests
|
||||
|
||||
These tests are an even more complete version of the Integration Tests mentioned above. Has all services of Onyx
|
||||
running, _including_ the Web Server.
|
||||
|
||||
Use these tests for anything that requires significant frontend <-> backend coordination.
|
||||
|
||||
Tests are located at `web/tests/e2e`. Tests are written in TypeScript. Spec-writing rules
|
||||
(Page Object Model, locator priority) live in `web/tests/e2e/README.md`.
|
||||
|
||||
To run them (the `playwright` script expands to `playwright test`; use it rather than
|
||||
`bunx`/`npx`, which can silently fetch an unpinned Playwright version):
|
||||
|
||||
```bash
|
||||
cd web && bun run playwright <TEST_NAME>
|
||||
```
|
||||
|
||||
For shared fixtures, best practices, and detailed guidance, see `backend/tests/README.md`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
**Always raise `OnyxError` from `onyx.error_handling.exceptions` instead of `HTTPException`.
|
||||
Never hardcode status codes or use `starlette.status` / `fastapi.status` constants directly.**
|
||||
|
||||
A global FastAPI exception handler converts `OnyxError` into a JSON response with the standard
|
||||
`{"error_code": "...", "detail": "..."}` shape. This eliminates boilerplate and keeps error
|
||||
handling consistent across the entire backend.
|
||||
|
||||
```python
|
||||
from onyx.error_handling.error_codes import OnyxErrorCode
|
||||
from onyx.error_handling.exceptions import OnyxError
|
||||
|
||||
# ✅ Good
|
||||
raise OnyxError(OnyxErrorCode.NOT_FOUND, "Session not found")
|
||||
|
||||
# ✅ Good — no extra message needed
|
||||
raise OnyxError(OnyxErrorCode.UNAUTHENTICATED)
|
||||
|
||||
# ✅ Good — upstream service with dynamic status code
|
||||
raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=upstream_status)
|
||||
|
||||
# ❌ Bad — using HTTPException directly
|
||||
raise HTTPException(status_code=404, detail="Session not found")
|
||||
|
||||
# ❌ Bad — starlette constant
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied")
|
||||
```
|
||||
|
||||
Available error codes are defined in `backend/onyx/error_handling/error_codes.py`. If a new error
|
||||
category is needed, add it there first — do not invent ad-hoc codes.
|
||||
|
||||
**Upstream service errors:** When forwarding errors from an upstream service where the HTTP
|
||||
status code is dynamic (comes from the upstream response), use `status_code_override`:
|
||||
|
||||
```python
|
||||
raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=e.response.status_code)
|
||||
```
|
||||
|
||||
## AI/LLM Integration
|
||||
|
||||
LLM calls go through LiteLLM; models are configurable per feature (chat, search, embeddings).
|
||||
|
||||
### Tracing — every LLM invocation must be tagged
|
||||
|
||||
Every LLM, embedding, rerank, image-generation, voice (STT/TTS), and intent-classification call must open a generation span tagged with a value from the `LLMFlow` registry in `backend/onyx/tracing/flows.py`. Use one of:
|
||||
|
||||
- `llm_generation_span(llm=..., flow=LLMFlow.X, input_messages=...)` for calls going through an `LLM` subclass.
|
||||
- `traced_llm_call(flow=LLMFlow.X, model=..., provider=..., input_messages=...)` for direct provider SDK / `litellm` / model_server HTTP calls that bypass the `LLM` abstraction.
|
||||
|
||||
Rules:
|
||||
|
||||
1. Add a new `LLMFlow` enum value before instrumenting a new operation. Don't pass raw strings.
|
||||
2. Flow tags name the **operation** (e.g. `IMAGE_EDIT`, `RERANK`) — not the provider. Provider lives in `model_config["model_provider"]`.
|
||||
3. The auto-wrap fallback in `onyx/llm/tracing_wrap.py` emits `LLMFlow.UNTAGGED_INVOKE` / `UNTAGGED_STREAM` for calls that reach `LLM.invoke` / `LLM.stream` without an explicit span. These sentinels are visible in dashboards and indicate missing instrumentation — fix the call site, don't rely on the fallback.
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
@@ -27,7 +27,7 @@ relevant env var or credential.
|
||||
Use when you need a real database or real API calls but want control over setup.
|
||||
|
||||
```bash
|
||||
python -m dotenv -f .vscode/.env run -- pytest backend/tests/external_dependency_unit
|
||||
uv run --env-file .vscode/.env pytest backend/tests/external_dependency_unit
|
||||
```
|
||||
|
||||
### Integration Tests (`tests/integration/`)
|
||||
@@ -47,7 +47,7 @@ contracts that do not have an HTTP API. Direct task/stub checks belong in
|
||||
`tests/external_dependency_unit/craft/`.
|
||||
|
||||
```bash
|
||||
python -m dotenv -f .vscode/.env run -- pytest backend/tests/integration
|
||||
uv run --env-file .vscode/.env pytest backend/tests/integration
|
||||
```
|
||||
|
||||
### Playwright / E2E Tests (`web/tests/e2e/`)
|
||||
@@ -55,7 +55,7 @@ python -m dotenv -f .vscode/.env run -- pytest backend/tests/integration
|
||||
Full stack including web server. Use for frontend-backend coordination.
|
||||
|
||||
```bash
|
||||
npx playwright test <TEST_NAME>
|
||||
cd web && bun run playwright <TEST_NAME>
|
||||
```
|
||||
|
||||
## Shared Fixtures
|
||||
|
||||
@@ -633,3 +633,12 @@ export function useToast() { ... }
|
||||
// ❌ Bad — user/session hook dumped in the global hooks directory
|
||||
// web/src/hooks/useSessionWatcher.ts
|
||||
```
|
||||
|
||||
# Tests
|
||||
|
||||
- Jest + React Testing Library guide for component tests: `web/tests/README.md`.
|
||||
- Playwright e2e specs live in `web/tests/e2e`; hard rules (Page Object Model, locator priority)
|
||||
are in `web/tests/e2e/README.md`.
|
||||
- Run an e2e test with the repo-pinned Playwright: `cd web && bun run playwright <TEST_NAME>`
|
||||
(the `playwright` script expands to `playwright test`; avoid `bunx`/`npx`, which can silently
|
||||
fetch an unpinned version).
|
||||
+8
-6
@@ -59,30 +59,32 @@ Don't run these tests if you don't want to do this!
|
||||
|
||||
Bring up the entire application.
|
||||
|
||||
0. Install playwright dependencies
|
||||
0. Install playwright dependencies (run `bun install` first so `bunx` resolves to the
|
||||
repo-pinned Playwright in `node_modules` instead of fetching the latest version)
|
||||
|
||||
```bash
|
||||
bun install
|
||||
bunx playwright install
|
||||
```
|
||||
|
||||
1. Run playwright
|
||||
1. Run playwright (the `playwright` script expands to `playwright test`)
|
||||
|
||||
```bash
|
||||
bunx playwright test
|
||||
bun run playwright
|
||||
```
|
||||
|
||||
To run a single test:
|
||||
|
||||
```bash
|
||||
bunx playwright test landing-page.spec.ts
|
||||
bun run playwright landing-page.spec.ts
|
||||
```
|
||||
|
||||
If running locally, interactive options can help you see exactly what is happening in
|
||||
the test.
|
||||
|
||||
```bash
|
||||
bunx playwright test --ui
|
||||
bunx playwright test --headed
|
||||
bun run playwright --ui
|
||||
bun run playwright --headed
|
||||
```
|
||||
|
||||
2. Inspect results
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Hard rules for tests under `web/tests/e2e/`. Read before adding or modifying a spec.
|
||||
|
||||
For the broader Onyx testing strategy and where Playwright fits among unit / external-dependency / integration tests, see `CLAUDE.md` ("Testing Strategy") and `backend/tests/README.md`. For Jest + React Testing Library guidance for component tests, see `web/tests/README.md`.
|
||||
For the broader Onyx testing strategy and where Playwright fits among unit / external-dependency / integration tests, see `backend/AGENTS.md` ("Testing Strategy") and `backend/tests/README.md`. For Jest + React Testing Library guidance for component tests, see `web/tests/README.md`.
|
||||
|
||||
## 1. Use the Page Object Model
|
||||
|
||||
|
||||
Reference in new issue
Block a user