diff --git a/.devcontainer/claude-code/CLAUDE.md b/.devcontainer/claude-code/CLAUDE.md index 8cd51d6b70..4616407555 100644 --- a/.devcontainer/claude-code/CLAUDE.md +++ b/.devcontainer/claude-code/CLAUDE.md @@ -6,9 +6,9 @@ Running **inside the Onyx dev container**. These notes are additive to the root ## No Docker daemon in here Don't use `docker` / `docker exec` / `docker compose`. Onyx services run as sibling -containers on the `onyx_default` network, reachable directly by hostname — so the root -guide's `docker exec -it onyx-relational_db-1 psql ...` won't work. Use -`psql -h relational_db -U postgres -c ""` instead. +containers on the `onyx_default` network, reachable directly by hostname — the root +guide's `psql` command works as-is here (the env vars below plus `POSTGRES_PASSWORD` +are exported); its `docker exec` fallback won't. ## Service hostnames (`onyx_default` network) diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index bfe663cf71..402aa4bc9e 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -26,6 +26,7 @@ "ONYX_DEVCONTAINER_FIREWALL": "${localEnv:ONYX_DEVCONTAINER_FIREWALL}", "OPENSEARCH_HOST": "opensearch", "POSTGRES_HOST": "relational_db", + "POSTGRES_PASSWORD": "${localEnv:POSTGRES_PASSWORD:password}", "REDIS_HOST": "cache", "S3_ENDPOINT_URL": "http://minio:9000", "SSH_AUTH_SOCK": "/tmp/ssh-agent.sock", diff --git a/AGENTS.md b/AGENTS.md index 22290f86a7..0b9675b374 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,131 +4,61 @@ This file provides guidance to AI agents when working with code in this reposito ## KEY NOTES -- Python deps live in a `uv`-managed virtualenv at `.venv` (repo root). If it doesn't exist yet, create it \ +- Python deps live in a `uv`-managed virtualenv at `.venv` (repo root). If it doesn't exist yet, create it with `uv sync --frozen`, then `source .venv/bin/activate`. -- To make tests work, check the `.env` file at the root of the project to find an OpenAI key. +- Test secrets (API keys etc.) are resolved by `backend/tests/utils/aws_secrets.py`, in order: process + env vars → the gitignored `.vscode/.env` (also used by the pytest commands in `backend/AGENTS.md`; + create it by copying `.vscode/env_template.txt`) → AWS Secrets Manager (requires `aws sso login`). + Tests declare what they need via `@pytest.mark.secrets(TestSecret.X)`. If a key you need still can't + be resolved, ask the user rather than skipping tests. - If using `playwright` to explore the frontend, log in with username `admin_user@example.com` and password `TestPassword123!` (the admin user created by the playwright global setup — see `web/tests/e2e/constants.ts`). If it doesn't exist yet, register it via the signup page; the first user registered automatically becomes admin. The app can be accessed at `http://localhost:3000`. - You should assume that all Onyx services are running. To verify, you can check the `backend/log` directory to make sure we see logs coming out from the relevant service. -- To connect to the Postgres database, use: `docker exec -it onyx-relational_db-1 psql -U postgres -c ""` +- To connect to the Postgres database, use: + `PGPASSWORD="${POSTGRES_PASSWORD:-password}" psql -h "${POSTGRES_HOST:-localhost}" -U postgres -c ""`. + This works on a host checkout and inside the devcontainer. If no `psql` client is available, fall back to + `docker exec onyx-relational_db-1 psql -U postgres -c ""` (no `-it` — agent shells have no TTY). - When making calls to the backend, always go through the frontend. E.g. make a call to `http://localhost:3000/api/persona` not `http://localhost:8080/api/persona` -- Put ALL db operations under the `backend/onyx/db` / `backend/ee/onyx/db` directories. Don't run queries - outside of those directories. ## Project Overview **Onyx** (formerly Danswer) is an open-source Gen-AI and Enterprise Search platform that connects to company documents, apps, and people. It features a modular architecture with both Community Edition (MIT licensed) and Enterprise Edition offerings. -### Background Workers (Celery) +### Technology Stack -Onyx uses Celery for asynchronous task processing with multiple specialized workers: +- **Backend**: Python 3.13, FastAPI, SQLAlchemy, Alembic, Celery +- **Frontend**: Next.js 16, React 19, TypeScript, Tailwind CSS +- **Database**: PostgreSQL with Redis caching +- **Search**: OpenSearch-backed keyword and vector document index +- **Auth**: OAuth2, SAML, multi-provider support +- **AI/ML**: LangChain, LiteLLM, multiple embedding models -#### Worker Types +### Repository Layout & Sub-project Guides -1. **Primary Worker** (`celery_app.py`) +Each sub-project has its own agents file with the standards for that area — read it before working +there: - - Coordinates core background tasks and system-wide operations - - Handles connector management, document sync, pruning, and periodic checks - - Runs with 4 threads concurrency - - Tasks: connector deletion, document-index sync, pruning, LLM model updates, user file sync +- `backend/` — FastAPI app + Celery workers. `onyx/` is the Community Edition core, `ee/` mirrors its + layout for Enterprise features, `alembic/` holds migrations, `tests/` the test suites. Standards + (Celery, migrations, testing, error handling, LLM tracing): `backend/AGENTS.md`. +- `web/` — Next.js frontend. Standards (also cover `desktop/`, the Tauri shell): `web/AGENTS.md`. +- `mobile/` — React Native + Expo app. Standards: `mobile/AGENTS.md`. Mobile differs from web on + several points (no DOM, NativeWind, expo-router), so do **not** assume the web rules apply there. -2. **Docfetching Worker** (`docfetching`) +Explore the tree with `ls` rather than relying on docs for the full package list. - - Fetches documents from external data sources (connectors) - - Spawns docprocessing tasks for each document batch - - Implements watchdog monitoring for stuck connectors - - Configurable concurrency (default from env) - -3. **Docprocessing Worker** (`docprocessing`) - - - Processes fetched documents through the indexing pipeline: - - Upserts documents to PostgreSQL - - Chunks documents and adds contextual information - - Embeds chunks via model server - - Writes chunks to the OpenSearch-backed document index - - Updates document metadata - - Configurable concurrency (default from env) - -4. **Light Worker** (`light`) - - - Handles lightweight, fast operations - - Tasks: document-index metadata sync, connector deletion, doc permissions upsert, checkpoint cleanup, index attempt cleanup - - Higher concurrency for quick tasks - -5. **Heavy Worker** (`heavy`) - - - Handles resource-intensive operations - - Tasks: connector pruning, document permissions sync, external group sync, CSV generation - - Runs with 4 threads concurrency - -6. **Monitoring Worker** (`monitoring`) - - - System health monitoring and metrics collection - - Monitors Celery queues, process memory, and system status - - Single thread (monitoring doesn't need parallelism) - - Cloud-specific monitoring tasks - -7. **User File Processing Worker** (`user_file_processing`) - - - Processes user-uploaded files - - Handles user file indexing and project synchronization - - Configurable concurrency - -8. **Beat Worker** (`beat`) - - Celery's scheduler for periodic tasks - - Uses DynamicTenantScheduler for multi-tenant support - - Schedules tasks like: - - Indexing checks (every 15 seconds) - - Connector deletion checks (every 20 seconds) - - Document-index metadata sync checks (every 20 seconds) - - Pruning checks (every 20 seconds) - - Monitoring tasks (every 5 minutes) - - Cleanup tasks (hourly) - -#### Key Features - -- **Thread-based Workers**: All workers use thread pools (not processes) for stability -- **Tenant Awareness**: Multi-tenant support with per-tenant task isolation. There is a - middleware layer that automatically finds the appropriate tenant ID when sending tasks - via Celery Beat. -- **Task Prioritization**: High, Medium, Low priority queues -- **Monitoring**: Built-in heartbeat and liveness checking -- **Failure Handling**: Automatic retry and failure recovery mechanisms -- **Redis Coordination**: Inter-process communication via Redis -- **PostgreSQL State**: Task state and metadata stored in PostgreSQL - -#### Important Notes - -**Defining Tasks**: - -- Always use `@shared_task` rather than `@celery_app` -- Put tasks under `background/celery/tasks/` or `ee/background/celery/tasks` -- Never enqueue a task without an expiration. Always supply `expires=` when - sending tasks, either from the beat schedule or directly from another task. It - should never be acceptable to submit code which enqueues tasks without an - expiration, as doing so can lead to unbounded task queue growth. - -**Defining APIs**: -When creating new FastAPI APIs, do NOT use the `response_model` field. Instead, just type the -function. - -**Testing Updates**: -If you make any updates to a celery worker and you want to test these changes, you will need -to ask me to restart the celery worker. There is no auto-restart on code-change mechanism. - -**Task Time Limits**: -Since all tasks are executed in thread pools, the time limit features of Celery are silently -disabled and won't work. Timeout logic must be implemented within the task itself. - -### Code Quality +## Code Quality ```bash # Install and run pre-commit hooks pre-commit install pre-commit run --all-files + +# Faster: run only on the files you touched +pre-commit run --files [ ...] ``` NOTE: Always make sure everything is strictly typed (both in Python and Typescript). @@ -136,161 +66,11 @@ NOTE: Always make sure everything is strictly typed (both in Python and Typescri NOTE: Keep comments brief and focused on information that stays relevant long-term. Don't write comments that only describe the instantaneous change (e.g. what was just added/removed/refactored). -## Architecture Overview +## Testing -### Technology Stack - -- **Backend**: Python 3.13, FastAPI, SQLAlchemy, Alembic, Celery -- **Frontend**: Next.js 15+, React 18, TypeScript, Tailwind CSS -- **Database**: PostgreSQL with Redis caching -- **Search**: OpenSearch-backed keyword and vector document index -- **Auth**: OAuth2, SAML, multi-provider support -- **AI/ML**: LangChain, LiteLLM, multiple embedding models - -OpenSearch is the current document index backend for search and indexing. Some -legacy modules, Celery task names, and migration helpers still mention Vespa; treat -those as compatibility or migration artifacts unless the active `DocumentIndex` -factory/config path explicitly uses them. - -### Directory Structure - -``` -backend/ -├── onyx/ -│ ├── auth/ # Authentication & authorization -│ ├── chat/ # Chat functionality & LLM interactions -│ ├── connectors/ # Data source connectors -│ ├── db/ # Database models & operations -│ ├── document_index/ # OpenSearch-backed DocumentIndex integration -│ ├── federated_connectors/ # External search connectors -│ ├── llm/ # LLM provider integrations -│ └── server/ # API endpoints & routers -├── ee/ # Enterprise Edition features -├── alembic/ # Database migrations -└── tests/ # Test suites - -web/ -├── src/app/ # Next.js app router pages -├── src/components/ # Reusable React components -└── src/lib/ # Utilities & business logic -``` - -## Frontend Standards - -Frontend standards for the `web/` and `desktop/` projects live in `web/AGENTS.md`. - -Standards for the **mobile** app (React Native + Expo) live in `mobile/AGENTS.md`. Mobile differs -from web on several points (no DOM, NativeWind instead of web Tailwind — e.g. spacing classes are -pixel-valued, not web's rem step scale — expo-router, RN primitives), so do **not** assume the web -rules apply when working in `mobile/`. - -## Database & Migrations - -### Running Migrations - -```bash -# Standard migrations -alembic upgrade head - -# Multi-tenant (Enterprise) -alembic -n schema_private upgrade head -``` - -### Creating Migrations - -```bash -# Create migration -alembic revision -m "description" - -# Multi-tenant migration -alembic -n schema_private revision -m "description" -``` - -Write the migration manually and place it in the file that alembic creates when running the above command. - -## Testing Strategy - -First, activate the virtualenv: `source .venv/bin/activate`. If `.venv` doesn't exist yet, create it first with `uv sync --frozen`. - -There are 4 main types of tests within Onyx: - -### Model choice for tests that make real LLM calls - -When a test makes a real LLM call (e.g. External Dependency Unit / integration tests -that hit a live provider), use the cheap-and-fast tier for each provider: - -- **OpenAI**: `gpt-5-mini` (never `gpt-4o` / `gpt-4o-mini`) -- **Anthropic**: `claude-haiku-4-5` - -### Unit Tests - -These should not assume any Onyx/external services are available to be called. -Interactions with the outside world should be mocked using `unittest.mock`. Generally, only -write these for complex, isolated modules e.g. `citation_processing.py`. - -To run them: - -```bash -pytest -xv backend/tests/unit -``` - -### External Dependency Unit Tests - -These tests assume that all external dependencies of Onyx are available and callable (e.g. Postgres, Redis, -MinIO/S3, OpenSearch are running + OpenAI can be called + any request to the internet is fine + etc.). - -However, the actual Onyx containers are not running and with these tests we call the function to test directly. -We can also mock components/calls at will. - -The goal with these tests are to minimize mocking while giving some flexibility to mock things that are flakey, -need strictly controlled behavior, or need to have their internal behavior validated (e.g. verify a function is called -with certain args, something that would be impossible with proper integration tests). - -A great example of this type of test is `backend/tests/external_dependency_unit/connectors/confluence/test_confluence_group_sync.py`. - -To run them: - -```bash -python -m dotenv -f .vscode/.env run -- pytest backend/tests/external_dependency_unit -``` - -### Integration Tests - -Standard integration tests. Every test in `backend/tests/integration` runs against a real Onyx deployment. We cannot -mock anything in these tests. Prefer writing integration tests (or External Dependency Unit Tests if mocking/internal -verification is necessary) over any other type of test. - -Tests are parallelized at a directory level. - -When writing integration tests, make sure to check the root `conftest.py` for useful fixtures + the `backend/tests/integration/common_utils` directory for utilities. Prefer (if one exists), calling the appropriate Manager -class in the utils over directly calling the APIs with a library like `requests`. Prefer using fixtures rather than -calling the utilities directly (e.g. do NOT create admin users with -`admin_user = UserManager.create(name="admin_user")`, instead use the `admin_user` fixture). - -A great example of this type of test is `backend/tests/integration/tests/streaming_endpoints/test_chat_stream.py`. - -To run them: - -```bash -python -m dotenv -f .vscode/.env run -- pytest backend/tests/integration -``` - -### Playwright (E2E) Tests - -These tests are an even more complete version of the Integration Tests mentioned above. Has all services of Onyx -running, _including_ the Web Server. - -Use these tests for anything that requires significant frontend <-> backend coordination. - -Tests are located at `web/tests/e2e`. Tests are written in TypeScript. - -To run them: - -```bash -bunx playwright test -``` - -For shared fixtures, best practices, and detailed guidance, see `backend/tests/README.md`. +There are 4 main types of tests: unit, external dependency unit, integration, and playwright e2e +(`web/tests/e2e`). Commands and guidance for all four live in `backend/AGENTS.md`; shared fixtures +and deeper detail in `backend/tests/README.md`. Prefer integration tests over the other types. ## Logs @@ -300,36 +80,14 @@ will be tailing their logs to this file. ## Security Considerations -- Never commit API keys or secrets to repository -- Use encrypted credential storage for connector credentials -- Follow RBAC patterns for new features -- Implement proper input validation with Pydantic models -- Use parameterized queries to prevent SQL injection - -## AI/LLM Integration - -- Multiple LLM providers supported via LiteLLM -- Configurable models per feature (chat, search, embeddings) -- Streaming support for real-time responses -- Token management and rate limiting -- Custom prompts and agent actions - -### Tracing — every LLM invocation must be tagged - -Every LLM, embedding, rerank, image-generation, voice (STT/TTS), and intent-classification call must open a generation span tagged with a value from the `LLMFlow` registry in `backend/onyx/tracing/flows.py`. Use one of: - -- `llm_generation_span(llm=..., flow=LLMFlow.X, input_messages=...)` for calls going through an `LLM` subclass. -- `traced_llm_call(flow=LLMFlow.X, model=..., provider=..., input_messages=...)` for direct provider SDK / `litellm` / model_server HTTP calls that bypass the `LLM` abstraction. - -Rules: - -1. Add a new `LLMFlow` enum value before instrumenting a new operation. Don't pass raw strings. -2. Flow tags name the **operation** (e.g. `IMAGE_EDIT`, `RERANK`) — not the provider. Provider lives in `model_config["model_provider"]`. -3. The auto-wrap fallback in `onyx/llm/tracing_wrap.py` emits `LLMFlow.UNTAGGED_INVOKE` / `UNTAGGED_STREAM` for calls that reach `LLM.invoke` / `LLM.stream` without an explicit span. These sentinels are visible in dashboards and indicate missing instrumentation — fix the call site, don't rely on the fallback. +- Never commit API keys or secrets to the repository +- Use the encrypted credential storage for connector credentials +- Follow existing RBAC patterns for new features ## Creating a Plan -When creating a plan in the `plans` directory, make sure to include at least these elements: +When creating a plan in the `plans` directory (gitignored — create it if it doesn't exist), make sure to +include at least these elements: **Issues to Address** What the change is meant to do. @@ -351,45 +109,6 @@ Keep it high level. You can reference certain files or functions though. Before writing your plan, make sure to do research. Explore the relevant sections in the codebase. -## Error Handling - -**Always raise `OnyxError` from `onyx.error_handling.exceptions` instead of `HTTPException`. -Never hardcode status codes or use `starlette.status` / `fastapi.status` constants directly.** - -A global FastAPI exception handler converts `OnyxError` into a JSON response with the standard -`{"error_code": "...", "detail": "..."}` shape. This eliminates boilerplate and keeps error -handling consistent across the entire backend. - -```python -from onyx.error_handling.error_codes import OnyxErrorCode -from onyx.error_handling.exceptions import OnyxError - -# ✅ Good -raise OnyxError(OnyxErrorCode.NOT_FOUND, "Session not found") - -# ✅ Good — no extra message needed -raise OnyxError(OnyxErrorCode.UNAUTHENTICATED) - -# ✅ Good — upstream service with dynamic status code -raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=upstream_status) - -# ❌ Bad — using HTTPException directly -raise HTTPException(status_code=404, detail="Session not found") - -# ❌ Bad — starlette constant -raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied") -``` - -Available error codes are defined in `backend/onyx/error_handling/error_codes.py`. If a new error -category is needed, add it there first — do not invent ad-hoc codes. - -**Upstream service errors:** When forwarding errors from an upstream service where the HTTP -status code is dynamic (comes from the upstream response), use `status_code_override`: - -```python -raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=e.response.status_code) -``` - ## Best Practices In addition to the other content in this file, best practices for contributing diff --git a/backend/AGENTS.md b/backend/AGENTS.md new file mode 100644 index 0000000000..6cb5b36082 --- /dev/null +++ b/backend/AGENTS.md @@ -0,0 +1,230 @@ +# BACKEND STANDARDS + +Guidance for the Python backend (`backend/`): the FastAPI app, Celery workers, database, and +tests. Additive to the root `AGENTS.md`. + +## Key Rules + +- Put ALL db operations under the `backend/onyx/db` / `backend/ee/onyx/db` directories. Don't run + queries outside of those directories. +- When creating new FastAPI APIs, do NOT use the `response_model` field. Instead, just type the + function. +- OpenSearch is the current document index backend for search and indexing. Some legacy modules, + Celery task names, and migration helpers still mention Vespa; treat those as compatibility or + migration artifacts unless the active `DocumentIndex` factory/config path explicitly uses them. + +## Background Workers (Celery) + +Onyx uses Celery for asynchronous task processing. Worker apps live in +`backend/onyx/background/celery/apps/`; the periodic schedule is defined in +`backend/onyx/background/celery/tasks/beat_schedule.py`. + +| Worker | Role | +| --- | --- | +| `primary` | Coordinates core background tasks: connector management/deletion, document-index sync, pruning checks, LLM model updates, user file sync | +| `docfetching` | Fetches documents from connectors, spawns docprocessing tasks; watchdog for stuck connectors | +| `docprocessing` | Indexing pipeline: upsert docs to Postgres, chunk, embed via model server, write chunks to the document index, update metadata | +| `light` | Fast lightweight ops: metadata sync, permissions upsert, checkpoint / index-attempt cleanup | +| `heavy` | Resource-intensive ops: pruning, document permissions sync, external group sync, CSV generation | +| `monitoring` | System health monitoring & metrics collection | +| `user_file_processing` | User-uploaded file indexing & project synchronization | +| `scheduled_tasks` | Executes user-scheduled (Craft) task runs | +| `beat` | Scheduler for periodic tasks; uses `DynamicTenantScheduler` for multi-tenant support | + +Key facts: + +- All workers use thread pools (not processes) — this is why Celery time limits don't work (see below). +- Multi-tenant: `DynamicTenantScheduler` explicitly adds `tenant_id` to each Celery Beat task's + kwargs; direct task sends must propagate it themselves (`TenantAwareTask` silently falls back to + the default schema when it's absent). +- Tasks route to named queues and carry a separate High/Medium/Low priority (`OnyxCeleryQueues` / + `OnyxCeleryPriority` in `backend/onyx/configs/constants.py`); Redis coordinates inter-process + communication; task state and metadata live in PostgreSQL. + +### Defining Tasks + +- Always use `@shared_task` rather than `@celery_app` +- Put tasks under `background/celery/tasks/` or `ee/background/celery/tasks` +- Never enqueue a task without an expiration. Always supply `expires=` when + sending tasks, either from the beat schedule or directly from another task. It + should never be acceptable to submit code which enqueues tasks without an + expiration, as doing so can lead to unbounded task queue growth. + +### Task Time Limits + +Since all tasks are executed in thread pools, the time limit features of Celery are silently +disabled and won't work. Timeout logic must be implemented within the task itself. + +### Testing Worker Changes + +If you make any updates to a celery worker and you want to test these changes, you will need +to ask the user to restart the celery worker. There is no auto-restart on code-change mechanism. + +## Database & Migrations + +Run all `alembic` commands from `backend/` (where `alembic.ini` lives) through `uv run`. + +### Running Migrations + +```bash +# Standard migrations +uv run alembic upgrade head + +# Multi-tenant (Enterprise) +uv run alembic -n schema_private upgrade head +``` + +### Creating Migrations + +```bash +# Create migration +uv run alembic revision -m "description" + +# Multi-tenant migration +uv run alembic -n schema_private revision -m "description" +``` + +Write the migration manually and place it in the file that alembic creates when running the above command. + +## Testing Strategy + +Run pytest through `uv run` from the repo root — no venv activation needed (`uv run` uses the +lockfile-pinned environment and creates/syncs `.venv` as needed). + +There are 4 main types of tests within Onyx: + +### Model choice for tests that make real LLM calls + +When a test makes a real LLM call (e.g. External Dependency Unit / integration tests +that hit a live provider), use the cheap-and-fast tier for each provider: + +- **OpenAI**: `gpt-5-mini` (never `gpt-4o` / `gpt-4o-mini`) +- **Anthropic**: `claude-haiku-4-5` + +### Unit Tests + +These should not assume any Onyx/external services are available to be called. +Interactions with the outside world should be mocked using `unittest.mock`. Generally, only +write these for complex, isolated modules e.g. `citation_processing.py`. + +To run them: + +```bash +uv run pytest -xv backend/tests/unit +``` + +### External Dependency Unit Tests + +These tests assume that all external dependencies of Onyx are available and callable (e.g. Postgres, Redis, +MinIO/S3, OpenSearch are running + OpenAI can be called + any request to the internet is fine + etc.). + +However, the actual Onyx containers are not running and with these tests we call the function to test directly. +We can also mock components/calls at will. + +The goal with these tests are to minimize mocking while giving some flexibility to mock things that are flakey, +need strictly controlled behavior, or need to have their internal behavior validated (e.g. verify a function is called +with certain args, something that would be impossible with proper integration tests). + +A great example of this type of test is `backend/tests/external_dependency_unit/connectors/confluence/test_confluence_group_sync.py`. + +To run them: + +```bash +uv run --env-file .vscode/.env pytest backend/tests/external_dependency_unit +``` + +### Integration Tests + +Standard integration tests. Every test in `backend/tests/integration` runs against a real Onyx deployment. We cannot +mock anything in these tests. Prefer writing integration tests (or External Dependency Unit Tests if mocking/internal +verification is necessary) over any other type of test. + +Tests are parallelized at a directory level. + +When writing integration tests, make sure to check the root `conftest.py` for useful fixtures + the `backend/tests/integration/common_utils` directory for utilities. Prefer (if one exists), calling the appropriate Manager +class in the utils over directly calling the APIs with a library like `requests`. Prefer using fixtures rather than +calling the utilities directly (e.g. do NOT create admin users with +`admin_user = UserManager.create(name="admin_user")`, instead use the `admin_user` fixture). + +A great example of this type of test is `backend/tests/integration/tests/streaming_endpoints/test_chat_stream.py`. + +To run them: + +```bash +uv run --env-file .vscode/.env pytest backend/tests/integration +``` + +### Playwright (E2E) Tests + +These tests are an even more complete version of the Integration Tests mentioned above. Has all services of Onyx +running, _including_ the Web Server. + +Use these tests for anything that requires significant frontend <-> backend coordination. + +Tests are located at `web/tests/e2e`. Tests are written in TypeScript. Spec-writing rules +(Page Object Model, locator priority) live in `web/tests/e2e/README.md`. + +To run them (the `playwright` script expands to `playwright test`; use it rather than +`bunx`/`npx`, which can silently fetch an unpinned Playwright version): + +```bash +cd web && bun run playwright +``` + +For shared fixtures, best practices, and detailed guidance, see `backend/tests/README.md`. + +## Error Handling + +**Always raise `OnyxError` from `onyx.error_handling.exceptions` instead of `HTTPException`. +Never hardcode status codes or use `starlette.status` / `fastapi.status` constants directly.** + +A global FastAPI exception handler converts `OnyxError` into a JSON response with the standard +`{"error_code": "...", "detail": "..."}` shape. This eliminates boilerplate and keeps error +handling consistent across the entire backend. + +```python +from onyx.error_handling.error_codes import OnyxErrorCode +from onyx.error_handling.exceptions import OnyxError + +# ✅ Good +raise OnyxError(OnyxErrorCode.NOT_FOUND, "Session not found") + +# ✅ Good — no extra message needed +raise OnyxError(OnyxErrorCode.UNAUTHENTICATED) + +# ✅ Good — upstream service with dynamic status code +raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=upstream_status) + +# ❌ Bad — using HTTPException directly +raise HTTPException(status_code=404, detail="Session not found") + +# ❌ Bad — starlette constant +raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied") +``` + +Available error codes are defined in `backend/onyx/error_handling/error_codes.py`. If a new error +category is needed, add it there first — do not invent ad-hoc codes. + +**Upstream service errors:** When forwarding errors from an upstream service where the HTTP +status code is dynamic (comes from the upstream response), use `status_code_override`: + +```python +raise OnyxError(OnyxErrorCode.BAD_GATEWAY, detail, status_code_override=e.response.status_code) +``` + +## AI/LLM Integration + +LLM calls go through LiteLLM; models are configurable per feature (chat, search, embeddings). + +### Tracing — every LLM invocation must be tagged + +Every LLM, embedding, rerank, image-generation, voice (STT/TTS), and intent-classification call must open a generation span tagged with a value from the `LLMFlow` registry in `backend/onyx/tracing/flows.py`. Use one of: + +- `llm_generation_span(llm=..., flow=LLMFlow.X, input_messages=...)` for calls going through an `LLM` subclass. +- `traced_llm_call(flow=LLMFlow.X, model=..., provider=..., input_messages=...)` for direct provider SDK / `litellm` / model_server HTTP calls that bypass the `LLM` abstraction. + +Rules: + +1. Add a new `LLMFlow` enum value before instrumenting a new operation. Don't pass raw strings. +2. Flow tags name the **operation** (e.g. `IMAGE_EDIT`, `RERANK`) — not the provider. Provider lives in `model_config["model_provider"]`. +3. The auto-wrap fallback in `onyx/llm/tracing_wrap.py` emits `LLMFlow.UNTAGGED_INVOKE` / `UNTAGGED_STREAM` for calls that reach `LLM.invoke` / `LLM.stream` without an explicit span. These sentinels are visible in dashboards and indicate missing instrumentation — fix the call site, don't rely on the fallback. diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/backend/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/backend/tests/README.md b/backend/tests/README.md index 33240cf151..2116fbb7a4 100644 --- a/backend/tests/README.md +++ b/backend/tests/README.md @@ -27,7 +27,7 @@ relevant env var or credential. Use when you need a real database or real API calls but want control over setup. ```bash -python -m dotenv -f .vscode/.env run -- pytest backend/tests/external_dependency_unit +uv run --env-file .vscode/.env pytest backend/tests/external_dependency_unit ``` ### Integration Tests (`tests/integration/`) @@ -47,7 +47,7 @@ contracts that do not have an HTTP API. Direct task/stub checks belong in `tests/external_dependency_unit/craft/`. ```bash -python -m dotenv -f .vscode/.env run -- pytest backend/tests/integration +uv run --env-file .vscode/.env pytest backend/tests/integration ``` ### Playwright / E2E Tests (`web/tests/e2e/`) @@ -55,7 +55,7 @@ python -m dotenv -f .vscode/.env run -- pytest backend/tests/integration Full stack including web server. Use for frontend-backend coordination. ```bash -npx playwright test +cd web && bun run playwright ``` ## Shared Fixtures diff --git a/web/AGENTS.md b/web/AGENTS.md index 28671a165d..8c94148e60 100644 --- a/web/AGENTS.md +++ b/web/AGENTS.md @@ -633,3 +633,12 @@ export function useToast() { ... } // ❌ Bad — user/session hook dumped in the global hooks directory // web/src/hooks/useSessionWatcher.ts ``` + +# Tests + +- Jest + React Testing Library guide for component tests: `web/tests/README.md`. +- Playwright e2e specs live in `web/tests/e2e`; hard rules (Page Object Model, locator priority) + are in `web/tests/e2e/README.md`. +- Run an e2e test with the repo-pinned Playwright: `cd web && bun run playwright ` + (the `playwright` script expands to `playwright test`; avoid `bunx`/`npx`, which can silently + fetch an unpinned version). diff --git a/web/README.md b/web/README.md index 2813c65751..f7048b4486 100644 --- a/web/README.md +++ b/web/README.md @@ -59,30 +59,32 @@ Don't run these tests if you don't want to do this! Bring up the entire application. -0. Install playwright dependencies +0. Install playwright dependencies (run `bun install` first so `bunx` resolves to the + repo-pinned Playwright in `node_modules` instead of fetching the latest version) ```bash +bun install bunx playwright install ``` -1. Run playwright +1. Run playwright (the `playwright` script expands to `playwright test`) ```bash -bunx playwright test +bun run playwright ``` To run a single test: ```bash -bunx playwright test landing-page.spec.ts +bun run playwright landing-page.spec.ts ``` If running locally, interactive options can help you see exactly what is happening in the test. ```bash -bunx playwright test --ui -bunx playwright test --headed +bun run playwright --ui +bun run playwright --headed ``` 2. Inspect results diff --git a/web/tests/e2e/README.md b/web/tests/e2e/README.md index a99dd99174..899e1cb82c 100644 --- a/web/tests/e2e/README.md +++ b/web/tests/e2e/README.md @@ -2,7 +2,7 @@ Hard rules for tests under `web/tests/e2e/`. Read before adding or modifying a spec. -For the broader Onyx testing strategy and where Playwright fits among unit / external-dependency / integration tests, see `CLAUDE.md` ("Testing Strategy") and `backend/tests/README.md`. For Jest + React Testing Library guidance for component tests, see `web/tests/README.md`. +For the broader Onyx testing strategy and where Playwright fits among unit / external-dependency / integration tests, see `backend/AGENTS.md` ("Testing Strategy") and `backend/tests/README.md`. For Jest + React Testing Library guidance for component tests, see `web/tests/README.md`. ## 1. Use the Page Object Model