mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-23 03:42:59 -05:00
63fcb62f20bea596268a1f40ff187bd56571663a
11109
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Evan LohnandClaude Opus 5.5
|
63fcb62f20 |
fix(connectors): normalize shared credential site URLs and report the use rule's refusal
A user who could see a shared credential was refused unless the bound values equalled a pair's exactly, so https://x.atlassian.net/wiki did not match a pair saved as https://x.atlassian.net/ although both index the same site. The binding check also skipped the use rule, so it passed a form that the run then refused with a 403, and the account list offered accounts the rule refuses (ENG-4380). - CredentialBinding.comparison_values gives the values the rule compares. site_url_fields compare by site_url_key (scheme and host in lower case, no trailing slash); Confluence Cloud also drops a trailing /wiki. Stored configs are unchanged. - credential_use_refusal returns the rule's refusal without raising, and assert_user_can_use_credential raises it, so one function decides. - The binding checks apply the rule before decrypting the credential and report a refusal as a shared_credential_settings rejection with its detail. - /manage/admin/similar-credentials/{source} adds usable and refusal_reason per credential, from the rule with no config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
d440e4ced2 |
fix(connectors): say a pair is required only for non-derived bound fields
The pair rule skips credential-derived fields such as Drive's credential_kind, so the docstring and the feature map now say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
821f35d1d7 |
fix(connectors): cover an unpaired shared Drive credential
Drive has a binding model whose only field is derived from the credential, so it takes the field-subtraction path. The Drive test created a pair first, so it would pass if that path stopped allowing unpaired credentials. The unbound-source test now runs for Drive too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
be93193916 |
fix(connectors): let a shared credential of an unbound source be used without a pair
A non-owner could use a credential shared with them only when the request kept the bound values of a cc-pair that already used it. For sources with no credential-bound fields (e.g. Linear, or Google Drive whose only bound field is derived from the credential) there are no values that could send the secret to another host, yet a shared credential with no pair was refused with INSUFFICIENT_PERMISSIONS, so pre-existing accounts failed checks in the add-connector form (ENG-4369). The matching-pair rule now applies only when the source has bound fields other than credential_derived_fields. Ownership, admin use of ownerless credentials, CREDENTIAL_PERMISSIONS_TO_IGNORE and the current_pair shortcut are unchanged. The refusal now names only the compared fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
3a5ca32c0f |
fix(indexing): assert fetch event times in the batch telemetry test
The test counted fetch events but not their times, so it would pass if the carried wait were dropped or never reset. It now drives a fixed clock and asserts each recorded duration, with the trailing wait dropped on a clean end and carried into the failed fetch on an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
7c80cc23f0 |
fix(indexing): count only document and hierarchy batches as fetch events
_timed_connector_runs recorded one CONNECTOR_FETCH event for every runner yield, including the trailing checkpoint and failure tuples. Steps that fetched no documents then grew the Fetch Source batch count, which the UI shows as batches. Only document and hierarchy batches now record an event; wait time of other yields goes into the next batch of the same step. The per-step checkpoint log moves to DEBUG because it printed the full checkpoint, which can hold a page of up to 999 OneDrive users. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
3545d4d13a |
fix(onedrive): skip drive-less users within one checkpoint step
In all-users mode each load_from_checkpoint call probed at most one user. A tenant with thousands of users without a OneDrive then ended thousands of empty checkpoint steps, and a run looked stuck for many minutes while it only skipped users. One step now keeps probing users until a drive opens, the listing ends, or MAX_USERS_PROBED_PER_CHECKPOINT users are probed. Progress stays in the checkpoint, so a step that uses up the budget still returns mid-run. Explicit-user mode does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
42d958e2aa |
fix(pruning): run a prune only under its own fence and refuse one for a paused pair
A stale generator could adopt a newer prune's fence and hold the pruning lock in its place. A manual prune of a paused pair was dispatched and then dropped by the generator while the route reported success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
01d981b5c7 |
fix(pruning): drop a queued prune under the pair row lock
A pause read and reset the prune fence after it released the row lock, so a generator could mark its start, or a replacement prune could take the fence, between the read and the reset. The pause now drops a queued prune before its status commit, and the generator checks the status and marks its start under the same lock. A generator whose fence holds a different prune exits without a reset. Revokes run after the cleanup, so a failed revoke no longer skips it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
13017698e5 |
fix(pruning): cancel an abandoned prune's sync record and keep a pause's stop signal
A prune inserts its PRUNING sync record IN_PROGRESS at dispatch. When the prune was abandoned before it ran (pause, deletion, fence validation), the fence was reset but the record stayed IN_PROGRESS forever, and the UI showed the prune as running. - Add reset_prune_and_cancel_sync_record: it marks the open PRUNING record CANCELED, then resets the fence. Use it where the deletion beat's stop timeout and validate_pruning_fence reset the fence. - The prune generator exits without enumerating when the pair is gone, PAUSED or DELETING. The pair's prune request stays, so the prune runs again on resume. - A pause revokes a queued (not started) prune and cancels its record. - The deletion beat cleared the stop fence of every pair that was not being deleted on each tick, which undid a pause's stop signal. It now clears the fence only for DELETING pairs, where the deletion set it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
50977b03ca |
fix: review follow-ups for the connector checks stack (#16320)
## Description
Fixes for the review comments on the connector checks stack (#16201 to
#16266). The fixes are in one PR at the top of the stack, so the PRs
under it do not have to run CI again.
**Row text and counts**
- GitHub: when every repository fails for different reasons, the row
names each kind of failure (#16201).
- Jira: counts of one use the singular. The four local `_counted`
helpers (GitHub, HubSpot, SharePoint, Slack) are now one shared
`counted()` in `onyx/connectors/models.py` (#16201).
- Linear: the sign-in summary stays within 40 characters (#16201).
- SharePoint: the library row shows "First N of M" only when the probe
cap cut the list, not when a site returned 404 (#16202).
- OneDrive: locked users (423) are reported apart from missing users. If
every user is locked, the row is Incomplete (#16203).
- Outlook: a group name that matches several groups reads as ambiguous,
not as "No group matches" (#16204).
**Outcomes**
- SharePoint: a certificate whose end date is past fails its row
(#16202).
- Outlook: the mail and calendar samples depend on the selected
mailboxes and groups rows. One unreachable mailbox now fails one row,
not three (#16204).
- Generic row: the HTTP status and the Error Details follow the same
exception chain (#16206).
**Config formats**
- Confluence page IDs take ASCII digits only (#16205).
- Notion page URLs must be on a Notion host, not a host that contains
one (#16212).
- SharePoint site URLs are checked for `/sites/`, `/teams/` or
`/personal/` in the path only (#16212).
**Draft runs and stored runs**
- A skipped, stopped or failed dependency makes a dependent's reused
pass show "Not run". This honours both `depends_on` and `depends_on_any`
(#16209).
- A Re-run does not keep a stale skipped discovery check (#16209).
- A stored run in progress shows its two-step rows from its progress,
and open checks keep their `row_id` and `step` (#16210).
**Tests**
- The wave test fails when the gateway check does not run beside the
connector checks (#16259).
## How Has This Been Tested?
- Each fix has a focused unit test. Most of the new tests fail without
their fix.
- Unit tests: `tests/unit/onyx/connectors` and `tests/unit/onyx/server`,
7269 passed.
- External dependency unit tests:
`tests/external_dependency_unit/connectors/capability_checks`, 95
passed.
- ruff, ty and pre-commit pass.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Applies review follow-ups for the connector checks stack, focused on row
text accuracy, check outcomes, config validation, and draft run
dependency handling.
**Row text and outcomes**
- GitHub names each failure kind when every repository fails for
different reasons.
- Jira uses singular count forms via a shared `counted()` helper in
`onyx/connectors/models.py`.
- Linear falls back to a short sign-in line when the email is too long.
- OneDrive reports locked users (423) separately from missing ones; all
locked stays Incomplete.
- Outlook names a group matching several entries as ambiguous, and its
samples run after the selected-scope checks; when nothing is selected
those checks do not apply and the samples still run.
- SharePoint shows the probe cap only when it cut the list, expired
certificates fail their row, and `_gateway()` reads its cached Graph
gateway once so a concurrent REST rebuild cannot return None.
- The generic row's HTTP status and Error Details now follow the same
exception chain.
**Draft runs, config validation, and tests**
- A reused pass shows "Not run" when a dependency is skipped, stopped,
or failed, honoring `depends_on` and `depends_on_any` once every member
has a result.
- Re-runs don't keep stale skipped discovery checks; stored runs in
flight show rows from their progress.
- Confluence page IDs take ASCII digits only, Notion URLs must be on a
Notion host, and SharePoint site markers are checked in the URL path
only.
- The wave test fails when the gateway check doesn't run alongside the
connector checks.
<sup>Written for commit
|
||
|
Evan LohnandClaude Opus 5.5
|
6bc0483325 |
chore: one row-text model for every capability check (#16266)
## Description
One row-text model and one summary-cut helper for every capability
check. No behavior changes.
- **`CheckRowText`** moves from `capability_checks/models.py` to
`onyx/connectors/models.py`, next to the `summary` and `description`
that `ValidationError` carries in `connectors/exceptions.py`. Every
caller imports it from there.
- **`MicrosoftRowText` is deleted.** It duplicated `CheckRowText`.
`microsoft_utils` could not import the checks package
(`test_microsoft_utils_import_direction.py`, unchanged). Now both share
the lower-level model.
- **Microsoft check text moves out of `microsoft_utils`.** Text that
only checks use moves to the new
`capability_checks/microsoft_check_text.py`: `sign_in`, `sign_in_text`,
`grant_line`, `raise_for_sign_in_error`, the sign-in rows and
`ENTRA_GROUPS_DENIED_ROW`. `MicrosoftSignIn` moves to
`capability_checks/models.py`. Everything `graph_errors.py` maps for the
Outlook connector's own validation stays in `microsoft_utils`.
- **One summary-cut helper.** `fit_summary` and
`MAX_ROW_SUMMARY_LENGTH`, next to `CheckRowText`, replace the copies in
`fit_line`, SharePoint, OneDrive and Outlook (`_line`), and Linear
(`_MAX_SUMMARY_CHARS`). A generic `shorten(text, limit)` in
`onyx/utils/text_processing.py` replaces the findings and advisory cuts.
- **Two helpers become public:** `raise_with_row` and
`is_transport_failure`, which the moved sign-in mapper reuses instead of
copying.
## How Has This Been Tested?
- Connector, server, EE and background unit suites: 5075 passed.
- `ty` is clean.
- The import-direction test and the 40-character row test pass. The row
test now also covers `microsoft_check_text`.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Consolidates the shared capability-check pieces into one home each: a
single row-text model, one summary-cut helper, and
`capability_checks/models.py` as the home of the check's Pydantic
models. No behavior changes.
- `CheckRowText` and `MAX_ROW_SUMMARY_LENGTH` move to
`onyx/connectors/models.py`; `MicrosoftRowText` is deleted, and
`microsoft_check_text.py` holds the check-only Microsoft text so
`microsoft_utils` still never imports check code.
- The draft-run, runner, and stored-run-progress models (state, plan,
cached result, readiness, outcome, progress) move into
`capability_checks/models.py`.
- A shared `shorten` in `text_processing.py` replaces the per-module row
and advisory truncation helpers.
<sup>Written for commit
|
||
|
Evan LohnandClaude Opus 5.5
|
d63481807a |
fix: a Google token refresh with no answer is not a rejection (#16265)
## Description
A Google OAuth or service-account token refresh that gets **no answer**
from Google is no longer read as a rejected credential.
- `google_auth.py` has a new `GoogleUnreachableError(ConnectionError)`.
Transport errors, connection errors and timeouts during the refresh
raise it, with the original error as its cause. A refresh that Google
rejects (`RefreshError`, e.g. `invalid_grant`) still returns None, then
`PermissionError`, as before.
- **Drive:** sign-in now reads "Cannot reach Google Drive" or "Sign-in
timed out". That is FAILED on a form run and INCOMPLETE on a stored run,
through the handler from #16255. Drive needed no other change.
- **Gmail** (generic row): "Cannot reach the source", where it used to
read "Unexpected response".
- **`GET /admin/connector/google-drive/check-auth/{id}`:** no answer
from Google returns `OnyxError(BAD_GATEWAY)`, not `authenticated=false`.
A rejected token still returns `authenticated=false`.
**For review:** after google-auth's own retries, a 5xx from Google's
token endpoint arrives as `RefreshError(retryable=True)` and still reads
as rejected.
## How Has This Been Tested?
New `test_google_token_refresh_errors.py`. It covers:
- a refresh with no answer;
- a timeout that stays detectable;
- a rejected refresh;
- the service-account path;
- Gmail's row;
- check-auth's 502 and `authenticated=false`.
Also Drive sign-in row tests and the gateway pass-through test. The
google_utils, google_drive and gmail unit tests pass.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
A Google token refresh that gets no answer from Google, or only a server
error, is no longer treated as a rejected credential; it now surfaces as
an unreachable error. A refresh Google rejects still behaves as before.
**What changed**
- A refresh with no answer or a token-endpoint 5xx (which google-auth
surfaces as a retryable `RefreshError`) now raises
`GoogleUnreachableError`; a rejected refresh still returns `None` and
later raises `PermissionError`.
- Drive sign-in reports "Cannot reach Google Drive" or "Sign-in timed
out" (FAILED on a form run, INCOMPLETE on a stored run); Gmail's generic
row reports "Cannot reach the source" instead of "Unexpected response".
- `GET /admin/connector/google-drive/check-auth/{id}` returns
`OnyxError(BAD_GATEWAY)` on no answer, not `authenticated=false`.
<sup>Written for commit
|
||
|
Evan LohnandClaude Opus 5.5
|
e5dea41986 |
feat: capability checks run in dependency waves (#16259)
## Description Capability checks now run in **dependency waves**, not one at a time. Each check starts as soon as all its dependencies have results and a slot is free, up to `MAX_PARALLEL_CHECKS = 4` per run. Runs take about as long as the depth of the chain (sign-in, then lists, then samples), not the sum of every check. Kept: - **"Not run" and rate-limit stops:** "Not run" outcomes stay. A RATE_LIMITED result stops new starts, and checks already running finish. - **Inputs and hang guards:** `prior_results` stays, and each check keeps its own hang guard. The coordinator's wait is bounded too. - **Ordering:** discovery checks still run last. The returned report keeps run order. - **Callbacks:** `on_result` runs on the calling thread, one call at a time, and gets the results in the order they finished. - **Threads:** workers come from `start_thread_future`, so contextvars (the tenant id) carry over. - **Draft runs:** the lease now renews for the longest hang guard among checks that can run at once. **Thread safety:** - This PR runs on top of #16264, which makes every source-operations gateway thread safe and documents that as a contract. - Connectors are not thread safe. So checks that need the connector instance (the fallback checks of sources without a gateway) run one at a time, while gateway checks run beside them. **For review:** - Progress still marks one check as RUNNING, while up to 4 may run. - Tests that assert execution order or the rate-limit stop of the next check use a new `one_check_at_a_time` fixture. - The draft-run fakes in the external-dependency tests now run after sign-in, as real checks do. Without that, cancel and supersede could not stop them. ## How Has This Been Tested? - New `test_check_waves.py`, built on barriers and events with no sleeps. It covers: - checks that call the connector never overlapping; - the parallel start; - the cap; - the rate-limit stop while a check is running; - report order versus completion order; - dependency stops in parallel; - discovery last; - the tenant context. - A stored-progress test for results that arrive out of order. - Unit suites pass at this tip. - Capability-check external-dependency tests: 94 passed, three runs of the draft-run file. - Capability-check integration tests against a local stack at this tip: 20 passed, 4 skipped. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. - [x] [Optional] Override Linear Check 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **New Features** - Capability checks now run concurrently, with up to four checks active at once. - Checks respect dependencies, ensuring prerequisites complete first. - Connector-based checks remain safely serialized when required. - Progress updates reflect checks as they finish. - Discovery checks begin only after validation checks complete. - **Bug Fixes** - Rate-limited runs stop starting new checks while allowing active checks to finish. - Checks that have not started are skipped after throttling. - Runs remain active long enough for unfinished checks to complete safely. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
37e8266c58 |
chore: source operations gateways are thread safe (#16264)
## Description
Source-operation gateways (`SourceOperations` subclasses) are now
**thread safe**: one gateway instance may be called from several threads
at once. The capability check runner relies on this, because the next PR
(#16259) runs a run's checks in parallel on one shared gateway. The
contract is in the `SourceOperations` docstring and in the connectors
feature map, with three patterns: lock lazy client builds, use one
client per thread for SDKs that are not thread safe, and refresh tokens
under a lock.
Audit of every gateway:
| Gateway | Finding | Change |
|---|---|---|
| GitHub | PyGithub keeps each request's URL and headers on one shared
connection object, so two threads could send each other's request. | One
`Github` client per thread. |
| SharePoint | The office365 `ClientContext` queues pending queries, and
its cache could hand a thread the context of another site. | One REST
context per thread. |
| Confluence | `probe_site` changed the shared client's timeout while
other threads used it. | Probe on a new client, then cache it. |
| HubSpot | Client and rate limiter were built without a lock. The
shared `rate_limit_builder` was not thread safe: once parallel calls
pushed the count past the limit, its `==` check never triggered again. |
Locked builds. `rate_limit_builder` locks its call history and uses
`>=`. This also fixes Drupal Wiki, ClickUp, Braintrust and Celery, which
share the helper. |
| Linear | With a static credentials provider, two threads could use the
same OAuth refresh token. | Refresh under a gateway lock. |
| Google Drive, Jira, Slack, Notion, YouTrack, OneDrive, Outlook,
Microsoft Graph | Already safe: locked builds, per-thread Google
clients, thread-safe SDK clients, or a harmless duplicate build. | None.
|
## How Has This Been Tested?
- One race test per fixed gateway, using threads and barriers. Each test
fails on the old code, except GitHub's, which uses the new per-thread
client store.
- `tests/unit/onyx/connectors` and
`tests/unit/ee/onyx/external_permissions`: 4855 passed.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Makes `SourceOperations` gateways safe to call from several threads at
once, so the capability check runner can share one gateway across
parallel checks. The contract now lives in the `SourceOperations`
docstring and the connectors feature map.
- GitHub keeps one `PyGithub` client per thread, since a shared
connection object could send each thread's request over another thread's
connection.
- SharePoint keeps one office365 REST context per thread, since a
context queues its pending queries on itself.
- Confluence `probe_site` probes on a new client, leaving the cached
client's connection settings untouched for other threads.
- HubSpot builds its client and rate limiter under a double-checked
lock; `rate_limit_builder` locks its call history and sleeps while `>=
max_calls`, which also fixes Drupal Wiki, ClickUp, Braintrust, and
Celery.
- Linear refreshes an OAuth token under a gateway lock, covering static
credential providers whose rotation lock is a no-op.
- Added one race test per fixed gateway plus one for the rate limiter;
all 4855 unit tests in the connector and external-permission suites
pass.
<sup>Written for commit
|
||
|
Evan LohnandClaude Opus 5.5
|
3de783d1a8 |
feat: checks can run after any one of several checks (#16258)
## Description
Checks can run after **any one** of several checks.
- `CapabilityCheck.depends_on` stays all-of. A new optional
`depends_on_any` holds one group of alternatives.
- **Runs when:** at least one member PASSED, or finished as a partial
result.
- **Ignored members:** members that do not apply, have no result, or did
not run because their own config is missing.
- **Stops when:** no member passed and at least one counted member
failed, is Incomplete, or was stopped by a gate. The not-run line names
the first such member in the order the check lists them.
- `depends_on` is evaluated first.
- Draft runs make a check wait while every applicable member waits.
Applied to:
- **HubSpot:** "List linked records" and "Sample notes" run after any
one of the selected object-type reads, not after sign-in only.
- **Notion:** "Sample page content" runs after "Find shared pages" or
"Find root page". "Find root page" again shows the design's "waiting for
settings" row on a run without a config. Before, it had no row, because
the old rule let it block page content.
## How Has This Been Tested?
- Runner unit tests:
- one member passes;
- a member is partial;
- all members fail;
- a member does not apply or needs config;
- a stopped member passes its cause on;
- all-of is checked before any-of;
- run order and draft waiting.
- HubSpot and Notion cases.
- The connector, server, EE and background unit suites pass at this tip.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Introduces `depends_on_any`, a new optional dependency type that lets a
check run after any one of several checks. A check runs when at least
one listed member passes or returns a partial result, and stops when no
member passed and at least one counted member failed, is incomplete, or
was stopped by a gate—the first such member in declaration order names
the cause. Members that do not apply, have no result, or were skipped
for lacking their own config or instance are left out, and `depends_on`
is still evaluated first.
**Behavior changes**
- HubSpot: "List linked records" and "Sample notes" run after any
selected object-type read passes instead of after sign-in only.
- Notion: "Sample page content" runs after "Find shared pages" or "Find
root page"; "Find root page" now shows the waiting-for-settings row on
runs without a config, where before it had no row.
- Draft runs make a check wait while every applicable member of its
any-of group waits.
<sup>Written for commit
|
||
|
Raunak Bhagat
|
31461f9c8c |
feat(admin): reorganise chat preferences and language models settings (#16435)
## Description
Reorganise the Chat Preferences and Language Models admin pages, tighten
their copy, and give their model selects one loading and failure
pattern. This PR also carries the former #3 (model load states and
hidden models), which was folded in. Each UI change is one bullet.
### Chat Preferences
- Split the settings into two cards: a **Chat** card and a **Search**
card.
- Moved **Default Language** to the top of the Chat card, as a
horizontal row.
- Chat card order: Default Language, Chat Naming Model, Multi-Model
Generation, Deep Research, Chat Auto-Scroll.
- Search card order: Search Selection Model, Search Mode, Auto-Detect
Search Filters.
- Removed the "Model selection" heading; the two task-model rows now sit
in the cards.
- Renamed "Chat naming model" to **Chat Naming Model**, and "Search
selection model" to **Search Selection Model**, with new copy.
- Removed the **Reset** buttons and the **(Optional)** suffix from both
task-model rows.
- Added an unset row at the top of both task-model selects, above a
divider; picking it clears the setting:
- **Chat Naming Model:** "Chat Default". Naming only runs in chats, on
the chat's own model.
- **Search Selection Model:** "Global Default", described as "Each
chat's model". Selection also runs in the Search UI, where the unset
fallback is the workspace default model.
- Clicking a task-model row's title opens its select.
- Removed the **beta** tag from Multi-Model Generation.
- Renamed "Disable Default Chat" to **Always Start with an Agent**, with
new copy and a link to featured agents.
- Made "without logging in" bold in the Allow Anonymous Users
description.
- Moved **Temperature Control** and **Reasoning Control** to the
Language Models page.
- New copy for Default Language and Search Selection Model.
### Language Models
- Card order: Default Model, Auto Mode, Router Model, a divider,
Reasoning Control, Temperature Control, Hide Provider Grouping.
- New copy for Default Model, Reasoning Control and Temperature Control.
- **Auto Mode** stays locked until the router list has loaded with at
least one router, with a "No router models found" tooltip when there are
none. Turning it off is always possible.
- The **Router Model** row shows only when a router model exists, and
its select can be cleared.
- With Auto Mode on and no usable router (none chosen, cleared, or
deleted since), an inline error under the Router Model row reads "Choose
a router model to enable users to use the auto routing mode." With no
router models at all, the error sits under Auto Mode instead.
### Settings (personal)
- **Display Language** now shares one component with the workspace
Default Language. It starts with a **Global Default** row (the workspace
language as its description) above a divider; picking it clears the
override.
### Model selects (task models and Router Model)
- **Loading:** a `DropdownLoader` stands in for the select.
- **Load failure:** the select stays, disabled with the error border,
and a persistent error toast offers a Retry button. The two task-model
rows share one toast.
- **Hidden models** are never offered, except a hidden model already in
use.
### Across these pages
- Removed **success toasts** from Chat Preferences, Language Models
(including cost overrides), Settings and Decision Models' provider
delete. Failures still show an error toast. e2e helpers wait for the
save response instead of the toast.
### Code changes behind the UI
- `LanguageSelect` (`lib/app/components.tsx`) serves both language
pickers.
- `filterModelConfigurations` loses its `visibleOnly` option; every
caller now lists visible models only (plus `keep`).
- `SimpleModelSelector`: its Global Default row takes a `label`, always
shows a divider (also in an ungrouped list), and the select takes
`isError`.
- Opal `InputHorizontal` and `InputVertical` take `error` and `warning`
props for inputs outside Formik.
## Changes
### Language Models
No "router model" configured, "Auto Mode" disabled:
<img width="872" height="452" alt="image"
src="https://github.com/user-attachments/assets/3894cb0f-9805-4909-99e9-f2bfe3ed92ab"
/>
"Auto Mode" on and no "Router model" configured:
<img width="877" height="526" alt="image"
src="https://github.com/user-attachments/assets/2ba22737-ddc7-4836-9ed8-83d2106aa54f"
/>
"Auto Mode" on and "Router model" configured:
<img width="863" height="506" alt="image"
src="https://github.com/user-attachments/assets/748db170-376b-4955-8d73-0c5a064d8a2b"
/>
### Chat Preferences
A few things have just been moved around here. The "beta" tag was also
removed from "Multi-Model Generation".
<img width="877" height="688" alt="image"
src="https://github.com/user-attachments/assets/e21336a8-4667-4110-a11e-e98f0488e0ad"
/>
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Reorganizes the admin Chat Preferences and Language Models pages and
tightens their copy. Chat Preferences now splits into Chat and Search
cards, Temperature Control and Reasoning Control move to Language
Models, and the task-model rows are renamed to Chat Naming Model and
Search Selection Model.
**Behavior changes**
- Task-model selectors start with a Global Default row that clears the
designation; chat naming reads Chat Default and search selection keys
off the workspace default. Reset buttons and the Optional suffix are
gone.
- Auto Mode stays locked until router models load and can only be
enabled when a router exists; it can always be turned off. The Router
Model row only shows when a router exists, and its select is clearable
with an inline error when Auto Mode needs a router.
- Failed model loads keep the select, disabled with an error border, and
raise a persistent error toast with a retry button; loading shows a
dropdown-loader shimmer.
- "Disable Default Chat" is renamed to "Always Start with an Agent" with
new copy and a link to featured agents.
- Success toasts are removed from these pages, including cost overrides
and Settings, and Decision Models no longer toasts provider deletion;
failures still show an error toast. e2e helpers wait for the save
response instead.
- `LanguageSelect` is shared between the workspace Default Language and
the personal Display Language; the personal picker offers a Global
Default row.
- Model pickers no longer list hidden models; they show visible models
only, plus the currently selected value.
<sup>Written for commit
|
||
|
Nikolas Garza
|
5b69702c7e |
fix(web): indexing status and connector page fixes from the v5 bug bash (#16471)
## Description Fixes on the indexing status page and the connector page from the v5 bug bash. Stack 3 of 3, on #16469. Fixes [ENG-4400](https://linear.app/onyx-app/issue/ENG-4400/status-badges-skip-scheduled-and-count-a-deleting-connector-as-failed) Fixes [ENG-4439](https://linear.app/onyx-app/issue/ENG-4439/recently-indexed-adds-up-docs-across-runs-and-the-status-page-numbers) Fixes [ENG-4440](https://linear.app/onyx-app/issue/ENG-4440/status-page-search-returns-no-attempts-and-the-page-icon-beside-it) Fixes [ENG-4398](https://linear.app/onyx-app/issue/ENG-4398/charts-use-utc-while-runs-use-local-time-and-the-throughput-chart) Fixes [ENG-4399](https://linear.app/onyx-app/issue/ENG-4399/manage-index-has-no-perm-sync-trigger-or-re-index-on-the-detail-page) **Why** - Status badges skipped Scheduled, a deleting connector counted as failed, and the doc figures added up every run with no labels. - Charts read in the server's UTC while runs read local time, a clicked bar drew nothing, Delete required a pause, and permission syncs had no trigger. **What** - A `deleting` bucket end to end. Every nonzero bucket gets a badge, and clicking one filters the list. - Documents held is the main figure on the overview card and each row, with processed counts labeled and explained on hover. - Chart labels, tooltip and span pill format in the browser's time zone. A picked slice keeps one neighbor each side so its bar draws. - Manage Index: Delete works on an active pair behind the confirm, and perm-synced pairs whose source syncs document permissions get a Sync Permissions row, disabled while one is queued or running. - The row search reads "Search run errors…", and the history panel scrolls into view below the 1232px breakpoint. ## How Has This Been Tested? Jest for the bucket tags, chart slices and time zone, the sync helper and the panel reveal, plus a backend test for the deleting bucket. Live in Chrome on the whole stack against a copy of a dev database: the Deleting badge filtered to the one deleting pair and Failed no longer counted it, chart labels read Pacific time, one clicked bar drew with its span pill, Sync Permissions toasted and then showed disabled as queued, and Delete was enabled on an active pair. Not live: the narrow-window scroll, because the browser window would not resize. The moving View Logs icon is unchanged, since keeping it in place needs a layout change. The web servers ran with `TZ=UTC`, as in a container. | Before | After | |---|---| |  |  | |  |  | |  |  | ## Additional Options - [x] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
Nikolas Garza
|
480840700a |
fix(indexing): record the real docfetching failure cause and cancel requested stops (#16469)
## Description A docfetching failure before the run's main loop records its real message, and a requested stop before the run starts reads as canceled. Stack 2 of 3, on #16467. Fixes [ENG-4396](https://linear.app/onyx-app/issue/ENG-4396/failure-summaries-hide-the-real-cause) **Why** - A failure in `validate_connector_settings`, such as an HTTP 500 from the site, skipped the in-process failure mark, so the run read "Spawned task exceptioned: exit_code=255" and the cause sat on the last line of the trace. - Exit 249 is a stop requested before the run started, but it was recorded as a failure, which drew red marks on the throughput charts. **What** - `mark_attempt_failed_unless_terminal` records `str(e)` and the trace, or the exception type when the message is empty. Both except branches of `_docfetching_task` and the in-loop branch call it. - The watchdog records exit 249 as canceled only when the attempt had a cancel request or its pair is paused or deleting. A stop fence left by a failed pause still fails, so repeated-error tracking sees it. - Both marks check for a terminal status under the row lock, so a watchdog INTERRUPTED write is never overwritten. - The dead "truncate long exception messages" block is gone. Its inner except always re-raised the original. ## How Has This Been Tested? External-dependency tests run the real watchdog, which spawns docfetching in a new process, on a throwaway Postgres and Redis: a local site answering 500 now fails the attempt with "Unexpected error validating ...: Internal Server Error (500)", a cancel request before start ends CANCELED, a stale stop fence on an active pair ends FAILED, and an INTERRUPTED attempt stays INTERRUPTED when a late failure or cancel is recorded. The first two failed on main. Not run against live Celery workers. ## Additional Options - [x] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
Nikolas Garza
|
aa6e8b3cfb |
fix(indexing): resolved and superseded runs close their errors, and a resolve clears the repeated-error flag (#16467)
## Description Resolved and superseded runs stop counting as open errors, and resolving a pair's latest failure clears its repeated-error flag. Stack 1 of 3. Fixes [ENG-4436](https://linear.app/onyx-app/issue/ENG-4436/retry-and-resolve-all-shown-write-two-different-resolved-flags) Fixes [ENG-4395](https://linear.app/onyx-app/issue/ENG-4395/errors-never-supersede-and-stay-open-forever) Fixes [ENG-4438](https://linear.app/onyx-app/issue/ENG-4438/failed-repeatedly-after-a-single-failure-and-the-warning-stays-after) **Why** - "Resolve all shown" wrote only `cc_pair_run_resolution`, while the open doc error count read only `is_resolved`, so a resolved run still said "N docs failed". - Perm-sync and prune error runs never closed, a superseded run looked the same as a manual resolve, and the "failed repeatedly" flag and alert survived a resolve. **What** - A doc error is open only when `is_resolved` is false and its run has no resolution. `is_resolved` is never bulk-written, since the poll cursor and Retry read it. - Full-pass kinds (permission sync, prune, group sync, checks) let a later succeeded or failed run supersede earlier error runs. Indexing keeps today's rule. A "Superseded" chip shows apart from "Resolved". - Resolving the latest failed attempt clears `in_repeated_error_state` and its alert. The beat heals a resolve that races it. Retry backoff is unchanged, and a Cloud auto-pause stays. - The status page row badge counts open errors, entity failures included. Threshold-1 alerts say "failed", not "failed repeatedly". ## How Has This Been Tested? External-dependency tests on a throwaway Postgres and Redis for supersede rules per kind, resolve clearing the flag and alert, the beat heal after a racing resolve, retry backoff after a resolve, and the badge with an entity-only failure, plus unit tests for the threshold copy. Each new test failed before its fix. Live in Chrome on the whole stack against a copy of a dev database: "Resolve all shown" turned "1 doc failed" into "1 doc resolved", resolving a flagged pair cleared the flag and its 725 alerts, and a completed-with-errors perm sync showed "Superseded" once a clean sync followed. The window query's plan was checked with EXPLAIN on synthetic data by a reviewer, not on a real tenant. | Before | After | |---|---| |  |  | |  |  | ## Additional Options - [x] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
Nikolas Garza
|
1541e3e593 |
fix(web): lock the SCIM and service-account pages without a license (#16470)
## Description `/admin/scim` and `/admin/service-accounts` now redirect with "This feature requires a license" when the license does not cover EE features, like the other paid admin pages. Fixes [ENG-4442](https://linear.app/onyx-app/issue/ENG-4442/scim-and-service-account-pages-are-not-locked-after-the-community). **Why** - Paid pages are locked by `web/src/proxy.ts:EE_ROUTES`, which rewrites them to the `/ee` tree whose layout turns an unlicensed deployment away. - These two paths were never in that list, so after a downgrade SCIM offered a token button that returns 402, and service accounts showed a load error. **What** - Both paths are added to `EE_ROUTES`. - Like `/admin/groups`, each has a one-line page stub in `app/admin/` (keeps typed `Link` hrefs valid) and `app/ee/admin/` (the copy the proxy serves). - The SCIM page body moves to `views/admin/ScimPage/`, with `interfaces.ts` renamed `types.ts` and imports made absolute. - `src/app/admin` type-coverage floor drops from 97.3 to 97.2 because typed code left it. Feature-map docs list the new EE routes. | Before | After | | --- | --- | |  |  | |  |  | ## How Has This Been Tested? Live in Chrome against a scratch API with no license (`ee_features_enabled` false, tier community): on main both pages load as above. On this branch both redirect to `/app` with the license toast, the same as `/admin/groups`. With a license seeded, both pages still load. Jest (`src/app/ee/__tests__/licenseGatedRoutes.test.ts`): the proxy rewrites both paths to `/ee`, and both cases fail against main's `proxy.ts`. `types:check` and pre-commit pass. ## Additional Options - [x] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
Raunak Bhagat
|
f47f48e052 |
feat(opal): error toast actions, DropdownLoader and dropdown fixes (#16432)
## Description
Opal changes for the admin settings stack. Nothing in the app uses the
new pieces until the third PR in the stack (model load states).
- **Fixed:** a dropdown that is disabled while it saves no longer
flashes open when it is re-enabled. Disabling an open dropdown now
closes it, and its list always plays its close animation.
- **`LineLoader` takes a `height`:** each bar sits centred in a line of
that height, so the shimmer lines up with the text beside it. `height`
is a new `LineHeight` type, a closed Spacing step matching the
`--height-line-*` tokens (default 5, 20px). The Plugins pages' two-line
loaders become about 8px taller.
- **Toasts take an `id`:** a toast with the id of a live one replaces it
in place.
- **Toasts take an `action`:** an icon button left of the close button,
with its label as the tooltip. A click runs it, and the toast stays
open.
- **New `useErrorToast` hook:** shows a persistent error toast, with an
optional Retry button, while a fetch is failing. A retry that fails
again updates the toast in place, so it does not flash or animate in
again; it is removed when the error clears.
- **New `DropdownLoader`:** a select that has not loaded, drawn as the
input's shell with a shimmering icon and label and a static chevron.
- **Disabled and invalid inputs:** a disabled input with an invalid
field shows the error border.
## Changes
### Additional States
- error state
- disabled state
- error + disabled state
<img width="725" height="245" alt="image"
src="https://github.com/user-attachments/assets/2600f053-6b78-46d1-9376-abb022e3b024"
/>
### Loader edits
- Added dropdown-loaders
- Added variable line-heights to line-loaders
https://github.com/user-attachments/assets/6687b15e-c3ff-495b-bc41-f85d8a87d418
### Disabled while loading
https://github.com/user-attachments/assets/3eda28e3-b8a6-43cf-b3f7-e5b12dda7d55
## Summary by cubic
Reorganizes the Chat Preferences and Language Models admin pages and
standardizes how their settings behave. Chat Preferences is now split
into Chat and Search cards, Temperature and Reasoning Control move to
Language Models, and "Disable Default Chat" becomes "Always Start with
an Agent."
**Behavior changes**
- Model selects (Chat Naming, Search Selection, personal language) start
with a Global Default row naming the fallback; picking it clears the
override.
- Model pickers never list hidden models, except the current value.
- Auto Mode is disabled, with a tooltip, when no router model exists;
the Router Model row hides then.
- Success toasts are removed from these pages and Settings; failures
still toast, and model-load failures show a persistent error toast with
a retry button while the select stays, disabled with an error border.
- e2e helpers wait for the save response instead of the toast.
**Opal**
- Toasts accept an `id` (replacing a live toast) and an `action` button;
`useErrorToast` raises one for a failed fetch.
- Adds `DropdownLoader`, and `LineLoader` bars now sit centered in a
configurable `LineHeight`.
- Fixes a disabled dropdown flashing open on re-enable.
<sup>Written for commit
|
||
|
roshan
|
ae4ca216e9 |
feat(sidebar): tag Decision Models and Craft as Beta (#16440)
## Description Adds a small blue **Beta** tag to Decision Models and Craft: - **Admin sidebar, Decision Models:** the tag sits flush right on the row (`SidebarTab` `rightChildren`). - **Admin sidebar, Craft:** the tag sits flush right on the Craft section heading, in the same column as the Decision Models tag. Access and Preferences do not each get a tag. - **App sidebar (`/app`), Craft row:** shows Beta when Craft is available. The upsell state still shows only Business, so the row never shows two tags. Changes: - `AdminRouteEntry` gets an optional `beta` flag. `DECISION_MODELS` sets it, and `buildItems` copies it onto `SidebarItemEntry`. - `BETA_SECTION_IDS` in `admin-sidebar-utils.ts` lists the sections with a Beta heading (`craft`). - Opal `SidebarLayouts.Section` gets an optional `tag?: TagProps`, rendered at the end of the header (`.opal-sidebar-section__tag`), aligned with tab `rightChildren`. - Opal `SidebarTab`: `.opal-sidebar-tab__actions` no longer catches clicks itself. Its direct children still do, except a passive `Tag` (no `onClick`/`onRemove`), so a click on a passive tag opens the row link. - New strings `sidebar.adminSidebar.betaTag.label` and `sidebar.appSidebar.craft.betaTag` in all 9 locales. Stacked on #16284. ## How Has This Been Tested? - Jest: `admin-sidebar-utils.test.ts` checks that only Decision Models is a beta entry; i18n catalog tests pass. - Pre-commit: type check and coverage, oxfmt, oxlint, i18n copy check. - UI at `898de231a3`, desktop 1440×1000 and mobile 390×844: - The admin Decision Models and Craft heading tags have the same right edge (216 px) and size (32×16 px) on both viewports. - A click on the center of the Decision Models Beta tag opens `/admin/decision-models`. A click on the `/app` Craft Beta tag opens the Craft link (`GET /craft/v1`). - `/app` shows only Beta when Craft is available, and only Business in the upsell state. - Regression: the chat-session `…` button in `rightChildren` still opens its menu and does not navigate. - Craft availability and tier were mocked locally in the browser settings response, because the test workspace is Community tier. Real Craft entitlement and the licensed Craft page were not tested. Left: desktop. Right: mobile (390×844). **Admin sidebar**  **`/app` Craft row** | Craft available | Upgrade required | |---|---| |  |  | **Tag clicks and chat-menu regression**   ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. Link to Devin session: https://app.devin.ai/sessions/39d77bde91c34b75922bb05c02b93b90 Open in Devin Desktop: https://app.devin.ai/desktop/session/39d77bde91c34b75922bb05c02b93b90?variant=devin Requested by: @rohoswagger <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **New Features** - Added Beta labels to relevant admin sidebar sections and items, including Decision Models. - Added Beta labeling to the Craft sidebar entry alongside its existing Business label. - Added support for optional tags in sidebar section headers. - Localized new Beta labels across supported languages. - **Bug Fixes** - Sidebar tags no longer block clicks on their underlying navigation items. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
roshan
|
ab944e79ac |
feat(llm): add Vercel AI Gateway decision support (#16284)
## Summary Expose Vercel AI Gateway decision models on **Admin > Decision Models** using the existing Gateway connection. Treat catalog entries with type `evaluation` as decision-only models. Verify their capability during save and connection testing. Send typed requests to `/v1/decisions`, including when a custom API base omits `/v1`. Language models retain their chat behavior. Decision-model selection remains opt-in and unset by default. No product workflow consumes the selected model yet. ## Validation - 35 focused Gateway and decision backend tests passed during implementation. - After the custom-base URL fix, all 29 decision tests passed. Regression tests inspect outgoing HTTP URLs for versioned, unversioned, and trailing-slash bases. - Nine shared-setup frontend tests, type checks, lint, and touched-file hooks passed. - The live Gateway catalog reports decision models as `evaluation`. - Current-head browser inspection confirmed the provider card and setup form. No Gateway key was entered, so live credential validation and a decision request were not run. ## Screenshots Provider page at `2ffd035`, with selection unset.  Gateway setup with empty credentials. Model discovery requires a key.  ## Stack testing (`29eec61fb4`) Tested the full stack at the top commit on a local dev stack, desktop and 390×844 mobile. OpenAI decisions and chat used a real key. TypeSafe, Zoom, OpenRouter, and Vercel AI Gateway used local mock upstreams; Onyx auth, validation, persistence, encryption, and UI were real. - Passed: Decision Models lists only `evaluation` catalog entries; save/test reject chat-only Gateway models. - Passed: an unversioned base (`.../gateway`) sends `/gateway/v1/models` and `/gateway/v1/decisions`. - Passed: Gateway refresh and chat auto-mode do not add decision selections; selected-model guard works. - Passed (regression): Gateway chat models stay selectable and complete a chat; evaluation models are absent from the chat picker. - Fixed in #16276 (`37c248ec04`): the configured heading no longer shows literal asterisks (retested at `78cf4ccb12`).     ## Stack PR 4 of 4: #16262 → #16276 → #16280 → **#16284**. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added Vercel AI Gateway as a supported decision-model provider. * Gateway discovery now distinguishes chat-capable and decision-capable models, including decision-only catalogs. * Added automatic decision endpoint handling for custom Gateway base URLs. * Decision model configuration now lists Vercel AI Gateway as an available provider. * **Improvements** * Model selection prioritizes compatible models based on context. * Provider configuration controls adapt to scoped model settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Link to Devin session: https://app.devin.ai/sessions/39d77bde91c34b75922bb05c02b93b90 Open in Devin Desktop: https://app.devin.ai/desktop/session/39d77bde91c34b75922bb05c02b93b90?variant=devin Requested by: @rohoswagger |
||
|
roshan
|
fcc0cae05a |
feat(llm): share Zoom setup across Voice and Decisions (#16280)
## Summary Add Zoom Decisions and Zoom Decisions Pro to **Admin > Decision Models**. Configure Zoom from either Voice or Decision Models. Both pages use one shared connection and one encrypted API key and secret. Zoom Decisions is opt-in: a Zoom decision provider exists only after an admin connects Zoom on Decision Models. A Voice-only setup never creates one. Decision setup does not activate Voice defaults or select a decision model. Read current credentials and sign a fresh JWT for each request. Keep Zoom decision models out of chat. Removing Zoom from Decision Models deletes only the decision provider and keeps Scribe. Disconnecting Zoom on Voice also removes its linked decision provider, after selected-model checks. The migration only adds the link column; it writes no rows. Both setup links open the [Zoom AI Services documentation](https://developers.zoom.us/docs/ai-services/). ## Validation - 336 focused backend tests and nine frontend component tests passed. - Type checks, lint, translations, deterministic migration checks, and touched-file hooks passed. - A live Zoom Decision API request returned HTTP 200 with a JWT signed using the existing Scribe credentials. - The current-head preview opens the credential form directly from Decision Models. No Voice setup is required first. - PostgreSQL tests cover opt-in linking from Decision Models only, rotation, rollback, relink and duplicate-link rejection, and deletion. ## Screenshot Shared Zoom setup from Decision Models at `9b7792d`. Credentials are empty. This screenshot demonstrates the form, not a completed connection test.  ## Stack testing (`29eec61fb4`) Tested the full stack at the top commit on a local dev stack, desktop and 390×844 mobile. OpenAI decisions and chat used a real key. TypeSafe, Zoom, OpenRouter, and Vercel AI Gateway used local mock upstreams; Onyx auth, validation, persistence, encryption, and UI were real. - Fixed during testing: migration `9071f636d5f2` now revises `4d8453213439` (#16351 on main). The old `down_revision` left two alembic heads and failed most CI jobs on this PR and #16284. - Passed: Voice-only Zoom setup creates no decision provider. Decision Models opens the shared credential form directly. - Passed: one encrypted Voice connection; the linked LLM provider stores no duplicate key. JWT validation and the Scribe WebSocket handshake worked against the mock. - Passed: bad credentials fail. Both setup links open the Zoom AI Services docs. - Passed: selection guards block Zoom removal and Voice disconnect. Removing Zoom from Decision Models keeps Voice/Scribe. Zoom decision models are absent from chat. - **Fixed in #16276 (`37c248ec04`):** after Zoom is disconnected on Voice, sidebar navigation to Decision Models now shows **Connect** without a reload (retested at `78cf4ccb12`). Before the fix, the card showed **Configure/Delete** from a stale `adminLlmProvidersPaged` SWR cache.      ## Stack PR 3 of 4: #16262 → #16276 → **#16280** → #16284. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Zoom AI Decisions with two models supporting text and image inputs. * Added shared Zoom AI Services setup across Voice and Decision Models, including connection selection, credential reuse, rotation, and removal. * Added provider discovery, model capability indicators, decision-specific validation, and localized interface text. * Renamed Zoom Scribe setup labels to Zoom AI Services. * **Bug Fixes** * Improved connection validation and rollback. * Improved credential handling and protection against deleting active shared connections. * Preserved existing shared connection settings when updating providers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Link to Devin session: https://app.devin.ai/sessions/39d77bde91c34b75922bb05c02b93b90 Open in Devin Desktop: https://app.devin.ai/desktop/session/39d77bde91c34b75922bb05c02b93b90?variant=devin Requested by: @rohoswagger |
||
|
Nikolas Garza
|
a0c949df51 |
fix(mcp): refuse OAuth connect when the provider rejects the client metadata document (#16460)
## Description The MCP OAuth modal now refuses the connect when the provider rejects Onyx's client metadata document. Before, the user landed on the provider's error page. Fixes [ENG-4412](https://linear.app/onyx-app/issue/ENG-4412/notion-mcp-connect-lands-on-a-bad-page-instead-of-refusing-the). **Why** - Auto-discovery uses CIMD whenever the provider supports it ([#13877](https://github.com/onyx-dot-app/onyx/pull/13877)). If the provider cannot fetch `WEB_DOMAIN/api/mcp/oauth/client-metadata` (private or IP-allowlisted Onyx, such as st-dev behind its WAF allowlist), `/authorize` answers with an error and the user is stuck there. - Onyx never sees that failure, and the switch from [#16227](https://github.com/onyx-dot-app/onyx/pull/16227) is easy to miss. **What** - Before it redirects with a CIMD client_id, the backend requests the authorization URL and reads only the status (SSRF-guarded, no redirects, 5s cap). A 4xx or 5xx raises `client_metadata_rejected`. Network errors fail open. - The check runs in both connect paths: server OAuth connect and the new staged setup (Add MCP). - `oauth_setup_error` passes nested `OnyxError`s through the SDK's exception groups. - The modal shows the error and a button that turns off "Use client metadata document". Copy is in all locales. - DCR and known-provider flows are unchanged. | Before | After | | --- | --- | |  |  | ## How Has This Been Tested? Unit tests cover a rejected (400), accepted (302) and unreachable check in both paths, and the grouped-error passthrough. A Jest test covers the button. Live: a local API with `WEB_DOMAIN=https://st-dev.onyx.app`, in Chrome. Editing a Notion server and adding Linear through Add MCP both showed the new error. The button turned the switch off, and Connect then redirected to the provider with a DCR client. Measured `/authorize` for an unreachable CIMD id: Notion 400, Linear 400, Sentry and Canva 500. Not yet tested on a deployed st-dev. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
Nikolas Garza
|
31ea4944b5 |
fix(helm): wait for the object store at startup and keep the MinIO PVC on retire (#16431)
## Description Fixes two Helm issues from the v5.0 bug bash: [ENG-4445](https://linear.app/onyx-app/issue/ENG-4445/helm-the-copy-job-and-api-server-crash-loop-at-startup-before) and [ENG-4447](https://linear.app/onyx-app/issue/ENG-4447/helm-retiring-minio-needs-a-manual-pvc-keep-annotation-and-leaves-pvcs). **Why** - The api server's startup bucket check and the legacy copy's `ensure_bucket` fail with `EndpointConnectionError` until SeaweedFS is ready, so on the 0.8 → 0.9 upgrade they crash-loop and the copy spends its `backoffLimit`. - An install upgraded with `--reuse-values` never gets `helm.sh/resource-policy: keep` on `<release>-minio`, so the upgrade that sets `minio.enabled: false` deletes the PVC and MinIO's data unless the admin annotates it by hand. **What** - A `wait-for-object-store` init container on the app image polls the object store's `/healthz` before the api server and the copy Job start. It takes the guarded container's resources and securityContext. External object storage gets none. - With MinIO disabled, `minio-retired-pvc.yaml` renders the existing MinIO claim from its live spec with `keep`, so Helm updates it instead of deleting it. MIGRATION.md drops the manual step and says how to delete the claim. - The copy Job's name hash changes, so a release still on MinIO reruns the copy once (idempotent). Chart 0.9.21. ## How Has This Been Tested? kind with Helm 4, starting from a MinIO-only install with no keep annotation, then `--reuse-values` upgrades. Main's chart: the copy Job restarted twice before SeaweedFS was ready, and retiring MinIO deleted `onyx-minio`. This branch: the api and copy Job waited about 18 s in `wait-for-object-store` (six one-line logs each) with 0 restarts, the copy completed, and retiring MinIO kept `onyx-minio` with the same UID, `keep` added and the MinIO data still on the volume. A second upgrade changed nothing. `test_object_store_wait_render.py` renders the init container in each case and runs the wait against a refused, then 503, then 200 store. All 81 render tests pass. ## Additional Options - [x] [Optional] Please cherry-pick this PR to the latest release version.helm/onyx-0.9.21 |
||
|
Evan LohnandClaude Opus 5.5
|
a4619ec743 |
fix(ci): install Terraform once and retry the dhi.io login (#16332)
## Description
This PR removes two sources of CI infrastructure flakes.
**Terraform provider acceptance tests.** The `acceptance` job installed
Go but not Terraform. It did not set `TF_ACC_TERRAFORM_PATH` or
`TF_ACC_TERRAFORM_VERSION`. So terraform-plugin-testing (v1.16.0,
`internal/plugintest/config.go`) looked up the latest CLI on
checkpoint-api.hashicorp.com and downloaded it from
releases.hashicorp.com on every `resource.Test`. One bad response (502,
500, wrong zip content type, timeout) exited the test binary with
`failed to find or install Terraform CLI`. This caused about 24% of the
failures.
- Add the pinned `hashicorp/setup-terraform` step (same SHA as
`docs-check`) with an exact `terraform_version: "1.16.6"` and
`terraform_wrapper: false`. 1.11 is the floor for the write-only tests.
- Export `TF_ACC_TERRAFORM_PATH` from `command -v terraform`. With this
variable set, the library uses only that binary and makes no network
calls.
**dhi.io login.** `.github/actions/dhi-base-images` logged in with
`docker/login-action` once, with no fallback. A dhi.io outage (`503` on
`/v2/`) failed every image build.
- Replace the login with a bash step: `docker login dhi.io
--password-stdin`, 4 attempts, backoff of 15s, 30s, 45s. The token goes
in on stdin and never in an argument.
- Export the DHI build args only when the login succeeds.
- Add a `required` input (default `"true"`). When it is `"true"`, a
failed login fails the step, as before. When it is `"false"`, the step
logs a `::warning::` and the build uses the public base images, as fork
PRs already do (#13915).
- `build-model-server-image` gets a `dhi-required` input (default
`"true"`) and passes it through.
- Only `pr-playwright-tests.yml` (web image and model server image) and
`pr-integration-tests.yml` (model server image) set it to `"false"`.
`deployment.yml`, `cve-alerts.yml` and the nightly workflows stay
strict.
- The export step keeps its exact format, because
`.github/scripts/base-image-ref.sh` parses it.
## How Has This Been Tested?
- `pre-commit run` on the touched files: zizmor, actionlint, check-yaml
and ripsecrets pass.
- `shellcheck` on the new login script passes.
- Ran the login script locally with a fake `docker` that returns 503.
With `required=false` it retries 4 times, warns, writes
`logged-in=false` and exits 0. With `required=true` it exits 1. An
invalid `required` value exits 1.
- `.github/scripts/base-image-ref.sh web` still reads the digest from
the action file.
- This PR's CI runs the changed jobs: Terraform provider `acceptance
(bootstrap)` and `acceptance (api-key)`, and the Playwright and
integration image builds. The `Point the acceptance tests at the
installed Terraform CLI` step prints the Terraform version.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes two sources of CI flakiness: the Terraform provider acceptance job
no longer downloads the latest Terraform CLI on every test (a bad
response from HashiCorp's APIs failed the whole run), and the dhi.io
login now retries instead of failing every image build during an outage.
- The acceptance job installs Terraform 1.16.6 once with
`hashicorp/setup-terraform` and exports `TF_ACC_TERRAFORM_PATH`, so
`terraform-plugin-testing` uses that binary only and makes no network
calls.
- The `dhi-base-images` action retries `docker login dhi.io` 4 times
with backoff. A new `required` input (default `"true"`) keeps release
and CVE workflows strict; `pr-playwright-tests.yml` and
`pr-integration-tests.yml` set it to `"false"` so PR test images fall
back to the public base images with a warning instead of failing.
<sup>Written for commit
|
||
|
Evan LohnandClaude Opus 5.5
|
87df52a60f |
fix(sharepoint): back off on Graph throttling instead of failing (#16374)
## Description
**Evidence.** The daily "Connector Tests" runs on main fail often. About
18 of 35 sampled red runs (Oct 6–19) end with `RuntimeError: Failed to
load documents ... Graph 429 activityLimitReached: The request has been
throttled`. The main test is
`test_sharepoint_connector_all_sites__docs_only`.
`hierarchy_node_permissions` and `perm_sync_checks_live` fail the same
way. The run log shows the 429 on the drive listing: `Failed to fetch
drives for site '.../sites/Permisisonsync': ('activityLimitReached',
..., '429 Client Error ...
/v1.0/sites/danswerai.sharepoint.com:/sites/Permisisonsync')`.
**Cause.** The `RuntimeError` comes from the test helper
`load_all_from_connector`, which raises when the connector yields
failures. The SharePoint gateway sends some reads through the office365
Graph SDK with a bare `execute_query()` and no retry:
- the root site (tenant domain)
- the site listing and its next pages
- the site by URL (`get_site_id`)
- the drive listing (`list_drives`)
A 429 on `list_drives` became a `MicrosoftGraphError`. Drive init caught
it, yielded an entity failure and skipped the whole site.
The other paths already back off: the raw Graph GET
(`graph_api_get_json`), the SharePoint REST and Entra reads
(`sleep_and_retry`) and the downloads. They honour Retry-After through
`backoff_seconds` and `bound_server_wait`.
**Fix.**
- `SharepointSourceOperations` now sends these SDK reads through the
shared `sleep_and_retry`. Each try builds the query on a new client,
because a failed query stays queued on its client.
- 429 and 503 wait for Retry-After. Without the header, they back off
exponentially with jitter (5s, 10s, 20s, capped at 30s). Each read tries
at most `GRAPH_API_MAX_RETRIES + 1` times, the same limit as the raw
GET.
- The existing server-wait policy (`bound_server_wait`) still applies.
It logs long Retry-After values and caps them on cloud.
- Other refusals (403, 404 and so on) are not retried and behave as
before.
- Indexing, permission sync and the capability checks all use this
gateway, so all three get the back-off.
- Indexing has no dedicated rate-limit error type. The existing signal
is `MicrosoftGraphError` with status 429, which OneDrive re-raises
through `fails_the_attempt`. When Graph still throttles after the
retries, the SharePoint site-level stages now re-raise it. Those stages
are drive init, drive access, delta page, drive iteration and site
pages. The attempt fails and the next attempt resumes from the
checkpoint. Before, the site was skipped.
`MicrosoftGraphError.is_throttled` names the check.
- The capability checks already map a 429 to
`RateLimitedValidationError`.
## How Has This Been Tested?
- New unit tests in `test_sharepoint_graph_throttling.py` use a fake SDK
client:
- a 429 with Retry-After waits that long, then succeeds
- a 503 backs off, then succeeds
- exhausted retries raise `MicrosoftGraphError` 429 after
`GRAPH_API_MAX_RETRIES + 1` tries
- 403 and 404 are not retried
- a throttled next page of the site listing is retried
- A new test in `test_load_checkpoint_resilience.py` checks that a
throttled drive init fails the attempt. It does not skip the site.
- All unit tests in
`tests/unit/onyx/connectors/{sharepoint,microsoft_utils,onedrive}` pass
(596).
- I did not run the live daily SharePoint test. The AWS SSO token had
expired, and the local `.vscode/.env` has no `SHAREPOINT_CLIENT_SECRET`.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Makes the SharePoint connector back off on Graph throttling instead of
skipping sites and failing the daily connector tests.
The gateway's office365 SDK reads (root site, site listing, site by URL,
drive listing) ran with no retry, so a Graph 429 skipped the whole site.
These reads now go through the shared `sleep_and_retry` back-off.
- 429 and 503 wait for Retry-After or back off with jitter, up to
`GRAPH_API_MAX_RETRIES` times; other refusals (403, 404) are not
retried.
- Each retry builds the query on a new client, because a failed query
stays queued on its client.
- When Graph still throttles, the site-level stages re-raise the 429 or
503 so the attempt fails and the next one resumes from the checkpoint,
instead of skipping the site.
- Indexing, permission sync, and capability checks all get the back-off.
- New unit tests cover Retry-After honoring, back-off with exhaustion,
non-retried refusals, throttled listing pages, single-use retried
queries, and checkpoint resilience.
<sup>Written for commit
|
||
|
Nikolas Garza
|
fac3c16ac5 |
ci(cherry-pick): cherry-pick a merged GitHub stack as one PR (#16463)
## Description **Bug:** when a GitHub stack merges, each upper PR's merge event carries its parent branch as `base.ref`. The `base.ref == 'main'` gate skips them, so only the bottom PR gets a cherry-pick. **Fix:** - Drop the `base.ref` gate. Check the merge commits are on main through the compare API. - Read the PR's stack through GraphQL. Layers merged within 30s of each other form one batch. Only the run for the top PR of a batch proceeds, after waiting out the window. - One `ods cherry-pick` call gets every SHA in the batch in stack order, so the batch opens one PR. A conflict fails the whole batch and alerts #cherry-pick-prs as today. - A partially merged stack cherry-picks only the layers in that merge batch. - One checked layer picks the whole batch. - Slack messages and the cherry-pick PR body show the stack. - `workflow_dispatch` with only `pr_number` resolves the batch the same way. - Gate command failures now send the Slack failure alert. ## How Has This Been Tested? - Ran the gate script locally against real PRs: Roshan's 6-layer stack (#16366 to #16371), the lower layers of that stack, a partially merged stack (#16322 to #16326), plain checked and unchecked PRs, a dispatch by SHA, a disallowed merger, and a GraphQL error. - Dispatched this branch's workflow with `pr_number=16371`. The gate resolved all 6 PRs in order with `GITHUB_TOKEN`: https://github.com/onyx-dot-app/onyx/actions/runs/39697078780. The cherry-pick step failed because all 6 were already on release/v5.0 by hand. `ods` fails on two empty cherry-picks in a row, which is a separate `ods` bug. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
Wenxi
|
29bc370294 |
fix(plugins): restore token entry after MCP URL changes (#16458)
## Description
Restore API-token entry after an administrator changes an MCP server
URL. Saving the new URL clears old header values before opening
Authentication. The retained empty header previously prevented the form
from creating its token field.
Restore the default `Authorization: Bearer {api_key}` template when all
API-token header values are blank. Administrators can then enter a
replacement personal or shared token. Existing nonempty custom headers
remain intact.
This follow-up targets main after the MCP setup stack merged. Its
release backport requires the MCP backport stack ending at #16455.
## How Has This Been Tested?
- 102 MCP authentication, configuration, and detail component tests
passed.
- Regression cases cover replacement personal and shared tokens, the new
server URL, and the submitted authentication template.
- Changed-file pre-commit checks passed, including TypeScript typing and
coverage, lint, formatting, and feature-map integrity.
## Additional Options
- [x] [Optional] Please cherry-pick this PR to the latest release
version.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the MCP API-token field disappearing after an administrator
changes a server URL. Saving the new URL clears old header values, and a
fully blank header previously prevented the form from creating its token
field. The form now restores the default `Authorization: Bearer
{api_key}` template whenever all API-token header values are blank, so
administrators can enter a replacement personal or shared token.
Existing nonempty custom headers remain intact.
<sup>Written for commit
|
||
|
roshan
|
bf62f4bf65 |
feat(admin): add opt-in Decision Models configuration (#16276)
## Summary Add Admin > Decision Models for TypeSafe AI, OpenAI, and OpenRouter. Decision-model selection starts unset and never falls back to the chat default. Decision providers have dedicated save and credential-test endpoints. Their request accepts connection fields and explicit `selected_model_names`. It rejects chat settings. The database operation changes decision selections while preserving shared chat settings, visibility, defaults, and access. The dedicated decision form separates catalog capability from saved selection. Catalog refresh and auto-mode synchronization do not select additional decision models. Clear the selected workspace decision model before removing its model or provider. Remove decision-scope flags and branches from the shared chat provider forms and updates. Shared model settings remain on Language Models. No classification or review workflow ships in this stack. ## Validation - 247 related backend unit tests passed, including stored-capability preservation. - 15 tests against isolated PostgreSQL passed. - 24 frontend component tests and full web type checks passed. - Live requests through the frontend proxy verified selection changes, chat preservation, and rejection of chat settings. - All applicable pre-commit hooks passed, including Python and TypeScript types, Ruff, oxlint, oxfmt, secret scanning, and feature-map integrity. - The standard local Playwright setup requires paid group APIs unavailable on this local deployment. The browser test did not reach its assertions. ## Stack testing (`29eec61fb4`) Tested the full stack at the top commit on a local dev stack, desktop and 390×844 mobile. OpenAI decisions and chat used a real key. TypeSafe, Zoom, OpenRouter, and Vercel AI Gateway used local mock upstreams; Onyx auth, validation, persistence, encryption, and UI were real. - Passed: decision selection starts **Not configured**, independent of the chat default. - Passed: live OpenAI decision validation saved `gpt-6-luna`. OpenRouter discovery and validation worked; bad keys fail visibly. - Passed: decision saves keep shared chat settings, visibility, defaults, and access (DB before/after compared). - Passed: save/test endpoints reject chat settings (422). Catalog refresh and chat auto-mode do not add decision selections. - Passed: model and provider removal are blocked until the workspace selection is cleared. - Passed (regression): live GPT-5 Mini chat works; decision-only models are absent from the chat picker. - **Fixed (`37c248ec04`):** two UI bugs found here. See *Fix retest* below.       ## Fix retest (`78cf4ccb12`) `37c248ec04` fixes two bugs found by the stack test: - Decision Models showed a deleted provider (e.g. Zoom removed on Voice) as Configure/Delete until a full reload. `useAdminLanguageModels` now takes `{ revalidateIfStale }`; Decision Models passes `true`, so the cached provider list refetches on mount. - Configure headings rendered literal asterisks (`Configure *Vercel AI Gateway*`). `DecisionProviderModal` now wraps the title in `markdown()`, like the shared LLM modal. Retested in the UI, no page reloads: - Passed: Zoom connected on Voice and Decision Models, selection cleared, Zoom disconnected on Voice. Sidebar navigation to Decision Models showed **Connect**. - Passed: a Gateway connection made on Language Models showed **Configure** on Decision Models. - Passed: Gateway and OpenAI Configure headings show no asterisks. - Passed: both pages load, and display-name edits save and persist. - Jest: new regression test for a stale cached provider list, and a heading assertion.    ## Stack PR 2 of 4: #16262 → **#16276** → #16280 → #16284. Link to Devin session: https://app.devin.ai/sessions/39d77bde91c34b75922bb05c02b93b90 Open in Devin Desktop: https://app.devin.ai/desktop/session/39d77bde91c34b75922bb05c02b93b90?variant=devin Requested by: @rohoswagger |
||
|
roshan
|
cc110e2bbf |
feat(llm): add TypeSafe AI provider setup (#16262)
## Summary Add TypeSafe AI as a model provider so admins can save System One credentials before decision-model support lands. - Add the real TypeSafe AI logo, provider card, and credential form in Language Models. - Test credentials through TypeSafe's native System One endpoint and reuse encrypted provider storage. - Support creation, editing, and deletion without creating chat models or setting a chat default. - Keep TypeSafe out of chat onboarding. Reject tests that send saved credentials to another provider. - Keep the setup form focused on credentials. Remove the roadmap and internal validation paragraph from all nine locales. This PR covers provider setup only. #16276 moves TypeSafe to the dedicated Decision Models page and adds discovery and optional selection. No review or classification workflow ships in this stack. ## Validation - Previously created and edited the provider in the live admin UI with a real TypeSafe key. Verified saved credentials against System One. - Confirmed that the provider stores no chat models, stays out of chat listings, and can be deleted through the API. - 232 backend unit tests, 25 PostgreSQL tests, and 12 frontend tests passed. - Touched-file hooks passed for typing, lint, formatting, translations, secrets, and feature-map checks. - After the copy removal, ten setup and locale tests passed on this PR. Thirteen passed on the full stack. Typecheck and lint passed. ## Screenshots Current provider-only setup at `475ae5d`, with the real TypeSafe AI logo and no implementation-status paragraph. Credentials are empty. This screenshot is separate from the earlier real-key validation.  ## Stack testing (`29eec61fb4`) Tested the full stack at the top commit on a local dev stack, desktop and 390×844 mobile. OpenAI decisions and chat used a real key. TypeSafe, Zoom, OpenRouter, and Vercel AI Gateway used local mock upstreams; Onyx auth, validation, persistence, encryption, and UI were real. - Passed: bad credentials show an error; good credentials reach `/v1/systemone`. - Passed: create, rename, refresh, model removal, and delete. - Passed: TypeSafe stores decision models only, never becomes the chat default, and is absent from the chat picker. - Passed: both credential-test endpoints reject reuse of saved TypeSafe credentials against another provider; no request reached the other base.    ## Stack PR 1 of 4: **#16262** → #16276 → #16280 → #16284. Link to Devin session: https://app.devin.ai/sessions/39d77bde91c34b75922bb05c02b93b90 Open in Devin Desktop: https://app.devin.ai/desktop/session/39d77bde91c34b75922bb05c02b93b90?variant=devin Requested by: @rohoswagger |
||
|
devin-ai-integration[bot]
|
054f786794 |
chore(mcp): remove helpers and strings left unused by staged setup (#16444)
## Description
Remove code that the staged MCP setup stack (#16419, #16413) left with
no production callers:
- `mcpOAuthReturnHref` in `web/src/lib/plugins/routes.ts`. Its only
caller, the old admin OAuth start in `MCPAuthenticationPage.tsx`, was
removed.
- `consumeMcpChatEnableIntent` in
`web/src/sections/actions/mcpChatEnableIntent.ts`. Its only caller, the
`onTriggerFetchTools` handler in `MCPPluginSetupModal.tsx`, was removed.
The private `checkIntent` helper is now inlined into
`verifyMcpChatEnableIntent`. Behavior does not change.
- The `actions.mcpAuthModal.errors.oauthStage` and `discoveryStage`
strings in all 9 locale files. They were read only through ``
t(`mcpAuthModal.errors.${connectionError.stage}`) ``, and that call was
removed.
Tests that used `consumeMcpChatEnableIntent` to check stored intent
state now use `verifyMcpChatEnableIntent`.
## How Has This Been Tested?
- `jest` for `mcpChatEnableIntent`, plugin routes,
`MCPServerDetailPage`, `sections/actions/modals`, and `src/i18n`: 214
tests passed.
- `bun run types:check`, oxlint, oxfmt, and pre-commit hooks passed.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
Link to Devin session:
https://app.devin.ai/sessions/55fba60b72e94e1282b8d8caad16b371
Open in Devin Desktop:
https://app.devin.ai/desktop/session/55fba60b72e94e1282b8d8caad16b371?variant=devin
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removes MCP helpers and locale strings left unused by the staged MCP
setup stack.
- `consumeMcpChatEnableIntent` becomes `verifyMcpChatEnableIntent`,
which no longer deletes the stored proof after checking; callers clear
it explicitly with `clearMcpChatEnableIntent`.
- `mcpOAuthReturnHref` and the `oauthStage`/`discoveryStage` strings in
all nine locale files had no remaining callers and are removed.
- Tests that relied on consuming to expire proof now verify and clear
separately.
<sup>Written for commit
|
||
|
devin-ai-integration[bot]
|
ad4534cffc |
chore(craft): remove unused viewActions translation key (#16433)
## Description
Remove `craft.actionPolicies.viewActions` from all nine message
catalogs. #16408 removed its only caller, `t(onChange ? "manageActions"
: "viewActions")` in `TaskActionPolicies.tsx`. No other code uses the
key, and no code builds `actionPolicies` keys dynamically.
## How Has This Been Tested?
- `bunx jest src/i18n/__tests__` (9 catalog tests) passes.
- Pre-commit passes: TypeScript type check, translation updates,
feature-map integrity.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
Link to Devin session:
https://app.devin.ai/sessions/70f47988b6014b07bdc5b1a3112017fc
Open in Devin Desktop:
https://app.devin.ai/desktop/session/70f47988b6014b07bdc5b1a3112017fc?variant=devin
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removes the unused `craft.actionPolicies.viewActions` translation key
from all nine message catalogs. #16408 removed its only caller, so the
key is no longer referenced by any code.
<sup>Written for commit
|
||
|
Wenxi
|
06b6fa03fb |
fix(plugins): open tools after MCP setup completes (#16413)
## Description Open the saved plugin's Tools tab after MCP setup succeeds, including OAuth connections with an existing valid grant. Setup previously returned to Configuration, which made successful authentication appear to restart setup. Separate authentication error notices from their recovery buttons with consistent spacing. This includes the Use your own OAuth client action after automatic registration fails. This PR builds on #16419. It uses the existing detail route with `tab=tools` and preserves the stack's staged validation and name-first focus. The stack includes the shared Chat update fix from main. The temporary exclusive scheduling workaround is removed. ## How Has This Been Tested? - Passed all 174 tests across the affected UI, MCP service, and translation suites after rebasing. - Verified completed setup selects Tools without repeating discovery. Legacy OAuth return coverage still selects Tools and discovers once. - Passed changed-file checks for TypeScript, coverage, formatting, lint, secrets, and feature-map integrity. - Playwright CI will rerun on this stacked revision, including the personal-key tests under normal scheduling. - The registration-error recovery test passes after the spacing fix. - Live external provider consent was not tested. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * MCP setup completion now opens the plugin detail page directly on the Tools tab. * OAuth returns and existing authenticated connections open the Tools tab, with legacy OAuth returns continuing to support tool discovery. * Direct navigation to the Tools tab is supported without triggering tool discovery. * **Bug Fixes** * Improved the MCP authentication error layout while preserving recovery options for registration and discovery failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Wenxi
|
d942a6d457 |
fix(plugins): keep MCP edits unsaved until connection succeeds (#16419)
## Description Keep MCP setup and connection edits unsaved until authentication and tool discovery succeed. In-form failures retain drafts and leave the saved connection unchanged. Changing Configuration again invalidates an older staged proposal. Explicit Chat choices take priority over previous enable intents. Successful authentication preserves unsaved tool edits. URL changes clear old header values, and intentional OAuth navigation avoids a discard warning. Remove unused legacy client wrappers and dead state. Full-page OAuth failures require restarting authorization; this change does not persist browser form drafts across redirects. Setup failures use a neutral translated connection error when an upstream response has no usable error detail. OAuth-specific errors and safe backend details remain intact. The shared browser-test helper follows staged setup responses and reads the saved server ID after OAuth completes. ## How Has This Been Tested? 139 component tests and 49 selected service tests passed. Full TypeScript checking/coverage, lint, formatting, and feature-map checks passed. Backend transaction test limits are recorded in the parent PR. Two focused Playwright helper checks passed for immediate connection and pending authorization. A live local smoke test stopped at the existing SSRF block on the local mock; security settings were unchanged. Full browser CI is rerunning. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added staged MCP server setup with validation and tool discovery before publishing. * Preserved unsaved settings and tool selections during authentication. * Added OAuth continuation for authentication-required setups. * Added explicit control over enabling MCP servers for Chat. * **Bug Fixes** * Failed or cancelled authentication no longer overwrites saved settings. * Provider-denied OAuth requests now display the correct cancellation state. * Improved connection-failure messaging across supported languages. * **Documentation** * Updated MCP setup and OAuth lifecycle documentation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Wenxi
|
1e15e42f71 |
fix(mcp): publish connection changes only after validation (#16418)
## Description Validate MCP authentication and tool discovery before saving a new connection or changing an existing connection. Failed attempts leave the saved URL, credentials, and tools unchanged. Save validated settings, credentials, tools, and requested Chat/Craft choices in one transaction. Reject stale policy/configuration edits while allowing ordinary token refresh. Preserve registered OAuth client settings and safely reuse an unexpired grant without rotating its refresh token during validation. Support SSE discovery with the exchanged bearer. Preserve omitted transports and valid grants without expiry. Lock policy and default Chat rows before the final stale-edit check, using the shared locking helper from main. Tool discovery now returns a structured authentication error for upstream HTTP 401 responses, including nested transport failures. Setup and saved-connection refresh share this mapping. Runtime credential rejection handling keeps the original transport errors. Async setup handlers await publication in a worker thread so database lock waits do not block the event loop. ## How Has This Been Tested? MCP and route unit tests passed, including SSE discovery, failed publication, safe token reuse, and immutable credential snapshots. Python typing, lint, formatting, and route permission checks passed. All 11 PostgreSQL setup/edit regression tests passed locally after the final fixture corrections. They cover failed publication, stale edits, OAuth registration metadata, and preservation of refreshed credentials. The expanded MCP, runtime, and route suite passed 375 tests. An unauthenticated request to the live GitHub MCP endpoint confirmed a sanitized 401 response without database writes. All 17 focused setup API tests pass after the worker-thread change, including both publication paths. CI is rerunning on the updated commit. Live provider sign-in remains unverified. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **New Features** - Added an MCP setup workflow to validate server settings, authenticate connections, discover tools, and review configuration before publishing. - Added OAuth authorization and callback support, including reuse of valid saved grants. - Setup changes are published atomically, protecting existing configurations when validation, authentication, or discovery fails. - Added safeguards against stale concurrent edits overwriting newer configuration changes. - **Bug Fixes** - Improved MCP authentication error handling, including clearer handling of unauthorized responses during tool discovery. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Wenxi
|
eee522c3fd |
feat(mcp): add temporary OAuth state for staged connections (#16417)
## Description Add temporary OAuth storage for proposed MCP connections. Attempts use Onyx’s credential-protection helper, expire after ten minutes, and can be consumed once by their owner. Encryption requires Enterprise Edition and a configured encryption key. Shared cache storage supports callbacks on any API replica. Reuse provider construction and discovery. Preserve registered client metadata and probe through an OAuth-capable transport before SSE sign-in. ## How Has This Been Tested? OAuth storage/provider tests and the full MCP/route unit suite passed. Redis concurrency and owner-binding tests passed. Python typing, lint, formatting, and feature-map checks passed. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added OAuth setup for MCP connections, including provider discovery and resumable authorization. * Added secure, temporary setup storage with expiration, single-use protection, and owner and tenant binding. * Added clear setup outcomes for connected servers and authorization-required states. * Added configuration options for availability, chat access, policy management, and action policies. * **Documentation** * Documented staged OAuth support for proposed MCP connections. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Wenxi
|
381123cae9 |
fix(plugins): improve MCP setup and configuration (#16414)
## Description MCP setup focuses Server Name when opened and when returning from Authentication. MCP setup shows neutral indicators until a valid server URL starts discovery. Spinners begin after the existing 700-millisecond debounce. Clearing or changing the URL resets both indicators and cancels the previous probe. OAuth Pass-through appears in the authentication dropdown when Onyx supports OAuth login, even with advanced settings collapsed. It remains an explicit choice. Configuration now shows the server URL. Users with edit permission can change it; other viewers see a read-only field. Saving a new URL uses the existing backend credential reset and opens Authentication to reconnect and discover tools. The saved response replaces cached server data before reloading. Whitespace-only edits preserve the existing connection, and metadata-only saves omit the URL. The preview also exposed missing client boundaries in Opal. The date-range picker and toast hooks now declare their client boundary at their source modules. ## How Has This Been Tested? - Setup focus regression: 40 component tests passed; signed-in browser confirmed Server Name receives initial focus. - MCP setup and authentication component suites: 79 tests pass. - MCP configuration and detail component suites: 51 tests pass, including invalid URLs, edit permissions, unchanged URL preservation, and a failed reload after saving a new URL. - Date-range picker component suite: 6 tests pass. - Changed-file pre-commit checks pass, including TypeScript and type coverage, formatting, lint, and feature-map integrity. - Live signed-in preview: Configuration shows the saved URL and permits admin edits, with no browser page errors. No live server URL was changed. - Live preview: `/app` and `/admin/plugins` render without page errors. Blank MCP setup shows both discovery stages idle. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * MCP server URLs can be edited with required HTTP(S) validation and remain read-only for viewers. * URL changes trigger authentication, reconnection, and tool rediscovery as needed. * OAuth pass-through is available whenever OAuth login is enabled. * Probe stages clearly show idle, pending, successful, and failed states. * Direct tab navigation and improved focus handling are supported in server setup and details. * **Bug Fixes** * Clearing or changing a URL resets probe status and cancels outdated checks. * Metadata-only saves preserve the existing endpoint. * Updated server details remain visible if reloading fails. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
roshan
|
1af99d61cb |
chore(craft): improve Kubernetes startup, image cleanup, and local debugging (#16376)
## Description
Craft development required repeated image builds, copied cluster
settings, and editor-specific Telepresence commands. Startup could
report success before services were usable.
- Share configuration resolution and launch commands between Make and VS
Code. Read deployed ConfigMaps and Secrets in memory, with personal
`.env.k8s` overrides.
- Add separate infrastructure and application readiness checks,
`craft-doctor`, and shared Telepresence connection/intercept handling.
- Reuse images, detect source changes, and provide checkout-scoped
cleanup previews. Add pause/resume and migration commands with
database-target checks.
- Default Kubernetes sandbox callbacks to the internal API Service.
Preserve explicit `ONYX_SERVER_URL` overrides and bump the Helm chart to
0.9.20.
- Update the local Kubernetes guide, Craft development skill, and
feature map.
- Give Helm chart CI a 100 GB runner disk. The previous runner exhausted
disk while unpacking OpenSearch.
- Upgrade the Storybook package family to 10.6.2 to fix the critical
audit finding GHSA-f2ph-cmg5-wxc8.
Existing `.env.k8s` files remain untouched; old service entries continue
to override chart settings. Image refresh updates the PodTemplate for
future sandboxes; existing sandbox pods retain their images.
## Review guide
[Read or download the Markdown review
guide](https://github.com/onyx-dot-app/onyx/blob/d3cbc42d89bfb150d1cfc815638645043c34b69e/docs/craft/dev/kubernetes-review-guide.md).
## How Has This Been Tested?
- Live kind and Telepresence: startup, intercept reuse, doctor, API and
worker hot reload, readiness success/failure, and migration
status/upgrade.
- Three real Craft smoke tests: sandbox provisioning, file
upload/readback, and web app preview through the frontend. No real LLM
calls were exercised.
- 76 Craft Helm tests and 27 development-tool regression tests passed.
- Pre-commit checks, explicit development-tool type checking, frontend
type coverage, and skill validation passed.
- The local web audit passes its critical-severity gate without
suppressions. The Storybook production build passes.
- Removed the disposable test cluster, local processes, and temporary
files.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Harden local Craft Kubernetes development by sharing configuration and
launch logic between Make and VS Code, and by checking service readiness
before startup reports success.
**What’s included**
- Resolve deployed ConfigMaps, Secrets, and container env in memory
through a shared launcher; personal `.env.k8s` values take precedence,
and resolved secrets never touch disk or terminal output.
- Add shared lifecycle commands (`craft-doctor`, staged
infrastructure/application readiness, Telepresence connect and
intercept, pause, resume, migration, cleanup previews) used by both Make
and VS Code.
- Reuse built images when source is unchanged, force a rebuild with
`--rebuild-sandbox-image`, and keep probe and command invocations
bounded while preserving actionable errors.
- Default sandbox callbacks to the internal API Service, preserving
explicit `ONYX_SERVER_URL` overrides; drop the now-unneeded validation
and bump the Helm chart to 0.9.20.
- Give the Helm CI runner a 100 GB disk, since Docker and kind each
store images and OpenSearch exhausted the default disk. Bump Storybook
tooling to 10.6.2 to patch a development server vulnerability.
- Update the local Kubernetes guide, Craft development skill, feature
map, and add a downloadable PR review guide.
**Validation**
- Passed 76 Craft Helm tests, 27 development-tool regression tests, and
new unit tests for the shared runtime and lifecycle CLI.
- Tested startup, readiness, Telepresence, hot reload, migrations, and
three Craft smoke tests against kind; no real LLM calls were exercised.
<sup>Written for commit
helm/onyx-0.9.20
|
||
|
Nikolas Garza
|
8767bbf110 |
fix(license): refuse a Community downgrade unless the license gates access (#16439)
## Description `POST /api/license/downgrade` now refuses unless the deployment is gated, the only state in which the UI offers it. Fixes [ENG-4422](https://linear.app/onyx-app/issue/ENG-4422/post-apilicensedowngrade-does-not-check-expiry-on-the-server). **Why** - The endpoint checked nothing. Any admin could call it on a live license and make every connector public. - The UI check alone is not a guard. **What** - `is_access_gated(metadata)` in `ee/onyx/server/settings/api.py`: an expired license, or no license with a perm-synced connector. - `/settings` and the endpoint both use it. The settings branches are otherwise unchanged. - The endpoint returns `CONFLICT` before any change when not gated. The step order is unchanged. - Known: with no license, a settings reset that fails after the commit cannot be retried, since the commit removed what gated it. ## How Has This Been Tested? Unit: ACTIVE, PAYMENT_REMINDER and GRACE_PERIOD licenses, and no license without perm-synced connectors, are refused with no step run (these fail on main). A gated license, and no license with a perm-synced connector, run every step. Settings tests pass unchanged. Live, lite mode on a scratch DB: 409 for no license, active, payment reminder and grace period, 200 for gated. With no license and a SYNC pair, a failure injected before the commit returned 500 and stayed gated, the retry returned 200, the next call 409. ## Additional Options - [x] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
devin-ai-integration[bot]
|
1c779b3288 |
chore(indexing): remove unused IndexingCoordination.request_cancellation (#16434)
## Description
#16391 changed `_cancel_index_attempt` in
`backend/onyx/server/documents/cc_pair.py` to
`_cancel_index_attempt__no_commit`. It now calls
`IndexingCoordination.request_cancellation__no_commit`. That was the
only caller of `IndexingCoordination.request_cancellation`, so this PR
removes it.
- Remove `IndexingCoordination.request_cancellation` from
`backend/onyx/db/indexing_coordination.py`.
- Change the reference in
`.agents/feature-map/components/indexing-pipeline.md` to
`request_cancellation__no_commit`.
## How Has This Been Tested?
- A search of the repo for `request_cancellation` finds no callers. The
search covered backend, ee, tests and mock-patch strings.
- `prek run` passes on the changed files (ty, ruff, feature map
integrity).
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
Link to Devin session:
https://app.devin.ai/sessions/a83c20c09124415fad8a6a274dce15d3
Open in Devin Desktop:
https://app.devin.ai/desktop/session/a83c20c09124415fad8a6a274dce15d3?variant=devin
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Removes the unused `IndexingCoordination.request_cancellation` method.
Its only caller now uses `request_cancellation__no_commit` directly, so
the wrapping method and its commit are dead code. Updates the
indexing-pipeline feature map to reference the remaining method.
<sup>Written for commit
|
||
|
Justin Tahara
|
f6249c71fa |
feat(llm): assume an AWS role for cross-account Bedrock (#16357)
## Description
Adds cross-account Bedrock. Onyx can run in one AWS account and call
Bedrock in another. The Bedrock provider gets two optional fields:
**Role ARN to assume** and **External ID**.
- **Chat calls:** `AWS_ROLE_NAME` / `AWS_EXTERNAL_ID` in `custom_config`
map to LiteLLM's `aws_role_name` / `aws_external_id`. LiteLLM 1.93 does
the STS AssumeRole, including from IRSA, and caches the credentials.
- **Model fetch:** `/admin/llm/bedrock/available-models` assumes the
role with the submitted credentials (IAM or access key), then lists
models with the temporary credentials.
- **Admin modal:** the two fields show for the IAM and access key auth
methods. They are cleared for the long-term API key method, because a
bearer token is bound to one account.
- **Cloud:** multi-tenant deployments reject these keys on provider
upsert, on provider test, and on model fetch. The pods' shared identity
would let any tenant assume any role that trusts it. The modal still
shows the fields on cloud; the backend rejects them (the same pattern as
Vertex Workload Identity).
The env var fallback (`AWS_ROLE_NAME` on the pods) works today without
this change. This PR makes the setting per-provider and fixes the model
fetch, which ignored the role.
## Screenshots
**Access key + role.** The role fields show below the credentials. Both
are optional.
<img width="600" alt="Access key auth with role ARN and external ID"
src="https://github.com/user-attachments/assets/1d934b1a-17ac-4524-8449-40091ffbc5f3"
/>
**Environment IAM role + role.** The pod's own role (for example IRSA)
assumes the role in the other account.
<img width="600" alt="IAM auth with role ARN and external ID"
src="https://github.com/user-attachments/assets/8631c344-4c46-4367-a658-16e92490367d"
/>
**Long-term API key.** The role fields are hidden and cleared, because a
bearer token works in only one account.
<img width="600" alt="Long-term API key auth hides the role fields"
src="https://github.com/user-attachments/assets/10d1ae9c-3ebb-4c1a-9600-e399b42e5ab1"
/>
**Model fetch with a role.** With dummy keys, the error comes from the
`AssumeRole` call. This shows the fetch assumes the role before it lists
models.
<img width="800" alt="Model fetch fails at AssumeRole with dummy keys"
src="https://github.com/user-attachments/assets/66f9c9e1-9c63-4136-afc3-73e08a29447b"
/>
## How Has This Been Tested?
- Unit: the role keys map to LiteLLM kwargs; the model fetch calls
`sts.assume_role` with the base credentials and lists models with the
temporary credentials; multi-tenant rejection in the model fetch and in
the write-path gate.
- `tsc`, `oxlint`, `oxfmt` run by hand (no `bun` locally); CI runs the
real hooks.
- Local run of this branch (fresh DB): the modal shows and clears the
fields for each auth method; the model fetch returns `400
CREDENTIAL_INVALID` from `AssumeRole` with dummy keys.
- Not yet tested against two real AWS accounts.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds cross-account Bedrock support so Onyx can run in one AWS account
and call Bedrock in another, with the Bedrock provider assuming a role.
- Adds optional Role ARN and External ID fields to the Bedrock provider.
- Chat calls map the fields to LiteLLM's `aws_role_name` and
`aws_external_id`, which performs the STS AssumeRole and caches
credentials.
- Only the uppercase `AWS_ROLE_NAME`/`AWS_EXTERNAL_ID` spellings are
accepted; lowercase spellings are dropped when an admin edits the
provider.
- `/admin/llm/bedrock/available-models` now assumes the role with the
submitted credentials before listing models; previously it ignored the
role.
- The admin modal shows the two fields for the IAM and access key auth
methods, and clears them when the long-term API key method is selected,
since a bearer token is bound to one account.
**Security**
- Multi-tenant deployments reject these keys on provider upsert,
provider test, and model fetch, because the shared server identity would
let any tenant assume roles that trust it.
<sup>Written for commit
|
||
|
roshan
|
17a632b05e |
fix(auth): honor the self-hosted schema for OAuth clients (#16387)
## Description
Self-hosted installations with a custom `POSTGRES_DEFAULT_SCHEMA`
currently create OAuth client registrations in `public`. This fails when
the migration role cannot create tables there.
Store OAuth clients in the configured application schema for
single-tenant installations. Keep cloud registrations in the shared
`public` catalog. Revision `e38f8193dd32` moves existing registrations
with `ALTER TABLE ... SET SCHEMA`, preserving client IDs, metadata,
timestamps, and indexes. Update the ORM schema to match.
Correct the original migration's schema placement so fresh installations
do not fail before reaching the new revision. Its downgrade accepts
either historical location. Already-applied databases use the new
revision to converge on the same layout. Intermediate revisions can have
different layouts depending on which code version created them.
Downgrade restores the registry to `public` for compatibility with older
application code and requires permission to create there.
## How Has This Been Tested?
- Eight PostgreSQL migration and storage tests passed, covering
preserved records and indexes, upgrade/downgrade/re-upgrade, cloud and
default-schema behavior, and a fresh custom-schema install without
CREATE permission on `public`.
- Client registration and lookup tests passed in separate processes
configured for a custom schema and cloud mode.
- Pre-commit, Ruff, and Ty checks passed.
- Fable reviewed the migration and runtime routing. Tests exercise
individual revisions rather than the complete migration chain.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Self-hosted installations with a custom `POSTGRES_DEFAULT_SCHEMA` now
store OAuth client registrations in that schema instead of `public`,
fixing failures when the migration role cannot create tables there.
Cloud registrations stay in the shared `public` catalog, and the ORM
model follows the same routing.
**Migration**
- Revision `e38f8193dd32` moves existing self-hosted registrations to
the configured schema with `ALTER TABLE ... SET SCHEMA`, preserving
client IDs, metadata, timestamps, and indexes.
- The original migration creates fresh installs in the configured
schema, and its downgrade handles either historical location.
- Downgrading restores the registry to `public` and requires permission
to create there.
<sup>Written for commit
|
||
|
Nikolas Garza
|
ec1ed831c0 |
fix(deployment): keep MinIO and its copy service stopped on a fresh Compose install (#16429)
## Description A fresh hand-run Compose install no longer starts MinIO or `object-store-copy`. Fixes [ENG-4429](https://linear.app/onyx-app/issue/ENG-4429/a-fresh-manual-compose-install-still-starts-minio). **Why** - With the v5 compose file and `env.template` defaults, MinIO started on a fresh install. Only onyx-cli set `MINIO_REPLICAS=0`. - Even with it set, `object-store-copy` started and idled forever. **What** - `env.template` sets `MINIO_REPLICAS=0`, since a fresh install never held files in MinIO. - `object-store-copy` follows `MINIO_REPLICAS` like `minio` does. - The compose default stays `1`, so an earlier release's `.env`, which lacks the variable, keeps MinIO and the copy running. - `env.prod.template` names the object store, the S3 key pair and `MINIO_REPLICAS=0`. A fresh prod `.env` defaulted the app to MinIO before. - The retire step stops MinIO with `docker compose up -d --no-deps minio object-store-copy`, since a plain `up -d` recreates every service that loads `.env`. - Not changed: Compose still creates the empty `minio_data` volume for a zero-replica service. Only a profile split across the CLI, ods, CI and installers would avoid it. Docs: https://github.com/onyx-dot-app/documentation/pull/521 ## How Has This Been Tested? Live on scratch Compose projects with the v5.0.0-beta.2 images. A `.env` copied from the new `env.template` ran only `object-store` and `relational_db`, with no `minio` or `object-store-copy` container. An upgraded `.env` without `MINIO_REPLICAS` ran `minio` and `object-store-copy`, and the copy started. `docker compose config` shows replicas 0 and 1 for the two cases. With `docker-compose.prod.yml` and a fresh `env.prod.template` plus keys, `api_server` points at `object-store` and `minio` and `object-store-copy` are at 0, and empty keys still refuse to start. `go test ./...` in `cli` passes. In `tools/ods`, `TestDevcontainerCommands` and `TestRunDevRebuild` fail locally because `SSH_AUTH_SOCK` adds a mount, which is unrelated. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. |
||
|
Justin Tahara
|
2bc84cf6bf |
ci: run only required checks in the merge queue (#16430)
## Description
The merge queue on `main` now runs only the jobs behind its 7 required
checks. Integration, Playwright and the Golang coverage snapshot move to
a post-merge run on `main`.
**Why**
- The queue merges as soon as the 7 required checks pass:
`quality-checks`, `playwright-required`, `database-tests`,
`backend-check`, `mypy-check`, `Jest Tests`, `required`. `required` and
`playwright-required` come from `merge-group.yml`, which reuses the PR's
verdict (#7080, tested-base #14682).
- 12 more workflows also ran on `merge_group`, and nothing waited for
them. They are ~27% of runs-on compute, and each queued PR started a
full CI run (~470 vCPU per group, 5 at a time). Those bursts fill the
vCPU quota and the spot pools.
- Last 14 days (640 merge groups, from ci.onyx.app): in the queue,
Integration failed 91 times, Ext-dep 81, Playwright 40, TF provider 29
and MCP 28. The PRs merged anyway, and none of it left `main` broken.
Push-to-main failures were only Connector Tests, Golang (3) and the Helm
release (1). So removing these runs does not lower protection; they
never gated a merge.
- Precedent: #11673 already moved Connector Tests out of the queue to PR
+ push-to-main, for the same reasons.
**Kept on `merge_group`** (required checks): `merge-group.yml`,
`pr-quality-checks`, `pr-database-tests`, `pr-python-tests`,
`pr-python-checks`, `pr-jest-tests`. The release-branch queue requires
only quality / database / backend / mypy, and release branches use their
own workflow files.
**Removed from `merge_group`:** mcp-compatibility, craft compose
integration, craft compose tests, craft k8s, desktop build,
external-dependency unit tests, golang, helm chart testing, storybook,
terraform provider, integration, playwright. Their PR triggers do not
change.
**Moved to push to `main`**
- **Integration:** full EE + MIT matrix. This keeps the MIT coverage
that only the queue ran (#5413). `reindex_port` is not run on main; PRs
(related changes) and the daily run still cover it.
- **Playwright:** full suite, the S3 visual-baseline update (only from a
fully green run, #8410), and the Okta job (informational on push).
- **Golang:** the coverage snapshot for `main` is published by the push
run instead of the queue run.
- **Concurrency:** pushes to `main` share one group and queue
(`cancel-in-progress: false` for push), with one run in progress and the
newest pending. A burst of merges gives about 2 runs, and every run
finishes. PRs still cancel older runs.
**Trade-off:** a conflict between two PRs that only integration or
Playwright catches shows up on `main` right after the merge, not before
it. That was already the case: nothing in the queue waited for these
jobs.
## How Has This Been Tested?
- `pre-commit` on the changed files: actionlint, zizmor and check-yaml
pass. All workflows parse. `merge_group` stays only in the 6 kept files.
- After merge, on ci.onyx.app: merge_group compute (Merge queue
dashboard) drops; queue time to merge is unchanged; post-merge
integration and Playwright runs appear on `main` (Workflows dashboard);
baselines and the golang snapshot keep updating.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
The merge queue now runs only its required checks. Twelve workflows
previously ran on `merge_group` without gating the merge and consumed
~27% of CI compute, so they are removed from the queue; Integration,
Playwright, and the Golang coverage snapshot move to post-merge runs on
`main`.
- Kept on `merge_group`: `merge-group.yml`, `pr-quality-checks`,
`pr-database-tests`, `pr-python-tests`, `pr-python-checks`, and
`pr-jest-tests`; all other workflows keep their PR triggers unchanged.
- Integration and Playwright now run the full suite on push to `main`;
Playwright also updates S3 baselines (only from a fully green run,
#8410) and runs the informational real-Okta job.
- The Golang coverage snapshot for `main` is published by the push run
instead of the queue run.
- Push-to-`main` runs share a concurrency group that queues instead of
cancelling, so a burst of merges still produces finished runs.
- Trade-off: conflicts only Integration or Playwright catch now surface
on `main` after the merge, not before — nothing in the queue waited on
those jobs either way.
<sup>Written for commit
|
||
|
Evan LohnandClaude Opus 5.5
|
582037d461 |
feat: every source's sign-in check unlocks the create form (#16257)
## Description
The create form unlocks the same way for every source: you fill in the
bound fields, run the checks, and the form unlocks when sign-in passes.
- `validates_binding=True` is now set on the sign-in check of GitHub,
Google Drive, HubSpot, Linear, Notion and YouTrack. Confluence, Jira,
Slack, SharePoint, OneDrive and Outlook already had it.
- Each source's base check class takes the flag.
- Discovery checks never unlock the form, so a slow count cannot hold
it.
- No frontend change is needed. `formUnlocked` in
`web/src/lib/connectors/checks/hooks.ts` already gates on
`validates_binding`. For a source without bound fields, the binding key
is just the credential.
**Behavior change:** the forms of the six sources above used to unlock
at once, without a run. They now stay locked until a run passes sign-in,
like the others.
## How Has This Been Tested?
Two new declaration tests:
- Every source with named checks has a sign-in check, and that check
sets `validates_binding`.
- Any other unlocking check runs after an unlocking check and is not a
discovery check.
The capability-check unit tests and the unit tests of every named-check
connector pass.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Unifies how connector create forms unlock: every source's form now stays
locked until a run passes a sign-in check that sets `validates_binding`,
instead of unlocking immediately.
- Adds `validates_binding=True` to the sign-in checks of GitHub, Google
Drive, HubSpot, Linear, Notion, and YouTrack; Confluence, Jira, Slack,
SharePoint, OneDrive, and Outlook already set it, and some add a listing
check (e.g., Jira's "List projects") that runs after sign-in.
- Behavior change: those six forms used to unlock without a run; they
now stay locked until sign-in passes, like the others. This also applies
to sources without bound fields, where the binding key is just the
credential.
- Updates the connector feature-map docs.
- Adds declaration tests verifying every source with named checks names
its sign-in checks, that each sets `validates_binding`, and that any
other unlocking check runs after an unlocking check and is not a
discovery check.
<sup>Written for commit
|
||
|
Evan LohnandClaude Opus 5.5
|
d6696aa66e |
fix: SharePoint site URLs follow the shared SSRF setting (#16256)
## Description One SSRF policy for every connector endpoint that an admin configures, and SharePoint site URLs now use it. - `web_connector_ssrf_enforced` is renamed to **`connector_ssrf_enforced`**. Its docstring now calls it the shared policy for connector endpoints. - Every caller uses the new name: web, GitHub, GitLab and YouTrack. - SharePoint's `validate_site_url` reads `get_security_settings().ssrf_protection_level` and passes `allow_private_network=not enforced, block_link_local_only=not enforced` to `validate_outbound_http_url`, as the other connectors do. One function covers saving (pairing), testing (the checks and the fallback) and calling (indexing). - **Behavior change:** at VALIDATE_LLM and the relaxed levels, SharePoint can now reach private hosts, the same as GitHub, GitLab and YouTrack. Cloud metadata stays blocked. ## How Has This Been Tested? New `test_site_url_ssrf_setting.py`: - VALIDATE_ALL blocks a private host. - VALIDATE_LLM, ALLOW_PRIVATE_NETWORK and DISABLED reach it. - Cloud metadata stays blocked at VALIDATE_LLM and DISABLED. The SharePoint, web, GitHub and YouTrack unit tests pass. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. - [x] [Optional] Override Linear Check 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Security** * Standardized SSRF protection across GitHub, GitLab, SharePoint, web scraping, and YouTrack connectors. * Connector URL validation now consistently follows the configured security level. * Strict protection blocks private-network destinations, while relaxed settings allow appropriate private-host access. * Cloud metadata endpoints remain blocked across protection levels. * **Tests** * Added coverage for SharePoint URL validation under different SSRF protection settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Evan LohnandClaude Opus 5.5
|
26d8b09d27 |
fix: Google Drive sign-in fails a form run when Google is unreachable (#16255)
## Description
Google Drive sign-in now follows the shared sign-in rule: on a form run
(a draft run or a pair dry run), it **fails** when Google is unreachable
or times out. Stored and scheduled runs keep it Incomplete, so a slow
Google never holds indexing.
- `_GoogleDriveCheck` accepts `fails_when_unreachable`, and
`google_drive_auth` sets it.
- New `GOOGLE_CONNECTION_ERRORS` and `is_google_timeout()` in
`source_operations.py`. They cover google-auth `TransportError`,
`httplib2.ServerNotFoundError`, requests connection and timeout errors,
and the built-in `ConnectionError` and `TimeoutError`.
- `_google_errors` maps these errors to `TimedOutValidationError` or
`UnreachableValidationError` with the design's rows: "Cannot reach
Google Drive" and "Sign-in timed out" for sign-in, "Timed out" for other
checks.
- The gateway still raises the raw errors, because indexing's retries
(`is_transient_google_error`) expect them.
**Known gap:** an OAuth account's token refresh
(`get_google_oauth_creds`) swallows network errors, so an unreachable
Google still reads "Google no longer accepts the account" for OAuth. The
fix also touches Gmail and two server endpoints, so it belongs in its
own PR.
## How Has This Been Tested?
- Unit tests:
- An unreachable Google (three error types) gives FAILED on a form run
and INCOMPLETE on a stored run, with kind `unreachable`.
- A sign-in timeout gives FAILED on a form run and "Sign-in timed out".
- A timeout in a later check stays INCOMPLETE.
- Connector, server and EE unit suites pass at this tip.
## Additional Options
- [ ] [Optional] Please cherry-pick this PR to the latest release
version.
- [x] [Optional] Override Linear Check
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Google Drive sign-in now fails a form run when Google is unreachable or
times out, matching the other sign-in checks. Stored and scheduled runs
keep the check Incomplete, so a slow Google never holds indexing.
- `source_operations.py` gains `GOOGLE_CONNECTION_ERRORS` and
`is_google_timeout()` to classify google-auth `TransportError`,
`httplib2.ServerNotFoundError`, requests connection and timeout errors,
and built-in `ConnectionError`/`TimeoutError`, including wrapped causes.
- `_google_errors` maps them to `UnreachableValidationError` or
`TimedOutValidationError`; a timeout in a later check stays Incomplete,
and `google_drive_auth` sets `fails_when_unreachable`.
- The gateway still raises the raw errors because indexing's retries
expect them. Known gap: OAuth token refresh swallows network errors, so
an unreachable Google still reads "Google no longer accepts the
account"; that fix touches Gmail and server endpoints and belongs in its
own PR.
<sup>Written for commit
|
