chore: remove Vespa and consolidate the document index interface (#15336)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

docs/craft/dev/local-kubernetes.md

@@ -120,7 +120,7 @@ per cluster. New clusters use the `kindest/node:v1.33.1` node image so Craft's
 native init sidecar pod shape is supported. Existing clusters are not recreated;
 set `KIND_NODE_IMAGE` to override the default for a newly created cluster.
 
-Watch pods (vespa and CNPG-postgres take a minute or two on first boot):
+Watch pods (opensearch and CNPG-postgres take a minute or two on first boot):
 
 ```bash
 kubectl -n onyx get pods -w
@@ -332,7 +332,7 @@ external-dependency-unit tests against a temp dir. See
 
 Run **`k8s: pause cluster`** (or `docker stop onyx-dev-control-plane`) to stop
 the kind node container. PVC data lives inside that container, so postgres,
-redis, opensearch, vespa, and minio state all survive. Resume with
+redis, opensearch, and minio state all survive. Resume with
 **`k8s: resume cluster`** — the kubelet reconciles pods automatically.
 
 Reach for **`k8s: cluster down (full teardown)`** only when you want a clean

docs/craft/kubernetes/craft-eks-runbook.md

@@ -14,7 +14,7 @@ Craft provisions a sandbox + snapshot/restore, with no manual kubectl/RBAC/node
 **Clean seam: Terraform = AWS infrastructure; Helm = everything inside Kubernetes. Terraform's outputs are Helm's inputs.**
 
 **Terraform** (`deployment/terraform/…`) creates ONLY AWS resources:
-- VPC / subnets / NAT; EKS **cluster** + **node groups** (main, vespa, **sandbox** — labeled/tainted/IMDSv2) + the **OIDC provider**; EKS add-ons + gp3 storage class.
+- VPC / subnets / NAT; EKS **cluster** + **node groups** (main, vespa — the legacy name of the document-index group, **sandbox** — labeled/tainted/IMDSv2) + the **OIDC provider**; EKS add-ons + gp3 storage class.
 - Managed data stores: **RDS** (Postgres), **ElastiCache** (Redis), **OpenSearch** domain; the main Onyx **S3 file-store bucket**; **WAF**.
 - IAM/IRSA **roles**: the workload role for Onyx application pods.
 - **Outputs** (the only thing Helm consumes): cluster name, RDS/Redis/OpenSearch endpoints, the file-store bucket name, the workload IRSA **role ARN**, the OIDC provider ARN/URL.
@@ -208,7 +208,7 @@ sandboxProxy:
 ```
 
 - ElastiCache: `REDIS_HOST`, `REDIS_SSL=true`, `REDIS_SSL_CERT_REQS=none`, `auth.redis.enabled=false` (no auth token).
-- OpenSearch (v4.0 search backend): `ONYX_DISABLE_VESPA=true`, `ENABLE_OPENSEARCH_INDEXING/RETRIEVAL_FOR_ONYX=true`, `USING_AWS_MANAGED_OPENSEARCH=true`, `OPENSEARCH_REST_API_PORT=443`, `OPENSEARCH_USE_SSL=true`, `OPENSEARCH_ADMIN_USERNAME=admin`.
+- OpenSearch (v4.0 search backend): `USING_AWS_MANAGED_OPENSEARCH=true`, `OPENSEARCH_REST_API_PORT=443`, `OPENSEARCH_USE_SSL=true`, `OPENSEARCH_ADMIN_USERNAME=admin`.
 - S3: `S3_FILE_STORE_BUCKET_NAME`, `S3_ENDPOINT_URL=""`, `AWS_REGION_NAME`.
 - Craft: `ENABLE_CRAFT=true`, `ONYX_SERVER_URL=http://onyx-api-service.onyx.svc.cluster.local:8080`, `auth.sandboxPushSecret.enabled=true`. (`SANDBOX_SERVICE_ACCOUNT_NAME`/`SANDBOX_CONTAINER_IMAGE` default correctly.)
 
@@ -424,7 +424,7 @@ The original lead infra TODO list is accounted for as:
 ## 7. Notes / gotchas (condensed)
 
 - us-west-1 has only 2 AZs (→ the slice fix). Shared account near the EIP quota → use `single_nat_gateway=true`.
-- The `vespa` node group is vestigial in v4.0 (OpenSearch replaced Vespa) — size it small or make it optional.
+- The `vespa` node group (legacy name) only matters for an in-cluster OpenSearch. With a managed OpenSearch domain, set `vespa_node_enabled = false`.
 - `cluster_endpoint_public_access_cidrs=[]` causes a perpetual no-op diff AWS rejects — set explicitly (e.g. `["0.0.0.0/0"]`).
 - Codified chart/terraform changes live in the **local** chart, not the published `onyx/onyx` — install from `.` until released.
 - LLM provider: configure via admin UI (encrypted in DB) — never `GEN_AI_API_KEY` in a ConfigMap.

docs/group-manager-scoped-permissions/00-index.md

@@ -26,7 +26,7 @@ live) · D2 admins-only create groups · D3 admin-or-manager-of-that-group assig
 everything **except delete** · **D7 attaching an agent to a group is controlled by `manage:agents`** (standard GATE 2 keyed on
 `MANAGE_AGENTS` — admins/global holders self-share to their groups, scoped managers to managed groups;
 `add:agents`-only users can't group-share). The reviews confirmed PAT,
-chat-runtime, and document/Vespa ACL are untouched, and refuted the backfill data-loss concern. Full
+chat-runtime, and document-index ACL are untouched, and refuted the backfill data-loss concern. Full
 case-by-case coverage + the boot-bug prerequisite are in **[03 §11](03-detailed-design.md)** — the
 authoritative implementation checklist.
 

docs/group-manager-scoped-permissions/02-high-level-design.md

@@ -206,6 +206,6 @@ The gate therefore loads the resource's **current** groups in the same transacti
 
 - Dropping `role` / `UserRole` / `is_curator` and migrating the 3 residual `user.role==ADMIN` readers
   (`persona_sharing.py:53`, `build_session.py:638`, `search/api.py:104`) — deferred cleanup release.
-- Any change to document-level Vespa ACL (`get_acl_for_user`) — unchanged; managers affect entity-level access
+- Any change to document-level document-index ACL (`get_acl_for_user`) — unchanged; managers affect entity-level access
   only.
 - CE behavior — Group Manager is an EE capability (custom groups are EE); CE has only Basic + Admins.

docs/group-manager-scoped-permissions/03-detailed-design.md

@@ -654,7 +654,7 @@ managed scope + PRIVATE); reject otherwise. Admins unaffected (global bypass).
   filter edits inside the `if get_editable:` block as correlated subqueries — do not perturb the shared joins
   above the split.
 - **Junction-only:** scope is purely `User__UserGroup.is_manager` + resource↔group junctions;
-  `permission_grant` stays global-only; Vespa ACL untouched. Backfill is safe — no upgrade migration nulls
+  `permission_grant` stays global-only; document-index ACL untouched. Backfill is safe — no upgrade migration nulls
   `role`, and `native_enum=False` stores `'CURATOR'`/`'GLOBAL_CURATOR'` literally. Ship the
   zero-managed-group migration report for the GLOBAL_CURATOR snapshot caveat.
 

docs/group-manager-scoped-permissions/04-implementation-plan.md

@@ -73,7 +73,7 @@ PRIVATE and strictly within their managed groups — enforced authoritatively at
   - **Corrections (§11.7):** feedback `db/feedback.py` = **no change** (admin-only; not in bundle);
     `recompute_user_permissions__no_commit` takes `(user_ids, db_session)` and must be extended to set
     `is_group_manager`.
-  - **Confirmed SAFE (§11.8):** PAT cap, chat runtime, and document/Vespa ACL are untouched — keep them so.
+  - **Confirmed SAFE (§11.8):** PAT cap, chat runtime, and document-index ACL are untouched — keep them so.
 
 ## Implementation Strategy