mirror of
https://github.com/onyx-dot-app/onyx.git
synced 2026-10-22 19:32:55 -05:00
Justin Tahara
2e6af4ae72
## Description Integration test shards spend ~23% of their time in "Install Python deps" (`uv sync` inside the devcontainer, with no cache). Across all shards that is ~14% of runs-on vCPU-minutes per day (~61k vCPU-min, ~$1.2k/mo). This PR caches the uv download cache for those shards. - `install-python-deps` mounts a host uv cache dir into the devcontainer (`UV_CACHE_DIR=/uv-cache`, `UV_LINK_MODE=copy`, since uv cannot hardlink across bind mounts). - The cache is stored in the RunsOn S3 cache (`runs-on/cache/restore` / `save`), keyed on OS + arch + `uv.lock`, with a prefix restore key. - Every caller restores it. Only `integration-tests` shard 0 saves, and only on a cache miss (new `save-cache` input). So a `uv.lock` change uploads once, not once per shard. - Both steps skip on GitHub-hosted runners, so the 10 GB GitHub cache is not used. Scope: this PR was first a wider change that also moved small jobs to GitHub-hosted runners. That part is dropped. It undid #15792 (Docker Hub rate limits) and the ECR pull-through cache. Only the cache change is left. ## How Has This Been Tested? - `pre-commit` on the changed files: actionlint, zizmor and check-yaml pass. - CI on this PR: the first run misses the cache and shard 0 saves it. Rerun the integration workflow and compare the "Install Python deps" step time on a warm cache against main. ## Additional Options - [ ] [Optional] Please cherry-pick this PR to the latest release version. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
54 lines
2.1 KiB
YAML
54 lines
2.1 KiB
YAML
name: "Install Python deps"
|
|
description: "Syncs Python deps inside the devcontainer into the bind-mounted workspace, so later `uv run --no-sync` steps share one venv."
|
|
inputs:
|
|
devcontainer-image:
|
|
description: "Fully qualified devcontainer image reference (registry:tag)"
|
|
required: true
|
|
save-cache:
|
|
description: "Save the uv cache when the key missed. Set it on one job per run, so parallel shards do not all upload."
|
|
required: false
|
|
default: "false"
|
|
runs:
|
|
using: "composite"
|
|
steps:
|
|
# The RunsOn S3 cache has no size cap. On GitHub-hosted runners runs-on/cache
|
|
# falls back to the 10 GB GitHub cache, so skip it there.
|
|
- name: Restore uv cache
|
|
id: uv-cache
|
|
if: runner.environment == 'self-hosted'
|
|
uses: runs-on/cache/restore@a5f51d6f3fece787d03b7b4e981c82538a0654ed
|
|
with:
|
|
path: ${{ runner.temp }}/uv-cache
|
|
key: ${{ runner.os }}-${{ runner.arch }}-uv-devcontainer-${{ hashFiles('uv.lock') }}
|
|
restore-keys: |
|
|
${{ runner.os }}-${{ runner.arch }}-uv-devcontainer-
|
|
|
|
# The cache is a separate bind mount from /workspace, so uv cannot hardlink
|
|
# across them. UV_LINK_MODE=copy skips the failed hardlink attempt.
|
|
- name: Sync deps in devcontainer
|
|
shell: bash
|
|
env:
|
|
DEVCONTAINER_IMAGE: ${{ inputs.devcontainer-image }}
|
|
WORKSPACE: ${{ github.workspace }}
|
|
UV_CACHE_HOST_DIR: ${{ runner.temp }}/uv-cache
|
|
run: |
|
|
mkdir -p "${UV_CACHE_HOST_DIR}"
|
|
docker run --rm \
|
|
-v "${WORKSPACE}:/workspace" \
|
|
-v "${UV_CACHE_HOST_DIR}:/uv-cache" \
|
|
-e UV_CACHE_DIR=/uv-cache \
|
|
-e UV_LINK_MODE=copy \
|
|
-w /workspace \
|
|
"${DEVCONTAINER_IMAGE}" \
|
|
uv sync --frozen --no-group model_server
|
|
|
|
- name: Save uv cache
|
|
if: >-
|
|
runner.environment == 'self-hosted' &&
|
|
inputs.save-cache == 'true' &&
|
|
steps.uv-cache.outputs.cache-hit != 'true'
|
|
uses: runs-on/cache/save@a5f51d6f3fece787d03b7b4e981c82538a0654ed
|
|
with:
|
|
path: ${{ runner.temp }}/uv-cache
|
|
key: ${{ steps.uv-cache.outputs.cache-primary-key }}
|