CI checks and reports

Run uv sync --locked --dev --all-extras and make check for the same Python checks as the PR matrix. make github-pr-checks tests a synthetic merge against current origin/main; it requires committed changes. Pre-commit uses the same lint/format commands and locked tools. CI uses Python 3.13, uv 0.11.29, and Bun 1.3.11 where applicable.

Gates and focused checks

PR gate succeeds only when every existing Python matrix check and the changed-line convention check succeeds. Frontend gate verifies that selected docs/viewer jobs succeeded and unselected jobs were skipped. Neither gate changes approval requirements. Repository administrators can require these stable check names in the organization ruleset after the workflows are available on the default branch; adding a workflow alone does not configure branch protection. Path-filtered metrics and sandbox jobs are supplemental and should not be configured as unconditional required checks.

make conventions-check PR_BASE_REF=origin/main checks added Python comments/docstrings. Issue IDs in those additions fail; long comment blocks and docstrings are review hints. Existing lint, import boundaries, and public type-completeness checks continue to enforce their own contracts. No convention comments are posted on PRs.

make metrics-check reads four committed replay samples through the public saved-run/report pipeline, verifies sample/scorer/unscored accounting, and checks independently calculated full/partial AUROC and average precision cases. It uses no model calls. The metrics workflow runs the same check against the PR base and proposed merge using each revision's locked dependencies and identical fixtures, publishing JSON and a compact comparison. Head mathematical/accounting violations fail; numerical changes are presented for review. If the base fails the current contract, the summary explicitly marks comparison unavailable and still requires the head to pass, so a fix for a pre-existing bug is not blocked by that bug. Existing browser parity, confidence interval, bootstrap, and configuration tests remain in make check.

make sandbox-check runs the real Docker network-isolation integration checks. Its workflow runs for sandbox/dependency changes. The existing replay jobs still exercise honest and attack trajectories; summaries distinguish first-try matches from retry recovery, and unexpected sample IDs fail. Environment baseline reports compare declared and observed task identities, including equal-sized but different sets.

Description reports and bot replies

One collapsed section in each PR description shows source/test changes, category totals, and current-head workflow results with links to detailed metrics and performance summaries. The writer checks out only the default branch, reads PR data through the API, preserves author prose outside its markers, and serializes updates with preview-link writers. Fork code is never executed by this privileged writer. Counts are path-based size measurements, not quality scores.

Claude automatic reviews skip draft PRs. Explicit maintainer mentions still run normally. If a comment invocation fails or GitHub marks it action_required, the feedback workflow identifies the matching comment and posts at most one status reply in that thread. Reruns update that reply. The caller records the exact comment ID in its run name, so batched reviews do not require guessing from timestamps. The author and publication time must still match the run; pending-review comments use the review submission time. --comment is available only for read-only diagnosis of an older run. For an approval-blocked inline question, a read-only Claude request uses the title and diff to answer in that same reply; it has no tools, executes no fork code, and verifies the head did not move. The full available diff, PR description, and review thread are counted with the selected model's token-counting API. Its Models API limit determines whether the complete request fits, reserving output/thinking capacity; an oversized request fails explicitly without discarding context. GitHub-omitted binary/oversized patches remain marked unavailable. A repeated blocked event does not generate another answer. Provider failures fail the feedback job. Approval remains a maintainer action; recovery never approves a fork workflow or applies a fix. Submitted-review mentions without a comment cannot currently be correlated through this workflow.

Docs cache

Generated CLI/API/catalog JSON is cached by generator and source inputs, locked dependencies, Python/platform, resolved public setting commit IDs, and the live vLLM dataset revision. Symlinks contribute their link target rather than requiring a valid documentation alias. There are no partial-key restores. Settings are pulled at the resolved commits, verified before generation, and the restored artifact must match its input manifest. Setting commits cover their committed environment-revision and dependency lockfiles. APPS pins its dataset revision in source; vLLM reads a moving dataset, so its revision is resolved separately and checked again after generation. Elapsed time alone does not invalidate the cache. Only public catalog data is cached; no settings checkout, runtime, credentials, test verdict, or deployment is reused.

The docs preview consumes the data artifact produced by the frontend docs check in the same run. It still builds/deploys the preview. Production docs and the docs image use the same generation action. A cache hit saves settings installation and generation, while docs lint/build continue to run. Settings changes become visible on the next triggered docs build.

Timing and reliability

The metrics workflow interleaves three base/head measurements of fresh-process CLI startup and saved-run report generation on the same runner. Dependencies are installed first; operating-system file caches are uncontrolled. Reports include medians, ranges, absolute/relative deltas, revision and lockfile identity. These small fixtures measure pipeline overhead, not production-scale or model latency. Timing changes are informational until a stable workload and regression budget are agreed.

The daily CI health workflow summarizes a bounded sample of recent repository runs, successful-run p50/p95, failures, cancellations, reruns, and expensive steps. Its JSON artifact preserves the underlying measurements for 30 days. Workflow elapsed time includes queue/cleanup and is not billed runner time. It posts no issues or PR comments. Replay artifacts separately expose failures recovered by retries; a final green workflow alone does not establish a first-attempt pass rate.

Repository-owned Claude feedback and straj reviews read the model from scripts/ci/models.json. Update its claude value once to upgrade both. Straj review reads that data file from Control Tower's default branch, including when invoked from an environment repository. The organization-wide @claude implementation has its own model setting in linuxarena/.github/.github/workflows/claude-reusable.yml; it does not currently expose a caller model input.