Run details

aiWorkspace

CI Pipeline Executions
- Current executionSucceeded
  fix/ai-mcp-widget-approval
- Runs and Agents
- TimelineBeta
- Configuration
- Self-Healing CI
- Project Graph
- Circular DependenciesEnterprise
- Resource Usage
- Analysis
- ConformanceEnterprise
Runs

Insights

Compare tasks
Analytics

feat(ci): coverage regression gate vs merge-base (#1036) * feat(ci): coverage reporting with a committed baseline ratchet Adds a `test:coverage` script to every package and a `Coverage` job to the PR workflow that fails when coverage drops. Coverage was previously unrunnable: `@vitest/coverage-v8` was pinned to 4.0.14 in all 50 packages while `vitest` resolved to 4.1.10, so vitest could not resolve the provider at all (`ERR_MODULE_NOT_FOUND`). The two existing `test:coverage` scripts (ai, ai-client) could not have worked. Aligning the provider on 4.1.10 fixes it. - every package gains `test:coverage`, measured over `src/**` with json-summary + text-summary reporters (flags live in the script so no per-package vitest config changes were needed) - `scripts/coverage-check.mjs` diffs the fresh summaries against `coverage-baseline.json` and exits non-zero on a >0.5pp drop in statements/branches/functions/lines; unmeasured packages are skipped rather than counted as 0% - root gains `test:coverage:all`, `test:coverage:check`, `test:coverage:update` - `coverage-baseline.json` seeded from a full run of the 48 packages whose suites pass on this machine - CONTRIBUTING.md documents the workflow and the two known gaps * feat(ci): surface the coverage table on the PR run summary The check previously only printed to the CI log, so the numbers were invisible unless the job went red. Writing the same per-package table with deltas to $GITHUB_STEP_SUMMARY puts it on the PR's Checks tab on every run, pass or fail, with no token and no `pull-requests: write` permission. * fix(ci): baseline the three platform-divergent packages from Linux The first CI run confirmed the gate works — 47 of 50 packages reproduced the locally generated numbers to the hundredth — and surfaced the three that could not be measured correctly on Windows: - ai-acp and ai-sandbox have tests that fail on Windows for reasons unrelated to coverage, so they were missing from the baseline entirely - ai-sandbox-local-process reads ~20pp higher on Windows because its process-spawn code branches on platform, which tripped the gate All three now carry the numbers from the Linux CI run. CONTRIBUTING.md spells out that the baseline must come from Linux, since CI is what enforces it. * docs(ci): make coverage explicitly CI-only Coverage was already absent from `test`, `test:pr`, `test:ci` and the git hooks, but the docs still presented it as part of the local workflow — four coverage rows in CONTRIBUTING's day-to-day command table, and an instruction to re-baseline with `pnpm test:coverage:update` on your own machine. That last one was an active trap: the committed numbers are Linux numbers and a few packages measure differently per platform, so following the doc on a Mac or Windows box reintroduces exactly the ai-sandbox-local-process failure the previous commit fixed. The check now prints a paste-ready JSON block for every package the baseline needs to absorb, in both the log and the run summary, so re-baselining is a copy out of CI rather than a local run. `test:coverage:update` stays as the maintainer escape hatch but is no longer the documented path. * fix(ci): quote the coverage include glob, correct stale lint references Review feedback: - quote `--coverage.include='src/**'` in all 50 test:coverage scripts so a POSIX shell can't expand the pattern before vitest parses it. Verified the quoted form behaves identically on Windows (pnpm normalises it) and yields byte-identical percentages, so it is safe cross-platform. - CONTRIBUTING referenced `pnpm test:eslint`, which does not exist — the repo moved to oxlint. Fixed that and the two other stale ESLint mentions, and added the Coverage gate to the list of what CI runs. * refactor(ci)!: compute the coverage baseline in CI instead of committing it The committed baseline had the maintenance problems it looked like it would: it needed manual syncing, it was platform-sensitive (ai-sandbox-local-process reads ~20pp higher on Windows, which broke the gate on this very PR), and adding or removing a package meant hand-editing a generated file. The Coverage job now measures each affected package twice in one job — on the PR head and on its merge-base with main — and compares those. Nothing to keep in sync, no platform skew (both sides run on the same runner), and package add/remove needs no action. - delete coverage-baseline.json, `test:coverage:check`, `test:coverage:update` and the paste-a-JSON-block flow; all of it existed only to tend the file - coverage-check.mjs gains `--collect <dir>` (snapshot summaries so they survive the checkout between runs) and `--base <dir> --head <dir>` - resolve the package list once, with `--with-target=test:coverage`, so both sides measure the same set and a docs-only PR resolves to an empty list rather than erroring in run-many; the base side intersects that list with the projects that exist there, so PR-added packages report as `new` - new coverage.yml runs test:coverage:all on pushes to main purely to populate the Nx Cloud cache: the base-side task hashes then already exist remotely and, because test:coverage declares its coverage dir as an output, restore without re-running tests Verified locally: collect/compare round-trip, a drop failing with only the out-of-tolerance metrics listed, PR-added packages reporting `new` without failing, removed packages ignored, missing base dir, empty head, bad args. * fix(ci): stage the coverage script outside the working tree The merge-base step ran `node scripts/coverage-check.mjs --collect` after checking out the base commit, where that file does not exist — this PR is what adds it: Error: Cannot find module '/home/runner/work/ai/ai/scripts/coverage-check.mjs' Not specific to this PR either: any PR that adds or moves the script would hit the same thing. Copying it to $RUNNER_TEMP before the checkout fixes it and also guarantees both runs and the comparison use the same version, which the previous `git checkout $GITHUB_SHA -- scripts/...` in the compare step only approximated. The base coverage run itself was fine, at 3/6 Nx cache hits before coverage.yml has ever run on main. * fix(ci): clear coverage dirs before each run so the base side is real The previous run passed for the wrong reason. `coverage/` is gitignored, so checking out the merge base left this PR's coverage files on disk, and the base-side `--collect` read them back: 47 packages were compared against themselves, which is why every delta was exactly 0.00. Both runs now clear `packages/*/coverage` first. Nx restores the directory for cached tasks, so genuine cache hits still work. On this PR the base side will now legitimately collect almost nothing — the merge base predates the json-summary reporter, so there is no base data to compare against and packages report as `new`. That resolves itself once this lands and main has the machinery. * fix(ci): pass the project list via env to satisfy zizmor zizmor flagged both `${{ steps.affected.outputs.list }}` uses inside run blocks as code injection via template expansion (pr.yml:69, pr.yml:85), and it's right: the list is built from package names in the PR's own files, so on a fork PR a package named with shell metacharacters would be interpolated straight into the script. Passing it through env keeps it out of the shell source entirely. Confirmed clean by running zizmor v1.28.0 locally over both workflows. * fix(ci): fail the coverage gate when a side cannot be compared Missing summaries were treated as out of scope, so this PR's Coverage job compared nothing against main. Collect now requires --expect, packages that vanish from head fail, empty head fails, and both CI runs pass json-summary so the merge-base writes coverage-summary.json. * ci: match coverage cache args and cover all public packages - coverage.yml now forwards the same vitest args as pr.yml. Nx hashes forwarded args, so before this the merge-base run never hit the main-branch cache. - Add test:coverage to 13 public packages that came from main without it: ai-cloudflare, ai-compaction, ai-lovable, ai-reactor, ai-remix, ai-sandbox-blaxel, ai-sandbox-boxd, ai-sandbox-e2b, ai-sandbox-upstash-box, ai-skills, ai-typesafe, ai-worldlabs, svelte-ai-devtools. - ai-angular: include ui/src in coverage. - coverage-check.test.ts: import readFileSync and existsSync instead of require. - CONTRIBUTING: correct the cache text and list svelte-ai-devtools as a package with no tests. * ci: build ai-memory and ai-sandbox before test:coverage Their testkit-subpath tests import the built package, so test:lib depends on build. test:coverage only had ^build, so the Coverage job failed on a missing dist. * ci: drop test:coverage from the UI packages that have no tests ai-react-ui, ai-solid-ui and ai-vue-ui report 0 statements, so the coverage collect step failed on them. Packages with no tests do not gate the job. * ci: print coverage task logs in the job output Nx sends failed task logs only to Nx Cloud, so the GitHub log shows no error. The static output style prints each task's output in the job log. It is an Nx flag, not a forwarded arg, so the task hash does not change. --------- Co-authored-by: Tom Beckenham <34339192+tombeckenham@users.noreply.github.com>

nx run-many --targets=build --exclude=examples/**,testing/**

Succeeded

4 CPU cores

read access token used

b447d60c1508

99%

Cache hits

69 of 70 tasks used cache.

Get faster results

Learn how to enable distribution.

Atomizer enabled

70 groups of tasks optimized.

© 2026 - Nx Cloud

Terms of Service Privacy Policy Changelog Status Docs Contact Nx Cloud Pricing Company @NxDevTools

aiWorkspace