Agent Architecture Weekly

Trying a test selector on a GitHub Actions job in shadow mode before trusting it

Shadow mode lets a test selector prove itself before it can skip anything.

Sponsored by Graphide

Agent Architecture Weekly Editors
· 8 min read
Cover illustration for “Trying a test selector on a GitHub Actions job in shadow mode before trusting it”
Features · November 13, 2026 · 8 min read · 1,736 words

A test selector on a GitHub Actions job should prove itself in shadow evaluation before it gets to decide what runs. That means keeping the full suite as the required check while recording the selector’s choices, reconciling every exclusion with actual results, and testing its response to uncertainty.

Why test-selection tools need a qualification period before they gate CI

A selector can produce a green check while withholding the test that would have exposed a regression. A path-glob approach can miss a downstream test whose own file has not changed. A graph-based approach can miss a dependency it never modeled. Neither mistake necessarily causes a job failure. The regression can therefore reach production without the suite ever testing the affected behavior on the branch.

That risk grows when pressure for faster feedback outruns the evidence for safe selection. Agents write code faster than full suites verify it, making CI throughput a direct constraint on development. Shadow mode separates the evaluation of a selector from permission to skip tests. The full suite supplies the ground truth, while the selector records what it would have chosen for the same change. Promotion then needs its own explicit approval after setup.

What shadow mode means in a GitHub Actions job

Shadow mode runs the selector’s planning pipeline while leaving the full-suite test step as it is. The selector computes its selection, records every exclusion and its reason, and publishes that information to a job summary or artifact. The existing test command still runs the whole suite, and its pass/fail verdict stays the same.

The mechanical boundary is more important than the name of a flag. A step described as “shadow” is not observational if its output controls whether the test step executes. In the intended arrangement, no required check changes name or status, and no job is skipped because of the selection. The output is evidence. Across commits, that evidence becomes a record of how the selector interprets changes.

Synapsium Labs’ ccrc-pwa shows this separation. It ran with CCRC_SELECTION set to shadow in ci.yml (PR #183), so the selector computed and reported its choice while CI still ran everything, and a one-line pull request (#211) switched it to enforce after a history replay was reported.

Running the selected subset and the full suite side by side before trusting the selector

For every pull request commit, the evaluation should record two signals that can be compared. One is the full-suite result, which remains the required check. The other is the selector’s proposed subset, including its exclusions. These can come from parallel jobs, with the advisory job kept out of the dependencies of required checks. Running a selected subset in that advisory job does not replace the independent full-suite result.

The comparison artifact should connect the proposed selection to what actually failed in the full run. For each failing test, the record must show whether the selector would have included it or left it out. An excluded failure is the decisive discrepancy. Publishing this comparison to the job summary makes it visible during PR review rather than leaving engineers to reconstruct it from separate artifacts. Both signals must describe the same commit so the comparison can show whether the selector would have excluded a failing test.

The qualifying period also needs a minimum set of runs that covers the repository’s meaningful change patterns. A long sequence of isolated feature edits does not substitute for shared-infrastructure changes, schema changes, and dependency updates. Throughout evaluation, the full suite remains required on PRs and runs on the default branch and every promotion to a release branch. After enforcement, a nightly or per-merge full run keeps that safety net in place. The ccrc-pwa plan illustrates that arrangement with a daily full run on main and a green full-suite requirement for promotion to stable.

Fail-closed behavior when the selector cannot be sure

When the selector cannot establish which tests a change reaches, it must run the full suite or expand conservatively. Uncertainty is not evidence that a test is unaffected. The shadow logs should state what was uncertain and why it led to each fallback.

For an audit, the relevant cases include an empty or malformed diff, an unseen changed path, and a change to CI or selector configuration. Dependency manifests, lockfiles, fixtures, and test helpers also require a full run or conservative expansion. Any internal error belongs in the same category. The audit must check both whether the selector reports a problem and whether its proposed selection responds safely.

jposluns / guardrails explicitly names empty diffs, bad path shapes, CI or selector files, unclaimed paths, and paths claimed only by ** as fail-closed triggers. Unreviewed entries always run. keyorixhq / keyorix PR #3781 similarly resolves doubt to KEEP, including timeouts, build failures, undetermined lines, and tests importing os/exec. An unexplained exclusion is a gap caught before enforcement, which is precisely what the qualification period is meant to reveal.

Edge cases that stress-test a selector through lockfiles, fixtures, and config changes

Lockfile updates test whether selection accounts for changes beyond source-file relationships. A dependency version bump can change the behavior of code that calls it even when that code is unchanged. An import graph alone may not express that connection. A shadow record that says “lockfile change triggers full run” makes the conservative response reviewable. A record that excludes tests merely because their source dependencies were untouched exposes the missing relationship.

Fixture edits pose a different challenge because tests can read fixtures through the filesystem at runtime. Import-graph analysis may miss that dependency, so vitest-affected has fullSuiteTriggers for paths that must force a full run, including fixtures read through fs, assets imported through bundler configuration, and generated data. The qualifying corpus should contain fixture changes that exercise this boundary.

Configuration changes can change how the suite runs even when no test file changes. Changes to tsconfig.json, vitest.config.*, package.json, or the workflow itself should trigger a full run. The same tool treats lockfile and config-file changes as full-suite triggers by default. The evaluation should include these changes on purpose and check which tests the selector chooses.

Requiring that every skipped test is listed with a reason

An aggregate skip count cannot support a safety audit. The shadow record must list every test left out and say why it was skipped. A reason such as “not reachable from changed paths” makes a claim that can be compared with the full-suite outcome. If that test fails, the reviewer has both the discrepancy and the selector’s explanation to investigate.

“Dependency graph incomplete for this package” describes uncertainty. Likewise, “lockfile change triggers full run” explains expansion. A log that records these phrases without showing the resulting decision is incomplete. Reviewers need to see how each reason affected the plan.

For any selector, the evaluation needs a summary containing the selection, the exclusions and a reason for each, plus a way to ask about one test against the same commit. For Vitest, vitest-affected’s explain trail option and the vitest-affected-explain CLI show the path from the seed change through graph edges to the selected test.

A concrete shadow setup on a GitHub Actions job

This setup runs two jobs in parallel on every pull request commit. The first is the existing full-suite job, with its test command, runner, and required check name unchanged. The second is an advisory job that runs only the subset the selector proposes, with the same test command. During qualification, the advisory job stays outside needs for every required check.

For a Go suite whose existing command is go test ./..., the advisory job runs the packages the selector chose with the same flags, and a summary step writes the selection log and per-test reasons to the GitHub Actions step summary. Running the advisory job does not, by itself, qualify the selector. The shadow evaluation is the separate comparison between its proposed choices and the complete run.

The summary should expose which tests would run, which would not, and why each exclusion was proposed. Those entries then need to be reconciled with failures from the required job. A reviewer who questions why a test was left out should be able to ask the selector about that test against the same commit.

The advisory job can reveal incomplete explanations, unsafe fallbacks, or excluded failures without suppressing the required suite. Its presence is not approval for enforcement. Approval depends on the accumulated comparison record and the documented response when selection cannot be trusted.

What a safety record looks like

A safety record counts every test run the selector left out and how many of those failed in the matching full run, over a stated number of pipeline runs and dates.

Such a record is bounded by one codebase’s dependencies, fixtures and configuration. Zero observed misses does not establish that the underlying miss rate is zero. A one-sided statistical bound, reported with its conditions, says how large the miss rate could still plausibly be, and is still not a promise.

The useful model is the form of the evidence. Every exclusion must be reconciled with the corresponding full-suite outcome before it can contribute to a safety record. A team’s comparison report should support a similarly explicit statement about its own repository, including the changes covered and any discrepancies found.

How much shadow evidence is enough before promoting the selector to enforcement

Qualification ends when the record covers the relevant change patterns, its exclusions can be audited, and enforcement has a documented fail-closed rollback path. The minimum corpus includes at least one lockfile change, fixture change, CI/workflow file change, and schema or config change. Those cases must appear in the results. More runs still leave an untested dependency boundary untested.

The comparison report must show zero cases where the selector would have excluded a test that failed in the full suite. Any such case disqualifies the current evaluation and restarts it. The buster14a/buster-legacy issue frames the goal as qualifying focused selection against full-suite failures and preventing coverage drift. After enforcement, the ongoing full runs provide the safety net for drift that the initial evaluation cannot rule out forever.

Promotion should be a separate mode change or PR that keeps the workflow structure as it is. Before activation, the rollback path should be documented so that reverting the mode change restores full-suite execution on PRs in one step. Full runs stay in place on the default branch and at every release-branch promotion.

More in Features