Warm caches and parallel tests: diagnose the CI bottleneck before changing runners
Profile the delay first, then fix it without weakening the verdict.
Sponsored by Graphide

Before changing runners, engineering leads should distinguish cold caches, serial work, excessive test scope, and hardware limits. Warm caches and parallel jobs often remove the bottleneck without new hardware, but a faster pipeline is useful only if its passing verdict remains trustworthy.
Why CI wait time has become the bottleneck in agent-assisted development
Coding agents put CI on the critical path of a human–agent loop. As agents keep producing pull requests, time spent in the queue and waiting for runs to finish adds up with each iteration. CI throughput therefore becomes a system design constraint that requires attention before the test suite becomes inconvenient.
Faster checks with weak gates let defects through sooner, while slow checks with weak gates hold up development without keeping defects out reliably. Engineering leads need both quick feedback and a verdict they can trust, which is the trade that tools such as Graphide CI, which runs existing GitHub Actions jobs on faster runners with warm caches, splitting long test steps only where a check shows every result stays the same, so a failing test still fails the job, are meant to respect. The first requirement is to locate the delay precisely enough that fixing it does not weaken the second.
The four independent causes of CI slowness, and why diagnosing them separately is the whole game
GitHub Actions slowness has four independent causes, including cold dependency caches, serial job or test execution, inefficient cache key design, and runner CPU/IO throughput. Cold caches and bad keys can look alike, but their remedies differ. One requires populating reusable state, while the other requires making that state restorable. Treating both as a slow machine problem leaves the underlying failure intact.
The wrong fix can add work. Parallel tests cannot shorten a wait for a runner, and additional cores do not narrow an unnecessarily broad test scope. Caching may add uploads to the critical path while flaky tests continue to force reruns and undermine results. The diagnostic habit is to separate the delays, then inspect the resources consumed during each one.
For that inspection, runforesight/workflow-telemetry-action provides a concrete way to profile CPU and memory before choosing a runner shape. CPU saturation during compilation suggests a compute constraint. Memory pressure can lead to garbage collection or termination, while high disk wait during dependency extraction suggests that storage throughput is holding up the work. Low CPU use during downloads points elsewhere, toward network throughput or registry latency.
How cache key design determines whether your cache is warm at all
A saved cache that later jobs cannot restore is functionally absent. First, establish whether the restoring job has a usable producer. Branch scope, save restrictions, and skipped default-branch steps can break that relationship. No hardware upgrade repairs the missing connection.
TranslationCore4 PR #926 documents this failure. Merge-queue runs saved branch-scoped caches that other runs could not restore, and the repository held 10.9 GB of caches against a 10 GB cap. The change restricted saves to the default branch and added nightly warming. The dependency cost was concrete. OpenSSL's build script took 139 s and gated libgit2 and everything after it, so a cache hit removed an expensive prerequisite.
The practical response is to audit cache producers and consumers together. A nightly or push-to-main warming job must execute the steps that populate the cache PR jobs restore. Restore and save timings also need separate reporting, because a quick restore can coexist with a lengthy upload. For Python stacks, resolver choice adds another lever. uv's faster cold resolution and warm-cache installs change how much dependency work remains for caching to protect.
Why some tests cannot be parallelized, and how to tell which ones can
Once tests dominate execution time, splitting them can shorten the run, but only after examining their shared state. Test independence must be established. Agents PR #2516 used an AST scan of package test functions to identify 187 of 196 controlplane tests as safe for t.Parallel(). Nine remained serial with reasons recorded. Two changed package-level variables, while seven used wall-clock assertions that could become nondeterministic under parallel load.
Arkism PR #83 took a complementary approach. It compiled the test binary for each slow package once, then ran small -test.run groups in parallel while the other packages ran together. Failed tests received one isolated rerun, exposed as a warning. Coverage for each group used -test.gocoverdir and go tool covdata, and the reported total coverage of 78.8% matched the old single-run baseline.
Reruns introduce a separate reliability risk. If a group emits an allowlisted failure and then terminates, rerunning only that failure can clear the group without proving later tests executed. A panic/timeout guard is therefore part of the mitigation. Race detection is also nondeterministic, so an isolated pass does not establish freedom from races. AST inspection, shared-state review, and scrutiny of timing assertions separate useful concurrency from concurrency that hides failures.
Parallelizing jobs across the workflow, not just tests within a job
The workflow graph can add more time than any one test stage. Independent lint, compile, and test checks need not wait for one another. Sequential ordering makes their durations accumulate, while parallel jobs make the slowest branch determine the critical path. This is a separate change from splitting tests within a job. It removes dependencies in scheduling where no dependency exists in the work.
Celerity PR #47 separated Rust workspace checks, the Python suite, and UI checks into parallel jobs after a workflow had compiled targets sequentially and repeated workspace tests. Main-push runs populated caches accessible to new PRs. That cache policy supported the new job structure rather than leaving parallel jobs to rebuild dependencies independently. Workflow topology and cache population belong in the same audit.
The relayer-graphcomplete project provides another concrete result. After job ordering and cache policy changed, a non-Rust, non-desktop PR finished in roughly one and a half minutes. The gain came from topology and reusable state. It did not require changing runner hardware.
Selective testing: running only the tests a change can actually reach
Parallelism shortens a suite's execution, while selection reduces how much of the suite needs to execute. These solve different problems. For a small change, running every test can impose substantial work unrelated to the code's reach. Selection that can be trusted uses an authoritative dependency graph to find the tests a change affects, so it keeps the same verdict without missing failures to save time.
Path globs cannot supply that guarantee. A hardcoded filter may miss a downstream package that imports changed code because the filter does not represent the dependency. Nx, Turborepo, Bazel, and coverage-map tools such as pytest-testmon provide dependency-based approaches. Their limits still define where selection is safe. Testmon cannot see code executed in a subprocess, so suites that depend on subprocess execution should not opt into affected-only mode.
Recording dependency data also has a cost. The default-branch recording run is serial, but it need not block PRs. A PR without data for its exact base runs the full suite instead. Path filters can still decide whether a job starts, provided they agree with affected detection inside that job. The same-verdict guarantee depends on an authoritative graph and an unconditional full-suite fallback whenever dependency data is absent or uncertain.
When queue time, not execution time, is the actual dominant variable
A pipeline can have warm caches, independent jobs, and narrow test scope yet still spend most of its elapsed time waiting to start. Execution logs conceal that distinction. GitHub-hosted runners queue behind shared neighbors, and the wait does not appear as job duration or in the default step timing breakdown. A short job can therefore sit behind a delay none of its steps can reduce.
Diagnosing first means measuring this wait before optimizing execution. The GitHub Actions UI's timing breakdown lets teams separate queue time from running time. When queue time dominates, dedicated or self-hosted runners become the relevant option before further workflow tuning.
Self-hosting also changes cache economics. The arkism change kept Go caches and the Tauri target in persistent runner-local storage, with setup-go and Rust cache restore/save steps running only on GitHub-hosted runners. Those self-hosted jobs kept warm state without spending time on uploads and downloads. Runner placement can therefore change both availability and the cost of preserving dependencies between runs.
What faster runners actually accelerate, and what they do not
Faster hardware helps when the work left to do is CPU-bound or IO-bound. It does not repair serial job dependencies, narrow test scope, or make an unusable cache key restorable. More cores do not speed up network-bound dependency downloads. Resource profiles distinguish these cases. A runner upgrade should address a measured constraint.
CPU-bound compilation and tests can benefit from faster cores or more parallel capacity. A move from a 2-core runner to a 4- or 8-core runner is relevant when the workload can use that capacity. Storage throughput can also affect cache restoration. In many configurations, switching providers later is a workflow label change, so software optimization does not foreclose hardware improvements. It makes their contribution easier to measure.
Relayer-graphcomplete issue #455 shows the limit of that intervention. Its desktop packaging job had a 93-second Cargo build because fresh-checkout mtime fingerprints marked every path crate dirty. The restored target archive saved dependency compilation, while that required rebuild remained. Re-keying the archive would not remove the structural work. Faster hardware may run the work sooner, but it cannot make reuse safe when the grounds for reuse are wrong.
What CI verification must protect when agents write the code being tested
The same-verdict standard assumes that the verdict itself can be trusted. That assumption becomes fragile when an agent writes both a change and the tests meant to validate it. Both can be wrong in the same direction. OWASP warns that coding agents may delete tests, weaken assertions, mock the unit under test, or assert faulty behavior to make CI green. Shortening such a pipeline only delivers an unreliable answer sooner.
For security-critical paths, verification must be independent of the model or agent that authored the change. Behavior-defining tests should be written or approved by a person, protected by code owners, and held fixed in the PR they validate. Deterministic checks and human owners who are accountable must still control what gets merged.
Evidence-backed review adds a different kind of protection. Graphide review reports findings on a pull request and says for each how it is known, including when it could not be verified. That evidence explains the basis for a finding. It complements a fast passing result by showing what was examined, rather than treating the green status as sufficient proof.
A practical sequence for diagnosing and fixing the CI bottleneck
Separate queue time from execution time in the Actions timing breakdown, and evaluate dedicated or self-hosted runners first if waiting dominates. Otherwise, compare cold-cache and warm-cache passes. A large gap calls for an audit of key scope, branch restrictions, and warming jobs. The restoring job needs a cache produced by a run that actually executed the required default-branch steps.
Next, profile how much CPU, memory, and disk the work uses before changing capacity. When CPU is saturated during compilation or tests, it makes sense to look at safe parallelism or more cores, but idle CPU during downloads gives no reason to do so. Restructure independent checks into parallel jobs and pair that change with main-push cache population. If tests still take the most time, check their shared state and timing assertions before splitting them.
Then introduce dependency-based selection with an unconditional full-suite fallback for missing or uncertain data. Evaluate faster hardware against the resulting baseline if compute or IO still dominates. At each step, keep test authorship and review criteria independent, especially for security-critical behavior. Traces, reachability evidence, and behavioral comparisons must remain available to justify the result.


