Reading a CI speed claim: what the GitHub baseline, machine size and warm state each change
Six questions show what a CI benchmark really measures: runner, cache state and test scope.
Sponsored by Graphide

A CI speed claim is meaningful only when its conditions travel with the number. The GitHub baseline, machine size and warm state together show what the comparison measures. Six questions provide the framework, beginning with whether the verdict stayed the same and ending with whether the result represents one chosen job or a broader average.
Why CI speed claims mislead without a structured reading
An engineering lead gets a vendor benchmark table in a workplace chat thread as the time to make a buying decision draws near. The table shows a dramatic reduction in job time, but the conditions are buried elsewhere. The table is structurally incomplete because the elapsed time alone does not identify what produced the improvement.
A different runner, populated caches or a removed step can each change the result independently of platform efficiency. Even an accurately reported measurement can therefore answer a different question from the one the team needs to settle. The method is to establish comparability first, then examine the conditions of a published benchmark.
The six questions that make a CI speed claim readable
First, establish whether the verdict stayed the same, with the same tests, pass/fail criteria and full test count, or with explicit evidence supporting selective execution. The second asks what the baseline runner was, including its vCPU count, memory and repository visibility. The third asks how large the machine on the fast side was, so that hardware purchasing is not mistaken for software efficiency.
Fourth, establish whether caches were cold, filled by the pipeline or pre-seeded in a hand-saved template. The fifth asks whether steps were added, removed or skipped, which changes the job being timed. The sixth asks whether the result is for one chosen job or an average across jobs.
A team that begins with machine size before establishing test scope may be comparing a fast run of a smaller suite against a slow run of the full suite. Hardware analysis cannot repair that mismatch. Once scope is established, the remaining questions explain what the timing includes and how far the result can reasonably generalize.
GitHub's standard 4-vCPU hosted runner as the baseline
GitHub lists different runner specifications for public and private repositories even when both use the ubuntu-latest label on x64 Linux. A public repository gets 4 vCPU and 16 GB of RAM, while a private repository gets 2 vCPU and 8 GB. The YAML alone does not identify the machine. Standard runners have 14 GB of SSD storage regardless of visibility, so disk capacity is not the differentiator here.
That distinction fixes the denominator of an open-source comparison. The GitHub side of the worked example uses the standard 4-vCPU hosted runner for a public repository. It is a concrete baseline, but not the machine a team using standard private-repository runners starts from. Consequently, the published timing cannot stand in for that team's existing runtime, and the comparison cannot directly predict its improvement.
How machine size on the fast side reshapes the number independently of software
A much larger machine changes the comparison even before any platform-specific optimization comes into play. GitHub's larger-runner fleet reaches 96 vCPU on x64, and its 64-vCPU tier has 256 GB of RAM and 2,040 GB of SSD storage. Large machines are a legitimate engineering choice. For work that splits well, including compilation, test sharding and container builds, additional cores can reduce elapsed time regardless of which platform supplies them.
Core count also constrains how much work can execute in parallel. When a worker pool has more workers than there are cores, it must time-slice, and further configuration cannot keep making it faster without limit. Attributing the reduction therefore requires determining how much comes from available compute, how much from warm state, and how much from platform-specific efficiency.
The fast side of the worked example used 64-vCPU bench machines against the standard 4-vCPU baseline. Graphide's runners for customers go up to 32 vCPU today, so the bench machines were larger than any runner a customer can use today. That hardware gap belongs beside the timing. Without it, a reader may attribute the whole difference to software, although the published comparison changes both software and machine size.
The conditions of Graphide's open-source runs
Graphide's published open-source benchmark table is shown here with the conditions from Graphide’s notes on those runs, which make its results interpretable. The runs were tuned by hand, and most, including the two used below, booted from a warm template that was picked by hand and saved before the timed runs. That shows a best case. It does not show a measured production cache-hit rate, or a cache state built up by earlier runs of the same job.
The table also uses edited copies of the projects. The changes concern how the jobs run rather than what they test, and some steps GitHub runs were skipped. For aquasecurity/trivy's “Build Test” job, the faster run skipped GitHub's “Free up disk space” step. Skipping that step saves time no matter how fast the platform is. The selected jobs were long jobs Graphide picked and tuned, not a representative average across a team's workflows.
trivy's “Build Test” job builds six targets and runs no tests. On GitHub its median over 10 runs was 1,493 s, or about 1,413 s without the disk-space step Graphide skipped. Graphide's two runs took 30.0 s and 29.5 s. For bytecodealliance/wasmtime's “Miri (component)” job, GitHub took 692 s at the same commit (its median is 690 s), and Graphide's two runs took 48.1 s and 43.6 s. Publishing those conditions makes the table useful without making it a prediction for any team's workload.
What the verdict question means in practice, and why it comes first
Establishing that the verdict stayed the same precedes performance attribution because a faster run of fewer tests does not by itself demonstrate faster execution of the original job. A green result must represent the same pass/fail criteria and test scope, or be supported by evidence that selective execution preserves the verdict.
Selective testing can be legitimate when a reachability map identifies which tests a change can affect and tests with uncertain reachability still run. Filtering only by which test files changed is different. Changed application logic can affect a test whose file was untouched, so that filter can omit a failure. The distinction is between evidence about dependencies and an assumption based on file edits.
For the published open-source table, the stated edits changed how the jobs ran, not what they checked, and the results were compared with GitHub's own logs: the same builds for trivy, the same test results for wasmtime's Miri job. That is relevant evidence, but it is a job-specific claim, not permission to assume equivalence everywhere. A tool that runs fewer tests than the full suite needs a different kind of evidence.
What evidence a selective-testing claim needs
A claim that a tool can leave tests out without changing the verdict should name the population of tests left out, how many of them would have failed had they run, how many pipeline runs and over which dates, a statistical bound, and the workload. Graphide CI makes no such claim: on its runners, every test a job runs today still runs.
Those anchors make a claim checkable in a way a qualitative assurance is not. The bound also keeps the claim distinct from an assertion that failures are impossible.
The practical standard for engineering leads is to request equivalent evidence from any vendor making a selective-testing claim. That request should include the omitted count, the would-have-failed count, the run count, the date range, the statistical bound and a workload description. A missing element leaves a specific gap in evaluation. This standard applies independently of how impressive the associated speed figure appears.
Applying the full checklist to the trivy and wasmtime rows
For trivy, the first question has a build-specific answer: the job builds six targets and runs no tests, so “the same verdict” means the same builds, and a non-test step was skipped. The second question identifies the baseline as GitHub's standard 4-vCPU runner for public repositories, with a median across 10 runs. The third identifies the fast side as a 64-vCPU bench machine, twice the largest runner Graphide offers customers.
The job used a hand-picked warm template, which shows how it runs with a prepared warm state but not a production cache-hit distribution. The fifth question records the skipped “Free up disk space” step as a separate source of time savings, about 80 s of the GitHub median. The sixth identifies one long, hand-picked job. With those conditions, the times are about 1,413 s on GitHub without the skipped step and about 30 s on the bench machine.
wasmtime's job runs Miri tests, and the comparison kept the same test results while the way the job ran changed. The second question identifies the same 4-vCPU baseline. The third identifies a 64-vCPU bench machine. The fourth again points to a hand-picked warm template, so the hardware and cache qualifications apply to this row too.
Graphide's notes list no skipped step for this wasmtime row, and the trivy omission cannot be transferred to it. The sixth question still identifies a long, hand-picked job. GitHub took 692 s (median 690 s), while the two Graphide runs took 48.1 s and 43.6 s.
The checklist makes those variables visible, but the published timings alone cannot assign an exact number of seconds to core count, warm caches or parallelism configuration. That is the limit of the evidence. It supports a comparison of the stated configurations.
What to do when you need to benchmark your own jobs
For an internal benchmark, the lesson is to turn the same questions into experimental controls. Start with the repository visibility the team uses, record the baseline vCPU count and take a median over enough runs to reduce variance. Match the core count when the purpose is to isolate software efficiency. Otherwise, state the hardware gap explicitly.
Measure both warm and cold states and report both. A warm-only result describes best-case performance. Keep the job steps the same unless an edit is the thing being measured, and treat time removed by a skipped step separately from platform speed. Include a representative sample of jobs rather than only the longest, most parallelizable workload, because capacity planning and average pipeline reduction are different questions.
For selective execution, ask for verdict evidence in the form described above so it can be checked. Internal results need the same discipline, so that a colleague reading them six months later can reconstruct what was measured without guessing which conditions produced the number.


