We Measured 141 CI Jobs. A Quarter of the Runner Time Was Setup.

Every team that has let coding agents loose on a repository has watched the same thing happen: more pull requests arrive, each one triggers the same pipeline, and continuous integration starts to feel expensive. The instinct is to make the tests faster. We spent this morning measuring where continuous integration time actually goes, and the tests are not the problem. A quarter of it is spent getting ready to run them.

TL;DR

• We timed 141 successful CI jobs across 12 popular open-source JavaScript and TypeScript repositories using the GitHub Actions API, which exposes per-step start and finish times.
• 25.4% of the measured runner time went on setup and teardown: runner provisioning, checkout, dependency installs, cache restores and post-steps. The median job spent 42 seconds there.
• On jobs shorter than two minutes the setup share was 43.5%. On jobs over five minutes it was 22.1%. 18% of all jobs spent longer setting up than working.
• Setup is a fixed cost per job, so splitting work into more jobs (the standard fix for slow pipelines) multiplies it.
• Eliminating that overhead entirely would buy 34% more CI on the same runner budget, before anyone optimises a single test.

The post that prompted this

Linear published a write-up this week on reworking their CI after their CTO filed an issue titled “CI costs are high”. Their framing is worth quoting because it is precise: agents made shipping code exponentially faster, validating it did not keep up, and every pull request still has to pass through the same gate. They brought pull request wait time from over six minutes to just over five and roughly halved runner time per test, while the test suite kept growing. They are adding around 2,000 tests a week, and agents now write the majority of them.

But the single largest saving in the whole article had nothing to do with making anything faster. Seven small independent checks, each booting a runner, checking out the repository and installing dependencies before doing a few seconds of useful work, were consolidated into two jobs. Based on June usage, that saved roughly 87,000 runner-minutes a month: 11.8% of their total CI spend, for zero improvement in how fast anything ran.

So we measured how common that is

The GitHub Actions REST API returns a steps array for every job, each entry carrying started_at and completed_at. That makes the split measurable on any public repository without permission or instrumentation.

We took each repository’s most recent completed push-event run of a CI, test or build workflow, across 12 active repositories (Vite, Svelte, Astro, Vue core, ESLint, Rollup, TypeORM, Fastify, pnpm, Hono, TanStack Query, Nest), kept only successful jobs, and classified each step by name. Runner provisioning, checkout, dependency installation, cache restore and every automatically injected post-step counted as setup. Anything naming a build, test, lint, typecheck, format or analysis step counted as work, and work words won ties. That produced 141 jobs and 8.8 hours of runner time, with 98.8% of job wall-clock accounted for by step timings.

Aggregate setup share: 25.4%. Median setup per job: 42 seconds. Jobs under two minutes spent 43.5% of their time on setup; jobs over five minutes spent 22.1%. Astro’s Windows build job ran 105 seconds and spent 93 of them getting ready. Twenty-five of the 141 jobs spent longer preparing than working.

Caveats we would rather print than bury: this is open-source JavaScript, one run per repository, successful jobs only. Step classification is a name heuristic, so a job that quietly builds inside an install step is misread. Open-source runs also hit cold caches more often than a busy private repository would. Treat 25% as an order of magnitude for this kind of codebase, not as your number. Getting your number takes about twenty minutes.

Why this is a structural problem, not a tuning problem

Test execution is a variable cost: it scales with how much code you have and how much of it you choose to run. Setup is a fixed cost: it is paid once per job, and it barely cares what the job then does. Almost every CI conversation we join is about the variable cost.

The fixed cost is the one that moves, because it is multiplied by job count, and job count is the thing teams keep increasing. Sharding a suite across more runners is the standard answer to “CI is slow”, and it works: Linear went from four shards to eight and made the critical job roughly 19% faster. But doubling the shards also doubles the setup you pay for. Linear are explicit about this, and it is the most transferable sentence in their post: at 110 to 140 seconds of setup per shard, eight shards would have burned 15 to 19 minutes of runner time on setup alone, more than the tests themselves. Eight shards only became affordable after they had cut setup to around 40 seconds. Before the setup work, four shards cost 8.3 minutes of setup; afterwards, eight shards cost 7.5 minutes.

That is the whole argument. Parallelism is rationed by your fixed cost per job. If you have not attacked setup, you cannot buy speed with more runners, because every runner you add arrives with a tax attached.

Two currencies, and you have to name one

CI is billed in two different units and they trade against each other. One is runner-minutes, which is money. GitHub’s published rates are $0.006 per minute for a Linux 2-core runner, $0.010 for Windows and $0.062 for macOS, plus $0.07 per GB-month for Actions cache storage. A team doing 200 CI runs a day with ten jobs each, at our median 42 seconds of setup, burns about 30,800 runner-minutes a month on preparation: roughly $185 on Linux, $308 on Windows. Real, but rarely the thing that gets a meeting.

The other unit is critical-path seconds, which is developer and agent wait, and it is where the damage actually lands. Sharding buys wall-clock and spends runner-minutes. Consolidating jobs buys runner-minutes and can spend wall-clock. Teams that skip the decision usually optimise whichever number their tooling happens to display.

Arrival rate is what makes this urgent rather than interesting. Queueing behaviour is non-linear: at 50% runner utilisation, average queue wait is roughly one service time; at 80% it is four; at 90% it is nine. Raising your arrival rate by 60% can quadruple the wait. Agents do not make your tests slower. They raise the arrival rate, and the queue does the rest. In our sample, on repositories with ordinary human commit rates, median queue wait before a job even started was 37 seconds, with a 90th percentile of 267 seconds. That is the floor, not the forecast.

What to actually do

1. Measure the split first. Call /repos/{owner}/{repo}/actions/runs/{run_id}/jobs, sum the duration of every step named “Set up job”, “Complete job”, anything beginning “Post “, and every checkout, install and cache step. Divide by total. That ratio decides everything below.
2. Consolidate trivial jobs. Any job doing under 30 seconds of real work is mostly overhead. Merge them and run the tasks concurrently inside one runner. This was Linear’s single biggest saving.
3. Bake the constant into the image. If every shard installs the same Postgres client or the same build headers, that belongs in a CI base image, not in a step.
4. Scope installs to the package under test. Installing an entire monorepo workspace to test one package took Linear from 44 to 73 seconds down to 16 to 18.
5. Delete caches that lose to a rebuild. A cache keyed on a volatile lockfile can take 28 seconds to restore against 7.5 seconds to rebuild, and you pay storage for the privilege. Time both.
6. Get checkout off your gate jobs. Change-detection jobs that block everything else usually need a diff, not a working tree. Capping fetch depth took Linear’s slowest gate from 94 seconds to 20; removing checkout entirely took others from 27 to 7.
7. Decide the currency before you shard. Write down whether this quarter you are buying wall-clock or runner-minutes, because the same change improves one and worsens the other.

The bill for a bottleneck we already named

We argued in July that the constraint had moved from writing code to verifying it. This is the infrastructure invoice for that shift, and it arrives in a form most teams are not set up to read: not a slow test suite, but a constant that was always there and never mattered until something started multiplying it. The fix is unglamorous. It is also available this week, needs no new budget line, and does not depend on anyone writing faster code.

REPTILEHAUS builds and maintains CI and deployment infrastructure for teams shipping at agent pace, and a pipeline audit is usually the cheapest performance work available to a codebase. If your CI bill has moved and nobody can say which part of it grew, get in touch.

📷 Photo by Paul Krüger on Unsplash