In August 2025, a data pipeline at Tailscale threw an error against one of its control plane databases. An engineer ran PRAGMA integrity_check, and the database came back corrupt. Over the following six months there were nineteen more corruption events, each taking a shard of the control plane offline while it was restored, with affected tailnets losing their admin console, their API, and the ability to learn about network changes.
The cause was a data race in SQLite’s checkpointing logic that had been sitting in the codebase since version 3.7.0, released on 21 July 2010. Sixteen years, in a project with roughly 92 million lines of test code, widely regarded as the most thoroughly tested open source software in existence.
The interesting question is not how a bug survived sixteen years in SQLite. It is why Tailscale was apparently the first organisation to work out what was happening.
TL;DR
- The SQLite WAL-reset bug affects every release from 3.7.0 (July 2010) through 3.51.2 (January 2026), and is fixed in 3.51.3 (13 March 2026), with backports to 3.44.6 and 3.50.7.
- It is a data race between a checkpoint and a concurrent WAL reset that causes silent, permanent corruption. It requires WAL mode, two or more connections on the same file in separate threads or processes, and writes or checkpoints landing at the same instant.
- SQLite’s own assessment puts the occurrence rate in the wild at or below the rate of SSD malfunctions and cosmic ray bit flips. The developers were never able to reproduce it organically.
- Tailscale found it because a backup pipeline ran integrity checks as a routine stage. Nearly everyone else who hit it over sixteen years almost certainly blamed the disk.
- The lessons are about detection, not SQLite: verify backups rather than merely taking them, inventory the database engines embedded in your dependencies, and treat non-standard configuration of boring technology as a risk you have consciously taken on.
What actually went wrong
In write-ahead logging mode, SQLite does not write changes directly into the main database file. Modified pages are appended to a separate WAL file, and a checkpoint later copies them back into the database, a process called backfilling. Once everything is backfilled and no reader needs the old content, the WAL is reset and reused from the start. Coordination depends on a salt value in a shared memory index, incremented on each reset.
The bug was that a checkpoint would read the WAL header, and a concurrent writer could then reset and wrap the WAL underneath it. The checkpoint carried on with its stale copy of the header, backfilling against a layout that no longer existed. It copied the wrong pages into the main database file and updated its progress counter as though it had done the right thing. Nothing errored. The database was simply, quietly wrong.
On 3 March 2026 a SQLite developer found and fixed it. The fix is a single comparison: check the salt has not changed since the header was read for this checkpoint, and abandon the backfill if it has.
Why it hid for sixteen years
Timing, first. The race window is narrow, and the SQLite team could never trigger it deliberately. They had to write test instrumentation that forced the interleaving in order to confirm the fix worked at all. Then the shape of the test coverage: alternative VFS implementations commonly run in exclusive locking mode, which sidesteps the code path entirely, so only the standard Unix and Windows layers exercised it, and only under genuine multi-connection concurrency.
Third, and the one worth sitting with: attribution. When a database comes back corrupt once, on one machine, with nothing in the logs, the overwhelmingly likely explanation is hardware. Bad disk. Bad memory. Filesystem misbehaving under a power cut. That is the correct prior and it is almost always right. SQLite now estimates the wild occurrence rate of this bug as no higher than that of SSD failures or cosmic ray strikes, which means that for sixteen years everyone who hit it had a perfectly reasonable explanation to hand, and used it.
Tailscale was exposed by a choice they made
Tailscale’s control plane takes manual control of checkpointing rather than letting SQLite handle it, so it can run fast and consistent backups, and it checkpoints very aggressively. That is documented, supported behaviour. It is also a long way off the path most SQLite deployments take.
Run an operation rarely and a one-in-many-millions race is theoretical. Run it constantly and it becomes an operational certainty. Tailscale did not find this bug because they were unlucky, but because they had turned a rare event into a frequent one, at scale, for years.
Their own conclusion travels well beyond databases: running boring technology in a non-standard way is a risk. The value of boring technology is the accumulated operational knowledge of everyone else running it the same way. Step off that path and you inherit the edge cases alone.
The part that should worry you more
Tailscale caught this because integrity verification was a stage in their backup pipeline, not a step in incident response. Automation found it, on a schedule, before a customer reported anything. Most teams have backups. Far fewer have a job that restores one, runs a structural integrity check against it, and alerts when the check fails. A backup you have never verified is not a backup. It is an untested assumption with a storage bill attached.
Silent corruption is the failure mode that defeats conventional monitoring. Uptime checks pass. Error rates stay flat. Latency graphs stay clean. Nothing in a standard observability stack is designed to notice that a row contains different bytes than it did yesterday. By the time it surfaces it has usually propagated into every backup inside your retention window, which is where a recoverable incident becomes permanent data loss.
You probably do not know which SQLite you are running
SQLite is not a database you choose. It is a database you inherit. It ships inside Android and iOS, inside every Electron application, inside browsers, inside the Python standard library, inside countless applications that bundle their own copy. Plex Media Server, to take one public example, bundles SQLite 3.39.4, squarely inside the affected range.
This is the same blind spot we have written about with embedded Chromium. Software composition analysis tooling reads your manifest files. It does not read the C library statically linked into a binary dependency three levels down. Your inventory says you depend on a package. It does not say that package contains a 2019 build of SQLite.
If you operate anything where SQLite holds durable state (edge deployments, sync engines, mobile clients, desktop applications, CI caches, local-first architectures) the audit is worth an afternoon: which components embed SQLite, which version each ships, whether they run in WAL mode, and whether more than one connection ever touches the same file.
A note on formal methods
There is a satisfying postscript. When the bug was disclosed, Canonical needed to know whether dqlite, their distributed SQLite derivative, was affected. Rather than reason informally, they wrote a TLA+ specification of SQLite’s locking and checkpoint behaviour. The model checker produced a counterexample in twenty states: a bug that sixteen years of production use and 92 million lines of tests never surfaced, reproduced almost immediately once the concurrency model was described formally rather than sampled empirically. dqlite turned out to be unaffected, because it takes a write lock before checkpointing.
The lesson is not that everyone should write TLA+. It is that tests sample a state space and formal models search it, and for concurrency bugs that distinction is the whole game.
What to actually do
- Upgrade. SQLite 3.51.3 or later, or the 3.44.6 and 3.50.7 backports if you are pinned. Then check what your dependencies bundle, because that is where the old versions hide.
- Make integrity verification a pipeline stage. Restore a backup on a schedule, run a structural check, alert on failure. Continuously, not quarterly.
- Measure detection latency. If a row silently changed today, how long until something told you? If the answer is “a customer would email us”, that is your number.
- Audit your non-standard configuration. Every place you have taken manual control of something a mature system normally handles itself is a deliberate trade, and should have a named owner who knows why.
- Extend your dependency inventory to embedded runtimes. Database engines, browser engines, media codecs, TLS libraries. The things that arrive inside other things.
The real takeaway
SQLite did not fail here in any meaningful sense. A sixteen-year-old race with an occurrence rate comparable to cosmic ray bit flips, found and fixed in one line, is an argument for boring technology rather than against it. What failed, for sixteen years, was detection. Tailscale’s advantage was not better engineers. It was a pipeline that checked its own work, and the willingness to keep investigating after the easy explanation had already been offered to them nineteen times.
At REPTILEHAUS we build and operate production systems where data integrity is not negotiable, from DevOps pipelines and backup verification through to database architecture and dependency governance. If you are not certain how quickly you would notice silent corruption in your own stack, that is a good conversation to have before you need to have it. Get in touch.
📷 Photo by Winston Chen on Unsplash
