Your AI Agent Just Wrote a Migration That Will Lock Your Production Table

On 4 May 2026, GitHub spent 55 minutes serving errors on pull requests. The cause was not a novel attack, a cloud provider failure or an exotic distributed systems bug. It was, in their own words, “a routine online schema migration running against a large, heavily-accessed database table”. The migration saturated connection capacity just as traffic peaked, roughly 1.3% of requests failed, and recovery took a further 33 minutes after the migration was paused.

GitHub employs some of the most experienced database engineers in the industry. They wrote gh-ost, the online schema change tool that half the MySQL world now depends on. If a routine migration can bite them, it will certainly bite a twelve-person product team whose migrations are generated by an ORM and, increasingly, written by a coding agent that has never seen the row count of the table it is altering.

TL;DR

• Schema migrations take sites down through lock queues, not slow SQL. An ACCESS EXCLUSIVE lock waiting behind one long query blocks every subsequent read of that table.
• GitHub’s May 2026 availability report documents two schema incidents in three days: a migration on a hot table (55 minutes) and a 32-bit integer key hitting its ceiling in a Vitess lookup table (3 hours 39 minutes, near-total failure of pull request thread creation).
• AI coding agents generate syntactically correct migrations with catastrophic lock behaviour, because nothing in the prompt tells them the table has 200 million rows.
• Expand and contract is the only pattern that survives contact with production. Every schema change becomes three deploys, and no deploy is ever backward-incompatible.
• Set lock_timeout on your migration role, lint migrations in CI with Squawk, and never give an autonomous agent production database credentials.

The lock queue is the thing nobody understands

Ask a developer why a migration is risky and they will usually say it locks the table for a long time. That is only half right, and the wrong half.

Most dangerous DDL in PostgreSQL takes an ACCESS EXCLUSIVE lock. That lock conflicts with everything, including plain SELECT statements. Here is the failure mode that actually causes outages:

1. An analytics query, a long transaction from a background worker or a forgotten psql session holds a weak lock on orders.
2. Your migration requests ACCESS EXCLUSIVE and joins the queue behind it.
3. Every query that arrives afterwards, including one-millisecond primary key lookups, queues behind your migration.

The migration itself might take 30 milliseconds. It does not matter. Your checkout page is down until that stale transaction finishes, and Postgres will politely let the pile grow until connections are exhausted. This is why the incident postmortem never says “the ALTER TABLE was slow”. It says “connection capacity saturated”, which is exactly what GitHub reported.

What is actually safe

On modern PostgreSQL, the following are cheap metadata operations:

• ADD COLUMN that is nullable, or has a non-volatile default (safe since PostgreSQL 11, which stopped rewriting the table)
• DROP COLUMN (fast in the database, though it will break any application code still selecting it)
• Widening varchar(50) to varchar(100) or to text
• RENAME at the catalogue level, which is instant and also the single most reliable way to break a running application

The following look equally innocent in a diff and are not:

• SET NOT NULL requires a full table scan while holding ACCESS EXCLUSIVE. The safe route is to add a CHECK (col IS NOT NULL) NOT VALID constraint, run VALIDATE CONSTRAINT under a weak lock, then SET NOT NULL, which PostgreSQL 12 and later will skip the scan for.
• CREATE INDEX without CONCURRENTLY blocks all writes for the duration. With CONCURRENTLY it cannot run inside a transaction, which is precisely why most migration frameworks (Rails, Django, Prisma, Alembic) need explicit configuration to allow it, and why so many teams silently do not.
• Adding a foreign key locks both tables while validating every row. Add it NOT VALID, then validate separately.
• Changing a column type generally rewrites the entire table and every index on it.

None of this is new. All of it is well documented. And almost none of it is encoded anywhere your team will actually encounter it at the moment the migration is written.

Why AI agents made this materially worse

A coding agent asked to “add a tenant_id to invoices and make it required” will produce correct, idiomatic, reviewable SQL. It will also produce ALTER TABLE invoices ADD COLUMN tenant_id uuid NOT NULL, because that is what the instruction said and that is what every tutorial in its training data looks like.

The agent has no idea that invoices has 180 million rows, that it is the hottest table in the system, or that the change will hold an exclusive lock for several minutes. Nothing in the repository tells it. Table statistics are not in the context window. This is the same class of problem as constraint decay: agents degrade sharply on backend work where correctness depends on state they cannot see.

The review process does not catch it either. The pull request contains three lines of SQL. It looks fine. It is fine, in the sense that it does what it says. The reviewer, who is already dealing with an AI-inflated review queue, approves it in forty seconds.

The agentic incidents that make headlines are the dramatic ones. One vendor roundup counts nine publicly documented cases between June 2025 and July 2026 in which a coding agent destroyed data outright, including a developer who connected an agent to a live Supabase instance and watched it run a Prisma command with --shadow-database-url pointed at production. Those stories are useful for arguing about credentials. But the quieter failure, the migration that is technically correct and operationally lethal, will happen to far more teams and will never be written up.

Expand and contract, properly

The pattern that works has been known for a decade and is still under-practised: never let the application and the schema be incompatible, in either direction, at any point.

Renaming user_name to username is not one migration. It is five steps across three deploys:

1. Expand. Add the new column. Deploy.
2. Dual write. Application writes both columns, reads the old one. Deploy.
3. Backfill. Copy historical data in batches, with sleeps, outside the migration transaction.
4. Switch reads. Application reads the new column, still writes both. Deploy.
5. Contract. Stop writing the old column, drop it. Deploy.

Yes, that is slower. It is also the difference between a rollback being “revert the deploy” and “restore from backup”. The critical property is that at every point, both the old and new application versions work against the current schema, which is what makes rolling deploys and instant rollback possible at all.

Tooling can automate a good deal of this. pgroll implements expand and contract natively for PostgreSQL by exposing multiple schema versions through views, so old and new application code each see the shape they expect. On MySQL, gh-ost and pt-online-schema-change perform the table rebuild out of band, gh-ost by reading the binary log rather than installing triggers, which makes it throttleable and pausable mid-flight.

Five things to put in place this week

1. Set a lock timeout on the migration role. SET lock_timeout = '3s' before DDL means a blocked migration fails instead of forming a queue. Wrap it in a retry loop with backoff. This one setting prevents the majority of migration-induced outages, and it is a one-line change.

2. Lint migrations in CI. Squawk is a free, Rust-based PostgreSQL migration linter with a GitHub Action that comments violations directly on the pull request. It catches missing CONCURRENTLY, volatile defaults and unsafe type changes before a human ever looks. Worth noting for anyone budgeting: Atlas moved its migrate lint command out of the free tier in October 2025, so the paid alternative now starts at $9 per developer per month plus $59 per CI project.

3. Separate schema deploys from code deploys. If migrations run automatically as part of application release, you have coupled two things with different risk profiles and different rollback semantics. Run them as a deliberate, observable step.

4. Test against production-scale row counts. A migration that takes 8 milliseconds against 400 seeded rows tells you nothing. Either use a branched copy of production data (Neon, PlanetScale and Turso all make this cheap now) or seed a staging database to realistic volume.

5. Keep autonomous agents away from production credentials. Agents can draft migrations. Migrations reach production through CI, with a human approving the plan and the linter having its say. The boundary is not “can the agent write SQL”, it is “can the agent execute SQL against a database that matters”.

The schema debt you have not noticed yet

Two days after the migration incident, GitHub had a second one: a 32-bit integer key reached its maximum value in a Vitess lookup table, and creation of new pull request review threads failed at close to 100% for three hours and 39 minutes. The fix was to move the lookup table definitions to 64-bit columns across every shard.

Every product built on a default integer primary key has this clock ticking. Two billion sounds enormous until you have an events table, a webhook log or an audit trail. And an int4 to int8 conversion on a large hot table is one of the hardest migrations there is: a full rewrite, every index rebuilt, every foreign key revalidated. It is a three-week project when planned and a four-hour outage when discovered.

Audit your integer primary keys. Check the current value against the ceiling. Do it before it is urgent, because the version of this problem that arrives on its own schedule is enormously more expensive than the version you scheduled.

Who owns the schema?

The uncomfortable answer at most growing companies is nobody. The ORM owns it, generation is automatic, review is cursory, and the first real conversation about a table’s shape happens during an incident. Adding AI agents to that process does not create the problem, but it does remove the last piece of friction that used to slow it down: a developer writing SQL by hand and pausing, briefly, to wonder how big the table was.

The teams that handle this well are not the ones with the best tooling. They are the ones who decided that schema changes are a distinct category of change with their own review path, their own guardrails and a named owner.

At REPTILEHAUS we do a lot of this work: database architecture reviews, migration strategy for teams moving fast with AI-assisted development, and the unglamorous DevOps plumbing that turns a risky release into a boring one. If your migrations currently go out with a held breath, get in touch and we will talk through what a safer pipeline looks like for your stack.

📷 Photo by Albert Stoynov on Unsplash