Two things came off the Hacker News front page on Monday night. The loud one was OpenAI’s Navier-Stokes announcement at 1,239 points, just behind Tristan Buckmaster’s own statement PDF at 1,571. The quiet one was a four-post thread from Terence Tao, published at 20:32 UTC on 8 September, which settled at 327 points. We are not going to adjudicate the mathematics. Tao’s thread is the more useful document anyway, because his argument is not really about mathematics: it describes something that has already happened to your backlog, and explains why your estimates stopped landing without anyone being able to say quite when.
TL;DR
- Terence Tao argues that AI has flattened the “difficulty landscape” that told researchers which open problems deserved attention at all.
- Software delivery has the same problem. Agentic tooling made one subset of tickets far faster and left another untouched, and nothing in the ticket tells you which you are holding.
- Tao names the mechanism that keeps the boundary invisible: AI companies publish wins and withhold negative results. Your suppliers do it to you, and your delivery process does it to itself.
- Most engineering organisations destroy the only data that would map that boundary: deleted spike branches, discarded agent transcripts, tickets recording the eventual fix and never the failed attempt.
- The response is measurement, not opinion: log agent attempt outcomes, estimate in review capacity rather than developer days, keep an unassisted control sample, and demand negative results at procurement.
What Tao actually said
The counter-intuitive part comes first. The set of open problems one could pose is infinite, so how can good problems become scarce? Tao’s analogy: “a country or region can suffer a critical shortage of drinking water while simultaneously being surrounded by a massive ocean.” You can generate questions at will, such as the ten-to-the-ten-to-the-tenth digit of pi. Almost none are worth anyone’s attention, because they reveal no insight and connect to nothing.
Deciding which questions are worth attention is, in his words, “a lengthy, deliberate, and subjective process”, and it depends on knowing the difficulty landscape of a field: which questions are easy with known methods, which are solvable with effort, and which are out of reach. Every genuine advance flattens that landscape somewhat, which is normally fine, because the same advance extends how far you can reach and opens new frontiers to explore.
What is different now, Tao argues, is the absence of any such frontier. AI tools have flattened large areas of the terrain without producing a legible boundary between the AI-feasible and the AI-hard, partly because the technology keeps moving, and partly because of what he calls “the refusal of AI companies to disclose their negative results, or reveal the process towards obtaining their solutions”. His conclusion is the line worth carrying into any engineering organisation: “it is now the identification of a promising problem which is the scarce and precious resource.”
Your backlog is a difficulty landscape
Every planning ritual your team runs is a difficulty landscape wearing a costume. Story points, t-shirt sizes, “that’s a Tuesday”, “that’s most of a quarter”. None of these are measurements. They are acts of recognition by someone who has seen forty tickets shaped like this one.
Agentic tooling flattened part of that terrain, and only part. Work that used to take a competent developer three days now lands in forty minutes. Other work, described in an issue that reads identically, still takes three days, or takes longer than it used to because the agent produced something plausible that passed review and failed six weeks later in a way nobody traced back.
The terrain moving is not the problem. Terrain moves whenever tooling improves, and teams recalibrate within a quarter or two. The problem is that the boundary is no longer visible from the ticket. Nothing in the issue text tells you which side of the line the work sits on, no supplier will tell you where the line runs, and the line moves with every model release you did not schedule.
We have made a related argument about mean time to restore as an average over two populations that have separated. This is the same shape with a nastier property. There, the average concealed something you could still recover from the data afterwards. Here the loss is prospective: you need the classification before you commit to a sprint or quote a fixed price, and the classification is precisely what has become unreadable.
Nobody publishes the failures, including you
Tao’s point about negative results applies at two levels. Read it first as a procurement observation. Every model release arrives with a card of benchmark wins. None arrives with the complementary list: the task classes where the tool burned an afternoon and produced nothing usable, and how the vendor established that. The missing half is the half you need in order to plan. A capability claim with no matching incapability claim is not an evaluation, it is a brochure, and it belongs on the shelf with the rest of your supplier counterparty risk.
Then turn it on your own organisation, where it stings, because you have the data and you throw it away. When an agent-assisted attempt fails, what survives it? The branch is deleted. The session transcript rolls out of history. The ticket is picked up by a human, fixed, and closed, and the permanent record says “two days” rather than “attempted with an agent, abandoned after ninety minutes, then two days by hand”. Multiply that across a year and you have deleted the map of your own difficulty landscape while believing you kept tidy tickets. It is the same artefact teams destroy when they lose an R&D tax credit claim, kept for a different reason. Nobody else will produce this map: it is specific to your codebase, your conventions and your review culture.
The cheapest fix is not always the most valuable one
The clearest evidence sits in the Equational Theories Project report (arXiv:2512.07087), a collaborative effort that resolved more than 22 million implications in universal algebra and formalised them in Lean. Section 14.1 records the opportunity cost of moving quickly. A few implications stumped the team and forced genuinely new techniques for constructing infinite counterexamples; an automated prover later found the same implications had relatively small finite counterexamples. Had the prover run first, the technique would never have been invented, and the authors suspect their early automated sweeps quietly obscured other fruitful problems.
Translated into a delivery organisation: the cheapest resolution to a ticket is not always the most valuable one, and you will not notice the difference, because the cheap resolution closes the ticket and the ticket is what you measure.
Five things to do about it
- Log the attempt, not just the diff. Three fields on every ticket: was an agent attempted, what was the outcome (merged as written, merged after rework, or abandoned), and how long until the change was accepted. One quarter of that beats any public benchmark, because it maps your system rather than somebody’s evaluation harness.
- Estimate agent-touched work in review capacity, not developer days. Production is the half that collapsed; verification is the half that did not, and it absorbs the difference. Convert effort into developer days and you are budgeting the cheap half while assuming the expensive half is free. Our timeline bands for 2026 hold for the same reason: the critical path was never the typing.
- Keep a deliberate sample of unassisted work. Not sentiment, calibration. If every ticket is attempted with the tool, you have no control group and no way to distinguish the landscape moving from your team’s judgement decaying.
- Ask suppliers for negative results, in writing, before you buy. “Which classes of task does this tool fail at, and how did you establish that?” A vendor who can only produce wins has told you something about their evaluation. Put it in the procurement pack beside data residency and exit terms.
- Separate “solved” from “understood” in your definition of done. This is not the failure mode where the agent games the measurement. Here the change is genuinely correct and nobody can explain why it works. Nominate the work where a raw solution carries negative value: the payments path, anything touching authentication, anything that will be modified again inside a year. An unexplained merge there is a liability carried at book value.
Tao closes with a comparison worth stealing. A modern food donation drive does not accept arbitrary contributions merely because they are technically edible; it publishes standards for what is actually wanted. Engineering organisations never wrote that standard down, because for thirty years a working solution and an understood solution arrived in the same commit. They no longer do.
REPTILEHAUS builds and maintains software for teams facing exactly this: agent-assisted delivery that stays reviewable, evaluation harnesses that measure your codebase rather than a leaderboard, and DevOps practice that treats model suppliers as the dependencies they are. If your estimates have stopped landing and nobody can say why, talk to us.
📷 Photo by Fabien Maurin on Unsplash
