Somewhere in your organisation there is an MCP server that took a quarter to build and gets called about twice a week. On this week’s “Ask HN: Who is using MCP in production?” thread, one engineer put it more bluntly: their company “spent six months creating an MCP server that no one will ever really use at scale”. The replies are full of the same shape. Teams built the integration, wired it up correctly, watched the agent quietly not use it, and then reached for the prompt file to try to talk it into compliance.
Prompting is not the fix, because this is not a prompting problem. Two pieces of research published in the last few days measure what agents actually do when you hand them a choice of tools, and the answer is uncomfortable for anyone who has been building tool surfaces on the assumption that the better tool wins.
TL;DR
- Across 16,893 measured agent sessions, Claude Code, Codex and Cursor picked the same tool in only 42% of cases, so “what will the agent choose” has no single answer you can design against.
- In a controlled retrieval experiment, LSP-backed semantic search hit 1.00 precision against grep’s 0.76, and it did not matter: recall stayed near 0.66 in both arms. Better precision bought no better outcome.
- Agents selected the semantic tool 0 to 6% of the time on simple lookups. Forcing them to use it raised token spend 12 to 19% for mixed results.
- Output shape beat capability: adding inline context to the same tool’s results lifted multi-file rename success from 0.67 to 0.83 pass@1 and cut follow-up reads from 15.2 to 3.2.
- Selection rate belongs on your dashboard next to latency and error rate. A tool that is never chosen is not neutral, it is a permanent tax on every request’s context window.
Three agents, 17,000 runs, 42% agreement
The team at Armature ran 16,893 sessions across 75 repositories in 10 languages, with 1,163 prompt variations and four user personas, from symptom-describing vibe coders to constraint-aware enterprise engineers. Crucially the agents implemented solutions rather than merely recommending them, so the measurement captures committed behaviour rather than opinion.
The headline is the disagreement. Given the same task in the same repository, the three leading coding agents converged on the same tool in only 42% of experimental cells. In voice work, Claude Code reached for Twilio, Codex for the OpenAI Realtime API, and Cursor for Vapi. The research method the agents used differed just as sharply: Codex ran a web search in 94% of sessions, often narrowing with site: operators to trusted domains, Cursor consulted the web in roughly two thirds, and Claude Code leaned on its priors and searched in about 30%, rising towards 80% for newer problem domains.
One finding deserves particular attention from anyone selling a developer tool. Claude Code chose to write an in-house implementation rather than adopt any third-party library about 19% of the time, roughly double the 10% rate for Codex and Cursor. Your competitor is not only the other vendor in the category. It is increasingly the agent deciding your problem is small enough to solve inline.
The number that should change how you build tools
The second study is the sharper one, because it isolates the variable. Researchers gave Claude models both grep and an LSP-backed semantic navigation tool, then measured code localisation, reference completeness and multi-file renames across Python and TypeScript repositories.
On reference completeness, the LSP path achieved perfect precision, 1.00 against grep’s 0.76, by eliminating textual false matches. Recall in both arms stayed around 0.66. Read that pairing again, because it is the whole argument: the semantic tool was strictly more correct about what it returned, and it did not find more of the true callers. The ceiling was not retrieval precision. It was how thoroughly the agent chose to work the problem. On clean repositories the LSP arm added nothing at all, since grep already resolved every reference; the gains, an F1 improvement of +0.246, appeared only on noisy codebases.
The agents seemed to know this. Offered both tools on simple localisation tasks, they picked semantic navigation between 0 and 6% of the time. On reference completeness, where it genuinely helps, selection rose to 45 to 57%. Forcing a semantic-first policy pushed token consumption up 12 to 19% without reliably better outcomes.
Then comes the result we keep returning to in our own agent work. Holding the tool’s capability constant and simply adding inline context to its output raised multi-file rename success from 0.67 to 0.83 pass@1, and dropped the number of follow-up file reads from 15.2 to 3.2. Nothing about the underlying analysis changed. The tool just stopped making the model go and fetch what it needed next.
What actually decides selection
Three properties predict whether an agent reaches for your tool, and capability is not among them.
Familiarity in the training corpus. Grep appears in decades of shell transcripts, tutorials, Stack Overflow answers and Makefiles. Language servers are consumed almost entirely through IDE interfaces, so the corpus contains very little of a human invoking one from a command line. The model is fluent in one and merely aware of the other. That asymmetry is not fixable through your tool description.
Output that carries the next decision. A precise answer that forces fifteen follow-up reads is worse, in a token-metered loop, than a slightly noisy answer that arrives with enough surrounding context to act on. Return the neighbourhood, not just the coordinate.
Zero setup and predictable failure. Grep runs on any tree in any language, honours .gitignore without being asked, and fails legibly. Several commenters in the HN thread described “LSP isn’t working” as a recurring hole they burn tokens climbing out of. An agent that has been burned by your tool’s setup path has learned something durable about it.
Designing a tool surface that gets used
Start by measuring selection rate per tool, per task type. Almost nobody does this, and it is the single most informative number about an agent deployment. Instrument which tools were available, which were chosen, and what the model did immediately afterwards. If a tool sits below a few per cent for its intended task category, you do not have an adoption problem to solve with better prose in the description, you have a tool that should be removed or merged.
Then treat the context window as a budget with a line item per tool. Every schema you expose is charged on every request, whether or not it is called. The HN thread is candid about this: teams are actively removing MCP servers in favour of CLI calls on token efficiency grounds, and several noted that agents skip tools rather than read a long list. A surface of eight sharp tools beats forty comprehensive ones.
Name and shape things the way the corpus already does. A tool called search_files that takes a regex and returns matching lines with surrounding context inherits every grep example the model has ever seen. A bespoke query language, however expressive, starts from zero. Fluency compounds; novelty does not.
Finally, put your semantic tooling where it wins rather than everywhere. Layered retrieval, broad cheap search first and precise symbol-level confirmation second, is the pattern the industry has converged on for good empirical reasons. Offering the expensive tool for the cheap task simply trains the agent to ignore it.
Tool choice is now a distribution channel
The Armature data has a commercial edge that is easy to miss. Stripe took roughly 90% of payment integrations, Neon 66% of databases, S3 45% of file storage, Resend 35.6% and Postmark 27.4% of email. Those shares were not won in a procurement meeting. They were won in the model’s priors, and they vary by repository language: Resend dominated TypeScript projects while SendGrid led in Python.
If you sell to developers, an increasing share of your funnel is an agent picking a default at three in the morning on behalf of someone who will never read your comparison page. That makes documentation quality, canonical naming, and the presence of clean idiomatic examples in public code into distribution strategy rather than developer relations hygiene.
Where this leaves your roadmap
We have argued before, in agent-native software design and the MCP roadmap, that the interface is the product. This week’s evidence makes it measurable. Capability is table stakes. Selection is the metric, and selection is governed by familiarity, output shape and reliability, in that order.
The practical move is unglamorous: audit the surface you already ship, delete what is never chosen, and rewrite the survivors to return context rather than answers. REPTILEHAUS builds and audits agent systems, MCP servers and internal tool surfaces for teams who need them to work in production rather than demo well. If you have an integration nobody’s agent calls, get in touch and we will tell you which half of it is worth building.
📷 Photo by Anton Savinov on Unsplash
