The Same Weights Are Not the Same Model: Why Self-Hosted LLMs Fail Tool Calls

There is a comfortable assumption sitting underneath most self-hosted AI plans: that a model is a file. You download the weights, you point an inference server at them, and you get the thing you read about in the benchmark table. The weights are the model, the model is deterministic, and the rest is plumbing.

That assumption is wrong, and the way it is wrong matters most precisely where teams are now betting the hardest: agents that call tools.

TL;DR

• Identical model weights produce measurably different output depending on quantisation, KV cache precision, attention backend and tensor parallelism settings. The file is not the model. The stack is the model.
• Independent testing on Qwen 3.6-27B across a real 100k-token workload found top-1 token disagreement rates ranging from roughly 15% (INT8 W8A16) to roughly 50% (NVIDIA NVFP4) against a BF16 reference.
• Prose degrades gracefully under this divergence. Tool calls do not. A single flipped token turns show arp into show run, or port 5432 into 543ql, and the agent executes it confidently.
• INT4 KV cache quantisation failed to complete tool calls outright. Some popular community fine-tunes showed a nearly 6% token flip rate and 36 invalid tool branches against the official baseline.
• The practical fix is not “use higher precision”. It is treating your inference configuration as a versioned production dependency, with an evaluation harness that exercises tool calling specifically.

What the testing actually found

In mid-August 2026, a Level1Techs forum contributor writing as thr3e published a multi-part investigation that deserves more attention than it has received. Rather than running synthetic benchmarks, they took a real 100k-token network automation workstream and measured how the same model behaved across different serving configurations, capturing full-vocabulary logits and measuring token-flip rates through teacher-forced decoding.

The hardware spanned RTX PRO 6000 Blackwell, RTX 5090, H200 and B200. The model was Qwen 3.6-27B, served five ways. Measured against a BF16 reference, top-1 token disagreement at 88k of context came out roughly as follows: official FP8 (W8A8) around 30%, INT8 W8A16 around 15%, AWQ W4A16 around 35%, and NVIDIA NVFP4 around 50%.

Those are large numbers, and the instinct is to discount them. A token flip is not necessarily an error. Two configurations can disagree on the next token and both produce a perfectly good paragraph, because natural language has enormous redundancy. If the model says “the next step” instead of “the following step”, nobody files a ticket.

The tool-calling results are where the argument stops being academic. Both NVFP4 and AWQ produced incorrect Cisco commands, substituting show run for show arp. The reference configuration did not. Same weights, same prompt, different arithmetic, wrong command sent to a production router.

Why identical weights give different answers

The underlying cause is boring and unavoidable: floating point arithmetic is not associative, and no two inference stacks execute it in exactly the same order.

Different attention backends (FlashAttention 2, Flash Inference, Triton Attention) use different CUDA kernels with different reduction orders, and produce divergent logits. KV cache quantisation compounds it: BF16 was bit-identical across runs, INT8 diverged then recovered, and INT4 broke tool calling entirely. Tensor parallelism adds another layer, and produced genuinely strange results in testing, with TP1 succeeding, TP2 failing, and TP4 succeeding again, which points at NCCL communication ordering rather than the weights.

Then there is the dependency surface. A single vLLM nightly build in the investigation pulled in 734 packages. Every one of those is a version that can change how a number gets rounded.

None of this is a bug. It is what running numerical software on heterogeneous hardware has always looked like. What changed is the consequence. Small logit differences compound across a generation, and the author’s framing is the clearest summary of the mechanism: when next-token probability shifts far enough, THE then NE then XT becomes THE then NE then W then DAY. One divergence early, and the two outputs never reconverge.

Tool calling is the failure surface

This is the part we would ask any client to internalise before signing off a self-hosted architecture.

For eighteen months the dominant self-hosted use case was retrieval and summarisation, and text output is forgiving. Slightly different phrasing is invisible. Slightly different emphasis is invisible. The tolerance is enormous, which is exactly why teams ran a few sample prompts, judged the output “fine”, and shipped.

Agentic workloads removed that tolerance. A tool call is a structured payload with no redundancy at all. Every character is load-bearing. The argument is either the right hostname or a broken one. The port is either 5432 or it is 543ql, which is a real corruption observed in testing, alongside hostnames collapsing to a single hyphen. There is no graceful degradation path for a malformed JSON argument being passed to a function that talks to your database.

So the quality signal your team has been using, “the output looks good”, is measuring the wrong thing. It measures the forgiving surface and tells you nothing about the unforgiving one.

The community fine-tune problem

The same investigation tested four popular “uncensored” or abliterated Qwen 3.8 derivatives against the official baseline. Two held up well, with token flip rates around 1.3% to 1.4% and zero invalid tool branches. The other two did not: one at roughly 5% flips with a single invalid branch, and one at 5.8% flips with 36 invalid tool branches, producing literal corruption in operational commands.

Those models are downloaded constantly. They are chosen for behavioural reasons, usually to remove refusals, and they are evaluated on whether the refusals are gone. Almost nobody re-runs a tool-calling evaluation afterwards. A modification made for one property silently damaged an unrelated one, which is the same class of problem as an untested dependency bump, and deserves the same governance.

What this means for your architecture

The correct conclusion is not “self-hosting is too risky” or “always run BF16”. Both are overcorrections. Higher precision costs memory and throughput, and in a market where memory pricing has moved sharply against buyers, that is a real budget line rather than a rounding error.

The correct conclusion is that your inference configuration is a production dependency with the same status as your database version, and it currently is not being managed like one. Most teams can tell you their model name. Far fewer can tell you their KV cache precision, their attention backend, or their tensor parallel degree, and fewer still have those values in version control.

Six things worth doing this quarter:

1. Pin and version the whole serving configuration. Quantisation method, KV cache precision, attention backend, TP degree, sampler settings and inference server version belong in the repository, not in a shell script on the GPU host.
2. Build an evaluation harness that exercises tool calls, not prose. Assert on the actual arguments produced against a known-good set. This is the single highest-value item on the list, because it is the only thing that catches the failure mode described here.
3. Re-run that harness on every stack change. A driver update, an inference server bump or a hardware migration is a model change. Treat it as one.
4. Read the model card and honour it. Published sampler settings exist for a reason. Setting temperature too low is a well-known cause of reasoning models looping in their own think output, and it is not a model quality problem.
5. Treat fine-tunes as forks requiring qualification. A derivative model needs the same evaluation gate as the base model, not a vibe check on the behaviour that motivated the swap.
6. Reserve low-precision configurations for low-consequence work. Aggressive quantisation for summarisation and classification is a reasonable trade. The same configuration behind an agent with database or infrastructure credentials is not.

The wider pattern

This is the same lesson the industry keeps relearning at every new layer of abstraction. Serverless made concurrency limits your problem. Managed CI made workflow permissions your problem. Self-hosted inference makes numerical reproducibility your problem, and hands it to teams whose testing culture was built for deterministic software.

The honest position is the one the author landed on: every local implementation diverges, including the good ones, and including the hosted APIs you are comparing against. Divergence is not the failure. Not measuring it is.

At REPTILEHAUS we build and operate AI agent systems, and this is exactly the kind of problem that surfaces after deployment rather than during it, usually as an intermittent production incident nobody can reproduce. If you are moving agents onto self-hosted models, or you have a tool-calling pipeline that fails in ways your test suite never catches, we can help you build the evaluation and DevOps discipline around it. Get in touch.

---

📷 Photo by Taylor Vick on Unsplash