Kernwerk's engineering blog ran the same SHA-256-identical Qwen2.5-1.5B-Instruct model file on a MacBook with Apple M1 and on an NVIDIA Jetson Orin Nano Super. Aggregate benchmark scores were almost identical, but some individual answers differed on these devices, and with int8 quantization, more than half of the math answers diverged. Conclusion: a benchmark score obtained on one device does not transfer to another, and the cause of discrepancies is the non-associativity of floating-point addition, which flips the choice of the next token when logits are nearly equal.
What happened
Kernwerk fixed the control conditions: one Qwen2.5-1.5B-Instruct file with an identical SHA-256 hash, identical input tokens, PyTorch in fp16 on a MacBook with Apple M1 versus NVIDIA Jetson Orin Nano Super — first under PyTorch CUDA in fp16, then under TensorRT-LLM in int8. Answers were compared line by line on three sets: MMLU with 1140 questions, HellaSwag with 1000, and GSM8K with 250 tasks. In fp16, the aggregate scores of the two devices diverged by no more than 0.5 percentage points, but 18 of 250 GSM8K math answers (7.2%) were different. When Jetson switched to int8, 53.2% of GSM8K answers were rewritten, and the score dropped by 11.2 points. The authors explain this by floating-point arithmetic itself: addition is not associative, different compute kernels sum products in different orders, and when two leading tokens are nearly equal in logit, a minimal numerical shift flips the answer.
Context
The basis of the effect is a fundamental property of floating-point arithmetic, not a failure of a specific build, and in the context of neural networks it has already been noted in research. Kernwerk ties the experiment to two works. ICML 2025 describes Chimera examples: the same input data is classified differently depending on the linear algebra library — MKL, Accelerate, or cuBLAS. NeurIPS 2026 — this is FloatDoor: a specially trained LoRA adapter turns a platform difference into a threat, and Qwen3-8B generates vulnerable code behind it in 49% of programs on A100 versus 16% on H200; such a gap between auditing on one platform and exploitation on another is described as time-of-check/time-of-use. Hence the methodological lesson: an aggregate score masks individual variance, and nearly matching aggregates coexist with dozens of differently answered positions, so checking only by the mean value does not show the model's target behavior.
Why this matters for the industry
For the industry, the conclusion is that a benchmark run on a laptop or cloud GPU guarantees almost nothing for a target board: even identical weights behave differently, and int8 shifts the result even more. For edge deployments of the Jetson class, this is a specific process: test the exact compiled engine on the device itself, compare answers element by element with a reference, hash and sign the engine, tokenizer, and runtime — TensorRT and CUDA — and verify the provenance of the weights. Since platform differences can become an attack, acceptance procedures will likely be reoriented from the model file to the entire actually executable stack: weights, adapter, engine, drivers, and runtime with verifiable provenance of artifacts. For edge business, this is a recalculation of testing deadlines and a new expense item; a direct continuation looks like demand for tools for element-by-element comparison of answers between devices and for pipelines where the entire set of inference artifacts is signed.
Why this matters for users
If you run an LLM on different hardware — Mac, Jetson, or cloud GPU — identical weights do not guarantee identical answers. The effect is amplified with int8 quantization and in long generation: in a math task, one digit shifted at the start leads the entire reasoning astray, and the result changes without any error in the user's code. Before deployment, a practical set of actions is this: run a benchmark on the target device and in the configuration where inference will take place, then compare answers element by element with a reference on a reference platform, rather than trusting an aggregate score obtained somewhere else. For int8 deployments, it is worth separately checking math and long generation tasks. When reading others' results, consider on what hardware and in what configuration they were obtained: without this data, a score fixes little.
What is still unknown / limitations
Direct attribution of the minus 11.2 points in int8 on Jetson to "platform nondeterminism" is incorrect: in the input data, it is not specified relative to which answers the 53.2% discrepancy is calculated — fp16 on the same Jetson or fp16 on MacBook. A significant part of the shift is probably explained by the TensorRT-LLM quantization error itself, not by the gap between platforms. The mechanism through non-associativity of summation and nearly equal logits is plausible and consistent with classical numerical analysis, but in the available material it is given in the authors' retelling, and independent replications — the experiment is cheap: an open model and two boards — are not described in the data. The discussion on Hacker News at the time of publication was limited to a link without comments, that is, there is no external confirmation from the discussion yet.
Sources
- Kernwerk — Same weights, different answers: why you test a model on its board and certify what runs (Maxime Carriere)
- Hacker News — Same LLM, same input tokens, different answers on a MacBook and a Jetson (1 point, no comments)
Author
Look at AI, editorial team