Why an LLM writes plausible code, not guaranteed code
What the scientific literature actually measures when a language model writes code: a success probability over a sample, never a guarantee on the delivered output. Article 1 of the “Research on determinism” series.
A language model optimizes likelihood, not correctness
A language model produces code token by token, picking at each step the most probable continuation given its training. That mechanism explains its strength — the output looks like code humans have written — and its structural limit: nothing in the training objective requires the program to be correct. Correctness is a hoped-for property of the result, not a constraint of the process.
The consequence shows in the metric the field adopted: pass@k, the probability that at least ONE solution among k attempts passes the tests. It's a sample metric — it says how many draws you need to get a working program, not whether THE program you were handed works.
The founding numbers: statistical success, not a guarantee
The field's founding paper — the one that introduced Codex (the initial engine behind GitHub Copilot) and the HumanEval benchmark — gives the order of magnitude: the model solves 28.8% of problems on the first try, and 70.2% if allowed to produce 100 candidates per problem and keep the good one. The gap between those two numbers IS the point: quality comes from repeated sampling more than from the reliability of each output.
But a product that generates an application doesn't deliver 100 candidates: it delivers one. Everything separating 28.8% from 100% must then be covered by something other than the model — tests, validators, or an architecture that removes from the model what it cannot guarantee.
- Chen et al., Evaluating Large Language Models Trained on Code (2021): arxiv.org/abs/2107.03374
Hallucination isn't anecdotal: the measured case of dependencies
The most cleanly quantified example of plausible invention is software packages. A large-scale study had 16 models generate 576,000 Python and JavaScript code samples: on average, about one fifth of the recommended packages did not exist — over 200,000 invented package names, credible and well-formed, yet nonexistent.
The phenomenon even spawned a documented attack class, “slopsquatting”: registering a malicious package under a frequently hallucinated name, betting that an AI-assisted developer will install it unchecked. Code hallucination is thus not only a quality problem — it's a security vector.
- Spracklen et al., We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (Univ. of Texas at San Antonio, Virginia Tech, Univ. of Oklahoma) — 16 models, 576,000 samples, ≈ 20% nonexistent packages
And the same prompt doesn't yield the same code twice
On top of quality variability comes variability itself: the reference study on non-determinism (ACM TOSEM, 829 problems, five generations each) documents significant dispersion in the code produced for an identical prompt — and more recent work shows numerical sources of randomness persist even at zero temperature. Details and references are on the series' entry page.
What Blueprint Maker takes from this
The architectural answer is to never put the model in a position to invent what must be guaranteed. At Blueprint Maker, the LLM doesn't write the delivered code: it designs a business schema (the AppSpec), and a deterministic engine writes the code. Component properties are set by the engine under fixed rules — never guessed. The generated application's dependencies come from a frozen, tested template: a hallucinated package is impossible by construction, since the model doesn't choose packages.
Where statistical sampling remains — schema design — it is framed by validators and correction gates, then the complete output is proven in real conditions before delivery. Those two layers are the subject of the next two articles in the series.