From pass@k to a binary gate: validating a generated application
Whatever the generator, a program's final proof is its execution. What research says about tests and self-repair — and why Blueprint Maker made a binary criterion its only automated judge. Article 3 of the “Research on determinism” series.
Benchmarks assume tests that real life doesn't provide
The field's whole metrology — pass@k first — rests on a quiet assumption: every problem comes with its correctness tests, written by the benchmark's authors. The model generates, the tests decide. But in real use, the user describing an application brings no test suite: if the system wants a verdict, it must manufacture its own proof.
That's the decisive shift: from correctness estimated over a sample (how many candidates pass provided tests) to verification performed on the delivered output (does THIS application, the one you receive, work).
Self-repair helps, but doesn't suffice
One closely studied avenue: asking the model to fix its own code from execution errors. The reference study (ICLR 2024) tempers the enthusiasm: gains exist but are often modest once weighed against the cost of extra calls, and depend heavily on the quality of the model's diagnosis of its own error — a model that got it wrong is not best placed to understand why.
The architectural lesson: the repair loop has its place, but as a mechanism framed by an EXTERNAL, deterministic judge — not as a guarantee in itself.
- Olausson et al., Is Self-Repair a Silver Bullet for Code Generation? (ICLR 2024): arxiv.org/abs/2306.09896
Blueprint Maker's choice: a binary criterion, actually executed
Every generated application passes, before delivery, a real-conditions validation: the code is compiled, the application is built then actually started in production mode, the database is seeded, and the navigation is walked screen by screen by a driven browser. The verdict is binary: the application runs, or it doesn't. No plausibility grade, no confidence score — a judge that cannot be charmed by code that “looks right”.
The gate is doubled over time: every night, a canary replays reference generations end to end, and the percentage of applications passing all checks over the last seven days is published, dated, on the Reliability page. Measurement precedes argument.
Beyond starting: values that cannot lie
Starting isn't enough: an application can run and display wrong numbers. The product's integrity layer extends the same doctrine to the data: derived values — totals, computed statuses, aggregates — are recomputed server-side from their inputs, so a displayed value cannot contradict what it stems from. And under ambiguity, abstention wins: coverage never comes before correctness.
The series in short
A language model produces the plausible (article 1); so we confine it to a verifiable specification (article 2); a deterministic engine builds, and real execution decides (this article). Three layers, one idea: never ask statistics for what architecture can guarantee.