Right the first time?

What we measured

A website check sees what anyone on the internet sees: the page, its headers, the scripts it loads, and the answers to a few plain requests. It only uses requests meant for reading, and it does not sign in or change anything. Some of these checks only run once the person who starts the check confirms they may check the app, and every case was written as if they had. The repository checks read the files of a connected repository as they are committed: migrations, lockfiles, configuration and .env files.

A case is one such app, written down. A website case holds the page, its headers, the scripts and the answer to every address a check might ask for, plus what public registries would say about the domain. A repository case holds the files. The checks read each case as they would read a real app, without the network, so a run can be repeated byte for byte.

About half the cases are broken. The other half are near misses: the same app, built safely. A Supabase table that answers the public key with an empty list instead of rows. A cookie that scripts may read because it holds a theme, not a session. A file called .env.example full of placeholders. A check that cannot tell those apart from the real thing raises the alarm on both.

How we measured

Each set was written from a short brief that says, for each check, what it should report and what it should deliberately stay quiet on. The brief describes behaviour, not code. It was written without access to the checks’ code, their tests or any score. Before a set was run, the checks were frozen, and all nine reports name the SHA-256 of the files they ran. The set was scored once, and that score is the one we report. After that, the set became a test that every later version must pass.

A broken case counts as found when the check reports it. A safe case counts as a false alarm when the check reports anything at all. We give two shares: the broken setups found, and the safe setups called broken. Every interval is a 95% Wilson score interval. With 8 to 20 cases per check, the intervals for a single check are wide; the totals are the numbers to read.

We also rewrote the 144 cases of set 8 into 311 variants that must not change an answer. They put headers in another case, other quotes around attributes, comments in the SQL and extra markup. No answer changed.

The first time

Two of the nine sets came out with nothing missed and no false alarm. The weakest was Set 8, the first blind test of 18 checks we had just written: it found 53 of 72 (73.6%). The noisiest was Set 3, the first blind test of the eleven website checks we had shipped without one: 6 of its 44 safe setups were called broken.

The checks that look at a live app found 93.1% of what was broken there, and called 3.2% of the safe setups there broken. The ones that read a repository found 84.3% and called 2.9% broken. 21 of the 51 checks were right on every case they were given. Each was given 8 to 20 cases, so that says more about those cases than about the check.

Supabase and Firebase

Many apps built with AI tools keep their data in Supabase, and the protection of each table is a row-level security rule the app has to write. In 2025, an insufficient row-level security policy in apps generated by one AI builder was recorded as a vulnerability of its own. The builder disputes it, because each of its customers is responsible for their app’s data.

We have 9 Supabase checks and 2 for Firebase in this benchmark. On the live app, one asks each table the app reads for one row with the public key and records whether a row came back, never the row. In the repository, others read the migrations for tables without row-level security and for rules that let anyone change every row. Together they found 69 of 74 broken setups (93.2%) and called none of the 74 safe ones broken. The table check on the live app found 8 of 8; the migration check found 9 of 10.

That is the part of the benchmark we trust most. It is also where a founder can act at once, because a table that answers strangers is closed by a policy on that table. Of the 9 Supabase checks, two missed a case the first time, and one of those misses is kept on purpose, as the next section explains.

After the fixes

Every miss and every false alarm was studied. 16 of the 33 misses and 9 of the 11 false alarms led to a fix. Scored again, the same 702 cases give 333 of 350 found and 2 false alarms. That is what checks that have learned from their test look like, and it says nothing about apps they have not seen. It is why each round got a new set instead of another pass over an old one.

The release that shipped the first two sets’ checks said they found 54 of 54 with no false alarm. That was true of the version that shipped, after the fixes. The first time, those sets found 51 of 54 and raised 2 false alarms. We should have given both numbers then, and we give them now.

17 cases are still missed, and two safe setups still get a note. We kept them on purpose, each with the source that decides it. The checks stay quiet on:

• a wildcard in the origins Next.js allows for Server Actions, which Next.js itself never matches (4 cases)
• GitHub Actions whose tags were restored after an incident, which the advisories call safe again (3 cases)
• SVG images through the Next.js image optimizer, which has sent them with a policy that stops scripts since version 12.3 (3 cases)
• subdomains pointing at services where taking over a name is not known to work, such as GitHub Pages (2 cases)
• a GraphQL server that only answers its schema to a POST, which our checks never send (1 case)
• a build that copies every variable into the browser code, which Next.js refuses to build (1 case)
• a database function that looks up one user by email, the usual way to send an invite (1 case)
• a version lookup that needs the owner’s OK, in a case written without it (1 case)
• a stand-in for a key that does not have the shape of a real one (1 case)

The two safe setups that still get a note sit behind a bot challenge page. Our check cannot see such an app, and it says so in the report instead of calling the app fine. The brief had asked for silence, and we think the note is right.

What we don’t know yet

Every case is an app we made up. The brief asked for the migrations, lockfiles and pages that apps built with Lovable, Bolt, v0, Cursor or Replit have. No real app’s data is in them, and real apps are messier. Our study of code checks showed how far a score on written cases can fall on real code. We expect less of a fall here, because a live page has far fewer ways to say the same thing than code does, but we have not measured it.

The cases and their labels come from inside Vallit, from writers who could not see the checks. No outside security expert has reviewed them. A check that scored every case right on eight cases has a 95% interval that reaches down to 51% for what it finds. Five of the nine reports were run on a working copy with changes not yet committed. For those, the SHA-256 of the checks’ files says what ran, and the commit only says where it started. Two cases were written without the owner’s OK that their checks need, so the check did not run on them; we count them as the run scored them.

Apps built with AI tools can only be measured when their owners ask, and that is the study we want to run next.

We publish the data behind this study. Every measured result on this page is computed from the benchmark’s raw reports and cases, whose SHA-256 it lists. Beside it are all 702 cases with their first and later answers and a readme that says what each file holds. The cases themselves and the checks are not public. If you research this problem and want to check our work more closely, write to info@vallit.net.

Citation

@techreport{vallit2026blindchecks,
  title       = {Right the first time? 51 checks on 702 cases they had never seen},
  author      = {{Vallit}},
  institution = {Vallit},
  year        = {2026},
  month       = oct,
  url         = {https://www.vallit.net/research/right-the-first-time}
}