What Vallit checks, and what not

In short

• This page shows how often the checks found a problem we planted, and how often an alert was wrong. Each row is one kind of problem the site and the app name.
• The numbers come from test apps and from one real repository, measured on 7 October 2026. They describe the checks, not your app.
• A finding confirmed in code your app runs is the kind to act on. Check a finding marked Possible, or one in an old file, before you act on it.
• The second half lists what Vallit does not check at all. A report without findings says nothing about those parts.

How well the checks find problems

We measure every check against apps with problems we know about, and against apps where the same code is safe. Three kinds of test go into the table.

• Test apps and fixtures: more than 5,000 recorded cases in small apps of two to eight files, each written to test one rule. Checks score higher on them than on real code.
• Whole apps: 36 complete apps with several files each. 16 of them were written for this round by an author who never saw how the checks work. On their first run the code checks found 5 of their 13 problems; after the misses were fixed, 12.
• The whole-app column counts after those fixes. These apps are no longer unseen, so read it as what the checks do now, not as a rate on a new app. The older 20 apps started at 6 of 16.
• A real repository: one large codebase of ours with 1,497 files. Each alert was read by hand. Of 56 alerts from the code checks, 6 were real.

On real code, most raw alerts from a first look are wrong. Two language models read each finding from the checks that follow a request or a rule through your code, and what they reject is left out (What we check). A finding they cannot settle stays in the report with Possible before its severity.

The review helps, and it does not catch everything. On the real repository it left out about half of the wrong alerts, and it dropped one real problem. Most of the alerts still in the report were wrong too.

Where the wrong ones were tells you what to trust:

• Every finding the review confirmed in code the app runs was real.
• Every wrong finding left in the report sat in old files that were no longer deployed, or carried Possible. The review cannot tell an old file from a live one either, so it confirmed some of those.
• Check a Possible finding, and any finding in a file you no longer deploy, before you act on it. If it does not apply, click Tell us it is not an issue in the finding. When we agree, we mark it Not an issue, and it keeps that mark on later checks.

• Not tested means that kind of test has no case for that row yet. The checks on your website were not run against a real site for this table.
• Some recorded cases could not be decided, such as a replayed answer that did not load. They count neither for nor against a row. They are most common in the build and cloud settings row.
• The 22 wrong access alerts in the real repository came from checks the analysis did not recognise, such as a membership lookup. 22 of the 26 wrong HTML alerts were in old code still in the repository but no longer deployed.

What Vallit does not check

A report covers only what its checks can read. These parts are outside them.

• Servers in other languages. The code checks read JavaScript and TypeScript. For a server in Python, Ruby, Go, PHP or Java, they show Not run and the report reads Partial.
• Code in packages your repository does not contain. A signature check or a merge function from a private package cannot be read. The checks then leave the route alone instead of guessing.
• The code inside route middleware in Express and Hono. Its name counts as a sign that a route is protected. What it does with a token or a key is not followed.
• Values a visitor's browser makes for itself, such as an id from Math.random() in browser code. The code checks follow what your server does with a request.
• Pages and APIs behind a sign-in. The checks on your website load what any visitor can load. They do not sign in.
• Your business rules. A wrong discount, a race between two requests or a permission your product should not grant are outside the rules the checks follow.
• Attacks. Vallit sends no attacks, runs no penetration test and does not test how much load your app takes.
• Your live cloud account. Only the configuration files in your repository are read, not the account behind them.
• Whether a vulnerable library is used. A library is reported by its version. Whether your code calls the vulnerable part is not checked.
• Deployed or left over. Code still in your repository counts as live, even when it is no longer deployed. That can cause wrong alerts, confirmed ones included. Deleting old files from the repository ends them.
• Mobile and desktop apps. Vallit reads websites and the repository you connect. The build of a mobile or desktop app is not read.

Related