# Label corrections

_Published with the study "How often is a code check right?" (vallit.net/research/how-often-is-a-code-check-right). Every label of a test set that changed after the set was first scored, with the rule sentence that decides it. Copied from `bench/source/ADJUDICATION.md` in the Vallit app repository. The first scores in lab-cases.csv are the recorded ones, before any of these corrections._

A corpus label changes only when the case, read against `ORACLE.md`, plainly
contradicts it, and only after that corpus has been scored once as a holdout.
Its first score stays as recorded in `results/`; a correction affects only its
later use as a regression set. Every change is listed here with the oracle
sentence that decides it.

| Case                                           | Before                                                     | After                                                                                   | Why (ORACLE.md)                                                                                                                                                                                                                     |
| ---------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| holdout-a / a1-021                             | `unverified-webhook` only                                  | adds `access-control:unauthenticated-write` (same route file)                           | The unverified Stripe webhook updates stored records with no signed-in user. That is "a request with no signed-in user can update … stored records", and the exception only covers a _verified_ webhook.                            |
| holdout-a / a1-022                             | `unverified-webhook` only                                  | adds `access-control:unauthenticated-write` (same route file)                           | Same shape as a1-021: a Lemon Squeezy webhook that updates subscriptions without checking the X-Signature HMAC.                                                                                                                     |
| holdout-b / b2-027                             | `unverified-webhook`                                       | adds `access-control:unauthenticated-write` and `payment-integrity:cancel-keeps-access` | The Paddle webhook only checks that a signature header is present, then updates rows. It grants access on `subscription.activated` and handles no cancellation, update or expiry event.                                             |
| holdout-b / b3-022                             | `unauthenticated-write` on `src/routes/webhooks/stripe.ts` | same rule on `src/db/orders.ts`                                                         | The case format scores the file holding the write. The `update orders` statement lives in `markOrderPaid` in `src/db/orders.ts`, not in the route.                                                                                  |
| holdout-c / c2-029                             | `cancel-keeps-access`                                      | adds `payment-integrity:client-set-price` on `server/routers/billing.ts`                | The tRPC `createCheckoutLink` sends `price_id: input.planId` to Paddle with no allow-list. That is "a price id from the request that is not checked against an allow-list". The webhook grants pro for any paid price.              |
| holdout-a / a2-021, a2-022; holdout-b / b2-021 | `unverified-webhook` only                                  | adds `access-control:unauthenticated-write` (same webhook file)                         | These have the same shape as a1-021: an unverified payment webhook that increments a stored balance. With the a1-021 correction, every unverified webhook that updates rows now carries the rule, as holdout D labels it (dd2-030). |

Holdout corpus D has no corrections.

## Real-world adjudications

`results/fresh-repos-v3.adjudication-*.json` hold independent reviews of every
finding detector v3 reported on ten open-source repositories it had never seen.
Each reviewer read the repository code for its finding and gave `TP`, `FP` or
`UNSURE` with a reason. They did not see the detector's code. `scripts/verify-eval.ts`
uses these labels to measure the model verifier.