# Source-check benchmark — labelling rules (the oracle) _Published with the study "How often is a code check right?" (vallit.net/research/how-often-is-a-code-check-right). The labelling rules for the two checks it measures, as written before any test case existed. Copied from `bench/source/ORACLE.md` in the Vallit app repository; later sections for other checks are left out._ These rules define what counts as a real problem for the two source checks. They describe behaviour of the application, not how any detector works. Corpus authors label cases with them; the scorer compares a detector's output with the labels. Written before any corpus was authored and not changed to fit a result. ## Scope TypeScript/JavaScript web backends: Next.js App Router route handlers, Server Actions (file-level or inline `'use server'`), Pages Router API routes, Express, Hono, tRPC procedures, Supabase Edge Functions. Data layers: Prisma, Drizzle, supabase-js, Mongoose/MongoDB, raw SQL (pg, postgres.js, `sql` tags), Firestore. Payments: Stripe, Lemon Squeezy, Paddle, Polar. A case is a small repository (1–8 files) that must read like real application code. Server Actions count as public POST endpoints: a check on the page that renders a form does not protect the action (Next.js data-security guide). ## Check `access-control` | Rule | It is a problem when… | It is NOT a problem when… | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `unauthenticated-write` | A request with no signed-in user can update, upsert or delete stored records (not just insert a new one). | The route is protected by middleware/wrapper/procedure that demands a session; the write is a verified webhook or a cron job checked against a server secret; the filter is a secret capability token from the request (password reset token, invite code); only harmless counters (view counts) change; the Supabase client acts as the user so row-level security decides. | | `missing-ownership` | A signed-in user can read (and receive) or change/delete a record chosen by an id/slug in the request, and nothing ensures the record belongs to them or their team, and no admin/role check applies. | The query filter includes the session user/team; the record is loaded and the code stops when its owner is not the caller; a membership/permission lookup for the caller and the resource stops the request when missing; the caller passed a role/admin check; the data is public by design (published blog posts, product catalogue). | | `privilege-escalation` | Request data can set a privilege field (role, isAdmin, permissions, emailVerified, …) on a user/account/membership record, either by naming the field from the request or by writing the whole request body/object (spread or passed as-is) into a table that has such a field. | Fields are picked explicitly and exclude privileged ones; a validation schema lists only safe fields; the caller passed an admin/role check. | | `client-chosen-owner` | The user id that decides whose record is created, read (and returned), updated or deleted comes from the request (body, query, params, form) instead of the session. | The id comes from the session; the caller is an admin; it is a verified webhook using provider metadata. | ## Check `payment-integrity` | Rule | It is a problem when… | It is NOT a problem when… | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `unverified-webhook` | A payment webhook endpoint changes stored data (grants access, marks paid, adds credits) without verifying the provider signature (Stripe `constructEvent`, Svix/standard-webhooks verify, Lemon Squeezy/Paddle HMAC with a constant-time or equality comparison against the header). | Signature verification happens before the writes. | | `webhook-body-parsed` | The signature is verified against a body that was parsed and re-serialised (`JSON.stringify(await req.json())`, `req.body` in a Pages API route with body parsing left on). Real events then fail verification. | Verification receives `await req.text()`, a raw buffer, or a Pages API route with `bodyParser: false` reading the raw stream. | | `client-set-price` | A checkout/payment is created with an amount (`unit_amount`, `amount`, custom price) taken from the request, or with a price id from the request that is not checked against an allow-list. | The server looks up the amount/price from its own constant table or database by a product/plan key; the request's price id is checked against an allow-list; donation/"pay what you want" flows that are explicitly designed so. | | `unpaid-upgrade` | Code outside a verified webhook grants paid access (sets plan/tier/isPro/premium/subscription status active, adds credits) without confirming a payment with the provider in that request (retrieving the checkout session/subscription/payment and checking it is paid/active). | Verified webhook; the provider object is retrieved and its paid/active status checked before granting; admin-only endpoints; downgrades, trials and credit deductions. | | `duplicate-grant` | A verified webhook adds credits/balance by incrementing, and nothing prevents processing the same event twice (no stored event id, no unique constraint lookup, no status guard on the order). | Processed event ids are stored/checked; the grant is an idempotent set (`plan = 'pro'`); the order row is only updated when its status is still pending. | | `cancel-keeps-access` | The app sells subscriptions and grants access when a subscription payment arrives, but no code handles cancellation/expiry/update events and nothing re-checks subscription status with the provider later. | Cancellation/deleted/updated/expired events are handled; access is derived from a live provider lookup; one-time purchases only. | ## Case format ``` / case.json # labels ``` `case.json`: ```json { "id": "a1-017", "summary": "Pages API route deletes a document by id after checking the session only", "stack": ["pages-api", "next-auth", "prisma"], "expected": [ { "check": "access-control", "rule": "missing-ownership", "file": "pages/api/documents/[id].ts" } ] } ``` `expected` is empty for a secure case. Every expected problem names the file that contains the database write/read, checkout call or signature verification the rule is about (for `cancel-keeps-access`: the webhook file that grants access). A vulnerable case contains exactly the listed problems; everything else in it is correct. ## Scoring A detection is a `(case, check, rule, file)` tuple. True positive: it matches an expected tuple. False positive: a detection matching no expected tuple. False negative: an expected tuple with no detection. Precision, recall and F1 are reported per check and overall, with Wilson 95% intervals. A case-level view (any problem found vs. none) is reported alongside.