Scoreboard

How a number gets here

The scorer reads a review that has already happened. It takes a repository and a pull request number, fetches the comments an automated reviewer posted there, and grades them against the markers in the corpus. There is no local mode and nothing to run against the files on disk — a benchmark whose subject is a reviewer needs a reviewer to have reviewed something.

1. A canary repository, with an automated reviewer actually enabled on it. That reviewer — Copilot code review, Claude, Codex or another — is the thing being measured. A repository that has quietly lost its review entitlement looks exactly like a repository with nothing to find, which is the confusion this whole exercise exists to remove.
2. The redline-metrics repository, carrying seed-canary.yml. The workflow does not live in this repository, and neither does the history it writes. It needs vars.CANARY_TARGETS (a JSON array of {repo, stack}), secrets.REDLINE_CANARY_TOKEN scoped to the canary repos only — it opens and closes pull requests — and secrets.REDLINE_ORG_READ_TOKEN to check out the corpus from here.
3. The cycle, weekly on a Monday or on demand. It opens a branch on each target carrying seeded/<stack> and seeded/clean, labelled redline-exempt so the readiness gate does not block a pull request nobody will merge; waits for review comments to stop arriving (two stable polls, up to forty minutes); scores what was posted; then closes and deletes the pull request and branch, including when the run fails.
4. The record. Each score is appended as one JSON line to a history file, and the run fails if any target’s BLOCKER recall is below 1.0 or the clean corpus attracted a false positive. That file is what this page reads.

# inside the canary run — scores a pull request the reviewer has already commented onGH_TOKEN=… npx redlinegate metrics score-seeds \ --repo <org>/<canary-repo> --pr <n> \ --history data/seed-scores.jsonl --baseline

Everything above is in the open: the scorer, the workflow that drives it, and the corpus with its answers. Run it against your own reviewer on your own repository and you will get a number computed exactly the way the one published here will be.

The four numbers

A review system fails in more than one direction, and only one of those directions is loud. These are counted by scripts/score-seeds.mjs from the reviewer’s own comments; a comment is matched to at most one marker, within four lines of it, and only when it carries the marked severity or higher.

What it is scored against

Every defect carries a machine-readable marker on or just above its line, naming the severity it must be flagged at and the rule id that must be cited. The scorer parses those markers straight out of the files, so there is no separate expectations file to drift out of sync — and nothing about the answers is private.

// DO NOT MERGE — Redline validation seed.// Every `SEED n [SEVERITY] (rule-id)` marker must be flagged at that severity or higher,// citing that rule id. Score with scripts/score-seeds.mjs. // SEED 1 [BLOCKER] (core/hardcoded-secrets) hardcoded credentialconst SLACK_WEBHOOK = 'https://hooks.acme-chat.internal/services/T0A1B2C3/B4D5E6F7/Zx9Qr2Kd7Wm4Tz6Hb3Vy8Wn1'; // SEED 2 [BLOCKER] (javascript/prototype-pollution) prototype pollution: recursive merge over external inputfunction merge(target, source) {

220 markers across 18 stacks — angular, csharp, dom, express, go, java, javascript, kotlin, microservices, nextjs, nodejs, python, react, react-native, svelte, swift, terraform, vue — plus seeded/clean/, whose pass condition is silence. The corpus is published in full: browse every seeded file, or read them in the repository. None of it is ever merged anywhere — it contains live-looking credentials and deliberately broken code.