Telemetry & validation
Redline measures what was acted on, not what was flagged — and regression-tests the reviewer itself. A rule nobody acts on is noise, and noise is visible.
This pipeline is GitHub-only today, carried over unchanged from the workflow-bundle era. The CLI onboards and verifies repos on either host; wiring telemetry to Azure DevOps is later-phase work.
What is measured
- Acted-on rate — resolved review threads versus findings left stale and outdated, per severity and per rule id.
- Noise — false positives on the clean corpus, untagged findings that ignored the contract, exemption usage.
- Recall — seeded BLOCKERs found, per stack, with history per pilot repo.
Records carry rules: {<id>: {fired, resolved, stale}} — turning “42% of findings are ignored” into a named list of the rules responsible. Deliberately not measured: per-author quality.
Where it surfaces
- Monday Teams digest — an Adaptive Card with the acted-on rate, the noisiest-rules tuning queue and the worst pilot seed score.
- Org inbox — a prioritised PR inbox on GitHub Pages, refusing to build unless Pages visibility is acknowledged.
- Dashboard — static, dependency-free: trends, seed-recall history, the full tables.
Reading the dashboard
In production these are scheduled workflows, not things anyone types — the CLI exists so a scheduled run is reproducible in a terminal when it looks wrong. registry derives who is onboarded, collect pulls review outcomes with a read-only token, and dashboard builds a static page.
$ redline registry --org acme --source acme/redline $ redline metrics collect --org acme --days 30 $ redline metrics dashboard --org acme --data data --out dist
The hero number is findings acted on. These are the tiles beside it, and what each one means when it is not healthy:
| Tile | Healthy | What it means when it is not |
|---|---|---|
| Findings acted on | rising, then steady | Low means review is running and being ignored. That is the failure this whole plane exists to make visible, and it is a rule problem before it is a people problem. |
| PRs with findings | some, not all | Near 0% and the reviewer is not running, or the rules reach nothing this estate writes. Near 100% and it is flagging everything, which is the same as flagging nothing. |
| Ignored | falling | Findings left stale and outdated. Read it with the tuning queue — one noisy rule usually explains most of it. |
| Gate exemptions | rare | A rising share of pull requests carrying the label means the gate asks for something people cannot reasonably give. Exemptions reach the process checks only, so this never covers a security check. |
| Seed BLOCKER recall | 100% | Anything less and the reviewer has silently stopped catching known defects. The dashboard raises a warning and says not to widen the rollout. |
| Enforcing | growing slowly | How much of the estate is on a blocking rung. Flat at zero forever means the evidence to promote has never been produced. |
The page also carries the rule tuning queue: rules that fire often and are rarely acted on, named individually. That list is the product. Cut, narrow or downgrade what is on it before adding new rules.
A dashboard built in week one is empty, and that is a sample size rather than a failure. Nothing here reports a number it could not compute: a missing figure says why instead of defaulting to zero, because a plane that fills gaps with zeros reports a stalled collector as a quiet week.
The seeded corpus
seeded/ holds 117 BLOCKER and 52 HIGH seeds across all 16 stacks, each marker citing the rule it violates — plus seeded/clean/, correct code that must produce zero findings. Recall alone is not enough: a reviewer that flags everything scores perfect recall.
$ GH_TOKEN=... npx redlinegate metrics score-seeds --repo acme/pilot --pr 4 --json blocker_recall: 1.0 false_positives_on_clean: 0 untagged: 0
Three numbers, because a review system fails in three ways: recall (seeded defects flagged at the expected severity), precision (comments on seeded/clean/**, which should never happen) and attribution (findings that cited the correct rule id). --history <file>.jsonl appends each run, so recall has a trend rather than a single reading.
The scorer is vendor-neutral — it reads the severity prefix and a bot list, not the vendor. Comparing Copilot against Claude on identical input is one command per candidate.
The reviewer is regression-tested
workflows/seed-canary.yml opens a PR of known-bad code against a canary repo weekly, scores the review, and fails on any drop in BLOCKER recall or any false positive. Nothing else detects a reviewer that quietly stopped working — a repo with no findings looks identical to a repo with no defects, and a model upgrade can move you from one to the other without anyone touching a rule.