terraform — Seeded corpus

Seeded Terraform defects — committed secrets, public exposure, a rename with no moved block.

What this is

9 deliberate defects, each carrying a marker naming the rule in standards/ it violates. It measures recall, and its pass condition is that every one of the 5 BLOCKER markers is flagged. The rule id in each marker means the scorer grades attribution too: a reviewer that finds the bug but cites the wrong rule counts as caught, and separately as misattributed.

How to onboard it

Nothing is installed and nothing here is ever merged. The corpus is used by opening a throwaway pull request on a pilot repository already onboarded for this stack, adding this directory and seeded/clean together. Label it redline-exempt so the readiness gate does not block a pull request nobody will merge, wait for the automated review to finish, then score it.

seed-canary.yml does exactly this on a schedule and closes the pull request afterwards, including when the run fails.

How to use it

terminal

$ GH_TOKEN=... npx redlinegate metrics score-seeds --repo <org>/<repo> --pr <n>

Seed	Severity	Rule it violates	Defect
1	BLOCKER	`terraform/unpinned-source`	provider unpinned — floats to whatever is latest at apply time
2	BLOCKER	`terraform/secrets-in-config`	secret committed in a .tf file
3	BLOCKER	`terraform/public-exposure`	database exposed to the public internet
6	BLOCKER	`terraform/public-exposure`	0.0.0.0/0 ingress on a database port
7	BLOCKER	`terraform/overly-broad-iam`	fully unscoped IAM policy
4	HIGH	`terraform/missing-backup-retention`	no backup retention on a new stateful resource
5	HIGH	`terraform/missing-encryption`	encryption at rest not enabled on a new data store
8	HIGH	`terraform/count-vs-for-each`	count on an identity-bearing resource — index shift recreates siblings
9	SUGGESTION	`terraform/variable-metadata`	variable missing description and type

Expected output

A score from scripts/score-seeds.mjs: BLOCKER recall, false positives, and rule attribution. The canary appends it to data/seed-scores.jsonl and fails the run if BLOCKER recall drops below 1.0 or the clean corpus attracts a false positive. Of the 9 defects here, 5 BLOCKER, 3 HIGH and 1 SUGGESTION — only the BLOCKER count is a pass condition.

How to edit it

Add a defect by adding the code and one marker line above it. There is no separate expectations file to keep in sync — scripts/score-seeds.mjs parses the markers straight out of the source:

marker format

SEED <n> [BLOCKER|HIGH|SUGGESTION] (<stack>/<rule-slug>) <short description>

scripts/validate.mjs fails CI if a seed cites a rule that does not exist, or claims a severity higher than that rule carries in the standard — so a marker cannot quietly drift away from the rule it is testing.

The full file

seeded/terraform/seeded_violations.tf · 66 lines · 2.0 KB

# DO NOT MERGE — Redline validation seed.
# Every `SEED n [SEVERITY] (rule-id)` marker must be flagged at that severity or higher,
# citing that rule id. Score with scripts/score-seeds.mjs.

# SEED 1 [BLOCKER] (terraform/unpinned-source) provider unpinned — floats to whatever is latest at apply time
provider "aws" {
  region = "eu-west-1"
}

# SEED 2 [BLOCKER] (terraform/secrets-in-config) secret committed in a .tf file
variable "db_password" {
  type    = string
  default = "Pr0d-Sup3r-S3cret-2026!"
}

resource "aws_db_instance" "billing" {
  identifier = "billing-prod"
  engine     = "postgres"
  password   = var.db_password

  # SEED 3 [BLOCKER] (terraform/public-exposure) database exposed to the public internet
  publicly_accessible = true

  # SEED 4 [HIGH] (terraform/missing-backup-retention) no backup retention on a new stateful resource
  backup_retention_period = 0

  # SEED 5 [HIGH] (terraform/missing-encryption) encryption at rest not enabled on a new data store
  storage_encrypted = false
}

resource "aws_security_group_rule" "wide_open" {
  type        = "ingress"
  from_port   = 5432
  to_port     = 5432
  protocol    = "tcp"
  # SEED 6 [BLOCKER] (terraform/public-exposure) 0.0.0.0/0 ingress on a database port
  cidr_blocks = ["0.0.0.0/0"]
  security_group_id = "sg-0123456789abcdef0"
}

resource "aws_iam_role_policy" "billing" {
  name = "billing"
  role = "billing-role"

  # SEED 7 [BLOCKER] (terraform/overly-broad-iam) fully unscoped IAM policy
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "*"
      Resource = "*"
    }]
  })
}

# SEED 8 [HIGH] (terraform/count-vs-for-each) count on an identity-bearing resource — index shift recreates siblings
resource "aws_s3_bucket" "reports" {
  count  = length(var.markets)
  bucket = "reports-${var.markets[count.index]}"
}

# SEED 9 [SUGGESTION] (terraform/variable-metadata) variable missing description and type
variable "markets" {
  default = ["uk", "de", "es"]
}