chore(ci): close three validate.yml coverage gaps #38

Merged

zach merged 3 commits from chore/validate-coverage into fix/seed-integrity-and-series 2026-07-25 21:47:14 +00:00

Conversation 0 Commits 3 Files changed 6 +396 -21

zach commented 2026-07-25 21:37:54 +00:00

Owner

Copy link

Stacked on #31 (fix/seed-integrity-and-series), with #30 merged in so both escape hatches exist and the new check can be verified against a complete tree. Merge #30 and #31 first.

Each of these three would have caught a defect already found in this review round.

1. escape-hatches-are-plumbed

A repo variable only reaches a script if the step running it lists the name in env:. Nothing enforced that, and the result was a documented override that did nothing: check-hardening-drift.sh honoured ALLOW_HARDENING_REGRESSION, both POLICY.md:296 and config-architecture.md:138 told the operator to set it, and the drift step's env: listed only WORKDIR and SERIES. The operator's only route past a drift failure was a commit to main — which the stale-tag guard then turns into "delete and re-cut the tag", mid-CVE-SLA.

This job asserts every ALLOW_*/REQUIRE_* name a script reads is both passed via env: vars.<NAME> by some workflow and documented under docs/operators/ or in POLICY.md.

Worth being precise: a workflow-shell linter would not have caught the original — a missing YAML env: key involves no shell at all. That's why this is a separate check rather than folded into (3).

It earned its place immediately. Run against the tree, it flagged ALLOW_CROSS_SERIES_SEED as plumbed but undocumented — a gap I'd left in #31. Fixed there, in the branch that introduced the hatch, rather than papered over here.

2. container-digests-pinned, replacing no-placeholder-digests

The old job grepped containers/ for the placeholder token. That token now appears only in shim/Dockerfile, which the job deliberately excludes — so it could not fail. A check that cannot fail is worse than no check, because it reads as coverage.

Inverted to a positive assertion that cannot go dormant: every FROM under containers/ must carry a concrete @sha256 digest. Verified that a placeholder there now does fail, so the new form covers the old form's stated purpose.

Folded in the adjacent free win: build-kernel.yml's container.image (line 34) and the CONTAINER_IMAGE that goes into every manifest (line 424) were coupled only by a "Keep this digest in sync" comment. If they diverge, every manifest records a build environment the build did not use, and repro-check faithfully rebuilds in an image that never produced the release — mismatches that look like tampering.

3. workflow-shell + tools/lint-workflow-shell.py

The shellcheck job covers build/ and tools/ only. The largest and least-tested shell in this repo is in workflow run: blocks — cve-watch.yml alone is ~150 lines with jq, comm, an api_call helper and pipefail interactions — and a parse error there surfaces only when that job actually runs, which for cve-watch or prune could be weeks later.

Gated at severity=error: 43 inline scripts, 0 findings, so it lands green and stays signal rather than a wall of style noise. The backlog above that is small and stated in the job comment rather than hidden: 1 finding at warning, 3 at info.

The linter lives in tools/ rather than as an inline heredoc precisely because the premise is that inline workflow shell goes unlinted — so the linter itself belongs where shellcheck and py_compile already cover it.

It substitutes a placeholder for each expression template (a raw template isn't valid shell), which produces one deliberate interaction: comparing a template in a run: body shows up as SC2050 "expression is constant". That isn't noise — it means a template is being used where workflow-expressions (#29) now requires an env: variable. The two checks reinforce each other.

Verification

Exit codes checked without pipeline masking (my first pass read head's status, not the script's):

case	exit	want
real tree — escape-hatches	0	0
real tree — container-digests	0	0
real tree — workflow-shell	0	0
unplumbed hatch added to a script	1	1
hatch renamed in the workflow only	1	1
a FROM de-pinned	1	1
placeholder digest under containers/	1	1
divergent CONTAINER_IMAGE	1	1
containers/ with no FROM lines (vacuity guard)	1	1
genuinely broken workflow shell (unclosed if, bad array)	1	1

The linter's positive control needed a real parse error — my first attempt used warning-level issues against an error-level gate and passed, which is exactly why the gate level is now stated explicitly in the job comment rather than left implicit.

shellcheck, py_compile, yamllint -d relaxed clean.

Note

prune.py's doctests and the lines.py smoke test — the fourth gap from the plan — are in #37 instead, since they belong with the prune changes.

🤖 Generated with Claude Code

> **Stacked on #31** (`fix/seed-integrity-and-series`), with **#30** merged in so both escape hatches exist and the new check can be verified against a complete tree. Merge #30 and #31 first. Each of these three would have caught a defect already found in this review round. ## 1. `escape-hatches-are-plumbed` A repo variable only reaches a script if the step running it lists the name in `env:`. Nothing enforced that, and the result was a documented override that **did nothing**: `check-hardening-drift.sh` honoured `ALLOW_HARDENING_REGRESSION`, both `POLICY.md:296` and `config-architecture.md:138` told the operator to set it, and the drift step's `env:` listed only `WORKDIR` and `SERIES`. The operator's only route past a drift failure was a commit to `main` — which the stale-tag guard then turns into "delete and re-cut the tag", mid-CVE-SLA. This job asserts every `ALLOW_*`/`REQUIRE_*` name a script reads is **both** passed via `env: vars.<NAME>` by some workflow **and** documented under `docs/operators/` or in `POLICY.md`. Worth being precise: a workflow-shell linter would **not** have caught the original — a missing YAML `env:` key involves no shell at all. That's why this is a separate check rather than folded into (3). **It earned its place immediately.** Run against the tree, it flagged `ALLOW_CROSS_SERIES_SEED` as plumbed but undocumented — a gap I'd left in #31. Fixed there, in the branch that introduced the hatch, rather than papered over here. ## 2. `container-digests-pinned`, replacing `no-placeholder-digests` The old job grepped `containers/` for the placeholder token. That token now appears **only** in `shim/Dockerfile`, which the job deliberately excludes — so it **could not fail**. A check that cannot fail is worse than no check, because it reads as coverage. Inverted to a positive assertion that cannot go dormant: every `FROM` under `containers/` must carry a concrete `@sha256` digest. Verified that a placeholder there now *does* fail, so the new form covers the old form's stated purpose. Folded in the adjacent free win: `build-kernel.yml`'s `container.image` (line 34) and the `CONTAINER_IMAGE` that goes into every manifest (line 424) were coupled only by a *"Keep this digest in sync"* comment. If they diverge, every manifest records a build environment the build did not use, and `repro-check` faithfully rebuilds in an image that never produced the release — mismatches that look like tampering. ## 3. `workflow-shell` + `tools/lint-workflow-shell.py` The `shellcheck` job covers `build/` and `tools/` only. The largest and least-tested shell in this repo is in workflow `run:` blocks — `cve-watch.yml` alone is ~150 lines with `jq`, `comm`, an `api_call` helper and pipefail interactions — and a parse error there surfaces only when that job actually runs, which for `cve-watch` or `prune` could be weeks later. Gated at `severity=error`: **43 inline scripts, 0 findings**, so it lands green and stays signal rather than a wall of style noise. The backlog above that is small and stated in the job comment rather than hidden: **1 finding at `warning`, 3 at `info`**. The linter lives in `tools/` rather than as an inline heredoc precisely because the premise is that inline workflow shell goes unlinted — so the linter itself belongs where `shellcheck` and `py_compile` already cover it. It substitutes a placeholder for each expression template (a raw template isn't valid shell), which produces one deliberate interaction: comparing a template in a `run:` body shows up as `SC2050` *"expression is constant"*. That isn't noise — it means a template is being used where `workflow-expressions` (#29) now requires an `env:` variable. The two checks reinforce each other. ## Verification Exit codes checked **without** pipeline masking (my first pass read `head`'s status, not the script's): | case | exit | want | |---|---|---| | real tree — escape-hatches | 0 | 0 | | real tree — container-digests | 0 | 0 | | real tree — workflow-shell | 0 | 0 | | unplumbed hatch added to a script | 1 | 1 | | hatch renamed in the workflow only | 1 | 1 | | a `FROM` de-pinned | 1 | 1 | | placeholder digest under `containers/` | 1 | 1 | | divergent `CONTAINER_IMAGE` | 1 | 1 | | `containers/` with no `FROM` lines (vacuity guard) | 1 | 1 | | genuinely broken workflow shell (unclosed `if`, bad array) | 1 | 1 | The linter's positive control needed a **real parse error** — my first attempt used warning-level issues against an error-level gate and passed, which is exactly why the gate level is now stated explicitly in the job comment rather than left implicit. `shellcheck`, `py_compile`, `yamllint -d relaxed` clean. ## Note `prune.py`'s doctests and the `lines.py` smoke test — the fourth gap from the plan — are in **#37** instead, since they belong with the prune changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

zach added 3 commits 2026-07-25 21:37:54 +00:00

fix(ci): make both safety guards fail loudly instead of vacuously ... f21868de50

The two checks that stand between a seed bump and a silently weakened kernel
could both report success while checking nothing. Verified empirically, not
inferred.

tools/check-hardening-drift.sh: run against an EMPTY baseline, and against a
50-line truncation, it printed

  hardening-drift: OK (no enabled hardening symbol regressed vs baseline)

and exited 0 both times. The shape is `while ... done < <(enabled_hardening
"$baseline")`: no matches means the loop never iterates, `regressed` stays 0,
and the success path runs. build-kernel.yml only tests that the baseline file
EXISTS, so a truncated or zero-length releases/<tag>/final.config would retire
the only guard covering the 31 inherited hardening symbols -- with a green log
and no output distinguishable from a real pass.

build/configure-kernel.sh's fidelity assertion: same shape, same outcome. An
empty intent.config yields "fidelity assertion: OK (0 intent lines, all
landed)". `set -e` cannot catch it, because the inline `$(grep -c ...)` in that
message is a command substitution used as an ARGUMENT, which never fails the
shell.

Fixes, following the anti-vacuous-pass pattern validate.yml already uses in its
intent-matches-policy and dep-list-parity jobs:

  - drift: a >=1000 CONFIG_-line floor on BOTH inputs (real configs carry
    ~6,800, so this only catches truncation); the baseline symbol set is
    materialized before the loop and floored at 15 (31 today) -- a second,
    distinct check, because exclude_re is built FROM intent.config and a
    mangled intent can collapse the comparison set against a perfectly good
    baseline; the success line now reports the count.
  - fidelity: parse once into a variable, then three guards -- nothing parsed,
    a >=60 line floor (92 today), and a candidates-vs-parsed equality check.
    The third catches what a floor cannot: CRLF endings, trailing whitespace
    after "is not set", "#CONFIG_X" without the space. Each of those silently
    drops that pin instead of failing. The CRLF case is not hypothetical -- it
    parses all 69 `=y` pins and drops all 23 off-pins, and the guard reports
    exactly which lines fell through.

Separately: ALLOW_HARDENING_REGRESSION was inert. check-hardening-drift.sh has
honoured it since it was written, and both configs/POLICY.md and
docs/operators/config-architecture.md tell the operator to set it as a repo
variable -- but the drift step's env: block listed only WORKDIR and SERIES, so
`vars.ALLOW_HARDENING_REGRESSION` never reached the script. Without it the only
route past a drift failure is a commit to main, which the stale-tag guard then
turns into "delete and re-cut the tag", mid-CVE-SLA. Plumb it the same way
ALLOW_STALE_TAG and REQUIRE_NET_ISOLATION already are, hard-fail an empty
SERIES (which would silently skip the guard), and make the no-baseline branch a
:⚠️: so a SKIPPED guard is visible in the run summary.

Document all three hatches in a repo-variables table in runner-setup.md, which
previously mentioned only REQUIRE_NET_ISOLATION, in prose.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Merge branch 'fix/guards-fail-loudly' into chore/validate-coverage 3fba88a63f

chore(ci): close three validate.yml coverage gaps ... 4d8b8180ff

Each of these would have caught a defect already found in this review round.

1. escape-hatches-are-plumbed.

A repo variable only reaches a script if the step running it lists the name in
env:. Nothing enforced that, and the result was a documented override that did
nothing: check-hardening-drift.sh honoured ALLOW_HARDENING_REGRESSION, both
POLICY.md and config-architecture.md told the operator to set it, and the drift
step's env: block listed only WORKDIR and SERIES. The operator's only route past
a drift failure was a commit to main -- which the stale-tag guard then turns into
"delete and re-cut the tag", mid-CVE-SLA.

This job asserts every ALLOW_*/REQUIRE_* name a script reads is BOTH passed via
env: vars.<NAME> by some workflow AND documented under docs/operators/ or in
POLICY.md. Worth noting a workflow-shell linter would NOT have caught the
original: a missing YAML env: key involves no shell at all.

It earned its place immediately -- run against the tree it flagged
ALLOW_CROSS_SERIES_SEED as plumbed but undocumented, a gap left in the seed PR.
Fixed there, in the commit that introduced the hatch.

2. container-digests-pinned, replacing no-placeholder-digests.

The old job grepped containers/ for the placeholder token. That token now appears
ONLY in shim/Dockerfile, which the job deliberately excludes -- so it could not
fail. A check that cannot fail is worse than no check, because it reads as
coverage.

Inverted to a positive assertion that cannot go dormant: every FROM under
containers/ must carry a concrete @sha256 digest (verified that a placeholder
there now fails, i.e. the new form covers the old form's stated purpose). Folded
in the adjacent free win: build-kernel.yml's container.image and the
CONTAINER_IMAGE that goes into every manifest were coupled only by a "keep this
digest in sync" comment. If they diverge, every manifest records a build
environment the build did not use, and repro-check faithfully rebuilds in an
image that never produced the release -- mismatches that look like tampering.

3. workflow-shell + tools/lint-workflow-shell.py.

The shellcheck job covers build/ and tools/ only. The largest and least-tested
shell in this repo is in workflow run: blocks -- cve-watch.yml alone is ~150
lines with jq, comm, an api_call helper and pipefail interactions -- and a parse
error there surfaces only when that job runs, which for cve-watch or prune could
be weeks later.

Gated at severity=error: 43 inline scripts, 0 findings, so it lands green and
stays signal rather than noise. The backlog above that is small and stated in the
job comment rather than hidden: 1 finding at warning, 3 at info.

The linter lives in tools/ rather than as an inline heredoc, precisely because the
premise is that inline workflow shell goes unlinted -- so the linter itself
belongs where shellcheck and py_compile already cover it. It substitutes a
placeholder for each expression template (a raw template is not valid shell),
which produces one deliberate interaction: comparing a template in a run: body
shows up as SC2050 "expression is constant". That is not noise -- it means a
template is being used where workflow-expressions now requires an env: variable,
so the two checks reinforce each other.

Verified in both directions throughout, checking exit codes without pipeline
masking: real tree passes all three; an unplumbed hatch, a renamed hatch, a
de-pinned FROM, a divergent CONTAINER_IMAGE, a containers/ dir with no FROM lines
(vacuity guard), and genuinely broken workflow shell each fail. The linter's own
positive control needed a real parse error -- my first attempt used
warning-level issues against an error-level gate and passed, which is worth
recording as the reason the gate level is stated explicitly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

zach referenced this pull request 2026-07-25 21:39:31 +00:00

feat(publish): un-draft the release as the final step #39

zach merged commit 417dfbc08e into fix/seed-integrity-and-series 2026-07-25 21:47:14 +00:00

zach referenced this pull request from a commit 2026-07-25 21:47:16 +00:00

Merge pull request 'chore(ci): close three validate.yml coverage gaps' (#38) from chore/validate-coverage into fix/seed-integrity-and-series

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

unredacted/linux-hardened-unredacted!38

No description provided.