feat(manifest): add schema_version and ci_run_number #33

Merged
zach merged 5 commits from feat/manifest-schema-v2 into main 2026-07-25 21:47:43 +00:00
Owner

Must land before the next build — manifests are append-only, so a release built without these fields can never gain them.

ci_run_number closes a provenance hole

build-kernel.yml:262 passes github.run_number to build-debs.sh as REVISION, which becomes KBUILD_BUILD_VERSION (build-debs.sh:232) and therefore UTS_VERSION inside vmlinux — while tools/repro-check.sh:146 hard-codes REVISION=1 and the manifest recorded the value nowhere (verified: no such key in any of the twelve manifests).

So every published release contains a number in its kernel banner that no verifier can discover, and no independent rebuild of linux-image or linux-image-*-dbg could ever match. A follow-up removes the build's dependency on it entirely (deriving KBUILD_BUILD_VERSION from the tag instead); this field preserves the traceability that change gives up.

schema_version lets a reader branch instead of assume

An absent schema_version marks the twelve legacy releases, which repro-check.sh must report as partially verifiable rather than failed — they are missing a field that cannot be reconstructed. Without this marker the nightly check stays permanently red for a reason nobody can act on, which is worse than having no check.

releases/README.md now documents what each version promises, including that v7.0.10-hardened1 and v7.0.10-hardened1.2 carry null for kernel_version/hardened_revision and are not checkable at all.

Verification

case result
--ci-run-number 58 schema_version: 2, ci_run_number: "58"
flag omitted key absent (not null), schema_version: 2
existing releases/*/manifest.json unchanged, still report as legacy

py_compile clean, yamllint -d relaxed exit 0. No behaviour change to existing manifests; both fields are additive.

🤖 Generated with Claude Code

**Must land before the next build** — manifests are append-only, so a release built without these fields can never gain them. ## `ci_run_number` closes a provenance hole `build-kernel.yml:262` passes `github.run_number` to `build-debs.sh` as `REVISION`, which becomes `KBUILD_BUILD_VERSION` ([build-debs.sh:232](build/build-debs.sh:232)) and therefore `UTS_VERSION` inside `vmlinux` — while `tools/repro-check.sh:146` hard-codes `REVISION=1` and the manifest recorded the value **nowhere** (verified: no such key in any of the twelve manifests). So every published release contains a number in its kernel banner that no verifier can discover, and no independent rebuild of `linux-image` or `linux-image-*-dbg` could ever match. A follow-up removes the build's dependency on it entirely (deriving `KBUILD_BUILD_VERSION` from the tag instead); this field preserves the traceability that change gives up. ## `schema_version` lets a reader branch instead of assume An **absent** `schema_version` marks the twelve legacy releases, which `repro-check.sh` must report as *partially* verifiable rather than failed — they are missing a field that cannot be reconstructed. Without this marker the nightly check stays permanently red for a reason nobody can act on, which is worse than having no check. `releases/README.md` now documents what each version promises, including that `v7.0.10-hardened1` and `v7.0.10-hardened1.2` carry `null` for `kernel_version`/`hardened_revision` and are not checkable at all. ## Verification | case | result | |---|---| | `--ci-run-number 58` | `schema_version: 2`, `ci_run_number: "58"` | | flag omitted | key **absent** (not `null`), `schema_version: 2` | | existing `releases/*/manifest.json` | unchanged, still report as legacy | `py_compile` clean, `yamllint -d relaxed` exit 0. No behaviour change to existing manifests; both fields are additive. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(manifest): add schema_version and ci_run_number
All checks were successful
validate / shellcheck (pull_request) Successful in 12s
validate / yamllint (pull_request) Successful in 11s
validate / workflow-expressions (pull_request) Successful in 5s
validate / pycompile (pull_request) Successful in 4s
validate / intent-matches-policy (pull_request) Successful in 5s
validate / dep-list-parity (pull_request) Successful in 4s
validate / no-placeholder-digests (pull_request) Has been skipped
d961e0f414
Must land before the next build: manifests are append-only, so a release built
without these fields can never gain them.

ci_run_number closes a provenance hole. build-kernel.yml passes
github.run_number to build-debs.sh as REVISION, which becomes
KBUILD_BUILD_VERSION and therefore UTS_VERSION inside vmlinux -- while
tools/repro-check.sh hard-codes REVISION=1 and the manifest recorded the value
nowhere. So for all twelve published releases the shipped kernel contains a
number no verifier can discover, and no independent rebuild of linux-image or
linux-image-dbg could ever match. A follow-up removes the build's dependency on
it entirely (deriving KBUILD_BUILD_VERSION from the tag instead); this field
preserves the traceability that change gives up.

schema_version lets a reader branch on which fields exist rather than assume.
The distinction matters immediately: an ABSENT schema_version marks the twelve
legacy releases, which repro-check must report as PARTIALLY verifiable rather
than failed -- they are missing a field that cannot be reconstructed. Without
this marker the nightly check is permanently red for a reason nobody can act on,
which is worse than no check. releases/README.md now documents what each version
promises, including that v7.0.10-hardened1 and v7.0.10-hardened1.2 carry null
kernel_version/hardened_revision and are not checkable at all.

No behaviour change to existing manifests; both new fields are additive and
ci_run_number is omitted rather than nulled when unset.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(build): pin the module-signing key to a fixed in-tree path
All checks were successful
validate / dep-list-parity (pull_request) Successful in 5s
validate / no-placeholder-digests (pull_request) Has been skipped
validate / shellcheck (pull_request) Successful in 12s
validate / yamllint (pull_request) Successful in 12s
validate / workflow-expressions (pull_request) Successful in 4s
validate / pycompile (pull_request) Successful in 4s
validate / intent-matches-policy (pull_request) Successful in 5s
aa38d1c685
CHANGES SHIPPED BYTES. Verify against the draft release before publishing.

Every kernel this project has shipped publishes a runner temp-directory name.
Confirmed by reading it off a running machine, not inferred:

  # zcat /proc/config.gz | grep MODULE_SIG_KEY
  CONFIG_MODULE_SIG_KEY="/tmp/tmp.TliL5BYWSL/signing.pem"

Mechanism: configure-kernel.sh appended a per-build `mktemp -d` path to the
TREE's .config after the fidelity assertion, and build-debs.sh normalized that
value back to the placeholder only in the separately-captured final.config. But
bindeb-pkg ships the tree's .config as /boot/config-<rel>, embeds it into
vmlinux via CONFIG_IKCONFIG (= /proc/config.gz), and packages it in the headers
.deb. Three consequences, ascending:

  1. An information leak: the runner's temp-dir name is in every kernel. The
     private key itself was never shipped.
  2. Same-tag rebuilds were never bit-identical, because mktemp differs per run
     -- directly contradicting build-debs.sh's own comment.
  3. Possibly broken DKMS, untested either way: MODULE_SIG_ALL=y and the headers
     package carries that .config, so an out-of-tree build going through the
     kernel's own modules_install signing path reaches for a key file that does
     not exist. docs/users/dkms.md already concedes that path is unvalidated
     against a published release, so this is an open interaction. The
     Verification section below says how to settle it.

Fix: make CONFIG_MODULE_SIG_KEY a CONSTANT in-tree path, so the value the
fidelity assertion checks is the value that ships. Nothing to rewrite, nothing
to normalize, no ordering requirement against the assertion.

certs/unredacted-signing.pem, deliberately NOT the stock certs/signing_key.pem:
upstream certs/Makefile guards a FORCE + if_changed,gen_key rule with
`ifeq ($(CONFIG_MODULE_SIG_KEY),certs/signing_key.pem)`, so under the default
name kbuild would REGENERATE our pre-placed key with a random one and ship a
kernel trusting a key nobody holds. That name is also on MRPROPER_FILES. With a
non-default name no generator rule exists, so a missing key fails loudly.

That kbuild behaviour is the load-bearing assumption here and cannot be tested
without a real build, so build-debs.sh now asserts the outcome instead of
trusting it. Three new hard failures, all post-`make`, so the cost of tripping
one is a rebuild and never a bad release:

  (a) cert identity: the DER kbuild extracted into certs/signing_key.x509 --
      which it links into .builtin_trusted_keys, the trust root for every module
      signature this kernel accepts -- must equal our PEM's certificate. This is
      what turns a wrong kbuild assumption into an aborted build rather than a
      silently mis-keyed kernel.
  (b) no private key in any .deb. We now put a key inside a tree bindeb-pkg
      packages from; mkdebian's file list should not match certs/*.pem, but the
      cost of being wrong is publishing a signing key, so assert it.
  (c) config parity: the /boot/config-* inside the image .deb must be
      byte-identical to the captured final.config. That divergence IS this bug,
      undetected for twelve releases; now it cannot recur silently.

Also removes the CI run number from the shipped bytes. It reached
KBUILD_BUILD_VERSION and so UTS_VERSION inside vmlinux while being recorded
nowhere and hard-coded to 1 in repro-check.sh -- three independent reasons a
rebuild could never match. KBUILD_BUILD_VERSION now derives from the tag
(PACKAGING_REVISION, default 1); traceability moved to manifest.ci_run_number.

repro-check.sh: passes PACKAGING_REVISION parsed from the tag, which makes the
five .P releases structurally verifiable for the first time (previously the
rebuild emitted ...-hardened1 .debs and every file reported MISSING, reading
like tampering rather than a missing input); cross-checks the manifest's
kernel_version/hardened_revision against the tag; and generates a throwaway
signing key, since without one the rebuild would now fail at `make` rather than
silently generating a random one. Byte-identity on the cert-embedding artifacts
(vmlinuz, System.map, -dbg) still needs the real private key -- handing over the
published .der cannot help, because signing needs the private half -- so
REPRO_SIGNING_KEY_PATH is honoured for operators who hold it. Localizing that
divergence properly is follow-up work on the comparison itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat(repro): content-addressed .deb hashing; per-package verdict
All checks were successful
validate / shellcheck (pull_request) Successful in 12s
validate / yamllint (pull_request) Successful in 19s
validate / workflow-expressions (pull_request) Successful in 4s
validate / pycompile (pull_request) Successful in 4s
validate / intent-matches-policy (pull_request) Successful in 5s
validate / dep-list-parity (pull_request) Successful in 4s
validate / no-placeholder-digests (pull_request) Has been skipped
62b0016e5a
repro-check could not pass for any release, for four independent reasons. PR #35
removed two (the unrecorded CI run number in UTS_VERSION, and the missing
PACKAGING_REVISION). This removes the other two.

3. The comparison hashed a REPACK.

Both sides ran strip-signatures.sh, which ends in `dpkg-deb -b` -- and that
recompresses data.tar with the local dpkg/liblzma defaults. The build side ran it
inside the pinned container; repro-check.sh ran it on the bare host. So a
verifier on any other distro got a different hash for EVERY package, including
the ~800-byte meta packages where the strip is a complete no-op, with nothing
explaining why. strip-signatures.sh's own header says "we never recompress:
xz/zstd output isn't byte-stable across tool versions" -- true for .ko, equally
true for data.tar.

New tools/canonical-deb-hash.py never repacks. Pure stdlib: parse the ar
container, decompress control/data tar, canonicalize each member's content,
digest a sorted description of the tree. Identical output on Debian, Arch or
macOS, and it drops the sbattach and diffoscope host dependencies entirely (the
PE certificate table is parsed directly, which also makes the transform
symmetric on both sides rather than relying on sbattach being a perfect inverse
of sbsign).

Deliberately excluded from the digest: tar/ar member order, compression, and
member mtimes -- all packaging metadata, and mtime clamping varies by dpkg
version, which is the false-positive class being removed. SOURCE_DATE_EPOCH is
recorded in the manifest and fed back into the rebuild, so it is checked by
other means. Included: path, type, permission bits, uid/gid, size, link target,
and sha256 of canonicalized content.

4. The verdict was one boolean per package, so "cannot check this" read as
   "this failed".

kbuild links the signing certificate's DER into .builtin_trusted_keys, so a
rebuild with a different key produces a different vmlinuz -- and, since a
different-length cert shifts what follows, a different System.map and -dbg
payload. Handing a verifier the published .der cannot fix that: sign-file needs
the private half, so a cert+foreign-key PEM aborts on X509_check_private_key.
That is inherent, so localize it instead of pretending otherwise.

Two digests per package: key_independent_sha256 excludes boot/vmlinuz-*,
boot/System.map-* and usr/lib/debug/**; canonical_sha256 covers everything. A
third party gets a HARD pass/fail on the first -- which still covers all ~5,000
modules, /boot/config-* and the modules.* indexes -- and an operator with
REPRO_SIGNING_KEY_PATH gets the second. Verdict is now a per-package table
(OK exact / OK* / FAIL / SKIP) with a summary line, and it refuses to report
success if nothing was comparable.

Also removed: the released-.deb fetch block, which was gated on a manifest field
(`canonical_path`) that build-manifest.py never wrote -- so it never ran,
$work/released/ was always empty, and repro-check.yml's "upload diff report on
failure" always uploaded nothing. A comment now records that comparing against
PUBLISHED bytes is a genuinely stronger check that is still not attempted, so
nobody reads its absence as coverage.

Manifest goes to schema_version 3. unsigned_deb_sha256 is kept as an alias of
canonical_sha256 so existing readers keep working; the two-column TSV form is
still accepted. Legacy manifests (absent schema_version, or < 3) are reported
SKIP/partial rather than FAIL: they lack fields no rebuild can reconstruct, and
a permanently red check nobody can act on is worse than an honest "not
verifiable".

Docs corrected where they overclaimed. verifying-packages.md said exit 0
"confirmed the published binaries correspond to the public source" -- it does
not, because nothing compares published bytes; it now says what exit 0 does and
does not establish, and names the gap. building-from-source.md described the
dead released-.deb fetch, and claimed source tarballs are attached as release
assets (they are not; only .debs, final.config, manifest.json and the .der are).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merge pull request 'feat(repro): content-addressed .deb hashing; per-package verdict' (#36) from feat/canonical-deb-hash into fix/in-tree-signing-key-path
All checks were successful
validate / intent-matches-policy (pull_request) Successful in 5s
validate / dep-list-parity (pull_request) Successful in 4s
validate / shellcheck (pull_request) Successful in 14s
validate / yamllint (pull_request) Successful in 12s
validate / workflow-expressions (pull_request) Successful in 4s
validate / pycompile (pull_request) Successful in 4s
validate / no-placeholder-digests (pull_request) Has been skipped
8bb62f4779
Merge pull request 'fix(build): pin the module-signing key to a fixed in-tree path' (#35) from fix/in-tree-signing-key-path into feat/manifest-schema-v2
All checks were successful
validate / shellcheck (pull_request) Successful in 13s
validate / yamllint (pull_request) Successful in 13s
validate / workflow-expressions (pull_request) Successful in 4s
validate / pycompile (pull_request) Successful in 4s
validate / intent-matches-policy (pull_request) Successful in 5s
validate / dep-list-parity (pull_request) Successful in 4s
validate / no-placeholder-digests (pull_request) Has been skipped
e30c906d01
zach merged commit 2020ae4010 into main 2026-07-25 21:47:43 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
unredacted/linux-hardened-unredacted!33
No description provided.