fix(build): pin the module-signing key to a fixed in-tree path #35

Merged

zach merged 3 commits from fix/in-tree-signing-key-path into feat/manifest-schema-v2 2026-07-25 21:47:38 +00:00

Conversation 0 Commits 3 Files changed 16 +860 -198

zach commented 2026-07-25 21:04:50 +00:00

Owner

Copy link

⚠️ This changes shipped bytes. Verify against the draft release before dispatching publish — see Verification.

Stacked on #33 (feat/manifest-schema-v2): this removes the build's dependency on the CI run number, and #33 is what records it in the manifest instead. Merge #33 first.

Every kernel we've shipped publishes a runner temp-directory name

Confirmed by reading it off a running machine, not inferred:

# zcat /proc/config.gz | grep MODULE_SIG_KEY
CONFIG_MODULE_SIG_KEY="/tmp/tmp.TliL5BYWSL/signing.pem"

configure-kernel.sh appended a per-build mktemp -d path to the tree's .config after the fidelity assertion, and build-debs.sh normalized it back to the placeholder only in the separately captured final.config. But bindeb-pkg ships the tree's .config as /boot/config-<rel>, embeds it into vmlinux via CONFIG_IKCONFIG (= /proc/config.gz), and packages it in the headers .deb. build-debs.sh:215's own comment says as much: "make clean … preserves .config. The merged .config from configure-kernel.sh is what bindeb-pkg uses."

Three consequences, ascending:

1. Information leak — the runner's temp-dir name is in every kernel. The private key itself was never shipped (cat on that path fails on a user's machine, as it must).
2. Same-tag rebuilds were never bit-identical, because mktemp differs per run — directly contradicting build-debs.sh:99-105.
3. Possibly broken DKMS, untested either way. CONFIG_MODULE_SIG_ALL=y and the headers package carries that .config, so an out-of-tree build going through the kernel's own modules_install signing path reaches for a key file that doesn't exist. docs/users/dkms.md:25 already concedes that path is unvalidated against a published release, so this is an open interaction, not a known-good one. See Verification for how to settle it.

The fix

Make CONFIG_MODULE_SIG_KEY a constant in-tree path, so the value the fidelity assertion checks is the value that ships. Nothing to rewrite, nothing to normalize, no ordering requirement against the assertion.

certs/unredacted-signing.pem, deliberately not the stock certs/signing_key.pem. Upstream certs/Makefile guards a FORCE + if_changed,gen_key rule with ifeq ($(CONFIG_MODULE_SIG_KEY),certs/signing_key.pem) — under the default name kbuild would regenerate our pre-placed key with a random one and ship a kernel trusting a key nobody holds. That name is also on MRPROPER_FILES. With a non-default name no generator rule exists, so a missing key fails loudly.

That kbuild behaviour is the load-bearing assumption here, and it can't be tested without a real build — so build-debs.sh now asserts the outcome instead of trusting it. Three hard failures, all post-make, so tripping one costs a rebuild and never a bad release:

• (a) cert identity — the DER kbuild extracted into certs/signing_key.x509, which it links into .builtin_trusted_keys (the trust root for every module signature this kernel accepts), must equal our PEM's certificate. This is what converts a wrong kbuild assumption into an aborted build rather than a silently mis-keyed kernel.
• (b) no private key in any .deb — we now place a key inside a tree bindeb-pkg packages from. mkdebian's file list shouldn't match certs/*.pem, but the cost of being wrong is publishing a signing key.
• (c) config parity — the /boot/config-* inside the image .deb must be byte-identical to the captured final.config. That divergence is this bug; now it can't recur silently.

Also: the CI run number leaves the shipped bytes

It reached KBUILD_BUILD_VERSION → UTS_VERSION inside vmlinux, while being recorded nowhere and hard-coded to 1 in repro-check.sh — three independent reasons a rebuild could never match. KBUILD_BUILD_VERSION now derives from the tag (PACKAGING_REVISION, default 1); traceability moved to manifest.ci_run_number (#33).

repro-check.sh also now: passes PACKAGING_REVISION parsed from the tag, which makes the five .P releases structurally verifiable for the first time (the rebuild used to emit ...-hardened1 .debs and report MISSING for every file — reading like tampering rather than a missing input); cross-checks the manifest against the tag it's filed under; and generates a throwaway signing key, since without one the rebuild would now fail at make.

Byte-identity on the cert-embedding artifacts (vmlinuz, System.map, -dbg) still needs the real private key — handing a verifier the published .der cannot help, because signing needs the private half — so REPRO_SIGNING_KEY_PATH is honoured for operators who hold it. Properly localizing that divergence is follow-up work on the comparison itself.

Verification

Ran the real configure-kernel.sh against a stub make + merge_config.sh harness:

case	result
production path (MODULE_SIG_KEY_PATH set)	key installed at the constant path, mode 0600, zero /tmp/ occurrences in .config, exactly one MODULE_SIG_KEY line
run twice	.config byte-identical (42b7bec5… both times) — the direct regression test for this bug
pre-existing in-tree key	accepted, not overwritten
no key at all	exit 1, naming all three legitimate cases
intent.config / .config desync	exit 2
PEM with no certificate	exit 1

Cert-identity assertion (pure openssl, so testable locally): matching cert → passes; a foreign cert → fails, which is exactly the kbuild-regenerated-key case it exists to catch; absent .x509 → the actionable-message branch, not a silent skip. Leak-guard grep verified both directions.

Tag parsing checked against all 12 release directories: correct PACKAGING_REVISION for the 5 .P releases, and the two null-version releases (v7.0.10-hardened1, …1.2) now exit 2 with an explanation instead of proceeding with "null" as a version string.

shellcheck, bash -n, py_compile, yamllint -d relaxed, intent-matches-policy all clean. Confirmed no REVISION= remains anywhere and ci_run_number recording is intact.

Not testable on this host (no dpkg-deb): the config-parity and leak-guard assertions need Linux or the pinned container. They are the two that run against real .debs, and both fail closed.

Draft gate before dispatching publish for v7.1.5

This is what makes landing it pre-tag safe — nothing is published until you dispatch publish.yml:

dpkg-deb --fsys-tarfile linux-image-*.deb | tar -xO --wildcards './boot/config-*' | grep MODULE_SIG_KEY

Expect certs/unredacted-signing.pem and no /tmp string. Then confirm the final.config asset is byte-identical to that, sbverify --cert unredacted-hardened.der the vmlinuz, and on a scratch VM boot it and modprobe a signed in-tree module to confirm MODULE_SIG_FORCE accepts it. If anything is wrong, delete the tag and re-cut.

Worth doing against v7.1.4 first: build any DKMS module on a VM with the current kernel + headers and watch for a sign-file / No such file or directory error naming the /tmp/… path. If v7.1.4 already fails that way, this PR is a user-facing bug fix and belongs in the release notes as one. If it passes, the interaction is benign and this stays hygiene + reproducibility.

🤖 Generated with Claude Code

> ⚠️ **This changes shipped bytes.** Verify against the draft release before dispatching publish — see Verification. > > **Stacked on #33** (`feat/manifest-schema-v2`): this removes the build's dependency on the CI run number, and #33 is what records it in the manifest instead. Merge #33 first. ## Every kernel we've shipped publishes a runner temp-directory name Confirmed by reading it off a running machine, not inferred: ``` # zcat /proc/config.gz | grep MODULE_SIG_KEY CONFIG_MODULE_SIG_KEY="/tmp/tmp.TliL5BYWSL/signing.pem" ``` `configure-kernel.sh` appended a per-build `mktemp -d` path to the **tree's** `.config` after the fidelity assertion, and `build-debs.sh` normalized it back to the placeholder only in the *separately captured* `final.config`. But `bindeb-pkg` ships the tree's `.config` as `/boot/config-<rel>`, embeds it into `vmlinux` via `CONFIG_IKCONFIG` (= `/proc/config.gz`), and packages it in the headers `.deb`. `build-debs.sh:215`'s own comment says as much: *"`make clean` … preserves .config. The merged .config from configure-kernel.sh is what bindeb-pkg uses."* Three consequences, ascending: 1. **Information leak** — the runner's temp-dir name is in every kernel. The private key itself was never shipped (`cat` on that path fails on a user's machine, as it must). 2. **Same-tag rebuilds were never bit-identical**, because `mktemp` differs per run — directly contradicting `build-debs.sh:99-105`. 3. **Possibly broken DKMS, untested either way.** `CONFIG_MODULE_SIG_ALL=y` and the headers package carries that `.config`, so an out-of-tree build going through the kernel's own `modules_install` signing path reaches for a key file that doesn't exist. `docs/users/dkms.md:25` already concedes that path is unvalidated against a published release, so this is an **open** interaction, not a known-good one. See Verification for how to settle it. ## The fix Make `CONFIG_MODULE_SIG_KEY` a **constant** in-tree path, so the value the fidelity assertion checks is the value that ships. Nothing to rewrite, nothing to normalize, no ordering requirement against the assertion. **`certs/unredacted-signing.pem`, deliberately not the stock `certs/signing_key.pem`.** Upstream `certs/Makefile` guards a `FORCE` + `if_changed,gen_key` rule with `ifeq ($(CONFIG_MODULE_SIG_KEY),certs/signing_key.pem)` — under the default name kbuild would **regenerate** our pre-placed key with a random one and ship a kernel trusting a key nobody holds. That name is also on `MRPROPER_FILES`. With a non-default name no generator rule exists, so a *missing* key fails loudly. **That kbuild behaviour is the load-bearing assumption here, and it can't be tested without a real build** — so `build-debs.sh` now asserts the outcome instead of trusting it. Three hard failures, all post-`make`, so tripping one costs a rebuild and never a bad release: - **(a) cert identity** — the DER kbuild extracted into `certs/signing_key.x509`, which it links into `.builtin_trusted_keys` (the trust root for every module signature this kernel accepts), must equal our PEM's certificate. This is what converts a wrong kbuild assumption into an aborted build rather than a silently mis-keyed kernel. - **(b) no private key in any `.deb`** — we now place a key inside a tree `bindeb-pkg` packages from. `mkdebian`'s file list shouldn't match `certs/*.pem`, but the cost of being wrong is publishing a signing key. - **(c) config parity** — the `/boot/config-*` inside the image `.deb` must be byte-identical to the captured `final.config`. That divergence *is* this bug; now it can't recur silently. ## Also: the CI run number leaves the shipped bytes It reached `KBUILD_BUILD_VERSION` → `UTS_VERSION` inside `vmlinux`, while being recorded nowhere and hard-coded to `1` in `repro-check.sh` — three independent reasons a rebuild could never match. `KBUILD_BUILD_VERSION` now derives from the tag (`PACKAGING_REVISION`, default 1); traceability moved to `manifest.ci_run_number` (#33). `repro-check.sh` also now: passes `PACKAGING_REVISION` parsed from the tag, which makes the **five `.P` releases structurally verifiable for the first time** (the rebuild used to emit `...-hardened1` `.debs` and report `MISSING` for every file — reading like tampering rather than a missing input); cross-checks the manifest against the tag it's filed under; and generates a throwaway signing key, since without one the rebuild would now fail at `make`. Byte-identity on the cert-embedding artifacts (`vmlinuz`, `System.map`, `-dbg`) still needs the real private key — handing a verifier the published `.der` **cannot** help, because signing needs the private half — so `REPRO_SIGNING_KEY_PATH` is honoured for operators who hold it. Properly localizing that divergence is follow-up work on the comparison itself. ## Verification Ran the real `configure-kernel.sh` against a stub `make` + `merge_config.sh` harness: | case | result | |---|---| | production path (`MODULE_SIG_KEY_PATH` set) | key installed at the constant path, mode `0600`, **zero `/tmp/` occurrences in `.config`**, exactly one `MODULE_SIG_KEY` line | | **run twice** | `.config` **byte-identical** (`42b7bec5…` both times) — the direct regression test for this bug | | pre-existing in-tree key | accepted, not overwritten | | no key at all | exit 1, naming all three legitimate cases | | `intent.config` / `.config` desync | exit 2 | | PEM with no certificate | exit 1 | Cert-identity assertion (pure `openssl`, so testable locally): matching cert → passes; a *foreign* cert → **fails**, which is exactly the kbuild-regenerated-key case it exists to catch; absent `.x509` → the actionable-message branch, not a silent skip. Leak-guard grep verified both directions. Tag parsing checked against all 12 release directories: correct `PACKAGING_REVISION` for the 5 `.P` releases, and the two `null`-version releases (`v7.0.10-hardened1`, `…1.2`) now exit 2 with an explanation instead of proceeding with `"null"` as a version string. `shellcheck`, `bash -n`, `py_compile`, `yamllint -d relaxed`, `intent-matches-policy` all clean. Confirmed no `REVISION=` remains anywhere and `ci_run_number` recording is intact. **Not testable on this host** (no `dpkg-deb`): the config-parity and leak-guard assertions need Linux or the pinned container. They are the two that run against real `.deb`s, and both fail closed. ## Draft gate before dispatching publish for v7.1.5 This is what makes landing it pre-tag safe — nothing is published until you dispatch `publish.yml`: ```bash dpkg-deb --fsys-tarfile linux-image-*.deb | tar -xO --wildcards './boot/config-*' | grep MODULE_SIG_KEY ``` Expect `certs/unredacted-signing.pem` and **no** `/tmp` string. Then confirm the `final.config` asset is byte-identical to that, `sbverify --cert unredacted-hardened.der` the vmlinuz, and on a scratch VM boot it and `modprobe` a signed in-tree module to confirm `MODULE_SIG_FORCE` accepts it. If anything is wrong, delete the tag and re-cut. **Worth doing against v7.1.4 first:** build any DKMS module on a VM with the current kernel + headers and watch for a `sign-file` / `No such file or directory` error naming the `/tmp/…` path. If v7.1.4 already fails that way, this PR is a **user-facing bug fix** and belongs in the release notes as one. If it passes, the interaction is benign and this stays hygiene + reproducibility. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

zach added 1 commit 2026-07-25 21:04:51 +00:00

fix(build): pin the module-signing key to a fixed in-tree path ... aa38d1c685

CHANGES SHIPPED BYTES. Verify against the draft release before publishing.

Every kernel this project has shipped publishes a runner temp-directory name.
Confirmed by reading it off a running machine, not inferred:

  # zcat /proc/config.gz | grep MODULE_SIG_KEY
  CONFIG_MODULE_SIG_KEY="/tmp/tmp.TliL5BYWSL/signing.pem"

Mechanism: configure-kernel.sh appended a per-build `mktemp -d` path to the
TREE's .config after the fidelity assertion, and build-debs.sh normalized that
value back to the placeholder only in the separately-captured final.config. But
bindeb-pkg ships the tree's .config as /boot/config-<rel>, embeds it into
vmlinux via CONFIG_IKCONFIG (= /proc/config.gz), and packages it in the headers
.deb. Three consequences, ascending:

  1. An information leak: the runner's temp-dir name is in every kernel. The
     private key itself was never shipped.
  2. Same-tag rebuilds were never bit-identical, because mktemp differs per run
     -- directly contradicting build-debs.sh's own comment.
  3. Possibly broken DKMS, untested either way: MODULE_SIG_ALL=y and the headers
     package carries that .config, so an out-of-tree build going through the
     kernel's own modules_install signing path reaches for a key file that does
     not exist. docs/users/dkms.md already concedes that path is unvalidated
     against a published release, so this is an open interaction. The
     Verification section below says how to settle it.

Fix: make CONFIG_MODULE_SIG_KEY a CONSTANT in-tree path, so the value the
fidelity assertion checks is the value that ships. Nothing to rewrite, nothing
to normalize, no ordering requirement against the assertion.

certs/unredacted-signing.pem, deliberately NOT the stock certs/signing_key.pem:
upstream certs/Makefile guards a FORCE + if_changed,gen_key rule with
`ifeq ($(CONFIG_MODULE_SIG_KEY),certs/signing_key.pem)`, so under the default
name kbuild would REGENERATE our pre-placed key with a random one and ship a
kernel trusting a key nobody holds. That name is also on MRPROPER_FILES. With a
non-default name no generator rule exists, so a missing key fails loudly.

That kbuild behaviour is the load-bearing assumption here and cannot be tested
without a real build, so build-debs.sh now asserts the outcome instead of
trusting it. Three new hard failures, all post-`make`, so the cost of tripping
one is a rebuild and never a bad release:

  (a) cert identity: the DER kbuild extracted into certs/signing_key.x509 --
      which it links into .builtin_trusted_keys, the trust root for every module
      signature this kernel accepts -- must equal our PEM's certificate. This is
      what turns a wrong kbuild assumption into an aborted build rather than a
      silently mis-keyed kernel.
  (b) no private key in any .deb. We now put a key inside a tree bindeb-pkg
      packages from; mkdebian's file list should not match certs/*.pem, but the
      cost of being wrong is publishing a signing key, so assert it.
  (c) config parity: the /boot/config-* inside the image .deb must be
      byte-identical to the captured final.config. That divergence IS this bug,
      undetected for twelve releases; now it cannot recur silently.

Also removes the CI run number from the shipped bytes. It reached
KBUILD_BUILD_VERSION and so UTS_VERSION inside vmlinux while being recorded
nowhere and hard-coded to 1 in repro-check.sh -- three independent reasons a
rebuild could never match. KBUILD_BUILD_VERSION now derives from the tag
(PACKAGING_REVISION, default 1); traceability moved to manifest.ci_run_number.

repro-check.sh: passes PACKAGING_REVISION parsed from the tag, which makes the
five .P releases structurally verifiable for the first time (previously the
rebuild emitted ...-hardened1 .debs and every file reported MISSING, reading
like tampering rather than a missing input); cross-checks the manifest's
kernel_version/hardened_revision against the tag; and generates a throwaway
signing key, since without one the rebuild would now fail at `make` rather than
silently generating a random one. Byte-identity on the cert-embedding artifacts
(vmlinuz, System.map, -dbg) still needs the real private key -- handing over the
published .der cannot help, because signing needs the private half -- so
REPRO_SIGNING_KEY_PATH is honoured for operators who hold it. Localizing that
divergence properly is follow-up work on the comparison itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

zach referenced this pull request from a commit 2026-07-25 21:27:33 +00:00

feat(repro): content-addressed .deb hashing; per-package verdict

zach referenced this pull request 2026-07-25 21:28:08 +00:00

feat(repro): content-addressed .deb hashing; per-package verdict #36

zach added 2 commits 2026-07-25 21:47:34 +00:00

feat(repro): content-addressed .deb hashing; per-package verdict ... 62b0016e5a

repro-check could not pass for any release, for four independent reasons. PR #35
removed two (the unrecorded CI run number in UTS_VERSION, and the missing
PACKAGING_REVISION). This removes the other two.

3. The comparison hashed a REPACK.

Both sides ran strip-signatures.sh, which ends in `dpkg-deb -b` -- and that
recompresses data.tar with the local dpkg/liblzma defaults. The build side ran it
inside the pinned container; repro-check.sh ran it on the bare host. So a
verifier on any other distro got a different hash for EVERY package, including
the ~800-byte meta packages where the strip is a complete no-op, with nothing
explaining why. strip-signatures.sh's own header says "we never recompress:
xz/zstd output isn't byte-stable across tool versions" -- true for .ko, equally
true for data.tar.

New tools/canonical-deb-hash.py never repacks. Pure stdlib: parse the ar
container, decompress control/data tar, canonicalize each member's content,
digest a sorted description of the tree. Identical output on Debian, Arch or
macOS, and it drops the sbattach and diffoscope host dependencies entirely (the
PE certificate table is parsed directly, which also makes the transform
symmetric on both sides rather than relying on sbattach being a perfect inverse
of sbsign).

Deliberately excluded from the digest: tar/ar member order, compression, and
member mtimes -- all packaging metadata, and mtime clamping varies by dpkg
version, which is the false-positive class being removed. SOURCE_DATE_EPOCH is
recorded in the manifest and fed back into the rebuild, so it is checked by
other means. Included: path, type, permission bits, uid/gid, size, link target,
and sha256 of canonicalized content.

4. The verdict was one boolean per package, so "cannot check this" read as
   "this failed".

kbuild links the signing certificate's DER into .builtin_trusted_keys, so a
rebuild with a different key produces a different vmlinuz -- and, since a
different-length cert shifts what follows, a different System.map and -dbg
payload. Handing a verifier the published .der cannot fix that: sign-file needs
the private half, so a cert+foreign-key PEM aborts on X509_check_private_key.
That is inherent, so localize it instead of pretending otherwise.

Two digests per package: key_independent_sha256 excludes boot/vmlinuz-*,
boot/System.map-* and usr/lib/debug/**; canonical_sha256 covers everything. A
third party gets a HARD pass/fail on the first -- which still covers all ~5,000
modules, /boot/config-* and the modules.* indexes -- and an operator with
REPRO_SIGNING_KEY_PATH gets the second. Verdict is now a per-package table
(OK exact / OK* / FAIL / SKIP) with a summary line, and it refuses to report
success if nothing was comparable.

Also removed: the released-.deb fetch block, which was gated on a manifest field
(`canonical_path`) that build-manifest.py never wrote -- so it never ran,
$work/released/ was always empty, and repro-check.yml's "upload diff report on
failure" always uploaded nothing. A comment now records that comparing against
PUBLISHED bytes is a genuinely stronger check that is still not attempted, so
nobody reads its absence as coverage.

Manifest goes to schema_version 3. unsigned_deb_sha256 is kept as an alias of
canonical_sha256 so existing readers keep working; the two-column TSV form is
still accepted. Legacy manifests (absent schema_version, or < 3) are reported
SKIP/partial rather than FAIL: they lack fields no rebuild can reconstruct, and
a permanently red check nobody can act on is worse than an honest "not
verifiable".

Docs corrected where they overclaimed. verifying-packages.md said exit 0
"confirmed the published binaries correspond to the public source" -- it does
not, because nothing compares published bytes; it now says what exit 0 does and
does not establish, and names the gap. building-from-source.md described the
dead released-.deb fetch, and claimed source tarballs are attached as release
assets (they are not; only .debs, final.config, manifest.json and the .der are).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Merge pull request 'feat(repro): content-addressed .deb hashing; per-package verdict' (#36) from feat/canonical-deb-hash into fix/in-tree-signing-key-path 8bb62f4779

zach merged commit e30c906d01 into feat/manifest-schema-v2 2026-07-25 21:47:38 +00:00

zach referenced this pull request from a commit 2026-07-25 21:47:39 +00:00

Merge pull request 'fix(build): pin the module-signing key to a fixed in-tree path' (#35) from fix/in-tree-signing-key-path into feat/manifest-schema-v2

zach referenced this pull request 2026-07-26 00:52:05 +00:00

fix(build): config-parity assertion selected the -dbg package #40

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

unredacted/linux-hardened-unredacted!35

No description provided.