feat(repro): content-addressed .deb hashing; per-package verdict #36

Merged

zach merged 1 commit from feat/canonical-deb-hash into fix/in-tree-signing-key-path 2026-07-25 21:47:34 +00:00

Conversation 0 Commits 1 Files changed 10 +557 -140

zach commented 2026-07-25 21:28:08 +00:00

Owner

Copy link

Stacked on #35 (fix/in-tree-signing-key-path), which is itself on #33. Merge those first.

repro-check could not pass for any release, for four independent reasons. #35 removed two (the unrecorded CI run number in UTS_VERSION, the missing PACKAGING_REVISION). This removes the other two.

3. The comparison hashed a repack

Both sides ran strip-signatures.sh, which ends in dpkg-deb -b — and that recompresses data.tar with the local dpkg/liblzma defaults. The build side ran it inside the pinned container; repro-check.sh ran it on the bare host. So a verifier on any other distro got a different hash for every package — including the ~800-byte meta packages where the strip is a complete no-op — with nothing explaining why.

strip-signatures.sh's own header says "we never recompress: xz/zstd output isn't byte-stable across tool versions". True for .ko; equally true for data.tar.

New tools/canonical-deb-hash.py never repacks. Pure stdlib: parse the ar container, decompress control/data tar, canonicalize each member's content, digest a sorted description of the tree. Identical output on Debian, Arch or macOS — and it drops the sbattach and diffoscope host dependencies entirely. The PE certificate table is parsed directly, which also makes the transform symmetric on both sides rather than relying on sbattach being a perfect inverse of sbsign (a concern I could not otherwise settle).

Deliberately not in the digest: ar/tar member order, compression, and member mtimes — all packaging metadata, and mtime clamping varies by dpkg version, which is exactly the false-positive class being removed. SOURCE_DATE_EPOCH is recorded in the manifest and fed back into the rebuild, so it is checked by other means. In the digest: path, type, permission bits, uid/gid, size, link target, sha256 of canonicalized content.

4. The verdict was one boolean, so "cannot check" read as "failed"

kbuild links the signing certificate's DER into .builtin_trusted_keys, so a rebuild with a different key produces a different vmlinuz — and since a different-length cert shifts what follows, a different System.map and -dbg payload. Handing a verifier the published .der cannot fix this: sign-file needs the private half, so a cert+foreign-key PEM aborts on X509_check_private_key. That's inherent, so localize it rather than pretend otherwise.

Two digests per package. key_independent_sha256 excludes boot/vmlinuz-*, boot/System.map-*, usr/lib/debug/**; canonical_sha256 covers everything. A third party gets a hard pass/fail on the first — which still covers all ~5,000 modules, /boot/config-* and the modules.* indexes — and an operator with REPRO_SIGNING_KEY_PATH gets the second. The verdict is now a per-package table (OK exact / OK* / FAIL / SKIP) with a summary, and it refuses to report success if nothing was comparable.

Removed: a fetch block that never ran

The released-.deb fetch was gated on a manifest field canonical_path that build-manifest.py never wrote — so $work/released/ was always empty, and repro-check.yml's "upload diff report on failure" always uploaded nothing (the exact "no files found" warning its comment was written to avoid). A comment now records that comparing against published bytes is a genuinely stronger check that is still not attempted, so nobody reads its absence as coverage.

Verification

The money test — three .debs with identical logical content but different compression, compressor, member order, and mtimes:

	raw sha256	canonical	key-independent
a.deb (xz -9)	e05e9ca6…	ff12e935…	ff12e935…
b.deb (xz -1, reversed order)	a1051752…	ff12e935…	ff12e935…
c.deb (gzip, different mtimes)	759f162c…	ff12e935…	ff12e935…

Raw hashes all differ; canonical hashes all match. That proves the strip-environment defect fixed without needing two distros.

Negative controls — the digest must still be sensitive to real differences:

case	result
one byte of file content changed	hashes differ ✓
identical content, rebuilt	hashes match ✓
permission bits 0644 → 0600	hashes differ ✓
module signature trailer	strip_module_signature recovers the body exactly; unsigned module untouched; a signed and unsigned .ko.xz hash the same ✓
PE: signed (cert + 8-byte padding + recomputed checksum) vs unsigned	canonicalize identically, despite differing by 305 raw bytes ✓
non-PE input	returned unchanged ✓

The key-independent split, with two packages built as if by different keys:

case	result
same modules + config, different vmlinuz/System.map/dbg	key_independent matches, canonical differs ✓
a tampered module	caught by key_independent ✓
a tampered /boot/config-*	caught by key_independent ✓

Integration: the 3-column TSV flows through build-manifest.py producing schema_version: 3 with correct repro_class per package; the legacy 2-column form still parses. dep-list-parity still passes — the review flagged that job's extraction anchors on repro-check.sh's apt block and would fold any new apt block into the extracted set; the pure-Python approach adds no packages, and the job reports 18 pkgs; repro-check omits: curl jq nodejs sbsigntool as before.

shellcheck, py_compile, yamllint -d relaxed all clean.

Docs corrected where they overclaimed

• verifying-packages.md said exit 0 "confirmed the published binaries correspond to the public source". It does not — nothing compares published bytes. It now states what exit 0 does and does not establish, and names the gap explicitly rather than leaving a stronger claim standing.
• building-from-source.md described the dead released-.deb fetch, and claimed source tarballs are attached as release assets (they are not — only .debs, final.config, manifest.json and the .der). Replaced with what the manifest actually provides for the GPLv2 §3 obligation.

🤖 Generated with Claude Code

> **Stacked on #35** (`fix/in-tree-signing-key-path`), which is itself on #33. Merge those first. `repro-check` could not pass for **any** release, for four independent reasons. #35 removed two (the unrecorded CI run number in `UTS_VERSION`, the missing `PACKAGING_REVISION`). This removes the other two. ## 3. The comparison hashed a repack Both sides ran `strip-signatures.sh`, which ends in `dpkg-deb -b` — and that **recompresses** `data.tar` with the local dpkg/liblzma defaults. The build side ran it inside the pinned container; `repro-check.sh` ran it on the bare host. So a verifier on any other distro got a different hash for **every** package — including the ~800-byte meta packages where the strip is a complete no-op — with nothing explaining why. `strip-signatures.sh`'s own header says *"we never recompress: xz/zstd output isn't byte-stable across tool versions"*. True for `.ko`; equally true for `data.tar`. New **`tools/canonical-deb-hash.py`** never repacks. Pure stdlib: parse the `ar` container, decompress `control`/`data` tar, canonicalize each member's content, digest a sorted description of the tree. Identical output on Debian, Arch or macOS — and it drops the `sbattach` and `diffoscope` host dependencies entirely. The PE certificate table is parsed directly, which also makes the transform **symmetric** on both sides rather than relying on `sbattach` being a perfect inverse of `sbsign` (a concern I could not otherwise settle). Deliberately **not** in the digest: `ar`/tar member order, compression, and member mtimes — all packaging metadata, and mtime clamping varies by dpkg version, which is exactly the false-positive class being removed. `SOURCE_DATE_EPOCH` is recorded in the manifest and fed back into the rebuild, so it is checked by other means. **In** the digest: path, type, permission bits, uid/gid, size, link target, sha256 of canonicalized content. ## 4. The verdict was one boolean, so "cannot check" read as "failed" kbuild links the signing certificate's DER into `.builtin_trusted_keys`, so a rebuild with a different key produces a different `vmlinuz` — and since a different-length cert shifts what follows, a different `System.map` and `-dbg` payload. **Handing a verifier the published `.der` cannot fix this**: `sign-file` needs the private half, so a cert+foreign-key PEM aborts on `X509_check_private_key`. That's inherent, so localize it rather than pretend otherwise. Two digests per package. `key_independent_sha256` excludes `boot/vmlinuz-*`, `boot/System.map-*`, `usr/lib/debug/**`; `canonical_sha256` covers everything. A third party gets a **hard pass/fail** on the first — which still covers all ~5,000 modules, `/boot/config-*` and the `modules.*` indexes — and an operator with `REPRO_SIGNING_KEY_PATH` gets the second. The verdict is now a per-package table (`OK exact` / `OK*` / `FAIL` / `SKIP`) with a summary, and it **refuses to report success if nothing was comparable**. ## Removed: a fetch block that never ran The released-`.deb` fetch was gated on a manifest field `canonical_path` that `build-manifest.py` **never wrote** — so `$work/released/` was always empty, and `repro-check.yml`'s "upload diff report on failure" always uploaded nothing (the exact "no files found" warning its comment was written to avoid). A comment now records that comparing against **published** bytes is a genuinely stronger check that is still *not* attempted, so nobody reads its absence as coverage. ## Verification The money test — three `.deb`s with identical logical content but different compression, compressor, member order, and mtimes: | | raw sha256 | canonical | key-independent | |---|---|---|---| | `a.deb` (xz -9) | `e05e9ca6…` | `ff12e935…` | `ff12e935…` | | `b.deb` (xz -1, reversed order) | `a1051752…` | `ff12e935…` | `ff12e935…` | | `c.deb` (gzip, different mtimes) | `759f162c…` | `ff12e935…` | `ff12e935…` | Raw hashes all differ; canonical hashes all match. That proves the strip-environment defect fixed **without needing two distros**. Negative controls — the digest must still be sensitive to real differences: | case | result | |---|---| | one byte of file content changed | hashes differ ✓ | | identical content, rebuilt | hashes match ✓ | | permission bits `0644` → `0600` | hashes differ ✓ | | module signature trailer | `strip_module_signature` recovers the body exactly; unsigned module untouched; a signed and unsigned `.ko.xz` hash the same ✓ | | PE: signed (cert + 8-byte padding + recomputed checksum) vs unsigned | canonicalize identically, despite differing by 305 raw bytes ✓ | | non-PE input | returned unchanged ✓ | The key-independent split, with two packages built as if by different keys: | case | result | |---|---| | same modules + config, different vmlinuz/System.map/dbg | `key_independent` **matches**, `canonical` **differs** ✓ | | a tampered **module** | caught by `key_independent` ✓ | | a tampered `/boot/config-*` | caught by `key_independent` ✓ | Integration: the 3-column TSV flows through `build-manifest.py` producing `schema_version: 3` with correct `repro_class` per package; the legacy 2-column form still parses. **`dep-list-parity` still passes** — the review flagged that job's extraction anchors on `repro-check.sh`'s apt block and would fold any new apt block into the extracted set; the pure-Python approach adds no packages, and the job reports `18 pkgs; repro-check omits: curl jq nodejs sbsigntool` as before. `shellcheck`, `py_compile`, `yamllint -d relaxed` all clean. ## Docs corrected where they overclaimed - `verifying-packages.md` said exit 0 *"confirmed the published binaries correspond to the public source"*. It does not — nothing compares published bytes. It now states what exit 0 does and does not establish, and names the gap explicitly rather than leaving a stronger claim standing. - `building-from-source.md` described the dead released-`.deb` fetch, and claimed source tarballs are attached as release assets (they are not — only `.debs`, `final.config`, `manifest.json` and the `.der`). Replaced with what the manifest actually provides for the GPLv2 §3 obligation. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

zach added 1 commit 2026-07-25 21:28:08 +00:00

feat(repro): content-addressed .deb hashing; per-package verdict ... 62b0016e5a

repro-check could not pass for any release, for four independent reasons. PR #35
removed two (the unrecorded CI run number in UTS_VERSION, and the missing
PACKAGING_REVISION). This removes the other two.

3. The comparison hashed a REPACK.

Both sides ran strip-signatures.sh, which ends in `dpkg-deb -b` -- and that
recompresses data.tar with the local dpkg/liblzma defaults. The build side ran it
inside the pinned container; repro-check.sh ran it on the bare host. So a
verifier on any other distro got a different hash for EVERY package, including
the ~800-byte meta packages where the strip is a complete no-op, with nothing
explaining why. strip-signatures.sh's own header says "we never recompress:
xz/zstd output isn't byte-stable across tool versions" -- true for .ko, equally
true for data.tar.

New tools/canonical-deb-hash.py never repacks. Pure stdlib: parse the ar
container, decompress control/data tar, canonicalize each member's content,
digest a sorted description of the tree. Identical output on Debian, Arch or
macOS, and it drops the sbattach and diffoscope host dependencies entirely (the
PE certificate table is parsed directly, which also makes the transform
symmetric on both sides rather than relying on sbattach being a perfect inverse
of sbsign).

Deliberately excluded from the digest: tar/ar member order, compression, and
member mtimes -- all packaging metadata, and mtime clamping varies by dpkg
version, which is the false-positive class being removed. SOURCE_DATE_EPOCH is
recorded in the manifest and fed back into the rebuild, so it is checked by
other means. Included: path, type, permission bits, uid/gid, size, link target,
and sha256 of canonicalized content.

4. The verdict was one boolean per package, so "cannot check this" read as
   "this failed".

kbuild links the signing certificate's DER into .builtin_trusted_keys, so a
rebuild with a different key produces a different vmlinuz -- and, since a
different-length cert shifts what follows, a different System.map and -dbg
payload. Handing a verifier the published .der cannot fix that: sign-file needs
the private half, so a cert+foreign-key PEM aborts on X509_check_private_key.
That is inherent, so localize it instead of pretending otherwise.

Two digests per package: key_independent_sha256 excludes boot/vmlinuz-*,
boot/System.map-* and usr/lib/debug/**; canonical_sha256 covers everything. A
third party gets a HARD pass/fail on the first -- which still covers all ~5,000
modules, /boot/config-* and the modules.* indexes -- and an operator with
REPRO_SIGNING_KEY_PATH gets the second. Verdict is now a per-package table
(OK exact / OK* / FAIL / SKIP) with a summary line, and it refuses to report
success if nothing was comparable.

Also removed: the released-.deb fetch block, which was gated on a manifest field
(`canonical_path`) that build-manifest.py never wrote -- so it never ran,
$work/released/ was always empty, and repro-check.yml's "upload diff report on
failure" always uploaded nothing. A comment now records that comparing against
PUBLISHED bytes is a genuinely stronger check that is still not attempted, so
nobody reads its absence as coverage.

Manifest goes to schema_version 3. unsigned_deb_sha256 is kept as an alias of
canonical_sha256 so existing readers keep working; the two-column TSV form is
still accepted. Legacy manifests (absent schema_version, or < 3) are reported
SKIP/partial rather than FAIL: they lack fields no rebuild can reconstruct, and
a permanently red check nobody can act on is worse than an honest "not
verifiable".

Docs corrected where they overclaimed. verifying-packages.md said exit 0
"confirmed the published binaries correspond to the public source" -- it does
not, because nothing compares published bytes; it now says what exit 0 does and
does not establish, and names the gap. building-from-source.md described the
dead released-.deb fetch, and claimed source tarballs are attached as release
assets (they are not; only .debs, final.config, manifest.json and the .der are).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

zach merged commit 8bb62f4779 into fix/in-tree-signing-key-path 2026-07-25 21:47:34 +00:00

zach referenced this pull request from a commit 2026-07-25 21:47:34 +00:00

Merge pull request 'feat(repro): content-addressed .deb hashing; per-package verdict' (#36) from feat/canonical-deb-hash into fix/in-tree-signing-key-path

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

unredacted/linux-hardened-unredacted!36

No description provided.