Reproducible Builds,
the first ten years and beyond!
Holger Levsen
MiniDebConfBerlin 2024
2024-05-19, C-Base, Berlin
Reproducible Builds,
the first ten years and beyond!
Preserving other build artifacts
Holger Levsen
MiniDebConfBerlin 2024
2024-05-19, C-Base, Berlin
Who am I
- Holger Levsen / holger@debian.org, located in Hamburg, Germany. Born at 329 ppm. He/him. 🏳️🌈🏳️⚧️🖤😷
- Debian user since 1995, contributing since 2001, Debian member since 2007. I ❤️ Debian.
- Working on Reproducible Builds since 2014.
Aiming to make all ❤️ Free Software reproducible.
- Ask me anything, anytime. This is a pretty complex topic.
About you
- Who knows about Reproducible Builds, why and how?
- Who knows something about the Debian archive kit (dak)?
- Who knows something about Debian buildd infrastucture?
- Who knows how automatic debug packages work?
- Who knows Julien Cristau and/or dpkg-distaddfile?
About this workshop
- I'll get to the topic at hand in 5 minutes.
- "how to preserve other build artifacts?
- But first, some context.
The problem
- Source code of free software available
- …most people install pre-compiled binaries
- No one really knows how they really correspond (even those building those binaries).
- As a result there are various classes of supply chain attacks.
https://reproducible-builds.org/docs/definition/
- When is a build reproducible?
- A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
- The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.
-
By 2024 Reproducible Builds has been widely understood:
-
https://reproducible-builds.org/resources/
https://reproducible-builds.org/docs/
https://reproducible-builds.org/docs/publications/
- https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
- requires "Software Bill of Material" (SBOM)s for govermental software
- so far only recommends reproducible builds / verified SBOMs
Debian unstable, 20150131
CI results for Debian trixie, 20240519
btw, kudos on t64 transition progress!
CI results for Debian trixie, 20240519
CI reproducibility of Debian amd64
| suite | reproducible | unreproducible | fails to build | other |
|---|
| stretch |
23040(93.2%) |
1514(6.1%) |
85(0.3%) |
80(0.4%) |
| buster |
26653(93.9%) |
1405(4.9%) |
232(0.8%) |
108(0.4%) |
| bullseye |
29698(96.2%) |
761(2.5%) |
274(0.9%) |
127(0.4%) |
| bookworm |
33240(96.9%) |
670(2.0%) |
260(0.8%) |
124(0.4%) |
| trixie |
33399(95.6%) |
619(1.8%) |
673(1.9%) |
135(0.4%) |
CI reproducibility of Debian amd64
| suite | reproducible | unreproducible | fails to build | other |
|---|
| stretch |
23040(93.2%) |
1514(6.1%) |
85(0.3%) |
80(0.4%) |
| buster |
26653(93.9%) |
1405(4.9%) |
232(0.8%) |
108(0.4%) |
| bullseye |
29698(96.2%) |
761(2.5%) |
274(0.9%) |
127(0.4%) |
| bookworm |
33240(96.9%) |
670(2.0%) |
260(0.8%) |
124(0.4%) |
| trixie |
33399(95.6%) |
619(1.8%) |
673(1.9%) |
135(0.4%) |
CI reproducibility of Debian amd64
| suite | reproducible | unreproducible | fails to build | other |
|---|
| stretch |
23040(93.2%) |
1514(6.1%) |
85(0.3%) |
80(0.4%) |
| buster |
26653(93.9%) |
1405(4.9%) |
232(0.8%) |
108(0.4%) |
| bullseye |
29698(96.2%) |
761(2.5%) |
274(0.9%) |
127(0.4%) |
| bookworm |
33240(96.9%) |
670(2.0%) |
260(0.8%) |
124(0.4%) |
| trixie |
33399(95.6%) |
619(1.8%) |
673(1.9%) |
135(0.4%) |
CI results for Debian unstable, 20240519
CI builds vs real world rebuilders
- CI builds help us to understand why packages are probably unreproducible.
- But we also want to compare against what we distribute on ftp.debian.org. We call those rebuilders.
rebuilders https://beta.tests.reproducible-builds.org/debian(these are not CI builds anymore)s
this is fine. 🔥
Fixing snapshot.debian.org is work in progress. ✊
- many thanks to ln5, weasel, jcristau, lucas, noah, lynxis
- several different approaches are WIP, all will take some more time.
- this is fine. 🤗
Debian testing migration
- Since the end of 2023, CI reproducible-builds results are included in the excuses output for Debian testing migration, but there is no penalty nor bonus yet.
- In 2025 for Debian 14 "forky" however there should penalties for violating:
- reproducible packages must not regress (to be allowed into
testing and therefore into stable).
- NEW packages must build reproducibly (to be allowed into
testing and therefore into stable).
- At first there could be whitelisting of some needed packages, but less over time until we can drop those exceptions.
Debian policy
- 2017: packages should build reproducibly.
- 2025? reproducible packages must not regress.
- 2025? NEW packages must build reproducibly.
- 2027? packages must build reproducibly.
100%!
- 100% reproducible is a political decision and nothing technical.
- We need to change
debian-policy!
- We can work around 'must-have-offenders' using whitelists in the beginning.
- The goal is still 100%, whitelists are just a way to achieve that goal eventually.
- Whitelists are delays but no fixes. Fixes are either fixes or removals.
Other build artifacts (and other significant problems)
- I'll ignore other significant problems today, but they exist: eg signed shim and modules, which rebuilders to trust, etc
- So let's get to those other build artifacts finally.
Build artifacts
- .deb files (and .changes and .buildinfo)
- the one and only build logfile
- nothing else, so people who need those artifacts invented unreproducible work arounds and store these build artifacts in .deb files.
https://wiki.debian.org/BuildArtifacts
- .deb files (and .changes and .buildinfo)
- the one and only build logfile
- nothing else, so people who need those artifacts invented unreproducible work arounds and store these build artifacts in .deb files.
https://wiki.debian.org/BuildArtifacts
https://wiki.debian.org/BuildArtifacts
src:binutilssrc:gcc-X- We cannot whitelist those packages forever, if we want 100% reproducible packages in Debian
testing and stable eventually!
- The wiki page has more usage examples, eg verbose build logs from GNOME packages. prepocessed sources in /tmp/cc*.out
- Many thanks to Matthias Klose for writing it. I do trust him that those artifacts are essential for maintaining those packages.
Preserving build artifacts
- dpkg-distaddfile can be used to add files to debian/files so that they are referenced in the .changes files and thus included in upload. (Thanks to Julien Cristau for pointing this out.)
- Now we only need
dak to accept those files and make them available somewhere.
- Easy.🥳🍾👍
- (Though this doesn't work for failed builds. But then we have nothing for those now neither.)
Preserving build artifacts, how to continue...
- So we only need
dak to accept those files and make them available somewhere... 💅
- Sounds like an easy plan and almost to good to be true.
- I'll send a mail to debian-devel@l.d.o with links to that wiki URL and this talk, to continue the discussion on the mailing list.
- I've also submitted another workshop on this topic for DebConf24 and both Matthias and myself will be there. AIUI this really needs the FTP team too.
-
future reproducibility of Debian amd64
| suite | reproducible | unreproducible |
|---|
| stretch |
23040(93.2%) |
1514 |
| buster |
26653(93.9%) |
1405 |
| bullseye |
29698(96.2%) |
761 |
| bookworm |
33240(96.9%) |
670 |
| trixie |
36000 |
256 |
| forky |
40000 |
77 |
| forky+1 |
45000 |
42 |
| forky+2 |
50000 |
0 |
Thank you
… and all contributors out there!
Any questions, suggestions, ...? 🤷
Holger Levsen <holger@reproducible-builds.org>
B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
