reproduce.debian.net:
rebuilding what is distributed from ftp.debian.org
Holger Levsen
MiniDebConf Hamburg 2025
May 3rd 2025
About you
- Who knows about Reproducible Builds, why and how?
The End
Thank you!
About you
- Who knows about Reproducible Builds, why and how?
- Who contribute(s|d) to Reproducible Builds?
- Who knows that Reproducible Builds have been known for more than 10 years? >30 years?
- Who knows about SBOM? (Software Bill of Materials) ~= our .buildinfo files designed in 2014!
Outline of this talk
- explain reproduce.debian.net
- current status and outlook
- (nothing outside Debian though there have been huge progresses too)
About me
- Holger Levsen / h01ger. Debian user since 1995. Working on Reproducible Builds since 2014.
- Aiming to make all ❤️ Free Software reproducible.
- (...)
according to https://reproducible-builds.org/who/people/
akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Jochen Sprickerhof • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Linus Nordberg • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Oejet • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo
FOSDEM, 11 years ago...
FOSDEM, 10 years ago
Lunar passed away last November...
lunarⒶdebian.org / https://lunar.anargeek.net
according to https://reproducible-builds.org/who/projects/
Alpine Linux, Apache Maven, Arch Linux, Baserock, Bitcoin Core, BitShares, Buildroot, Civil Infrastructure Platform, coreboot, Debian, ElectroBSD, F-Droid, FreeBSD, Freedesktop SDK, Fedora, GNU Guix, Go, In-toto, MirageOS, Monero, NetBSD, NixOS, OpenEmbedded, openSUSE, OpenWrt, openEuler, Qubes OS, SecureDrop, Symfony, Tails, Talos Linux, TREZOR, Tor Browser, Webconverger, Yocto Project, Trisquel GNU/Linux, rattler-build, IzzyOnDroid
Introduction
The problem
- Source code of free software available
- …most people install pre-compiled binaries
- No one really knows how they really correspond (even those building those binaries).
- As a result there are various classes of supply chain attacks.
https://reproducible-builds.org/docs/definition/
- When is a build reproducible?
- A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
- The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.
Our mission
- Enable anyone to independently verify that a given source produces bit by bit identical results.
- Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
- (Un)secure software build reproducibly still remains (un)secure software. However, with reproducible builds you can be sure that you are running the software you want to be running, built from the sources you want to be using.
Our mission
- Enable anyone to independently verify that a given source produces bit by bit identical results.
- Most people will probably say: what does that even mean?
Our new slogan in the making...
- Enabling supply chain security.
By 2025 Reproducible Builds has been widely understood:
-
https://reproducible-builds.org/resources/ (incl. these slides)
https://reproducible-builds.org/docs/
https://reproducible-builds.org/docs/publications/ - https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
- requires "Software Bill of Material" (SBOM)s for govermental software
- so far only recommends reproducible builds / verified SBOMs
Common reasons for unreproducibilities:
Reproducible Builds Summits
- 2015 Athens
- 2016/2017 Berlin
- 2018 Paris
- 2019 Marrakech
- 2022 Venice
- 2023/2024 Hamburg
- 2025 Vienna?!?
Projects at Reproducible Builds Summits
Alpine Linux, Apache Maven, Apache Security, Arch Linux, baserock, Bazel, bootstrappable.org, Buildroot, CHAINS (KTH Royal Institute of Technology), coreboot, CoyIM, Debian, Eclipse Adoptium, EdgeBSD, ElectroBSD, F-Droid, Fedora, FreeBSD, GitHub, GNU Guix, GNU Mes, Google, Guardian Project, Homebrew, Huawei, Indiana University (IU), in-toto, IPFS, JustBuild, LEAP, LEDE, LibreOffice, Linux, MacPorts, Max Planck Institute for Security and Privacy (MPI-SP), Microsoft, MirageOS, Mobian, NetBSD, New York University (NYU), NixOS, Octez / Tezos, openSUSE, OpenWrt, pantsbuild.org, phosh, pkgsrc, privoxy, Project, Pure OS, Qubes OS, Quinel Ltd, rebuilderd, Red Hat, repeatr.io, riot-os.org, Rust, Software Freedom Conservancy, spytrap-adb, subuser.org, systemd, Tails, Tor Project, Ubuntu, University of Pennsylvania (UPenn) and Warpforge.
(There were more but we were asked to only mention these.)
Short summary of Reproducible Debian
Reproducible Builds for some parts of Debian are a reality today:
- reproducible docker/podman images: docker.debian.net
- reproducible live images: cdimage.debian.org
- individual packages, useful for both developers and some users
We are sorry to interrupt this talk for an important announcement
https://osuosl.org/blog/osl-future/
(show tab)
CI builders from 2015 until today and beyond
CI results for Debian unstable, 20250202
4015 reprodubility related bugs fixed (mostly upstreamed), 256 patches pending...
45045 bugs in 11.5 years ~= 11 per day
we rebuild constantly and find lots of FTBFS bugs
https://reproduce.debian.net
- a
rebuilderdinstance, running since Q3 2024 - rebuilding and comparing against what Debian distributes on
ftp.debian.org.
about rebuilderd
- support for rebuilding Arch, Fedora, Debian and Tails
- rebuilderd, rebuilderd-worker, rebuilderctl
- development started in 2019 during Marrakech summit
- several instances for Arch exist (about 5)
- one instance for Fedora exists
- written in Rust by kpcyrd
- available at https://github.com/kpcyrd/rebuilderd
- installation with apt, pacman -S, apk add, sudo make install, soon with dnf too
https://reproduce.debian.net
- Attempts to bit-for-bit identically rebuild each Debian binary package found in the distribution archive, using the .buildinfo file produced when the buildd originally built the package.
- For each distributed package, rebuilderd calls debrebuild that calls debootsnap, mmdebstrap and finally sbuild to build that package within a user namespace.
reproduce.debian.net vs tests.r-b.o/debian
- The goal of reproduce.debian.net is to replicate the same build process that is used by Debian during package publication -- not to seek out additional sources of variance.
- Variance testing, used to find factors that can prevent packages from rebuilding reproducibly, will continue at https://tests.reproducible-builds.org/debian/reproducible.html.
https://all.reproduce.debian.net
https://amd64.reproduce.debian.net
https://arm64.reproduce.debian.net
https://armel.reproduce.debian.net
https://armhf.reproduce.debian.net
https://i386.reproduce.debian.net
https://ppc64el.reproduce.debian.net
https://riscv64.reproduce.debian.net
missing
https://s390x.reproduce.debian.net
https://mips64el.reproduce.debian.net
unstable
experimental
-security
reproduce.debian.net
is trixie only atm
https://reproduce.debian.net
- We are very happy to use the same tool for Debian as Archlinux, and recently Fedora too.
- However the Debian setup is still its infancy and scaling is more of an issue: 8 architectures instead of 1, 2 suites needed instead of 1 (and 3 once trixie has been released), and 3 times as many packages as Arch tests.
- https://github.com/fepitre/package-rebuilder from Frédéric Pierre also exists. We love rebuilderd, but we also love software diversity.
https://reproduce.debian.net
more help much welcome!
There's a workshop explaining how to setup rebuilderd instances tomorrow.
Because do you really want to put all your trust in me???
The diff between theory and practice?
We soon will be more reproducible in practice than in theory ;p
The diff between theory and practice today?
96.6% in CI
vs
96.54% on amd64.reproduce.d.n
but we've only been doing this for 8 months...
switch tab
- Show numbers on r.d.n....
- Show reasons on r.d.n....
- continue presentation
How to reach 100% in practice
- 100% reproducible is a political decision and nothing technical.
- We need to change
debian-policy! - We can work around 'must-have-offenders' using allowlists in the beginning.
- The goal is still 100%, allowlists are just a way to achieve that goal eventually.
- Penalizing testing migration is a means to enforce
debian-policythough it can be done before it's policy.
Debian policy
- 2017: packages should build reproducibly.
- 2025? reproducible packages must not regress.
- 2025? NEW packages must build reproducibly.
- 2027? packages must build reproducibly.
- In practice the release team will probaby enforce this before it becomes policy. ☺️
The path to 100%
| suite | reproducible | unreproducible |
|---|---|---|
| stretch | 23040(93.2%) | 1514 |
| buster | 26653(93.9%) | 1405 |
| bullseye | 29698(96.2%) | 761 |
| bookworm | 33240(96.9%) | 670 |
| trixie | 35000 | 256 |
| forky | 40000 | 128 (but no regressions or new pkgs) |
| forky+1 | 45000 | 42 policy violations left |
| forky+2 | 50000 | 0 (?!?!!! that's probably 2031) |
and then came the MiniDebConf Hamburg...
- ...and Paul (elbrus) and me talked about
forky.
...so for forky...
- britney might be configured to not let unreproducible packages migrate to testing.
- full stop. applause. big grin too.
- with allowlists for some unreproducible packages...
What you can do on trixie today:
sudo apt install debian-repro-statusdebian-repro-status > /dev/nullINFO debian-repro-status > 60/2268 packages are not reproducible. INFO debian-repro-status > Your system is 97.35% reproducible.- That is better in practice than in theory. :-D
Thank you
… and all contributors out there!
Holger Levsen <holger@reproducible-builds.org>
Thank you
… and all contributors out there!
Any questions? 🤷