reproduce.debian.net:
rebuilding what is distributed from ftp.debian.org

Holger Levsen
Debconf25
July 15th 2025

About you

• Who knows about Reproducible Builds, why and how?

The End

Thank you!

About you

• Who knows about Reproducible Builds, why and how?
• Who contribute(s|d) to Reproducible Builds?
• Who knows that Reproducible Builds have been known for more than 10 years? >30 years?
• Who was at the first reproducible builds BoF at DebConf13?
• Who knows about SBOM? (Software Bill of Materials) ~= our .buildinfo files designed in 2014!

Outline of this talk

• explain reproduce.debian.net
• current status and outlook
• (nothing outside Debian though there have been huge progresses too)

Who am I

1. Holger Levsen / holger@debian.org, located in Hamburg, Germany. Born at 329 ppm. He/him. 🏳️‍🌈🏳️‍⚧️🖤😷
2. Debian user since 1995, contributing since 2001, Debian member since 2007. I ❤️ Debian.
3. Working on Reproducible Builds since 2014. Aiming to make all ❤️ Free Software reproducible.
4. Ask me anything, anytime. This is a pretty complex topic.
5. I'm here to present the work of many people:

according to https://reproducible-builds.org/who/people/

akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Jarl Gullberg • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Jochen Sprickerhof • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Linus Nordberg • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Oejet • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo

Lunar passed away last November...

• Often people thank me for Reproducible Builds
  
  and I reply thats wrong, that Reproducible Builds is the work of *many* people.
• And that is true and lunar would be the first to say that and actually he *was* the first to say that!
• So now, i'd like you all to be really REALLY LOUD and cheer all our work, which we've done, because we should celebrate the our work, which is here because Lunar started this!
• Thank you.

• ...and now i'd like to ask you for a moment of silence
• so that we can think about all the great work and great people we should thank while they and us are still alive!
• YOU ROCK. Thank you all!

FOSDEM, 11 years ago...

FOSDEM, 10 years ago

according to https://reproducible-builds.org/who/projects/

Alpine Linux, Apache Maven, Arch Linux, Baserock, Bitcoin Core, BitShares, Buildroot, Civil Infrastructure Platform, coreboot, Debian, ElectroBSD, F-Droid, FreeBSD, Freedesktop SDK, Fedora, GNU Guix, Go, In-toto, MirageOS, Monero, NetBSD, NixOS, OpenEmbedded, openSUSE, OpenWrt, openEuler, Qubes OS, SecureDrop, Symfony, Tails, Talos Linux, TREZOR, Tor Browser, Webconverger, Yocto Project, Trisquel GNU/Linux, rattler-build, IzzyOnDroid

Introduction

The problem

• Source code of free software available
• …most people install pre-compiled binaries
• No one really knows how they really correspond (even those building those binaries).
• As a result there are various classes of supply chain attacks.

https://reproducible-builds.org/docs/definition/

• When is a build reproducible?
• A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
• The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.

Our mission

• Enable anyone to independently verify that a given source produces bit by bit identical results.
• Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
• (Un)secure software build reproducibly still remains (un)secure software. However, with reproducible builds you can be sure that you are running the software you want to be running, built from the sources you want to be using.

Our mission

• Enable anyone to independently verify that a given source produces bit by bit identical results.
• Most people will probably say: what does that even mean?

By 2025 Reproducible Builds has been widely understood:

• 
  https://reproducible-builds.org/resources/ (incl. these slides)
  https://reproducible-builds.org/docs/
  https://reproducible-builds.org/docs/publications/
• https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
• requires "Software Bill of Material" (SBOM)s for govermental software
• so far only recommends reproducible builds / verified SBOMs

Common reasons for unreproducibilities:

Reproducible Builds Summits

• 2015 Athens
• 2016/2017 Berlin
• 2018 Paris
• 2019 Marrakech
• 2022 Venice
• 2023/2024 Hamburg
• 2025 Vienna

Projects at Reproducible Builds Summits

Alpine Linux, Apache Maven, Apache Security, Arch Linux, baserock, Bazel, bootstrappable.org, Buildroot, CHAINS (KTH Royal Institute of Technology), coreboot, CoyIM, Debian, Eclipse Adoptium, EdgeBSD, ElectroBSD, F-Droid, Fedora, FreeBSD, GitHub, GNU Guix, GNU Mes, Google, Guardian Project, Homebrew, Huawei, Indiana University (IU), in-toto, IPFS, JustBuild, LEAP, LEDE, LibreOffice, Linux, MacPorts, Max Planck Institute for Security and Privacy (MPI-SP), Microsoft, MirageOS, Mobian, NetBSD, New York University (NYU), NixOS, Octez / Tezos, openSUSE, OpenWrt, pantsbuild.org, phosh, pkgsrc, privoxy, Project, Pure OS, Qubes OS, Quinel Ltd, rebuilderd, Red Hat, repeatr.io, riot-os.org, Rust, Software Freedom Conservancy, spytrap-adb, subuser.org, systemd, Tails, Tor Project, Ubuntu, University of Pennsylvania (UPenn) and Warpforge.

(There were more but we were asked to only mention these.)

Short summary of Reproducible Debian

Reproducible Builds for some parts of Debian are a reality today:

• reproducible docker/podman images: docker.debian.net
• reproducible live images: cdimage.debian.org
• individual packages, useful for both developers and some users
• mmdebstrap --variant=apt trixie
• debvm, though !MR/45

How did we get there?

CI builders from 2015 until today and beyond

CI results for Debian unstable, 20250712

4347 reprodubility related bugs fixed (mostly upstreamed), 262 patches pending...

47362 bugs in 12 years ~= 11 per day

we rebuild constantly and find lots of FTBFS bugs

https://reproduce.debian.net

• a rebuilderd instance, running since Q3 2024
• rebuilding and comparing against what Debian distributes on ftp.debian.org.

about rebuilderd

• support for rebuilding Arch, Fedora, Debian and Tails
• rebuilderd, rebuilderd-worker, rebuilderctl
• written in Rust by kpcyrd, development started in 2019 during Marrakech summit
• available at https://github.com/kpcyrd/rebuilderd - installation with apt, pacman -S, apk add, sudo make install, soon with dnf too
• several instances for Arch exist (about 5), one instance for Fedora exists and so far, AFAIK, three for Debian

https://reproduce.debian.net

• Attempts to bit-for-bit identically rebuild each Debian binary package found in the distribution archive, using the .buildinfo file produced when the buildd originally built the package.
• For each distributed package, rebuilderd calls debrebuild that calls debootsnap, mmdebstrap and finally sbuild to build that package within a user namespace.

reproduce.debian.net vs tests.r-b.o/debian

• The goal of reproduce.debian.net is to replicate the same build process that is used by Debian during package publication -- not to seek out additional sources of variance.
• Variance testing, used to find factors that can prevent packages from rebuilding reproducibly, will continue at https://tests.reproducible-builds.org/debian/reproducible.html.

https://all.reproduce.debian.net: 92.6%

https://amd64.reproduce.debian.net: 96.5%

https://arm64.reproduce.debian.net: 96.7%

https://armel.reproduce.debian.net: 92.0%

https://armhf.reproduce.debian.net: 94.8%

https://i386.reproduce.debian.net: 97.1%

https://ppc64el.reproduce.debian.net: 89.6%

https://riscv64.reproduce.debian.net: 94.5%

missing

https://s390x.reproduce.debian.net

unstable

experimental

-security

forky

reproduce.debian.net
is trixie only atm

https://reproduce.debian.net

• We are very happy to use the same tool for Debian as Archlinux, and recently Fedora too.
• However the Debian setup is still its infancy and scaling is more of an issue: 8 architectures instead of 1, 2 suites needed instead of 1 (and 3 once trixie has been released), and 3 times as many packages as Arch tests.
• https://github.com/fepitre/package-rebuilder from Frédéric Pierre also exists. We love rebuilderd, but we also love software diversity.

https://reproduce.debian.net

another frontend is possible:

https://reproduce.debian.net

more help much welcome!

Also please setup rebuilderd instances!

Because do you really want to put all your trust in me???

The diff between theory and practice?

We soon will be more reproducible in practice than in theory ;p

The diff between theory and practice today?

96.6% in CI

vs

96.54% on amd64.reproduce.d.n

and we've only been doing this for 10 months & the freeze...

Find out for yourself:

• sudo apt install debian-repro-status
• debian-repro-status > /dev/null
• INFO debian-repro-status > 60/2268 packages are not reproducible.
INFO debian-repro-status > Your system is 97.35% reproducible.
• That is better in practice than in theory. :-D

How to reach 100% in practice

• 100% reproducible is a political decision and nothing technical.
• We need to change debian-policy!
• We can work around 'must-have-offenders' using allowlists in the beginning.
• The goal is still 100%, allowlists are just a way to achieve that goal eventually.
• Penalizing testing migration is a means to enforce debian-policy though it can be done before it's policy.

Debian policy

• 2017: packages should build reproducibly.
• 2025? reproducible packages must not regress.
• 2025? NEW packages must build reproducibly.
• 2027? packages must build reproducibly.
• In practice the release team will probaby enforce this before it becomes policy. ☺️

The path to 100%

The path to 100% (using old CI numbers...)

and then came the MiniDebConf Hamburg...

• ...and Paul (elbrus) and me talked about forky.

...so for forky...!

• britney will be configured to not let unreproducible packages migrate to testing!

  🥳 🤗 🫠 🥳

• with allowlists for some unreproducible packages...

switch tab

• Show numbers on r.d.n....
• Show reasons on r.d.n....
• continue presentation

Thank you
… and all contributors out there!

Holger Levsen <holger@reproducible-builds.org>

Thank you
… and all contributors out there!

Any questions? 🤷

#debian-reproducible on irc.oftc.net
#reproducible-builds on irc.oftc.net
rb-general@lists.reproducible-builds.org