a tale of several distros joining forces
for a common goal: Reproducible Builds
Holger Levsen, Jelle van der Waa, kpcyrd
FOSDEM 2025
2025-02-02, Brussels, Belgium
Outline of this talk
- a brief history of the Reproducible Builds project
- brief status reports about several distros
About you
- Who knows about Reproducible Builds, why and how?
- Who contribute(s|d) to Reproducible Builds?
- Who knows that Reproducible Builds have been known for more than 10 years? >30 years?
- Who knows about SBOM? (Software Bill of Materials) ~= our .buildinfo files designed in 2014!
About us
About us
- kpcyrd. Security Researcher. 🦝 🏴 Reproducible Builds since 2017, Debian and Alpine Linux since 2018, Arch Linux since 2019. Creator of whatsrc.org.
- Jelle van der Waa. Arch Linux since 2012, Arch Linux Developer.
- Holger Levsen / h01ger. Debian user since 1995. Working on Reproducible Builds since 2014.
- We're aiming to make all ❤️ Free Software reproducible.
according to https://reproducible-builds.org/who/people/
akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Jochen Sprickerhof • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Linus Nordberg • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Oejet • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo
Yesterday, 11 years ago...
Yesterday, 10 years ago
Lunar passed away last November...
lunarⒶdebian.org / https://lunar.anargeek.net
according to https://reproducible-builds.org/who/projects/
Alpine Linux, Apache Maven, Arch Linux, Baserock, Bitcoin Core, BitShares, Buildroot, Civil Infrastructure Platform, coreboot, Debian, ElectroBSD, F-Droid, FreeBSD, Freedesktop SDK, Fedora, GNU Guix, Go, In-toto, MirageOS, Monero, NetBSD, NixOS, OpenEmbedded, openSUSE, OpenWrt, openEuler, Qubes OS, SecureDrop, Symfony, Tails, Talos Linux, TREZOR, Tor Browser, Webconverger, Yocto Project, Trisquel GNU/Linux, rattler-build, IzzyOnDroid
Introduction
The problem
- Source code of free software available
- …most people install pre-compiled binaries
- No one really knows how they really correspond (even those building those binaries).
- As a result there are various classes of supply chain attacks.
https://reproducible-builds.org/docs/definition/
- When is a build reproducible?
- A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
- The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.
Our mission
- Enable anyone to independently verify that a given source produces bit by bit identical results.
- Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
- (Un)secure software build reproducibly still remains (un)secure software. However, with reproducible builds you can be sure that you are running the software you want to be running, built from the sources you want to be using.
Our mission
- Enable anyone to independently verify that a given source produces bit by bit identical results.
- Most people will probably say: what does that even mean?
Our new slogan in the making...
- Enabling supply chain security.
By 2025 Reproducible Builds has been widely understood:
-
https://reproducible-builds.org/resources/ (incl. these slides)
https://reproducible-builds.org/docs/
https://reproducible-builds.org/docs/publications/ - https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
- requires "Software Bill of Material" (SBOM)s for govermental software
- so far only recommends reproducible builds / verified SBOMs
How did we get there?
Why money?
Why Snowden
How did we really get there?
2013 and 2014
- Lunar hosted a brainstorming meeting at DebConf13.
- and another one at DebConf14
2013 and 2014
- Lunar hosted a brainstorming meeting at DebConf13,
- and another one at DebConf14, and a talk at FOSDEM 14!
- Patches for
dpkg: sorting fixes and .buildinfo files (SBOM!) - In September 2014 Holger started systematic builds of Debian packages, twice. First just 100 packages, then all of them.
- Mike Perry and Seth Schoen gave a presentation at Chaos Communication Congress in December 2014 (31C3) explaining the problem space very well.
2015
- FOSDEM talk by Lunar and Holger, inviting the free software world to collaborate and tackle this problem.
- CCCamp presentation by Lunar, showing many problems and their solutions.
- 1st Reproducible Builds Summit in Athens.
- SOURCE_DATE_EPOCH spec
- debbindiff by Lunar
2015
- FOSDEM talk by Lunar and Holger, inviting the free software world to collaborate and tackle this problem.
- CCCamp presentation by Lunar, showing many problems and their solutions.
- 1st Reproducible Builds Summit in Athens.
- SOURCE_DATE_EPOCH spec
- diffoscope by Lunar and ~84 other contributors
Common reasons for unreproducibilities:
Resources about unreproducibilities:
- Lunar's talk at CCCamp 2015
- https://reproducible-builds.org/docs/ & /resources
- It's much easier to show common pitfalls making a package unreproducible than the opposite:
- https://github.com/bmwiedemann/theunreproduciblepackage
- 430 known issue types in reproducible-notes.git
SOURCE_DATE_EPOCH
- Build time stamps are largly meaningless. SOURCE_DATE_EPOCH describes the time of the last modification of the source (in seconds since the Unix epoch).
- The specification is from 2015 and was updated in 2017.
- https://reproducible-builds.org/docs/source-date-epoch/
- Supported by a lot of software today.
diffoscope
- Who uses or has used diffoscope?
- diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them.
- https://try.diffoscope.org
- https://diffoscope.org
diffoscope
diffoscope
diffoscope example output
https://reproducible-builds.org
Reproducible Builds Summits
- 2015 Athens
- 2016/2017 Berlin
- 2018 Paris
- 2019 Marrakech
- 2022 Venice
- 2023/2024 Hamburg
- 2025 Vienna?!?
Projects at Reproducible Builds Summits
Alpine Linux, Apache Maven, Apache Security, Arch Linux, baserock, Bazel, bootstrappable.org, Buildroot, CHAINS (KTH Royal Institute of Technology), coreboot, CoyIM, Debian, Eclipse Adoptium, EdgeBSD, ElectroBSD, F-Droid, Fedora, FreeBSD, GitHub, GNU Guix, GNU Mes, Google, Guardian Project, Homebrew, Huawei, Indiana University (IU), in-toto, IPFS, JustBuild, LEAP, LEDE, LibreOffice, Linux, MacPorts, Max Planck Institute for Security and Privacy (MPI-SP), Microsoft, MirageOS, Mobian, NetBSD, New York University (NYU), NixOS, Octez / Tezos, openSUSE, OpenWrt, pantsbuild.org, phosh, pkgsrc, privoxy, Project, Pure OS, Qubes OS, Quinel Ltd, rebuilderd, Red Hat, repeatr.io, riot-os.org, Rust, Software Freedom Conservancy, spytrap-adb, subuser.org, systemd, Tails, Tor Project, Ubuntu, University of Pennsylvania (UPenn) and Warpforge.
(There were more but we were asked to only mention these.)
Reproducible-builds.org funding
- r-b.o is a Software Freedom Conservancy (SFC) project since 2018, currently funding Chris Lambs, Mattia Rizzolo, Vagrant Cascadian, myself & kpcyrd.
- Funding needed to support our continous work: community work, fixing upstreams, developing software, designing processes, the yearly summit...
- Thank you, CIP, OTF & STF & all past financial sponsors and all hardware sponsors too ❤️
about rebuilderd
- support for rebuilding Arch, Debian and Tails
- rebuilderd, rebuilderd-worker, rebuilderctl
- development started in 2019 during Marrakech summit
- several instances for Arch exist (about 5)
- written in Rust by kpcyrd
- available at https://github.com/kpcyrd/rebuilderd
- installation with pacman -S, apk add, sudo make install, soon with sudo apt install (the worker already is in trixie)
about rebuilderd
{
"name": "rust-spytrap-adb",
"version": "0.3.3-2",
"distro": "debian",
"suite": "main",
"architecture": "amd64",
"input_url": "https://buildinfos.debian.net/buildinfo-pool/r/
rust-spytrap-adb/rust-spytrap-adb_0.3.3-2_amd64.buildinfo",
"artifacts": [
{
"name": "librust-spytrap-adb-dev",
"version": "0.3.3-2",
"url": "http://deb.debian.org/debian/pool/main/r/
rust-spytrap-adb/librust-spytrap-adb-dev_0.3.3-2_amd64.deb"
},
{
"name": "spytrap-adb",
"version": "0.3.3-2",
"url": "http://deb.debian.org/debian/pool/main/r/
rust-spytrap-adb/spytrap-adb_0.3.3-2_amd64.deb"
}
]
},
about some distros...
Arch Linux 2015-2025
- 2015 - pacman records BUILDINFO
- 2017 - pacman S_D_E support & archlinux-repro
- 2019 - started archiving packages required for rebuilds
- 2020 - rebuilderd instance, [core] 86%
- 2024 - reproducible minimal container userland
- 2025 - 12% left to make reproducible (4 for minimal bootable install)
https://gitlab.archlinux.org/archlinux/rebuilderd-website
https://dashboards.archlinux.org/d/PKkRg-FGz/rebuilderd
Short summary of Reproducible Debian
Reproducible Builds for some parts of Debian are a reality today:
- reproducible docker/podman images: docker.debian.net
- reproducible live images: cdimage.debian.org
- individual packages, useful for both developers and some users
CI builders from 2015 until today and beyond
CI results for Debian unstable, 20250202
4015 reprodubility related bugs fixed (mostly upstreamed), 256 patches pending...
45045 bugs in 11.5 years ~= 11 per day
we rebuild constantly and find lots of FTBFS bugs
Debian testing migration, soon we'll be getting real!
- 2023: CI reproducible-builds results included in excuses output for Debian testing migration, but there is no penalty nor bonus yet.
- July 2024:
snapshot.debian.orggot fixed and we can now do rebuilds where the build is compared against what Debian distributes on ftp.debian.org instead of CI builds. - September 2024:
debootsnapanddebrebuild(both fromdevscripts) fixed for good. - October 2024: work on https://reproduce.debian.net began.
https://reproduce.debian.net
- a
rebuilderdinstance, running since Q3 2024 - rebuilding and comparing against what Debian distributes on
ftp.debian.org.
https://amd64.reproduce.debian.net
https://i386.reproduce.debian.net
https://arm64.reproduce.debian.net
https://armhf.reproduce.debian.net
https://riscv64.reproduce.debian.net
missing
https://armel.reproduce.debian.net
https://ppc64el.reproduce.debian.net
https://mips64el.reproduce.debian.net
https://s390x.reproduce.debian.net
https://reproduce.debian.net
- We are very happy to use the same tool for Debian as Archlinux.
- However the Debian setup is still its infancy and scaling is more of an issue: 8 architectures instead of 1, 2 suites needed instead of 1 (and 3 once trixie has been released), and 3 times as many packages as Arch tests.
- https://github.com/fepitre/package-rebuilder from Frédéric Pierre also exists. We love rebuilderd, but we also love software diversity.
https://reproduce.debian.net
more help much welcome!
The diff between theory and practice?
10% at the moment!
The diff between theory and practice?
96.6% in CI
vs
86.0% on reproduce.d.n
but we've only been doing this for 3 months...
How to reach 100% in practice
- 100% reproducible is a political decision and nothing technical.
- We need to change
debian-policy! - We can work around 'must-have-offenders' using allowlists in the beginning.
- The goal is still 100%, allowlists are just a way to achieve that goal eventually.
- Penalizing testing migration is a means to enforce
debian-policythough it can be done before it's policy.
Debian policy
- 2017: packages should build reproducibly.
- 2025? reproducible packages must not regress.
- 2025? NEW packages must build reproducibly.
- 2027? packages must build reproducibly.
- In practice the release team will probaby enforce this before it becomes policy. ☺️
The path to 100%
| suite | reproducible | unreproducible |
|---|---|---|
| stretch | 23040(93.2%) | 1514 |
| buster | 26653(93.9%) | 1405 |
| bullseye | 29698(96.2%) | 761 |
| bookworm | 33240(96.9%) | 670 |
| trixie | 35000 | 256 |
| forky | 40000 | 128 (but no regressions or new pkgs) |
| forky+1 | 45000 | 42 policy violations left |
| forky+2 | 50000 | 0 (?!?!!! that's probably 2031) |
NixOS
- https://luj.fr/blog/is-nixos-truly-reproducible.html - blog post by Julien Malko, summarizing his research article https://hal.science/hal-04913007.
- The article explores the proportion of bitwise reproducible packages in the Nix package repository and its evolution between 2017 and 2023.
- "Our most important finding is that the reproducibility rate in nixpkgs has increased steadily from 69% in 2017 to about 91% in April 2023."
NixOS
- Talk yesterday in the Nix and NixOS track:
https://fosdem.org/2025/schedule/event/fosdem-2025-4430-how-reproducible-is-nixos-/
FreeBSD
- Talk at FOSDEM 2016 by Baptiste Daroussin: Reproducible builds in FreeBSD packages
- FreeBSD base system continously tested on jenkins.debian.net since 2015. Just as NetBSD is :)
FreeBSD
- Talk at FOSDEM 2016 by Baptiste Daroussin: Reproducible builds in FreeBSD packages
- FreeBSD base system continously tested on tests.reproducible-builds.org since 2015. Just as NetBSD is :)
- In 2016 there was WIP for reproducing ports and achieved 80%. And then this efford got stalled...
- until now: https://freebsdfoundation.org/blog/zero-trust-builds-for-freebsd/
FreeBSD
- the zero-trust build project is scheduled from jan-aug 2025 and centers on the freebsd build process, and in particular, release building. the primary goal of this work is to enable the entire release process to run without requiring root access, and that build artifacts build reproducibly – that is, that a third party can build bit-for-bit identical artifacts.
- [this] is one of five initiatives that together are aimed at advancing zero trust builds, software bill of materials (sbom), ci/cd automation, security controls in ports and packages, and technical debt reduction.
NetBSD
- Jan-Benedict Glaw wrote in November 2024:
- On Linux, of 82 of all tested 94 port/arch combinations built successfully, with 78 building reproducible on two consecutive builds. [...]
- Building on NetBSD current, 83 (of 94) combinations build successfully, of those 68 were reproducible. [...]
- 44 (of 94) port/arch combinations are totally reproducible, creating bit-identical output on NetBSD and Linux.
R-B-OS
- Bernhard M. Wiedemann has been working on Reproducible Builds for OpenSUSE since 2016 as a QA/research project, producing arround 2000 patches, half of them sent upstream.
- R-B-OS is a PoC built upon this work (and funded by NLNet) and is 100% reproducible! Some fixes are "not distro ready yet" though.
- minimal VM image:
https://en.opensuse.org/openSUSE:Reproducible_openSUSE/Part1 - small DVD with some graphical UI:
https://en.opensuse.org/openSUSE:Reproducible_openSUSE/Part2
Fedora
- Zbigniew Jędrzejewski-Szmek started to do rebuilds of Fedora in 2024.
- https://in.waw.pl/~zbyszek/fedora/builds-f42-after-mass-rebuild.amd64.txt
- AIUI: 5838 src rpms, of which 4799 / 82% built reproducible. (Not sure if CI or rebuilds.)
- add-determinism https://github.com/keszybz/add-determinism
- matrix channel: #reproducible-builds:fedora.im
Summary: theory vs practice
- In theory, we are done. In practice, we have shown that Reproducible Builds can be done in theory.
- Now we need to close the gap between theory and practice.
- And those missing 4-5% in CI are also crucial however, or at least 1% of them. For Debian, 1% means 370 source packages...
Summary, looking forward
- Many projects support or aim for Reproducible Builds today. This is a huge success. We wanted to change the (software) world, and we did.
- Next: finish those last 1-5% upstream. (And there are some dragons too, eg PGO.)
- Next: create rebuilderd infrastructure(s), processes, tools.
- Also crucial: project-level consensus and commitment to Reproducible Builds in practice.
Thank you
… and all contributors out there!
Holger Levsen <holger@reproducible-builds.org>
Jelle van der Waa <jelle@archlinux.org>
kpcyrd, @kpcyrd@chaos.social, github.com/kpcyrd
Thank you
… and all contributors out there!
Any questions? 🤷