Reproducible Builds,
the first eleven years and beyond!



Holger Levsen
DebConf24
2024-07-29, Busan, South Korea

Reproducible Builds,
the first eleven years and beyond!



Holger Levsen
DebConf24
2024-07-29, Busan, South Korea

Who am I

  1. Holger Levsen / holger@debian.org, located in Hamburg, Germany. Born at 329 ppm. He/him. 🏳️‍🌈🏳️‍⚧️🖤😷
  2. Debian user since 1995, contributing since 2001, Debian member since 2007. I ❤️ Debian.
  3. Working on Reproducible Builds since 2014. Aiming to make all ❤️ Free Software reproducible.
  4. Ask me anything, anytime. This is a pretty complex topic.

people working on this - TTBOMK

akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Linus Nordberg • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo

according to https://reproducible-builds.org/who/people/

akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Linus Nordberg • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo

About you

  • Who knows about Reproducible Builds, why and how?
  • Who contribute(s|d) to Reproducible Builds?
  • Who knows that Reproducible Builds have been known for more than 10 years? >30 years?
  • Who knows about SBOM? (Software Bill of Materials) ~= our .buildinfo files from 2014!

We need you!
Please support these efforts

  • Do you think reproducible builds should happen?
    If so, please help. We need your help and support.
  • The goals of this talk it to recap what we have done and to celebrate 11 years of awesomeness of many with the aim to get you informed, excited & involved.
    Because a lot of work and support is still needed. We are still far from being done, despite all the progress and successes so far!
  • It's doable, we can do it together! 💪
    •

Introduction

The problem

  • Source code of free software available
  • …most people install pre-compiled binaries
  • No one really knows how they really correspond (even those building those binaries).
  • As a result there are various classes of supply chain attacks.

some ancient history (>10 years ago)

  • Thread on debian-devel@lists.debian.org from 2007. Deemed undoable by many.

Ancient history (>10 years ago)

  • Thread on debian-devel@lists.debian.org from 2007. Deemed undoable by many.
  • Before that a similar idea appeared in 2000 on debian-devel@l.d.o.
  • And then in 2017 we learned from John Gilmore on rb-general@lists.reproducible-builds.org that GCC was reproducible in the early 1990s on several architectures!

Fast forward to 2023

    https://lists.zx2c4.com/pipermail/wireguard/2023-April/008045.html
    Wireguard (VPN app for Android) builds are now reproducible, their release is identical on their website, Google Play Store and F-Droid. 🎯🎯🎯🥳
    (it's more complicated than that, see their mail.)

    We were not even informed. 🥲 People just do reproducible builds as normal part of their work nowadays. 🤗

People just do reproducible builds as normal part of their work nowadays.

🤗

https://reproducible-builds.org/docs/definition/

  • When is a build reproducible?
  • A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
  • The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.

Our mission

  • Enable anyone to independently verify that a given source produces bit by bit identical results.
  • Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
  • (Un)secure software build reproducibly still remains (un)secure software. However, with reproducible builds you can be sure that you are running the software you want to be running, built from the sources you want to be using.

By 2024 Reproducible Builds has been widely understood:


  • https://reproducible-builds.org/resources/
    https://reproducible-builds.org/docs/
    https://reproducible-builds.org/docs/publications/
  • https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
    • requires "Software Bill of Material" (SBOM)s for govermental software
    • so far only recommends reproducible builds / verified SBOMs

How did we get there?

  • Money
  • Edward Snowden

    • Why money?

  • Bitcoin
  • Bitcoin (the software) was made reproducible in 2011.

    • Why Snowden

  • Well...
  • Torbrowser was made reproducible in 2013 by Mike Perry.
  • That's Firefox. One of the biggest software projects in the world.

    • How did we really get there?

  • Money / Bitcoin
  • Edward Snowden / Torbrowser
  • ...and a LOT of work by MANY people over MANY years.

    • 2013 and 2014

    • Lunar hosted a brainstorming meeting at DebConf13.
    • and another one at DebConf14

    2013 and 2014

    • Lunar hosted a brainstorming meeting at DebConf13.
    • and another one at DebConf14
    • patches for dpkg: sorting fixes and .buildinfo files (SBOM!)
    • in September 2014 I started systematic builds of Debian packages, twice. First just 100 packages, then all of them.
    • Mike Perry and Seth Schoen gave a presentation at CCCongress in December 2014 showing "my" graphs. Wow.

    Debian unstable, 20150131

    2015

  • FOSDEM talk by Lunar and myself, inviting the free software world to collaborate and tackle this problem.
  • CCCamp presentation by Lunar, showing many problems and their solutions.
  • 1st Reproducible Builds Summit in Athens.
  • SOURCE_DATE_EPOCH spec
  • diffoscope

    • Common reasons for unreproducibilities:

  • timestamps, timestamps, timestamps
  • timestamps, timestamps, timestamps
  • build paths, build paths
  • all the rest

    • SOURCE_DATE_EPOCH

    • Who knows about SOURCE_DATE_EPOCH?
    • Build time stamps are largly meaningless. SOURCE_DATE_EPOCH describes the time of the last modification of the source (in seconds since the Unix epoch).
    • Supported by a lot of software today.
    • The specification is from 2015 and was updated in 2017.
    • https://reproducible-builds.org/docs/source-date-epoch/

    diffoscope

    • Who knows about diffoscope?
    • Who uses diffoscope?
    • diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them.

    diffoscope

  • Text and HTML ouput
  • File formats supported include: Android APK files, Android boot images, Android package resource table (ARSC), Apple Xcode mobile provisioning files, ar(1) archives, ASM Function, Berkeley DB database files, bzip2 archives, character/block devices, ColorSync colour profiles (.icc), Coreboot CBFS filesystem images, cpio archives, Dalvik .dex files, Debian .buildinfo files, Debian .changes files, Debian source packages (.dsc), Device Tree Compiler blob files, directories, ELF binaries, ext2/ext3/ext4/btrfs/fat filesystems, Flattened Image Tree blob files, FreeDesktop Fontconfig cache files, FreePascal files (.ppu), Gettext message catalogues, GHC Haskell .hi files, GIF image files, Git repositories, GNU R database files (.rdb), GNU R Rscript files (.rds), Gnumeric spreadsheets, GPG keybox databases, Gzipped files, Hierarchical Data Format database, HTML files (.html), ISO 9660 CD images, Java class files, Java .jmod modules, JavaScript files,

    • diffoscope

  • JPEG images, JSON files, Linux kernel images, LLVM IR bitcode files, local (UNIX domain) sockets and named pipes (FIFOs), LZ4 compressed files, lzip compressed files, macOS binaries, Microsoft Windows icon files, Microsoft Word .docx files, Mono ‘Portable Executable’ files, Mozilla-optimized .ZIP archives, Multimedia metadata, OCaml interface files, Ogg Vorbis audio files, OpenOffice .odt files, OpenSSH public keys, OpenWRT package archives (.ipk), PDF documents, PE32 files, PGP signatures, PGP signed/encrypted messages, PNG images, PostScript documents, Public Key Cryptography Standards (PKCS) files (version #7), Python pyc files, RPM archives, Rust object files (.deflate), Sphinx inventory files, SQLite databases, SquashFS filesystems, symlinks, tape archives (.tar), tcpdump capture files (.pcap), text files, TrueType font files, U-Boot legacy image files, WebAssembly binary module, XML binary schemas (.xsb), XML files, XMLB files, XZ compressed files, ZIP archives and Zstandard compressed files.
  • Fallback on hexdump comparison, fuzzy-matching to handle renamings, and much more!

    • diffoscope example output

  • Example diffoscope output for https-everywhere 5.0.6 vs 5.0.7
  • https://try.diffoscope.org
  • https://diffoscope.org

    • 3919 reprodubility related bugs fixed (mostly upstreamed), 298 patches pending...

    Resources about unreproducibilities:

    • 430 known issue types in reproducible-notes.git
    • Lunar's talk at CCCamp 2015
    • It's much easier to show common pitfalls making a package unreproducible than the opposite:
      • https://github.com/bmwiedemann/theunreproduciblepackage
      •
    • https://reproducible-builds.org/docs/

    Detour: additional benefits of reproducible builds

    • Lower development costs and increased development speed through less developer time wasted on waiting for builds.
    • Software development: does this change really have no effect / the desired effect only?
    • Licence compliance: you can only be sure a binary is Free Software if it can be (re-)build reproducibly from a given source.
    • Reproducible verified SBOMs.

    37400 bugs in 11 years ~= 9 per day

    we rebuild constantly and find lots of FTBFS bugs

    https://reproducible-builds.org

    Reproducible Builds Summits

  • 2015 Athens
  • 2016 Berlin
  • 2017 Berlin
  • 2018 Paris
  • 2019 Marrakech
  • 2022 Venice
  • 2023 Hamburg

    • Projects at Reproducible Builds Summits

    Alpine Linux, Apache Maven, Apache Security, Arch Linux, baserock, Bazel, bootstrappable.org, Buildroot, CHAINS (KTH Royal Institute of Technology), coreboot, CoyIM, Debian, Eclipse Adoptium, EdgeBSD, ElectroBSD, F-Droid, Fedora, FreeBSD, GitHub, GNU Guix, GNU Mes, Google, Guardian Project, Homebrew, Huawei, Indiana University (IU), in-toto, IPFS, JustBuild, LEAP, LEDE, LibreOffice, Linux, MacPorts, Max Planck Institute for Security and Privacy (MPI-SP), Microsoft, MirageOS, Mobian, NetBSD, New York University (NYU), NixOS, Octez / Tezos, openSUSE, OpenWrt, pantsbuild.org, phosh, pkgsrc, privoxy, Project, Pure OS, Qubes OS, Quinel Ltd, rebuilderd, Red Hat, repeatr.io, riot-os.org, Rust, Software Freedom Conservancy, spytrap-adb, subuser.org, systemd, Tails, Tor Project, Ubuntu, University of Pennsylvania (UPenn) and Warpforge.

    (There were more but we were asked to only mention these.)

    Detour: more unexpected benefits of reproducible builds

    • https://bootstrappable.org began as breakout session at the Reproducible Builds Summit 2016 in Berlin.
    • as I understand it is about bootstrapping toolchain binaries from sources only, except starting from around 500 handwritten bytes of machine code, a very simple assembler is build and then assembled some more, until it can build mes, which can build tinyCC, which can build an ancient GCC which then can build another ancient GCC, which then can be used to build modern GCC and the rest of the universe.

    Detour: more unexpected benefits of reproducible builds

    • https://bootstrappable.org began as breakout session at the Reproducible Builds Summit 2016 in Berlin.
    • Since October 2019, Guix bootstraps by using MesCC—the small C compiler that comes with Mes—to build TinyCC, which is used to build GCC 2.95.0, which then builds GCC 4.7.4. Version 4.7 is the last version of GCC to not require a C++ compiler.(quoted from bootstrappable.org)

    Reproducible Builds Summit

  • 2024
  • September 17-19 in Hamburg
  • We want you!
  • Sponsors still wanted too.

    • Reproducible-builds.org funding

    • r-b.o is a Software Freedom Conservancy (SFC) project since 2018, currently funding Chris Lambs, Mattia Rizzolo, Vagrant Cascadian and myself.
    • Funding needed to support our continous work: community work, fixing upstreams, developing software, designing processes, the yearly summit...
    • Thank you, CIP, OTF & STF & all past sponsors too ❤️

    Short summary of Reproducible Debian

    CI results for Debian unstable, 20240727

    Debian unstable, 20150131

    CI results for Debian unstable, 20240727

    CI reproducibility of Debian amd64

    suitereproducibleunreproduciblefails to buildother
    stretch 23040(93.2%) 1514(6.1%) 85(0.3%) 80(0.4%)
    buster 26653(93.9%) 1405(4.9%) 232(0.8%) 108(0.4%)
    bullseye 29698(96.2%) 761(2.5%) 274(0.9%) 127(0.4%)
    bookworm 33240(96.9%) 670(2.0%) 260(0.8%) 124(0.4%)
    trixie 34275(96.2%) 764(2.1%) 379(1.1%) 207(0.5%)

    CI reproducibility of Debian amd64

    suitereproducibleunreproduciblefails to buildother
    stretch 23040(93.2%) 1514(6.1%) 85(0.3%) 80(0.4%)
    buster 26653(93.9%) 1405(4.9%) 232(0.8%) 108(0.4%)
    bullseye 29698(96.2%) 761(2.5%) 274(0.9%) 127(0.4%)
    bookworm 33240(96.9%) 670(2.0%) 260(0.8%) 124(0.4%)
    trixie 34275(96.2%) 764(2.1%) 379(1.1%) 207(0.5%)

    CI results for Debian trixie, 20240727

    Debian testing migration

    • Since the end of 2023, CI reproducible-builds results are included in the excuses output for Debian testing migration, but there is no penalty nor bonus yet.
    • In 2025 for Debian 14 "forky" however there could penalties for violating:
      • reproducible packages must not regress (to be allowed into testing and therefore into stable).
      • NEW packages must build reproducibly (to be allowed into testing and therefore into stable).
    • At first there could be whitelisting of some needed packages, but less over time until we can drop those exceptions.

    Debian policy

    • 2017: packages should build reproducibly.
    • 2025? reproducible packages must not regress.
    • 2025? NEW packages must build reproducibly.
    • 2027? packages must build reproducibly.

    100%!

    • 100% reproducible is a political decision and nothing technical.
    • We need to change debian-policy!
    • We can work around 'must-have-offenders' using whitelists in the beginning.
    • The goal is still 100%, whitelists are just a way to achieve that goal eventually.
    • Penalizing testing migration IMO is a means to enforce debian-policy though it can be done before it's policy.

    future reproducibility of Debian amd64

    suitereproducibleunreproducible
    stretch 23040(93.2%) 1514
    buster 26653(93.9%) 1405
    bullseye 29698(96.2%) 761
    bookworm 33240(96.9%) 670
    trixie 35000 256
    forky 40000 128 (but no regressions or new pkgs)
    forky+1 45000 42 policy violations left
    forky+2 50000 0 (?!?!!! that's probably 2031)

    Debian testing migration, an even better outlook

    • 2023: CI reproducible-builds results included in excuses output for Debian testing migration, but there is no penalty nor bonus yet.
    • 2025: (for Debian 14 "forky"): testing migration penalities for reproducibility regressions or new unreproducible packages.
    • July 2024: snapshot.debian.org got fixed and thus it seems we'll finally be able to base this on rebuilder instead of CI builds.

    snapshot.debian.org got fixed!

    🥳

    huge thanks to Linux Nordberg and DSA!

    the following two slides are outdated but everybody loves comic sans so I kept them

    they also help to illustrate why/how we were stuck the last few years:

    https://beta.tests.reproducible-builds.org/

    reproducibility in theory is not enough

    • Then we need rebuilders. Remember: we only have had CI builders for the last 10 years, but what we really want is to be able to rebuild what is distributed on ftp.debian.org.
    • Thus we need a working snapshot.debian.org service.
    • Without snapshot.d.o we cannot recreate the exact same environments...
    • but snapshot was buggy: #1050815, #1031628, #1029744, #1034000, #1012559, #979115, #969603…
    • And there we had been stuck for more than five years... (as the bugs above weren't fixed until last month.)

    snapshot.debian.org got fixed!

    🥳

    huge thanks to Linux Nordberg and DSA!

    IMO this deserves a proper announcement with the technical details...!

    debootsnap and debrebuild from src:devscripts in unstable

    • wget https://buildinfos.debian.net/ftp-master.debian.org/buildinfo/2024/01/16/crun_1.13-1_amd64.buildinfo
    • debrebuild --builder=sbuild libaacs_0.11.1-3_amd64-source.buildinfo
      •
    • debootsnap and debrebuild need a working snapshot.debian.org thus this didn't really work until last month.
    • Please try it out and report bugs the BTS.

    and so now finally we can rebuild and compare with what we distribute on ftp.debian.org:

    • needs re-setup...
    • archlinux rebuilderd could also be used

    another quick side quest, an idea from the summit 2023: do we need all of snapshot.d.o?

    • 70000 binary packages in Debian $suite
    • these build-depend on only 30000 packages, so 40000 packages are never used as build-depends.
    • let's analyze all those .buildinfo files!
    • those 30000 packages are only used in 100000 variations!
    • that's less than 100 GB per arch and suite!
    • https://rebuilder-snapshot.debian.net was born.

    https://rebuilder-snapshot.debian.net

    • a cache for snapshot.debian.org, which stores only the packages used as build-depends today and makes them available via SHA256, path and an API.
    • 1 TB instead of 150 TB (for all release archs and stable+testing+unstable)
    • needs metasnap.debian.net
    • still in development
    • will be useful to take load from snapshot.d.o and for offline rebuilds

    more Debian Reproducible Builds successes

    • reproducible docker/podman images: docker.debian.net
    • reproducible live images: cdimage.debian.org
    • reproducible debian-installer (in theory, not tested atm)

    Short overview of reproducibility of various projects (AIUI)

    • This section is a bit outdated and incomplete. I'm sorry. And I'm very happy there's so much great stuff going on!
    • I'll probably skip this in the talk but will leave it here for those reading the slides.

    Short overview of reproducibility of various projects (AIUI)

    • Tails: "easy", pragmatically solved.
    • Arch Linux: has rebuilders and snapshot binary archive, though lacks further infrastructure and thus user tools like pacman-bintrans are PoCs.
      • Arch Linux is 86.4% reproducible with 1701 bad and 10849 good packages.
[core] repository is 93.3% reproducible with 17 bad and 238 good packages.
[extra] repository is 94.1% reproducible with 171 bad and 2860 good packages.
[community] repository is 83.8% reproducible with 1481 bad and 7674 good packages.
    • SuSE: active development, by one person, not enabled in official builds

    Short overview of reproducibility of various projects, continued

    • nixOS: https://reproducible.nixos.org: 1570 out of 1572 (99.87%) paths in the minimal installation image are reproducible.
    • GNU Guix: also reproducible by design (like nixOS) - guix-challenge
    • Yocto: support for reproducible images.
    • F-Droid: supports reproducible builds though no UI (manual web crawling needed) nor promises.

    Short overview of reproducibility of various projects, continued

    • Alpine: basic support.
    • ElectroBSD/FreeBSD/NetBSD/OpenBSD: basic support.
    • Fedora/Redhat/Ubuntu: not interested it seems.
      • though Fedora 38 (April 2023) enabled clamping mtimes of package files using SOURCE_DATE_EPOCH from changelog when building packages. YAY!

    Summary of various projects

      Today many projects support reproducible builds, but it's often still unclear what that means in detail, how it's enforced and how users can know and be confident.

      I call it reproducible in theory or in CI.

      This is a massive success! This was thought impossible not long ago!

    Theory vs Praxis

    • In theory, we are done. In practice, we have shown that reproducible builds can be done in theory.
    • For Debian we now need to setup rebuilders (!= CI builders) and we need to store the results somewhere and we need to define criterias how tools should treat that data, and then we need those tools...
    • And those missing 5% are also crucial however, or at least 1% of them. For Debian, 1% means 300 softwares...

    Summary, looking forward

    • Many projects support or aim for reproducible builds today. This is a huge success.
    • Next: finish those last 1-5% upstream. (And there are some dragons too, eg PGO.)
    • Next: create infrastructure, processes and tools to use those results...
    • Also crucial: project-level consensus and commitment to reproducible builds in practice.

    Thank you
    … and all contributors out there!

    Any questions? 🤷

    Holger Levsen <holger@reproducible-builds.org>
    B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

    22 unreproducible src packages out of the 1337 most popular

    rdma-core jpeg-xl graphviz qtwebengine-opensource-src pipewire tracker colord nss qtbase-opensource-src grub2 libu2f-host apg libdmapsharing libjcat bluez vlc coinor-cgl lynx underscore gegl ffmpeg bind9

    96 build-essential-depends unreproducible source packages (out of 5500)

    maven-shared-utils rdma-core rustc php8.2 jpeg-xl subversion systemtap graphviz r-base pipewire pandas camlp-streams statsmodels colord chromium qt6-declarative nss qtbase-opensource-src python3.12 python3.13 cdi-api gdb pupnp pacemaker scim gcc-13-cross qemu efl bsh cxxtest codenarc fop jsoup hevea libnative-platform-java javaparser black pstoedit lucene4.10 gmetrics emoslib combblas node-function-bind gdcm xmlbeans yarl nbsphinx groovy fltk1.3 ruby-pygments.rb libapache-poi-java ldc doxygen ghc lombok h2database freetds jsch jboss-jdeparser2 dejagnu jts python-nacl servlet-api jaxb ocaml-topkg odc qtremoteobjects-everywhere-src mpich lucene8 secilc bluez vlc valgrind linux86 golang-golang-x-net coinor-cgl parallel lynx underscore asmtools rocm-hipamd libcamera nbconvert mypy petsc node-d3 qtconnectivity-opensource-src eckit eccodes frozenlist extlib emacs ffmpeg bind9 meson-python ipyparallel