Reproducible Builds,
the first ten years and beyond!



Holger Levsen
FOSDEM 2024
2024-02-03, Brussels

Reproducible Builds,
the first ten years and beyond!



Holger Levsen
FOSDEM 2024
2024-02-03, Brussels

Who am I

  1. Holger Levsen / holger@debian.org, located in Hamburg, Germany. Born at 329 ppm. He/him. 🏳️‍🌈🏳️‍⚧️🖤😷
  2. Debian user since 1995, contributing since 2001, Debian member since 2007. I ❤️ Debian.
  3. FOSDEM 2005 was my first love^wFOSDEM. In 2014 we managed to do video for all the rooms for the 1st time.
  4. Working on Reproducible Builds since 2014. Aiming to make all ❤️ Free Software reproducible.
  5. Ask me anything, anytime. This is a pretty complex topic.

people working on this - TTBOMK

akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo

according to https://reproducible-builds.org/who/people/

akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo

About you

  • Who knows about Reproducible Builds, why and how?
  • Who contribute(s|d) to Reproducible Builds?
  • Who knows that Reproducible Builds have been known for more than 10 years? >30 years?
  • Who knows about SBOM?
    (Software Bill of Materials) ~= our .buildinfo files from 2014!

We need you!
Please support these efforts

  • Do you think reproducible builds should happen?
    If so, please help. We need your help and support.
  • The goals of this talk it to recap what we have done and to celebrate 10 years of awesomeness of many with the aim to get you informed, excited & involved.
    Because a lot of work and support is still needed. We are still far from being done, despite all the progress and successes so far!
  • It's doable, we can do it together! 💪
    •

Introduction

The problem

  • Source code of free software available
  • …most people install pre-compiled binaries
  • No one really knows how they really correspond (even those building those binaries).
  • As a result there are various classes of supply chain attacks.

some ancient history (>10 years ago)

  • Thread on debian-devel@lists.debian.org from 2007. Deemed undoable by many.

Ancient history (>10 years ago)

  • Thread on debian-devel@lists.debian.org from 2007. Deemed undoable by many.
  • Before that a similar idea appeared in 2000 on debian-devel@l.d.o.
  • And then in 2017 we learned from John Gilmore on rb-general@lists.reproducible-builds.org that GCC was reproducible in the early 1990s on several architectures!

Fast forward to 2023

    https://lists.zx2c4.com/pipermail/wireguard/2023-April/008045.html
    Wireguard (VPN app for Android) builds are now reproducible, their release is identical on their website, Google Play Store and F-Droid. 🎯🎯🎯🥳
    (it's more complicated than that, see their mail.)

    We were not even informed. 🥲 People just do reproducible builds as normal part of their work nowadays. 🤗

People just do reproducible builds as normal part of their work nowadays.

🤗

https://reproducible-builds.org/docs/definition/

  • When is a build reproducible?
  • A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
  • The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.

Our mission

  • Enable anyone to independently verify that a given source produces bit by bit identical results.
  • Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
  • (Un)secure software build reproducibly still remains (un)secure software. However, with reproducible builds you can be sure that you are running the software you want to be running, built from the sources you want to be using.

By 2024 Reproducible Builds has been widely understood:


  • https://reproducible-builds.org/resources/
    https://reproducible-builds.org/docs/
    https://reproducible-builds.org/docs/publications/
  • https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
    • requires "Software Bill of Material" (SBOM)s for govermental software
    • so far only recommends reproducible builds / verified SBOMs

How did we get there?

  • Money
  • Edward Snowden

    • Why money?

  • Bitcoin
  • Bitcoin (the software) was made reproducible in 2011.

    • Why Snowden

  • Well...
  • Torbrowser was made reproducible in 2013 by Mike Perry.
  • That's Firefox. One of the biggest software projects in the world.

    • How did we really get there?

  • Money / Bitcoin
  • Edward Snowden / Torbrowser
  • ...and a LOT of work by MANY people over MANY years.

    • 2013 and 2014

    • Lunar's BoF at DebConf13.
    • another BoF at DebConf14
    • patches for dpkg: sorting fixes and .buildinfo files (SBOM!)
    • in September 2014 I started systematic builds of Debian packages, twice. First just 100 packages, then all of them.
    • Mike Perry and Seth Schoen gave a presentation at CCCongress in December 2014 showing "my" graphs. Wow.

    Debian unstable, 20150131

    2015

  • FOSDEM talk by Lunar and myself, inviting the free software world to collaborate and tackle this problem.
  • CCCamp presentation by Lunar, showing many problems and their solutions.
  • 1st Reproducible Builds Summit in Athens.
  • SOURCE_DATE_EPOCH spec
  • diffoscope

    • Common reasons for unreproducibilities:

  • timestamps, timestamps, timestamps
  • timestamps, timestamps, timestamps
  • build paths, build paths
  • all the rest

    • SOURCE_DATE_EPOCH

    • Who knows about SOURCE_DATE_EPOCH?
    • Build time stamps are largly meaningless. SOURCE_DATE_EPOCH describes the time of the last modification of the source (in seconds since the Unix epoch).
    • Supported by a lot of software today.
    • The specification is from 2015 and was updated in 2017.
    • https://reproducible-builds.org/docs/source-date-epoch/

    diffoscope

    • Who knows about diffoscope?
    • Who uses diffoscope?
    • diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them.

    diffoscope

  • Text and HTML ouput
  • File formats supported include: Android APK files, Android boot images, Android package resource table (ARSC), Apple Xcode mobile provisioning files, ar(1) archives, ASM Function, Berkeley DB database files, bzip2 archives, character/block devices, ColorSync colour profiles (.icc), Coreboot CBFS filesystem images, cpio archives, Dalvik .dex files, Debian .buildinfo files, Debian .changes files, Debian source packages (.dsc), Device Tree Compiler blob files, directories, ELF binaries, ext2/ext3/ext4/btrfs/fat filesystems, Flattened Image Tree blob files, FreeDesktop Fontconfig cache files, FreePascal files (.ppu), Gettext message catalogues, GHC Haskell .hi files, GIF image files, Git repositories, GNU R database files (.rdb), GNU R Rscript files (.rds), Gnumeric spreadsheets, GPG keybox databases, Gzipped files, Hierarchical Data Format database, HTML files (.html), ISO 9660 CD images, Java class files, Java .jmod modules, JavaScript files,

    • diffoscope

  • JPEG images, JSON files, Linux kernel images, LLVM IR bitcode files, local (UNIX domain) sockets and named pipes (FIFOs), LZ4 compressed files, lzip compressed files, macOS binaries, Microsoft Windows icon files, Microsoft Word .docx files, Mono ‘Portable Executable’ files, Mozilla-optimized .ZIP archives, Multimedia metadata, OCaml interface files, Ogg Vorbis audio files, OpenOffice .odt files, OpenSSH public keys, OpenWRT package archives (.ipk), PDF documents, PE32 files, PGP signatures, PGP signed/encrypted messages, PNG images, PostScript documents, Public Key Cryptography Standards (PKCS) files (version #7), Python pyc files, RPM archives, Rust object files (.deflate), Sphinx inventory files, SQLite databases, SquashFS filesystems, symlinks, tape archives (.tar), tcpdump capture files (.pcap), text files, TrueType font files, U-Boot legacy image files, WebAssembly binary module, XML binary schemas (.xsb), XML files, XMLB files, XZ compressed files, ZIP archives and Zstandard compressed files.
  • Fallback on hexdump comparison, fuzzy-matching to handle renamings, and much more!

    • diffoscope example output

  • Example diffoscope output for https-everywhere 5.0.6 vs 5.0.7
  • https://try.diffoscope.org
  • https://diffoscope.org

    • 3852 reprodubility related bugs fixed (mostly upstreamed), 301 patches pending...

    32000 bugs in 10 years ~= 8 per day

    Resources about unreproducibilities:

    • https://reproducible-builds.org/docs/
    • 422 known issue types in reproducible-notes.git
    • Lunar's talk at CCCamp 2015
    • It's much easier to show common pitfalls making a package unreproducible than the opposite:
      • https://github.com/bmwiedemann/theunreproduciblepackage

    Detour: some unexpected benefits of reproducible builds

    • Lower development costs and increased development speed through less developer time wasted on waiting for builds.
    • Software development: does this change really have no effect / the desired effect only?
    • Licence compliance: you can only be sure a binary is Free Software if it can be (re-)build reproducibly from a given source.
    • Reproducible verified SBOMs.

    https://reproducible-builds.org

    Reproducible Builds Summits

  • 2015 Athens
  • 2016 Berlin
  • 2017 Berlin
  • 2018 Paris
  • 2019 Marrakech
  • 2022 Venice
  • 2023 Hamburg
  • 2024 ...

    • Projects at Reproducible Builds Summits

    Alpine Linux, Apache Maven, Apache Security, Arch Linux, baserock, Bazel, bootstrappable.org, Buildroot, CHAINS (KTH Royal Institute of Technology), coreboot, CoyIM, Debian, Eclipse Adoptium, EdgeBSD, ElectroBSD, F-Droid, Fedora, FreeBSD, GitHub, GNU Guix, GNU Mes, Google, Guardian Project, Homebrew, Huawei, Indiana University (IU), in-toto, IPFS, JustBuild, LEAP, LEDE, LibreOffice, Linux, MacPorts, Max Planck Institute for Security and Privacy (MPI-SP), Microsoft, MirageOS, Mobian, NetBSD, New York University (NYU), NixOS, Octez / Tezos, openSUSE, OpenWrt, pantsbuild.org, phosh, pkgsrc, privoxy, Project, Pure OS, Qubes OS, Quinel Ltd, rebuilderd, Red Hat, repeatr.io, riot-os.org, Rust, Software Freedom Conservancy, spytrap-adb, subuser.org, systemd, Tails, Tor Project, Ubuntu, University of Pennsylvania (UPenn) and Warpforge.

    (There were more but we were asked to only mention these.)

    Detour: more unexpected benefits of reproducible builds

    • https://bootstrappable.org began as breakout session at the Reproducible Builds Summit 2016 in Berlin.
    • as I understand it is about bootstrapping toolchain binaries from sources only, except starting from around 500 handwritten bytes of machine code, a very simple assembler is build and then assembled some more, until it can build mes, which can build tinyCC, which can build an ancient GCC which then can build another ancient GCC, which then can be used to build modern GCC and the rest of the universe.

    Detour: more unexpected benefits of reproducible builds

    • https://bootstrappable.org began as breakout session at the Reproducible Builds Summit 2016 in Berlin.
    • Since October 2019, Guix bootstraps by using MesCC—the small C compiler that comes with Mes—to build TinyCC, which is used to build GCC 2.95.0, which then builds GCC 4.7.4. Version 4.7 is the last version of GCC to not require a C++ compiler.(quoted from bootstrappable.org)

    Reproducible Builds Summit

  • 2024
  • where?
  • when?
  • We need a location for 50 people.
  • We need sponsors to cover the costs.
  • We need you!

    • Reproducible-builds.org funding

    • r-b.o is a Software Freedom Conservancy (SFC) project since 2018, currently funding Chris Lambs, Mattia Rizzolo, Vagrant Cascadian and myself.
    • Funding needed to support our continous work: community work, fixing upstreams, developing software, designing processes, the yearly summit...
    • Thank you! ❤️

    Short summary of Reproducible Debian

    CI results for Debian trixie, 20240201

    CI reproducibility of Debian amd64

    suitereproducibleunreproduciblefails to buildother
    stretch 23040(93.2%) 1514(6.1%) 85(0.3%) 80(0.4%)
    buster 26653(93.9%) 1405(4.9%) 232(0.8%) 108(0.4%)
    bullseye 29698(96.2%) 761(2.5%) 274(0.9%) 127(0.4%)
    bookworm 33240(96.9%) 670(2.0%) 260(0.8%) 124(0.4%)
    trixie 33399(95.6%) 619(1.8%) 673(1.9%) 135(0.4%)

    CI reproducibility of Debian amd64

    suitereproducibleunreproduciblefails to buildother
    stretch 23040(93.2%) 1514(6.1%) 85(0.3%) 80(0.4%)
    buster 26653(93.9%) 1405(4.9%) 232(0.8%) 108(0.4%)
    bullseye 29698(96.2%) 761(2.5%) 274(0.9%) 127(0.4%)
    bookworm 33240(96.9%) 670(2.0%) 260(0.8%) 124(0.4%)
    trixie 33399(95.6%) 619(1.8%) 673(1.9%) 135(0.4%)

    CI results for Debian unstable, 20240201

    Debian policy

    • 2017: packages should build reproducibly.
    • 2025? reproducible packages must not regress.
    • 2025? NEW packages must build reproducibly.
    • 2027? packages must build reproducibly.

    100%!

    • 100% reproducible is a political decision and nothing technical.
    • We need to change debian-policy!
    • We can work around 'must-have-offenders' using whitelists in the beginning.
    • The goal is still 100%, whitelists are just a way to achieve that goal eventually.

    future reproducibility of Debian amd64

    suitereproducibleunreproducible
    stretch 23040(93.2%) 1514
    buster 26653(93.9%) 1405
    bullseye 29698(96.2%) 761
    bookworm 33240(96.9%) 670
    trixie 36000 256
    forky 40000 77
    forky+1 45000 42
    forky+2 50000 0

    Debian testing migration

    • Since the end of 2023, CI reproducible-builds results are included in the excuses output for Debian testing migration, but there is no penalty nor bonus yet.
    • In 2025 for Debian 14 "forky" however there should penalties for violating:
      • reproducible packages must not regress (to be allowed into testing and therefore into stable).
      • NEW packages must build reproducibly (to be allowed into testing and therefore into stable).
    • At first there could be whitelisting of some needed packages, but less over time until we can drop those exceptions.

    rebuilders https://beta.tests.reproducible-builds.org/debian
    (these are not CI builds anymore)s

    100% reproducibility in theory is not enough, by far.

    • Then we need rebuilders.
    • Thus we need a working snapshot.debian.org service.
    • Without snapshot.d.o we cannot recreate the exact same environments...
    • but snapshot is buggy: #1050815, #1031628, #1029744, #1034000, #1012559, #979115, #969603…
    • And there we have been stuck for more than five years...

    rebuilders https://beta.tests.reproducible-builds.org/

    fixing snapshot.d.o

    • 150 TB of data for 20 years
    • 4 pushes per day, adding 70 GB each.
    • a fun project to fix, so they gave me git commit access and made me a member of the Debian snapshot LDAP group. 🙈
    • I'm honored but we need something soon.

    an idea at the summit 2023: do we need all of snapshot.d.o?

    • 70000 binary packages in Debian $suite
    • these build-depend on only 30000 packages, so 40000 packages are never used as build-depends.
    • let's analyze all those .buildinfo files!
    • those 30000 packages are only used in 100000 variations!
    • that's less than 100 GB per arch and suite!
    • https://rebuilder-snapshot.debian.net was born.

    https://rebuilder-snapshot.debian.net

    • a cache for snapshot.debian.org, which stores only the packages used as build-depends today and makes them available via SHA256, path and an API.
    • each arch takes roughly a week to seed from snapshot.d.o
    • each arch only takes hours to seed from another rebuilder-snapshot instance
    • we already run two instances and our goal is to allow many instances

    https://rebuilder-snapshot.debian.net

    • a cache in for snapshot.debian.org, which stores only the packages used as build-depends today and makes them available via SHA256, path and an API.
    • so debrebuild can use debootsnap together with metasnap to establish trust.
    • one blocking bug currently: issue #40
    • hopefully usable RSN. Many thanks to lynxis and josch!
    • also: snapshot.d.o is still awesome, despite it's flaws today. We need (to fix) it! Many thanks weasel & DSA!

    Debian 2024

    • testing migration can and will be used to enforce policy also in regards of reproducible builds (probably only enforcing for real in 2025...)
    • for a sensible setup of that, we need real rebuilders, aiming to rebuild what Debian distributes.
    • for that, we need a better working snapshot.d.o, which with rebuilder-snapshot finally is there.
    • CI builds will stay, to find issues. Rebuilders are needed to show the absence of issues.

    Short overview of reproducibility of various projects (AIUI)

    • this section is a bit outdated and incomplete...
    • I'm sorry.
    • and very happy there's so much great stuff going on!

    Short overview of reproducibility of various projects (AIUI)

    • Tails: "easy", pragmatically solved.
    • Arch Linux: has rebuilders and snapshot binary archive, though lacks further infrastructure and thus user tools like pacman-bintrans are PoCs.
      • Arch Linux is 86.4% reproducible with 1701 bad and 10849 good packages.
[core] repository is 93.3% reproducible with 17 bad and 238 good packages.
[extra] repository is 94.1% reproducible with 171 bad and 2860 good packages.
[community] repository is 83.8% reproducible with 1481 bad and 7674 good packages.
    • SuSE: active development, by one person, not enabled in official builds

    Short overview of reproducibility of various projects, continued

    • nixOS: https://reproducible.nixos.org: 1570 out of 1572 (99.87%) paths in the minimal installation image are reproducible.
    • GNU Guix: also reproducible by design (like nixOS) - guix-challenge
    • Yocto: support for reproducible images.
    • F-Droid: supports reproducible builds though no UI (manual web crawling needed) nor promises.

    Short overview of reproducibility of various projects, continued

    • Alpine: basic support.
    • ElectroBSD/FreeBSD/NetBSD/OpenBSD: basic support.
    • Fedora/Redhat/Ubuntu: not interested it seems.
      • though Fedora 38 (April 2023) enabled clamping mtimes of package files using SOURCE_DATE_EPOCH from changelog when building packages.

    Summary of various projects

      Today many projects support reproducible builds, but it's often still unclear what that means in detail, how it's enforced and how users can know and be confident.

      I call it reproducible in theory or in CI.

      This is a massive success! This was thought impossible not long ago!

    Theory vs Praxis

    • In theory, we are done. In practice, we have shown that reproducible builds can be done in theory.
    • Now we also need many rebuilders (!= CI builders) and we need to store the results somewhere and we need to define criterias how tools should treat that data, and then we need those tools...
    • And those missing 5% are also crucial however, or at least 1% of them. For Debian, 1% means 300 softwares...

    Summary, looking forward

    • Many projects support or aim for reproducible builds today. This is a huge success.
    • Next: finish those last 1-5% upstream.
    • Next: create infrastructure of rebuilders in practice.
    • Next: create infrastructure, processes and tools to securely use those results...
    • Next: project-level consensus and commitment to reproducible builds in practice.
    • Next: ... !!!

    Thank you
    … and all contributors out there!

    Any questions? 🤷

    Holger Levsen <holger@reproducible-builds.org>
    B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

    unreproducible packages out of the 1337 most popular

    bind9 bluez ffmpeg gegl graphviz grub2 guile-3.0 libdmapsharing libjcat libu2f-host libzstd linux lynx nss numpy python3.11 qtdeclarative-opensource-src qtquickcontrols2-opensource-src qtsensors-opensource-src qtspeech-opensource-src qtsvg-opensource-src qttools-opensource-src qtwayland-opensource-src qtwebchannel-opensource-src underscore vlc wireplumber

    build-essential-depends unreproducible source packages

    black bluez codenarc cxxtest dejagnu eccodes eckit efl emacs emoslib ffmpeg fish fltk1.3 freetds gdb ghc gmetrics graphviz groovy guile-3.0 h2database hevea javaparser ldc libcamera libzstd linux linux86 lombok lucene4.10 lucene8 lynx mpich mrmpi mypy nbsphinx nss numpy odc oxygen-icons5 pandas parallel pmix pstoedit python3.11 python3.12 python-django python-jsonschema qemu qt6-5compat qt6-declarative qt6-multimedia qt6-quick3d qt6-remoteobjects qtconnectivity-opensource-src qtdeclarative-opensource-src qtremoteobjects-everywhere-src qtsensors-opensource-src qtserialport-opensource-src qtspeech-opensource-src qtsvg-opensource-src qttools-opensource-src qtwayland-opensource-src qtwebchannel-opensource-src qtwebsockets-opensource-src r-base ruby-pygments.rb scikit-learn scons secilc sqlalchemy statsmodels stunnel4 sympy systemtap underscore valgrind vlc