reproduce.debian.net:
Reproducing Debian in the real world



Holger Levsen
MiniDebConf Hamburg 2026
May 9th 2026

About you

  • Who knows about Reproducible Builds, why and how?

The End

Thank you!

About you

  • Who knows about Reproducible Builds, why and how?
  • Who contribute(s|d) to Reproducible Builds?
  • Who knows that Reproducible Builds have been known for more than 10 years?  >30 years?
  • Who was at the first reproducible builds BoF at DebConf13?
  • Who knows about SBOM? (Software Bill of Materials) ~= our .buildinfo files designed in 2014!

Outline of this talk

  • explain reproduce.debian.net
  • current status and outlook for forky and duke
  • (nothing outside Debian though there have been huge progresses too)

Who am I

  1. Holger Levsen / holger@debian.org, located in Hamburg, Germany. Born at 329 ppm. He/him. 🏳️‍🌈🏳️‍⚧️🖤😷
  2. Debian user since 1995, contributing since 2001, Debian member since 2007. I ❤️ Debian.
  3. Working on Reproducible Builds since 2014. Aiming to make all ❤️ Free Software reproducible.
  4. Ask me anything, anytime. This is a pretty complex topic.
  5. I'm here to present the work of many people:

according to https://reproducible-builds.org/who/people/

akira • Alexander Bedrossian • Alexander Borkowski • Alexander Couzens (lynxis) • Alexis Bienvenüe • Alex Wilson • Allan Gunn (gunner) • Amit Biswas • Anders Kaseorg • Andrew Ayer • anonmos1 • Anoop Nadig • Arnout Engelen • Asheesh Laroia • Atharva Lele • Ben Hutchings • Benjamin Hof • Bernhard M. Wiedemann • Boyuan Yang • Brett Smith • Calum McConnell • Carl Dong • Ceridwen • Chris Lamb • Chris Smith • Christoph Berg • Christopher Baines • Chris West • Cindy Kim • Clemens Lang • Clint Adams • Dafydd Harries • Daniel Edgecumbe • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David A. Wheeler • David Bremner • David del Amo • David Prévot • David Suarez • Dhiru Kholia • Dhole • Drakonis • Drew Fisher • Ed Maste • Edward Betts • Eitan Adler • Elio Qoshi • Eli Schwartz • Emanuel Bronshtein • Emmanuel Bourg • Esa Peuha • Evangelos Ribeiro Tzaras • Fabian Keil • Fabian Wolff • Felix C. Stegerman • Feng Chai • Frédéric Pierret (fepitre) • Georg Faerber • Georg Koppen • Gonzalo Bulnes Guilpain • Graham Christensen • Greg Chabala • Guillem Jover • Hannes Mehnert • Hans-Christoph Steiner • Harlan Lieberman-Berg • heinrich5991 • Helmut Grohne • Hervé Boutemy • Holger Levsen (h01ger) • HW42 • Ian Muchina • intrigeri • jajajasalu2 • Jakub Wilk • James Fenn • Jan Nieuwenhuizen • Jan-Benedict Glaw • Jarl Gullberg • Javier Jardón • Jelle van der Waa • Jelmer Vernooij • Jérémy Bobbio (lunar) • Jochen Sprickerhof • Johannes Schauer Marin Rodrigues • John Neffenger • John Scott • Joshua Lock • Joshua Watt • Juan Picca • Juri Dispan • Justin Cappos • kpcyrd • Kushal Das • Levente Polyak • Linus Nordberg • Liyun Li • Ludovic Courtès • Lukas Puehringer • Maliat Manzur • marco • Marco Villegas • MarcoFalke • Marcus Hoffmann (bubu) • Marek Marczykowski-Górecki • Maria Glukhova • Mariana Moreira • marinamoore • Martin Suszczynski • Mathieu Bridon • Mathieu Parent • Mattia Rizzolo • Michael Pöhn • Mike Perry • Morten Linderud • Muz • Mykola Nikishov • Nick Gregory • Nicolas Boulenguez • Nicolas Vigier • Niels Thykier • Niko Tyni • Oejet • Omar Navarro Leija • opi • Orhun Parmaksiz • Oskar Wirga • Paul Gevers • Paul Spooren • Paul Wise • Peter Conrad • Peter De Wachter • Peter Wu • Philip Rinn • Pol Dellaiera • Profpatsch • Rahul Bajaj • Reiner Herrmann • Richard Purdie • Robbie Harwood • Roland Clobus • Russ Cox • Santiago Torres • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Sebastian Crane • Seth Schoen • Simon Butler • Simon Josefsson • Simon Schricker • Snahil Singh • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Adger • Steven Chamberlain • Sune Vuorela • Sylvain Beucler • Thomas Vincent • Tianon Gravi • Tim Jones • Tobias Stoeckmann • Tom Fitzhenry • Ulrike Uhlig • Vagrant Cascadian • Valentin Lorentz • Valerie R Young • Vipul • Wookey • Ximin Luo

FOSDEM, 12 years ago...

FOSDEM, 11 years ago

according to https://reproducible-builds.org/who/projects/

Alpine Linux, Apache Maven, Arch Linux, Baserock, Bitcoin Core, BitShares, Buildroot, Civil Infrastructure Platform, coreboot, Debian, ElectroBSD, F-Droid, FreeBSD, Freedesktop SDK, Fedora, GNU Guix, Go, In-toto, MirageOS, Monero, NetBSD, NixOS, OpenEmbedded, openSUSE, OpenWrt, openEuler, Qubes OS, SecureDrop, Symfony, Tails, Talos Linux, TREZOR, Tor Browser, Webconverger, Yocto Project, Trisquel GNU/Linux, rattler-build, IzzyOnDroid

Introduction

The problem

  • Source code of free software available
  • …most people install pre-compiled binaries
  • No one really knows how they really correspond (even those building those binaries).
  • As a result there are various classes of supply chain attacks.

https://reproducible-builds.org/docs/definition/

  • When is a build reproducible?
  • A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
  • The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.

Our mission

  • Enable anyone to independently verify that a given source produces bit by bit identical results.
  • Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
  • (Un)secure software build reproducibly still remains (un)secure software. However, with reproducible builds you can be sure that you are running the software you want to be running, built from the sources you want to be using.

Our mission

  • Enable anyone to independently verify that a given source produces bit by bit identical results.
  • Most people will probably say: what does that even mean?
  • Our new slogan in the making:


Enabling supply chain security.

By 2026 Reproducible Builds has been widely understood:


  • https://reproducible-builds.org/resources/ (incl. these slides)
    https://reproducible-builds.org/docs/
    https://reproducible-builds.org/docs/publications/
  • https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
    • requires "Software Bill of Material" (SBOM)s for govermental software
    • so far only recommends reproducible builds / verified SBOMs

Common reasons for unreproducibilities:

  • timestamps, timestamps, timestamps
  • timestamps, timestamps, timestamps
  • build paths, build paths
  • all the rest
  • (and somewhere in there there might be backdoors...)

    • Reproducible Builds Summits

    • 2015 Athens
    • 2016/2017 Berlin
    • 2018 Paris
    • 2019 Marrakech
    • 2022 Venice
    • 2023/2024 Hamburg
    • 2025 Vienna
    • 2026 Gothenburg

    Projects at Reproducible Builds Summits

    Alpine Linux, Apache Maven, Apache Security, Arch Linux, baserock, Bazel, bootstrappable.org, Buildroot, CHAINS (KTH Royal Institute of Technology), coreboot, CoyIM, Debian, Eclipse Adoptium, EdgeBSD, ElectroBSD, F-Droid, Fedora, FreeBSD, GitHub, GNU Guix, GNU Mes, Google, Guardian Project, Homebrew, Huawei, Indiana University (IU), in-toto, IPFS, JustBuild, LEAP, LEDE, LibreOffice, Linux, MacPorts, Max Planck Institute for Security and Privacy (MPI-SP), Microsoft, MirageOS, Mobian, NetBSD, New York University (NYU), NixOS, Octez / Tezos, openSUSE, OpenWrt, pantsbuild.org, phosh, pkgsrc, privoxy, Project, Pure OS, Qubes OS, Quinel Ltd, rebuilderd, Red Hat, repeatr.io, riot-os.org, Rust, Software Freedom Conservancy, spytrap-adb, subuser.org, systemd, Tails, Tor Project, Ubuntu, University of Pennsylvania (UPenn) and Warpforge.

    (There were more but we were asked to only mention these.)

    Short summary of Reproducible Debian

    Reproducible Builds for some parts of Debian are a reality today:

    • reproducible docker/podman images: docker.debian.net
    • reproducible live images: cdimage.debian.org
    • individual packages, useful for both developers and some users
    • mmdebstrap --variant=apt trixie
    • debvm (the fix for !MR/45 got uploaded today)

    How did we get there?

    CI builders from 2015 until today and beyond

    CI results for Debian unstable, 20260508

    4347 reprodubility related bugs fixed (mostly upstreamed), 262 patches pending...

    47362 bugs in 12 years ~= 11 per day

    we rebuild constantly and find lots of FTBFS bugs

    CI builders from 2015 until today and beyond

    https://tests.reproducible-builds.org/debian

    This is the old setup, which is still useful for now.

    and now for the new stuff

    (started in November 2024)

    https://reproduce.debian.net

    • a rebuilderd instance, running since Q3 2024
    • rebuilding and comparing against what Debian distributes on ftp.debian.org.

    about rebuilderd

    • support for rebuilding Arch, Fedora, Debian and Tails
    • rebuilderd, rebuilderd-worker, rebuilderctl
    • written in Rust by kpcyrd, development started in 2019 during Marrakech summit
    • available at https://github.com/kpcyrd/rebuilderd - installation with apt, pacman -S, apk add, sudo make install, soon with dnf too
    • several instances for Arch exist (about 5), one instance for Fedora exists and so far, 5-6 for Debian

    https://reproduce.debian.net

    • Attempts to bit-for-bit identically rebuild each Debian binary package found in the distribution archive, using the .buildinfo file produced when the buildd originally built the package.
    • For each distributed package, rebuilderd calls debrebuild that calls debootsnap, mmdebstrap and finally sbuild to build that package within a user namespace.

    reproduce.debian.net vs tests.r-b.o/debian

    • The goal of reproduce.debian.net is to replicate the same build process that is used by Debian during package publication -- not to seek out additional sources of variance.
    • Variance testing, used to find factors that can prevent packages from rebuilding reproducibly, will continue at https://tests.reproducible-builds.org/debian/reproducible.html.

    https://reproduce.debian.net

    • We are very happy to use the same tool for Debian as Archlinux and Fedora. 🤗
    • ...and Tails, Qubes, FreeBSD evenutally..?!

    https://reproduce.debian.net

    another frontend is possible:

    https://reproduce.debian.net

    more help much welcome!

    please setup rebuilderd instances and add them to https://wiki.debian.org/ReproducibleBuilds/rebuilderd-instances

    Because do you really want to put all your trust in me???

    future work: distributing and comparing trust using transparency logs.

    Find out about your system today:

    • sudo apt install debian-repro-status
    • debian-repro-status
    • INFO debian-repro-status > 60/2268 packages are not reproducible.
      INFO debian-repro-status > Your system is 97.35% reproducible.

    How to reach 100% in practice

    • 100% reproducible is a political decision and nothing technical.
    • We need to change debian-policy!
    • We can work around 'must-have-offenders' using allowlists in the beginning.
    • The goal is still 100%, allowlists are just a way to achieve that goal eventually.
    • Penalizing testing migration is a means to enforce debian-policy though it can be done before it's policy.

    Debian policy

    • 2017: packages should build reproducibly.
    • 2026? reproducible packages must not regress.
    • 2026? NEW packages must build reproducibly.
    • 2027? packages must build reproducibly.
    • In practice the release team will enforce this before it becomes policy. ☺️

    Debian's release team policy

    • 2017: packages should build reproducibly.
    • 2026 reproducible packages must not regress.
    • 2026 NEW packages must build reproducibly.
    • 2026 packages must build reproducibly.
    • In practice the release team will enforce this before it becomes official Debian policy. ☺️

    and then at the MiniDebConf Hamburg 2025

    • Paul (elbrus) and me talked about forky in future...

    ...so for forky...!

    • britney now finally has been configured to not let (most) unreproducible packages migrate to testing!

      🥳 🤗 🫠 🥳

    Testing migration is now blocked for

    • unreproducible NEW packages
    • regressions: unreproducible packages which have been reproducible before

    Testing migration is now blocked for

    • NEW unreproducible packages and regressions
    • can be unblocked on request
    • allowlisting possible, eg udebs atm
    • results on certain architectures can be ignored too

    the release team's stance is:

    • all packages must be reproducible
    • but for now, only NEW unreproducible packages and regressions are blocked
    • unreproducible packages are not yet considered RC buggy and won't be autoremoved yet

    Debian's release team policy

    • 2017: packages should build reproducibly.
    • 2026 reproducible packages must not regress.
    • 2026 NEW packages must build reproducibly.
    • 2026 packages must build reproducibly.
    • In practice the release team will enforce this before it becomes official Debian policy. ☺️

    The old path to 100% (using old CI numbers...)

    suitereproducibleunreproducible
    stretch 23040(93.2%) 1514
    buster 26653(93.9%) 1405
    bullseye 29698(96.2%) 761
    bookworm 33240(96.9%) 670
    trixie 35000 256
    forky 40000 128 (but no regressions or new pkgs)
    duke 45000 42 policy violations left
    duke+1 50000 0 (?!?!!! that's probably 2031)

    The old path to 100% (using old CI numbers...)

    is boring and has been obsoleted

    https://reproduce.debian.net

    • has everything (=main and non-free-firmware) since trixie, IOW, we have:
      experimental, unstable, forky, trixie, trixie-security, trixie-updates, trixie-proposed-updates, trixie-backports
    • has all archs except s390x and loong64. (We do have armel for trixie.)
    • once trixie becomes LTS, we'll add trixie LTS.

    https://reproduce.debian.net

    forky

    https://reproduce.debian.net/forky
    arch:all: 98.3% good, 419 bad sources

    https://reproduce.debian.net/forky
    amd64: 97.2% good, 512 bad sources

    https://reproduce.debian.net/forky
    arm64: 97.7% good, 408 bad sources

    https://reproduce.debian.net/forky
    armhf: 97.1% good, 507 bad sources

    https://reproduce.debian.net/forky
    i386: 97.0% good, 523 bad sources

    https://reproduce.debian.net/forky
    ppc64el: 91.1% good, 531 bad sources (*)

    https://reproduce.debian.net/forky
    riscv64: 91.6% good, 338 bad sources (*)

    https://reproduce.debian.net

    unstable

    https://reproduce.debian.net/unstable
    arch:all: 98.0% good, 515 bad sources

    https://reproduce.debian.net/unstable
    amd64: 97.6% good, 469 bad sources

    https://reproduce.debian.net

    trixie

    https://reproduce.debian.net/trixie
    arch:all: 91.7% good, 2001 bad sources

    https://reproduce.debian.net/trixie
    amd64: 95.6% good, 803 bad sources

    https://reproduce.debian.net

    trixie-security

    https://reproduce.debian.net/trixie-security
    arch:all: 82.9% good, 17 bad sources

    https://reproduce.debian.net/trixie-security
    amd64: 82.7% good, 15 bad sources

    https://reproduce.debian.net

    trixie-updates

    https://reproduce.debian.net/trixie-updates
    arch:all: 100% good, 1 good, 0 bad sources

    https://reproduce.debian.net/trixie-updates
    amd64: 100% good, 2 good, 0 bad sources

    https://reproduce.debian.net

    trixie-proposed-updates

    https://reproduce.debian.net/trixie-proposed-updates
    arch:all: 91.6% good, 9 bad sources

    https://reproduce.debian.net/trixie-proposed-updates
    amd64: 85.0% good, 19 bad sources

    https://reproduce.debian.net

    trixie-backports

    https://reproduce.debian.net/trixie-backports
    arch:all: 79.5% good, 53 bad sources

    https://reproduce.debian.net/trixie-backports
    amd64: 83.5% good, 32 bad sources

    Reasons for
    unreproducibilities in the real world

    ...
    (I'll get there in a second.)

    duke nukem?!?

    • for forky we will be rather liberal with accepting unreproducible packages.
    • but for duke... shall we get more ambitious and more aggressive?
    • we could declare zero unreproducible packages in main as our goal for duke. shall we?

    Reasons for
    unreproducibilities in the real world

    1. switch to r.d.n.
    2. switch to forky arch:all reasons
    3. switch to forky amd64 reasons

    Thank you
    … and all contributors out there!

    Any questions? 🤷

    Holger Levsen <holger@reproducible-builds.org>
    #debian-reproducible on irc.oftc.net
    #reproducible-builds on irc.oftc.net
    rb-general@lists.reproducible-builds.org