Reproducible Builds in October 2023

← View all our monthly reports


Welcome to the October 2023 report from the Reproducible Builds project. In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.

Reproducible Builds Summit 2023

Between October 31st and November 2nd, we held our seventh Reproducible Builds Summit in Hamburg, Germany!

Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort, and this instance was no different.

During this enriching event, participants had the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. A number of concrete outcomes from the summit will documented in the report for November 2023 and elsewhere.

Amazingly the agenda and all notes from all sessions are already online.

The Reproducible Builds team would like to thank our event sponsors who include Mullvad VPN, openSUSE, Debian, Software Freedom Conservancy, Allotropia and Aspiration Tech.


Reflections on Reflections on Trusting Trust

Russ Cox posted a fascinating article on his blog prompted by the fortieth anniversary of Ken Thompson’s award-winning paper, Reflections on Trusting Trust:

[…] In March 2023, Ken gave the closing keynote [and] during the Q&A session, someone jokingly asked about the Turing award lecture, specifically “can you tell us right now whether you have a backdoor into every copy of gcc and Linux still today?”

Although Ken reveals (or at least claims!) that he has no such backdoor, he does admit that he has the actual code… which Russ requests and subsequently dissects in great but accessible detail.


Ecosystem factors of reproducible builds

Rahul Bajaj, Eduardo Fernandes, Bram Adams and Ahmed E. Hassan from the Maintenance, Construction and Intelligence of Software (MCIS) laboratory within the School of Computing, Queen’s University in Ontario, Canada have published a paper on the “Time to fix, causes and correlation with external ecosystem factors” of unreproducible builds.

The authors compare various response times within the Debian and Arch Linux distributions including, for example:

Arch Linux packages become reproducible a median of 30 days quicker when compared to Debian packages, while Debian packages remain reproducible for a median of 68 days longer once fixed.

A full PDF of their paper is available online, as are many other interesting papers on MCIS’ publication page.


NixOS installation image reproducible

On the NixOS Discourse instance, Arnout Engelen (raboof) announced that NixOS have created an independent, bit-for-bit identical rebuilding of the nixos-minimal image that is used to install NixOS. In their post, Arnout details what exactly can be reproduced, and even includes some of the history of this endeavour:

You may remember a 2021 announcement that the minimal ISO was 100% reproducible. While back then we successfully tested that all packages that were needed to build the ISO were individually reproducible, actually rebuilding the ISO still introduced differences. This was due to some remaining problems in the hydra cache and the way the ISO was created. By the time we fixed those, regressions had popped up (notably an upstream problem in Python 3.10), and it isn’t until this week that we were back to having everything reproducible and being able to validate the complete chain.

Congratulations to NixOS team for reaching this important milestone! Discussion about this announcement can be found underneath the post itself, as well as on Hacker News.


CPython source tarballs now reproducible

Seth Larson published a blog post investigating the reproducibility of the CPython source tarballs. Using diffoscope, reprotest and other tools, Seth documents his work that led to a pull request to make these files reproducible which was merged by Łukasz Langa.


New arm64 hardware from Codethink

Long-time sponsor of the project, Codethink, have generously replaced our old “Moonshot-Slides”, which they have generously hosted since 2016 with new KVM-based arm64 hardware. Holger Levsen integrated these new nodes to the Reproducible Builds’ continuous integration framework.


Community updates

On our mailing list during October 2023 there were a number of threads, including:

  • Vagrant Cascadian continued a thread about the implementation details of a “snapshot” archive server required for reproducing previous builds. […]

  • Akihiro Suda shared an update on BuildKit, a toolkit for building Docker container images. Akihiro links to a interesting talk they recently gave at DockerCon titled Reproducible builds with BuildKit for software supply-chain security.

  • Alex Zakharov started a thread discussing and proposing fixes for various tools that create ext4 filesystem images. […]

Elsewhere, Pol Dellaiera made a number of improvements to our website, including fixing typos and links […][…], adding a NixOS “Flake” file […] and sorting our publications page by date […].

Vagrant Cascadian presented Reproducible Builds All The Way Down at the Open Source Firmware Conference.


Distribution work

distro-info is a Debian-oriented tool that can provide information about Debian (and Ubuntu) distributions such as their codenames (eg. bookworm) and so on. This month, Benjamin Drung uploaded a new version of distro-info that added support for the SOURCE_DATE_EPOCH environment variable in order to close bug #1034422. In addition, 8 reviews of packages were added, 74 were updated and 56 were removed this month, all adding to our knowledge about identified issues.

Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE.


Software development

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

In addition, Chris Lamb fixed an issue in diffoscope, where if the equivalent of file -i returns text/plain, fallback to comparing as a text file. This was originally filed as Debian bug #1053668) by Niels Thykier. […] This was then uploaded to Debian (and elsewhere) as version 251.


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In October, a number of changes were made by Holger Levsen:

  • Debian-related changes:

    • Refine the handling of package blacklisting, such as sending blacklisting notifications to the #debian-reproducible-changes IRC channel. […][…][…]
    • Install systemd-oomd on all Debian bookworm nodes (re. Debian bug #1052257). […]
    • Detect more cases of failures to delete schroots. […]
    • Document various bugs in bookworm which are (currently) being manually worked around. […]

  • Node-related changes:

  • Monitoring-related changes:

    • Remove unused Munin monitoring plugins. […]
    • Complain less visibly about “too many” installed kernels. […]

  • Misc:

    • Enhance the firewall handling on Jenkins nodes. […][…][…][…]
    • Install the fish shell everywhere. […]

In addition, Vagrant Cascadian added some packages and configuration for snapshot experiments. […]


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports