Reproducible Builds in May 2023

← View all our monthly reports


Welcome to the May 2023 report from the Reproducible Builds project

In our reports, we outline the most important things that we have been up to over the past month. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.

Holger Levsen gave a talk at the 2023 edition of the Debian Reunion Hamburg, a semi-informal meetup of Debian-related people in northern Germany. The slides are available online.

In April, Holger Levsen gave a talk at foss-north 2023 titled Reproducible Builds, the first ten years. Last month, however, Holger’s talk was covered in a round-up of the conference on the Free Software Foundation Europe (FSFE) blog.

Pronnoy Goswami, Saksham Gupta, Zhiyuan Li, Na Meng and Daphne Yao from Virginia Tech published a paper investigating the Reproducibility of NPM Packages. The abstract includes:

When using open-source NPM packages, most developers download prebuilt packages on npmjs.com instead of building those packages from available source, and implicitly trust the downloaded packages. However, it is unknown whether the blindly trusted prebuilt NPM packages are reproducible (i.e., whether there is always a verifiable path from source code to any published NPM package). […] We downloaded versions/releases of 226 most popularly used NPM packages and then built each version with the available source on GitHub. Next, we applied a differencing tool to compare the versions we built against versions downloaded from NPM, and further inspected any reported difference.

The paper reports that “among the 3,390 versions of the 226 packages, only 2,087 versions are reproducible,” and furthermore that multiple factors contribute to the non-reproducibility including “flexible versioning information in package.json file and the divergent behaviors between distinct versions of tools used in the build process.” The paper concludes with “insights for future verifiable build procedures.”

Unfortunately, a PDF is not available publically yet, but a Digital Object Identifier (DOI) is available on the paper’s IEEE page.

Elsewhere in academia, Betul Gokkaya, Leonardo Aniello and Basel Halak of the School of Electronics and Computer Science at the University of Southampton published a new paper containing a broad overview of attacks and comprehensive risk assessment for software supply chain security.

Their paper, titled Software supply chain: review of attacks, risk assessment strategies and security controls, analyses the most common software supply-chain attacks by providing the latest trend of analyzed attack, and identifies the security risks for open-source and third-party software supply chains. Furthermore, their study “introduces unique security controls to mitigate analyzed cyber-attacks and risks by linking them with real-life security incidence and attacks”. (arXiv.org, PDF)

NixOS is now tracking two new reports at reproducible.nixos.org. Aside from the collection of build-time dependencies of the minimal and Gnome installation ISOs, this page now also contains reports that are restricted to the artifacts that make it into the image. The minimal ISO is currently reproducible except for Python 3.10, which hopefully will be resolved with the coming update to Python version 3.11.

On our rb-general mailing list this month:

David A. Wheeler started a thread noting that the OSSGadget project’s oss-reproducible tool was measuring something related to but not the same as reproducible builds. Initially they had adopted the term “semantically reproducible build” term for what it measured, which they defined as being “if its build results can be either recreated exactly (a bit for bit reproducible build), or if the differences between the release package and a rebuilt package are not expected to produce functional differences in normal cases.” This generated a significant number of replies, and several were concerned that people might confuse what they were measuring with “reproducible builds”. After discussion, the OSSGadget developers decided to switch to the term “semantically equivalent” for what they measured in order to reduce the risk of confusion.

Vagrant Cascadian (vagrantc) posted an update about GCC, binutils, and Debian’s build-essential set with “some progress, some hope, and I daresay, some fears…”.

Lastly, kpcyrd asked a question about building a reproducible Linux kernel package for Arch Linux (answered by Arnout Engelen). In the same, thread David A. Wheeler pointed out that the Linux Kernel documentation has a chapter about Reproducible kernel builds now as well.

In Debian this month, nine reviews of Debian packages were added, 20 were updated and 6 were removed this month, all adding to our knowledge about identified issues. In addition, Vagrant Cascadian added a link to the source code causing various ecbuild issues. […]

The F-Droid project updated its Inclusion How-To with a new section explaining why it considers reproducible builds to be best practice and hopes developers will support the team’s efforts to make as many (new) apps reproducible as it reasonably can.

In diffoscope development this month, version 242 was uploaded to Debian unstable by Chris Lamb who also made the following changes:

  • If binwalk is not available, ensure the user knows they may be missing more info. […]
  • Factor out generating a human-readable comment when missing a Python module. […]

In addition, Mattia Rizzolo documented how to (re)-produce a binary blob in the code […] and Vagrant Cascadian updated the version of diffoscope in GNU Guix to 242 […].

reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, Holger Levsen uploaded versions 0.7.24 and 0.7.25 to Debian unstable which added support for Tox versions 3 and 4 with help from Vagrant Cascadian […][…][…]

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

In addition, Jason A. Donenfeld filed a bug (now fixed in the latest alpha version) in the Android issue tracker to report that generateLocaleConfig in Android Gradle Plugin version 8.1.0 generates XML files using non-deterministic ordering, breaking reproducible builds. […]

Testing framework

The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In May, a number of changes were made by Holger Levsen:

  • Update the kernel configuration of arm64 nodes only put required modules in the initrd to save space in the /boot partition. […]
  • A huge number of changes to a new tool to document/track Jenkins node maintenance, including adding --fetch, --help, --no-future and --verbose options […][…][…][…] as well as adding a suite of new actions, such as apt-upgrade, command, deploy-git, rmstamp, etc. […][…][…][…] in addition a significant amount of refactoring […][…][…][…].
  • Issue warnings if apt has updates to install. […]
  • Allow Jenkins to run apt get update in maintenance job. […]
  • Installed bind9-dnsutils on some Ubuntu 18.04 nodes. […][…]
  • Fixed the Jenkins shell monitor to correctly deal with little-used directories. […]
  • Updated the node health check to warn when apt upgrades are available. […]
  • Performed some node maintenance. […]

In addition, Vagrant Cascadian added the nocheck, nopgo and nolto when building gcc-* and binutils packages […] as well as performed some node maintenance […][…]. In addition, Roland Clobus updated the openQA configuration to specify longer timeouts and access to the developer mode […] and updated the URL used for reproducible Debian Live images […].


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports

Follow us on Twitter @ReproBuilds, Mastodon @reproducible_builds@fosstodon.org & Reddit and please consider making a donation. • Content licensed under CC BY-SA 4.0, style licensed under MIT. Templates and styles based on the Tor Styleguide. Logos and trademarks belong to their respective owners. • Patches for this website welcome via our Git repository (instructions) or via our mailing list. • Full contact info