Reproducible Builds in March 2025

← View all our monthly reports


Welcome to the third report in 2025 from the Reproducible Builds project. Our monthly reports outline what we’ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As usual, however, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.

Table of contents:

  1. Debian bookworm live images now fully reproducible from their binary packages
  2. “How NixOS and reproducible builds could have detected the xz backdoor”
  3. LWN: Fedora change aims for 99% package reproducibility
  4. Python adopts PEP standard for specifying package dependencies
  5. OSS Rebuild real-time validation and tooling improvements
  6. SimpleX Chat server components now reproducible
  7. Three new scholarly papers
  8. Distribution roundup
  9. An overview of “Supply Chain Attacks on Linux distributions”
  10. diffoscope & strip-nondeterminism
  11. Website updates
  12. Reproducibility testing framework
  13. Upstream patches

Debian bookworm live images now fully reproducible from their binary packages

Roland Clobus announced on our mailing list this month that all the major desktop variants (ie. Gnome, KDE, etc.) can be reproducibly created for Debian bullseye, bookworm and trixie from their (pre-compiled) binary packages.

Building reproducible Debian live images does not require building from reproducible source code, but this is still a remarkable achievement. Some large proportion of the binary packages that comprise these live images can (and were) built reproducibly, but live image generation works at a higher level. (By contrast, “full” or end-to-end reproducibility of a bootable OS image will, in time, require both the compile-the-packages the build-the-bootable-image stages to be reproducible.)

Nevertheless, in response, Roland’s announcement generated significant congratulations as well as some discussion regarding the finer points of the terms employed: a full outline of the replies can be found here.

The news was also picked up by Linux Weekly News (LWN) as well as to Hacker News.


How NixOS and reproducible builds could have detected the xz backdoor”

Julien Malka aka luj published an in-depth blog post this month with the highly-stimulating title “How NixOS and reproducible builds could have detected the xz backdoor for the benefit of all”.

Starting with an dive into the relevant technical details of the XZ Utils backdoor, Julien’s article goes on to describe how we might avoid the xz “catastrophe” in the future by building software from trusted sources and building trust into untrusted release tarballs by way of comparing sources and leveraging bitwise reproducibility, i.e. applying the practices of Reproducible Builds.

The article generated significant discussion on Hacker News as well as on Linux Weekly News (LWN).


LWN: Fedora change aims for 99% package reproducibility

Linux Weekly News (LWN) contributor Joe Brockmeier has published a detailed round-up on how Fedora change aims for 99% package reproducibility. The article opens by mentioning that although Debian has “been working toward reproducible builds for more than a decade”, the Fedora project has now:

…progressed far enough that the project is now considering a change proposal for the Fedora 43 development cycle, expected to be released in October, with a goal of making 99% of Fedora’s package builds reproducible. So far, reaction to the proposal seems favorable and focused primarily on how to achieve the goal—with minimal pain for packagers—rather than whether to attempt it.

The Change Proposal itself is worth reading:

Over the last few releases, we [Fedora] changed our build infrastructure to make package builds reproducible. This is enough to reach 90%. The remaining issues need to be fixed in individual packages. After this Change, package builds are expected to be reproducible. Bugs will be filed against packages when an irreproducibility is detected. The goal is to have no fewer than 99% of package builds reproducible.

Further discussion can be found on the Fedora mailing list as well as on Fedora’s Discourse instance.


Python adopts PEP standard for specifying package dependencies

Python developer Brett Cannon reported on Fosstodon that PEP 751 was recently accepted. This design document has the purpose of describing “a file format to record Python dependencies for installation reproducibility”. As the abstract of the proposal writes:

This PEP proposes a new file format for specifying dependencies to enable reproducible installation in a Python environment. The format is designed to be human-readable and machine-generated. Installers consuming the file should be able to calculate what to install without the need for dependency resolution at install-time.

The PEP, which itself supersedes PEP 665, mentions that “there are at least five well-known solutions to this problem in the community”.


OSS Rebuild real-time validation and tooling improvements

OSS Rebuild aims to automate rebuilding upstream language packages (e.g. from PyPI, crates.io, npm registries) and publish signed attestations and build definitions for public use.

OSS Rebuild is now attempting rebuilds as packages are published, shortening the time to validating rebuilds and publishing attestations.

Aman Sharma contributed classifiers and fixes for common sources of non-determinism in JAR packages.

Improvements were also made to some of the core tools in the project:

  • timewarp for simulating the registry responses from sometime in the past.
  • proxy for transparent interception and logging of network activity.
  • and stabilize, yet another nondeterminism fixer.


SimpleX Chat server components now reproducible

SimpleX Chat is a privacy-oriented decentralised messaging platform that eliminates user identifiers and metadata, offers end-to-end encryption and has a unique approach to decentralised identity. Starting from version 6.3, however, Simplex has implemented reproducible builds for its server components. This advancement allows anyone to verify that the binaries distributed by SimpleX match the source code, improving transparency and trustworthiness.


Three new scholarly papers

Aman Sharma of the KTH Royal Institute of Technology of Stockholm, Sweden published a paper on Build and Runtime Integrity for Java (PDF). The paper’s abstract notes that “Software Supply Chain attacks are increasingly threatening the security of software systems” and goes on to compare build- and run-time integrity:

Build-time integrity ensures that the software artifact creation process, from source code to compiled binaries, remains untampered. Runtime integrity, on the other hand, guarantees that the executing application loads and runs only trusted code, preventing dynamic injection of malicious components.

Aman’s paper explores solutions to safeguard Java applications and proposes some novel techniques to detect malicious code injection. A full PDF of the paper is available.


In addition, Hamed Okhravi and Nathan Burow of Massachusetts Institute of Technology (MIT) Lincoln Laboratory along with Fred B. Schneider of Cornell University published a paper in the most recent edition of IEEE Security & Privacy on Software Bill of Materials as a Proactive Defense:

The recently mandated software bill of materials (SBOM) is intended to help mitigate software supply-chain risk. We discuss extensions that would enable an SBOM to serve as a basis for making trust assessments thus also serving as a proactive defense.

A full PDF of the paper is available.


Lastly, congratulations to Giacomo Benedetti of the University of Genoa for publishing their PhD thesis. Titled Improving Transparency, Trust, and Automation in the Software Supply Chain, Giacomo’s thesis:

addresses three critical aspects of the software supply chain to enhance security: transparency, trust, and automation. First, it investigates transparency as a mechanism to empower developers with accurate and complete insights into the software components integrated into their applications. To this end, the thesis introduces SUNSET and PIP-SBOM, leveraging modeling and SBOMs (Software Bill of Materials) as foundational tools for transparency and security. Second, it examines software trust, focusing on the effectiveness of reproducible builds in major ecosystems and proposing solutions to bolster their adoption. Finally, it emphasizes the role of automation in modern software management, particularly in ensuring user safety and application reliability. This includes developing a tool for automated security testing of GitHub Actions and analyzing the permission models of prominent platforms like GitHub, GitLab, and BitBucket.


Distribution roundup

In Debian this month:


The IzzyOnDroid Android APK repository reached another milestone in March, crossing the 40% coverage mark — specifically, more than 42% of the apps in the repository is now reproducible

Thanks to funding by NLnet/Mobifree, the project was also to put more time into their tooling. For instance, developers can now run easily their own verification builder in “less than 5 minutes”. This currently supports Debian-based systems, but support for RPM-based systems is incoming. Future work in the pipeline, including documentation, guidelines and helpers for debugging.


Fedora developer Zbigniew Jędrzejewski-Szmek announced a work-in-progress script called fedora-repro-build which attempts to reproduce an existing package within a Koji build environment. Although the project’s README file lists a number of “fields will always or almost always vary” (and there are a non-zero list of other known issues), this is an excellent first step towards full Fedora reproducibility (see above for more information).


Lastly, in openSUSE news, Bernhard M. Wiedemann posted another monthly update for his work there.


An overview of Supply Chain Attacks on Linux distributions

Fenrisk, a cybersecurity risk-management company, has published a lengthy overview of Supply Chain Attacks on Linux distributions. Authored by Maxime Rinaudo, the article asks:

[What] would it take to compromise an entire Linux distribution directly through their public infrastructure? Is it possible to perform such a compromise as simple security researchers with no available resources but time?


diffoscope & strip-nondeterminism

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 290, 291, 292 and 293 and 293 to Debian:

  • Bug fixes:

    • file(1) version 5.46 now returns XHTML document for .xhtml files such as those found nested within our .epub tests. […]
    • Also consider .aar files as APK files, at least for the sake of diffoscope. […]
    • Require the new, upcoming, version of file(1) and update our quine-related testcase. […]

  • Codebase improvements:

    • Ensure all calls to our_check_output in the ELF comparator have the potential CalledProcessError exception caught. […][…]
    • Correct an import masking issue. […]
    • Add a missing subprocess import. […]
    • Reformat openssl.py. […]
    • Update copyright years. […][…][…]

In addition, Ivan Trubach contributed a change to ignore the st_size metadata entry for directories as it is essentially arbitrary and introduces unnecessary or even spurious changes. […]


Website updates

Once again, there were a number of improvements made to our website this month, including:


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In March, a number of changes were made by Holger Levsen, including:

  • reproduce.debian.net-related:

    • Add links to two related bugs about buildinfos.debian.net. […]
    • Add an extra sync to the database backup. […]
    • Overhaul description of what the service is about. […][…][…][…][…][…]
    • Improve the documentation to indicate that need to fix syncronisation pipes. […][…]
    • Improve the statistics page by breaking down output by architecture. […]
    • Add a copyright statement. […]
    • Add a space after the package name so one can search for specific packages more easily. […]
    • Add a script to work around/implement a missing feature of debrebuild. […]

  • Misc:

    • Run debian-repro-status at the end of the chroot-install tests. […][…]
    • Document that we have unused diskspace at Ionos. […]

In addition:

And finally, node maintenance was performed by Holger Levsen […][…][…] and Mattia Rizzolo […][…].


Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports