Reproducible Builds in April 2025

← View all our monthly reports


Welcome to our fourth report from the Reproducible Builds project in 2025. These monthly reports outline what we’ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. Lastly, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.

Table of contents:

  1. reproduce.debian.net
  2. Fifty Years of Open Source Software Supply Chain Security
  3. 4th CHAINS Software Supply Chain Workshop
  4. Mailing list updates
  5. Canonicalization for Unreproducible Builds in Java
  6. OSS Rebuild adds new TUI features
  7. Distribution roundup
  8. diffoscope & strip-nondeterminism
  9. Website updates
  10. Reproducibility testing framework
  11. Upstream patches

reproduce.debian.net

The last few months have seen the introduction, development and deployment of reproduce.debian.net. In technical terms, this is an instance of rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there.

This month, however, we are pleased to announce that reproduce.debian.net now tests all the Debian trixie architectures except s390x and mips64el.

The ppc64el architecture was added through the generous support of Oregon State University Open Source Laboratory (OSUOSL), and we can support the armel architecture thanks to CodeThink.


Fifty Years of Open Source Software Supply Chain Security

Russ Cox has published a must-read article in ACM Queue on Fifty Years of Open Source Software Supply Chain Security. Subtitled, “For decades, software reuse was only a lofty goal. Now it’s very real.”, Russ’ article goes on to outline the history and original goals of software supply-chain security in the US military in the early 1970s, all the way to the XZ Utils backdoor of 2024. Through that lens, Russ explores the problem and how it has changed, and hasn’t changed, over time.

He concludes as follows:

We are all struggling with a massive shift that has happened in the past 10 or 20 years in the software industry. For decades, software reuse was only a lofty goal. Now it’s very real. Modern programming environments such as Go, Node and Rust have made it trivial to reuse work by others, but our instincts about responsible behaviors have not yet adapted to this new reality.

We all have more work to do.


4th CHAINS Software Supply Chain Workshop

Convened as part of the CHAINS research project at the KTH Royal Institute of Technology in Stockholm, Sweden, the 4th CHAINS Software Supply Chain Workshop occurred during April. During the workshop, there were a number of relevant workshops, including:

The full listing of the agenda is available on the workshop’s website.


Mailing list updates

On our mailing list this month:

  • Luca DiMaio of Chainguard posted to the list reporting that they had successfully implemented reproducible filesystem images with both ext4 and an EFI system partition. They go on to list the various methods, and the thread generated at least fifteen replies.

  • David Wheeler announced that the OpenSSF is building a “glossary” of sorts in order that they “consistently use the same meaning for the same term” and, moreover, that they have drafted a definition for ‘reproducible build’. The thread generated a significant number of replies on the definition, leading to a potential update to the Reproducible Build’s own definition.

  • Lastly, kpcyrd posted to the list with a timely reminder and update on their repro-env” tool. As first reported in our July 2023 report, kpcyrd mentions that:

    My initial interest in reproducible builds was “how do I distribute pre-compiled binaries on GitHub without people raising security concerns about them”. I’ve cycled back to this original problem about 5 years later and built a tool that is meant to address this. […]


Canonicalization for Unreproducible Builds in Java

Aman Sharma, Benoit Baudry and Martin Monperrus have published a new scholarly study related to reproducible builds within Java. Titled Canonicalization for Unreproducible Builds in Java, the article’s abstract is as follows:

[…] Achieving reproducibility at scale remains difficult, especially in Java, due to a range of non-deterministic factors and caveats in the build process. In this work, we focus on reproducibility in Java-based software, archetypal of enterprise applications. We introduce a conceptual framework for reproducible builds, we analyze a large dataset from Reproducible Central and we develop a novel taxonomy of six root causes of unreproducibility. We study actionable mitigations: artifact and bytecode canonicalization using OSS-Rebuild and jNorm respectively. Finally, we present Chains-Rebuild, a tool that raises reproducibility success from 9.48% to 26.89% on 12,283 unreproducible artifacts. To sum up, our contributions are the first large-scale taxonomy of build unreproducibility causes in Java, a publicly available dataset of unreproducible builds, and Chains-Rebuild, a canonicalization tool for mitigating unreproducible builds in Java.

A full PDF of their article is available from arXiv.


OSS Rebuild adds new TUI features

OSS Rebuild aims to automate rebuilding upstream language packages (e.g. from PyPI, crates.io and npm registries) and publish signed attestations and build definitions for public use.

OSS Rebuild ships a text-based user interface (TUI) for viewing, launching, and debugging rebuilds. While previously requiring ownership of a full instance of OSS Rebuild’s hosted infrastructure, the TUI now supports a fully local mode of build execution and artifact storage. Thanks to Giacomo Benedetti for his usage feedback and work to extend the local-only development toolkit.

Another feature added to the TUI was an experimental chatbot integration that provides interactive feedback on rebuild failure root causes and suggests fixes.


Distribution roundup

In Debian this month:

  • Roland Clobus posted another status report on reproducible ISO images on our mailing list this month, with the summary that “all live images build reproducibly from the online Debian archive”.

  • Debian developer Simon Josefsson published another two reproducibility-related blog posts this month, the first on the topic of Verified Reproducible Tarballs. Simon sardonically challenges the reader as follows: “Do you want a supply-chain challenge for the Easter weekend? Pick some well-known software and try to re-create the official release tarballs from the corresponding Git checkout. Is anyone able to reproduce anything these days?” After that, they also published a blog post on Building Debian in a GitLab Pipeline using their multi-stage rebuild approach.

  • Roland also posted to our mailing list to highlight that “there is now another tool in Debian that generates reproducible output, equivs”. This is a tool to create trivial Debian packages that might Depend on other packages. As Roland writes, “building the [equivs] package has been reproducible for a while, [but] now the output of the [tool] has become reproducible as well”.

  • Lastly, 9 reviews of Debian packages were added, 10 were updated and 10 were removed this month adding to our extensive knowledge about identified issues.

The IzzyOnDroid Android APK repository made more progress in April. Thanks to funding by NLnet and Mobifree, the project was also to put more time into their tooling. For instance, developers can now easily run their own verification builder in “less than 5 minutes”. This currently supports Debian-based systems, but support for RPM-based systems is incoming.

  • The rbuilder_setup tool can now setup the entire framework within less than five minutes. The process is configurable, too, so everything from “just the basics to verify builds” up to a fully-fledged RB environment is also possible.

  • This tool works on Debian, RedHat and Arch Linux, as well as their derivates. The project has received successful reports from Debian, Ubuntu, Fedora and some Arch Linux derivates so far.

  • Documentation on how to work with reproducible builds (making apps reproducible, debugging unreproducible packages, etc) is available in the project’s wiki page.

  • Future work is also in the pipeline, including documentation, guidelines and helpers for debugging.

NixOS defined an Outreachy project for improving build reproducibility. In the application phase, NixOS saw some strong candidates providing contributions, both on the NixOS side and upstream: guider-le-ecit analyzed a libpinyin issue. Tessy James fixed an issue in arandr and helped analyze one in libvlc that led to a proposed upstream fix. Finally, 3pleX fixed an issue which was accepted in upstream kitty, one in upstream maturin, one in upstream python-sip and one in the Nix packaging of python-libbytesize. Sadly, the funding for this internship fell through, so NixOS were forced to abandon their search.

Lastly, in openSUSE news, Bernhard M. Wiedemann posted another monthly update for their work there.


diffoscope & strip-nondeterminism

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading a number of versions to Debian:

  • Use the --walk argument over the potentially dangerous alternative --scan when calling out to zipdetails(1). […]
  • Correct a longstanding issue where many >-based version tests used in conditional fixtures were broken. This was used to ensure that specific tests were only run when the version on the system was newer than a particular number. Thanks to Colin Watson for the report (Debian bug #1102658) […]
  • Address a long-hidden issue in the test_versions testsuite as well, where we weren’t actually testing the greater-than comparisons mentioned above, as it was masked by the tests for equality. […]
  • Update copyright years. […]

In strip-nondeterminism, however, Holger Levsen updated the Continuous Integration (CI) configuration in order to use the standard Debian pipelines via debian/salsa-ci.yml instead of using .gitlab-ci.yml. […]


Website updates

Once again, there were a number of improvements made to our website this month including:


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In April, a number of changes were made by Holger Levsen, including:

  • reproduce.debian.net-related:

    • Add armel.reproduce.debian.net to support the armel architecture. […][…]
    • Add a new ARM node, codethink05. […][…]
    • Add ppc64el.reproduce.debian.net to support testing of the ppc64el architecture. […][…][…]
    • Improve the reproduce.debian.net front page. […][…]
    • Make various changes to the ppc64el nodes. […][…]9[…][…]
    • Make various changes to the arm64 and armhf nodes. […][…][…][…]
    • Various changes related to the rebuilderd-worker entry point. […][…][…]
    • Create and deploy a pkgsync script. […][…][…][…][…][…][…][…]
    • Fix the monitoring of the riscv64 architecture. […][…]
    • Make a number of changes related to starting the rebuilderd service. […][…][…][…]

  • Backup-related:

    • Backup the rebuilder databases every week. […][…][…][…]
    • Improve the node health checks. […][…]

  • Misc:

    • Re-use existing connections to the SSH proxy node on the riscv64 nodes. […][…]
    • Node maintenance. […][…][…]

In addition:

  • Jochen Sprickerhof fixed the risvc64 host names […] and requested access to all the rebuilderd nodes […].

  • Mattia Rizzolo updated the self-serve rebuild scheduling tool, replacing the deprecated “SSO”-style authentication with OpenIDC which authenticates against salsa.debian.org. […][…][…]

  • Roland Clobus updated the configuration for the osuosl3 node to designate 4 workers for bigger builds. […]


Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:



Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports