Reproducible Builds in October 2024

Welcome to the October 2024 report from the Reproducible Builds project.

Our reports attempt to outline what we’ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Table of contents:

1. Beyond bitwise equality for Reproducible Builds?
2. ‘Two Ways to Trustworthy’ at SeaGL 2024
3. Number of cores affected Android compiler output
4. On our mailing list…
5. diffoscope
6. IzzyOnDroid passed 25% reproducible apps
7. Distribution work
8. Website updates
9. Reproducibility testing framework
10. Supply-chain security at Open Source Summit EU
11. Upstream patches

---

Beyond bitwise equality for Reproducible Builds?

Jens Dietrich, Tim White, of Victoria University of Wellington, New Zealand along with Behnaz Hassanshahi and Paddy Krishnan of Oracle Labs Australia published a paper entitled “Levels of Binary Equivalence for the Comparison of Binaries from Alternative Builds”:

The availability of multiple binaries built from the same sources creates new challenges and opportunities, and raises questions such as: “Does build A confirm the integrity of build B?” or “Can build A reveal a compromised build B?”. To answer such questions requires a notion of equivalence between binaries. We demonstrate that the obvious approach based on bitwise equality has significant shortcomings in practice, and that there is value in opting for alternative notions. We conceptualise this by introducing levels of equivalence, inspired by clone detection types.

A PDF of the paper is freely available.

‘Two Ways to Trustworthy’ at SeaGL 2024

On Friday 8th November, Vagrant Cascadian will present a talk entitled Two Ways to Trustworthy at SeaGL in Seattle, WA.

Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture. Vagrant’s talk:

[…] delves into how two project[s] approaches fundamental security features through Reproducible Builds, Bootstrappable Builds, code auditability, etc. to improve trustworthiness, allowing independent verification; trustworthy projects require little to no trust.

Exploring the challenges that each project faces due to very different technical architectures, but also contextually relevant social structure, adoption patterns, and organizational history should provide a good backdrop to understand how different approaches to security might evolve, with real-world merits and downsides.

Number of cores affected Android compiler output

Fay Stegerman wrote that the cause of the Android toolchain bug from September’s report that she reported to the Android issue tracker has been found and the bug has been fixed.

the D8 Java to DEX compiler (part of the Android toolchain) eliminated a redundant field load if running the class’s static initialiser was known to be free of side effects, which ended up accidentally depending on the sharding of the input, which is dependent on the number of CPU cores used during the build.

To make it easier to understand the bug and the patch, Fay also made a small example to illustrate when and why the optimisation involved is valid.

On our mailing list…

On our mailing list this month:

• Following-up to previous work, James Addison informed the list that the recently-released Sphinx documentation generator includes improvements to the next copyright notice substitutions.

• Pol Dellaiera wrote to the list in order to seek advice around introducing the concept of reproducibility to computer science Masters students at the University of Mons, Belgium.

• James Addison also followed-up to a previous thread on “CONFIG_MODULE_SIG and the unreproducible Linux Kernel” to add: “I wonder whether it would be possible to use the Linux kernel’s Integrity Policy Enforcement to deploy a policy that would prevent loading of anything except a set of expected kernel modules.” […]

• There were also two informative replies from David Wheeler to a broad-based discussion on Reproducible Builds being defined in various standards. […][…]

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 279, 280, 281 and 282 to Debian:

• Ignore errors when listing .ar archives (#1085257). […]
• Don’t try and test with systemd-ukify in the Debian stable distribution. […]
• Drop Depends on the deprecated python3-pkg-resources (#1083362). […]

In addition, Jelle van der Waa added support for Unified Kernel Image (UKI) files. […][…][…] Furthermore, Vagrant Cascadian updated diffoscope in GNU Guix to version 282. […][…]

IzzyOnDroid passed 25% reproducible apps

The IzzyOnDroid project has reached a good milestone by reaching over 25% of the ~1,200 Android apps provided by their repository (of official APKs built by the original application developers) having been confirmed to be reproducible by a rebuilder.

Distribution work

In Debian this month:

• Holger Levsen uploaded devscripts version 2.24.2, including many changes to the debootsnap, debrebuild and reproducible-check scripts. This is the first time that debrebuild actually works (using sbuild’s unshare backend). As part of this, Holger also fixed an issue in the reproducible-check script where a typo in the code led to incorrect results […]

• Recently, a news entry was added to snapshot.debian.org’s homepage, describing the recent changes that made the system stable again:

  The new server has no problems keeping up with importing the full archives on every update, as each run finishes comfortably in time before it’s time to run again. [While] the new server is the one doing all the importing of updated archives, the HTTP interface is being served by both the new server and one of the VM’s at LeaseWeb.

  The entry list a number of specific updates surrounding the API endpoints and rate limiting.

• Lastly, 12 reviews of Debian packages were added, 3 were updated and 18 were removed this month adding to our knowledge about identified issues.

Elsewhere in distribution news, Zbigniew Jędrzejewski-Szmek performed another rebuild of Fedora 42 packages, with the headline result being that 91% of the packages are reproducible. Zbigniew also reported a reproducibility problem with QImage.

Finally, in openSUSE, Bernhard M. Wiedemann published another report for that distribution.

Website updates

There were an enormous number of improvements made to our website this month, including:

• Alba Herrerias:

  • Improve consistency across distribution-specific guides. […]
  • Fix a number of links on the Contribute page. […]

• Chris Lamb:

  • Correct the name of Civil Infrastructure Platform name and update image on the Projects page. […]
  • Update broken link on the Value Initialization page. […]
  • Try and make pipeline/branch builds of the website easier to browse. […][…][…][…]

• hulkoba

  • Contribute to the new ‘Success stories’ page. […]

• James Addison:

  • Huge and significant work on a (as-yet-merged) quickstart guide to be linked from the homepage […][…][…][…][…]
  • On the homepage, link directly to the Projects subpage. […]
  • Relocate “dependency-drift” notes to the Volatile inputs page. […]

• Ninette Adhikari:

  • Add a brand new ‘Success stories’ page that “highlights the success stories of Reproducible Builds, showcasing real-world examples of projects shipping with verifiable, reproducible builds”. […][…][…][…][…][…]

• Pol Dellaiera:

  • Update the website’s README page for building the website under NixOS. […][…][…][…][…]
  • Add a new academic paper citation. […]

Lastly, Holger Levsen filed an extensive issue detailing a request to create an overview of recommendations and standards in relation to reproducible builds.

Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, a number of changes were made by Holger Levsen, including:

• Add a basic index.html for rebuilderd. […]
• Update the nginx.conf configuration file for rebuilderd. […]
• Document how to use a rescue system for Infomaniak’s OpenStack cloud. […]
• Update usage info for two particular nodes. […]
• Fix up a version skew check to fix the name of the riscv64 architecture. […]
• Update the rebuilderd-related TODO. […]

In addition, Mattia Rizzolo added a new IP address for the inos5 node […] and Vagrant Cascadian brought 4 virt nodes back online […].

Supply-chain security at Open Source Summit EU

The Open Source Summit EU took place recently, and covered plenty of topics related to supply-chain security, including:

• Public Sector & OpenSSF: Principles for Package Repository Security
• The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency and Usability in AI
• Structured Scorecard Results: Tailor Your Own Supply-Chain Security Policies
• Lightning Talk: Elephant in the Room: How Supply Chain Security Standards Are Not Standard and What to Do About It
• Lightning Talk: Charting the Course for Secure Software Supply Chain with Guac-AI-Mole!
• TPMs, Merkle Trees and TEEs: Enhancing SLSA with Hardware-Assisted Build Environment Verification
• Accountability Taxonomy for AI Software Bill of Materials
• Securing Your Supply Chain with an Open Source Ecosystem
• OSS Supply Chain Threats and Why You Need a Holistic Security Strategy
• A Step Closer to in-Toto’lly Secure: Using in-Toto and OPA Gatekeeper to Verify Artifact Integrity
• Panel Discussion: Improving Supply Chain Integrity with OpenSSF Technologies
• Case Study: 10+ Years of Developing an SBOM System and the Dos and Don’ts
• SBOM in SaaS Environments: An Update
• Securing Git Repositories with Gittuf

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann

  • apache-ivy (.zip modification time)
  • ccache (build failure)
  • colord (CPU)
  • efivar (CPU/march=native)
  • gsl (no check)
  • libcamera (date/copyright year)
  • libreoffice (possible rpm/build toolchain corruption bug)
  • moto (.gz modification time)
  • openssl-1_1 (date-related issue)
  • python-pygraphviz (benchmark)
  • sphinx/python-pygraphviz (benchmark)
  • python-panel (package.lock has random port)
  • python-propcache (random temporary path)
  • python314 (.gz-related modification time)
  • rusty_v8 (random .o files)
  • scapy (date)
  • wine (parallelism)
  • ibmtss (FTBFS-2026)
  • pymol (date)
  • pandas (ASLR)
  • linutil (drop date)
  • lsof (also filed in openSUSE: uname -r in LSOF_VSTR)
  • schily (also filed in openSUSE: uname -r)
  • superlu (nocheck)
  • util (random test failure)
  • ceph (year-2038 variation from embedded boost)

• Chris Lamb:

  • #1085097 filed against python-roborock.
  • #1085280 filed against pywayland.
  • #1085283 filed against readsb.
  • #1085381 filed against xraylarch.

• James Addison:

  • #1085112 filed against distro-info.

• Zbigniew Jędrzejewski-Szmek:

  • calibre (two sort issues) […][…]

---

Finally, If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.

• Mastodon: @reproducible_builds@fosstodon.org

• Mailing list: rb-general@lists.reproducible-builds.org

• Twitter: @ReproBuilds