Reproducible Builds in December 2024

← View all our monthly reports


Welcome to the December 2024 report from the Reproducible Builds project!

Our monthly reports outline what we’ve been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security when relevant. As ever, however, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.

Table of contents:

  1. reproduce.debian.net
  2. debian-repro-status
  3. On our mailing list
  4. “Enhancing the Security of Software Supply Chains”
  5. diffoscope
  6. Supply-chain attack in the Solana ecosystem
  7. Website updates
  8. Debian changes
  9. Other development news
  10. Upstream patches
  11. Reproducibility testing framework

reproduce.debian.net

Last month saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project. rebuilderd is our server designed monitor the official package repositories of Linux distributions and attempts to reproduce the observed results there.

This month, however, we are pleased to announce that not only does the service now produce graphs, the reproduce.debian.net homepage itself has become a “start page” of sorts, and the amd64.reproduce.debian.net and i386.reproduce.debian.net pages have emerged. The first of these rebuilds the amd64 architecture, naturally, but it also is building Debian packages that are marked with the ‘no architecture’ label, all. The second builder is, however, only rebuilding the i386 architecture.

Both of these services were also switched to reproduce the Debian trixie distribution instead of unstable, which started with 43% of the archive rebuild with 79.3% reproduced successfully. This is very much a work in progress, and we’ll start reproducing Debian unstable soon.

Our i386 hosts are very kindly sponsored by Infomaniak whilst the amd64 node is sponsored by OSUOSL — thank you! Indeed, we are looking for more workers for more Debian architectures; please contact us if you are able to help.


debian-repro-status

Reproducible builds developer kpcyrd has published a client program for reproduce.debian.net (see above) that queries the status of the locally installed packages and rates the system with a percentage score. This tool works analogously to arch-repro-status for the Arch Linux Reproducible Builds setup.

The tool was packaged for Debian and is currently available in Debian trixie: it can be installed with apt install debian-repro-status.


On our mailing list

On our mailing list this month:

  • Bernhard M. Wiedemann wrote a detailed post on his “long journey towards a bit-reproducible Emacs package.” In his interesting message, Bernhard goes into depth about the tools that they used and the lower-level technical details of, for instance, compatibility with the version for glibc within openSUSE.

  • Shivanand Kunijadar posed a question pertaining to the reproducibility issues with encrypted images. Shivanand explains that they must “use a random IV for encryption with AES CBC. The resulting artifact is not reproducible due to the random IV used.” The message resulted in a handful of replies, hopefully helpful!

  • User Danilo posted an in interesting question related to their attempts in trying to achieve reproducible builds for Threema Desktop 2.0. The question resulted in a number of replies attempting to find the right combination of compiler and linker flags (for example).

  • Longstanding contributor David A. Wheeler wrote to our list announcing the release of the “Census III of Free and Open Source Software: Application Libraries” report written by Frank Nagle, Kate Powell, Richie Zitomer and David himself. As David writes in his message, the report attempts to “answer the question ‘what is the most popular Free and Open Source Software (FOSS)?’”.

  • Lastly, kpcyrd followed-up to a post from September 2024 which mentioned their desire for “someone” to implement “a hashset of allowed module hashes that is generated during the kernel build and then embedded in the kernel image”, thus enabling a deterministic and reproducible build. However, they are now reporting that “somebody implemented the hash-based allow list feature and submitted it to the Linux kernel mailing list”. Like kpcyrd, we hope it gets merged.


Enhancing the Security of Software Supply Chains: Methods and Practices

Mehdi Keshani of the Delft University of Technology in the Netherlands has published their thesis on “Enhancing the Security of Software Supply Chains: Methods and Practices”. Their introductory summary first begins with an outline of software supply chains and the importance of the Maven ecosystem before outlining the issues that it faces “that threaten its security and effectiveness”. To address these:

First, we propose an automated approach for library reproducibility to enhance library security during the deployment phase. We then develop a scalable call graph generation technique to support various use cases, such as method-level vulnerability analysis and change impact analysis, which help mitigate security challenges within the ecosystem. Utilizing the generated call graphs, we explore the impact of libraries on their users. Finally, through empirical research and mining techniques, we investigate the current state of the Maven ecosystem, identify harmful practices, and propose recommendations to address them.

A PDF of Mehdi’s entire thesis is available to download.


diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 283 and 284 to Debian:

  • Update copyright years. […]
  • Update tests to support file 5.46. […][…]
  • Simplify tests_quines.py::test_{differences,differences_deb} to simply use assert_diff and not mangle the test fixture. […]


Supply-chain attack in the Solana ecosystem

A significant supply-chain attack impacted Solana, an ecosystem for decentralised applications running on a blockchain.

Hackers targeted the @solana/web3.js JavaScript library and embedded malicious code that extracted private keys and drained funds from cryptocurrency wallets. According to some reports, about $160,000 worth of assets were stolen, not including SOL tokens and other crypto assets.


Website updates

Similar to last month, there was a large number of changes made to our website this month, including:

  • Chris Lamb:

    • Make the landing page hero look nicer when the vertical height component of the viewport is restricted, not just the horizontal width.
    • Rename the “Buy-in” page to “Why Reproducible Builds?” […]
    • Removing the top black border. […][…]

  • Holger Levsen:

  • hulkoba:

    • Remove the sidebar-type layout and move to a static navigation element. […][…][…][…]
    • Create and merge a new Success stories page, which “highlights the success stories of Reproducible Builds, showcasing real-world examples of projects shipping with verifiable, reproducible builds. These stories aim to enhance the technical resilience of the initiative by encouraging community involvement and inspiring new contributions.”. […]
    • Further changes to the homepage. […]
    • Remove the translation icon from the navigation bar. […]
    • Remove unused CSS styles pertaining to the sidebar. […]
    • Add sponsors to the global footer. […]
    • Add extra space on large screens on the Who page. […]
    • Hide the side navigation on small screens on the Documentation pages. […]


Debian changes

There were a significant number of reproducibility-related changes within Debian this month, including:

  • Santiago Vila uploaded version 0.11+nmu4 of the dh-buildinfo package. In this release, the dh_buildinfo becomes a no-op — ie. it no longer does anything beyond warning the developer that the dh-buildinfo package is now obsolete. In his upload, Santiago wrote that “We still want packages to drop their [dependency] on dh-buildinfo, but now they will immediately benefit from this change after a simple rebuild.”

  • Holger Levsen filed Debian bug #1091550 requesting a rebuild of a number of packages that were built with a “very old version” of dpkg.

  • Fay Stegerman contributed to an extensive thread on the debian-devel development mailing list on the topic of “Supporting alternative zlib implementations”. In particular, Fay wrote about her results experimenting whether zlib-ng produces identical results or not.

  • kpcyrd uploaded a new rust-rebuilderd-worker, rust-derp, rust-in-toto and debian-repro-status to Debian, which passed successfully through the so-called NEW queue.

  • Gioele Barabucci filed a number of bugs against the debrebuild component/script of the devscripts package, including:

    • #1089087: Address a spurious extra subdirectory in the build path.
    • #1089201: Extra zero bytes added to .dynstr when rebuilding CMake projects.
    • #1089088: Some binNMUs have a 1-second offset in some timestamps.

  • Gioele Barabucci also filed a bug against the dh-r package to report that the Recommends and Suggests fields are missing from rebuilt R packages. At the time of writing, this bug has no patch and needs some help to make over 350 binary packages reproducible.

  • Lastly, 8 reviews of Debian packages were added, 11 were updated and 11 were removed this month adding to our knowledge about identified issues.


Other development news

In other ecosystem and distribution news:

  • Lastly, in openSUSE, Bernhard M. Wiedemann published another report for the distribution. There, Bernhard reports about the success of building ‘R-B-OS’, a partial fork of openSUSE with only 100% bit-reproducible packages. This effort was sponsored by the NLNet NGI0 initiative.


Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In November, a number of changes were made by Holger Levsen, including:

  • reproduce.debian.net-related:

    • Add a new i386.reproduce.debian.net rebuilder. […][…][…][…][…][…]
    • Make a number of updates to the documentation. […][…][…][…][…]
    • Run i386.reproduce.debian.net run on a public port to allow external workers. […]
    • Add a link to the /api/v0/pkgs/list endpoint. […]
    • Add support for a statistics page. […][…][…][…][…][…]
    • Limit build logs to 20 MiB and diffoscope output to 10 MiB. […]
    • Improve the frontpage. […][…]
    • Explain that we’re testing arch:any and arch:all on the amd64 architecture, but only arch:any on i386. […]

  • Misc:

    • Remove code for testing Arch Linux, which has moved to reproduce.archlinux.org. […][…]
    • Don’t install dstat on Jenkins nodes anymore as its been removed from Debian trixie. […]
    • Prepare the infom08-i386 node to become another rebuilder. […]
    • Add debug date output for benchmarking the reproducible_pool_buildinfos.sh script. […]
    • Install installation-birthday everywhere. […]
    • Temporarily disable automatic updates of pool links on buildinfos.debian.net. […]
    • Install Recommends by default on Jenkins nodes. […]
    • Rename rebuilder_stats.py to rebuilderd_stats.py. […]
    • r.d.n/stats: minor formatting changes. […]
    • Install files under /etc/cron.d/ with the correct permissions. […]

… and Jochen Sprickerhof made the following changes:

Lastly, Gioele Barabucci also classified packages affected by 1-second offset issue filed as Debian bug #1089088 […][…][…][…], Chris Hofstaedtler updated the URL for Grml’s dpkg.selections file  […], Roland Clobus updated the Jenkins log parser to parse warnings from diffoscope […] and Mattia Rizzolo banned a number of bots and crawlers from the service […][…].


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports