Reproducible Builds in November 2024

← View all our monthly reports


Welcome to the November 2024 report from the Reproducible Builds project!

Our monthly reports outline what we’ve been up to over the past month, and highlight items of news from elsewhere in the world of software supply-chain security where relevant. As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.

Reproducible Builds mourns the passing of Lunar

The Reproducible Builds community sadly announced it has lost its founding member. Jérémy Bobbio aka ‘Lunar’ passed away on Friday November 8th in palliative care in Rennes, France.

Lunar was instrumental in starting the Reproducible Builds project in 2013 as a loose initiative within the Debian project. Many of our earliest status reports were written by him, and many of our key tools in use today are based on his designs. Lunar’s creativity, insight and kindness were often noted. He will be greatly missed.

You can view our full tribute elsewhere on our website.


Introducing reproduce.debian.net…

This month saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse (see below for more information), reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project. rebuilderd is our server designed monitor package repositories of Linux distributions and attempt to reproduce the actual (i.e. observed) results there.

In November, reproduce.debian.net was only rebuilding Debian unstable on the amd64 architecture, but throughout the MiniDebConf it had attempted to rebuild 66% of the official archive.

However, the results as-of writing, it can be determined that it is currently possible to bit-for-bit reproduce and corroborate approximately 78% of the actual binaries distributed by Debian — that is, using the .buildinfo files hosted by Debian itself.

reproduce.debian.net also contains instructions how to setup one’s own rebuilderd instance, and we very much invite everyone with a machine to spare to setup their own version and to share the results. Whilst rebuilderd is still in development, it has been used to reproduce Arch Linux since 2019. We are especially looking for installations targeting Debian architectures other than i386 and amd64.


New landing page design

FIXME

Zig programming language demonstrated reproducible

Motiejus Jakšty posted an interesting and practical blog post on his successful attempt to reproduce the Zig programming language without using the pre-compiled binaries checked into the repository, and despite the circular dependency inherent in its bootstrapping process.

As a summary, Motiejus concludes that:

I can now confidently say (and you can also check, you don’t need to trust me) that there is nothing hiding in zig1.wasm [the checked-in binary] that hasn’t been checked-in as a source file.

The full post is full of practical details, and includes a few open questions.


SBOMs for Python packages

The Python Software Foundation has announced a new “cross-functional project for SBOMs and Python packages”. Seth Michael Larson writes that the project is “specifically looking to solve these issues”:

  • Enable Python users that require SBOM documents (likely due to regulations like CRA or SSDF) to self-serve using existing SBOM generation tools.
  • Solve the “phantom dependency” problem, where non-Python software is bundled in Python packages but not recorded in any metadata. This makes the job of software composition analysis (SCA) tools difficult or impossible.
  • Make the adoption work by relevant projects such as build backends, auditwheel-esque tools, as minimal as possible. Empower users who are interested in having better SBOM data for the Python projects they are using to be able to contribute engineering time towards that goal.

A GitHub repository for the initiative is available, and there are a number of queries, comments and remarks on Seth’s Discourse forum post.


Reproducible builds by default in Maven 4

On our mailing list this month, Hervé Boutemy reported the latest release of Maven (4.0.0-beta-5) has reproducible builds enabled by default. In his mailing list post, Hervé mentions that “this story started during our Reproducible Builds summit in Hamburg”, where he created the upstream issue that builds on a “multi-year” effort to have Maven builds configured for reproducibility.


PyPI now supports digital attestations

Elsewhere in the Python ecosystem and as reported on LWN and elsewhere, the Python Package Index (PyPI) has announced that it has finalised support for PEP 740 (“Index support for digital attestations”).

Trail of Bits, who performed much of the development work, has an in-depth blog post about the work and its adoption, as well as what is left undone:

One thing is notably missing from all of this work: downstream verification. […]

This isn’t an acceptable end state (cryptographic attestations have defensive properties only insofar as they’re actually verified), so we’re looking into ways to bring verification to individual installing clients. In particular, we’re currently working on a plugin architecture for pip that will enable users to load verification logic directly into their pip install flows.

There was an in-depth discussion on LWN’s announcement page, as well as on Hacker News.


Debian updates

There was significant development within Debian this month. Firstly, at the recent MiniDebConf in Toulouse, France, Holger Levsen gave a Debian-specific talk on rebuilding packages distributed from ftp.debian.org — that is to say, how to reproduce the results from the official Debian build servers:

Holger described the talk as follows:

For more than ten years, the Reproducible Builds project has worked towards reproducible builds of many projects, and for ten years now we have build Debian packages twice—with maximal variations applied—to see if they can be build reproducible still.

Since about a month, we’ve also been rebuilding trying to exactly match the builds being distributed via ftp.debian.org. This talk will describe the setup and the lessons learned so far, and why the results currently are what they are (spoiler: they are less than 30% reproducible), and what we can do to fix that.

Holger’s slides and video in .webm format are available.

FIXME: https://bits.debian.org/2024/12/bits-from-the-dpl-december.html


Next, rebuilderd is the server to monitor package repositories of Linux distributions and attempt to reproduce the observed results. This month, version 0.21.0 released, most notably with improved support for binNMUs by Jochen Sprickerhof and updating the rebuilderd-debian.sh integration to the latest debrebuild version by Holger Levsen. There has also been significant work to get the rebuilderd package into the Debian archive, in particular, both rust-rebuilderd-common version 0.20.0-1 and rust-rust-lzma version 0.6.0-1 were packaged by kpcyrd and uploaded by Holger Levsen.

Related to this, Holger Levsen submitted three additional issues against rebuilderd as well:

  • rebuildctl should be more verbose when encountering issues. […]
  • Please add an option to used randomised queues. […]
  • Scheduling and re-scheduling multiple packages at once. […]

… and lastly, Jochen Sprickerhof submitted one an issue requested that rebuilderd downloads the source package in addition to the .buildinfo file […] and kpcyrd also submitted and fixed an issue surrounding dependencies and clarifying the license […]


Separate to this, back in 2018, Chris Lamb filed a bug report against the sphinx-gallery package as it generates unreproducible content in various ways. This month, however, Dmitry Shachnev finally closed the bug, listing the multiple sub-issues that were part of the problem and how they were resolved.


Elsewhere, Roland Clobus posted to our mailing list this month, asking for input on a bug in Debian’s ca-certificates-java package. The issue is that the Java key management tools embed timestamps in its output, and this output ends up in the /etc/ssl/certs/java/cacerts file on the generated ISO images. A discussion resulted from Roland’s post suggesting some short- and medium-term solutions to the problem.


Holger Levsen uploaded some packages with reproducibility-related changes:


Lastly, 12 reviews of Debian packages were added, 5 were updated and 21 were removed this month adding to our knowledge about identified issues in Debian.


“Dependency Challenges in OSS Package Registries”

At BENEVOL, the Belgium-Netherlands Software Evolution workshop in Namur, Belgium, Tom Mens and Alexandre Decan presented their paper, “An Overview and Catalogue of Dependency Challenges in Open Source Software Package Registries”.

The abstract of their paper is as follows:

While open-source software has enabled significant levels of reuse to speed up software development, it has also given rise to the dreadful dependency hell that all software practitioners face on a regular basis. This article provides a catalogue of dependency-related challenges that come with relying on OSS packages or libraries. The catalogue is based on the scientific literature on empirical research that has been conducted to understand, quantify and overcome these challenges. […]

A PDF of the paper is available online.


Website updates

Yet again, there were an enormous number of chages made to our website this month, including:

  • Alex Feyerke and Mariano Giménez:

  • Bernhard M. Wiedemann:

    • Update the “System images” page to document the e2fsprogs approach. […]

  • Chris Lamb:

  • FC (Fay) Stegerman:

    • Replace more inline markdown with HTML on the “Success stories” page. […]
    • Add some links, fix some other links and correct some spelling errors on the “Tools” page. […]

  • Holger Levsen:

    • Add a historical presentation (“Reproducible builds everywhere eg. in Debian, OpenWrt and LEDE”) from October 2016. […]
    • Add jochensp and Oejet to the list of known contributors. […][…]

  • Julia Krüger:

  • Ninette Adhikari & hulkoba:

    • Add/rework the list of success stories into a new page that clearly shows milestones in Reproducible Builds. […][…][…][…][…][…]

  • Philip Rinn:

    • Import 47 historical weekly reports. […]

  • hulkoba:

    • Add alt text to almost all images (!). […][…]
    • Fix a number of links on the “Talks”. […][…]
    • Avoid so-called ‘ghost’ buttons by not using <button> elements as links, as the affordance of a <button> implies an action with (potentially) a side effect. […][…]
    • Center the sponsor logos on the homepage. […]

    • Move publications and generate them instead from a data.yml file with an improved layout. […][…]

    • Make a large number of small but impactful stylisting changes. […][…][…][…]

    • Expand the “Tools” to include a number of missing tools, fix some styling issues and fix a number of stale/broken links. […][…][…][…][…][…]


Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


Misc development news


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In November, a number of changes were made by Holger Levsen, including:

  • reproduce.debian.net-related changes:

    • Create and introduce a new reproduce.debian.net service and subdomain […]
    • Make a large number of documentation changes relevant to rebuilderd. […][…][…][…][…]
    • Explain a temporary workaround for a specific issue in rebuilderd. […]
    • Setup another rebuilderd instance on the o4 node and update installation documentation to match. […][…]
    • Make a number of helpful/cosmetic changes to the interface, such as clarifying terms and adding links. […][…][…][…][…]
    • Deploy configuration to the /opt and /var directories. […][…]
    • Add an infancy (or ‘alpha’) disclaimer. […][…]
    • Add more notes to the temporary rebuilderd documentation. […]
    • Commit an nginx configuration file for reproduce.debian.net’s “Stats” page. […]
    • Commit a rebuilder-worker.conf configuration for the o5 node. […]

  • Debian-related changes:

    • Grant jspricke and jochensp access to the o5 node. […][…]
    • Build the qemu package with the nocheck build flag. […]

  • Misc changes:

    • Adapt the update_jdn.sh script for new Debian trixie systems. […]
    • Stop installing the PostgreSQL database engine on the o4 and o5 nodes. […]
    • Prevent accidental reboots of the o4 node because of a long-running job owned by josch. […][…]

In addition, Mattia Rizzolo addressed a number of issues with reproduce.debian.net […][…][…][…]. And lastly, both Holger Levsen […][…][…][…] and Vagrant Cascadian […][…][…][…] performed node maintenance.


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports

Follow us on Twitter @ReproBuilds, Mastodon @reproducible_builds@fosstodon.org & Reddit and please consider making a donation. • Content licensed under CC BY-SA 4.0, style licensed under MIT. Templates and styles based on the Tor Styleguide. Logos and trademarks belong to their respective owners. • Patches for this website welcome via our Git repository (instructions) or via our mailing list. • Full contact info

Conservancy Logo

Reproducible Builds Logo