Reproducible Builds in March 2023

← View all our monthly reports


Welcome to the March 2023 report from the Reproducible Builds project.

In these reports we outline the most important things that we have been up to over the past month. As a quick recap, the motivation behind the reproducible builds effort is to ensure no malicious flaws have been introduced during compilation and distributing processes. It does this by ensuring identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

If you are interested in contributing to the project, please do visit our Contribute page on our website.


News

There was progress towards making the Go programming language reproducible this month, with the overall goal remaining making the Go binaries distributed from Google and by Arch Linux (and others) to be bit-for-bit identical. These changes could become part of the upcoming version 1.21 release of Go. An issue in the Go issue tracker (#57120) is being used to follow and record progress on this.


Arnout Engelen updated our website to add and update reproducibility-related links for NixOS to reproducible.nixos.org. […]. In addition, Chris Lamb made some cosmetic changes to our presentations and resources page. […][…]


Intel published a guide on how to reproducibly build their Trust Domain Extensions (TDX) firmware. TDX here refers to an Intel technology that combines their existing virtual machine and memory encryption technology with a new kind of virtual machine guest called a Trust Domain. This runs the CPU in a mode that protects the confidentiality of its memory contents and its state from any other software.


A reproducibility-related bug from early 2020 in the GNU GCC compiler as been fixed. The issues was that if GCC was invoked via the as frontend, the -ffile-prefix-map was being ignored. We were tracking this in Debian via the build_path_captured_in_assembly_objects issue. It has now been fixed and will be reflected in GCC version 13.


Holger Levsen will present at foss-north 2023 in April of this year in Gothenburg, Sweden on the topic of Reproducible Builds, the first ten years.


Anthony Andreoli, Anis Lounis, Mourad Debbabi and Aiman Hanna of the Security Research Centre at Concordia University, Montreal published a paper this month entitled On the prevalence of software supply chain attacks: Empirical study and investigative framework:

Software Supply Chain Attacks (SSCAs) typically compromise hosts through trusted but infected software. The intent of this paper is twofold: First, we present an empirical study of the most prominent software supply chain attacks and their characteristics. Second, we propose an investigative framework for identifying, expressing, and evaluating characteristic behaviours of newfound attacks for mitigation and future defense purposes. We hypothesize that these behaviours are statistically malicious, existed in the past, and thus could have been thwarted in modernity through their cementation x-years ago. […]


On our mailing list this month:

  • Mattia Rizzolo is asking everyone in the community to save the date for the 2023’s Reproducible Builds summit which will take place between October 31st and November 2nd at Dock Europe in Hamburg, Germany. Separate announcement(s) to follow. […]

  • ahojlm posted an message announcing a new project which is “the first project offering bootstrappable and verifiable builds without any binary seeds.” That is to say, a way of providing a verifiable path towards trusted software development platform without relying on pre-provided binary code in order to prevent against various forms of compiler backdoors. The project’s homepage is hosted on Tor (mirror).


The minutes and logs from our March 2023 IRC meeting have been published. In case you missed this one, our next IRC meeting will take place on Tuesday 25th April at 15:00 UTC on #reproducible-builds on the OFTC network.


… and as a Valentines Day present, Holger Levsen wrote on his blog on 14th February to express his thanks to OSUOSL for their continuous support of reproducible-builds.org. […]


Debian

Vagrant Cascadian developed an easier setup for testing debian packages which uses sbuild’s “unshare mode” along and reprotest, our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. […]


Over 30 reviews of Debian packages were added, 14 were updated and 7 were removed this month, all adding to our knowledge about identified issues. A number of issues were updated, including the Holger Levsen updating build_path_captured_in_assembly_objects to note that it has been fixed for GCC 13 […] and Vagrant Cascadian added new issues to mark packages where the build path is being captured via the Rust toolchain […] as well as new categorisation for where virtual packages have nondeterministic versioned dependencies […].

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

In addition, Vagrant Cascadian filed a bug with a patch to ensure GNU Modula-2 supports the SOURCE_DATE_EPOCH environment variable.

Testing framework

The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In March, the following changes were made by Holger Levsen:

  • Arch Linux-related changes:

    • Build Arch packages in /tmp/archlinux-ci/$SRCPACKAGE instead of /tmp/$SRCPACKAGE. […]
    • Start 2/3 of the builds on the o1 node, the rest on o2. […]
    • Add graphs for Arch Linux (and OpenWrt) builds. […]
    • Toggle Arch-related builders to debug why a specific node overloaded. […][…][…][…]

  • Node health checks:

    • Detect SetuptoolsDeprecationWarning tracebacks in Python builds. […]
    • Detect failures do perform chdist calls. […][…]

  • OSUOSL node migration.

    • Install megacli packages that are needed for hardware RAID. […][…]
    • Add health check and maintenance jobs for new nodes. […]
    • Add mail config for new nodes. […][…]
    • Handle a node running in the future correctly. […][…]
    • Migrate some nodes to Debian bookworm. […]
    • Fix nodes health overview for osuosl3. […]
    • Make sure the /srv/workspace directory is owned by by the jenkins user. […]
    • Use .debian.net names everywhere, except when communicating with the outside world. […]
    • Grant fpierret access to a new node. […]
    • Update documentation. […]
    • Misc migration changes. […][…][…][…][…][…][…][…]

  • Misc changes:

    • Enable fail2ban everywhere and monitor it with munin […].
    • Gracefully deal with non-existing Alpine schroots. […]

In addition, Roland Clobus is continuing his work on reproducible Debian ISO images:

  • Add/update openQA configuration […], and use the actual timestamp for openQA builds […].
  • Moved adding the user to the docker group from the janitor_setup_worker script to the (more general) update_jdn.sh script. […]
  • Use the (short-term) ‘reproducible’ source when generating live-build images. […]

diffoscope development

diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats as well. This month, Mattia Rizzolo released versions 238, and Chris Lamb released versions 239 and 240. Chris Lamb also made the following changes:

  • Fix compatibility with PyPDF 3.x, and correctly restore test data. […]
  • Rework PDF annotation handling into a separate method. […]

In addition, Holger Levsen performed a long-overdue overhaul of the Lintian overrides in the Debian packaging […][…][…][…], and Mattia Rizzolo updated the packaging to silence an include_package_data=True […], fixed the build under Debian bullseye […], fixed tool name in a list of tools permitted to be absent during package build tests […] and as well as documented sending out an email upon  […].

In addition, Vagrant Cascadian updated the version of GNU Guix to 238 [… and 239 […]. Vagrant also updated reprotest to version 0.7.23. […]

Other development work

Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE



If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports