Reproducible Builds in October 2022

← View all our monthly reports


Welcome to the Reproducible Builds report for October 2022! In these reports we attempt to outline the most important things that we have been up to over the past month.

As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.


Our in-person summit this year was held in the past few days in Venice, Italy. Activity and news from the summit will therefore be covered in next month’s report!

A new article related to reproducible builds was recently published in the 2023 IEEE Symposium on Security and Privacy. Titled Taxonomy of Attacks on Open-Source Software Supply Chains and authored by Piergiorgio Ladisa, Henrik Plate, Matias Martinez and Olivier Barais, their paper:

[…] proposes a general taxonomy for attacks on opensource supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution.

Taking the form of an attack tree, the paper covers 107 unique vectors linked to 94 real world supply-chain incidents which is then mapped to 33 mitigating safeguards including, of course, reproducible builds:

Reproducible Builds received a very high utility rating (5) from 10 participants (58.8%), but also a high-cost rating (4 or 5) from 12 (70.6%). One expert commented that a ”reproducible build like used by Solarwinds now, is a good measure against tampering with a single build system” and another claimed this ”is going to be the single, biggest barrier”.

It was noticed this month that Solarwinds published a whitepaper back in December 2021 in order to:

[…] illustrate a concerning new reality for the software industry and illuminates the increasingly sophisticated threats made by outside nation-states to the supply chains and infrastructure on which we all rely.

The 12-month anniversary of the 2020 “Solarwinds attack” (which SolarWinds Worldwide LLC itself calls the “SUNBURST” attack) was, of course, the likely impetus for publication.

Whilst collaborating on making the Cyrus IMAP server reproducible, Ellie Timoney asked why the Reproducible Builds testing framework uses two remarkably distinctive build paths when attempting to flush out builds that vary on the absolute system path in which they were built. In the case of the Cyrus IMAP server, these happened to be:

  • /build/1st/cyrus-imapd-3.6.0~beta3/
  • /build/2/cyrus-imapd-3.6.0~beta3/2nd/

Asked why they vary in three different ways, Chris Lamb listed in detail the motivation behind to each difference.

On our mailing list this month:

The Reproducible Builds project is delighted to welcome openEuler to the Involved projects page […]. openEuler is Linux distribution developed by Huawei, a counterpart to it’s more commercially-oriented EulerOS.

Debian

Colin Watson wrote about his experience towards making the databases generated by the man-db UNIX manual page indexing tool:

One of the people working on [reproducible builds] noticed that man-db’s database files were an obstacle to [reproducibility]: in particular, the exact contents of the database seemed to depend on the order in which files were scanned when building it. The reporter proposed solving this by processing files in sorted order, but I wasn’t keen on that approach: firstly because it would mean we could no longer process files in an order that makes it more efficient to read them all from disk (still valuable on rotational disks), but mostly because the differences seemed to point to other bugs.

Colin goes on to describe his approach to solving the problem, including fixing various fits of internal caching, and he ends his post with “None of this is particularly glamorous work, but it paid off”.

Vagrant Cascadian announced on our mailing list another online sprint to help “clear the huge backlog of reproducible builds patches submitted” by performing NMUs (Non-Maintainer Uploads). The first such sprint took place on September 22nd, but another was held on October 6th, and another small one on October 20th. This resulted in the following progress:

41 reviews of Debian packages were added, 62 were updated and 12 were removed this month adding to our knowledge about identified issues. A number of issue types were updated too. [1][…]

Lastly, Luca Boccassi submitted a patch to debhelper, a set of tools used in the packaging of the majority of Debian packages. The patch addressed an issue in the dh_installsysusers utility so that the postinst post-installation script that debhelper generates the same data regardless of the underlying filesystem ordering.

Other distributions

F-Droid is a community-run app store that provides free software applications for Android phones. This month, F-Droid changed their documentation and guidance to now explicitly encourage RB for new apps […][…], and FC Stegerman created an extremely in-depth issue on GitLab concerning the APK signing block. You can read more about F-Droid’s approach to reproducibility in our July 2022 interview with Hans-Christoph Steiner of the F-Droid Project.

In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

diffoscope

diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 224 and 225 to Debian:

  • Add support for comparing the text content of HTML files using html2text. […]
  • Add support for detecting ordering-only differences in XML files. […]
  • Fix an issue with detecting ordering differences. […]
  • Use the capitalised version of “Ordering” consistently everywhere in output. […]
  • Add support for displaying font metadata using ttx(1) from the fonttools suite. […]

  • Testsuite improvements:

    • Temporarily allow the stable-po pipeline to fail in the CI. […]
    • Rename the order1.diff test fixture to json_expected_ordering_diff. […]
    • Tidy the JSON tests. […]
    • Use assert_diff over get_data and an manual assert within the XML tests. […]
    • Drop the ALLOWED_TEST_FILES test; it was mostly just annoying. […]
    • Tidy the tests/test_source.py file. […]

Chris Lamb also added a link to diffoscope’s OpenBSD packaging on the diffoscope.org homepage […] and Mattia Rizzolo fix an test failure that was occurring under with LLVM 15 […].

Testing framework

The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:

  • Run the logparse tool to analyse results on the Debian Edu build logs. […]
  • Install btop(1) on all nodes running Debian. […]
  • Switch Arch Linux from using SHA1 to SHA256. […]
  • When checking Debian debstrap jobs, correctly log the tool usage. […]
  • Cleanup more task-related temporary directory names when testing Debian packages. […][…]
  • Use the cdebootstrap-static binary for the 2nd runs of the cdebootstrap tests. […]
  • Drop a workaround when testing OpenWrt and coreboot as the issue in diffoscope has now been fixed. […]
  • Turn on an rm(1) warning into an “info”-level message. […]
  • Special case the osuosl168 node for running Debian bookworm already. […][…]
  • Use the new non-free-firmware suite on the o168 node. […]

In addition, Mattia Rizzolo made the following changes:

  • Ensure that 2nd build has a merged /usr. […]
  • Only reconfigure the usrmerge package on Debian bookworm and above. […]
  • Fix bc(1) syntax in the computation of the percentage of unreproducible packages in the dashboard. […][…][…]
  • In the index_suite_ pages, order the package status to be the same order of the menu. […]
  • Pass the --distribution parameter to the pbuilder utility. […]

Finally, Roland Clobus continued his work on testing Live Debian images. In particular, he extended the maintenance script to warn when workspace directories cannot be deleted. […]


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports

Follow us on Twitter @ReproBuilds, Mastodon @reproducible_builds@fosstodon.org & Reddit and please consider making a donation. • Content licensed under CC BY-SA 4.0, style licensed under MIT. Templates and styles based on the Tor Styleguide. Logos and trademarks belong to their respective owners. • Patches for this website welcome via our Git repository (instructions) or via our mailing list. • Full contact info