Welcome to the Reproducible Builds report for October 2022! In these reports we attempt to outline the most important things that we have been up to over the past month.
As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
Our in-person summit this year was held in the past few days in Venice, Italy. Activity and news from the summit will therefore be covered in next month’s report!
A new article related to reproducible builds was recently published in the 2023 IEEE Symposium on Security and Privacy. Titled Taxonomy of Attacks on Open-Source Software Supply Chains and authored by Piergiorgio Ladisa, Henrik Plate, Matias Martinez and Olivier Barais, their paper:
[…] proposes a general taxonomy for attacks on opensource supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution.
Taking the form of an attack tree, the paper covers 107 unique vectors linked to 94 real world supply-chain incidents which is then mapped to 33 mitigating safeguards including, of course, reproducible builds:
Reproducible Builds received a very high utility rating (5) from 10 participants (58.8%), but also a high-cost rating (4 or 5) from 12 (70.6%). One expert commented that a ”reproducible build like used by Solarwinds now, is a good measure against tampering with a single build system” and another claimed this ”is going to be the single, biggest barrier”.
It was noticed this month that Solarwinds published a whitepaper back in December 2021 in order to:
[…] illustrate a concerning new reality for the software industry and illuminates the increasingly sophisticated threats made by outside nation-states to the supply chains and infrastructure on which we all rely.
The 12-month anniversary of the 2020 “Solarwinds attack” (which SolarWinds Worldwide LLC itself calls the “SUNBURST” attack) was, of course, the likely impetus for publication.
Whilst collaborating on making the Cyrus IMAP server reproducible, Ellie Timoney asked why the Reproducible Builds testing framework uses two remarkably distinctive build paths when attempting to flush out builds that vary on the absolute system path in which they were built. In the case of the Cyrus IMAP server, these happened to be:
Asked why they vary in three different ways, Chris Lamb listed in detail the motivation behind to each difference.
On our mailing list this month:
Daniel Garcia from WalletScrutiny.com started a thread asking for input on buttons with the Reproducible Builds logo, requesting design suggestions or other feedback. […]
Arch Linux contributor kpcyrd wrote to our list this month with the news that “multiple people in Arch Linux noticed the output of our
git archivecommand doesn’t match the tarball served by GitHub anymore. In his post, kpcyrd narrows the change to a specific commit in Git. […]
Akihiro Suda wrote to a share a new tool called
repro-get. According to Akihiro’s post, “repro-get is a tool to install a specific snapshot of apt/dnf/apk/pacman packages using SHA256SUMS files”. This is needed in order to install specific (or “pinned”) dependencies needed to validate a build.
Finally, Janneke Nieuwenhuizen announced the release of GNU Mes 0.24.1, which represents 23 commits over five months by four people. GNU Mes is a Scheme interpreter and C compiler for bootstrapping the GNU System. […]
The Reproducible Builds project is delighted to welcome openEuler to the Involved projects page […]. openEuler is Linux distribution developed by Huawei, a counterpart to it’s more commercially-oriented EulerOS.
Colin Watson wrote about his experience towards making the databases generated by the
man-db UNIX manual page indexing tool:
One of the people working on [reproducible builds] noticed that man-db’s database files were an obstacle to [reproducibility]: in particular, the exact contents of the database seemed to depend on the order in which files were scanned when building it. The reporter proposed solving this by processing files in sorted order, but I wasn’t keen on that approach: firstly because it would mean we could no longer process files in an order that makes it more efficient to read them all from disk (still valuable on rotational disks), but mostly because the differences seemed to point to other bugs.
Colin goes on to describe his approach to solving the problem, including fixing various fits of internal caching, and he ends his post with “None of this is particularly glamorous work, but it paid off”.
Vagrant Cascadian announced on our mailing list another online sprint to help “clear the huge backlog of reproducible builds patches submitted” by performing NMUs (Non-Maintainer Uploads). The first such sprint took place on September 22nd, but another was held on October 6th, and another small one on October 20th. This resulted in the following progress:
ascii2binary(Fixed #1020812, #998758 & #1007421)
bibclean(Fixed #829754 & #929036)
leave(Fixed #777403, #967002 & #999259)
mailto(Fixed #998978 & #777413)
remote-tty(Fixed #829721 & #977280)
xcolmix(Fixed #1020748, #999219 & #988018)
z80asm(Fixed #939775 & #1020875)
elvis-tiny(Fixed #829755 & #901345)
hannah(Fixed #845782 & #901260)
mod-dnssd(Submitted alternate fix for #828752)
snake4(Fixed #829715 & #913734)
zephyr(Investigated #828867 & #1021374)
checkpw(Fixed #777299 & #1020887)
41 reviews of Debian packages were added, 62 were updated and 12 were removed this month adding to our knowledge about identified issues. A number of issue types were updated too. […]
Lastly, Luca Boccassi submitted a patch to
debhelper, a set of tools used in the packaging of the majority of Debian packages. The patch addressed an issue in the
dh_installsysusers utility so that the
postinst post-installation script that
debhelper generates the same data regardless of the underlying filesystem ordering.
F-Droid is a community-run app store that provides free software applications for Android phones. This month, F-Droid changed their documentation and guidance to now explicitly encourage RB for new apps […][…], and FC Stegerman created an extremely in-depth issue on GitLab concerning the APK signing block. You can read more about F-Droid’s approach to reproducibility in our July 2022 interview with Hans-Christoph Steiner of the F-Droid Project.
In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Bernhard M. Wiedemann:
fastjet-contrib(sort nondeterminstic filesystem ordering)
forge(Sphinx “doctree” issue)
gau2grid(output varies with
tcl(fails to build in 2038)
vectorscan(output varies with
xz2/lzma(Rust-related filesystem ordering)
- #891263 filed against
puppetback in early 2018 was finally merged into Puppet and was released in Puppet 7.20.0.
- #1021198 filed against
- #1022777 filed against
- #891263 filed against
- #1021331 filed against
- #1021373 filed against
- #1021374 filed against
- #1021452 filed against
- #1021454 filed against
- #1021456 filed against
- #1021457 filed against
- #1021458 filed against
- #1021461 filed against
- #1021463 filed against
- #1021464 filed against
- #1021466 filed against
- #1021469 filed against
- #1021470 filed against
- #1021471 filed against
- #1021472 filed against
- #1021473 filed against
- #1021498 filed against
- #1021509 filed against
- #1021512 filed against
- #1021513 filed against
- #1021514 and #1021516 filed against
- #1021518 filed against
- #1021520 filed against
- #1021521 and #1021522 filed against
- #1021751 filed against
- #1021789 filed against
- #1021792 and #1021793 filed against
- #1021799 and #1021800 filed against
- #1021860 filed against
- #1021893 filed against
- #1022130 filed against
- #1021331 filed against
diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions
225 to Debian:
- Add support for comparing the text content of HTML files using
- Add support for detecting ordering-only differences in XML files. […]
- Fix an issue with detecting ordering differences. […]
- Use the capitalised version of “Ordering” consistently everywhere in output. […]
- Add support for displaying font metadata using
ttx(1)from the fonttools suite. […]
- Temporarily allow the
stable-popipeline to fail in the CI. […]
- Rename the
order1.difftest fixture to
- Tidy the JSON tests. […]
get_dataand an manual assert within the XML tests. […]
- Drop the
ALLOWED_TEST_FILEStest; it was mostly just annoying. […]
- Tidy the
- Temporarily allow the
Chris Lamb also added a link to diffoscope’s OpenBSD packaging on the diffoscope.org homepage […] and Mattia Rizzolo fix an test failure that was occurring under with LLVM 15 […].
The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:
- Run the
logparsetool to analyse results on the Debian Edu build logs. […]
btop(1)on all nodes running Debian. […]
- Switch Arch Linux from using SHA1 to SHA256. […]
- When checking Debian
debstrapjobs, correctly log the tool usage. […]
- Cleanup more task-related temporary directory names when testing Debian packages. […][…]
- Use the
cdebootstrap-staticbinary for the 2nd runs of the
- Drop a workaround when testing OpenWrt and coreboot as the issue in diffoscope has now been fixed. […]
- Turn on an
rm(1)warning into an “info”-level message. […]
- Special case the
osuosl168node for running Debian bookworm already. […][…]
- Use the new
non-free-firmwaresuite on the
In addition, Mattia Rizzolo made the following changes:
- Ensure that 2nd build has a merged
- Only reconfigure the
usrmergepackage on Debian bookworm and above. […]
bc(1)syntax in the computation of the percentage of unreproducible packages in the dashboard. […][…][…]
- In the
index_suite_pages, order the package status to be the same order of the menu. […]
- Pass the
--distributionparameter to the
Finally, Roland Clobus continued his work on testing Live Debian images. In particular, he extended the maintenance script to warn when workspace directories cannot be deleted. […]
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via: