Reproducible Builds in September 2021

The goal behind “reproducible builds” is to ensure that no deliberate flaws have been introduced during compilation processes via promising or mandating that identical results are always generated from a given source. This allowing multiple third-parties to come to an agreement on whether a build was compromised or not by a system of distributed consensus.

In these reports we outline the most important things that have been happening in the world of reproducible builds in the past month:

First mentioned in our March 2021 report, Martin Heinz published two blog posts on sigstore, a project that endeavours to offer software signing as a “public good, [the] software-signing equivalent to Let’s Encrypt”. The two posts, the first entitled Sigstore: A Solution to Software Supply Chain Security outlines more about the project and justifies its existence:

Software signing is not a new problem, so there must be some solution already, right? Yes, but signing software and maintaining keys is very difficult especially for non-security folks and UX of existing tools such as PGP leave much to be desired. That’s why we need something like sigstore - an easy to use software/toolset for signing software artifacts.

The second post (titled Signing Software The Easy Way with Sigstore and Cosign) goes into some technical details of getting started.

There was an interesting thread in the /r/Signal subreddit that started from the observation that Signal’s apk doesn’t match with the source code:

Some time ago I checked Signal’s reproducibility and it failed. I asked others to test in case I did something wrong, but nobody made any reports. Since then I tried to test the Google Play Store version of the apk against one I compiled myself, and that doesn’t match either.

BitcoinBinary.org was announced this month, which aims to be a “repository of Reproducible Build Proofs for Bitcoin Projects”:

Most users are not capable of building from source code themselves, but we can at least get them able enough to check signatures and shasums. When reputable people who can tell everyone they were able to reproduce the project’s build, others at least have a secondary source of validation.

Distribution work

Frédéric Pierret announced a new testing service at beta.tests.reproducible-builds.org, showing actual rebuilds of binaries distributed by both the Debian and Qubes distributions.

In Debian specifically, however, 51 reviews of Debian packages were added, 31 were updated and 31 were removed this month to our database of classified issues. As part of this, Chris Lamb refreshed a number of notes, including the build_path_in_record_file_generated_by_pybuild_flit_plugin issue.

Elsewhere in Debian, Roland Clobus posted his Fourth status update about reproducible live-build ISO images in Jenkins to our mailing list, which mentions (amongst other things) that:

• All major configurations are still built regularly using live-build and bullseye.
• All major configurations are reproducible now; Jenkins is green.
  • I’ve worked around the issue for the Cinnamon image.
  • The patch was accepted and released within a few hours.
• My main focus for the last month was on the live-build tool itself.
  • It will properly use the proxy for all HTTP traffic.

Related to this, there was continuing discussion on how to embed/encode the build metadata for the Debian “live” images which were being worked on by Roland Clobus.

Ariadne Conill published another detailed blog post related to various security initiatives within the Alpine Linux distribution. After summarising some conventional security work being done (eg. with sudo and the release of OpenSSH version 3.0), Ariadne included another section on reproducible builds: “The main blocker [was] determining what to do about storing the build metadata so that a build environment can be recreated precisely”.

Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

Community news

On our website this month, Bernhard M. Wiedemann fixed some broken links […] and Holger Levsen made a number of changes to the Who is Involved? page […][…][…]. On our mailing list, Magnus Ihse Bursie started a thread with the subject Reproducible builds on Java, which begins as follows:

I’m working for Oracle in the Build Group for OpenJDK which is primary responsible for creating a built artifact of the OpenJDK source code. […] For the last few years, we have worked on a low-effort, background-style project to make the build of OpenJDK itself building reproducible. We’ve come far, but there are still issues I’d like to address. […]

diffoscope

diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 183, 184 and 185 as well as performed significant triaging of merge requests and other issues in addition to making the following changes:

• New features:

  • Support a newer format version of the R language’s .rds files. […]
  • Update tests for OCaml 4.12. […]
  • Add a missing format_class import. […]

• Bug fixes:

  • Don’t call close_archive when garbage collecting Archive instances, unless open_archive definitely returned successfully. This prevents, for example, an AttributeError where PGPContainer’s cleanup routines were rightfully assuming that its temporary directory had actually been created. […]
  • Fix (and test) the comparison of R language’s .rdb files after refactoring temporary directory handling. […]
  • Ensure that “RPM archives” exists in the Debian package description, regardless of whether python3-rpm is installed or not at build time. […]

• Codebase improvements:

  • Use our assert_diff routine in tests/comparators/test_rdata.py. […]
  • Move diffoscope.versions to diffoscope.tests.utils.versions. […]
  • Reformat a number of modules with Black. […][…]

However, the following changes were also made:

• Mattia Rizzolo:

  • Fix an autopkgtest caused by the androguard module not being in the (expected) python3-androguard Debian package. […]
  • Appease a shellcheck warning in debian/tests/control.sh. […]
  • Ignore a warning from h5py in our tests that doesn’t concern us. […]
  • Drop a trailing .1 from the Standards-Version field as it’s required. […]

• Zbigniew Jędrzejewski-Szmek:

  • Stop using the deprecated distutils.spawn.find_executable utility. […][…][…][…][…]
  • Adjust an LLVM-related test for LLVM version 13. […]
  • Update invocations of llvm-objdump. […]
  • Adjust a test with a one-byte text file for file version 5.40. […]

And, finally, Benjamin Peterson added a --diff-context option to control unified diff context size […] and Jean-Romain Garnier fixed the Macho comparator for architectures other than x86-64 […].

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:

  • gtk4 (date-related issue)
  • build-compare (random tempfile problem)
  • itinerary (time-related build failure)

• Chris Lamb:

  • #993950 filed against lcalc.
  • #993952 filed against htscodecs.
  • #994123 filed against osdlyrics.
  • #994976 filed against xtermcontrol.
  • #994978 filed against rust-insta.
  • #994979 filed against python-tomli.
  • #995258 filed against python-pairix.
  • #995259 filed against python-pybedtools (forwarded upstream).

• Simon McVittie:

  • #993673 filed against clazy.
  • #993674 filed against ga.
  • #993675 filed against opensmtpd.
  • #993681 filed against sphinxcontrib-programoutput.

• Vagrant Cascadian:

  • #995092 filed against guile-3.0.
  • #995131 filed against nut.
  • #995143 filed against xdmf.
  • #995145 filed against cyrus-sasl2.
  • #995211 filed against libbrahe.
  • #995217 filed against maildirsync.
  • #995401 filed against canna.
  • #990246 filed against vlc (forwarded upstream […][…])

Testing framework

The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

• Holger Levsen:

  • Drop my package rebuilder prototype as it’s not useful anymore. […]
  • Schedule old packages in Debian bookworm. […]
  • Stop scheduling packages for Debian buster. […][…]
  • Don’t include PostgreSQL debug output in package lists. […]
  • Detect Python library mismatches during build in the node health check. […]
  • Update a note on updating the FreeBSD system. […]

• Mattia Rizzolo:

  • Silence a warning from Git. […]
  • Update a setting to reflect that Debian bookworm is the new testing. […]
  • Upgrade the PostgreSQL database to version 13. […]

• Roland Clobus (Debian “live” image generation):

  • Workaround non-reproducible config files in the libxml-sax-perl package. […]
  • Use the new DNS for the ‘snapshot’ service. […]

• Vagrant Cascadian:

  • Also note that the armhf architecture also systematically varies by the kernel. […]

Contributing

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.

• Twitter: @ReproBuilds

• Mailing list: rb-general@lists.reproducible-builds.org