Welcome to the August 2022 report from the Reproducible Builds project! In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.
As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
As announced last month, registration is currently open for our in-person summit this year which is due to be held between November 1st → November 3rd. The event will take place in Venice (Italy). Very soon we intend to pick a venue reachable via the train station and an international airport. However, the precise venue will depend on the number of attendees. Please see the announcement email for information about how to register.
The US National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA) and the Office of the Director of National Intelligence (ODNI) have released a document called “Securing the Software Supply Chain: Recommended Practices Guide for Developers” (PDF) as part of their Enduring Security Framework (ESF) work.
The document expressly recommends having reproducible builds as part of “advanced” recommended mitigations, along with hermetic builds. Page 31 (page 35 in the PDF) says:
Reproducible builds provide additional protection and validation against attempts to compromise build systems. They ensure the binary products of each build system match: i.e., they are built from the same source, regardless of variable metadata such as the order of input files, timestamps, locales, and paths. Reproducible builds are those where re-running the build steps with identical input artifacts results in bit-for-bit identical output. Builds that cannot meet this must provide a justification why the build cannot be made reproducible.
The full press release is available online.
On our mailing list this month, Marc Prud’hommeaux posted a feature request for diffoscope which additionally outlines a project called The App Fair, an autonomous distribution network of free and open-source macOS and iOS applications, where “validated apps are then signed and submitted for publication”.
Author/blogger Cory Doctorow posted published a provocative blog post this month titled “Your computer is tormented by a wicked god”. Touching on Ken Thompson’s famous talk, “Reflections on Trusting Trust”, the early goals of “Secure Computing” and UEFI firmware interfaces:
This is the core of a two-decade-old debate among security people, and it’s one that the “benevolent God” faction has consistently had the upper hand in. They’re the “curated computing” advocates who insist that preventing you from choosing an alternative app store or side-loading a program is for your own good – because if it’s possible for you to override the manufacturer’s wishes, then malicious software may impersonate you to do so, or you might be tricked into doing so. [..] This benevolent dictatorship model only works so long as the dictator is both perfectly benevolent and perfectly competent. We know the dictators aren’t always benevolent. […] But even if you trust a dictator’s benevolence, you can’t trust in their perfection. Everyone makes mistakes. Benevolent dictator computing works well, but fails badly. Designing a computer that intentionally can’t be fully controlled by its owner is a nightmare, because that is a computer that, once compromised, can attack its owner with impunity.
In Debian this month, the
required package sets became 100% reproducible in Debian bookworm on the
arm64 architectures. These two subsets of the full Debian archive refer to Debian package “priority” levels as described in the §2.5 Priorities section of the Debian Policy — there is no canonical “minimal installation” package set in Debian due to its diverse methods of installation.
As it happens, these package sets are not reproducible on the
i386 architecture because the
ncurses package on that architecture is not yet reproducible, and the
sed package currently fails to build from source on
armhf too. The full list of reproducible packages within these package sets can be viewed within our QA system, such as on the page of
required packages in
amd64 and the list of
essential packages on
arm64, both for Debian bullseye.
It recently has become very easy to install reproducible Debian Docker containers using
podman on Debian bullseye:
$ sudo apt install podman $ podman run --rm -it debian:bullseye bash
Related to this, it has also become straightforward to reproducibly bootstrap Debian using mmdebstrap, a replacement for the usual debootstrap tool to create Debian root filesystems:
$ SOURCE_DATE_EPOCH=$(date --utc --date=2022-08-29 +%s) mmdebstrap unstable > unstable.tar
Work has also taken place to ensure that the canonical debootstrap and cdebootstrap tools are also capable of bootstrapping Debian reproducibly, although it currently requires a few extra steps:
“Clamping” the modification time of files that are newer than
$SOURCE_DATE_EPOCHto be not greater than
Deleting a few files. For debootstrap, this requires the deletion of
/var/log/bootstrap.log, and for cdebootstrap we also need to delete the
/var/log/apt/term.logfiles as well.
This process works at least for unstable, bullseye and bookworm and is now being tested automatically by a number of QA jobs setup by Holger Levsen […][…][…][…][…][…]. As part of this work, Holger filed two bugs to request a better initialisation of the
/etc/machine-id file in both debootstrap […] and cdebootstrap […].
Elsewhere in Debian, 131 reviews of Debian packages were added, 20 were updated and 27 were removed this month, adding to our extensive knowledge about identified issues. Chris Lamb added a number of issue types, including:
nondeterministic_ids_in_html_output_generated_by_python_sphinx_panels […]. Lastly, Mattia Rizzolo removed the
deterministic flag from the
captures_kernel_variant flag […].
Vagrant Cascadian posted an update of the status of Reproducible Builds in GNU Guix, writing that:
Ignoring the pesky unknown packages, it is more like ~93% reproducible
and ~7% unreproducible... that feels a bit better to me!
These numbers wander around over time, mostly due to packages moving
back into an "unknown" state while the build farms catch up with each
other... although the above numbers seem to have been pretty consistent
over the last few days.
The post itself contains a lot more details, including a brief discussion of tooling.
Elsewhere in GNU Guix, however, Vagrant updated a number of packages such as
directfb […] and
mm-common […], as well as updated the version of reprotest to 0.7.21 […].
In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.
diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions
221 to Debian, as well as made the following changes:
external_tools.pyto reflect changes to
- Depend on the dedicated
xxdpackage now, not the
- Don’t crash if we can open a PDF file using the PyPDF library, but cannot subsequently parse the annotations within. […]
The Reproducible Builds project aims to fix as many currently-unreproducible packages as possible as well as to send all of our patches upstream wherever appropriate. This month we created a number of patches, including:
Bernhard M. Wiedemann:
at-spi-sharp(build failure when build on a multiprocessor machine).
borgbackup(fails to build in 2038, fix)
memcached(fails to build in 2038)
nim(fails to build in 2038)
perl-Time-Moment(fails to build in 2038)
python-bson(fails to build in 2038)
python-heatclient(fails to build in 2038)
python3.8(fails to build in 2038)
The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, Holger Levsen made the following changes:
- Temporarily add Debian unstable
deb-srclines to enable test builds a Non-maintainer Upload (NMU) campaign targeting 708 sources without
.buildinfofiles found in Debian unstable, including 475 in bookworm. […][…]
- Correctly deal with the Debian Edu packages not being installable. […]
- Finally, stop scheduling stretch. […]
- Make sure all Ubuntu nodes have the
linux-image-generickernel package installed. […]
- Temporarily add Debian unstable
Health checks & view:
- Add a new job to test reproducibility of mmdebstrap bootstrapping tool. […][…][…][…]
- Run our new mmdebstrap job remotely […][…]
- Improve the output of the mmdebstrap job. […][…][…]
- Adjust the mmdebstrap script to additionally support debootstrap as well. […][…][…]
- Work around mmdebstrap and debootstrap keeping logfiles within their artifacts. […][…][…]
- Add support for testing cdebootstrap too and add such a job for unstable. […][…][…]
- Use a reproducible value for
SOURCE_DATE_EPOCHfor all our new bootstrap jobs. […]
- Send the
- Send the
In addition, Roland Clobus re-enabled the tests for live-build images […] and added a feature where the build would retry instead of give up when the archive was synced whilst building an ISO […], and Vagrant Cascadian added logging to report the current target of the
/bin/sh symlink […].
As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via: