Welcome to the June 2019 report from the Reproducible Builds project! In our reports we outline the most important things that we have been up to over the past month.
In order that everyone knows what this is about, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.
In June’s report, we will cover:
- Media coverage — Lego bricks, pizza and… Reproducible Builds‽
- Upstream news — Is Trusting Trust close to a ‘rebuttal’?
- Events — What happened at MiniDebConf Hamburg, the OpenWrt Summit, etc.
- Software development — Patches patches patches, etc.
- Misc news — From our mailing list…
- Getting in touch… and how to contribute.
- The Prototype Fund, an initiative to “aid software developers, hackers and creatives in furthering their ideas from concept to demo” produced a video featuring Holger Levsen explaining Reproducible Builds… using Lego bricks and pizza!
- Joseph Devietti from Cloudseal published a post titled An introduction to reproducible builds on their blog. It gives a brief overview of the problem and what we are trying to solve, additionally noting the practical point that:
One key motivation for reproducible builds is to enable peak efficiency for the build caches used in modern build systems.
- Carl Dong gave a presentation entitled Bitcoin Build System Security at the Breaking Bitcoin conference in Amsterdam, Netherlands.
- The Monero cryptocurrency now offers full reproducibility for all compiled binaries, a feature first requested in October 2017.
- The Fedora project debated setting the
SOURCE_DATE_EPOCHenvironment variable in all builds via
rpm, an idea that was accepted and merged on the 27th by Igor Gnatenko.
- Jeremiah Orians announced that version 1.0 of the
mescc-tools-seedcompiler has been released. For those not familiar with the project, it is the full bootstrap of a cross-platform compiler for the C programming language (written in C itself) from hex, the ultimate goal being able to demonstrate fully-bootstrapped compiler from hex to the GCC GNU Compiler Collection. This has many implications in and around Ken Thompson’s Trusting Trust attack he outlined in his 1983 Turing Award Lecture.
There were a number of events that included or incorporated members of the Reproducible Builds community this month. If you know of any others, please do get in touch. In addition, a number of members of the Reproducible Builds project will be at DebConf 2019 in Curitiba, Brazil and will present on the status of their work.
MiniDebConf Hamburg 2019
Holger Levsen, Jelle van der Waa, kpcyrd and Alexander Couzens attended MiniDebConf Hamburg 2019 and worked on Reproducible Builds. As part of this, Holger gave a status update on the Project with a talk entitled Reproducible Builds aiming for bullseye, referring to the next Debian release name:
Jelle van der Waa kindly gifted Holger with a Reproducible Builds display:
In addition, Lukas Puehringer gave a talk titled Building reproducible builds into apt with in-toto:
As part of various hacking sessions:
Jelle van der Waa:
- Improved the
reproducible_json.pyscript to generate distribution-specific JSON, leading to the availability of an ArchLinux JSON file.
- Investigated why the Arch Linux kernel package is not reproducible, finding out that
KGBUILD_BUILD_TIMESTAMPshould be set. The enabling of
CONFIG_MODULE_SIG_ALLcauses the kernel modules to be signed with a (non-deterministic) build-time key if none is provided, leading to unreproducibility.
- keyutils was fixed with respect to it embedding the build date in its binary. […]
- nspr was made reproducible in Arch Linux. […]
- Improved the
- Alexander Couzens:
Holger Levsen was on-hand to review and merge all the above commits, providing support and insight into the codebase. He additionally split out a
README.development from the regular, more-generic
Here, Holger participated in the discussions regarding
.buildinfo build-attestation documents. As a result of this, Paul Spooren (aparcar) made a pull request to introduce/create a
feeds.buildinfo (etc) for reproducibility in OpenWrt.
Chris Lamb spent significant time working on
buildinfo.debian.net, his experiment into how to process, store and distribute
.buildinfo files after the Debian archive software has processed them. This included:
Started making the move to Python 3.x (and Django 2.x) […][…][…][…][…][…][…] additionally performing a large number of adjacent cleanups including dropping the authentication framework […], fixing a number of flake8 warnings […], adding a
setup.cfgto silence some warnings […], moving to
%-style interpolation and
u"Unicode"strings […], etc.
Added a number of (as-yet unreleased…) features, including caching the expensive landing page queries. […]
Took the opportunity to start migrating the hosting from its current GitHub home to a more-centralised repository on salsa.debian.org, moving from the Travis to the GitLab continuous integration platform, updating the URL to the source in the footer […] and many other related changes […].
There was a significant amount of effort on our website this month.
Moved the remaining site to the newer website design. This was a long-outstanding task (#2) and required a huge number of changes, including moving all the event and documentation pages to the new design […] and migrating/merging the old
_layouts/page.htmlinto the new design […] too. This could then allow for many cleanups including moving/deleting files into cleaner directories, dropping a bunch of example layouts […] and dropping the old “home” layout. […]
Added reports to the homepage. (#16)
Re-ordered and merged various top-level sections of the site to make the page easier to parse/navigate […][…] and updated the documentation for
SOURCE_DATE_EPOCHto clarify that the alternative
date(1)is for compatibility with BSD variants of UNIX […].
Thomas Vincent added a huge number of videos and slides to the Resources page […][…][…][…][…][…] etc. as well as added a button to link to subtitles […] and fixing a bug when displaying metadata links […].
- Alexander Couzens (OpenWrt):
- Holger Levsen:
- Jelle van der Waa:
- Change Arch Linux and Alpine
- Add a Jenkins job to generate Arch Linux HTML pages. […]
- Fix the Arch Linux suites in the
- Add an Arch JSON export Jenkins job. […]
- Create per-distribution reproducible JSON files. […]
- Change Arch Linux and Alpine
- Start adding an Alpine theme. […]
- Add an Alpine website. […][…][…][…]
KGBchat bot. […]
- Use the
apkversion instead of
- Install/configure various parts of the chroot including passing in Git options […], adding the
abuildgroup onto more servers […][…], installing GnuPG […]
- Build packages using its own scheduler. […] […][…]
- Misc maintenance and fixups. […][…]
- Mattia Rizzolo:
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
- Bernhard M. Wiedemann:
- argus-client (Parallelism race condition and silent build failure)
- benchmark (Version upgrade, fixing failing to build from FTBFS with
- ck (FTBFS with
-j1, also fixed upstream)
- herbstluftwm (Embedded build date)
- HSAIL-Tools (Sort Perl hash)
- lcov (A file’s modification time was being updated by sed)
- linphone (Sort Python
readdir(3), submitted but ignored upstream)
- mypaint (Sort call to
readdir(3), probably upstream)
- MozillaFirefox+Thunderbird (Report parallelism race condition)
- ndpi (Fix a date, already merged upstream)
- oyranos (Drop build kernel version, already merged upstream)
- perl-Alien-SDL (Sort Perl
readdir(3)`, orphaned upstream)
- perl-Email-Date-Format (Fix a rare breakage)
- perl-OLE-Storage_Lite (Fix FTBFS when built in 2020, ignored upstream)
- plowshare (Date, already upstream)
- python-hyper (Avoid build getting stuck with
-j1, not upstream)
- python-nautilus (Python date)
- python-pgmagick (Sort
readdir(3)call, already merged upstream)
- python-qt5 (Sort a Python
os.walk, submitted upstream)
- python-xmlsec (Sort a Python glob call, already merged upstream)
- rakkess (Fix a date and time call in
- rclone/cobra (sort Go programming language hash)
- rubygem-rice (Drop unreproducible files)
- sawfish (Version update to get all upstream reproducibility fixes)
- surfraw (Fix a date, already upstream)
- terraform (Report FTBFS when built in 2030)
- thunarx-python (Fix a date, not yet upstream)
- uperf (Date, already upstream)
- vboot (Uses a shell date, not yet upstream)
- wine (Report the use of random file names)
- Chris Lamb:
- Morten Linderud submitted a patch for libpod, a library used to create container pods to fix a date-related reproducibility issue which has subsequently been merged.
- Richard Biener submitted a patch for the GCC GNU Compiler Collection to fix differences in the runtime debugging info between builds in its D programming language support.
Chris Lamb also did more work testing of the reproducibility status of Debian Installer images. In particular, he was working around and patching an issue stemming from us testing builds far into the “future”. (#926242)
In addition, following discussions at MiniDebConf Hamburg, Ivo De Decker reviewed the situation around Debian bug #869184 again (“dpkg: source uploads including
_amd64.buildinfo cause problems”) and updated the bug with some recommendations for the next Debian release cycle.
In diffoscope (our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues) Chris Lamb documented that
run_diffoscope should not be considered a stable API […] and adjusted the configuration to build our Docker image from the current Git checkout, not the Debian archive […]
On our mailing list this month Lars Wirzenius continued conversation regarding various questions about reproducible builds and their bearing on building a distributed continuous integration system which received many replies (thread index for May & June). In addition, Sebastian Huber asked whether anyone has attempted a reproducible build of a GCC compiler itself.
If you are interested in contributing the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
This month’s report was written by Alexander Borkowski, Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, heinrich5991, Holger Levsen, Jelle van der Waa, kpcyrd & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.