Welcome to the June 2019 report from the Reproducible Builds project! In our reports we outline the most important things that we have been up to over the past month.
In order that everyone knows what this is about, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.
In June’s report, we will cover:
- Media coverage — Lego bricks, pizza and… Reproducible Builds‽
- Upstream news — Is Trusting Trust close to a ‘rebuttal’?
- Events — What happened at MiniDebConf Hamburg, the OpenWrt Summit, etc.
- Software development — Patches patches patches, etc.
- Misc news — From our mailing list…
- Getting in touch… and how to contribute.
Media coverage
- The Prototype Fund, an initiative to “aid software developers, hackers and creatives in furthering their ideas from concept to demo” produced a video featuring Holger Levsen explaining Reproducible Builds… using Lego bricks and pizza!
- Joseph Devietti from Cloudseal published a post titled An introduction to reproducible builds on their blog. It gives a brief overview of the problem and what we are trying to solve, additionally noting the practical point that:
One key motivation for reproducible builds is to enable peak efficiency for the build caches used in modern build systems.
- Carl Dong gave a presentation entitled Bitcoin Build System Security at the Breaking Bitcoin conference in Amsterdam, Netherlands.
Upstream news
- The Monero cryptocurrency now offers full reproducibility for all compiled binaries, a feature first requested in October 2017.
- The Fedora project debated setting the
SOURCE_DATE_EPOCHenvironment variable in all builds viarpm, an idea that was accepted and merged on the 27th by Igor Gnatenko.
- Jeremiah Orians announced that version 1.0 of the
mescc-tools-seedcompiler has been released. For those not familiar with the project, it is the full bootstrap of a cross-platform compiler for the C programming language (written in C itself) from hex, the ultimate goal being able to demonstrate fully-bootstrapped compiler from hex to the GCC GNU Compiler Collection. This has many implications in and around Ken Thompson’s Trusting Trust attack he outlined in his 1983 Turing Award Lecture.
Events
There were a number of events that included or incorporated members of the Reproducible Builds community this month. If you know of any others, please do get in touch. In addition, a number of members of the Reproducible Builds project will be at DebConf 2019 in Curitiba, Brazil and will present on the status of their work.
MiniDebConf Hamburg 2019
Holger Levsen, Jelle van der Waa, kpcyrd and Alexander Couzens attended MiniDebConf Hamburg 2019 and worked on Reproducible Builds. As part of this, Holger gave a status update on the Project with a talk entitled Reproducible Builds aiming for bullseye, referring to the next Debian release name:
Jelle van der Waa kindly gifted Holger with a Reproducible Builds display:
In addition, Lukas Puehringer gave a talk titled Building reproducible builds into apt with in-toto:
As part of various hacking sessions:
-
Jelle van der Waa:
- Improved the
reproducible_json.pyscript to generate distribution-specific JSON, leading to the availability of an ArchLinux JSON file. - Investigated why the Arch Linux kernel package is not reproducible, finding out that
KBUILD_BUILD_HOSTandKGBUILD_BUILD_TIMESTAMPshould be set. The enabling ofCONFIG_MODULE_SIG_ALLcauses the kernel modules to be signed with a (non-deterministic) build-time key if none is provided, leading to unreproducibility. - keyutils was fixed with respect to it embedding the build date in its binary. […]
- nspr was made reproducible in Arch Linux. […]
- Improved the
- kpcyrd:
- Created various Jenkins jobs to generate Alpine build chroots, schedule new packages and to ultimately build them. […][…][…]
- Created an Alpine reproducible testing overview page.
- Provided a proof of concept
SOURCE_DATE_EPOCHpatch forabuildto fix timestamp issues in Alpine packages. […]
- Alexander Couzens:
- Rewrote the database interaction routines for OpenWrt.
- Migrated the OpenWrt package parser to use Python 3.x as Python 2.x will be reaching end-of-life at the end of this year.
- Setup a test environment using a new
README.developmentfile.
Holger Levsen was on-hand to review and merge all the above commits, providing support and insight into the codebase. He additionally split out a README.development from the regular, more-generic README file.
OpenWrt summit
The OpenWrt project is a Linux operating system targeting embedded devices, particularly wireless network routers. In June, they hosted a summit that took place from 10th to 12th of the month.
Here, Holger participated in the discussions regarding .buildinfo build-attestation documents. As a result of this, Paul Spooren (aparcar) made a pull request to introduce/create a feeds.buildinfo (etc) for reproducibility in OpenWrt.
Software development
buildinfo.debian.net
Chris Lamb spent significant time working on buildinfo.debian.net, his experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them. This included:
-
Started making the move to Python 3.x (and Django 2.x) […][…][…][…][…][…][…] additionally performing a large number of adjacent cleanups including dropping the authentication framework […], fixing a number of flake8 warnings […], adding a
setup.cfgto silence some warnings […], moving to__str__andstr.format(...)over%-style interpolation andu"Unicode"strings […], etc. -
Added a number of (as-yet unreleased…) features, including caching the expensive landing page queries. […]
-
Took the opportunity to start migrating the hosting from its current GitHub home to a more-centralised repository on salsa.debian.org, moving from the Travis to the GitLab continuous integration platform, updating the URL to the source in the footer […] and many other related changes […].
-
Applied the Black “uncompromising code formatter” to the codebase. […]
Project website
There was a significant amount of effort on our website this month.
-
Chris Lamb:
-
Moved the remaining site to the newer website design. This was a long-outstanding task (#2) and required a huge number of changes, including moving all the event and documentation pages to the new design […] and migrating/merging the old
_layouts/page.htmlinto the new design […] too. This could then allow for many cleanups including moving/deleting files into cleaner directories, dropping a bunch of example layouts […] and dropping the old “home” layout. […] -
Added reports to the homepage. (#16)
-
Re-ordered and merged various top-level sections of the site to make the page easier to parse/navigate […][…] and updated the documentation for
SOURCE_DATE_EPOCHto clarify that the alternative-rcall todate(1)is for compatibility with BSD variants of UNIX […]. -
Made a large number of visual fixups, particularly to accommodate the principles of responsive web design. […][…][…][…][…]
-
Updated the lint functionality of the build system to check for URIs that are not using
/foo/-style relative URLs. […]
-
-
Jelle van der Waa updated the Events page to correct invalid Markdown […] and fixed a typo of “distribution” on a previous event page […].
-
Thomas Vincent added a huge number of videos and slides to the Resources page […][…][…][…][…][…] etc. as well as added a button to link to subtitles […] and fixing a bug when displaying metadata links […].
In addition, Atharva Lele added the Buildroot embedded Linux project to the “Who’s involved” page. […]
Test framework
We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. The following changes were done in the last month:
- Alexander Couzens (OpenWrt):
- Holger Levsen:
- Jelle van der Waa:
- Change Arch Linux and Alpine
BLACKLISTstatus toblacklist[…] andGOODtoreproducible[…] respectfully. - Add a Jenkins job to generate Arch Linux HTML pages. […]
- Fix the Arch Linux suites in the
reproducible.inifile. […] - Add an Arch JSON export Jenkins job. […]
- Create per-distribution reproducible JSON files. […]
- Change Arch Linux and Alpine
-
kpcyrd (Alpine):
- Start adding an Alpine theme. […]
- Add an Alpine website. […][…][…][…]
- Add
#alpine-reproducibleto theKGBchat bot. […] - Use the
apkversion instead ofvercmp. […] - Install/configure various parts of the chroot including passing in Git options […], adding the
abuildgroup onto more servers […][…], installing GnuPG […] - Build packages using its own scheduler. […] […][…]
- Misc maintenance and fixups. […][…]
- Mattia Rizzolo:
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
- Bernhard M. Wiedemann:
- argus-client (Parallelism race condition and silent build failure)
- benchmark (Version upgrade, fixing failing to build from FTBFS with
-j1) - ck (FTBFS with
-j1, also fixed upstream) - herbstluftwm (Embedded build date)
- HSAIL-Tools (Sort Perl hash)
- lcov (A file’s modification time was being updated by sed)
- linphone (Sort Python
readdir(3), submitted but ignored upstream) - mypaint (Sort call to
readdir(3), probably upstream) - MozillaFirefox+Thunderbird (Report parallelism race condition)
- ndpi (Fix a date, already merged upstream)
- oyranos (Drop build kernel version, already merged upstream)
- perl-Alien-SDL (Sort Perl
readdir(3)`, orphaned upstream) - perl-Email-Date-Format (Fix a rare breakage)
- perl-OLE-Storage_Lite (Fix FTBFS when built in 2020, ignored upstream)
- plowshare (Date, already upstream)
- python-hyper (Avoid build getting stuck with
-j1, not upstream) - python-nautilus (Python date)
- python-pgmagick (Sort
readdir(3)call, already merged upstream) - python-qt5 (Sort a Python
readdir(3)/os.walk, submitted upstream) - python-xmlsec (Sort a Python glob call, already merged upstream)
- rakkess (Fix a date and time call in
Makefile) - rclone/cobra (sort Go programming language hash)
- rubygem-rice (Drop unreproducible files)
- sawfish (Version update to get all upstream reproducibility fixes)
- surfraw (Fix a date, already upstream)
- terraform (Report FTBFS when built in 2030)
- thunarx-python (Fix a date, not yet upstream)
- uperf (Date, already upstream)
- vboot (Uses a shell date, not yet upstream)
- vtk:
- Fix a date date (merged)
- Sort a Perl hash (merged)
- Report a parallelism race from an unspecified dependency in CMake
- wine (Report the use of random file names)
- Chris Lamb:
- #930768 filed against node-d3-fetch.
- #930769 filed against node-d3-contour.
- #930814 filed against node-d3-hierarchy.
- #930911 filed against node-d3-scale-chromatic.
- #931102 filed against combblas.
- Morten Linderud submitted a patch for libpod, a library used to create container pods to fix a date-related reproducibility issue which has subsequently been merged.
- Richard Biener submitted a patch for the GCC GNU Compiler Collection to fix differences in the runtime debugging info between builds in its D programming language support.
Distribution work
In Debian, 39 reviews of packages were added, 3 were updated and 8 were removed this month, adding to our knowledge about identified issues.
Chris Lamb also did more work testing of the reproducibility status of Debian Installer images. In particular, he was working around and patching an issue stemming from us testing builds far into the “future”. (#926242)
In addition, following discussions at MiniDebConf Hamburg, Ivo De Decker reviewed the situation around Debian bug #869184 again (“dpkg: source uploads including _amd64.buildinfo cause problems”) and updated the bug with some recommendations for the next Debian release cycle.
Bernhard M. Wiedemann posted his monthly Reproducible Builds status update for the openSUSE distribution.
Other tools
In diffoscope (our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues) Chris Lamb documented that run_diffoscope should not be considered a stable API […] and adjusted the configuration to build our Docker image from the current Git checkout, not the Debian archive […]
Lastly, Chris Lamb added support for the clamping of tIME chunks in .png files […] to strip-nondeterminism, our tool to remove specific non-deterministic results from a completed build.
Misc news
On our mailing list this month Lars Wirzenius continued conversation regarding various questions about reproducible builds and their bearing on building a distributed continuous integration system which received many replies (thread index for May & June). In addition, Sebastian Huber asked whether anyone has attempted a reproducible build of a GCC compiler itself.
If you are interested in contributing the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
-
IRC:
#reproducible-buildsonirc.oftc.net. -
Twitter: @ReproBuilds
-
Mailing list:
rb-general@lists.reproducible-builds.org
This month’s report was written by Alexander Borkowski, Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, heinrich5991, Holger Levsen, Jelle van der Waa, kpcyrd & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.
