Reproducible Builds in September 2020

← View all our monthly reports


Welcome to the September 2020 report from the Reproducible Builds project. In our monthly reports, we attempt to summarise the things that we have been up to over the past month, but if you are interested in contributing to the project, please visit our main website.

This month, the Reproducible Builds project was pleased to announce a donation from Amateur Radio Digital Communications (ARDC) in support of its goals. ARDC’s contribution will propel the Reproducible Builds project’s efforts in ensuring the future health, security and sustainability of our increasingly digital society. Amateur Radio Digital Communications (ARDC) is a non-profit which was formed to further research and experimentation with digital communications using radio, with a goal of advancing the state of the art of amateur radio and to educate radio operators in these techniques. You can view the full announcement as well as more information about ARDC on their website.


In August’s report, we announced that Jennifer Helsby (redshiftzero) launched a new reproduciblewheels.com website to address the lack of reproducibility of Python ‘wheels’. This month, Kushal Das posted a brief follow-up to provide an update on reproducible sources as well.

The Threema privacy and security-oriented messaging application announced that “within the next months”, their apps “will become fully open source, supporting reproducible builds”:

This is to say that anyone will be able to independently review Threema’s security and verify that the published source code corresponds to the downloaded app.

You can view the full announcement on Threema’s website.

Events

Sadly, due to the unprecedented events in 2020, there will be no in-person Reproducible Builds event this year. However, the Reproducible Builds project intends to resume meeting regularly on IRC, starting on Monday, October 12th at 18:00 UTC (full announcement). The cadence of these meetings will probably be every two weeks, although this will be discussed and decided on at the first meeting. (An editable agenda is available.)

On 18th September, Bernhard M. Wiedemann gave a presentation in German titled Wie reproducible builds Software sicherer machen (“How reproducible builds make software more secure”) at the Internet Security Digital Days 2020 conference. (View video.)

On Saturday 10th October, Morten Linderud will give a talk at Arch Conf Online 2020 on The State of Reproducible Builds in the Arch Linux distribution:

The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.

During the Reproducible Builds summit in Marrakesh, GNU Guix, NixOS and Debian were able to produce a bit-for-bit identical binary when building GNU Mes, despite using three different major versions of GCC. Since the summit, additional work resulted in a bit-for-bit identical Mes binary using tcc and this month, a fuller update was posted by the individuals involved.


Development work

In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update.

Debian

Chris Lamb uploaded a number of Debian packages to address reproducibility issues that he had previously provided patches for, including cfingerd (#831021), grap (#870573), splint (#924003) & schroot (#902804)

Last month, an issue was identified where a large number of Debian .buildinfo build certificates had been ‘tainted’ on the official Debian build servers, as these environments had files underneath the /usr/local/sbin directory to prevent the execution of system services during package builds. However, this month, Aurelien Jarno and Wouter Verhelst fixed this issue in varying ways, resulting in a special policy-rcd-declarative-deny-all package.

Building on Chris Lamb’s previous work on reproducible builds for Debian .ISO images, Roland Clobus announced his work in progress on making the Debian Live images reproducible. […]

Lucas Nussbaum performed an archive-wide rebuild of packages to test enabling the reproducible=+fixfilepath Debian build flag by default. Enabling the fixfilepath feature will likely fix reproducibility issues in an estimated 500-700 packages. The test revealed only 33 packages (out of 30,000 in the archive) that fail to build with fixfilepath. Many of those will be fixed when the default LLVM/Clang version is upgraded.

79 reviews of Debian packages were added, 23 were updated and 17 were removed this month adding to our knowledge about identified issues. Chris Lamb added and categorised a number of new issue types, including packages that captures their build path via quicktest.h and absolute build directories in documentation generated by Doxygen`, etc.

Lastly, Lukas Puehringer’s uploaded a new version of the in-toto to Debian which was sponsored by Holger Levsen. […]

diffoscope

diffoscope is our in-depth and content-aware diff utility that can not only locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds too.

In September, Chris Lamb made the following changes to diffoscope, including preparing and uploading versions 159 and 160 to Debian:

  • New features:

    • Show “ordering differences” only in strings(1) output by applying the ordering check to all differences across the codebase. […]

  • Bug fixes:

    • Mark some PGP tests that they require pgpdump, and check that the associated binary is actually installed before attempting to run it. (#969753)
    • Don’t raise exceptions when cleaning up after guestfs cleanup failure. […]
    • Ensure we check FALLBACK_FILE_EXTENSION_SUFFIX, otherwise we run pgpdump against all files that are recognised by file(1) as data. […]

  • Codebase improvements:

    • Add some documentation for the EXTERNAL_TOOLS dictionary. […]
    • Abstract out a variable we use a couple of times. […]

  • diffoscope.org website improvements:

    • Make the (long) demonstration GIF less prominent on the page. […]

In addition, Paul Spooren added support for automatically deploying Docker images. […]

Website and documentation

This month, a number of updates to the main Reproducible Builds website and related documentation. Chris Lamb made the following changes:

In addition, Holger Levsen re-added the documentation link to the top-level navigation […] and documented that the jekyll-polyglot package is required […]. Lastly, diffoscope.org and reproducible-builds.org were transferred to Software Freedom Conservancy. Many thanks to Brett Smith from Conservancy, Jérémy Bobbio (lunar) and Holger Levsen for their help with transferring and to Mattia Rizzolo for initiating this.

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of these patches, including:

Bernhard M. Wiedemann also reported issues in git2-rs, pyftpdlib, python-nbclient, python-pyzmq & python-sidpy.

Testing framework

The Reproducible Builds project operates a Jenkins-based testing framework to power tests.reproducible-builds.org. This month, Holger Levsen made the following changes:

  • Debian:

    • Shorten the subject of “nodes have gone offline” notification emails. […]
    • Also track bugs that have been usertagged with usrmerge. […]
    • Drop abort-related codepaths as that functionality has been removed from Jenkins. […]
    • Update the frequency we update base images and status pages. […][…][…][…]

  • Status summary view page:

    • Add support for monitoring systemctl status […] and the number of diffoscope processes […].
    • Show the total number of nodes […] and colourise critical disk space situations […].
    • Improve the visuals with respect to vertical space. […][…]

  • Debian rebuilder prototype:

    • Resume building random packages again […] and update the frequency that packages are rebuilt. […][…]
    • Use --no-respect-build-path parameter until sbuild 0.81 is available. […]
    • Treat the inability to locate some packages as a debrebuild problem, and not as a issue with the rebuilder itself. […]

  • Arch Linux:

  • System health checks:

    • Highlight important bad conditions in colour. […][…]
    • Add support for detecting more problems, including Jenkins shutdown issues […], failure to upgrade Arch Linux packages […], kernels with wrong permissions […], etc.

  • Misc:

    • Delete old schroot sessions after 2 days, not 3. […]
    • Use sudo to cleanup diffoscope schroot sessions. […]

In addition, stefan0xC fixed a query for unknown results in the handling of Arch Linux packages […] and Mattia Rizzolo updated the template that notifies maintainers by email of their newly-unreproducible packages to ensure that it did not get caught in junk/spam folders […]. Finally, build node maintenance was performed by Holger Levsen […][…][…][…], Mattia Rizzolo […][…] and Vagrant Cascadian […][…][…].


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports