Reproducible Builds in March 2021

← View all our monthly reports


Welcome to the March 2021 report from the Reproducible Builds project!

In our monthly reports, we try to outline the most important things that have happened in the reproducible builds community. If you are interested in contributing to the project, though, please visit our Contribute page on our website.


F-Droid is a large repository of open source applications for the Google Android platform. This month, Felix C. Stegerman announced apksigcopier, a new tool for copying signatures for .apk files from a signed .apk file to an unsigned one which is necessary in order to verify reproducibly of F-Droid components. Felix filed an Intent to Package (ITP) bug in Debian to include it in that distribution as well (#986179).

On 9th March, the Linux Foundation announced the sigstore project, which is a centralised service that allows developers to cryptographically sign and store signatures for release artifacts. sigstore attempts to help developers who don’t wish to manage their own signing keypairs.

A discussion was started on Hacker News this month regarding OpenSSF, a broad technical initiative aiming to focus on vulnerability disclosures, security tooling as well other related threats to open source projects. At the time of writing, the HN discussion has over 70 comments, including input from members of OpenSSF itself.

On our mailing list, Felix C. Stegerman followed-up to a thread in January 2021 regarding reproducible Python .pyc files. In addition, Jan Nieuwenhuizen announced the release of GNU Mes version 0.23. Mes, a Scheme interpreter and C compiler designed for bootstrapping a base GNU system, was ported to the ARM architecture and can now be used in the GNU Guix “Reduced Binary Seed” bootstrap.

Elsewhere in supply-chain security news, it was discovered that hackers added backdoors to the source code for the PHP programming language after breaching an internal Git server. The malicious code would have made websites vulnerable to a complete takeover including stealing credit card and other sensitive personal information. (ArsTechnica story).


Software development

Distribution work

Coreboot is a project that provides a fast, secure and free software alternative boot experience for modern computers and embedded systems.

This month, Alexander “lynxis” Couzens worked on improving support for Coreboot’s payloads to be reproducible. Whilst Coreboot itself is reproducible, not all of its firmware payloads are. However, lynxis’s new patches now pass build environment variables (e.g. TZ, SOURCE_DATE_EPOCH, LANG, etc.) to the build systems of the respective payloads. […]


When building Debian packages, dpkg currently passes options to the underlying build system to strip out the build path from generated binaries. However, many binaries still end up including the build path because they embed the entire compiler command-line which includes, ironically, the very flags that specify the build path to facilitate stripping it out. Vagrant Cascadian therefore filed a bug against the Debian dpkg package to use GCC’s .spec files to specify the fixfilepath and fixdebugpath options. This supplies the build path to GCC via the DEB_BUILD_PATH environment variable, thus avoid passing the path on the command-line itself. Related to this, it was noticed that Debian unstable reached 85% reproducibility for the first time since enabling variations in the build path.

Frédéric Pierret has been working on a partial copy of the snapshot.debian.org “wayback machine” service, limited to the packages needed to rebuild Debian bullseye on the amd64 architecture. This is to workaround some limitations of snapshot.debian.org. Whilst the mirror itself is reachable at debian.notset.fr, the software to creating it is available in Frédéric’s Git repository. Currently, Frédéric’s service has mirrored 4 months of the archive over two weeks, but needs approximately 3-5 years of content in order to fully rebuild bullseye. To that end, a request was made to the Debian system administrators to obtain better access to snapshot.debian.org to accelerate the initial seeding.

53 reviews of Debian packages were added, 25 were updated and 22 were removed this month adding to our extensive knowledge of identified issues.


Bernhard M. Wiedemann posted his monthly reproducible builds status report for the openSUSE distribution.


NixOS continues to inch towards their milestone of having a fully-reproducible minimal installation ISO: the work by Frederik Rietdijk to make the Python packages reproducible has been merged and the PR to build GCC reproducibly is also progressing. In the mean time a problem with ruby was found and fixed by Tom Berek and a fix for a problem with the gi-docgen generation of Pango documentation is in progress.


diffoscope

diffoscope is the Reproducible Build’s project in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it provides human-readable diffs from many kinds of binary formats. This month, Chris Lamb made a large number of changes (including releasing version 169 version 170 and version 171:

  • New features:

    • If zipinfo(1) shows a difference but we cannot uncover a difference within the underlying .zip or .apk file, add a comment to the output and actually show the binary comparison. (#246)
    • Ensure all our temporary directories have useful names. […]
    • Ignore --debug and similar arguments when creating a (hopefully-useful) temporary directory suffix. […]

  • Optimisations:

    • Avoid frequent long lines in RPM header outputs that cause extremely slow HTML output generation. (#245)
    • Use larger read buffer block sizes when extracting files from archives. […]
    • Use a much-shorter HTML class name instead of diffponct to optimise HTML output. […]

  • Output improvements:

    • Make error extracting X, falling back to binary comparison 'Y' error message in diffoscope’s output nicer. […]
    • Don’t emit “Unable to stat file” debug messages at all. We have entirely-artificial directory “entries” such as ELF sections which, of course, will never exist as files. […]

  • Logging improvements:

    • Add the target directory when logging which directory we are extracting containers to. […]
    • Format report size messages when generating HTML reports. […]
    • Don’t emit a Returning a FooContainer logging message too, as we already emit Instantiating a FooContainer log message. […]
    • Reduce “Unable to stat file” warnings to debug messages as these are sometimes by design. […]

  • Misc improvements:

    • Clarify a comment regarding not extracting excluded files. […]
    • Remove trailing newline from updated test file (re: #243). […]
    • Fix test_libmix_differences failure on openSUSE Tumbleweed. (#244)
    • Move test_rpm to use the assert_diff utility helper.

In addition Hans-Christoph Steiner added a diffoscope.tools.get_tools method to support programmatically fetch diffoscope’s internal config […], Mattia Rizzolo updated the tests to not require a tool when it wasn’t required as well as to correct a misleading reason for skipping, Roland Clobus made diffoscope more tolerant of malformed Debian .changes. files […] and Vagrant Cascadian updated a test so that it would not be run if a required too was not available […].

Website and documentation

Several changes were made to the main Reproducible Builds website and documentation this month. Arnout Engelen, for example, updated the configuration to avoid a conflict between jekyll-polyglot and sass […] as well as replacing an outdated NixOS-related link to a pull request […].

In addition, Chris Lamb fixed some links in old reports […], Frédéric Pierre updated the entry for QubesOS on our list of partner projects, adding an external tests page […], and Vagrant Cascadian added a ‘light’ variant of the Reproducible Builds logo […].

Upstream patches

Testing framework

The Reproducible Builds project operates a Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, the following changes were made:


  • Frédéric Pierret (Qubes-OS):

    • Improve the scripts to host .buildinfo files in a Debian-style “pool” directory structure. […][…][…]
    • Improve handling of temporary files. […]
    • Create package sets in a public folder. […] the
    • Merge a suite-specific script into the main one. […]
    • Fix an awk script. […][…]

  • Holger Levsen:

    • Fix regular expression in host “health check” to correctly detect Lintian issues in diffoscope builds […] as well as APT failures caused by broken dependencies […].
    • Schedule armhf architecture bullseye packages in Debian more often than unstable as the release is near. […]
    • Further work on prototype Debian rebuilder tool to correct a typo in a debrebuild option […], to fail correctly even during when using “pipes” […][…] and make the debug output more readable in general […].
    • Handle temporary files files in the scripts to host .buildinfo files in a Debian-style “pool” directory structure […][…]
    • Declare any pool_buildinfos_suites jobs as “zombies” jobs. […]

  • Vagrant Cascadian:

    • Add a new virt32a-armhf-rb.debian.net and virt64a-armhf-rb.debian.net builders. […][…][…][…][…]
    • Re-enable armhf architecture nodes, now that they have built the pbuilder tarballs. […] […]
    • Add a new package set for “debian-on-mobile-maintainers”. […]

Elsewhere in our infrastructure, Mattia Rizzolo updated the Mailman mailing list configuration to move the automated backups to run 10 minutes after midnight […] and to fix an Ansible warning regarding Python str and int` types […]. Lastly, build node maintenance was performed by Holger Levsen […][…][…], Mattia Rizzolo […] and Vagrant Cascadian […][…][…][…].


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports