Reproducible Builds in November 2020

← View all our monthly reports


Greetings and welcome to the November 2020 report from the Reproducible Builds project. In our monthly reports, we point out the most important things that have happened in and around our community.

  • Jifeng Xuan gave an online presentation titled Localization of Unreproducible Builds to introduce a technique and tool called RepLoc that can identify the actual problematic, unreproducible files:

    RepLoc features a query augmentation component that utilizes the information extracted from the build logs and a heuristic rule-based filtering component that narrows the search scope. By integrating the two components with a weighted file ranking module, RepLoc is able to automatically produce a ranked list of files that are helpful in locating the problematic files for the unreproducible builds.

    A recording of Xuan’s talk is available, as is a PDF of the associated academic article which was co-written by Zhilei Ren, He Jiang and Zijiang Yan and Zijiang Yang.

  • The Precursor project aims to make a complete hardware and software solution for secure and private communications. It is based on the RISC-V platform, and at the time of writing it is close to reaching its crowdfunding target. This month, a post on Andrew “bunnie” Huang’s blog describes more about the technical details of the project, highlighting that its builds are entirely reproducible.

  • diffware is a new diffoscope-like tool that provides a summary of changes between two files or directories. It can be configured to retain only the changes that matter to the user, and can actually be combined with diffoscope itself to dive deeper into differences it finds.

  • The Corona Warn App is a fork of the German Corona App, where data is stored locally on each user’s device, preventing authorities or other parties from accessing and controlling the data. It doesn’t use the Google-provided services, yet it remains compatible with the official app. It will shortly be available on the F-Droid free-software app store, and it is has been reported that, once available, it will be bit-for-bit reproducible. (German press coverage: Heise.de & Golem.de)

  • The rebuilderd project released three new versions this month, adding support for diffoscope, better build log handling and dramatically improving the prioritisation of new and failed builds. rebuilderd has been powering Arch Linux’s reproducible efforts since April 2020 where it has been used to determine that approximately 80% of Arch Linux’s packages are reproducible.

Distribution work

The Yocto Project has been quietly working on improving reproducibility. As reported in January 2020, its core-image-minimal target packages are bit-for-bit reproducible regardless of the build system’s distribution or the directory used to perform the build. Starting with the first milestone release in the current development cycle, the entire world packages target for all 11,271 packages in OpenEmbedded-Core are now reproducible, with the exception of 65 packages. New targets will be added to the existing automated testing to ensure regressions can be spotted quickly.

In recent months there has been preparatory work to enable the reproducible=+fixfilepath build flag by default. Enabling this fixfilepath feature flag will fix reproducibility issues in an estimated 500-700 packages. After previous discussion a discussion on the debian-devel mailing list, Vagrant Cascadian filed a bug to explicitly propose a patch for the dpkg developers.

Vagrant Cascadian also disabled parallel builds in Debian’s guix package in order to fix a number of reproducibility issues, filing a separate upstream bug report pertaining to embedded build paths. Vagrant additionally made non-maintainer uploads of the texi2html […] and intltool […] packages to Debian in order to fix two toolchain issues.

We also added to our knowledge about identified issues, as 171 reviews of Debian packages were added, 22 were updated and 25 were removed this month. As part of this, Chris Lamb identified and categorised three new toolchain issues: build_path_captured_by_pyuic5, build_path_captured_by_octave & build_path_captured_by_nim.

In the openSUSE distribution, Bernhard M. Wiedemann published his monthly Reproducible Builds status update.

Upstream patches

The following patches were created this month:

Tools

diffoscope is the Reproducible Build’s project in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it provides human-readable diffs from many kinds of binary format.

This month, Chris Lamb uploaded version 162 to Debian (later backported by Mattia Rizzolo), as well as made the following changes:

  • Improvements:

    • Move the slightly-confusing behaviour if a single file is passed to diffoscope on the command-line to a new --load-existing-diff command. […]
    • Ensure the new diffoscope-minimal package that was introduced by Mattia Rizzolo has a different short description from the primary diffoscope one. […]
    • Refresh the long and short descriptions of all of the Debian packages. […]

  • Bug fixes:

    • Don’t depend on radare2 in the Debian ‘autopkgtests’ as it will not be in bullseye due to security considerations. (#975313)
    • Avoid some incorrectly-formatted error messages. This was caused by diffoscope raising an artificial CalledProcessError exception in a generic handler. […]

  • Codebase improvements:

    • Add a comment regarding Java tests to help diffoscope contributors who are not developing using Debian […] and don’t use the old-style super(...) call […].

In addition, Conrad Ratschan added a comparator for “legacy” uboot uImage files to diffoscope (!69), Mattia Rizzolo split the diffoscope package into a diffoscope-minimal package which excludes the larger packages from its Recommends (#975261) and Jelmer Vernooij added a missing space to an error message […].

Elsewhere in our tooling, Holger Levsen also bumped the Standards-Version headers in strip-nondeterminism […], diffoscope […], disorderfs […] and reprotest […], as well as updated the tox.ini test configuration for reprotest and filed a bug after noticing that its testsuite is not run during the build (#975094)

Testing framework

The Reproducible Builds project operates a large Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, Holger Levsen made the following changes:

  • Debian-related changes:

    • Stop testing the Debian buster distribution, except for new package versions. […]
    • Fix a typo when setting up logs to run diffoscope. […]
    • Delete old bullseye and unstable build environments even sooner. […][…]
    • Detect failures to update Debian’s “chdist”. […]

  • Node provisioning scripts:

    • Add debug output if daemon-reload calls fails, etc. […][…]
    • Drop the bring_back_node.sh script; using vim is simpler here. […]
    • Improve documentation of builtin-pho database setup. […]
    • Add more fine-tuned colour indication of filesystem usage on the Jenkins shell monitor. […]

  • Other distributions:

    • Ensure that that FreeBSD test virtual machines are upgraded to version 12.2. […]
    • Enable building of all OpenWrt packages again. […]
    • Detect failure to update Arch Linux build environments. […]

  • System health checks & notifications:

    • Detect etckeeper system service failures. […]
    • Update diskspace warnings. […][…]
    • Provide empty placeholders for machines going down. […]
    • Don’t alert if the version of diffoscope in Debian is behind PyPi. […]
    • Move some IRC notifications to #reproducible-changes. […][…]
    • Suppress noise when showing offline nodes in the Jenkins shell monitor. […]

  • Documentation:

Build node maintenance was also performed by Holger Levsen […][…][…][…][…], Mattia Rizzolo […][…][…][…] and Vagrant Cascadian […][…].

Community changes

Chris Lamb updated the main Reproducible Builds website and documentation to clarify that the SOURCE_DATE_EPOCH environment variable is not Debian specific […], and made a number of miscellaneous cosmetic changes […][…].

There was significant IRC activity during November too. Not only did we create a new IRC channel to capture notifications […], we also hosted a total four meetings: the first were on general topics […][…] as well as specific session on how to debug various distributions. We then held our first ‘Ask Me Anything’ (AMA) as an opportunity for people to ask introductory questions […]. Another AMA session will be held on 7th January 2021.

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports