Reproducible Builds in September 2022

← View all our monthly reports


Welcome to the September 2022 report from the Reproducible Builds project! In our reports we try to outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. If you are interested in contributing to the project, please visit our Contribute page on our website.

David A. Wheeler reported to us that the US National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA) and the Office of the Director of National Intelligence (ODNI) have released a document called Securing the Software Supply Chain: Recommended Practices Guide for Developers (PDF).

As David remarked in his post to our mailing list, it “expressly recommends having reproducible builds as part of ‘advanced’ recommended mitigations”. The publication of this document has been accompanied by a press release.

Holger Levsen was made aware of a small Microsoft project called oss-reproducible. Part of, OSSGadget, a larger “collection of tools for analyzing open source packages”, the purpose of oss-reproducible is to:

analyze open source packages for reproducibility. We start with an existing package (for example, the NPM left-pad package, version 1.3.0), and we try to answer the question, Do the package contents authentically reflect the purported source code?

More details can be found in the README.md file within the code repository.

David A. Wheeler also pointed out that there are some potential upcoming changes to the OpenSSF Best Practices badge for open source software in relation to reproducibility. Whilst the badge programme has three certification levels (“passing”, “silver” and “gold”), the “gold” level includes the criterion that “The project MUST have a reproducible build”.

David reported that some projects have argued that this reproducibility criterion should be slightly relaxed as outlined in an issue on the best-practices-badge GitHub project. Essentially, though, the claim is that the reproducibility requirement doesn’t make sense for projects that do not release built software, and that timestamp differences by themselves don’t necessarily indicate malicious changes. Numerous pragmatic problems around excluding timestamps were raised in the discussion of the issue.

Sonatype, a “pioneer of software supply chain management”, issued a press release month to report that they had found:

[…] a massive year-over-year increase in cyberattacks aimed at open source project ecosystems. According to early data from Sonatype’s 8th annual State of the Software Supply Chain Report, which will be released in full this October, Sonatype has recorded an average 700% jump in repository attacks over the last three years.

More information is available in the press release.

A number of changes were made to the Reproducible Builds website and documentation this month, including Chris Lamb adding a redirect from /projects/ to /who/ in order to keep old or archived links working […], Jelle van der Waa added a Rust programming language example for SOURCE_DATE_EPOCH […][…] and Mattia Rizzolo included Protocol Labs amongst our project-level sponsors […].


Debian

There was a large amount of reproducibility work taking place within Debian this month:

  • The nfft source package was removed from the archive, and now all packages in Debian bookworm now have a corresponding .buildinfo file. This can be confirmed and tracked on the associated page on the tests.reproducible-builds.org site.

  • Vagrant Cascadian announced on our mailing list an informal online sprint to help “clear the huge backlog of reproducible builds patches submitted” by performing NMU (Non-Maintainer Uploads). The first such sprint took place on September 22nd with the following results:

    • Holger Levsen:

      • Mailed #1010957 in man-db asking for an update and whether to remove the patch tag for now. This was subsequently removed and the maintainer started to address the issue.
      • Uploaded gmp to DELAYED/15, fixing #1009931.
      • Emailed #1017372 in plymouth and asked for the maintainer’s opinion on the patch. This resulted in the maintainer improving Vagrant’s original patch (and uploading it) as well as filing an issue upstream.
      • Uploaded time to DELAYED/15, fixing #983202.

    • Vagrant Cascadian:

      • Verify and updated patch for mylvmbackup (#782318)
      • Verified/updated patches for libranlip. (#788000, #846975 & #1007137)
      • Uploaded libranlip to DELAYED/10.
      • Verified patch for cclive. (#824501)
      • Uploaded cclive to DELAYED/10.
      • Vagrant was unable to reproduce the underlying issue within #791423 (linuxtv-dvb-apps) and so the bug was marked as “done”.
      • Researched #794398 (in clhep).

    The plan is to repeat these sprints every two weeks, with the next taking place on Thursday October 6th at 16:00 UTC on the #debian-reproducible IRC channel.

  • Roland Clobus posted his 13th update of the status of reproducible Debian ISO images on our mailing list. During the last month, Roland ensured that the live images are now automatically fed to openQA for automated testing after they have been shown to be reproducible. Additionally Roland asked on the debian-devel mailing list about a way to determine the canonical timestamp of the Debian archive. […]

  • Following up on last month’s work on reproducible bootstrapping, Holger Levsen filed two bugs against the debootstrap and cdebootstrap utilities. (#1019697 & #1019698)

Lastly, 44 reviews of Debian packages were added, 91 were updated and 17 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated too, including the descriptions of cmake_rpath_contains_build_path […], nondeterministic_version_generated_by_python_param […] and timestamps_in_documentation_generated_by_org_mode […]. Furthermore, two new issue types were created: build_path_used_to_determine_version_or_package_name […] and captures_build_path_via_cmake_variables […].

Other distributions

In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

diffoscope

diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 222 and 223 to Debian, as well as made the following changes:

  • The cbfstools utility is now provided in Debian via the coreboot-utils package so we can enable that functionality within Debian. […]

  • Looked into Mach-O support.

  • Fixed the try.diffoscope.org service by addressing a compatibility issue between glibc/seccomp that was preventing the Docker-contained diffoscope instance from spawning any external processes whatsoever […]. I also updated the requirements.txt file, as some of the specified packages were no longer available […][…].

In addition Jelle van der Waa added support for file version 5.43 […] and Mattia Rizzolo updated the packaging:

  • Also include coreboot-utils in the Build-Depends and Test-Depends fields so that it is available for tests. […]
  • Use `pep517 and pip to load the requirements. […]
  • Remove packages in Breaks/Replaces that have been obsoleted since the release of Debian bullseye. […]

Reprotest

reprotest is our end-user tool to build the same source code twice in widely and deliberate different environments, and checking whether the binaries produced by the builds have any differences. This month, reprotest version 0.7.22 was uploaded to Debian unstable by Holger Levsen, which included the following changes by Philip Hands:

  • Actually ensure that the setarch(8) utility can actually execute before including an architecture to test. […]
  • Include all files matching *.*deb in the default artifact_pattern in order to archive all results of the build. […]
  • Emit an error when building the Debian package if the Debian packaging version does not patch the “Python” version of reprotest. […]
  • Remove an unneeded invocation of the head(1) utility. […]

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Testing framework

The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. This month, however, the following changes were made:

  • Holger Levsen:

    • Add a job to build reprotest from Git […] and use the correct Git branch when building it […].

  • Mattia Rizzolo:

    • Enable syncing of results from building live Debian ISO images. […]
    • Use scp -p in order to preserve modification times when syncing live ISO images. […]
    • Apply the shellcheck shell script analysis tool. […]
    • In a build node wrapper script, remove some debugging code which was messing up calling scp(1) correctly […] and consquently add support to use both scp -p and regular scp […].

  • Roland Clobus:

    • Track and handle the case where the Debian archive gets updated between two live image builds. […]
    • Remove a call to sudo(1) as it is not (or no longer) required to delete old live-build results. […]

Contact

As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:



← View all our monthly reports

Follow us on Twitter @ReproBuilds, Mastodon @reproducible_builds@fosstodon.org & Reddit and please consider making a donation. • Content licensed under CC BY-SA 4.0, style licensed under MIT. Templates and styles based on the Tor Styleguide. Logos and trademarks belong to their respective owners. • Patches for this website welcome via our Git repository (instructions) or via our mailing list. • Full contact info

Conservancy Logo

Reproducible Builds Logo