Reproducible Builds: Weekly report #195

Here’s what happened in the Reproducible Builds effort between Sunday January 13th and Saturday January 19th 2019:

• In the Rust programming language community there was an interesting discussion on the /r/rust subreddit around the ripgrep utility becoming reproducible in Debian. In addition, Tony Arcieri opened a issue in the Rust’s Secure Code Working Group enquiring about reproducible builds tooling.

• Last week, Chris Lamb opened Debian bug #919207 requesting that the squashfs-tools package (which creates and manipulates read-only compressed file systems) applies a patch to remove non-deterministic data introduced by a “fragmentation deflator” thread. This was the final patch required for reproducible images for (at least) Tails.

  Whilst Laszlo Boszormenyi applied the patch, he subsequently reverted the change as it was breaking LZO compression. However, Chris subsequently updated and fixed the issue which was then uploaded in version 1:4.3-11.

• As part of the Debian Long Term Support (LTS) effort it was noticed that an old package was failing to build beyond ~2015.

• Holger Levsen released and uploaded disorderfs (our FUSE-based filesystem that deliberately introduces non-determinism into filesystems) version 0.5.6-1 to Debian unstable […] and Chris Lamb released/uploaded strip-nondeterminism (our tool that post-processes files to remove known non-deterministic output) version 1.1.0-1 to Debian unstable […] too.

• Chris Lamb added 8 Debian package reviews but 12 were also updated and 14 were removed in this week, adding to our knowledge about identified issues.

• There were a number of interesting discussions on our mailing list this week including:

  • Hervé Boutemy posted a brief introduction to “reproducible-central” after a number of discussions and documentation regarding Java Virtual Machine rebuilder attestations and the Apache Maven build tool.
  • Elio Qoshi from Ura Design asking whether we would be interested in updating our style guide.
  • Lastly, Eli Schwartz posted an update regarding reproducible package archives in Arch Linux.

Packages reviewed and fixed, and bugs filed

• Bernhard M. Wiedemann:
  • python-cmarkgfm (merged, sort python glob)
• Chris Lamb:
  • #919566 filed against satpy (forwarded upstream).

diffoscope development

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages. There were a few updates this week including contributions from:

• Chris Lamb:
  • Fix inverted logic and invalid reference to file in the FreePascal comparator. […]
  • Use str.format over + for string concatenation. […]
  • Re-enable Gnumeric Build-Depends. […] […]
• Jelle van der Waa:
  • Remove an unused re import in the WebAssembly comparator. (MR: !18)

Version 108 was then uploaded to Debian unstable by Chris Lamb and was subsequently backported to the stretch-backports distribution by Mattia Rizzolo.

Website development

There were a number of updates to the reproducible-builds.org project website this week, including:

• Hervé Boutemy:
  • Large number of changes to the Java Virtual Machine page including adding the build-root property for multi-module builds […], adding instructions on Apache Maven rebuild arguments […], amongst many others […] […] […].
• Holger Levsen:
  • Drop LEDE as the project has re-merged with OpenWrt. […]
• Peter Wu:
  • Mention QT_RCC_SOURCE_DATE_OVERRIDE and add some more CMake, RPATH and Qt notes on the deterministic build systems page. […] […] […].
  • Document the use of -fmacro-prefix-map and -ffile-prefix-map on the build path page. […]
  • Fix some links and typos on the contribute page, some dead links to Salsa and correct some link formatting issues. […] […] […]

Test framework development

We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. This week:

• Arch Linux is the first project being built on nodes dedicated from OSUOSL.

  Interestingly, these new nodes are running 4.19 Linux kernels from the stretch-backports distribution as Qt in Arch needs a newer kernel than the kernel in Debian stretch to build. As a result of this we are now seeing 1,736 builds of Arch packages in the last 24h, meaning our subset of packages are being fully rebuilt every 5 or 6 days.

• F-Droid became the second project to be tested on these new nodes after Holger Levsen increased the size of various partitions to accommodate the builds, as well as to provide a Squid proxy for all our OSUOSL nodes.

The following more-specific changes were made:

• Eli Schwartz:
  • Import Arch Linux GnuPG keys before running makepkg. […]
  • Perform a giant cleanup of trailing whitespaces. […]
• Holger Levsen:
  • Arch Linux-specific changes:
    • Adjust the rescheduling of packages which have been tested X days ago. […] […] […]
    • Adopt maintenance job to work with the new OSUOSL nodes. […]
    • Support OpenSSH running on ports other than 22. […]
    • Fix the path to the Arch Linux mirrorlist. […]
  • Debian-specific changes:
    • Fix warning message to include the name of broken package sets […] and also show the total number of packages in a package set […].
    • Don’t update pbuilder and Debian schroots on OSUOSL nodes. […]
    • Clarify “stalled” status of the LeMaker HiKey960 boards. […]
    • Document how to access Codethink’s arm64 nodes. […]
  • F-Droid-specific changes:
    • Remove duplicate job definitions. […]
  • Misc/generic changes:
    • Update the “job health page”, adding a helpful footer. […] […]
    • Use time.osuosl.org as the NTP server for OSUOSL nodes, de.pool.ntp.org for the rest. […]
    • Warn if we detect the wrong [Maximum Transmission Unit (MTU))[https://en.wikipedia.org/wiki/Maximum_transmission_unit). […]
    • Drop another mention of LEDE. […]
  • Node maintenance. ([…], […], […], […], […], […], […], […], […], etc.)
• Mattia Rizzolo:
  • Fix a variable name in the “deploy Jenkins” script. […]
  • Fix a non-fatal syntax error in the “health check” script. […]
  • Node maintenance. ([…], etc.)
• Vagrant Cascadian:
  • Node maintenance. ([…], […], etc.)

---

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, heinrich5991, Holger Levsen, Mattia Rizzolo, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.