Reproducible Builds: Weekly report #185

Here’s what happened in the Reproducible Builds effort between Sunday November 4 and Saturday November 10 2018:

• We are excited to announce that the Reproducible Builds project has joined the Software Freedom Conservancy!

  

  Conservancy is a not-for-profit organisation that helps promote, develop and defend free software projects. We can now can take directed donations and the Conservancy can also provide projects us with basic legal services. The Reproducible Builds project is delighted and honoured to be associated with Conservancy’s outreach work and other work of the project and look forward to a long and mutually beneficial relationship.

• The month-long session of students from the Application Security course at New York University, cataloguing, submitting and merging reproduciblity bugs concluded this week. This year, students made 55 tags and issues for Debian and Arch Linux packages and sent 18 pull requests upstream of which 4 have been merged.

• Richard Parkins posted a detailed message to our mailing list on the topic of algorithms used for comparing binary files in a way that makes the result easily consumable by humans. Most binary file comparators just compare bytes and thus do not semantically detect deletions or insertions. This is relevant to our work on diffoscope. He linked to some example code on GitHub.

• There was further discussion on Debian bug #869184 which relates to dpkg generating source uploads that include architecture in the name of the .buildinfo file (eg. _amd64.buildinfo). This week, Salvatore Bonaccorso reported that the Debian Security Team were hit by this issue again.

• On Tuesday 6th November, Chris Lamb hosted a seminar and a lengthy Q&A session at the William Gates Building at the University of Cambridge on reproducible builds as part of the Computer Laboratory NetOS Group.

• Simon McVittie kindly provided a patch to our Jenkins-based testing framework that powers tests.reproducible-builds.org to vary whether we apply the “merged /usr” directory scheme between builds. This is where the /{bin,sbin,lib}/ directories are symbolic links to /usr/{bin,sbin,lib}/. It was subsequently merged by Holger Levsen and resulted in some variations in (at least) quilt and systemd.

• Chris Lamb updated strip-nondeterminism (our tool to post-process files to remove known non-deterministic output) to catch invalid ZIP “local” field lengths — we were previously blindly trusting the value supplied in the ZIP file (#803503). In addition, he applied a patch from Emmanuel Bourg to update the Javadoc handler to handle OpenJDK 11 (#913132). He then subsequently uploaded version 0.044-1 to Debian unstable.

• Agustin Henze announced in a mail to the debian-devel mailing list that the new Debian CI pipeline includes support testing for reproducibility using reprotest. These tests are currently available on-demand and need to be set up individually.

• 33 Debian package reviews were added, 14 were updated and 33 were removed in this week, adding to our knowledge about identified issues. Chris Lamb also updated the dc_created_timestamp_in_javadoc issue and added a new cflags_recorded_in_in_ada_ali_files toolchain issue.

• We have received more than 45 registrations for the upcoming Reproducible Builds summit in Paris between 11th—13th December 2018 and thus are in the process of closing registrations. If you are interested in attending and are contributing to a project not yet represented, please do get in touch as registrations will close shortly.

• Our report from last week was quoted in LWN’s “Distribution quotes of the week”.

Packages reviewed and fixed, and bugs filed

• Bernhard M. Wiedemann:
  • geany (Don’t use inode numbers in useless ways)
  • python-qscintilla-qt5 (sort readdir(2), submitted upstream)
  • pesign-obs-integration (bug)
  • grep (PGO/parallelism)
• Chris Lamb:
  • #912957 filed against python-multipletau (https://github.com/FCS-analysis/multipletau/pull/16)
  • Chris’s previously-authored patches for GNU mtools to ensure the Debian Installer images could become reproducible which were sent upstream last week (1 & 2) were merged and should appear in the upcoming mtools 4.0.20 release.
• Oskar Wirga:
  • python-octaviaclient (Set PYTHONHASHSEED)
  • felix-osgi-obr (Timestamp in documentation)
  • easyconf (Ended up being a bug in strip-nondeterminism)
  • commons-daemon (Ended up being a bug in strip-nondeterminism)
• Snahil Singh:
  • #913195: Please make netmrg reproducible.

diffoscope development

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages. This week, version 105 was uploaded to Debian unstable by Mattia Rizzolo. It included contributions already covered in previous weeks as well as new ones from:

• Chris Lamb:
  • Don’t assume all files called “.a” are ELF binaries because we specified a FILE_EXTENSION_SUFFIX. This prevents a potential “Unrecognized archive format” traceback. (#903446)
  • Prevent errors when obtaining PDF metadata from files with multiple PDF metadata dictionary definition entries. (#913315)
  • Display the reason when cannot extract metadata from PDF files.
• Daniel Shahaf:
  • Fix test failures with upcoming file(1) 5.35. Thanks to Christoph Biedl for the heads-up in advance. (#912756)
• Mattia Rizzolo:
  • Update GPG key.
  • Fix a number of flake8 and other deprecation warnings.
• Will Thompson:
  • Add a --list-missing-tools option.

Website updates

There were a large number of changes to our website this week:

• Bernhard M. Wiedemann:
  • Update the “CMake” info to our page about the SOURCE_DATE_EPOCH environment variable.
• Chris Lamb:
  • Use our “contribute to Salsa” page in the footer.
  • Add Cambridge University seminar.
• Holger Levsen:
  • Add Huawei, Alpine Linux and Bazel to Paris summit event page. (1, 2, 3)
• Mattia Rizzolo:
  • Prepend site.baseurl (not site.url) in our logo.
  • Add a donation page.

In addition to that we had contributions from Deb Nicholson, Chris Lamb, Georg Faerber, Holger Levsen and Mattia Rizzolo et al. on the press release regarding joining the Software Freedom Conservancy:

Test framework development

There were a large number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org by Holger Levsen this week (see below). The most important work was done behind the scenes outside of Git which was a long debugging session to find out why the Jenkins Java processes were suddenly consuming all of the system resources whilst the machine had a load of 60-200. This involved temporarily removing all 1,300 jobs, disabling plugins and other changes. In the end, it turned out that the underlying SSH/HDD performance was configured poorly and, after this was fixed, Jenkins returned to normal.

• Debian-specific changes:
  • Merge patch by Simon McVittie to apply the “merged /usr” directory scheme between builds (#901473). […]
  • Document that we vary by installing the usr-merge package […] and add a link to the corresponding Debian Wiki page […].
  • Use pbuilder from the “backports” repositories everywhere, to achieve that also force installation of pbuilder from backports on Ubuntu 16.04
  • Deal with flaky armhf boards. (1, 2, 3, 4)
  • Remove java and depends from all 49 build nodes manually. Also clean up cruft from the jessie2stretch upgrades on armhf nodes.

• Misc/generic changes:

  • Increase heap size further and drop all other Java arguments.
  • Do not recover schroot sessions. Thanks to Helmut Grohne.
  • Run our health check less often.
  • Make jenkins schroot configuration the common/default.
  • Drop code related to volume groups.

In addition, Mattia Rizzolo fixed an issue in the web-based package rescheduling tool by encoding a string before passing to subprocess.run and to fix the parsing of the “issue” selector option.

---

This week’s edition was written by Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Oskar Wirga, Santiago Torres, Snahil Singh & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.