Reproducible Builds: Weekly report #179

Here’s what happened in the Reproducible Builds effort between Sunday September 23 and Saturday September 29 2018:

• Another reminder that the fourth Reproducible Builds summit will take place in December 11th—13th 2018 at Mozilla’s offices in Paris, France. If you are interested in attending please register by sending an email to holger@layer-acht.org. More details may be found on the associated event page.

• Holger Levsen is requesting review and comments on a multi-project syntax notes proposal. He hopes to implement this in the next few weeks.

• A pull request was proposed to CPython to ensure SOURCE_DATE_EPOCH only influences the default type of .pyc files (yet does not enforce it).

• Guillem Jover uploaded dpkg 1.19.1 to Debian unstable, adding support for dumping database package records in alphabetical order in order to provide reproducible status and available database files, and further make it possible to output other “deb822“-formatted data in a deterministic way. In addition, the Dpkg::Vendor::Debian library gained support for a reproducibility-related fixfilepath feature.

• Jelle van der Waa started a TODO list to fix unreproducible PKGBUILD files in Arch Linux which use the Imagemagick graphics library’s convert(1) utility for image manipulation which embeds the date:create and date:modify file headers.

• disorderfs version 0.5.4-1 (our FUSE-based filesystem that deliberately introduces non-determinism into filesystem metadata) was uploaded to Debian unstable by Chris Lamb. It included a number of contributions covered previously.

• A fix to ensure to that fontconfig — a library for configuring and customizing font access — landed in Debian unstable (via upstream).

• Chris Lamb added four Debian package reviews. In addition, three were updated and 11 were removed in this week, adding to our knowledge about identified issues.

• Bernhard M. Wiedemann discovered problems from arch-dependent noarch packages in openSUSE; there are actually over 1,000 of these.

Patches filed

• Bernhard M. Wiedemann:

  • build-compare (better quoting of strings)
  • cpio (drop date using gettextize --no-changelog)
  • foma/malaga-suomi (merged, segfault bug, noarch)
  • gromacs (merged, drop user+host+date+CPU info)
  • kubernetes (merged, sort sets for reproducible manual pages)
  • ldc (merged, drop CPU type from manpage)
  • mapcrafter (use convert -strip)
  • nfoview (version-update, filesyssystem)
  • obs-build (use virtio-serial)
  • plasma5-desktop (parallelism/race)
  • post-build-checks (bug from arch-dependent noarch package)
  • something-for-rabbit (make it noarch)
  • vit (date)
  • yast-x11 (merged, sort hash, noarch)

• Chris Lamb:

  • #909476 filed against botan.
  • #909568 filed against infnoise.
  • #909782 filed against libinput.

diffoscope development

diffoscope (our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages) was updated in Debian unstable by Mattia Rizzolo. It included contributions already covered in previous weeks but also included new changes from:

• Chris Lamb:
  • Strip trailing whitespace from ssconvert(1) output to support gnumeric 1.12.43,
• Mattia Rizzolo:
  • Ignore continuous-integration failures in Debian testing.
  • Adapt tests to the output of gnumeric 1.12.43.
  • In Debian, list liblz4-tool as an alternative to the lz4 package.

Test framework development

There were a huge number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org by Holger Levsen this week, including:

• Arch Linux-specific changes:

  • Introduce user/group variations in order to detect reproducibility issues.
  • Add a script to manually schedule packages.
  • Detect failure to download from Git repositories.
  • Perform a chown(1) call on the build directory to the build2 user to ensure it becomes writeable.
  • Announce summary of scheduled packages again.
  • Fix scheduling of old packages.
  • Fix CSV statistic storage.
  • Reschedule “depwait” and “404” packages after 24 hours has elapsed, packages in 404_ state after two days and DEPWAIT state after two days.
  • Ignore long-living schroots in maintenance job which was finding old schroots by accident.
  • Correctly detect and report if pacman cannot lock its database.
  • Detect “The requested URL returned error: 504” (gateway timeout) build failures.
  • Create pages of packages in “repository X in Y” states.
  • Create a job to update Archlinux webpages every thirty minutes as well as create links between pages.
  • Use the ${PKG_ID} variable consistently

• Debian GNU/Linux-specific changes:

  • Mark odxu4a as offline and then back up (!)
  • Extend the check_node_is_up() utility. This ensures that nodes not accidentally marked as offline
  • Set man-db/auto-update debconf to false and configure dpkg to use the force-unsafe-io flag to speed up builds.

• Misc changes:

  • Shorten IRC messages (1 & 2).
  • Provide overviews of monthly graphs from Munin as well as yearly.
  • Mark nodes as offline after hours of failures.
  • Configure iptables to drop incoming UDP packets - thanks, Bernhard M. Wiedemann!
  • Ignore Archlinux builds when looking for breakages.
  • Fix the MacPorts diffoscope check.
  • Add a new “job health overview” page.
  • Split html_nodes_info job into two jobs for flexibility.
  • Keep fewer log files.
  • Add notes about which tables Debian is using as some might be useful for Arch Linux too.

In addition, Eli Schwartz refactored the grepping of build logs into a helper function, and Mattia Rizzolo made the following changes:

• Enable all the reproducible-related build flags from dpkg by exporting DEB_BUILD_OPTIONS=reproducible=+all.
• Fix the check for patched packages since we moved from Alioth to Salsa.
• Don’t consider “unknown” suites while building the history page.
• Aid debugging by defining a __str__ Python “magic” method.
• Fix a typo in build.sh.

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, Daniel Shahaf, Holger Levsen, Jelle van der Waa & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.