Task suggestions for Debian

Join the discussion

To get help, feel free to ask on the IRC channel or the mailing list. We want to be friendly, supportive, and have fun experimenting together.

Upgrade to a modern debhelper style

If you maintain a package for Debian, you can make sure that your package uses a modern debhelper style (e.g. one-liner debian/rules with overrides as needed). We aim to fix many causes of non-deterministic builds in the debhelper suite directly, so packages that use debhelper will be much easier to make reproducible with just an upgrade of the toolchain.

Inventory known issues

Inventory issues found by the continuous integration platform.

Fix reproducibility issues

Fix known reproducibility issues. See the inventory of identified issues.

NMUs for packages with existing patches

Do responsible NMUs of packages with existing patches by uploading to DELAYED/15.

Contribute to common tools

Improve our common tools: diffoscope, strip-nondeterminism, disorderfs.

Redesign reproducibility status pages

Help us redesign Debian’s reproducible builds status pages using a CSS toolkit like Bootstrap.

Reporting issues

Reporting issues is a great way to contribute. Please read our guide below on how to report bugs in Debian.

How to report bugs in Debian

Overview of all bug reports concerning reproducible builds

All bugs relevant to the reproducible builds project should use usertags with user reproducible-builds@lists.alioth.debian.org. Also use X-Debbugs-Cc to notify the list, but please use our reproducible-bugs@lists.alioth.debian.org list for this header.

To usertag a bug after it has been submitted use:

bts user reproducible-builds@lists.alioth.debian.org . usertag XXXXXX + timestamps toolchain

Usertags

• toolchain: affects a tool used by other package build systems

• infrastructure: affects the whole Debian infrastructure or policies

• timestamps: time of build in recorded during the build process

• fileordering: build output varies with readdir() order

• buildpath: path of sources is recorded during the build process

• username: username is recorded during the build process

• hostname: hostname is recorded during the build process

• uname: uname output is recorded during the build process

• environment: environment variables are recorded during the build process

• randomness: some build aspects are dependent on (pseudo-)randomness

• cpu: some build aspects are dependent on CPU features or computation speed

• signatures: uses a cryptographic signatures as part of the build process

• umask: permissions depend on current umask

• buildinfo: issues related to .buildinfo control files

• ftbfs: fails to build from source

• locale: varying locales lead to differing behavior (e.g. sorting)

Example email to submit a patch:

From: J. Random Hacker <jrhacker@example.org>
To: submit@bugs.debian.org
Subject:

Source:
Version: <VERSION>
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: timestamps fileordering
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

Hi!

While working on the “reproducible builds” effort [1], we have noticed
that

The attached patch removes extra timestamps from the build system and
ensure a stable file order when creating the source archive. Once applied,
it can be built reproducibly in our current experimental framework.


 [1]: https://wiki.debian.org/ReproducibleBuilds

Inventorying issues

The easiest way to find issues is to examine the list of packages failing to build reproducibly as found by continuous integration. The first packages in the list are the one who have been tried most recently.

Notes about packages are kept in the notes. Git repository in packages.yml. The list of known common issues is kept in the issues.yml file.

The page for a given package should open on the diffoscope output. Read the list of known issues to get an idea of what you may found. Here is some more advice:

• Timestamps in gzip headers are a no-brainer.

• When there’s a mismatching Build ID in an executable, it means a variation happens during the compilation. Investigation can be done using sources.debian.org (see link at the top).

• First step should be a search for the __DATE__, __TIME__ or __TIMESTAMP__ using codesearch. Otherwise, try to locate calls to date in configure.ac, Makefile.am, etc.

The clean-notes script in the misc repository will detect outdated notes and re-order packages by alphabetical order. It should be run before committing changes to the notes repository.

Fixing issues

Fixing reproducibility issues falls into two categories: either the problem is specific to a single package or the cause is the output of another package (then referenced as “toolchain” package).

Fixing a single package

The usual steps are:

1. Use debcheckout or apt-get source to retrieve the source code.

2. Do the changes. With packages using the 3.0 (quilt) format, dpkg-source --commit can be useful.

3. Update debian/changelog. New version is usually original version with .0~reproducible1.

4. Use dpkg-buildpackage -S to create source package.

5. Use reprotest to test reproducibility. If the package is not reproducible, examine the diffoscope output logs/PACKAGE.diffoscope.html or compare build logs logs/PACKAGE.build1 and logs/PACKAGE.build2, then repeat from step 2 unless the issue comes from another package. In that case, see about “toolchain” packages below.

6. Use debdiff or git format-patch to create patches.

7. Create a new bug report (read our guide on how to report bugs in Debian), and don’t forget to attach the patch!

8. Add an entry or reference the bug in packages.yml in reproducible-notes.

Fixing a toolchain package

Fixing an issue in a package that affects the reproducibility of other packages requires some more steps, but the general process is the same:

1. Use debcheckout or apt-get source to retrieve the source code.

2. Do the changes. With packages using the 3.0 (quilt) format, dpkg-source --commit can be useful.

3. Update debian/changelog. New version is usually original version with .0~reproducible1.

4. Use pdebuild or gbp buildpackage to build the package.

5. Backup base-reproducible.tgz.

6. Use pbuilder --login --save-after-exec --basetgz base-reproducible.tgz to install the newly built package.

7. Test a package affected with reprotest. If the issue is still not fixed, repeat from step 2.

8. If the package is in Git, create a new repository on salsa.debian.org. Push your changes to a (rebasable) pu/reproducible_builds branch.

9. Subscribe to the upload-source notification for the package on the Package Tracking System. This is needed so you don’t forget to update the custom package when a new version hits the archive.

10. Upload the package to the reproducible APT repository.

11. Document the changes on the wiki.

12. Reference the bug in issues.yml in reproducible-notes and on the wiki page about the issue if there is one.

13. Everybody with a Debian SSO client certificate (both DDs and guests) can schedule source packages to be rebuilt by using the authenticated endpoint at https://tests.reproducible-builds.org/cgi-bin/schedule. There are handy icons (♻)in every package page that links to that cgi-bin script with the correct parameters. If you don’t have a valid client certificate or have any other trouble you can find somebody in the #debian-reproducible IRC channel to help you. Also, if you need to mass schedule many packages (even over the daily limit imposed by the public scheduling script described here), you can find the jenkins administrators in that channel that can do mass scheduling for you.

14. If the changes don’t break anything, create a new bug report. Don’t forget to attach patches and to use the toolchain usertag.

Continuous integration platform

Several jobs have been created to regularly test packages (from main/sid) on jenkins.debian.net. As a result there is the reproducible build overview of packages.

The setup is explained in this blog post only, but this post is somewhat outdated by now and needs to be amended.

See the various reproducible_* scripts in the Jenkins Git repository

Working on installation media or live systems

Having installation and live systems which can be built reproducibly would also be great.

In Debian, #900918 is being used to track the progress of reproducible installation images. There is an analyze_image Bash script that creates sha512 hashes of all files included within an image, access rights, symlinks, partition table, bootloader and more. Doing this with two images that should match and comparing the reports the script creates can help to identify sources of non-determinism in images. It does not have iso support yet. The author (Patrick Schleizer) is interested to generalize the script for more generic, Debian use cases

External links

• Announcing Whonix’s First Implementation of Verifiable Builds
• Whonix Verifiable Builds
• Tails reproducible builds blueprint
• reproducible debootstrap