The Reproducible Builds project relies on several projects, supporters and sponsors for financial support, but they are also valued as ambassadors who spread the word about our project and the work that we do.
This is the ninth installment in a series featuring the projects, companies and individuals who support the Reproducible Builds project. We started this series by featuring the Civil Infrastructure Platform project, and followed this up with a post about the Ford Foundation as well as recent ones about ARDC, the Google Open Source Security Team (GOSST), Bootstrappable Builds, the F-Droid project, David A. Wheeler, Simon Butler and Kees Cook.
Today, however, we will be talking with Jochen Sprickerhof, one of the newer members of the Reproducible Builds project core team.
Vagrant Cascadian: Could you tell me a bit about yourself? What sort of things do you work on?
Jochen Sprickerhof: I am a freelance programmer working on Open Source. Mainly doing Debian, F-Droid and some smaller software projects. In general I made it a habit to look into every software I use and try to fix bugs or add features I need. In Debian I maintain about 180 packages with topics from home banking over build systems to robotics. Most of my time I currently work on reproduce.debian.net where we try to bit by bit reproduce the packages distributed by Debian.
Vagrant: Could you describe the path that lead you to working on reproducible-builds?
Jochen: I started my Debian journey as a pupil, converting my school to Debian and serving as it’s system administrator for 13 years. After studying applied system science I joined the universities robotics labs where I worked on the robot operating system ROS and the point cloud library PCL. In the end I enjoyed programming more then writing papers so I eventually left academia for a robotics startup. Some years ago I realized that the open source work I was doing in my spare time was actually the work I cared most about. Nowadays I am really grateful that I can spend my days working on things I find important and have lots of fun with.
Vagrant: What projects did you recently make big progress on?
Jochen: A recent example is metasnap.debian.net. It is a meta archive of snapshot.debian.org that is an archive of all packages in Debian. Let me explain it the other way round. With reproduce.debian.net we try to reproduce the packages as they are distributed by the Debian archive. For that we need the same build environment (compilers, build tools..) as was used by Debian back when the original package was compiled. Luckily snapshot.debian.org has all those but they are not easily accessible via apt, Debians package manager. So metasnap provides a mapping from a package name and version to the apt repo on snapshot.debian.org to download it from. It was created by josch some time ago and is an awesome work. But when we tried to reproduce more and more packages on reproduce.debian.net we found that some where missing packages from the build environment even though they where visible on snapshot.debian.org. We found that metasnap excluded some archive areas because they where not expected to be needed. Reimporting all the data took more then two months and surfaced a couple more flaws.
With this fixed we where able to build more packages only to find out that metasnap also needs better support for version numbers. Luckily we where able to rewrite the data in a day instead of starting the import again.
Vagrant: You have been working on infrastructure to support reproducible builds for a while, has recent adoption of reproduce.debian.net into the Debian release tooling changed the focus of your work?
Jochen: Quite a bit. When we started reproduce.debian.net in 2024 only around 33% of the packages could be reproduced successfully. Today we are above 98%. Most were not bugs in the packages themselves, but in the infrastructure. Like the metasnap thing I talked about earlier or packages just needed a rebuild because something else, like the toolchain, was fixed in the meantime. In May people from the Debian release team and reproducible builds sat together and decided that the overall state is good enough and now packages that regress on reproducibility are blocked from entering the next Debian release. But that does not mean all the work is on the shoulders of the Debian package maintainers. Since then I have been constantly looking at the migration tooling to spot regressions and provide fixes. Also a couple of maintainers reached out to us for help and I hope more will do so in future.
Vagrant: What is one small thing you (or others) have not yet gotten to that you would really like to see?
Jochen: The central tool to reproduce Debian packages is debrebuild, also written by josch. Currently it has two ways to retrieve the build dependencies of a package. Either it uses metasnap.debian.net as explained above or it can access the Debian unstable apt archive directly. This allows to test packages locally before everything is indexed on metasnap by compiling against against unstable. But actually there are many other apt repositories to query, like Debian stable or even derivatives. Adding support for an optional list of apt repositories in debrebuild would be great. That would also be a big step to support reproducing other Debian distributions.
Vagrant: … one big thing?
Jochen: It would be great to integrate metasnap.debian.net into snapshot.debian.org. There is some discussion on it already in bugs.debian.org/650783.
Vagrant: What are the tools you use the most?
Jochen: According to my fish shell history:
$ history | awk '{print $1}' | sort | uniq --count | sort --numeric-sort --reverse | head -10
33937 git
19760 vi
11593 rm
8099 cd
7487 ls
6002 apt
5207 grep
3874 mv
3491 dpkg
2692 cp
Vagrant: So, is the fish shell reproducible? I remember it did not used to…
Jochen: You can check for yourself, it is last time I did: reproduce.debian.net/excuses.html?source_name=fish But looking through the other commands, neovim sadly is not: reproduce.debian.net/excuses.html?source_name=neovim I hope we can fix that at some point.
Vagrant: Oh, that’s a nice URL to check for reproducible package… you can just pass the source package name to check the current results?
Jochen: Yes. An other one is udd.debian.org/reproducibility/ where you can list all packages of a Debian maintainer. It also lists source reproducibility and has nice filter.
Vagrant: What tools do you use specifically working on reproducible builds?
Jochen:
I don’t have statistics for that but I would say sbuild, to build the
package, debrebuild, to reproduce it, and diffoscope to analyze the
differences. Obviously I also need apt source
Vagrant: So how many packages are left to build reproducibly, and once those are finished, what is next?
Jochen: Right now reproduce.debian.net shows over 98% reproducibility though there are still over 650 package left and some will probably need a lot of work. But actually I think making packages reproducible is just the first step. For me this is a project to gain trust into the system. To reproduce a package we have two parts: The source of the package and the build environment. Fixing the packages means gaining trust into the first part but we still rely on the individual build environments for each package, as we need to use the same compiler that was used when the package was build initially. So we have to keep around all historical versions of all toolchain packages. I really would like to remove this extra archive which means we would have to rebuild all of Debian around release time. I am dreaming of a Debian release where you could bit by bit reproduce every package just from the released versions. Due to how Debian works this is not a trivial rebuild but will need some work on the infrastructure. By the way, initially there was a third component to pay attention to: any connection to the outside world during the build. Luckily we fixed the Debian build daemons to not allow network connections during the build some time ago.
Vagrant: Thanks for all that work, and taking the time to tell us a bit about yourself!
Jochen: Thanks a lot for the interview!
For more information about the Reproducible Builds project, please see our website at
reproducible-builds.org. If you are interested in
ensuring the ongoing security of the software that underpins our civilisation
and wish to sponsor the Reproducible Builds project, please reach out to the
project by emailing
contact@reproducible-builds.org.
