Reproducible Buster, Bullseye & Bookworm where we come from and where we are going
Holger Levsen
the last mile and other lightyears ahead
I miss you, lovely Debian people!
...and I miss DebConf!
...and this is all wrong.
However, it is what it is... sigh.
Anyway.
The incomplete team, with apologies to $YOU
akira • Alexis Bienvenüe • Alexander Couzens • Andrew Ayer • Asheesh Laroia • Bernhard M. Wiedemann • Boyuan Yang • Ceridwen • Chris Lamb • Chris West • Christoph Berg • Clint Adams • Dafydd Harries • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David Suarez • Dhole • Drew Fisher • Emmanuel Bourg • Emanuel Bronshtein • Esa Peuha • Fabian Wolff • Frédéric Pierret • Guillem Jover • Hans-Christoph Steiner • Harlan Lieberman-Berg • Helmut Grohne • Holger Levsen • HW42 • Intrigeri • Jelmer Vernooij • josch • Juan Picca • Justin Cappos • kpcyrd • Lunar • Maria Glukhova • Mathieu Bridon • Mattia Rizzolo • Nicolas Boulenguez • Niels Thykier • Niko Tyni • Paul Wise • Peter De Wachter • Philip Rinn • Reiner Herrmann • Robbie Harwood • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Stefano Rivera • Stéphane Glondu • Steven Chamberlain • Tom Fitzhenry • Vagrant Cascadian • Valerie Young • Valentin Lorentz • Wookey • Ximin Luo
- Sadly this talk is not team prepared and thus misses the updates since last DebConf section and more. It's the 2nd and hopefully the last time that only myself is presenting this massive team work.
- Vagrant gives another talk about Reproducible Builds at DebConf21:
- https://debconf21.debconf.org/talks/89-looking-forward-to-reproducible-builds/
- Saturday, August 28, 12:30 UTC
Who am I
- Holger Levsen / holger@debian.org
- Debian user since 1995
- Debian member since 2007
- Working on Reproducible Builds since 2014
- Located in Hamburg, Germany
Who am I
- Holger Levsen / holger@debian.org
- Debian user since 1995
- Debian member since 2007
- Working on Reproducible Builds since 2014
- Located in Hamburg, Germany
- Responsible for more than 10% of all source packages in Debian bullseye
Introduction
The problem
- Source code of free software available
- …most people install pre-compiled binaries
- No one knows whether they really correspond.
- As a result there are various classes of supply chain attacks.
The solution
- Enable anyone to independently verify that a given source produces bit by bit identical results.
- As a side effect: you can only be sure a binary is free software if it has been reproduced. It's only free software if it's reproducible!
- Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
The definition
- When is a build reproducible?
- A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
- The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.
- https://reproducible-builds.org/docs/definition/
I'll mostly ignore why and how to do such builds now.
I'll just mention that this has been widely understood as a problem now: https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
So I will focus on how to distribute and verify builds today. First I will give an overview about various projects and then I'll explain more about the situation in Debian.
https://reproducible-builds.org
Short overview of reproducibility of other projects (all AIUI)
- Tails: "easy", pragmatically "solved" but not systematically...
- Arch Linux: has rebuilders, though also lacks user tools and/or other integration
- SuSE: active development, by one person, not enabled in offial builds
Short overview of reproducibility of other projects (all AIUI), continued
Short overview of reproducibility of other projects (all AIUI), continued
Summary of reproducibility of other projects (all AIUI)
Many project support reproducible builds by now, but it's unclear what that means, how it's enforced and how users can know and be confident...
I probably didn't backdoor this
The unreproducible package
Debian
My goals / wishes for DebConf20 / last year
-
To share and widen the understanding of the status of reproducible bullseye:
- CI versus rebuilds
- issues with buildinfos.debian.net/org
- thousands of packages without .buildinfo files in bullseye
- fix and improve debrebuild (from src:devscripts)
- other issues
- using reproducible builds and user interfaces are not even on my radar (anymore), but we will need those too.
Status of those goals / wishes today
- CI versus rebuilds: some progress
- issues with buildinfos.debian.net/org: better
- thousands of packages without .buildinfo files in bullseye: solved
- fix and improve debrebuild (from src:devscripts): partly addressed, huge infrastructure progress
- other issues: always
- using reproducible builds and user interfaces: getting closer
Though first, my frustration (from 2020)
I feel I have given warnings that the next Debian release will not be reproducible for years.
Debian 9 / stretch
The "reproducible in theory but not in practice" release
Debian 10 / buster
The "we could be reproducible but we are not" release
Debian 11 / bullseye
The "we are almost there but still haven't sorted out some requirements" release
Debian 9 / stretch
The "reproducible in theory but not in practice" release
Debian 10 / buster
The "we could be reproducible but we are not" release
Debian 11 / bullseye
The "we are almost made it" release
Debian 12 / bookworm
The first Debian release with some meaningful reproducibility?
share and widen understanding of the status of reproducible Debian
CI versus rebuilds
Debian is wrong
93% reproducibility is a lie.
or rather: 93% are CI results.
CI versus rebuilds:
- We have no Debian infrastructure rebuilding Debian packages. The reproducible-builds.org rebuilders are builders, not rebuilders.
- Up until recently we had two main blockers for rebuilders:
- >3000 packages without .buildinfo files, fixed by myself end of February 2021.
- snapshot.debian.org was (and is) unusable for rebuilds, fixed by Frédéric Pierret and josch since June 2021, by providing a partial mirror for amd64 only and only going back until January 2017.
- see their talk on Thursday, August 26 at 21 UTC: "Making use of snapshot.debian.org for fun and profit"
That number (93%) was wrong/from last year
- we are at 95.5% (29599 out of 30896 source packages) CI reproducibiliy for bullseye now.
- that's almost 2% up compared to buster (93.9%)
- or almost 3000 more reproducible packages (29599 instead of 26682 in buster)
- or even more impressive: we've solved one third of the remaining 6% buster had...
"Solved" problems with .buildinfo files
- buildinfos.debian.net is just a proof of concept, but it kinda works.
- we had >3000 packages without .buildinfo files... (solved).
- #862073 ftp.debian.org: Please POST .buildinfo files to buildinfo.debian.net (worked around)
- #763822 ftp.debian.org: please include .buildinfo file in the archive (worked around)
Remaining problems with .buildinfo files
- #862538 security.debian.org: Please POST .buildinfo files to buildinfo.debian.net: security updates only show up at point releases
- #929397 ftp.d.o: please upload LTS .buildinfo files to ftp-master: we have some time to fix this, bookworm will become LTS in 3 years or so
- GPG keys expire...
Debian rebuilders
- Last year we had to fix and improve debrebuild(from src:devscripts)
- most of them have addressed...
- but rebuilding needs a working snapshot.debian.org service and we found that snapshot.debian.org doesn't scale...
- snapshot.debian.org was (and is) unusable for rebuilds, fixed by Frédéric Pierret and josch since June 2021, by providing a partial mirror for amd64 only and only going back until January 2017.
- see their talk on Thursday, August 26 at 21 UTC: "Making use of snapshot.debian.org for fun and profit"
Debian rebuilders / snapshot.debian.org
- now that we have https://debian.notset.fr/snapshot/ (and soon snapshot.reproducible-builds.org hosted at OSUOSL) we can setup rebuilders
- rebuilders for both bullseye and bookworm!
- Frédéric Pierret has a proof of concept rebuilder service too. I'm looking forward to integrate that into tests.reproducible-builds.org in the coming months!
meaningful reproducibilty of Debian
- all 21 essential packages are reproducible.
- 26 out of 29 required packages are reproducible.
- 50 out of 1216 packages of the most installed packages are not reproducible.
- 28 out of 687 packages of the basic GNOME packages are not reproducible.
- Debian installer images are not reproducible.
- Debian Live images are not reproducible.
Eventually...
- #863622: apt: warn when installing packages that are not reproducible
other issues, release team related
- We are very happy that testing migration is blocked for binary uploads
- We very much like the idea of accellerating migration for reproducibility.
- Debian policy: probably too early for "must", but maybe time for "must not regress"? (This needs rebuilders first.)
Thank you
… and all the contributors out there!
Do you think reproducible builds should happen?
If so, please pick one of these bugs and help fixing it.
We need your help.
https://wiki.debian.org/ReproducibleBuilds