Reproducible Builds
for Bullseye, Bookworm & beyond
Holger Levsen
the last mile and other lightyears ahead
The incomplete team, with apologies to $YOU
akira • Alexis Bienvenüe • Alexander Couzens • Andrew Ayer • Asheesh Laroia • Bernhard M. Wiedemann • Boyuan Yang • Ceridwen • Chris Lamb • Chris West • Christoph Berg • Clint Adams • Dafydd Harries • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David Suarez • Dhole • Drew Fisher • Emmanuel Bourg • Emanuel Bronshtein • Esa Peuha • Fabian Wolff • Frédéric Pierret • Guillem Jover • Hans-Christoph Steiner • Harlan Lieberman-Berg • Helmut Grohne • Holger Levsen • HW42 • Intrigeri • Jelmer Vernooij • josch • Juan Picca • Justin Cappos • kpcyrd • Lunar • Maria Glukhova • Mathieu Bridon • Mattia Rizzolo • Nicolas Boulenguez • Niels Thykier • Niko Tyni • Paul Wise • Peter De Wachter • Philip Rinn • Reiner Herrmann • Robbie Harwood • Roland Clobus • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Stefano Rivera • Stéphane Glondu • Steven Chamberlain • Tom Fitzhenry • Vagrant Cascadian • Valerie Young • Valentin Lorentz • Wookey • Ximin Luo
Who am I
- Holger Levsen / holger@debian.org
- Debian user since 1995, contributing since 2001, Debian member since 2007
- Located in Hamburg, Germany
- Working on Reproducible Builds since 2014
- Uploaded >10% of all source packages in Debian bullseye
- Uploaded <10% of all source packages in Debian bookworm...
- May 2022 was quite extraordinary... little time to prepare this.
Introduction
Introduction
- Who doesn't know about Reproducible Builds, why and how?
The problem
- Source code of free software available
- …most people install pre-compiled binaries
- No one knows whether they really correspond.
- As a result there are various classes of supply chain attacks.
The solution
- Enable anyone to independently verify that a given source produces bit by bit identical results.
- Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
- As a side effect: you can only be sure a binary is free software if it has been reproduced. Someone elses binary is only certainly free software if it's reproducible!
The definition
- When is a build reproducible?
- A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
- The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.
- https://reproducible-builds.org/docs/definition/
I'll mostly ignore why and how to do such builds today.
I'll just mention that by now this has been widely understood as a problem:
https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
So today I will give a short overview about various projects and then I'll explain the situation in Debian.
https://reproducible-builds.org
Short overview of reproducibility of other projects (all AIUI)
- Tails: "easy", pragmatically "solved" but not systematically...
- Arch Linux: has rebuilders, though also lacks user tools and/or other integration
Arch Linux is 86.4% reproducible with 1701 bad and 10849 good packages. [core] repository is 93.3% reproducible with 17 bad and 238 good packages. [extra] repository is 94.1% reproducible with 171 bad and 2860 good packages. [community] repository is 83.8% reproducible with 1481 bad and 7674 good packages.
Short overview of reproducibility of other projects (all AIUI), continued
- "Corona Contract Tracing German": update problem due to unreproducibility
Short overview of reproducibility of other projects (all AIUI), continued
Summary of reproducibility of other projects (all AIUI)
Many projects support reproducible builds by now, but it's unclear what that means, how it's enforced and how users can know and be confident...
I probably didn't backdoor this
The unreproducible package
Debian
Reproducible Builds were first discussed at DebConf13...
..in a BoF hosted by Lunar sparking all of this. DebConf14 had another BoF.
Automated test builds at the end of 2014.
FOSDEM 2015: getting the wider FLOSS community involved.
diffoscope!
First summit at the end of 2015 in Athens.
DebConf15 had four people giving the talk...
How can we get this done...???
we wondered at the beginning of the Stretch development cycle...
Reproducible talks at least...?
DebConf16
DebConf17
DebConf18
DebConf19
DebConf20
DebConf21
"I feel I have given warnings that the next Debian release will not be reproducible for years." is a quote from last year.
...and I feel fine! 😀
Schrödingers h01ger: frustrated and happy.
Indeed I have given warnings that the next Debian release will not be reproducible for years...
...and I feel fine! 😀
Let me explain. First the frustration...
Debian 9 / stretch
The "reproducible in theory but not in practice" release
Debian 10 / buster
The "we could be reproducible but we are not" release
Debian 11 / bullseye
The "we are almost there but still haven't sorted out some requirements" release
Debian 9 / stretch
The "reproducible in theory but not in practice" release
Debian 10 / buster
The "we could be reproducible but we are not" release
Debian 11 / bullseye
The "we are almost made it" release
Debian 12 / bookworm
The first Debian release with some meaningful reproducibility?
The previous two slides were from last year...
Debian 12 / bookworm
The first Debian release with some meaningful/usable reproducibility?!?
Debian 13 / trixie
I still haven't found what I'm looking for
Debian issues in depth
93% reproducibility is a lie.
or rather: 93% are CI results.
CI versus rebuilds:
- We have no Debian infrastructure rebuilding Debian packages. The reproducible-builds.org rebuilders are builders, not rebuilders.
https://beta.tests.reproducible-builds.org/debian is showing rebuilds of ftp.debian.org - huge thanks to Frédéric Pierret.
- Up until recently we had two main blockers for rebuilders:
- >3000 packages without .buildinfo files, fixed by myself end of February 2021.
- snapshot.debian.org was (and is) unusable for rebuilds, fixed by Frédéric Pierret and josch since June 2021, by providing a partial mirror for amd64 only and only going back until January 2017.
That number (93%) was wrong/from two years ago
- we are at 96.0% (29674 out of 30895 source packages) CI reproducibiliy for bullseye now.
- that's almost 2% up compared to buster (93.9%)
- or almost 3000 more reproducible packages (29674 instead of 26682 in buster)
- or even more impressive: we've solved one third of the remaining 6% buster had...
- but we are at 94.8% (30482 out of 32153 source packages) CI reproducibiliy for bookworm! :/
94.8% reproducibility is neither a lie nor useless...
94.8% reproducibility is neither a lie nor useless...
https://beta.tests.reproducible-builds.org/debian
https://beta.tests.reproducible-builds.org/debian
https://beta.tests.reproducible-builds.org/debian
https://beta.tests.reproducible-builds.org/debian
-
unreproducible in build-essential:
- linux
- gcc
- isl
https://beta.tests.reproducible-builds.org/debian
- amd64 only, also because our snapshot mirror is amd64 only
- one rebuilder only, not several
- one person maintaining this, thank you very much, Frédéric Pierret!
- one person maintaining this, I'm so sorry... so someone, please do something, please help!
snapshot.debian.org
- snapshot.debian.org was (and is) unusable for rebuilds, fixed by Frédéric Pierret and josch since June 2021, by providing a partial mirror for amd64 only and only going back until January 2017.
- so now we have snapshot.reproducible-builds.org hosted at OSUOSL and mirroring from https://debian.notset.fr/snapshot/
- arm64 snapshot wanted too (though it needs more than just HW)
- https://salsa.debian.org/freexian-team/project-funding/-/merge_requests/14
"Solved" problems with .buildinfo files
- buildinfos.debian.net is just a proof of concept, but it kinda works. (though "it's called a commit because my code is a crime"...)
- we had >3000 packages without .buildinfo files, I NMUed all of them and I've found 782 more of those... I'll fix those too, but NEW ones will keep coming...
- #862073 ftp.debian.org: Please POST .buildinfo files to buildinfo.debian.net (worked around poorly)
- #763822 ftp.debian.org: please include .buildinfo file in the archive (worked around poorly,
dakknows nothing about them)
Remaining problems with .buildinfo files
- #862538 security.debian.org: Please POST .buildinfo files to buildinfo.debian.net: security updates only show up at point releases
- #929397 ftp.d.o: please upload LTS .buildinfo files to ftp-master: we have some time to fix this, bookworm will become LTS in 2 years or so
And then, meaningful reproducibilty of Debian is not possible yet:
- linux, gcc and apt are our current blockers getting build-essential reproducible.
- Debian installer images are not reproducible in bullseye.
- Debian Live images are not reproducible in bullseye.
meaningful reproducibilty of Debian is possible for: (amd64 only)
- Debian installer images, are reproducible when build from git, as shown by Roland Clobus. The problem here is that automated testing of d-i images fails almost constantly in sid and testing...
- Debian Live images are reproducible using live-build as shown by Roland Clobus..
- reproducible package installation != reproducible packages
- future of Debian live images uncertain, though we have 3 choices now: none, unreproducible or reproducible.
Reproducible d-i and live images?
- Roland Clobus has a talk tomorrow at 10:40 localtime titled Reproducible builds as applied to non-compiler output.
- Roland and Phil Hands are working together to get those images tested for functionality as well, using https://openqa.debian.net.
other issues, release team area
- We are very happy that testing migration is blocked for binary uploads.
- We very much like the idea of accellerating migration for reproducibility.
- Debian policy: too early for "must", but maybe for trixie we can have "must not regress"?
other issues, salsa CI related
- "btw", reprotest is basically unmaintained upstream.
bullseye goals, 6-12 months left
- 0 packages without .buildinfo files..
- build-essential reproducible.
- d-i images reproducible.
- live images reproducible.
- arm64 on our snapshot mirror.
- a 2nd rebuilder of ftp.debian.org
trixie goals
- I still haven't found what I'm looking for, these are rather long term goals, but nothing strategic yet:
- 0 bugs with patches unuploaded. Currently there are 313 of these. 2 NMUs per week, uploaded to DELAYED/14.
- snapshot.debian.org usable for mass rebuilds of all architectures by many users.
- .buildinfo files known and used by
dak. - #863622: apt: warn when installing packages that are not reproducible
Thank you
… and all the contributors out there!
Do you think reproducible builds should happen?
If so, please pick one of these bugs and help fixing it.
We need your help.
https://wiki.debian.org/ReproducibleBuilds
I still haven't found what I'm looking for
but I'm confident we'll get there, eventually!