Reproducible Builds
for Bullseye, Bookworm & beyond
Holger Levsen
where we come from and where we are going
or
the last mile and other lightyears ahead
or
I still haven't found what I'm looking for
This is a pre-recorded remote talk.
I miss you, lovely Debian people!
Please enjoy DebConf22!
very incomplete list of people
who have been working on this
akira • Alexis Bienvenüe • Alexander Couzens • Allen Gunn • Andrew Ayer • Asheesh Laroia • Bernhard M. Wiedemann • Boyuan Yang • Ceridwen • Chris Lamb • Chris West • Christoph Berg • Clint Adams • Dafydd Harries • Daniel Kahn Gillmor • Daniel Shahaf • Daniel Stender • David Suarez • Dhole • Drew Fisher • Emmanuel Bourg • Emanuel Bronshtein • Esa Peuha • Fabian Wolff • Frédéric Pierret • Guillem Jover • Hans-Christoph Steiner • Harlan Lieberman-Berg • Helmut Grohne • Holger Levsen • HW42 • Intrigeri • Jan Nieuwenhuizen • Jelle van der Waa • Jelmer Vernooij • josch • Juan Picca • Justin Cappos • kpcyrd • Levente 'anthraxx' Polyak • Lunar • Maria Glukhova • Mathieu Bridon • Mattia Rizzolo • Morten Linderud • Nicolas Boulenguez • Niels Thykier • Niko Tyni • Paul Wise • Peter De Wachter • Philip Rinn • Reiner Herrmann • Robbie Harwood • Roland Clobus • Santiago Vila • Sascha Steinbiss • Satyam Zode • Scarlett Clark • Stefano Rivera • Stefano Zacchiroli • Stéphane Glondu • Steven Chamberlain • Tom Fitzhenry • Vagrant Cascadian • Valerie Young • Valentin Lorentz • Wookey • Ximin Luo
(Huge sorry if YOU are missing, please lets fix this. I'd think there should probably be 50 more people on this list at least..!)
Who am I
- Holger Levsen / holger@debian.org
- Debian user since 1995, contributing since 2001, Debian member since 2007
- Located in Hamburg, Germany
- Working on Reproducible Builds since 2014
- Uploaded >10% of all source packages in Debian bullseye
- Uploaded <10% of all source packages in Debian bookworm...
Introduction
Introduction
- Who doesn't know about Reproducible Builds, why and how?
- Ah, well, this is a remote talk and I have no direct feedback. 😷 🤔 🤗 😇
The problem
- Source code of free software available
- …most people install pre-compiled binaries
- No one knows whether they really correspond.
- As a result there are various classes of supply chain attacks.
The solution
- Enable anyone to independently verify that a given source produces bit by bit identical results.
- Reproducible Builds are an important building block in making supply chains more secure. Nothing more, nothing less.
- As a side effect: you can only be sure a binary is free software if it has been reproduced. Someone elses binary is only certainly free software if it's reproducible!
The definition
- When is a build reproducible?
- A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
- The relevant attributes of the build environment, the build instructions and the source code as well as the expected reproducible artifacts are defined by the authors or distributors. The artifacts of a build are the parts of the build results that are the desired primary output.
- https://reproducible-builds.org/docs/definition/
I'll mostly ignore why and how to do such builds today.
I'll just mention that by now this has been widely understood as a problem:
https://www.whitehouse.gov/briefing-room/statements-releases/2021/06/08/...
So today I will give a short overview about various projects and then I'll explain the situation in Debian.
https://reproducible-builds.org
Short overview of reproducibility of other projects (all AIUI)
- Tails: "easy", pragmatically "solved" but not systematically...
- Arch Linux: has rebuilders, though also lacks user tools and/or other integration
Arch Linux is 86.4% reproducible with 1701 bad and 10849 good packages. [core] repository is 93.3% reproducible with 17 bad and 238 good packages. [extra] repository is 94.1% reproducible with 171 bad and 2860 good packages. [community] repository is 83.8% reproducible with 1481 bad and 7674 good packages.
Short overview of reproducibility of other projects (all AIUI), continued
- "Corona Contract Tracing German": update problem due to unreproducibility
Short overview of reproducibility of other projects (all AIUI), continued
Summary of reproducibility of other projects (all AIUI)
Many projects support reproducible builds by now, but it's unclear what that means, how it's enforced and how users can know and be confident...
We still haven't found what we're looking for.
Summary of reproducibility of other projects (all AIUI)
Many projects support reproducible builds by now, but it's unclear what that means, how it's enforced and how users can know and be confident...
We mostly still haven't found what we're looking for, because it's really hard.
For example: without 100% it's basically impossible to do a sensible user experience.
Some more tips
I probably didn't backdoor this
The unreproducible package
https://reproducible-builds.org/docs
Debian
Reproducible Builds were first discussed at DebConf13...
..in a BoF hosted by Lunar sparking all of this. DebConf14 had another BoF.
Automated test builds at the end of 2014.
FOSDEM 2015: getting the wider FLOSS community involved.
diffoscope!
First summit at the end of 2015 in Athens.
DebConf15 had four people giving the talk...
“How can we get this done...???”
We wondered at the beginning of the Stretch development cycle.
Reproducible talks at least...?
DebConf16
DebConf17
DebConf18
DebConf19
DebConf20
DebConf21
“I feel I have given warnings that the next Debian release will not be reproducible for years.” is a quote from last years.
...and I feel fine! 😀
Schrödingers h01ger: frustrated and happy.
Indeed I have given warnings that the next Debian release will not be reproducible for years...
...and I feel fine! 😀
Let me explain. First the frustration...
Debian 9 / stretch
The "reproducible in theory but not in practice" release
Debian 10 / buster
The "we could be reproducible but we are not" release
Debian 11 / bullseye
The "we are almost there but still haven't sorted out some requirements" release
Debian 9 / stretch
The "reproducible in theory but not in practice" release
Debian 10 / buster
The "we could be reproducible but we are not" release
Debian 11 / bullseye
The "we are almost made it" release
Debian 12 / bookworm
The first Debian release with some meaningful reproducibility?
The previous two slides were from last year...
Debian 12 / bookworm
The first Debian release with some meaningful/usable reproducibility?!?
Debian 13 / trixie
I still haven't found what I'm looking for
Debian issues in depth
93% reproducibility is a lie.
or rather: 93% are CI results.
I explain what's "wrong" with CI results in a moment...
93% reproducibility is neither a lie nor useless...
93% reproducibility is neither a lie nor useless...
That number (93%) was wrong/from two years ago
- we are at 96.0% (29674 out of 30895 source packages) CI reproducibiliy for bullseye now.
- that's almost 2% up compared to buster (93.9%)
- or almost 3000 more reproducible packages (29674 instead of 26682 in buster)
- or even more impressive: we've solved one third of the remaining 6% buster had...
Did I say bullseye?
- 4 weeks ago we were at 94.8% (30482 out of 32153 source packages) CI reproducibiliy for bookworm.
- YAY.
Then, since "Hamburg", something broke and we're at:
- 93.0% for bookworm/amd64
- 93.7% for bookworm/arm64
- but why ???
why 93.x% for bookworm now?
why 93.x% for bookworm now?
because haskell FTBFS...
CI versus rebuilds:
- We have no Debian infrastructure rebuilding Debian packages. The reproducible-builds.org rebuilders are builders, not rebuilders.
- That's why I called 93% (or whatever) a "lie".
- Up until recently we had two main blockers for rebuilders:
- >3000 packages without .buildinfo files, fixed by myself in February 2021 and in June 2022.
- snapshot.debian.org was (and is) unusable for rebuilds, fixed by Frédéric Pierret and josch since June 2021, by providing a partial mirror for amd64 only and only going back until January 2017.
CI versus rebuilds:
- We have no Debian infrastructure rebuilding Debian packages. The reproducible-builds.org rebuilders are builders, not rebuilders.
- https://beta.tests.reproducible-builds.org/debian is showing rebuilds of ftp.debian.org - huge thanks to Frédéric Pierret for this PoC.
- Sadly, Frédéric's rebuilder is down atm...
- And one rebuilder is not good enough also. It's a start though:
https://beta.tests.reproducible-builds.org/debian
https://beta.tests.reproducible-builds.org/debian
https://beta.tests.reproducible-builds.org/debian
https://beta.tests.reproducible-builds.org/debian
-
unreproducible in build-essential:
- linux
- gcc
https://beta.tests.reproducible-builds.org/debian
- amd64 only, also because our snapshot mirror is amd64 only
- one rebuilder only, not several (and at least some should run on Debian ressources)
- one person maintaining this so far. Thank you very much, Frédéric Pierret, and sorry too.
working around snapshot.debian.org
- snapshot.debian.org was (and is) unusable for rebuilds, fixed by Frédéric Pierret and josch since June 2021, by providing a partial mirror for amd64 only and only going back until January 2017.
- without "a working" snapshot.debian.org (it works, "just" not for our usecases) we cannot have reproducible Debian...
- sadly snapshot.notset.fr is currently down and snapshot.reproducible-builds.org ist not yet up... :/
improvements to our snapshot.debian.org mirror
- https://salsa.debian.org/freexian-team/project-funding/-/merge_requests/14
- soon to be hosted at OSUOSL as snapshot.reproducible-builds.org
- we want at least arm64 too, though that needs more than just HW. See the MR above.
"Solved" problems with .buildinfo files
- we had >3000 packages without .buildinfo files, I NMUed all of them (with the help of David Bremner!) 😇 Just NEW ones will keep coming...
- buildinfos.debian.net is just a proof of concept, but it works around #862073, #763822, #862538, #929397 well enough.
- GPG keys expire, so we just ignore signatures...
And then, meaningful reproducibilty of Debian is still not possible because:
- linux, gcc and glibc are our current blockers getting build-essential reproducible in bookworm.
- Debian installer images are not reproducible in bullseye.
- Debian Live images are not reproducible in bullseye.
- Sadly "bullseye" was not a typo in the last two lines. :(
meaningful reproducibilty of Debian d-i images
(amd64 only)
- Debian installer images, are reproducible when build from git, as shown by Roland Clobus. The problem here is that automated testing of d-i images fails almost constantly in sid and testing...
meaningful reproducibilty of Debian live images
(amd64 only)
- Debian Live images are reproducible using live-build as shown by Roland Clobus..
- reproducible package installation != reproducible packages
- future of Debian live images uncertain, though we have 3 choices now: none, unreproducible or reproducible.
more on d-i and live images
- Roland Clobus gave a talk at the Debian Reunion Hamburg about his efforts to revive live-images.
- Roland and Phil Hands are working together to get those images tested for functionality as well, using https://openqa.debian.net.
- There's a "Debian installer and images team BoF" happening now, though I don't know if live images will be covered there.
other issues, release team area
- We are very happy that testing migration is blocked for binary uploads.
- We very much like the idea of accellerating migration for reproducibility.
- Debian policy: too early for "must", but maybe for trixie we can have "must not regress"?
other issues, salsa CI related
- "btw", reprotest is basically unmaintained upstream.
bookworm goals
6 months until the freeze.- 0 packages without .buildinfo files..
- build-essential reproducible.
- d-i images reproducible.
- live images reproducible.
- more archs on our snapshot mirror (arm64?).
- a 2nd rebuilder of ftp.debian.org. and a 3rd...
trixie goals
- snapshot.debian.org usable for mass rebuilds by many users for all architectures.
- more rebuilders! (instead of more CI builders)
- 0 bugs with patches unuploaded. Currently there are 292 of these. 2 NMUs per week, uploaded to DELAYED/15.
- #863622: apt: warn when installing packages that are not reproducible
- .buildinfo files known and used by
dak.
post trixie goals
- I still haven't found what I'm looking for...!
- 100% reproducible packages and distributed images for
trixie+1? - What else?
- A liveable planet would also be really really nice. 🥵😱 Kinda off-topic here, but I still wanted to at least once mention the big elephant in every room. 🐘
Thank you
… and all the contributors out there!
Do you think reproducible builds should happen?
If so, please pick one of these bugs and help fixing.
We need your help.
https://wiki.debian.org/ReproducibleBuilds
I still haven't found what I'm looking for
but I'm confident we'll get there, eventually!