Reproducible Builds Summit IV Event Documentation

December 11 – 13, 2019. Paris, France

Agenda

[Day 1]

[Tuesday, December 11]

10:00 Opening session (No notes taken in this session)

The Summit started with participant introductions, a welcome message from the event co-organizers, and an overview of the meeting agenda.

10:30 Getting to know each other

Each participant was invited to talk with another participant they had not met yet.

10:45 Sharing knowledge about reproducible builds (No notes taken in this session)

Participants gathered in small groups which included both folks who were at previous Reproducible Builds Summits and newcomers.

11:05 Break

11:20 Project updates

Several facilitators hosted short discussions, sharing updates about different projects and topics.

12:30 Lunch

13:30 Agenda brainstorming

Participants generated a list of topics and questions that they wanted to discuss at the meeting. The result of the brainstorming was then taken into account to identify the working sessions that had emerged as the most compelling for those in attendance.

14:40 Break

15:00 Strategic working sessions I

16:10 Break

16:25 Closing session

The meeting day ended with an invitation to all participants to share their goals and wishes for the following days, and words of gratitude for a productive start of the Summit.

16:55 Adjourn

17:00 Hack time (No notes taken)

[Day 2]

[Wednesday, December 12]

10:00 Opening session

The second day of the Summit opened with a summary of the work done during the previous day, and a brief overview of the plan for the day ahead.

10:20 Strategic working sessions II

11:45 Break

12:00 Skill share sessions (No notes taken in these sessions)

12:45 Lunch

13:45 Group picture

14:00 Strategic working sessions III

15:30 Closing plenary

The day was brought to a close with the invitation to share proposals for the next day’s working sessions, and the invitation to use the meeting space to hack together during the rest of the afternoon.

16:00 Adjourn

16:05 Hack time

[Day 3]

[Thursday, December 13]

[10:00 Opening session]

[The final day of the Summit opened with the encouragement to focus the last day of the meeting on drafting action plans to be carried out during the following weeks and months.]

[10:20 Strategic working sessions IV]

[12:30 Lunch]

**[13:30 Strategic working sessions V

[15:30 Closing plenary]

[The Summit ended with notes of appreciation for everyone who participated in the meeting and contributed to its success.]

16:00 Adjourn

16:05 Hack time

Session notes

Day 1

Project updates

###

Sustainability of the Reproducible Builds project

Discussion focus: Governance, where the project is, sustainability.

Initially the project didn't need to be an organization. The Linux Foundation funded a few members of the core team for the first few years, CII provided support too.

Recently the Reproducible Builds became a member project of Software Freedom Conservancy: this helps to manage funding, and it helps image-wise since they are a good organization with good reputation. Also, they support providing useful connections and networking opportunities.

Reproducible openSUSE

Discussion focus: scripts to help make openSUSE builds reproducible

github.com/bmwiedemann/theunreproduciblepackage

The Unreproducible Package is meant as a practical way to demonstrate the various ways that software can break reproducible builds using just low level primitives without requiring external existing programs that implement these primitives themselves.

It is structured so that one subdirectory demonstrates one class of issues in some variants observed in the wild.

github.com/bmwiedemann/reproducibleopensuse

Reproducible openSUSE tools

github.com/bmwiedemann/reproducibleopensuse/blob/master/autoprovenance

autoprovenance

Toolsets

Discussion focus: Diffoscope, strip-nondeterminisms, tools for debugging.

reproducible-builds.org/tools/

Boostrap ability

Discussion focus: Bootstrap ability Ask me Anything (e.g. Is my favorite distro bootstapable?)

bootstrappable.org/

Connect through mailing list and/or the IRC channel #bootstrappable

gitlab.com/janneke/mes

Reduced Binary Seed Bootstrap joyofsource.com/reduced-binary-seed-bootstrap.html

Wish: every language developer to tell their bootstrap able success story.

How to achieve user trust and adoption?

Discussion focus: Once we have reproducible builds, how can we help users trust them and use them.

Project: How to protect Debian users from backdoors? Response: Reproducible builds can help with that!

So we need to have a system to ensure that all new packages are built reproducibly.

Also needed along the line: operated diversity.

Open questions, specifically about update processes: question of user privacy, trust between organizations (sharing infrastructure, agreeing on APIs etc.)

Debian APT archive. See www.debian.org/distrib/archive

Testing framework

Discussion focus: How we test distros for reproducibility

tests.reproducible-builds.org/

Current needs:

Reproducibility by design

Discussion focus: Looking at distros which have elements of reproducibility by design

Agenda brainstorming

Specific projects/ languages

User

Specific problems

Bootstrapping

Reach out

Funding/sustainability

Doc

Out of scope

[Column with blank label]

[Column with blank label]

[Column with blank label]

[Column with blank label]

[Column with blank label]

[Column with blank label]

[Post-its not belonging to any column]

User verification

Cross distro

Language issues

Bootstrap

Developers Developers…Developers!

Sources

Trust & Delivery

Things to do here

Build infrastructure

Android/Java

[Post-its not belonging to any column]

Metadata standards

Communications

Rebuilders

End user/ Interfaces

[Post-its not belonging to any column]

Strategic working sessions I

####Tooling

Tools in r-b:

diffoscope

strip-nondeterminism

disorderfs

reprotest

autoclassify (partially specific rpm/OBS)

autoprovenance

filterdiff? github.com/bmwiedemann/reproducibleopensuse/blob/master/filterdiff

buildinfo.debian.net

github.com/bmwiedemann/theunreproduciblepackage

obs-build for cross-distro buildenv setup

rebuilders ?

missing tools:

trace back issues to source line

disorderfs:

problems when applying it to whole build-chroot

diffoscope:

running 30 minutes for an ISO image - e.g. diff sub-parts

hard to see the interesting parts / changes in large diffs

tool interconnect / integration - similar to github+CI

r-b test aaS

packaging for tools

matrix: what is packaged for which OS

Where to maintain/develop the tools - outside salsa?

git repo, bug tracking, mailing lists, documentation, how to contribute

go through the "good OSS project" checklist

how to get more people involved

doc: structure for different personas

single dev

doc: +cookbook do this to test that your software builds reproducibly

distribution maintainer

product/business manager wanting to learn about the benefits

rebuilders:

how to incentivise running rebuilders

what software is it? apply OSS checklist

prepared public cloud rebuilder images

Nix / OBS:

could quantify money or CO2 saved by build-tree-cutoff from r-b

Compilers

BPPM = BUILD_PATH_PREFIX_MAP

Specification defined in reproducible-builds.org/specs/build-path-prefix-map/.

Why do compilers embed buildpath? Do they need it?

Go compiler: stores _both_ buildpath and debug symbols.

Example package: seqtk (?) differs only in build path embedding.

Compiler option -fdebug-prefix-map.

GCC8 -ffile-prefix-map, used to manipulate __FILE__

Why BPPM needs to be a map (and not just "trim the part before the project root").

Clang has debug-prefix-map, but not file-prefix-map.

GCC rejected the BPPM patch: too many env ars, don't want any more.

Old idea: use this variable only if a special flag is provided? Modify debug-prefix-map or file-prefix-map.

On Nix/Guix (?) just use the same virtual path. Why not in Debian? Because we want full reproducibility.

On Go: problem with static libraries.

Do we want to go with BPPM or not?

Do we want to rely on build flags instead?

User verification

End user verification

Motivation / Use cases

Tasks:

UI

*

Tasks:

(Before discussing UI, we should discuss the user stories)

downloaded from GitHub or have someone verify the build from

r-b.orgs and show the task color)

Process

Tasks:

Bootstrapping trust

Tasks:

Trust

Tasks:

TODO:

downloaded from GitHub or have someone verify the build from

r-b.orgs and show the task color)

Terminology

Vocabulary

Words that we might want to define or that felt relevant for our work:

Questions that arised from looking at these terms:

/!\ There should also be a photo of the post-its board somewhere /!
(see above)

Bootstrapping

All things bootstrapping (<2018-12-11 Tue 15:15>)

POSIX systems need to have an early C bootstrap. On top of that we have other languages that each need their own bootstrap story that may assume the presence of a POSIX system.

Outreach: should we add a “tainted” metric to packages that directly or indirectly depend on binary blobs? Produce a ranking of languages and their “ecosystems” wrt to the amount of “taintedness” or the cumulative size of the binary seed?

In some aspects bootstrappability is orthogonal to reproducibility, but on some axes there’s a clear connection. Reproducibility is important for auditing purposes, but if there’s a dependency on an opaque binary blob, the audit cannot be complete.

Bootstrapping allows you to convince yourself of the trustworthiness of a binary. Reproducibility allows you to transfer trust to others.

---------------------------

Trascription of post-it notes

---------------------------

Theory and philosophy

Compiler writer outreach

Bootstrappability=convince yourself. Reproducibility=convince others

Too late for outreach?

Sell it better

Supporting old & new architectures

Defining "bootstrappable" clarifying arguments on the website

Policy on generated non-binaries (e.g. fixing build steps, prep+using source)

From a common starting bootstrapping line, taking new architecture make them split, try to rejoin them the most

Growing

Rust long boostrap chain

Rust port to a new architecture

Replay history vs create new route

build system creators + bootstrappable

Circular dependency

GHC (now for real)

Audit

attack vectors

Add "taintedness" fornon-bootstrappable languages/tools

DCC for bootstrap?

Auditability (of long chains)

Source proviname/proof of age

Audit initial seed

Getting off the ground

Sharing & cooperation: GNU; Musl + Clang; BSD + Clang; …

Format for steps for bootstrapping? Text/ shell scripd/ Guix/ Nix/…

Really do C first

GNU from scratch. GCC->MesCC->…

Minimal (sharable?) code for bootstrapping

Rebuilders

Input: source code; Build instructions

Output: sha of source; build info??

Rebuilder repo: salsa.debian.org/reproducible-builds/debian-rebuilder-setup

Short term: project specific rebuilders. long term:project agnostic

Generic rebuilders (debian/etc)

Setup tools (available) & documentation

Rebuilder architecture

DB for rebuilder results

Small scale rebuilders

Centralized rebuilder results

result format?

Scheduler – monitor for new buildinfo files

Worker(s) – buildstuff

More workers

“visualizer" (-> rename this) collects results from builders

What information should rebuilders provide?

How? From whom? Why? → API

Hash of package? buildinfo?

Not all projects have buildinfo files

Are .buildinfo files without binary checksums useless? (Arch Linux has those)

Is it enough / useful if rebuilders publish a stripped down version of .buildinfo files?

Current Debian .buildinfo files are unreproducible

Publish only successful rebuilds or all artifacts?

Package manager integration

Theat model

Compromised: by sources build; by hackers

Harden rebuilders

Rebuilder diversify for security protection

Can't be crowdsourced without trusting the crowd

Mapping out hardware level issues

There are more and more architectures (x86, ARM, GPUs, FPGA, machine learning

chips, DSPs, …) with interesting aspects that cause unreproducible builds:

Observations are that a fair amount of number of bulilds fail with high thread counts, main reason are incorrect Makefiles. The current work around is to limit the number of threads to 1 - which yields in a max 2% utiliztion in the above example. In many cases builds are somewhat timing critical (to release fixes etc) so having an efficient and scalable parallel build environment is important

Also, how can we trust the hardware? Somebody mentioned backdoors in FPGAs. (Could there be a Thompson attack on hardware? E.g. could the software used to build the hardware (VHDL compiler) be backdoored to insert/propagate a backdoor in the hardware?)

Solutions:

Random shoutout to MIPS and Cray-1 instruction sets.

Java reproducibility

objective: verify that dependencies are corresponding to source code

not only Java (jars) but also Android (apk)

source code as tarball, run build

but build is "transparent"/"opaque" => don't know all the inputs: dependencies are pulled from (Maven) Central repository or others as binary

how to check everything that was pulled?

Bootstrapping issues in addition

Maven is now available in Gix as bootstrap, but not plugins

Gradle in progress, but more complex since even more dependencies

Java ecosystem uses Maven Central as repository

how to rebuild Central repository content in a separate location to check against original?

sources required in central are not for build, but for IDE users = -sources.jar for .jar

see central.sonatype.org/pages/requirements.html#supply-javadoc-and-sources for reference

but only sometimes -source-release.zip or .tar.gz (= source tarballs that we need to rebuild) are provided, since not defined as checked requirement to publish

Android is globally using Gradle and download from many repositories, mainly "Google’s Maven Repository" (dl.google.com/dl/android/maven2/index.html))

some code in there is proprietary

reproducibility requires to store which JDK version is used, since JDK 7, 8, 9 … don't generate the same bytecode

=> absolutely need to record JDK version used to be able to later rebuild by someone else

recording platform is questionable

how can we move forward and help future artifacts published to Central to be really reproducible and rebuilt?

Record:

=> need to talk about publishing the build recipe (as Buildinfo…)

then how to verify dependencies at buildtime, because they might come from other repository, or changed in the middle

exists Gradle witness plugin that records in Gradle buildfile hashes of dependencies used, and when hash is already present checks that the calculated hash is the same

but does not the work for plugins,

and some people complain that build is slower

Day 2

Strategic working sessions II

User stories

End User Stories (Dec. 12, 2018)

** Distro and package manager users

** Users installing non-distro software off the Internet

thing; this allows security auditors to look for backdoors in

binaries.

** Company employees installing internal software

Rebuilder security

scheduler

builders (use fresh chroot)

API for results

buildinfo can be uploaded without any permission - increased attack surface

assets:

attackers:

out of scope:

Debian generate buildinfo inside (potentially compromised) chroot

todo:

compromised binary package could compromise further builds on all rebuilders

rebuilders should use only packages confirmed to reproduce

maybe: have "unstable-confirmed" with only confirmed packages for (re)builders to use

Summary of state of bootstrapping in Nix/Guix.

Classic bootstrap tools set in Nix/Guix: gcc, binutils, bash, coreutils, … (250+ MB of opaque binaries)

Current state in Guix: replaces gcc, glibc, binutils (roughly 150 MB of binaries) by linux-libre kernel headers, mescc-tools, bootstrap-mes (www.gnu.org/software/mes/)..) mescc-tools provides among other things a Scheme interpreter that is used to build a patched tinycc, which in turn builds gcc. mes also contains a small C library. The Scheme interpreter is currently delivered in binary form, so the next step is to bootstrap that from source.

Should be fairly straightforward to use this in the Nixpkgs bootstrap. However, currently no ARM support.

Have to use old gcc (pre 4.something) because newer gcc requires a C++ compiler.

in guix just now:

mes (scheme interpreter / C compiler) → tinycc → ancient binutils (2.2.0.18) → ancient gcc-core (2.95) that doesn't depend on (most of) glibc → ancient glibc (2.2.5) → full ancient gcc → gcc 4.7.4 (last pre-C++ gcc) [all of the previous steps are i686-linux] → crossbuild of current version of gcc for x86_64-linux

Kernel dependency? Bootstrapping on bare hardware without an OS?

Overhead of the new bootstrap approach: 3 hours or so?

Mes bootstrap tarballs: alpha.gnu.org/gnu/guix/bootstrap/i686-linux/20181020/

Java and JVM languages

JMV languages bootstrap

1. situation on guix:

1.1 bootstrapping

1.2 JVM: bootstrapped

1.3 jdk: version 11

1.4 JAVA: bootstrapped

1.4.1 ant: bootstrapped

1.4.1.1 have build system

1.5 clojure: bootstrapped

1.5.1 have build system

1.5.2 missing leiningen

*** unbundle asm and lang

** groovy: bootstrapped

** scala: compiler bootstrapping problem

** maven bootstrapped

** keep dependency count low

*** antlr

*** bcel

*** ant

** grammar for scala

*** ast done

** empty classes build

** inheritance all done

** ignore type checking -- as far as we can

** target something recent

** write it in compcert style

** keep it uptodate

*** involve other people when done

** twitter might be interested

** too much dependencies

** depends on itself

** fials to build the dependency list

** google’s repo, combine all repos, build in one round

Hardware II

[mfl] I think incorrect results, or in other words undetected user level data corruption are extremely low probability. All CPU vendors are putting super efforts into validation as wrong results are an absolute no-go. The cache and memory sub-systems are probably most exposed to random bit-flips, but are protected through SECDED (Single Bit Error Correction Double Bit Error Detection). In addition the majority of external memories has a chip-kill feature, i.e. the ECC bits are laid out in way across the multiple memory chips that an entire chip can fail.

RAS (Reliability Availability and Serviceability) have become common in HW and SW design,there are different techniques in predicting errors and changing corresponding components.

It should however be clear that HW is failing due to a multitude of errors and these errors are expected and handled by design. The same is true for disk drives which have a number of reserved sectors as spares, replacement is (almost) transparant to the user.

Communication interfaces (copper or fibre) are using advanced algorithms to eliminate errors, again this is totally expected.

More relevant are "new" bugs as demonstrated by SPECTRE and MELTDOWN.

But then remember the old saying "Any piece of HW can fail at any time" :)

[mfl] Certain instructions are micro-coded. Micro-code is SW (although a bit special) and the implication is that a build may yield different behaviour. At least there is no way to integrate micro-code into hashes. The sme is true for other "invisible" SW components as the BIOS and BMC

[mfl] The point here is that certain HW elemenents can have a life-cycle of decades. We need to pro-actively research potential pitfalls

mfl to some extend unpredictable behaviour also occurs on HW caches. It probably can be safe to assume that a FPGA implementation is operating when the VHDL/Verilog or whatever HDL has synthesized and placed&routed succesful. Any error there should probably be seen similar to a compiler error generating wrong code.

Need to evaluate if it is feasible to create hashes for bitstreams.

I'm lost here which HW doesn't guarantee reproduceable results? There are a lot of customers actually validating bit-wise correct results on many different platforms.

mfl again I don't see bitflips as something which will show up at the user level. If there is other proof on this we obviously need to re-consider

Cross distribution participation

representatives from Arch Linux, openSUSE, Guix, Nix, Alpine Linux, Debian

“package finder tool" interface for searching infomation about packages, links to distribution trackers, etc.:

maintainer.zq1.de

sharing between distros:

salsa.debian.org/reproducible-builds/reproducible-notes/blob/master/ideas_on_sharing_notes_between_distros

what is needed to create a database across distros to compare reproducibility across distros:

www.softwareheritage.org/archive/

Debian documentation: wiki.debian.org/UpstreamGuide

repology as source of information about packages in various distros: repology.org/

PostgreSQL database dump of the reproducible status of Debian, Arch Linux and ?

tests.reproducible-builds.org/reproducible.sql.xz (MB)

Schema tests.reproducible-builds.org/reproducibledb.html

issues around go packaging:

database of patches submitted upstream to avoid duplicate work

Trust models

trust model for reproducible builds and rebuilders

the trust is in the builder (which holds the key)

now, adding rebuilders to the picture, who rebuild the package, and should be checked

want to remove the blind trust in the authority!

requirements:

solution:

TUF/notary

adjustments to package manager

process:

verification:

if one of the signing keys gets compromised, with a CT you can see whether some package was signed out of the official loop

for the n+1 threshold (n should be chosen by the user), the negative matters as well

whenever you query the rebuilder: three-state:

rebuilder signs with their key the signature of the package

client retrieves a list of checksums, and verifies that it meets

distribution of hashes vs trust model

“score system" for rebuilders?

policy vs long-term audit

“official" mirror list + "official" rebuilder list

reputation is part of the model

malicious rebuilders could get official checksums, and publish it to get high reputation

another model: builder only publishes package when some rebuilders have successfully rebuild

what should the user configure?

non-nacks:

timing and monotonicity:

(a) alert, have to audit

(b) don't install

what are the types for request/response for rebuilder?

for a given package name and version, what should be the result?

TUF (the update framework)

what we're getting here

Tools matrix

Package Matrix for reproducible-builds.org/tools/

Strategic working sessions III

Reproducible file systems

Use cases

System images include e.g. cloud images, live media, installation media.

Solved already

report from Tails.

⇒ we'll focus on non-solved.

FAT filesystem
Overview

2 ways to create:

2 ways to copy:

Until today, Qubes OS and Tails did not manage to get 100%

reproducibility of FAT filesystems.

WIP

(Thanks!)

Creating filesystem

Solved!

As of 2018-12-12, on Debian sid, this generates 2 identical FAT32 filesystems:

#!/bin/sh

set -eu

SRC\_DIR=\"\$1\"

[ -d \"\$SRC\_DIR\" \] \|\| exit 2

export MTOOLS\_SKIP\_CHECK=1

TIMESTAMP=\'2008-12-24 08:15:42\'

for i in 1 2 ; do

IMG=\$(readlink -f \$i.img)

rm -f \$i.img

mkfs.msdos \--invariant -v -i 1234ACAB -C \$i.img \$((1300 \* 1024))

(

cd \"\$SRC\_DIR\"

find ! -name . -type d -print0 \| LC\_ALL=C sort -z \| \\

xargs -0 -n1 -I\_ faketime \"\$TIMESTAMP\" mmd -i \$IMG ::\_

find -type f -print0 \| LC\_ALL=C sort -z \| \\

xargs -0 -n1 -I\_ faketime \"\$TIMESTAMP\" mcopy -i \$IMG \_ ::\_

)

[ \$i -eq 2 \] \|\| sleep 61

done

IPFS distribution

Using IPFS to distribute package binaries (Dec. 12th)

Each peer in the IPFS network has a “peer ID” (the hash of a public

key) and connects to other peers. Together, the peer form a

distributed file system. The file system can be mounted using FUSE.

Content is unencrypted; applications who need it are expected to do

it by themselves.

implements DHT communication. Users look up content by hash in the

DHT, which gives them the contact information (IP address, etc.) of

the peers that provide that content.

Non-pinned data—e.g., data a peer fetch from other peers—can be

garbage-collected.

the data is content-addressed.

the following procedure:

1. Split the file in several chunks, by default 256 KiB.

2. The set of chunks leads to a Merkle tree.

3. The end result is the SHA256 of the root of the tree.

“trickle”) that is more appropriate for retrieving data

incrementally.

identical chunks within files.

localhost:8080/ipfs/HASH/file.txt. ipfs.io provides that

gateway, although it’s obviously centralized and mostly useful for

testing.

support, etc. etc.

localhost:8080:/ipfs/HASH, which is immutable.

mDNS/DNS-SD), you’d end up downloading directly from that peer.

servers by an HTTP server such as ftp.debian.org, and because it

exposes an HTTP interface (the “gateway”), it could be integrated

in a package manager without making any modifications

1. you still go to, say, ftp.debian.org, to get a list of package

name/IPFS hash pairs;

2. from there you fetch the package itself directly over IPFS, using

its hash.

update it every time you publish new content, with URLs like

/ipns/PEERID/file-name.txt. Updates are published via

publish/subscribe (“pubsub”).

workaround is to add a TXT record to a DNS entry, with

“_dnslink=/ipfs/HASH”.

User stories II

User Stories

[Note: these are in rough priority order]

1. Someone uploads a malicious* package version to a package repository. End user types "apt-get install <package>". The malicious package version does *not* get installed.

1b. Same as above, but for a user who chose an alternate package repository which may be less trustworthy than the central repository.

1c. Same as above, but for a user not using a package manager. For example, downloading static binary from GitHub or a Docker image from Dockerhub.

1d. User configures the definition of what trust means, e.g. at least N rebuilders have "certified" the binary, a particular rebuilder has certified, etc. [TODO: define "certified".]

2. User wants to contribute to help build trust. They rebuild packages and publish their results so that other users can detect bad binaries.

3. End user chooses operating system based on the trustworthiness of its binaries. How can they gain insight?

3b. Distribution publishes statistics about trustworthiness, e.g. % of reproducible packages in repo, to give others trust in their distribution. Also to compete and push industry toward better place.

4. Developer builds something locally, which they trust, and remotely (CI/CD), which they don't trust. By comparing hashes they don't need to upload large amounts of data, possibly over a weak link, to gain trust in the remotely built binaries. Example use case: Tor Browser, which takes ages to build and has huge artifacts.

5. System administrator wants to verify that none of the binaries running on their system are affected by known vulnerabilities. Can use a blacklist of "bad" artifact hashes.

6. System administrator applies a security policy on the source code that run on their system, e.g. "all source code must contain sign off by another person" or "do not deploy AGPL binaries." The deployer proves that the binary If you can prove that a binary came from particular sources, you can apply the policy on the source to know whether it’s OK to allow the binary.

7. Package maintainer’s system is compromised. Malicious binaries built on their system are not accepted by the package repository.

8. Author of software performs necessary steps to make their product reproducible, with tests to prevent regressions. Doing so should be easy.

[TODO: any other error cases?]

Out of scope

N. A user wants to rebuild the entire operating system from source. [If they rebuild everything, why do they need reproducibility?]

Alpine

Alpine link added in <maintainer.zq1.de/#bash> JavaScript part

Tools matrix II

Bootstrapping III

Goal: Use mes as binary seed for Nix

Looking through the boostrappable documentation:

The website sources are available here: git.savannah.gnu.org/cgit/guix/bootstrappable.git/

Looking at Debian:

build profiles to build minimal package variants used to simplify the bootstrap process

cross-building bootstrap of architectures

TinyCC bootstrap (via guile + mescc) in Nix: github.com/edolstra/nixpkgs/commit/800bd373c0958d3fda355df9d1d0a49eea5815c8 \o/

Trust models II

Trust models II

The goal was to identify possible trust models and architectures: how can we protect users from non-reproducible packages and identify the global consensus on what binary package hash corresponds to a source package. We defined two models.

Manipulations of the source code by the uploading maintainer may be allowed.

Questions

what may be malicious:

Models/architectures

supply chain steps:

1. upstream

2. downstream/distribution maintainer

3. a benevolent archive builds twice to make sure package is reproducible

4. create binary package

5. install on client and some rebuilder checks (to be defined later)

We now describe two models with different assumptions and security properties. In a paranoid world we could want both models simultaneously.

based on a transparency log

goal: create a global consensus on what constitutes the "correct" binary to a source package

notable properties:

⇒ only detects, does not prevent reproducibility failures

based on supply chain-security

notable properties:

further considerations

trustworthy rebuilders

How do we get reliable third-party rebuilders, NOT operated by the distribution? How do we get enough (how many?) of them?

can we integrate git into the log server Merkle tree?

Not sure, we don't know the documented security guarantees of git.

Checking reproducibility is not enough

We should get guarantees from further up the supply chain, e.g. that someone vouches for the source code that is used by all rebuilders

Resources

Rebuilder NYU (server)

Rebuilder University Bergen (server)

in-toto (website)

in-toto apt transport (repo)

software transparency architecture

software transparency at DebConf18

On Omitting Commits and Committing Omissions: Preventing Git Metadata Tampering That (Re)introduces Software Vulnerabilities

Test infrastructure

Test infrastructure

tests.reproducible-builds.org/reproducible.sql.xz (MB)

Schema tests.reproducible-builds.org/reproducibledb.html

M started adding a distribution field in the test database model.

OPAM reproducibility

opam hooks → preinstall, postinstall

opam package install → list of binaries and locations

local reproducibility (storing hashes on the first run and check on the second round)

check reproducibility w.r.t. hashes provided by independent rebuilders

opam should have a way to clearly specify the external dependencies in order to be able to exactly record and reproduce the

build environment, including versions.

independent rebuilders should be provided with buildinfo files to be able to reproduce exactly

the build environment

opam2 runs on a sandbox that at the moments binds /usr in the bubblewrap chroot. The idea

would be to restrict the sandbox to only the declared dependencies.

I did a bit more work on the opamrepro script. You can find my last version here :

gitlab.com/abate/opam-repro

Initial findings are:

This analysis was done on a small set of packages and ignoring *.cmt* files.

it would be interesting to generate output for all opam packages.

Next steps

gist.github.com/hannesm/224e777fb29cef1b1a9fdda2ea6881fe

Hack time

Day 3

Strategic working sessions IV

###

MirageOS

1st phase: specify the artifacts

2nd phase: specify the transformations

3rd phase: specify the things to reproductibilytification

From the first input, using cp / mirage-configure & opam (todo), generates all informations for the rebuild input. Once we get these, using "simple" commands, it is possible to generate the rebuild and then compare unikernel binary & hypervisor config

Specification of

TODO:

reprotest (see salsa.debian.org/reproducible-builds/reprotest also a debian package) → run on opam repo ?

Rebuilders III

Rebuilders take 3

Recap:

Two models for security checks for reproducible builds

The log server shows the entire history of the rebuilt packages.

The rebuilder has a list of packages

Upstream problems:

Discussion about verifying the correct source code is being used…

If bad source code gets in, then a bad binary comes out…

The source package is changed to look at a different, compromised upstream repository

F-Droid produces binaries in some cases that bit-for-bit match the upstream binary release.

For Debian, the client fetches metadata. It checks with the individual rebuilders to find out if they agree on the resulting binaries.

Discussion of binaries being misrepresented (a binary produced for nginx, being misrepresented to the user as apache)

List of things we want to talk about:

in-toto (is a thing?) It’s a framework for creating and verifying things

buildinfo.debian.net/

It’s a service, a bit hacky, different distros have different requirements

When a client queries the rebuilder, it’s an online operation.

Fetching metadata from rebuilders shouldn't necessarily be an online operation, at least at install time.

Where does the source code come from for rebuilders? The buildinfo file contains the hash for the source.

The rebuilder creates a in-toto file.

What to standardise and share?

Should rebuilders work for multiple distributions?

Maybe a standardised approach could help those who don't already have a concept

Existing rebuilders

We want to produce documentation on how rebuilders work.

Maybe a desired outcome is some advertising to encourage people to run rebuilders.

Switch to the client side

Rebuilders building twice… you should only have one version

Rebuilders building multiple times, is that a good idea?

Publishing history from rebuilders could be interesting, like a certificate transparency log

Policy or architecture…

Links:

PGO Profile-guided optimization

PGO

Profile guided optimization are currently unreproducible and for Arch Linux enabled for some packages such as Firefox. Python.

Bernhard has written an example of unreproducible PGO, and has actually fixed PGO issues for some packages.

build.opensuse.org/request/show/647618

github.com/bmwiedemann/theunreproduciblepackage/tree/master/pgo

A simple PGO example with a fixed input is reproducible on at least two different sets of hardware. The question is, if a more complicated program with different input is reproducible.

Worked on identifying if grep can be PGO'd and reproducible such as openSUSE has achieved. The grep build isn't always reproducible but sometimes is. This might depend on how long it takes to run the test suite which is fed to the

pkgbuild.com/~jelle/grep-unrepro/

PGO works

with our grep example:

Filesets

---

Can be input or output

Can be recursively composed

Output of build targets

Serializable

Reminds some people of source, rather than build results.

Immutable

has a _handle_/reference

Synonymous with snapshot?

Definition should clarify attributes: e.g. unix bits, symlinks. But we can recognize many things as roughly this.

Concrete

Artefact/Artifact

---

Opaque object (binary, ml model, bytes)

Noun is reminiscent of "archeology"!

Generated Source?

Implies one object (non-traversable); Is a fileset that has been turned into a "tar" archive an artifact?

Product of some process

Packages

---

A package could be the "least granular unit".

Are packages all past/present/future versions or just a particular version?

Cannot be recursive

Not subdivisble which has implications for debugging reproducibility

Indicates dependencies (runtime/buildtime/install)?

Organization unit with semantics

Docker images: they're files, but also have a mainfest.json which contains semantics. So is this a package? An application?

In a programming language the concept of a "package" or "module" to define scope. Files are less interesting.

Is a deb a package by itself or is it only a package when it is installed?

A bunch of packages coming together… is a "distro"… Unless they're language packages (e.g. npm).

Likely contains configuration data.

We tend to think of packages as something that is named rather than immutable. The name must be resolved to actual "things".

Side effects?

Wish list

---

We need a term for "named intent to produce a thing"… which does NOT carry "content has semantics" like the word "package". What is the thing the rebuilder _does_?

Potential terms: "build target", "rule (output)", "catalog name"

Is **transitive closure** part of this? Do we have two missing terms?

“Rule/Transform" → Fileset → Other Rules/Transforms

Need to reference the output of an unevauluated rule/transform.

Build

---

Builds are a function of a fileset to a fileset

tl;dr;

---

Debugging for reproducibility issues

We discussed the 10 known sources of nondeterminism documented in github.com/bmwiedemann/theunreproduciblepackage/

and covered Bernhard’s how-to-debug list:

to be added into github.com/bmwiedemann/reproducibleopensuse where the useful (OBS/openSUSE-related) debugging tools are

Getting openSUSE on testing infrastructure

openSUSE added to t.r-b.o db and html is being generated.

For this, openSUSE’s published rb.zq1.de/compare.factory/reproducible.json is imported into reproducibledb

salsa.debian.org/qa/jenkins.debian.net/commit/5e429a958763d1ba2fd90aafffeb64ab25e7e396

salsa.debian.org/qa/jenkins.debian.net/commit/673fd148299a943de9af0daf1d13678a9617e1b1

salsa.debian.org/qa/jenkins.debian.net/commit/502138c149494047eb19900b7ac408d64079cbe3

salsa.debian.org/qa/jenkins.debian.net/commit/d216ef4f0e1e0015e3a0180680e0d8faee7809f4

salsa.debian.org/qa/jenkins.debian.net/commit/9704f22211fc1a9a04d6fc0335b76191fec89438

salsa.debian.org/qa/jenkins.debian.net/commit/19b01d7ea611ebd84bdf1a76a64a396ff7e9a5fa

salsa.debian.org/qa/jenkins.debian.net/commit/68e0d31da013b7b7b6d63aeebbb6c062cdbd2800

salsa.debian.org/qa/jenkins.debian.net/commit/dd60da7d54f551c5d08e8f75607fff137b130fa2

salsa.debian.org/qa/jenkins.debian.net/commit/c891a86566efbd6c4aed32628dd0d94f0717a363

salsa.debian.org/qa/jenkins.debian.net/commit/a30c3c18f9fc9e6adcbf53fdcf2dfed9a0452a96

Strategic working sessions V

Qubes OS reproducible hacking

Qubes OS hacking:

packagage building - it works!

github.com/QubesOS/qubes-builder/pull/66

github.com/QubesOS/qubes-builder-rpm/pull/41 Fix handling components with multiple packages

github.com/QubesOS/qubes-vmm-xen/pull/49

github.com/QubesOS/qubes-builder-debian/pull/26

Transparency log for Debian and Tor Browser

Torbrowser and Debian can follow a similar model

for each supported Torbrowser version, a signed tree root which covers that release

Hack time