Collaborative Working Sessions - Metrics

Reproducible Builds Summit 2022

Notes

We defined 4 columns:

  • distributions ( ex. debian )
  • projects ( ex. tor browser )
  • compilers ( ex. gcc )
  • ecosystem ( ex. go libraries ) this is of course related to a language

For each column we defined :

  • qualitative metrics QL ( yes / no / maybe ) that can be manually sourced from a website ( with citation ) and updated from time to time.
  • quantitative metrics QN ( a number ) that should be scraped from an external source and updated regularly and automatically

For distributions:

  • QN number of reproducible packages / vs n. of total packages ( for each release )
  • QN number of reproducible packages / vs n. of total packages ( for each arch )
  • QL does the distribution has a reproducible base system ( bootstrap )
  • QL what kind of reproducibility test is used ?
  • QN how many “reproducible friendly compilers” are shipped by this distro ?

For Projects:

  • QL does the project ships reproducible binanries / images / things ?
  • QN how many of the project dependencies are reproducible ?
  • what kind of reproducible test is used ?

For Compilers:

  • QL does the compiler repruducibly bootstrap itself
  • QL does the compiler produce reproducible binaries ( under defined cirmumstances )

For ecosystems:

  • QN How many reproducible packages / total number of installable packages ?

yaml file formats:

Compiler yaml file format:

name of the compiler: string
language: string
build systems: [ string ]
reproducible binaries on differen archs: [string, bool]
bootstrap reproducibly: bool
compile reproducible binaries: bool

Distribution yaml file format:

name of the distribution: string
type of reproducibility (reprotest/rebuild): string
reproducible status: [architecture]
reproducible status of build essential dependencies: [architecture]

architecture:

name: string
reproducible: float
unreproducible: float
unknown: float

Examples:

name of the compiler: ocamlc
language: ocaml
build systems:
    - dune
    - opam
 reproducible binaries on differen archs: 
    - amd64: yes
    - arm64: yes
bootstrap reproducibly: yes
compile reproducible binaries: true

name of the distribution: alpine
type of reproducibility (reprotest/rebuild/none): none
reproducible status: []
reproducible status of build essential dependencies: []