Variations in the build environment

We identify 16 variations in the environment variables that might lead to unreproducible builds. Each environment variable serves as a valuable resource for understanding and addressing the challenges associated with achieving reproducible builds.

Archive Metadata

When working with compressed file formats like zip and tar, it is important to consider the presence of Archive Metadata. This metadata includes information such as file owners, permissions, and timestamps.

Extracting compressed files within a build environment introduces the risk of inconsistencies in file ownership and permissions compared to the original source. This variation in file metadata can result in files being assigned different owners and permissions during the extraction process.

Additionally, when these compressed files are uncompressed, the resulting files inherit timestamps that may differ from the originally generated file timestamps. These inconsistencies further contribute to the unreproducibility of packages.

In certain scenarios, compressed files store modified timestamps (mtimes) that can undergo changes during the build process. When multiple distinct builds are executed, the inconsistent timestamps from compressed files are incorporated into the resulting executables, leading to unreproducible builds. For further details on the archive metadata environment variation, please refer to the provided documentation.

Architecture Information

Architecture information refers to crucial details concerning the Linux kernel version and hardware architecture name, which are obtained through the use of the uname utility. When builds are conducted on different build systems, invoking the uname utility may yield diverse hardware architecture and kernel versions, which are then compiled into the resulting artifacts. This variation in architecture information leads to unreproducible builds, as the artifacts produced on different build systems will differ due to the discrepancies in the kernel version and hardware architecture used during the build process.

For instance, during the build process of the systemd package, a call is made to the uname utility for debugging purposes. This call retrieves the hardware architecture of the build system. In one scenario, the build was performed on a system with the i686 architecture, while in another, it was executed on a system with the x86_64 architecture. Consequently, this difference in architectural information is reflected in the resulting artifacts, causing the builds to become unreproducible due to the varying hardware architectures used during the build process. For more details, you can refer to the issue.

Build ID

The Build ID is a special hash code generated during the build process, derived from specific portions of the software binary content. Its primary function is to generate identical hash codes for identical binaries, enabling unique identification based on their identity rather than their contents. If different builds of the same code artifacts produce distinct Build IDs in their resulting build artifacts, it indicates an unreproducible build process.

It is important to note that the generation of the Build ID can be influenced by various factors, including noise, as highlighted in a bug report. Notably, when builds are executed on different build systems, inconsistent UUIDs can result in varying Build IDs, leading to unreproducible builds.

Build Path

The build path is a critical component in achieving reproducible builds as it provides the necessary build configurations and dependencies for the compiler. It is important to understand that discrepancies in the build path can result in unreproducible builds.

For instance, in a specific scenario, one build utilized a relative build path while another adopted an absolute path. As a consequence, variations in the recorded build paths within the resulting build artifacts led to unreproducible builds.

Workaround: In order to address and mitigate the challenges related to unreproducible builds caused by variations in the build path, a collaborative effort was undertaken with developers involved in the GCC community. This collaborative effort resulted in the introduction of a flag BUILD_PATH_PREFIX_MAP that facilitate the usage of relative paths, enabling the reproducibility of distinct builds. For further details on the flag and their implementation, please refer to the provided documentation.

Build Timestamp

The build timestamp refers to the information associated with the date and time of a specific build execution. It is important to consider that during the build process, any files that are generated, modified, or accessed may embed compile-time timestamps in the form of logs within the resulting build artifacts. These timestamps can lead to differences in the content of the build artifacts when distinct builds are performed due to changes in build time.

However, it is essential to recognize that relying solely on timestamps provides limited insight into the software build itself. This is because builds can be executed on an older version of the software while still having a more recent timestamp.

An example of a timestamp variation would be the C pre-defined macros such as _Date_ and _Time_ are utilized to output the current time. It is important to note that when these macros are invoked by distinct build systems, different timestamps are incorporated into the compiled code, resulting in variations in the generated build artifacts.

Workaround: The SOURCE_DATE_EPOCH environment variable has been introduced as a solution to address the challenges related to build timestamps and facilitate reproducible builds. The value assigned to the SOURCE_DATE_EPOCH variable represents the timestamp of the most recent modification made to the source code for a specific release. This timestamp is usually derived from the source changelog file, ensuring consistent and accurate build time determination across various build systems.

For comprehensive details on the usage and implementation of the SOURCE_DATE_EPOCH variable, we refer to the provided specification.

File Encoding

File Encoding refers to the specific encoding scheme used for files, playing a critical role in ensuring the reproducibility of builds. When builds are executed on different build systems, employing distinct encoding schemes can result in variations in build artifact patterns, potentially leading to unreproducible packages.

In one scenario, during the build process of a package, files on different machines were built using different encoding schemes. Specifically, one build utilized a non-UTF encoder, while the other employed a UTF-8 encoder. These differing encoding strategies led to distinct content in the resulting build artifacts, rendering the builds unreproducible.

Workaround: To ensure the reproducibility of builds, it is crucial to proactively manage and harmonize the encoding schemes across various build systems. By standardizing encoding practices, developers can mitigate the risks associated with unreproducible builds, promoting consistent and reliable outcomes in the build process.

Filesystem Ordering

The order in which files are created and displayed within the filesystem can have a significant impact on the reproducibility of artifacts. When distinct builds are executed, variations in the file order can occur, which, in turn, leads to a different ordering of segments inside the generated artifacts.

For instance, in the case of Ruby 2.3, the presence of mkmf.rb is notable. This script is responsible for automatically generating Makefiles for multiple Ruby applications. However, a critical issue arises from the fact that the generated Makefiles do not sort the list of object files. Consequently, when distinct builds are performed, the resulting build logs capture the compilation process in an unordered manner. This lack of order in the compilation can directly impact the resulting artifacts, rendering them unreproducible.

File Permission

During a software build, new files are created, inheriting predefined file permissions from the containing folder. However, the default file permissions assigned to these new files can vary across different build systems. This discrepancy in default permissions can introduce inconsistencies when attempting to reproduce the build process, ultimately affecting the reliability and trustworthiness of the resulting software.

For instance, during the execution of distinct builds, the usage of the umask utility has been observed to introduce unreproducibility. When the umask value varies across different build systems, the default permissions assigned to files during the build process can differ. This discrepancy is documented in this issue. For example, one build system may have a more permissive umask value, resulting in wider permissions for files, while another build system may have a more restrictive umask value. These disparities in file permissions become embedded in the compiled artifacts, making it challenging to reproduce identical builds.

Locale

Locale plays a crucial role in enabling users to utilize language-specific settings, which are subsequently translated into the corresponding binary code. Each locale maps words to distinct binary codes, facilitating language-specific functionality. However, it is essential to note that variations in the locale settings during the execution of distinct builds can lead to unreproducible outcomes.

When different locales are employed between two build systems during the execution of distinct builds, the resulting build artifact will exhibit varying content. This discrepancy arises due to the mapping of words to different binary codes within each locale. Consequently, the builds become unreproducible, hindering the consistent generation of build artifact.

In certain cases, unreproducible builds have been observed due to discrepancies in locale settings during the build process. Specifically, when default parameter values for functions are set according to the user’s locale rather than the build system’s locale, the builds can become unreproducible.

For comprehensive details on the usage of locales, we refer to the provided documentation.

Workaround: To ensure the reproducibility of builds, one should carefully manage and synchronize the locale settings across different build systems. By standardizing the locale configurations, one can minimize the risks associated with unreproducible builds, promoting consistent and predictable outcomes in the build process.

Package Dependency

In the context of reproducible builds, package dependencies refer to critical software components that must be present for a package to operate efficiently. By preventing code duplication within the source package, they contribute to maintaining a consistent build process. Nevertheless, if packages do not explicitly define the precise versions of their dependencies, it can lead to various complications concerning build dependencies. Challenges related to package dependencies encompass absent dependencies, conflicting dependencies, and the utilization of incompatible or outdated dependencies during the build process. As a result, these issues can cause the builds to become unreproducible. Furthermore, the behavior or execution of a build dependency can also introduce disparities in the build process, adding to the complexity of achieving reproducibility.

For instance, ftbfs_due_to_virtual_dependencies highlights specific cases where packages encounter build failures due to their inability to locate or satisfy essential virtual dependencies. These failures could arise from inadequately specified virtual dependencies or a scarcity of available packages in the build environment, which are meant to provide the required functionalities or features expected by these virtual dependencies.

Randomness

Randomness introduces an element of unpredictability to data stored in data structures and tasks executed in parallel. During the build process on different build systems, the order in which parallel jobs are executed may vary. As a consequence, the generated logs of these parallel build executions can be captured differently, leading to unreproducible builds in the resulting artifacts.

One specific manifestation of this randomness can be observed in the context of software packaging in Debian. Here, control.tar.gz is a compressed file that contains metadata about the package such as the list of files in the package and their respective checksums (md5sums). This is crucial for verifying the integrity of the files in the package.

When creating a Debian package, non-determinism may arise due to the varying order of files listed in md5sums between different builds, found in the control.tar.gz file. This usually happens when the package does not use dh_md5sums, and the find command is used to list the files, which does not guarantee a consistent order. More information can be found in this issue.

For comprehensive details on the randomness, we refer to the provided documentation.

Workaround: To mitigate the issues caused by randomness in unordered data structures or file listings from commands like find, developers are strongly advised to implement a sorting mechanism when retrieving data from these data structures. By applying the sort operation, the data can be arranged in a specific and consistent order, regardless of the inherent randomness.

Reference to Memory Address

In the context of reproducible builds, a reference to a memory address pertains to numerical representations of particular memory locations within build environments. These memory addresses are utilized by data structures in various programming languages like C and Python to access specific locations in memory. The issue arises when, during the execution of separate builds, the same object is allocated different memory addresses, leading to varying content stored in the resulting artifacts. This disparity in memory allocation causes the build process to become unreproducible, as the output artifacts are no longer identical.

For instance, the ldaptor package relies on a python module called weakref. Within this weakref python module, the “repr” function is utilized. This particular function is responsible for generating the memory address of the instance passed to it. However, the instances provided to this function yield different memory addresses, which ultimately become part of the compiled artifacts. As a consequence, this discrepancy in memory addresses leads to unreproducible builds, as documented in this bug report.

Snippet Encoding

Snippet Encoding is the process of encoding strings or specific segments of a file using random numbers. These random numbers, functioning as security keys, are utilized to encode data and prevent unauthorized usage. Within the build execution, these randomized digits are incorporated into the resulting artifacts. Since distinct build systems generate different sets of randomized digits, the resulting build artifacts exhibit varying content.

For instance, during the execution of distinct builds, the usage of the srandom utility has been observed to introduce unreproducibility. This utility is employed to provide a seed value to the randomization function, ensuring randomness in the generated output. However, it was found that the seed value stored in the resulting build artifact differed for each build, leading to unreproducible outcomes as seen in this issue.

System DNS Name

System DNS Name refers to the hostname of a host computer within a specified network, serving as the identifier for that system. This name, also known as the system’s hostname, is crucial in distinguishing the host within the network. However, variations in DNS names can occur across different build systems leading to potential unreproducibility in the resulting artifacts.

Uninitialized Memory

In the context of reproducible builds, uninitialized memory refers to the unutilized memory assigned to resources like data structures or file systems. For example, data structures in various programming languages may receive larger memory allocations than necessary, and to optimize performance, this extra memory is filled with randomized padding. The issue arises when resources utilizing this uninitialized memory are stored in files, which could be linked into the resulting artifact during the execution of different builds. As a consequence, this variation in the inclusion of uninitialized memory in the files leads to unreproducible builds, as the resulting artifacts differ due to the random padding introduced during the build process.

For instance, during the build process of the ipadic package, a .dat file is created, which includes uninitialized memory. To address this, randomized padding is applied to fill the uninitialized memory. However, when different build systems execute builds for the ipadic package, the resulting artifacts contain the .dat files with varying randomized padding. As a consequence, this discrepancy in the content of the .dat files causes the ipadic package to become unreproducible on Debian and openSUSE platforms as seen in this bug report.

User Information

User information refers to data that discloses a user’s identity, including their username, which has the potential to be included in the build logs. If this user information becomes captured in the resulting artifacts, it can lead to divergent build outputs, causing the builds to become unreproducible.

For instance, during the build process of the gnustep-base package, the string generated by $USER is executed, where $USER represents the name of the system’s user who is executing the build. The issue arises when builds are performed on different build systems, as the $USER variable outputs distinct usernames for each system. As a result, this variation in the captured user information causes the builds to become unreproducible, as the resulting artifacts will contain different usernames due to the diverse build environments as seen in this issue.

---

← Documentation index

Documentation home

Introduction

• Definitions
• History
• Buy-in
• Making plans
• Academic publications

Achieve deterministic builds

• Commandments of reproducible builds
• Variations in the build environment
• SOURCE_DATE_EPOCH
• Deterministic build systems
• Volatile inputs can disappear
• Stable order for inputs
• Stripping of unreproducible information
• Value initialization
• Version information
• Timestamps
• Timezones
• Locales
• Archive metadata
• Stable order for outputs
• Randomness
• Build path
• System images
• JVM

Define a build environment

• What's in a build environment?
• Recording the build environment
• Definition strategies
• Proprietary operating systems

Distribute the environment

• Building from source
• Virtual machine drivers
• Formal definition

Verification

• Cryptographic checksums
• Embedded signatures
• Sharing certifications

Specifications

• SOURCE_DATE_EPOCH
• BUILD_PATH_PREFIX_MAP (WIP)