Reproducibility Troubleshooting

To identify the origin of non-deterministic build outputs:

Try to isolate the build steps involved. For example: it may be possible to perform a partial rebuild of the affected files, and/or to temporarily comment-out irrelevant source code until the problem disappears.
- Smaller builds are generally quicker, and this is likely to allow you to experiment more rapidly.
- Removing irrelevant code allows you and collaborators to narrow the focus of your investigation.
- Minimal examples are often helpful as test cases and for communication purposes.
Compare the differing output files, using a tool such as diffoscope.
- Sometimes the nature of the content difference itself will provide clues (see the difference patterns below).
- Context surrounding the difference may help you to locate relevant source code.
Try to pinpoint the variance factor that causes the difference. Tools such as reprotest can help you do this.
- This will allow you to focus your experimentation on the relevant factor.
- Knowing the variance factor should also make it easier to consult relevant documentation.

With some luck and/or persistence, you should obtain an idea of the affected build step, the different values that appear, and the input factors that introduce them.

We now need to inspect the differences and to try to understand what could have caused each of them.

In some cases, the differences themselves may provide some hints about possible fixes. For example:

Difference patterns

Before	After	Seems to be a…	Possible remedy
2021-04-10	2026-09-01	date	Configure `SOURCE_DATE_EPOCH`
APK file A	APK file B	Android build difference	Unpack the APK and examine the contents
00 00 A4 81	00 00 B4 81	Zip file permission	Configure reproducible file permissions

Reproducibility Troubleshooting

Difference patterns

Introduction

Achieve deterministic builds

Managing variance

Define a build environment

Distribute the environment

Verification

Specifications