Detect CI reproducible builds and behave accordingly?

Support Android
1

Hi,

this aims at generating the same output bytes during build runs. Currently, we have some extra options set by the android build environment on github CI and fdroidserver build machines to achieve this.

Summary of what we’ve already done:

  • ‘EXTRA_LDFLAGS’: ‘-buildid=’
  • environment “SOURCE_DATE_EPOCH”, getSourceDateEpoch()
  • environment “BUILD_HOST”, “Catfriend1-syncthing-android”
  • environment “BUILD_USER”, “reproducible-build”

Those together already produce a reproducible build within the F-Droid infrastructure (Syncthing-Fork Reproducibility Status).

image
image1125×399 19.8 KB

Today, I’ve found more issues comparing my GitHub CI builds of the app to the F-Droid outputs and opened a topic on their support forum to work together with them ( Syncthing-Fork reproducible build - work in progress - Apps - F-Droid Forum ).

One remaining problem is:

Syncthing’s test data: which is not compiled in, but deleted by F-Droid’s build process as binary files are detected by their security scanner and disallowed in their build process (syncthing/lib/model/testdata at main · syncthing/syncthing · GitHub)

I’ve workaround by agreeing on “scanignore” with them, and got asked, if we could handle this more nicely within Syncthing’s own repo or build script here ( Update com.github.catfriend1.syncthingandroid.yml: testdata is not compiled in (!24168) · Merge requests · F-Droid / Data · GitLab ).

image

Before, we’ve just deleted the “testdata” file, resulting in the git checkout of the syncthing submodule no longer be clean. That threw differencing bytes into the built “libsyncthingnative.so” module. I’ve examined using 7zip and found “libsyncthing.native/.go.buildinfo” to contain the “dirty” flag string “vcs.modified=true”. While the github CI did not delete the “testdata” file and compiled the byte string to “vcs.modified=false”.

A quick research on Google turned out, we maybe could set “GOFLAGS=-buildvcs=false” for both F-Droid and GitHub CI builds of “Syncthing-Fork”?

===

Thinking about it, I would also welcome a central solution - if possible and sensible - in syncthing’s “build.go” script. We then could detect the Android build and act accordingly or set the above adjustments automatically if an env var like “REPRODUCIBLE_CI_BUILD=1” is present, as F-Droid’s team member Licaon_Kter suggested.

What do you think? Should we work together on this?

Kind regards, Catfriend1

2

I think the build should be reproducible by default, given the env vars you mention in the first paragraph. Any steps to make that more true are welcome. (E.g. we should probably disable buildid by default if necessary.) No Android specific hacks.

If people want to modify the repository and avoid dirty flags they can do the needful (there are git commands to mark the changes as ignored, for example).

1 Like
3

Thanks, I did not know this. Noted in case I need it in the future :slightly_smiling_face:.

git update-index --assume-unchanged syncthing/src/github.com/syncthing/syncthing/lib/model/testdata

4

For all builds or just CI? I think you refer to build.go?

5

Side-question from looking at the linked MR:

Oh, now I understand why “since years” the testdata file was excluded. It does not seem because of saving space in the output APK, but to avoid triggering the fdroid security mechanism for a file that isn’t actually baked in.

Is it bundling the syncthing repo in the APK?! Afaik we only copied the build artifacts and that also seems sensible - bundling testfiles or sources is pretty wasteful as useless.

6

I’ve checked using 7zip to analyse what was built, and from what I’ve seen it does not package the testfile in. (This is nothing I decide from my build scripts.)

7

Thank you all who were involved today to help me on improving the reproducible build :clap::slightly_smiling_face:!

I’ve written a short summary of my lessons learned at the F-Droid forum: Syncthing-Fork reproducible build - work in progress - #11 by Catfriend1 - Apps - F-Droid Forum

tl;dr: One remarkable finding

Sidenote: The build is reproducible as long as Linux-and-Linux is compared. Linux-and-Windows produces different bytecode, e.g. some elements within “libSyncthingnative.so” are different by CRLF vs. LF, caused by the NDK itself.

1 Like
8

Might want to make sure git checks out files with native line endings and doesn’t convert them to CRLF on Windows.

1 Like
9

Yes. I think that when you run build.go build (or tar, zip, etc) that should be a reprodudible build by default, as far as it’s possible to make it so. If that means disabling the build ID, then we do that.

(The default build.go action, install, is more for local development and I don’t think we care as much about reproducibility. If it has build ID and other things enabled that enables the Go compiler to identify things that do not require to be rebuilt and build faster, that’s better.)

1 Like
10

PR openend at Update build.go by Catfriend1 · Pull Request #10203 · syncthing/syncthing · GitHub

11

Offtopic: btw I’ve found an interesting fork of syncthing-android here ( Commits · bpavuk/syncthing-android-rx · GitHub ) It’s based on the Syncthing-Fork and seems to be like a complete rewrite in Kotlin plus Material 3 and compose. :clap: I hope the author will reach out and help with the project, so we get more hands on it. :folded_hands:

2 Likes