[SOLVED] Possible tweaks to speed up the compilation process?

Until now, I have just been compiling Syncthing using the default Go settings as follows.

go run build.go

However, it can take a few minutes to compile, especially when starting from scratch. One thing that bothers me a little is that the CPU usage while compiling is very low, e.g. only ~10% with Ryzen 4350G (4 cores + 4 threads). Everything is located on a fast SSD RAID, so storage is not the culprit either.

Are there any possible tweaks to speed up the compilation process, e.g. by increasing the multithreading or such?

The first compilation needs to populate module cache (dependencies), so I guess what you see is a network/disk io being the limit.

I am not aware of any tweaks.

Hmm, I was thinking of something in the vein of make -jX to increase the number of threads when compiling the Linux kernel and such.

It would the network here, as downloading from GitHub is kind of slow where I live.

It’s multithreaded by default.

Yeah, I was just wondering whether it was possible to maybe increase the number of threads, as you can do when compiling the C languages.

I have another question related to the compilation though. It is possible to make the output more verbose when building Syncthing? Right now, especially on successive builds, there is basically no output on the screen if there have been no file changes, and the command prompt just hangs with a blank screen for about 1-2 minutes with no visible progress until the build is ready.

A ./build.sh or go run build.go when there have been no changes takes less than a second for me, on my computer from 2015. A full git clean -fxd && go clean -cache && go run build.go takes about 25s, and that’s with rebuilding everything including the Go standard library. This also produces a lot of output because it’s doing a lot of things. There’s something strange with your setup that has nothing to do with the number of compilation threads. If you’re on Windows the first guess is obvious.

You can see what precisely takes time though, set BUILDDEBUG=1.

1 Like

I know what you mean, but not this time :sweat_smile:. Windows Defender is disabled with GPO and there is no other AV software on this machine.

This is what the actual output looks like with BUILDDEBUG=1.

go run build.go -goos windows -no-upgrade -goarch 386 zip

runError: git describe --always --dirty --abbrev=8
... in 83.0196ms
runError: git describe --always --abbrev=0
... in 60.0133ms
runError: git branch -a --contains
... in 116.0256ms
Notice: Next generation GUI will not be built; see --with-next-gen-gui.
rm -r syncthing.exe
runError: git describe --always --dirty --abbrev=8
... in 65.0155ms
runError: git describe --always --abbrev=0
... in 72.0152ms
runError: git branch -a --contains
... in 107.0247ms
runError: git describe --always --dirty --abbrev=8
... in 93.0199ms
runError: git describe --always --abbrev=0
... in 96.0223ms
runError: git branch -a --contains
... in 125.0302ms
runError: goversioninfo -o X:\syncthing\cmd\syncthing\resource.syso
... in 52.8538259s
runError: git show -s --format=%ct
... in 57.0119ms
runPrint: go build -v -trimpath -tags noupgrade -ldflags -w -X github.com/syncthing/syncthing/lib/build.Version=v1.14.0-rc.1.dev.133.gfb078068 -X github.com/syncthing/syncthing/lib/build.Stamp=1613411453 -X github.com/syncthing/syncthing/lib/build.User=tomasz86 -X github.com/syncthing/syncthing/lib/build.Host=tomasz86 -X github.com/syncthing/syncthing/lib/build.Tags=noupgrade github.com/syncthing/syncthing/cmd/syncthing
... in 2.7686189s
runError: signtool.exe sign /fd sha1 syncthing.exe
... in 5.0008ms
Codesign: signing failed:
syncthing-windows-386-v1.14.0-rc.1.dev.133.gfb078068.zip
... build completed in 57.8219399s

Apparently goversioninfo takes forever to run :slightly_frowning_face:.

Indeed! Now you know where to continue debugging.

Yeah, I will try to see what the problem is about. I am not sure about the git errors either.

At least with this knowledge, I can compile my own test builds without using goversioninfo altogether, which will reduce the process to ~7s.

Edit 1:

Without goversioninfo

... build completed in 861.1932ms

so this is the culprit for sure.

Edit 2:

Just for the record, building the binary for Android also takes 40+ seconds, but there the toolchain from the Android NDK is the culprit.

Interestingly, after updating Go to 1.16, the compilation time has dropped to only 5-10 seconds now (with goversioninfo enabled). If this stands, then I have no more complaints then :slightly_smiling_face:.

Hello, I’ve made an alternative to goversioninfo. It has a little more features and allows proper multilingual VersionInfo.

It can be used as a library too, so you don’t need a temporary JSON file. If you want to try it, simply replace the shouldBuildSyso function in build.go with:

func shouldBuildSyso(dir string) (string, error) {
	rs := winres.ResourceSet{}

	info := vi.Info{}
	info.SetFileVersion(version)
	info.SetProductVersion(version)
	info.Set(winres.LCIDDefault, vi.CompanyName, "The Syncthing Authors")
	info.Set(winres.LCIDDefault, vi.FileDescription, "Syncthing - Open Source Continuous File Synchronization")
	info.Set(winres.LCIDDefault, vi.FileVersion, version)
	info.Set(winres.LCIDDefault, vi.InternalName, "syncthing")
	info.Set(winres.LCIDDefault, vi.LegalCopyright, "The Syncthing Authors")
	info.Set(winres.LCIDDefault, vi.OriginalFilename, "syncthing")
	info.Set(winres.LCIDDefault, vi.ProductName, "Syncthing")
	info.Set(winres.LCIDDefault, vi.ProductVersion, version)
	rs.SetVersionInfo(info)

	icon, err := loadICO("assets/logo.ico")
	if err != nil {
		return "", err
	}
	rs.SetIcon(winres.ID(1), icon)

	sysoPath := filepath.Join(dir, "cmd", "syncthing", "resource.syso")

	syso, err := os.Create(sysoPath)
	if err != nil {
		return "", err
	}
	defer syso.Close()

	err = rs.WriteObject(syso, winres.Arch(goarch))
	if err != nil {
		return "", err
	}

	return sysoPath, nil
}

func loadICO(path string) (*winres.Icon, error) {
	ico, err := os.Open("assets/logo.ico")
	if err != nil {
		return nil, err
	}
	defer ico.Close()

	icon, err := winres.LoadICO(ico)
	if err != nil {
		return nil, errors.New("failed to load " + path + ": " + err.Error())
	}

	return icon, nil
}

If it works better for you, I might propose a PR.

I forgot imports :slight_smile:

"github.com/tc-hib/winres"
vi "github.com/tc-hib/winres/version"

Thanks for the alternative, but there’s no general problem with goversioninfo itself I think; it runs in 165 ms on our extremely slow Windows build VM. Tomasz’ problem is something else.

As I said a few posts above, the problem got fixed on its own after upgrading from Go 1.15.x to 1.16. The whole compilation now takes just a few seconds, so I can only confirm that the original issue has been fixed :slightly_smiling_face:.

Well, you actually had (have?) a subtle bug of goversioninfo: several tools like Resource Hacker or UPX would not be able to read Version Info from syncthing.exe. This is because of a bug in rsrc that was there from the start. I signaled it and akavel has fixed it in v0.10. But I suppose goversioninfo.exe still doesn’t correctly work because the author hasn’t updated go.mod?

Try to compress syncthing.exe with UPX. It will lose a few icon sizes and Version Info. This is not a bug in UPX, it’s a bug in goversioninfo, a simple lack of alignment. I wrote my own tool to free my mind after waiting for a long time for a fix from the authors.