Recently I am struggling a lot with not consistently reproducible test fails (examples https://github.com/syncthing/syncthing/pull/3996, https://github.com/syncthing/syncthing/pull/3986, https://github.com/syncthing/syncthing/issues/4017). The jenkings logs of these failures are often not very helpful, as they only print up to log level info.
Is there any reason not to enable both debug and verbose logging for testing?
Ridiculous amounts of log output, mostly. Usually it should be possible to reproduce a failure somehow locally?
Usually yes, but in case of both #3986 and #3996 I can’t reproduce the failures locally. I tried loading cpu and disk heavily during testing in the hopes of triggering it like this, but I can’t. The errors are anyway not (obviously) timing related, so I am currently at a complete loss as to what may cause them. That’s why I personally would rather filter through tons of output in 9 out of 10 cases, but have more information in the 10th case where it is (may be) relevant.
The kind of thing I am talking about just manifested nicely by not being consistent on jenkins either: I pushed merging master and a minor string change in javascript code to https://github.com/syncthing/syncthing/pull/3996. So the code after these two commits is essentially the same. Still the test failed on the first and succeeded on the second. That seems like a race to me, but there is nothing potentially racy (unless Sync in os does not actually block until it is really flushed on disk), there are no new goroutines and only blocking channel sends/receives involved in that test.
Is it reproducible (-ish) with the “syncthing-branch” job? You can enter “pr-1234” as the branch to build PR 1234. If so I can add a thing to set STTRACE to taste.
Edit: ah, let me set you up with an account on the build server or you can’t do that. I’ll do that this evening.