I had a look at codecov.io and it is very simple to set up. I think it could be useful to visualize our test coverage, especially for diffs on PRs. I am aware that coverage doesn’t mean, that the code actually does what it should, but no coverage means it’s certainly untested and that maybe warrants special attention then.
All that’s required is to run tests with coverage recording enabled and a little bit of scripting and calling a bash script to upload to codecov. That can either be a separate build step/job on the CI or adding the necessary options to one of the existing test stages. Both could be done as an additional command in build.go or a separate script. I tested it for unit tests, but it should even be doable on the integration tests as well.
What does it need proprietary binaries for? You just need to run a bash script, for which the “source” code is known, and apache licensed: https://codecov.io/bash
And sure, I don’t propose it and expect others to do it
I probably wont do it with high prio right now myself, just want to know where I stand if I find myself with some time and motivation for it.
This is really simple: Either we just enable a travis build for it (i.e. no new work for teamcity) or add a build step/job to teamcity (which requires to put a secret token somewhere on the build server). In either scenario it can just “run in the background”, providing coverage data, or a bot can be activated which posts on PRs with information about the coverage change and how much of the diff is covered (example: https://github.com/imsodin/syncthing/pull/4). So two things to decide:
Travis or TeamCity?
PR-bot or not?
PS: Our current coverage of lib/ is just shy over 50%.
@calmh@AudriusButkevicius Is there maybe a way to stop codecov from posting to github and show up in the checks, but still work, i.e. on https://codecov.io/gh/syncthing/syncthing commits don’t show up as “Unknown”. I think the info can be useful at times and I have slights hope of reducing the flakyness at some point
That prompted my post, but I shouldn’t have asked and just looked it up myself. I’ll change codecov to not comment and show statuses, then it’d be nice if you could give the permissions again.
Codecov interface shows that our repository was affected by their recent blunder. It looks like our TeamCity setup didn’t allow it to exfiltrate anything important, but I’m not too much versed in TeamCity to be sure about that.
