I had a look at codecov.io and it is very simple to set up. I think it could be useful to visualize our test coverage, especially for diffs on PRs. I am aware that coverage doesn’t mean, that the code actually does what it should, but no coverage means it’s certainly untested and that maybe warrants special attention then.
All that’s required is to run tests with coverage recording enabled and a little bit of scripting and calling a bash script to upload to codecov. That can either be a separate build step/job on the CI or adding the necessary options to one of the existing test stages. Both could be done as an additional command in build.go or a separate script. I tested it for unit tests, but it should even be doable on the integration tests as well.
What do you think?
I think it needs proprietary binaries or some
curl foo | sudo bash which sucks. I guess if I get this “for free” without having to do anything, sure.
What does it need proprietary binaries for? You just need to run a bash script, for which the “source” code is known, and apache licensed: https://codecov.io/bash
And sure, I don’t propose it and expect others to do it
I probably wont do it with high prio right now myself, just want to know where I stand if I find myself with some time and motivation for it.
This is really simple: Either we just enable a travis build for it (i.e. no new work for teamcity) or add a build step/job to teamcity (which requires to put a secret token somewhere on the build server). In either scenario it can just “run in the background”, providing coverage data, or a bot can be activated which posts on PRs with information about the coverage change and how much of the diff is covered (example: https://github.com/imsodin/syncthing/pull/4). So two things to decide:
- Travis or TeamCity?
- PR-bot or not?
PS: Our current coverage of
lib/ is just shy over 50%.
@calmh @AudriusButkevicius Is there maybe a way to stop codecov from posting to github and show up in the checks, but still work, i.e. on https://codecov.io/gh/syncthing/syncthing commits don’t show up as “Unknown”. I think the info can be useful at times and I have slights hope of reducing the flakyness at some point
I just revoked its permissions for syncthing/syncthing as the comments were … less than illuminating. (No comment on the rest.)
That prompted my post, but I shouldn’t have asked and just looked it up myself. I’ll change codecov to not comment and show statuses, then it’d be nice if you could give the permissions again.
Sure! I’ve reinstated it. I’m keeping an eye on it though.
It better stay calm.
Codecov interface shows that our repository was affected by their recent blunder. It looks like our TeamCity setup didn’t allow it to exfiltrate anything important, but I’m not too much versed in TeamCity to be sure about that.
We are safe, right?
Yes, we’ve discussed this internally and came to the same conclusion.