Syncthing verifies device ID of the other peer before accepting connections.
The device ID is indeed a hash of certificate in some format.
Mutual TLS is an industry standard. Why not use GO stack on mTLS and check the TLS certificate of the other peer?
The advantage is that, Syncthing will not do custom security. It will be standard mTLS, standard tools parsing and verifying TLS certificates (not device IDs). Essentially an SSL VPN.
You’d have to carry around the entire certificate of the other peer for verification. They are inconvenient to type and tedious to transfer over the phone.
I would disagree here. In my understanding, mTLS isn’t a “standard”, it’s an operating mode of TLS (which syncthing already uses). RFC8705 is supposedly one of the closest things to an actual standard for mTLS. (It describes mTLS in combination with OAuth). And it just happenes to describe a hash-based certificate authentication method in relation to mTLS that is quite similar to syncthing (except that syncthing doesn’t use JWTs). RFC 8705: OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens
I agree that syncthing shouldn’t roll their own crypto, but that’s not what happens here: all of that happens in normal TLS stacks. The only thing that happens outside of the standard stack is certificate verification, which is actually quite common outside of webPKI environments. Syncthing also relies on having temporarily unauthenticated TLS connections in order to receive new device prompts.
The risk here is low:
syncthing could forget to verify hashes
syncthing could improperly compute hashes
Both of these are easily caught with simple tests. Cryptographic issues such as side channels are not relevant in hash based certificate authentication.
The downside of ripping everything out is that new device prompts will get really ugly and more users will struggle to distribute full certs across devices. In my opinion this is a usability vs security question, which in this case has a minor security risk vs a large usability problem.