I had two devices setup. Both had syncthing installed and running. They were on the same network and also happened to both have hamachi installed and be connected on that. Syncthing decided to send the file over the network interface of Hamachi.
Can I somehow just tell syncthing to not connect or do anything over that network interface?