To uninstall, you need to remove;
- The
syncthing.exebinary, wherever you put it - The syncthing config dir in
%LocalAppData%/Syncthingor%AppData%/Syncthing(depending on Windows version)
That leaves an .stfolder marker (as well as a possible .stignores) in your actual older as well, which is hidden. It doesn’t do any harm, but removing it would be the final step.
All of this could/should be handled by an installer as you mentioned, or for that matter by a syncthing -uninstall option as syncthing itself knows about all this…