Unfortunately I didn’t manage to find the culprit of my problem.
Out of a sudden Syncthing on my Raspberry got stuck and does not want anymore.
It is not able to connect to any other device, neither via internet nor in the local network.
Firewall is non active and the required ports are obviously open and even connections established.
As a sanity check, can you actually reach Device 4 (or any troubled device/external-network really) from the raspberry, consecutively and without issues? There are several no route to host log-lines present which could imply some underlying but local connectivity issue (outside of Syncthing). A simple long-lasting ping would suffice.
Ping to 34 does also works from the Pi without problems, only the latency with 18-200ms is surprisingly long, but no drop out.
Could let it run for more than 5 minutes maybe.
The only thing fixing it was deleting the Syncthing config and setting it up again - but with copying the old .pem files to not needing to set it up on the other clients again.
The only difference I found between the old and the new config so far was the backup mechanisms per folder.
Nothing else (e.g. firewall, network config, permissions) changed.
Maybe in the index file there was an issue.
Will try to investigate further that this bug could possibly be avoided in the future.
There is no network address information in the database.
Your instances are most likely all llstening on the same port, and try to map the same port on the router, which causes non-determinism, and potential routing to itself.
That sounds very plausible.
Any hints on how to avoid it?
I have not changed anything network related, especially not on Syncthing on the Pi.
But I have to admit I did one mistake:
On the Pi a file conflict appeared (that happens occasionally) that stops it from further syncing. It always stops at one single file and never gets the conflict resolved until I manually delete it on the Pi.
On the Pi all files are encrypted as it is my “untrusted server”. It’s also only smaller files, no big ones; additional the trusted instances do not run into similar problems (like this unsolvable file conflict, nor the self connect).
The latter part for information in case it could potentially have caused it.
If devices are in the same lan, you probably want them to listen on different ports. Docs cover that.
But also the fact that you are hitting this issue suggests that your network is misconfigured, and they can’t connect directly and are trying to connect via “internet” (router in this case I guess), instead of directly.
But regarding issues with the LAN I would not agree.
Following an example log (the one on the Pi looks identical); everything is discovered nicely via local discovery:
2023-10-17 22:37:14 My ID: <1>
2023-10-17 22:37:15 Hashing performance is 79.87 MB/s
2023-10-17 22:37:15 Overall send rate is unlimited, receive rate is unlimited
2023-10-17 22:37:15 TCP listener ([::]:22000) starting
2023-10-17 22:37:15 Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
2023-10-17 22:37:15 QUIC listener ([::]:22000) starting
2023-10-17 22:37:15 Listing network interfaces: route ip+net: netlinkrib: permission denied
2023-10-17 22:37:15 Detected 0 NAT services
2023-10-17 22:37:15 Ready to synchronize Sync (sendreceive)
2023-10-17 22:37:15 Using discovery mechanism: global discovery server https:/ /discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
2023-10-17 22:37:15 ...
2023-10-17 22:37:15 Using discovery mechanism: global discovery server https:/ /discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
2023-10-17 22:37:15 Using discovery mechanism: global discovery server https:/ /discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
2023-10-17 22:37:15 Using discovery mechanism: IPv4 local broadcast discovery on port 21027
2023-10-17 22:37:15 Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
2023-10-17 22:37:15 Ready to synchronize "dcim" ... (sendreceive)
2023-10-17 22:37:15 GUI and API listening on 127.0.0.1:8384
2023-10-17 22:37:15 Access the GUI via the following URL: https:/ /127.0.0.1:8384/
2023-10-17 22:37:15 My name is "xyz"
2023-10-17 22:37:15 Device <2> is "rpi" at [dynamic]
2023-10-17 22:37:15 Device <3 offline> is "sSSD" at [dynamic]
2023-10-17 22:37:15 Device <4 offline> is "t450s" at [dynamic]
2023-10-17 22:37:15 Device <5 offline>is "s9s" at [dynamic]
2023-10-17 22:37:15 Device <6 offline>is "sUSB" at [dynamic]
2023-10-17 22:37:15 Device <7 offline> is "abc" at [dynamic]
2023-10-17 22:37:15 Device <8> is "s7" at [dynamic]
2023-10-17 22:37:21 Established secure connection to <2> at 192.168.178.39:22000-192.168.178.33:22000/tcp-client/TLS1.3-TLS_CHACHA20_POLY1305_SHA256/WAN-P30
2023-10-17 22:37:21 Device <2> client is "syncthing v1.25.0" named "rpi" at 192.168.178.39:22000-192.168.178.33:22000/tcp-client/TLS1.3-TLS_CHACHA20_POLY1305_SHA256/WAN-P30
2023-10-17 22:37:21 Established secure connection to <8> at 192.168.178.39:22000-192.168.178.34:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30
2023-10-17 22:37:21 Device <8> client is "syncthing v1.23.7" named "s9s" at 192.168.178.39:22000-192.168.178.34:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30
2023-10-17 22:37:35 quic:/ /0.0.0.0:22000 detected NAT type: Port restricted NAT
2023-10-17 22:37:35 quic:/ /0.0.0.0:22000 resolved external address quic:/ /109.250.26.112:63478 (via stun.syncthing.net:3478)
2023-10-17 22:37:37 Completed initial scan of sendreceive folder Sync
2023-10-17 22:37:57 Joined relay relay://77.22.105.45:22067
2023-10-17 22:38:13 Completed initial scan of sendreceive folder "dcim" (xgk6v-v8a6z)
In which case, the connect to self might be a red herring somehow.
But effectively I assume when you’ll see connected to self, it will be a non-lan ip that points to yourself, due to not being able to connect via lan.