2.0.0-rc.22 Mostly Hangs On Saving Changes

Moisie · June 25, 2025, 8:28am

Hello!

I don’t know if this is a problem specific to my 2.0.0-rc.22 installation, but I’m having an issue where an action which brings up the ‘Syncthing is saving changes. Please wait…’ message (e.g. adding a new remote device to a folder, or unpausing a paused folder) seems to largely hang the entire operation.

I’ve left Syncthing in this state for hours - and it hasn’t changed; I’ve refreshed the web interface, and it loads into the all-folders-are-unknown state - and then never updates. Only a close-and-restart restores operation.

CPU and RAM load do not look at all excessive during this state; not sure if the disk activity is unusual - but the databases for any of the folders I’m testing with aren’t extraordinarily large.

Here’s a log - I unpaused folder y9qce-aaaaa at 07:11:30 (with a bit of preamble for context):

Wed Jun 25 06:30:50 IST 2025
Begin service_prestart
End service_prestart
Starting syncthing command /volume1/@appstore/syncthing/bin/syncthing serve --config=/volume1/@appdata/syncthing --data=/volume1/@appdata/syncthing
[start] 2025/06/25 06:30:50 INFO: syncthing v2.0.0-rc.22 "Hafnium Hornet" (go1.24.4 linux-amd64) builder@github.syncthing.net 2025-06-20 07:08:06 UTC [stguiassets]
[start] 2025/06/25 06:30:50 INFO: Automatic upgrade is always enabled for candidate releases.
[M45F6] 2025/06/25 06:30:51 INFO: My ID: M45F6VX-AAAAAAA-BBBBBBB-CCCCCCC-DDDDDDD-EEEEEEE-FFFFFFF-GGGGGGG
[M45F6] 2025/06/25 06:30:51 INFO: Hashing performance is 60.75 MB/s
[M45F6] 2025/06/25 06:30:51 INFO: Overall send rate is unlimited, receive rate is unlimited
[M45F6] 2025/06/25 06:30:51 INFO: Using discovery mechanism: global discovery server https://discovery-lookup.syncthing.net/v2/?noannounce
[M45F6] 2025/06/25 06:30:51 INFO: Using discovery mechanism: global discovery server https://discovery-announce-v4.syncthing.net/v2/?nolookup
[M45F6] 2025/06/25 06:30:51 INFO: Using discovery mechanism: global discovery server https://discovery-announce-v6.syncthing.net/v2/?nolookup
[M45F6] 2025/06/25 06:30:51 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[M45F6] 2025/06/25 06:30:51 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[M45F6] 2025/06/25 06:30:51 INFO: Anonymous usage reporting is always enabled for candidate releases.
[M45F6] 2025/06/25 06:30:51 INFO: TCP listener ([::]:22000) starting
[M45F6] 2025/06/25 06:30:51 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
2025/06/25 06:30:51 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
[M45F6] 2025/06/25 06:30:51 INFO: GUI and API listening on [::]:7070
[M45F6] 2025/06/25 06:30:51 INFO: Access the GUI via the following URL: http://127.0.0.1:7070/
[M45F6] 2025/06/25 06:30:51 INFO: My name is "My-NAS"
...
[M45F6] 2025/06/25 07:11:08 INFO: Additional connection (+161) for device SEPDJWL at 192.168.1.4:22000-192.168.1.52:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H7C47VV2QIBTS1RH415PH2
[M45F6] 2025/06/25 07:11:08 INFO: Established secure connection to UD7AGR2 at [::]:22000-192.168.0.31:22000/quic-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P20-62C6H7C4657A4D4CUMNI35GKI0
[M45F6] 2025/06/25 07:11:08 INFO: Additional connection (+150) for device UD7AGR2 at [::]:22000-192.168.0.31:22000/quic-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P20-62C6H7C4657A4D4CUMNI35GKI0
[M45F6] 2025/06/25 07:11:09 INFO: Abandoning old index handler for SEPDJWL (62C6H6SOTPORKD9H6QKU35E33C) in favour of 62C6H7C47VV2QIBTS1RH415PH2
[M45F6] 2025/06/25 07:11:09 INFO: Abandoning old index handler for 6FLFQJH (62C6H63K458LKTN6PA8B5VNH50) in favour of 62C6H78SQH0VID93BD01I0IMPK
[M45F6] 2025/06/25 07:11:09 INFO: Abandoning old index handler for UD7AGR2 (62C6H4ELLH5VQF9S0G606DTLS4) in favour of 62C6H7C4657A4D4CUMNI35GKI0
[M45F6] 2025/06/25 07:11:09 INFO: Abandoning old index handler for NN3I4DJ (62C6H664LVU64AI3KC82KTRPII) in favour of 62C6H7BE2SB3QBNEJTSQ8RJPOK
[M45F6] 2025/06/25 07:11:09 INFO: Device UD7AGR2 folder "Folder-Name-1" (y9qce-aaaaa) has a new index ID (0xAF160C3BD7312815)
[M45F6] 2025/06/25 07:11:13 INFO: Established secure connection to BJ6O5JO at [::]:22000-194.aaa.bbb.ccc:1053/quic-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P40-62C6H821Q2I1GG41GISITMP40E
[M45F6] 2025/06/25 07:11:13 INFO: Additional connection (+93) for device BJ6O5JO at [::]:22000-194.aaa.bbb.ccc:1053/quic-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P40-62C6H821Q2I1GG41GISITMP40E
[M45F6] 2025/06/25 07:11:14 INFO: Abandoning old index handler for BJ6O5JO (62C6H558UGDNCSSM8SA9RNM6IU) in favour of 62C6H821Q2I1GG41GISITMP40E
[M45F6] 2025/06/25 07:11:14 INFO: Device BJ6O5JO folder "Folder-Name-1" (y9qce-aaaaa) has a new index ID (0x80B824C3F86E6D72)
[M45F6] 2025/06/25 07:11:14 INFO: Established secure connection to UD7AGR2 at 192.168.1.4:22000-192.168.0.31:22000/quic-server/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P20-62C6H8502EPFSKD4QG9DMFJ9J8
[M45F6] 2025/06/25 07:11:14 INFO: Additional connection (+151) for device UD7AGR2 at 192.168.1.4:22000-192.168.0.31:22000/quic-server/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P20-62C6H8502EPFSKD4QG9DMFJ9J8
[M45F6] 2025/06/25 07:11:15 INFO: Abandoning old index handler for UD7AGR2 (62C6H7C4657A4D4CUMNI35GKI0) in favour of 62C6H8502EPFSKD4QG9DMFJ9J8
[M45F6] 2025/06/25 07:11:15 INFO: Device UD7AGR2 folder "Folder-Name-1" (y9qce-aaaaa) has a new index ID (0xAF160C3BD7312815)
[M45F6] 2025/06/25 07:11:18 INFO: Established secure connection to SEPDJWL at 192.168.1.4:22000-192.168.1.52:22000/quic-server/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P20-62C6H8K3ITCDCLFMEJMGO6OUOA
[M45F6] 2025/06/25 07:11:18 INFO: Additional connection (+162) for device SEPDJWL at 192.168.1.4:22000-192.168.1.52:22000/quic-server/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P20-62C6H8K3ITCDCLFMEJMGO6OUOA
[M45F6] 2025/06/25 07:11:19 INFO: Abandoning old index handler for SEPDJWL (62C6H7C47VV2QIBTS1RH415PH2) in favour of 62C6H8K3ITCDCLFMEJMGO6OUOA
[M45F6] 2025/06/25 07:11:29 INFO: Established secure connection to SEPDJWL at 192.168.1.4:22000-192.168.1.52:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9R1KG638TAFK1G2DAEQSI
[M45F6] 2025/06/25 07:11:29 INFO: Additional connection (+163) for device SEPDJWL at 192.168.1.4:22000-192.168.1.52:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9R1KG638TAFK1G2DAEQSI
[M45F6] 2025/06/25 07:11:29 INFO: Established secure connection to UD7AGR2 at 192.168.1.4:22000-192.168.0.31:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9R0EI6RILHDGJ36GMGQA4
[M45F6] 2025/06/25 07:11:29 INFO: Additional connection (+152) for device UD7AGR2 at 192.168.1.4:22000-192.168.0.31:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9R0EI6RILHDGJ36GMGQA4
[M45F6] 2025/06/25 07:11:29 INFO: Established secure connection to SDP4AYM at 192.168.1.4:22000-192.168.1.54:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9R29QOPETCNG09QCTI0T2
[M45F6] 2025/06/25 07:11:29 INFO: Additional connection (+47) for device SDP4AYM at 192.168.1.4:22000-192.168.1.54:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9R29QOPETCNG09QCTI0T2
[M45F6] 2025/06/25 07:11:29 INFO: Established secure connection to 6FLFQJH at 192.168.1.4:22000-192.168.1.101:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9RDMBUL6IS5STN0IMMVV4
[M45F6] 2025/06/25 07:11:29 INFO: Additional connection (+3) for device 6FLFQJH at 192.168.1.4:22000-192.168.1.101:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6H9RDMBUL6IS5STN0IMMVV4
[M45F6] 2025/06/25 07:11:30 INFO: Abandoning old index handler for SEPDJWL (62C6H8K3ITCDCLFMEJMGO6OUOA) in favour of 62C6H9R1KG638TAFK1G2DAEQSI
[M45F6] 2025/06/25 07:11:30 INFO: Abandoning old index handler for UD7AGR2 (62C6H8502EPFSKD4QG9DMFJ9J8) in favour of 62C6H9R0EI6RILHDGJ36GMGQA4
[M45F6] 2025/06/25 07:11:30 INFO: Abandoning old index handler for 6FLFQJH (62C6H78SQH0VID93BD01I0IMPK) in favour of 62C6H9RDMBUL6IS5STN0IMMVV4
[M45F6] 2025/06/25 07:11:30 INFO: Abandoning old index handler for SDP4AYM (62C6GUR7IQ36KC8M8VSL239LUA) in favour of 62C6H9R29QOPETCNG09QCTI0T2
[M45F6] 2025/06/25 07:11:30 INFO: Device SDP4AYM folder "Folder-Name-1" (y9qce-aaaaa) has a new index ID (0x08F718FA0D279036)
[M45F6] 2025/06/25 07:11:30 INFO: Device UD7AGR2 folder "Folder-Name-1" (y9qce-aaaaa) has a new index ID (0xAF160C3BD7312815)
[M45F6] 2025/06/25 07:11:30 INFO: Ready to synchronize "Folder-Name-2" (nzhy6-aaaaa) (sendreceive)
[M45F6] 2025/06/25 07:11:30 INFO: Device SEPDJWL folder "Folder-Name-2" (nzhy6-aaaaa) has mismatching index ID for us (0xCD7A8A514B1DCB28 != 0x42A675B41BEB4886)
[M45F6] 2025/06/25 07:11:30 INFO: Device SEPDJWL folder "Folder-Name-2" (nzhy6-aaaaa) has a new index ID (0x692030890D65307D)
[M45F6] 2025/06/25 07:11:31 INFO: Established secure connection to SEPDJWL at 192.168.1.4:22000-192.168.1.52:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256/LAN-P10-62C6HA3DLDS1IMDML7B14INN8C
[M45F6] 2025/06/25 07:13:50 INFO: Unexpected folder "Folder-Name-3" (zafbk-aaaaa) sent from device "BJ6O5JO-AAAAAAA-BBBBBBB-CCCCCCC-DDDDDDD-EEEEEEE-FFFFFFF-GGGGGGG"; ensure that the folder exists and that this device is selected under "Share With" in the folder configuration.
[M45F6] 2025/06/25 07:15:07 INFO: Unexpected folder "Folder-Name-3" (zafbk-aaaaa) sent from device "BJ6O5JO-AAAAAAA-BBBBBBB-CCCCCCC-DDDDDDD-EEEEEEE-FFFFFFF-GGGGGGG"; ensure that the folder exists and that this device is selected under "Share With" in the folder configuration.

The log would then continue with these last two messages periodically (folder zafbk-aaaaa had previously been removed for troubleshooting, but now I can’t add it back!), but with no other activity visible.

Am I alone with these issues? Happy to try and produce more detailed logs if that would help.

tomasz86 · June 25, 2025, 8:39am

What are your settings regarding multiple connections? By default in Syncthing v2, it should be set to 3, however in your log there are some crazy numbers recorded.

calmh · June 25, 2025, 8:57am

Yes, that. If you can take a goroutine trace / support bundle when it’s in this state that would be helpful as well.

Generally speaking, what it’s waiting for is for folders to cease doing what they’re doing and become idle. That can in some cases be a little while (on the order of a few seconds), but apparently something pathologic is going on here.

Hundreds of connections per device won’t help, one of the things a folder might be waiting for is outstanding requests, not sure those are hard cancelled. In which case there may be hundreds when there are this many connections.

Moisie · June 25, 2025, 9:15am

Yes - I saw those connection counts, and was wondering about those. At this stage I wasn’t sure if was the same connection count as had been increased in v2.

To the best of my knowledge I’ve not made any changes to the number of connections - for example, here are the settings for one of the noted devices:

I don’t think it’s going to allow me to do so - I’ve been waiting five minutes since clicking on the Support Bundle option so far, and nothing has come forth. I’ll keep waiting just in case…

Moisie · June 25, 2025, 9:29am

Oooh we’re in luck! DM’ing the support bundle to you (@calmh) now.

Moisie · June 25, 2025, 9:53am

Hmmmm sods law… Since unpausing the folder in order to provide fodder for the Support Bundle, Syncthing has now leapt into action, just to refute this thread.

I swear it was hung for hours and hours on previous attempts - but it’s possible you won’t find anything useful in the Support Bundle this time.

calmh · June 25, 2025, 10:42am

No real smoking gun in there, but maybe a couple of unlucky circumstances;

There are full index transfers going on for several devices, including dropping the indexes for those devices when they connect, this is likely the fix for the index issue in rc.21 but it makes for a lot of database activity, activity which can’t be interrupted in the middle.
There are a lot of (peer) devices doing this.
It appears to be a real slow computer, the reported hash rate is 18 MB/s. This is about the 5th percentile of reporting devices, meaning about 95% of the computers out there are faster. The median for the current population is 383 MB/s. This could just be that it had a lot of other stuff going on when the measurement was happening, but it still means Syncthing is or was rather CPU starved.
I suspect the ridiculous connection numbers are a bug/artifact of connections being blocked on these things when they come in, so they don’t actually start and therefore don’t count towards the total. That should be fixed, but also probably doesn’t make much difference here.

Moisie · June 25, 2025, 10:51am

Many thanks @calmh - and apologies if I’ve taken up your time for no reason. It was increasingly looking like a showstopper here, and I wasn’t sure how many others are testing v2 on a large dataset (not that I’ve yet set all of it running under v2 though).

It sounds like I just need to be patient and let these all complete, ideally before making any changes to the config - does that sound about right? Happy to do so if that’s the answer.

Moisie · June 27, 2025, 8:01am

Ok - so I got itchy…

I changed numConnections from 0 to 1 for all my remote devices and tried again. Now tasks which prompt the ‘Syncthing is saving changes. Please wait…’ message complete in just a few minutes - adding a new remote device to a folder, unpausing a paused folder, renaming a remote device all working well.

So - I have a workaround that’s good for me - thanks for pointing me in the right direction.

Based on this experiment, I’m less convinced about this not making much difference. I am minded by your note in Connection accounting doesn't properly account closed connections · Issue #10150 · syncthing/syncthing · GitHub however…

While this gauge is not that important, it may be a symptom of a more serious problem in connection close handling, since the accounting code itself is quite straightforward.

calmh · June 27, 2025, 9:30am

Yes, I believe you’re right. There’s a bug lurking