I’m using Syncthing to sync an external HDD with another computer. The HDD is mounted on /Volumes/Elements SE (typical macOS mounting point). Which “Elements SE” is the label of the HDD. The file system I’m using on the disk is ExFAT.
After I removed the HDD, Syncthing immediately knew there was something wrong and told me “Folder path missing”. But I need to manually rescan the disk after putting it back. Otherwise, Syncthing won’t know it’s back. This is the problem I described in that issue. It’s normal in the current version. If I do that manually, it will pick the folder again.
After I left the disk unattached for a day, maybe after some system reboots, I found that Syncthing started to scan the folder, with the progress bar at about 10%. (But there is no progress text after the “Scanning” string.) Even though the disk is unmounted, therefore the folder does not exist at that time.
Then I tried to put the disk back. But Syncthing was still stuck at the scanning state and didn’t back to normal for a long time. So I tried restarting Syncthing, pause resume, etc. I even attempted to edit the ignore pattern, because it can trigger a rescan. However, it also stuck at “Loading…” forever.
Finally, the real scanning started with a (0%). But sadly, it didn’t just finish like before. It started to reindex the whole disk all over again. It will take a whole day to finish, and still working on it while I’m typing this. (Now it’s 90%)
So the problem is:
Why did Syncthing stuck on scanning even though there is no folder? (It should just say stopped until I put the disk back.)
Why did it try to reindex the disk all over again?
I think generally yanking drives half way through io operations can probably lead to processes to hang on io. It’s quite hard to understand what exactly happened here, but I suspect just disconnecting the drive at the “wrong time” can lead to sadness of this sort. I suspect just restarting syncthing could probably recover it, and pausing the folder before removing the drive is probably the right approach here.
I did remember I disconnected the drive while seeing syncing was in a ready state though.
By the way, even if I pause the folder before disconnecting and resume after connecting, it can still consume a long time. Just walking through the drive cost about 10 minutes. So every time I want to add a file to it and sync it. I need to wait 10 minutes before Syncthing is ready. Even I just want to sync a 1MB file.
I was just switched from Resilio Sync. I can disconnect the drive any time. And it can back to work immediately after I connect my drive. I kinda want a smooth experience like this. There are lots of stuff that need to improve when dealing with removable drives.
However walking the filesystem should not take too long, if you don’t have a crazy amount of files.
Also don’t know much about exFAT, but if it has similar issues with timestamp accuracy as FAT32 does, it could be that syncthing sees everything as changed and spends 10min rescanning everything.
Plugging/unplugging drives is not really the target use case, but we welcome any contributions/PRs that makes it better.
It didn’t take that long before. I remember it was like less 1 minute to walk through the drive. But after Syncthing did that whole day indexing, now it’s longer. I don’t quite know why.
I’d like to contribute. But I never wrote golang. Maybe in the future, I will give it a try. Now I’m using a script to pause the folder when I unmounted the drive, resume when I mounted the drive.
There is no way an automation script knows I want to unmount a drive before I do that. But because I’m monitoring system events, it almost happens immediately. I tried several times today, and it works.
Syncthing does not differentiate between internal and external drives, but other than safety checks to handle drives that have been unplugged (i.e. when the sync folder suddenly disappears), use-cases where drives are disconnected for relatively long periods of time require more manual intervention.
According to the screenshot, the Syncthing folder is set to “Send Only”, so it is going to ignore any changes from “Home-Server”.
Of the files that are being missed by Syncthing, are they new files added to the external drive or are they updates to existing files? Do any of the missing files match the ignore pattern(s)?
Also, what is the model number of your Western Digital Elements SE drive?
On this side, it’s send only. On the other side (Home-Server), it’s receive only. Because this sync is for backup purposes. This should not be a problem.
I made many changes recently. Including add, edit, delete, rename, etc. But the state of the folder on “Home-Server” is almost staying at the old state several days ago.
My ignore patterns are just for system files like (.DS_Stores). So this should also not be a problem.
The model for my drive is “WD Elements SE 25FE Media”. I’m kinda wondering if this might be an ExFAT problem. So I’m trying to re-format the drive with APFS and copy all files back. To see if this fixes the problem. This takes forever though.
Years ago, I recall an issue with BTSync on early macOS where it was configured to replicate a mounted volume – call the volume “archive”. When “archive” was unmounted while BTSync was running, BTSync would create a new file at the mount-point /Volumes/archive. This led to all sorts of crazy, unexpected behavior such as delayed file reads, etc.
And so to your issue, while Syncthing is running and your external volume is unmounted, does ls -la /Volumes show an entry for your unmounted drive anyway?
I’m quite sure Syncthing is not trying to create the same folder. But what you said reminded me of something. Thank you so much!
I did use network sharing to access the synced folder on “Home Server”. When mounting a network-shared folder, macOS mounts the disk as a volume in /Volumes. And because it has the same name as the source drive. Syncthing might just think they are the same! Then it started to scan the remote disk as if it were the removable HDD! This can also explain the slow scanning speed.
This is likely the reason I had this particular problem. I’ll try to change the name of the shared folder from “Home Server”, and see if this happens again.
But because I’m monitoring system events, it almost happens immediately. I tried several times today, and it works.
