tl;dr I was wondering if there were ways to accelerate the scanning process on a new Synching instance set to receive only? Since the sending device has backed up most things I am OK with only receiving “new things” going forward but I am not sure if it’s possible to distinguish new vs old (and if so how to do it.) My main obj is to accelerate getting back to “actively syncing” and to reduce the time scanning is taking right now since Syncthing is currently not syncing anything and at the current rate it will be weeks before it starts up again.
Longer story:
I’ve been running Syncthing on my Synology but decided it’s probably time to set it up on my Mac mini (since it’s no longer supported on DSM 7 and I’ll likely want to upgrade sooner or later.)
I have Syncthing running on a hosted cloud service (set to send only) with ~1TB of data. I’d say 99% of the data has already been synced down to my Synology. I recently disabled Synchthing on my Synology NAS and am set it up on my Mac mini (set to receive only) and pointed it at the pre-existing folder on my Synology NAS with all the backed up local data Synchthing on my Synology had been managing. When I setup Syncthing on my Mac mini it turns out scanning is a lengthy process! Current Scan Time Remaining ~ 16d 7h. Yikes!
For the time being I have decided to re-enable synchthing on my synology (just to have coverage during the scanning process on my Mac mini.)
Question:
Is there a way to tell the Mac mini “the synology folder is up-to-date with the cloud based folder” please stop scanning and only pickup new items? If so, how would I do that? Thanks for any tips/suggestions. And, sorry for now stating this more eloquently. Syncthing has been rock solid and one of my favorite pieces of open source software. True life saver.
If you’re up for some hackishness that may or may not work, you could in principle copy the entire database from one device to another. Any deviations in the actual local data will then be picked up as new changes, which you can then revert because you’re in receive-only mode.
I haven’t tried this myself for a long time. There may be dragons.
Oh, and if the data is only “almost” the same (say, the modification timestamps are different because the filesystem on one side supports nanosecond precision and the other side just microsecond precision, or the permissions are different) then that data will anyway be rescanned since it doesn’t match what’s in the database. You could possibly set the mod time window and ignore permissions options to counter this.
You haven’t given specifics on the hardware in question, but is the Mac mini a new machine or something old and slow? If it’s the latter, you may want to check https://forum.syncthing.net/t/optimising-syncthing-for-low-end-hardware/14885 (written by myself, in full disclosure) for some tips how to optimise Syncthing for slower hardware in general.
In particular, judging by the scan time remaining, I’d assume you’re dealing with spinning drives here, which means you may want to limit maxFolderConcurrency to something like 2 or even 1. Rotating disks don’t play nicely with a lot of threads trying to scan files in different locations, which is normally the case with modern multi-core CPUs.
Also, what I personally like to do is to split data into many smaller folders rather than using fewer large ones. This way a single large folder stuck in scanning won’t block the others from operation.
Syncthing is running on a Mac mini (2018) with 3.2 GHz 6-Core Intel Core i7 and 32 GB 2667 MHz DDR4. It’s connected via ethernet to a Synology ds-418play with 2gb memory and ~ 24tb of WD Red (spinning) drives. There was approx 2tb of data in the cloud (and a local folder on the synology with a lot of folders and probably hundreds of GBs maybe a TB of data.)
Scanning was blocking a MacOS security update (amongst other things) and I decided after watching it slowly making progress over many many days to just take an extreme step. I deleted a bunch of the backed up data in the cloud and moved local files to another folder.
This had the desired outcome of significantly speeding up scanning (it more or less become a several minute exercise.) My main obj here is to just migrate Syncthing to running on my Mac so I can consider updating my Synology (which will sadly prevent me from running Syncthing on it - which would be my preferred local host.) I’ll see how this runs for awhile before committing to this approach. I can’t determine precisely why Syncthing runs more efficiently on my synology as the Mac mini isn’t that old. I def could be wrong but I assume the network transport layer is causing the majority of the lag. While I’ve setup the synology to use both nics as one I doubt that is doing much good here.
As an aside I did try the config proposals (thanks for linking to them) before making these changes but they seemed to have no noticeable impact. So, I reverted to the defaults for now. I am kinda curious how this will run over time…
I do hope someone from the community is inspired to make a package that runs on the latest syno update.
Eek. Now my speed issues appear to be during writing!
I guess running syncthing on MacOS connected to a NAS via ethernet is not the ideal setup.
I sure miss having Syncthing running locally on Synology. Performance wise, this is a massive step backwards (for me). I seem to have gone from the ability to download and have Mbps writes to kbps.
I can’t compare this with the setups I’m currently running because I don’t own a Synology. I’m the owner of the Syncthing macOS project and I use multiple homebuild NAS systems. One is running Debian 64bit with 4GB RAM on intel atom. The other one is a FreeBSD machine with 16GB RAM. Both systems have western digital red disks. The Debian one runs BTRFS filesystem for Syncthing and the FreeBSD one uses ZFS. I think you are stuck with the fact how your Synology disk setup (software RAID) or something else is causing disk I/O or ethernet latencies. This can be a deep problem in the configuration and even the Linux kernel.
My answer is not much of help because you can not run custom firmware/inspect all the things on the synology which is the whole point of user friendlyness. Have you tried getting help on the synology forum about your RAID5 setup?
I have seen some issues past sunday with my Debian BTRFS NAS which caused the whole system to break down and be unreachable. BTRFS does some cleaning and scanning but the scanning kernel process was eating my CPU and RAM away. So I had to disable btrfs qgroup on the volume. With your setup this is probably not the case, but can be a problem if the RAID5 is made with BTRFS or mdraid. Not sure which variant Synology is using.
I don’t follow here. As written in another thread, there is nothing wrong with Syncthing on Synology’s DSM 7 update. It works fine just as always.*
So you are not actually running it there anymore? I understood you were going to update the DS418play when the sync has finished? If not, how are you actually using the DS then? If by any chance you are using Syncthing over a network file system (Samba, network drive, whatever they call it on MacOS) then you’re certain to get an awful performance.
It would help if you describe your setup in more detail, as it’s getting more and more confusing for me at least.
(*) If you upgrade from DSM 6 to 7 with Syncthing installed, you just need to make sure not to click “Repair Package”.
I misunderstood and didn’t realize DSM 7 had synocommunity Syncthing support. I just updated to DSM and synching is working much better than running on my Mac mini. I am back to getting consistent 20+MiB/s download speeds! Somehow the latest synchthing update did wipe my prior settings but that’s fine, I am super happy to have Syncthing running on my synology again. Only took a few min to update, fix some permissions that changed and get back up and running.
