I have just installed Syncthing on my I5 3470 unRAID Server & on my I32100 HP Gen8 Microserver with the intention of syncing all my media folders (approx 10TB) from the unRAID server to the Microserver.
I have configured the folders but am confused as to what it is actually doing? (it has been running about a day now and reporting scanning 22% on the unRAID server and syncing 97% on the Microserver.
Can anyone please tell me what it is scanning? why is it taking so long? I have used freefilesync/rsync in the past and I could have completed the job twice by now using those programs?
Lastly, I always like to verify newly copied files (I do not want to copy all the same files each time) can someone please explain how this works?
Syncthing is hashing all of your files. It builds up a database of all of your files, including some hashes which can be used to see if the content of a file has changed. This will take a long time if you have a lot of data, a slow processor, or both.
Once it’s finished doing this, it can get to work synchronising.
Although this approach means you may have to wait for a while when you first set up syncthing, it means that Syncthing can easily tell which files have changed, or even which parts of a file have changed. This means that it can quickly determine what data it needs to transfer, and will only transfer data which actually needs transferring.
Sure, you could have run rsync several times in the time taken for Syncthing to complete it’s hashing phase. But after this, Syncthing will be able to synchronise both sides in a very short period of time, whereas rsync will always take a relatively long time.
I guess I will just have to wait a few days then! it does only do the hashing once then? also, can you please comment on the file verification? does it only verify new files? is that what the hashing is doing?
After the initial scan/hash Syncthing will check periodically if the size or timestamp of the files have changed. If so it will hash those files again to detect what changed.
Also, after detecting a change, Syncthing will send the file info and the list of hashes to the other nodes. The other node will then request the data to get the new file / update an existing file. When receiving the data, it will than check if the hash is correct.
Sorry to be a pain but it finished scanning a 4-5 hours ago the percentage went to 100% and then it changed to just “scanning” and is still sat there like that? it is not syncing any new files, what is it doing now?
My best guess it’s that it’s reconciling it’s view of the repo with the other side, which can probably take a while given the size. You might want to change the scanning frequency to something way higher than a minute as well…
I have been looking for answers on the scanning frequency but googled out! there are so many different opinions on the frequency and also depending on what version I have… very confused by it all! I was hoping this program would allow me to set a source & destination directory and I would not have to worry about it but all I have done for days is watch it say scanning… I don’t mean to sound negative so sorry if I do but I am doing an awful lot of waiting without any on-screen notifications/updates… what should I change the frequency to? will it take over a day each time it runs? it will kill my drives if that is the case surely?
It should only have to so it once.
The speed of this is relative to the CPU speed.
If the files don’t change often, set it to once a day. If they do change often, then perhaps using syncthing-inotify utility alongside (or synctrayzor) is a better option.
Let me amend that. 34k files really isn’t that many, so the periodic scan should be quite fast - less than a second if the metadata is in disk cache. However 6618 GB is a shitload of blocks which makes a lot of database operations and the index exchange heavy. You can probably expect some sluggish performance all around. Stay tuned for a future version with variable block size better tuned for handling many largeish files.
ok, so my files range from 1mb to 5GB so basically you are saying my data is too much for Syncthing in it’s present state? should I switch back to an alternative program until Syncthing is upgraded? my hard disks are just being hammered constantly… I have changed the scan to 1 day now though so basically any files that I change will not be picked up until the 24 hour cycle occurs which is more than I would like as I was after something more instant like freefilesync livesync…
Sorry but if you have explained how to have “live sync” I cannot see that anywhere?
I hope you can see my point of view as I am an inexperienced user and although you and calmh are developers, calmh says I have a shitload of blocks but then you are telling there are users with larger amounts of data… I don’t mean to frustrate you but I am not up to speed on the terminology, so sorry for that… I think I misunderstand because synctrayzor is a Windows app but I am on Linux (unRAID)
After several days of my hard disks and CPU being stressed to the hilt it finally started syncing last night… I woke to find that after syncing all night it had only managed to sync 200GB and my CPU was still be stressed out (this is primarily a Plex Server) so I have now decided to abandon Syncthing and am syncing with FreeFileSync which is trickling my CPU (which is what i need) so maybe Syncthing was just not the right product for me but I wanted to thank you for your very fast support and may see you in the future if Sycthing develops into something I need.
it does only do the hashing once then? also, can you please comment on the file verification? does it only verify new files? is that what the hashing is doing?