I am in the process of moving from BT Sync to Syncthing and am wondering why I observe the transfer of files although the very same files are already present. There is a chance though that I misinterpreted the info of the web UI but different files show up as “Last File Received” and also the “Download Rate” counter seems rather high.
Besides that question I have, I want to offer to support in narrowing down why this is happening, but I need some expert advice on what to log in which way. I have not yet moved all folders from BT Sync to Syncthing and I would expect to see the same behavior (transmission of files which are already present) again.
During the switch form BT Sync to Syncthing I struggled with some issues which I seem to have fixed because after the files are synced again everything seems to work fine.
My setup:
iMac running OS X 10.9.5 and Syncthing v0.11.7, Mac OS X (64 bit)
DiskStation1 (Synology DS412+) running DSM 5.2 update 1 and Syncthing v0.11.7, Linux (64 bit)
DiskStation2 (Synology DS415+) running DSM 5.2 update 1 and Syncthing v0.11.7, Linux (64 bit)
On the DiskStations I installed the Syncthing package of SynoCommunity which installs v0.10.30-3. The automatic update did not work at first, therefore I logged in via SSH and ran Syncthing with the --upgrade option. That installed v0.11.6 which then automatically updated to v0.11.7.
Preparing access rights on the DiskStations:
The folders and files to be synced are already present on all devices. That’s been done with BT Sync. On the iMac both BT Sync and Syncthing run as the logged in user, therefore no ownership or access right had to be changed. On the DiskStations BT Sync and Syncthing run under their own user and therefore all files created by them are owned by their user. Therefore I chowned all files and folders created by BT Sync to be owned by the user syncthing (had to learn this first by failing sync attempts). Also for the so called shared folder on the DiskStations I granted change access to the group sc-syncthing.
Preparing the devices:
First I removed the folders from BT Sync control and deleted the .sync folder used by BT Sync. Then I created an .stignore file on all devices in the folder to be synced ensuring the correct owner on the DiskStations. On each device I then added the new folder and gave it the same folder ID without sharing it with the other devices. After all devices completed scanning the folder, I verified that they showed the same number of items in the “Local State”.
Sharing the folders:
After the devices where prepared in the above way, I started on the iMac to share the folder with the DiskStations. Interestingly DiskStation2 accepted the sharing without the need to acknowledge it while DiskStation1 asked the user on the web UI to accept the sharing from both iMac and DiskStation2. I could not see where there is a different configuration explaining the different behavior.
Then after a little while the sync process bar dropped to a low percentage and files started to show up at “Last File Received”. It took a while to complete and I am sure that only a view files showed up at “Last File Received”, but I would have expected that no files needed to be transferred between the devices.
Before I continue with the remaining folders I decided to share this observation. Maybe some of the developers are interested in having some logs to understand what is going on. Or maybe it’s just the normal behavior?
You can set STTRACE=model environment variable which will give verbose logs of whats happening, I assume it will print out a lot of taking shortcut.
Also, at the point you start syncing, you can click on the Out of sync text to open a modal which shows progress for each file, given it’s actually being retransferred.
If you have a lot of files, it’s likely that the network usage went up because we were exchanging index, the fact that Last file received is changing does not mean that we are transferring data, it could mean that we are just transfering metadata (such as modification times etc) which is part of the index.
Thanks a lot for the quick reply! I’m learning a lot these days. It seems as the network usage is caused by exchanging index. I’m seeing a lot of taking shortcut in the log. Will let it complete and get back if I have more questions.
Regarding the automatic accept by DiskStation2 of sharing requests:
I found the difference in configuration which might be responsible for that. On DiskStation2 I marked iMac to be an Introducer. This setting was missing on DiskStation1. The short description on the web UI for Introducer says only something about devices, but it seems to be valid for folders as well.
However to start syncing, DiskStation2 needed a restart of Syncthing which was not proposed by the web UI after it automatically accepted the sharing.
AFAIK we never auto-accept folders.
Given two nodes connect, they both express what they share with the other, which causes prompts in the UI saying X wants to share Y with you, accept? which you might have clicked to add the share.
But the folder has to be shared with the remote device to start with.
Yes, I saw this dialogue on DiskStation1, where I accepted it, but I did not see it on DiskStation2. That’s why I was confused and mentioned it on my original post.
I set the Introducer setting on DiskStation1 to Yes after I shared the latest folder, but there are more folders to be moved from BT Sync to Syncthing, so I will be able to see if this setting will also make DiskStation1 to auto-accept the sharing.
From my user point of view it seems to behave differently.
Thats what I observed when I added the next folder (Video) to the sync setup:
as a preparation on all devices the folder is added to Syncthing without sharing it first
after scanning is complete on all devices, on iMac I enabled sharing of folder Video with the other devices
all other devices have set the iMac as the Introducer
all other devices seem to automatically accept the sharing offer, because without any user action the status of the folder switches to a green up-to-date in the web UI and the Shared With info lists the other devices (see screenshot below)
after things have settled down, all devices show an incomplete syncing status (see screenshot below)
I assume that the syncing will complete properly as soon as I restart Syncthing manually on DiskStation1, DiskStation2 and MacBook
(Note: In the meantime I added MacBook to the sync setup. MacBook will only sync some of the folders. The new folder Video I have just added is also synced with MacBook.)
From my point of view this is not a bad thing, but as I understand you it is not the behavior you expect. Also what is missing is the modal on the web UI of the other devices, which tells the user that he should restart Syncthing to complete the setup.
I took screenshots of the web UI (see below) and also should have logs (model, scanner) of all devices if that would be of any help.
Now I am going to restart Syncthing on Diskstation1, Diskstation2 and MacBook and would expect that after a while syncing will complete.
Here are the screenshots:
iMac (the Introducer from where the sharing of Video is initiated)
Edit/Update:
Restart of Syncthing on DiskStation1, DiskStation2 and MacBook is complete, now the sync status shows up-to-date on all devices. Going to shutdown Syncthing now and save the logs in case they would be of any help. Just let me know.
