If possible, please upload screenshots of the GUI from both sides, and also the log files from all the involved devices.
Here you go - I hope this is the right stuff…
the two machines are “T540” my laptop, and “coolbox” the desktop machine
Not sure why the log on the T540 is so much shorter - hope it is OK…
Will probably be heading for bed soon, so replies may be delayed…
ex-Gooserider
I’m not an expert, but I personally cannot see anything unusual in the logs. You have tons of folders though. Are the Out of Sync items limited to only some of them, or are they rather spread around all the folders?
If it is the former, then it would probably help to limit the scope to only the affected folders. If you can do this, then please try to remove such a folder, re-add it, and see if anything changes.
Also, when you were adding the folders for the first time, were the files already present on the other device, or were starting from scratch?
On the T540, it says Coolbox has two folders w/ OOS items, on Coolbox it says T540 has 5 folders, two of which are the same on both machines.
I have a such a huge number of folders because it isn’t safe to back up my /home directory, and these are the folders that are in /home (I REALLY want to back up everything, w/ the exception of the few machine specific config files (i.e. monitor specs for the window manager)) Is there a better way to set this up?
This is what I was asking about in one of the earlier messages in the thread (#4?) but hadn’t gotten an answer, so this is what I ended up with.
I had done an earlier install of ST and managed to break it badly by attempting to sync my /home. I thought I had done a ‘scorched earth’ removal of the previous install, and was restarting from scratch. However I believe that the folders I am seeing problems with had been synced during the previous install before things broke.
I will try removing one of the problem folders and putting it back in and see what happens.
ex-Gooserider
There are different approaches. I personally like to have a multi-folder setup like yourself, because this way you have total control over what is synced, and what is not. A different method is to add a whole large folder, and then ignore everything with selected exceptions, e.g.
!/folder1
!/folder2
!/folder3
*
Which approach is better will likely vary depending on the situation, such as the total number of folders and files, hardware performance, etc.
Of course, this is assuming that you have a lot of stuff that you don’t want to sync. If you only want to ignore a few items, then adding a whole folder with a few ignore patterns will be much simpler.
I am having mixed results w/ removing and re-adding folders…
It seemed to help in some cases, but not in others… Now I’m getting an error about inotify limits on the T540, and need to chase down the docs on that (I’ve seen them, just need to find it again) and fix it.
The T540 is down to one folder w/ OOS items, and it is the one w/ the error.
Coolbox is down to 3 folders, one w/ the error, one that I’ve removed and re-added a couple of times (Music), and one (Pictures) I haven’t touched yet…
I had a major mess w/ Music, for some reason (possibly user error) It put all it’s contents into my home directory instead of ~/Music, which made for a lot of cleanup effort to get back to previous state…
However it seems to be making slow progress…
ex-Gooserider
I feel like I am chasing my tail on this, I have now removed and added the problem folders multiple times, and it seems to have gotten rid of SOME of the problem folders, but not others…
On Coolbox, ~/Music seems to be the problem - 13,844 items, 14.7 GiB - However the list only shows 1385 items, which is probably the number of files in that folder.
On T540, ~/Pictures and ~/Reference-docs are the problem files listed 6,293 items, ~4.17 GiB. ~/Reference-docs says 48 problem items, which is a subset of the directory, with a mix of files, subdirs and files in the subdirs. ~/Pictures seems to be a similar mix w/ 625 files - strangely this does NOT add up to the number of files listed as problems…
When I’m looking at the list of bad files on either machine, they are ALL 128 B - so likely “metadata” rather than the actual files…
I also did a spot check of a few of the problem files and they appear to be the same on both machines…
I noticed in the FAQ a section on dealing w/ a corrupted database, the symptoms in that section don’t quite match what I see here, but is that process of deleting the database and making ST regenerate it a possible cure?
Alternatively, where is this metadata stored? Is that a file that can be deleted and regenerated?
ex-Gooserider
This sounds like an issue that I have had recently, but I haven’t been able to reproduce in a clean environment yet. I would bet that those 128 B “files” are actually directories, not files. Can you confirm this?
I have only had the issue when there were pre-existing files on the receiving side though. Could you please try to set one of such folders to Send Only on one side, then remove it (i.e. completely wipe out the files) on the other side, and then try to sync it again? This is the only “solution” that has worked for me here. Once synced properly for the first time, the folders should continue synchronising without any issues.
Mixed on the files being directories…
It is a bit unclear in the ST display just what the OOS items means - The OOS count only shows in the “Remote Device” info section, but does that mean the files are OK on the remote device, and bad on the local one, or vice versa???
In my case, all of the files were originally made on the T540, (at least for now, I’m hoping this will change…)
The Reference-files directory, on both machines, all the “bad” files listed are directories or at least I didn’t see any that weren’t.
For the Music directory it seems to be a mix of files and directories. In doing a more detailed check, it looks like I was mistaken, the OOS items list is NOT all 128 B in size, the items that are files show the about the same size in the list and in Dolphin (difference probably due to rounding)
In addition, probably as a result of whatever it was that caused the contents of ~/Music to get dumped into my home directory, it seems that many of the OTHER subdirs in home ended up getting copied into ~/Music, and those seem to be at least a good chunk of the OOS items.
On the positive side, /Pictures seems to have fixed itself, at least neither machine lists it…
I will give your suggestion a try, probably starting w/ ~/Music after doing the major cleanup I seem to need there…
I have as you suggested, Paused everything on both machines, and removed the Music folder entry from ST on Coolbox, and deleted the entire /Music directory… I then went to the T540, and cleaned up the /Music directory by deleting or moving the non-music directories and files to a new (not synced) directory for evaluation and probable deletion…
I then unpaused ST on both machines… I may not have waited long enough but even though the folder on T540 was set to share, it didn’t try to recreate it on Coolbox. When I tried to recreate the folder on Coolbox, some how I almost ended up putting Music’s contents into the home directory (again). Fortunately I caught this before it had a chance to start syncing files! I then paused both machines, and removed the Music folder entries on BOTH machines, and added it back again to T540 as a new folder. However when I got the request to accept it on Coolbox, it was trying to put it in /home rather than /home/Music… I edited the creation window to fix this and it has been syncing since, and appears to be putting the files in the right place…
One minor thing I’ve noticed, not sure if it is a bug or not, is that in the ST device displays both the local and remote devices show the same upload and download rates at any given moment… It seems to me that if one machine is uploading, the other should be downloading, so the rates ought to be inverted between the two machines?
I THINK I have finally gotten all the folders synced w/o any Out of Sync items showing on either machine, so all seems to be good (at last…)
FWIW, this is what I’ve found seems to work as a procedure…
- Decide which machine is the best source, and which will be the “target”
- On BOTH running instances of ST, hit “Pause All” under both folders and devices.
- On the SOURCE machine, remove the problem folder.
- On the TARGET machine, remove the problem folder AND delete the corresponding directory and all it’s contents.
- On the TARGET machine create a new empty directory w/ an appropriate name.
- On the SOURCE machine, create a NEW folder with the same source directory as in the folder removed in step 3.
- UNPAUSE all the folders and devices on both machines.
- When you get the message on the TARGET machine about the newly created folder, accept it. VERIFY THAT IT POINTS TO THE DESIRED DIRECTORY!!! Otherwise it may put everything in the Target’s /Home, which will cause a real mess…
- Wait for the files to sync over from the source to target…
Hope this helps - ex-Gooserider
2 Likes
I have now gotten all three of the machines that I want to be synced talking to each other and MOSTLY doing the right thing, or so it appears…
However I’m still having Out of Sync item problems… The third machine “Bill-box” is a fairly new machine (for me, I got it as cruft, but had to put a brand new HD in it, so effectively blank), with a Debian Buster install, KDE, Firefox and a few other packages, but I hadn’t done any major data transfers or other work on it.
As I’m entering this, T540 says it is up to date with both Bill-Box and Coolbox. Coolbox says it is up to date with T540, but has 254 items that are NOT in sync w/ Bill-Box… Unfortunately I am not in the same location as Bill-Box at this time, so I can’t see what it says for state.
All of the OOS items are in either ~/Downloads, or ~/Music, and it appears that all are files. They are in different subdirectories of those two, sometimes multiple levels, and the OOS items in any given subdir are a subset of the files that are in that subdir… I.e ~/Music/[artist]/[album]/ might have 15 songs, but only 3 or 4 are shown as OOS.
I have the machines set up as a mesh, where each machine has the other two as remote devices… I based this in part on the forum thread https://forum.syncthing.net/t/best-multi-devices-sync-config/13449/4
It seems that there is a logic problem if A = B, and A = C, but B =/= C 
One of the remaining things I need to figure out is that all of the machines go to sleep when I’m not using them, and it appears they have trouble syncing when they are asleep… It seems nice and “green” to have unused machines using less power, but I need to figure out how to keep them synced when what I am doing is unlikely to have more than one awake at any given time… Possibly setting up a cron job?
I also need to figure out how to keep all the software programs matched, and share some of the data files on things like my calendars and firefox, but that is a separate issue, I’m probably going to have to set up some individual sharing for those items.
ex-Gooserider
I would personally suggest to keep at least one of the machines online all the time. When idle, a desktop can use as low as 20~30W, which shouldn’t really affect your electricity bills, and this should prevent potential conflicts.
In Windows, I use the Task Scheduler to change the power plan to Power Save when idle, which I have customised to limit the CPU speed to minimum. The same can be done with CPU governor tweaks in Linux.
Since Syncthing is completely peer-to-peer, there is no way to sync files without at least 2 machines up and running at the same time (which makes it different from other, cloud-based solutions).
I’m working on solving the machine sleeping, and hopefully am making progress on that front.
I am running into another issue, and am wondering what is going on, and how to fix it.
I use the KOrganizer in KDE to keep my calendar, and in an effort to keep my calendar synced between the machines, I created an .ics file in one of the directories that is synced and set it to be the file that all the machines use. The calendar configuration has a setting “File monitoring” w the note
If file monitoring is enabled the resource will reload the file when changes are made by other programs. It also tries to create a backup in case of conflicts whenever possible.
I have this enabled on all three machines. If I understand the description, if I make a change in the calendar on the machine I’m using, the updated ICS file should sync to the other machines, (overwriting the copy on those machines) and then KOrganizer should simply see the file change on disk and update itself…
Instead I seem to get a cascade of sync conflicts that add another conflict file every few seconds… What is happening and how do I fix it?
ex-Gooserider
I have a feeling that this has more to do with the calendar application than Syncthing.
I would pause the Syncthing folder in question on all devices, and then just observe the ICS file itself to see what really is going on with it (i.e. does the file itself keep changing, does the mtime get updated, etc.).
If there is nothing obvious, then I would suggest doing more testing with just 2 devices first.
I have paused it on the two machines I am at right now, I don’t see an obvious way to pause it on the remote machine? (am I missing something?)
I watched the ics file on one of the paused machines and it didn’t seem to change except when I created a test appointment, which updated it once to the time of the new appointment (which I think is the expected behavior)
ex-Gooserider
Do you still get those conflicts when trying to change the calendar with just the 2 devices syncing?
If you don’t have access to the third device and cannot pause the folder there, you can temporarily pause the whole device to do the testing.
I just paused Bill-box on T540 and Coolbox, and deleted the existing conflict files.
I then made a change in the calendar on Coolbox, which updated the .ics file, as expected. Since the folder was paused, it didn’t propogate the changes to T540, again as expected.
Then I unpaused the folder on it and T540, and almost immediately got 3 conflict files on both machines. Two were dated from the previous file (which would have been the one on T540) one each for the .ics file and it’s backup. The third was a backup for the current .ics file. About 5 minutes later, I got a 4th CF file for the backup. The regular .ics and backup updated on T540
Since then, without making any changes to the calendar about once a minute, I see ST doing a scan of the folder and go into syncing for a few seconds, and the M-stamp on the .ics file updates to the current time. (The time does not change when ST isn’t attempting to keep the file synced)
I also tried making a new test appointment on each machine with ST working and it appears to have synced to the other w/o creating any new conflict files in each case the first time I did it, but did the same set of three files when I tried it a second time.
I also tried pausing / resuming the folder on Coolbox without making any changes in the calendar, and did not get new conflict files.
So the syncing seems to be doing the right thing, sort of except for the frequent rescans when there are no changes. and creating the conflict files in some cases but not others when making changes to the calendar.
ex-Gooserider
It seems as if the calendar software treats accessing the file by Syncthing (via the scanning process) as a “modification”, and then it also updates its modification time accordingly, which then triggers Syncthing too.
I guess that you could possibly work around this by setting Max Conflicts to 0, but really, this sounds like an issue with KOrganizer rather than anything that Syncthing can handle.
However, even considering the above, Syncthing should normally only rescan all files every hour (3600s), which is not that often, unless you have disabled the file watcher and/or changed the default settings.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.