This week I stumbled across a - probably rather unusual - bug. Basically it fails to delete a folder which name contains a German umlaut.
[EVUK3] 15:52:09 INFO: Connected to device LPAQ2DG with a newer version (current "v1.1.1" < remote "v1.1.2-rc.1"). Checking for upgrades.
[EVUK3] 16:09:26 INFO: Puller (folder "Bilder 2016" (Bilder-2016), file "03 März"): delete dir: directory is not empty; files within are probably ignored on connected devices only
[EVUK3] 16:09:26 INFO: Folder "Bilder 2016" (Bilder-2016) isn't making sync progress - retrying in 1h4m0s.
[EVUK3] 16:09:27 INFO: Puller (folder "Bilder 2015" (Bilder-2015), file "03 März"): delete dir: directory is not empty; files within are probably ignored on connected devices only
[EVUK3] 16:09:27 INFO: Folder "Bilder 2015" (Bilder-2015) isn't making sync progress - retrying in 1h4m0s.
[EVUK3] 17:13:27 INFO: Puller (folder "Bilder 2015" (Bilder-2015), file "03 März"): delete dir: directory is not empty; files within are probably ignored on connected devices only
[EVUK3] 17:13:27 INFO: Folder "Bilder 2015" (Bilder-2015) isn't making sync progress - retrying in 1h4m0s.
To give a bit of background: I have a network of 3 machines (windows, windows, FreeBSD) syncing folders with each other. This week I decided to upgrade storage on one machine and move the pictures folder to that drive. I then realized that I need to re-create all the sync folders setup in Syncthing to point to the new (moved) directory. While re-indexing the folders, Syncthing crashed and I had to manually stop the Windows process. However, that (probably) corrupted the Syncthing database, so upon restart of the service, it thought that the folders each were only a couple MB instead of GB’s…
Consequently, it started deleting all the pictures - and, to my horror - also the one in the second node which was online.
Therefore, I made sure that when I took the third node online, that Syncthing wasn’t running and my pictures remained intact. From there, I copied everything back to the second machine (FreeBSD) and it synced back to the first machine alright.
I am now at the stage on when I took the third machine with Syncthing online again. Nothing out of the ordinary happened, until I noticed the above, strange, console entry and noticed that two (out of about 12) folders were marked as “out of sync”.
Does anybody have any ideas as to
a) why it still tries to delete the folder?
b) how to tell it that the folder is alright?
a) No idea, I don’t know what you did exactly. If you just changed the config file manually to point Syncthing at the new directory, which did not contain anything, then yes, it will start deleting everything and that’s not a bug. You just can’t do that. The proper thing to do is to remove and re-add the folder. In that case nothing will get deleted.
b) This error means, that someone else wants this client to delete the directory, but the directory still contains valid, known files. What you need to do is to see what’s actually in this directory, compare it to whats in the same directory on other connected devices and to ignore patterns on all devices - there should be something fishy. If there isn’t, please still report back with that information, maybe there’s another scenario we didn’t think of.
a) removing and adding folders in Syncthing, that is what I did. I didn’t dare to tamper with the config file, because, as you say, that would be a no-go. I suspect that by adding ~200GB of folders to be scanned of an microSD card was probably too much at once and brought Syncthing down.
b) There are no ignore patterns and there are definitely files in that folder (my precious pictures ;). So in a way I am quite happy that it didn’t go ahead with deleting it. What stroke me as odd was the fact that all the other subfolders on that level (01 January ~ 12 December) didn’t throw the same error, only the one containing an umlaut threw this error. And it happened for two different years (parent folders). The request for deletion came from the FreeBSD server (which however does contain these folders and the files in it).
Ok. It just got a bit weirder: I did check the specific folders on the server. They are partly as they should be, but partly the folders in question are empty according to SSH. Yet Windows Explorer shows that the files are still there, but made into temporary Syncthing files (.syncthing.IMG_xxx.JPG.tmp) with proper filesize. That also might explain why I have 46GB “unsynced” on the server. Which brings me to the next question:
c) I can change permissions on the server, but am quite unsure in which direction it will go. That is, will it continue to delete the files OR will it restore the files back from tmp status?
a) Then you should check logs. Except for running out of memory, Syncthing isn’t generally brought down by a lot of data and even if, that just means no sync, not data deletion.
b) That would be indeed a weird coincidence. However I use and never had problems with Umlaute (on linux though), but who knows.
Temporary Syncthing files mean, that it synced them, but then there was a problem so the originals haven’t been replaced with the temporary copies. They will be removed after 24h and except for maybe given clues to what might happen, you can essentially consider them as non-existent.
c) I don’t understand what permissions you mean.
In general, please provide screenshots, that contains lots of info on your setup and current state, and thus helps for getting an overview and I can ask more specific questions.
a) I will see if I can retrieve these logs (any specific ones you are looking for?). It might be that I just thought that Syncthing was down and by manually stopping the system process corrupted the database. The corrupted database however then resulted in the deletion process (because the “global index” was set to a the “last known state” (I assume) and that one was much lower than the actual status. Therefore, in order to get the local status in sync with the global status, it started deleting the files.
b) That is what I was thinking as well. However, the problem persisted over a day or so and it only affected the mentioned “Umlaut folders”. However, since then, the system levelled out again and the error disappeared.
The temp files being deleted after 24hours isn’t happening here. I still have them…
(In the screenshot, 2011 should be 29~GB, so the actual filesize is correct).
c) Permission to the files. Looking at the (server) logs, I see a lot of “chmod not permitted”. I can give 777 to these folders, but am hesitant as to what might happen
(If you let me know which screenshots would be most useful, I can provide them).
Regarding logs and screenshots: With the information I have I don’t know yet what to specifically look for, so any screenshots and logs from the same time and multiple devices would be a good start.
If you have one “good copy” of the data, it’s probably a good idea to change the folders on that device to “send-only”. Then that copy is secured and you can fix your permission problems.
Another point I didn’t catch yet: For the paths connected to the “dir not empty” errors, check the paths on disk on all devices, specifically looking for case only differences or similar things.
/2013/01 Januar/Florin/.syncthing.IMG_0739.JPG.tmp: operation not permitted
2019-04-28 19:55:06 Puller (folder Bilder-2013, file "06 Juni/Familie/IMG_0039.JPG"): setting perms on temp file: chmod /media/bilder/2013/06 Juni/Familie/.syncthing.IMG_0039.JPG.tmp: operation not permitted
2019-04-28 19:55:06 Puller (folder Bilder-2013, file "06 Juni/Crea/IMG_0097.JPG"): setting perms on temp file: chmod /media/bilder/2013/06 Juni/Crea/.syncthing.IMG_0097.JPG.tmp: operation not permitted
2019-04-28 19:55:07 Puller (folder Bilder-2013, file "06 Juni/Familie/IMG_9865.JPG"): setting perms on temp file: chmod /media/bilder/2013/06 Juni/Familie/.syncthing.IMG_9865.JPG.tmp: operation not permitted
In the first set, the good files were created by user Samuel. My guess would be that these are the files I manually copied over to the server (using Samuel) via Windows Explorer as a first counter measure to the disappearing files.
So we are just talking about permission errors now? And you do have backups? Then just go ahead and fix the permissions.
Also you refer refer to windows explorer in conjunction with server/FreeBSD: Do you share them with samba? If so what’s the relation of that to Syncthing syncing to windows?
Ok, I think I got it solved. It was indeed a permission problem.
That is, I share these folders via Samba to Windows and apparently Windows likes to set ACL (visible as + symbol in FreeBSD). Removing (cleaning) the ACL using “setfacl -b” made the files accessible for Syncthing again and it could clean up the mess. So well done for Syncthing to keeping track of what there was and what wasn’t.
One thing I noticed which is still a bit misterious is Syncthing trying to sort it out by again and again uploading GB’s of data. That is, whenever I switched on a node (laptop or desktop), Syncthing uploaded some GB of data. The amount was random. I assume that Syncthing realized that there were files missing on the server, but although it uploaded it, it was not able to actually replace the (seemingly) missing files due to the ACL permission? SO it tried again and again.
That sounds like whatever these ACL permissions were, they did not prevent Syncthing from creating a new, temporary file, but it did prevent it from renaming that temporary file to it’s final name, overwriting the old file in the process. Therefore Syncthing synced to temporary files, then failed, then after 1 day it automatically cleaned the temporary files out, and the dance began from the start.
)
