I’ve been using Syncthing for a while now and most of the time it’s transparent, but in the past few days it’s created zombie directories that took me a while to track down.
I have a folder on 2 MacMinis for development for the ESP32 SoC. While getting it set up, I used git to clone some repositories from Github in ~/Documents/Dev/ESP32. Then, as I decided what I wanted to work with and how to rearrange things, I deleted some of the cloned repos.
And within 3-4 minutes, they were back. The entire directory structure and all the files were recreated on my Study computer. (For reference, my main system is in my study, so I’ll call it “Study,” and the other is in my workshop, so I’ll call that “Shop.”) Due to a back injury, for the past few days I have been working ONLY on Study, not on Shop.
At first, by coincidence, I’d delete the directories and see them reappear after I ran Visual Studio Code, and thought that maybe it was somehow tracking repos I had opened in it and was restoring them - assuming I made mistakes. But that’s ruled out. Then, since I haven’t used git before (CVS systems were new and not everyone used them back when I was doing most of my dev work), I thought maybe it was somehow set to automatically keep the repos up to date.
But now it looks like it’s Syncthing. I’m deleting a directory tree and the main directory in that tree is reappearing within 2 minutes. It’s like fighting zombies or Jason Voorhees - every time I kill these directories, they come back. I was finally able to get rid of them by watching for them to reappear and deleting them as soon as they showed up again.
I’m not seeing anything in logs, but it’s happening in directories setup with Syncthing and not in other directories.
So here’s a detailed history.
Over 24 hours ago I was testing in ~/Documents/Dev/ESPHome/openHASP/Source. I created a repo using git clone --recursive https://github.com/HASwitchPlate/openHASP. It creates a new directory for the repo, ~Documents/Dev/ESPHome/openHASP/Source/openHASP and creates a full repo in there. (Please note there are TWO openHASP folders - just getting used to git and didn’t realize it’d check the repo out in another folder, not in Source.
Then, later (as in more than 5 minutes, maybe half an hour or an hour later), I am in ~/Documents/Dev/ESPHome/openHASP/Source and I use rm -Rf openHASP/*; rm -Rf openHASP/.* and everything is deleted. (I use the default setting of 3 minutes for folder scans in Syncthing.)
I use ls -la openHASP and it’s empty.
About 4 minutes later I do ls -la openHASP and the files have returned.
Repeat until frustrated. Every time I delete the directory tree, it comes back.
Early this morning, or this afternoon, since I realized my ESP32 dev work was all in ESPHome, and that I’m using ESPHome, openHASP, and Tasmota, so I won’t confuse myself, I create ~/Documents/Dev/ESP32 and copy everything in there from ~/Documents/Dev/ESPHome and remove that directory (ESPHome) from my Syncthing configuration. I rename the ESPHome directory to ~/Documents/Dev/oldESPHome.
I added the directory ~/Documents/Dev/ESP32 to Syncthing on this Mac, that I’m working on and on my workshop system, which is very close to this Mac in structure and apps. It syncs up okay.
I go to the old ESPHome directory, delete the openHASP repo I had been using and wait half an hour and double-check. It’s gone. Wait an hour to be sure. Yes, it’s gone. So once I remove ESPHome from Syncthing and delete a repo, it goes away and does not come back.
I go into ~/Documents/Dev/ESP32/Source and check out the openHASP directory with the same command I used earlier. Now I have ~/Documents/Dev/ESP32/Source/openHASP, with the full repo in it. I leave it there while out and running errands and come back and delete it, using the same 2 rm -Rf commands I used earlier. At this point, it’s after midnight, which gives me a good line on the logs - look at only after midnight for whatever Syncthing did with this repo.
I check immediately, from the directory ~/Documents/Dev/ESP32/Source and there is nothing in ~Documents/Dev/ESP32/Source/openHASP and there are no files in ther eat all. (List again, looking at dot files - nothing there.)
I check again, about 5 minutes later and the repo is back again.
I check the logs for Syncthing activity since midnight and this is all I see:
2025-02-19 00:15:23 New external port opened: external TCP address(es) [0.0.0.0:14978] to local address [::]:22000. 2025-02-19 00:15:23 New external port opened: external TCP address(es) [0.0.0.0:45196] to local address [::]:22000. 2025-02-19 00:19:28 Sent usage report (version 3) 2025-02-19 00:26:49 Folder "Development-ESP32" (~/Documents/Dev/ESP32) isn't making sync progress - retrying in 1m19s.
I find, when looking back through logs, only errors are mentioned. No logging of sync events that worked.
So, in short, if the repo is in a directory configured for Syncthing, it comes back. If it’s not, it does not come back - but it does not show anything in the logs.
Also, addressing this, now that I’ve looked at the thread:
I may misunderstand the situation and issues in that thread. I think I’m doing something different. I don’t want to use one of my Macs as a server for git. I want the repos I’m working with to be identical on both my Macs, so I can work on it on my computer in the house and, when I’m down in the workshop, can pick up where I let off and work on the same thing. Or, if everything is done and I am down in the shop testing things, and find an error, I can fix it in the shop, without having to come back to the study to fix it.
Basically, the reason why it’s not recommended to sync Git repositories is because a repository comprises of a large number of files that all need to go together in a consistent state, or otherwise it becomes corrupted easily. Syncthing doesn’t know that the folder you’re trying to sync is a Git repository, so it just treats those files as any others (e.g. like documents or video files that can exist and be used on their own without relying on the whole folder structure).
However, whether you sync Git repositories or not, this still shouldn’t lead to files or folders getting undeleted like that .
This makes sense. I don’t think it’s a factor in my case for a few reasons:
Syncthing is set to scan and sync every 3 minutes.
I’m the only one using both systems, and never at the same time.
It takes me over 3 minutes just to walk from my study to the workshop, so if I make changes on one system, the other will have time to update before I get to it.
Exactly!
Last night, for the first time, when testing, I looked at the other computer. (I’m using the computer in my study this week and not the one in the workshop until a back injury recovers, so, for this discussion, “Study” is my prime computer and “Shop” is secondary.) I added a repo, waited over 30 minutes, then deleted it. It was back in under 5 minutes and it was also on the Shop computer.
I deleted the files on Study again and looked in 1-2 minutes later. Some were back. Not all, but some directory trees were back. It was apparently still syncing them. So I used rm -Rf to delete everything. Then I waited and checked again. This time only one directory showed up. That’s when it occurred to me to check the Shop computer and only that directory was there.
I could do a lot more testing on that, but, for now, it looks to me like when I deleted the other files, during sync, that one directory (lib), was the one that missed, probably due to timing. So, at that point, the only zombie that was showing up was lib. I finally deleted it on both systems, first on Study, then on Shop. And the zombies disappeared.
Typically, when a directory is undeleted by Syncthing it’s because when it tried to delete it on the other side it failed, because there were unknown ignored files in it or something similar – icon metadata, hidden directories created by the system itself, etc. Alternatively, because you’ve enabled syncing of extended attributes or ownership, and these differed when Syncthing was going to delete the directory and it was resurrected as a sync conflict.
.