Well, that is part of my problem. There is no error message on the GUI I can report and the log on TrueNAS fails with “unable to retrieve container logs for containerd://05a6d51663fb0a625dc952b9de02ede47d2316904d0599eb9666f778d24b6398”
So, I figured “OK. No problem. I can just shell into the container and locate a log there.”. Sadly, that doesn’t work either and I am getting “ERROR: cannot exec into into a container in a completed pod; current phase is failed.”
Yes, I wish I would have a log to analyze the problem, but I haven’t been able to locate one yet.
You can follow the log from within Syncthing’s Web GUI. And also enable different debug facilities. Check out the About dialog as well, if a log file is being written, it will be listed there.
You haven’t really described the symptoms though. What exactly is the “local additions issue”?
It is only listing folders as Local Additions which makes no sense since the folders do exist in the Send only folder.
Thank you! Finally, some progress.
In the log, I am seeing the following message relevant to this post: “2024-06-18 18:53:51 Revert: directory has been deleted on a remote device but is not empty; the contents are probably ignored on that remote device, but not locally”
So, I am guessing my initial hunch that it might be a permission problem is correct?
Just to clarify: The machine, the problem occurs on is running on TrueNAS Scale version Dragonfish-24.04.1.1 and I am using Syncthing (TrueNAS catalog) version 1.27.8.
I don’t see a connection to permissions here. The message tells you what is wrong: The folder contains some files locally (on the receive-only side), but the sender side informed about the deletion of the whole folder. The problem is, it didn’t send a deletion for the remaining files within, which are still in the receiver’s folder. Thus they don’t get deleted and therefore the parent directory cannot either.
You should look closely at any ignore patterns you have set (which is not clear yet). They should match on all involved nodes. And you might need to prefix some patterns with (?d) to allow Syncthing to delete the parent folder if these matching files are still inside. See Ignoring Files — Syncthing documentation for the specification, including special case prefixes.
To work around the issue very easily, just delete the offending folders manually.
I have adjusted my Ignore pattern rule, but that didn’t fix the issue. So, I deleted one folder and then the system re-synced that one folder without re-adding it to the Local Additions count.
I can now resolve the issue by deleting all folders with issues and then wait on the re-sync. It will take about 2 days, but it is doable. So, the ability to fix it is the good news and the time it takes to do so is the bad news.
I also have no idea why the problem suddenly occurred, but I am guessing it is related to my dragonfish upgrade…
Anyhow, I will report back once the re-sync is done.
Good news is that the re-sync went faster than expected.
Bad news is that I ended up with the Local Additions issue again. Once again it shows roughly 1100 folders that are not in sync with the source.
I am using the file browser application to validate the folders and files. Is it possible that the app is not showing me what has been added locally?
I did notice two issues this time.
The Receive Only target shows that it has received a Thumbs.db file, but I cannot find this file in the Send Only folder.
But there is an even bigger problem which also explains why the re-sync went so fast. The Send Only folder has more folders/files than the Receive Only folder even though it shows as Up to date and Local Additions on the other end. I mean like 580GB are missing. What’s going on here?
The Send only folder as seen from Windows. Fairly close to the Send only folder on the Synology.
Since I am very confused by the results, I am retracing my steps by deleting a folder from the Receive only folder. It initially disappears and a few minutes later it got re-synced from the Send only folder as expected. So far, so good.
Now, I am waiting for it to reappear with Local Additions!
Are you sure the folder IDs match? That is not visible in the screenshots. Are the devices showing as connected? What are the ignore patterns? Have you checked the list of locally added items? How do they differ?
Note that manually deleting something within a receive-only folder also counts as a “local addition”. So your problem might go away if you just press the Revert Local Changes button.
Mismatching global state is usually a matter of different ignore patterns. And that includes stuff that was ignored on the sending side after having previously been announced to other peers. (Probably not the case here, as your sender has more data on global state.)
That isn’t really expected except when you press the Revert Local Changes button.
I noticed your screenshots both show a connection to a common “nas1” device, the rest is blanked out. Are these two devices actually sharing the folder with each other directly? Or just through that common third party?
What kind of “network mount”? What about the ignore patterns?
If you still want to just work around the issue, note that you don’t need to delete and re-sync any files. Just remove the receive-only folder from the config, restart Syncthing (just to be extra careful), and then accept the invitation again, pointing to the same folder path. Anything that is missing will then be downloaded, anything superfluous should show up as local addition.
In my home, I have a Ubiquiti UDMPRO which is connected to a remote Sophos SG (legacy) via a Site-To-Site VPN (IPSec). The remote network is 10.10.x.y and my local network is 192.168.x.y. So, I just “mounted” it using built-in UDMPRO capabilities. That has been working for years and I have done some recent testing to confirm that there are no issues.
I am using the same Ignore patterns on all folders.
I have just double checked it once again.
I believe my troubles have started after my recent TrueNAS bluefin upgrade. All upgrade steps (including manual steps and scripts) completed successfully, but I did lose my syncthing settings which I quickly re-created using old (<1y) screenshots.
Yes, I have been thinking about temporarily removing the folder as outlined in your post above as well, but isn’t that likely going to give me the same result as deleting all Receive only folders yesterday?
As a person with an IT background, I am also stunned that it is not possible to accurately determine what has been added locally. I mean some of my other folders are being accessed by local TrueNAS applications, but the folder giving the issue is not used by any application at the moment. It is purely a backup folder.
I don’t understand this part of your setup. Are you “mounting” it as some kind of mapped network drive? NFS or Samba / CIFS, WebDAV or something? How does that fit together with using Syncthing?
The ignore patterns definitely don’t need to include the .stfolder marker, nor the .stignore file. Both are not synced anyway. Marking them as (?d) would also be incorrect, because if they prevented a folder from being deleted, that would be the configured root folder, which Syncthing obviously wouldn’t try to delete. And finally, be aware that such a pattern matches the name in any subdirectory, recursively. You’d need to anchor it like /.stignore if you actually mean just that one top-level file.
The difference is in Syncthing’s database, the so-called index. When removing a folder from the configuration, anything known about it in the database (especially files which did exist but were deleted) is erased. Thus it will start with exactly the same global state as announced from your send-only device when they do start sharing again.
To determine what the local additions are about, just click on the count and it will show you a list of files.
To increase log level, go to Actions > Logs and enable specific debugging facilities. model is usually interesting, scanner, walkfs and fs might be relevant to your case, maybe others.
The added log information shows some more data when pressing Revert changes. I do see a “NoPermissions:true”. Is that relevant for revert not working?
