Hi, I’m experiencing the following issue very regularly:
Within a synced directory, I move a file from any one folder to another, and when doing so, more often than not the remote device will create a file in the destination directory, but fail to delete the file from its previous location with the following error: syncing: delete file: file modified but not rescanned; will try again later.
To recap, in this scenario, LocalDevice/SyncedDir/MyCoolFile is moved to LocalDevice/SyncedDir/AnotherDir/MyCoolFile.
The remote device will then have the following layout:
So the move operation stays within the Syncthing folder, which should be detected by Syncthing and result in a similar result instead of the file being re-transferred in whole.
hmmm… if the TrueNAS is running Syncthing, where does the SMB share fit in? Is it just providing the Syncthing folder via SMB for some other devices?
With the TrueNAS in the middle and the problem occurring when a file is moved on macOS or Windows, it sounds to me like it’s a permissions issue on the TrueNAS.
TrueNAS Core supports both apps with and without Docker – which way is Syncthing running on the TrueNAS?
With the current set of devices, there’s really no benefit to enabling “Sync Extended Attributes”, and actually adds overhead to the sync process (Syncthing also doesn’t support it if two endpoints are different OSes).
It’s because extended attributes are generally not cross-platform. Samba in TrueNAS might or might not even have support for extended attributes enabled, so the attributes get “lost in translation” as they pass from macOS thru TrueNAS to Windows.
Correct, any small devices that don’t have the storage capacity are just accessing those same directories over SMB. In this case, each syncing pair of folders is immediately below the root of the SMB share, like so:
MyCoolShare
|_SyncedDir1
|_SyncedDir2
|_etc.
It’s currently running through the official TrueNAS Core plugin, which operates as a FreeBSD Jail.
I wasn’t really sure whether or not I needed this. My biggest concern was whether updating a file on one machine would result in the extended attributes being lost on another. It’s not necessarily a “need to have” if so, mostly just a “want to have”.
So based on this info, you think it may have something to do with the synced files residing in the SMB share. I could definitely believe that as I ran into a lot of file permission issues when I was initially setting it up. So that said, when accessing it as a share, I don’t seem to encounter any issues, leading me to believe that could be a possibility. Unfortunately, making this change will more or less require me to create this dataset fresh, transfer all files, and then set up syncthing again, so if possible, I would like to save that as a last resort.
To be honest, if I have to go to those lengths, then I might just remove syncthing from the picture and just use the share on each device. Not really ideal, but I could live with it.
So let’s say that a file is moved on the Mac – e.g. macOS/SyncedDir/MyCoolFile to macOS/SyncedDir/AnotherDir/MyCoolFile – and on Windows, Syncthing has trouble removing Windows/SyncedDir/MyCoolFile, what does it look like at that moment in the SMB share on the TrueNAS?
If a device pair is macOS-to-macOS, Linux-to-Linux or Windows-to-Windows, and the underlying filesystem supports it, then syncing extended attributes is fine.
It’s when mixing OSs where it becomes “hit or miss” (usually “miss”). There are niche setups such as WSL (Windows Subsystem for Linux) where Windows stores the regular Linux file attributes in the extended attributes of NTFS (I don’t know if extended attributes for Linux are also supported by the same setup).
Samba normally switches to a less privileged user after startup (at least that’s how it is on Linux; it’s been a while since I’ve used FreeBSD, so things might have changed). Either all the files are owned by the Samba daemon, or Samba users were created.
Syncthing running in a FreeBSD jail definitely makes managing the permissions and file access more complicated because the files in the SMB share are outside of the jail. Syncthing in the jail needs read/write access to files that are either owned by the Samba daemon, or possibly by multiple Samba users.
In this scenario, the SMB Share will have both TrueNAS/SyncedDir/MyCoolFile and TrueNAS/SyncedDir/AnotherDir/MyCoolFile, and syncthing will report SyncedDir as “Out of Sync” with 1 failed item marked with the following error: syncing: delete file: file modified but not rescanned; will try again later.
In this scenario, syncthing is configured to run as the same user who will have access to all the files in the specific folder in the dataset. To clarify, syncthing is being used to sync several directories in my home directory between these three machines. My user on TrueNAS, (let’s call it cshook with a group of cshook) exists within the Jail and syncthing has been configured to launch as this user. The dataset which contains this directory is owned by root:wheel, but the directory itself is owned by my user. This is the same directory that is mounted as the SMB share, as well as the one mounted to the syncthing jail, under mnt/cshook.
So in TrueNAS (outside of the jail) from the root directory of the dataset, the following permissions are reported:
That cshook directory and associated user/group are the same ones which syncthing runs under inside of its jail. Not sure if this gives you any more insight, but hopefully it at least provides some more context.
Okay, so that helped confirm my suspicions. With TrueNAS basically acting as a hub between the Mac and Windows devices, the SMB share showing the same issue means it’s definitely a permissions problem on the TrueNAS.
If I were to guess, I’d say that the only files that are affected are files that weren’t originally uploaded from the Mac/Windows devices to TrueNAS via Syncthing. They might have different ownership compared to files that Syncthing creates.
Mmmm… that’s going to be all of them then. They all have rwx permissions for the user, but nothing for the group or other, though they do also all have ACL on them from the SMB share looking like this:
Since syncthing is running as my user, and since my user is the owner of the file, and since I’m ignoring permissions on the files, in theory everything seems like it should work, but the proof is in the pudding as they say.
I may end up just recreating this dataset and not even touching SMB.
Although Syncthing is running as you, it’s also bounded by a FreeBSD jail…
Syncthing’s “Ignore Permissions” settings is whether or not it replicates permissions that are sent by an upstream peer, so file/directory permissions on local storage still impacts what Syncthing is allowed to read/write.
So I removed the SMB Share and stripped the ACLs back to what I would expect them to be, and that on its own didn’t seem to make any difference. I’m not sure if it’s relevant or not, but for future users who come across this post I wanted to include this just in case it was part of the fix.
What did seem to actually fix things was disabling “Sync Extended Attributes”. I don’t know exactly why that was causing the issue (at least based on the way that I’ve read up on how that syncing is implemented), but for whatever reason they were. Maybe weird leftovers from the SMB ACLs, but at this point I’m just happy that it’s actually working.
Thank you for your help in getting me to a solution!