Hi,
Not always but sometimes when I delete a file and scan the file from the API the file reappears.
I read somewhere that this might be an issue if user is using a network file system?
Is there any work around this issue ?
Thank you in advance.
1 Like
I am not aware of that issue you are referring to. In general the answer is no, scanning should not make a deleted file reappear under any circumstances. Please explain what exactly you do and on what setup.
So, I have multiple syncthing nodes and scan/inotify is disabled. When I make a change I do an API call to scan. Each syncthing node has a different nfs mount point. File permission sync is also disabled.
It is not scanning the makes the file reappear. The scan goes through fine but after a while the file reappears from some other node. Generally there are many of scans happening so the file deletion is not propagated right away on the other nodes but takes some time.
Not sure what other information can I give.
What does that exactly mean? E.g. Syncthing runs on device A with a folder at path /foo/bar, and a partition on device B is mounted with nfs to /foo/bar on A. Or Syncthing runs on device A with a folder at path /foo/bar, which is also mounted with nfs on other devices.
Also do any of the devices “connected” by nfs mounts share any Syncthing folders?
When you say the file reappears, does it get synced from other devices? Do you get any indication of what’s going on? Essentially as I said I am not aware of any such problem, so I have no clues what might be going on, so anything that’s related to your problem and setup might provide such a clue, even if it doesn’t seem immediately important. Also syncthing version and OS.
each nfs partition is unique and is only mounted to one syncthing node (1-1).
I have folder XXXXXXX which is shared with all nodes.
Yes the file gets synced from another device which ( one that deletion did not happen) and timestamp of the file is changed to the latest date.
I forgot to mention that I only do changes to one syncthing node and scan only to that node. the other nodes can be read only but in the future I plan to change that.
The folder shared is around 40TB.
I’m thinking that something like this might happen but have not been able to reproduce 100%.
From node A: I delete file X/X from disk I scan for file X/X I add file X/Y I scan for file X/Y
File X/X deletion has not been propagated to node B but scan of file X/Y has and syncthing node B thinks that it has the latest files so it sees that node A does not have file X/X and tries to sync ?
I have a theory around something being amiss with modtimes: B receives the notice to delete X/X, looks on disk (nfs mount) whether what is there is what it expects from db, sees a new modtime, thus doesn’t delete but instead rescans and bumps the version. This creates a conflict with the deletion on A, but as in conflicts between changes and deletions changes always win, you just see the file being recreated.
Could you check the following to test that:
- Does the modtime on B change when this happens.
- Query the https://docs.syncthing.net/rest/db-file-get.html endpoint after the file is recreated on A and post the output.
Just out of curiosity: Why do you run Syncthing on nfs mounts like that? The reason not to run on a network share but directly on the server is that full scans (traversing the filesystem and checkung modtimes and sizes) are usually slow, and for 40TB they likely take ages.
yes the modtime changes on syncthing B.
which endpoint should I query ? the one that I send the scan to delete ?
I would stay away from the conversation on why this is done like this 
Yes that one (even though it might not actually matter which).
My useless-but-entertaining-and-time-consuming-story spider-sense is triggered - I am all ears 
1 Like
The file was created long time ago and shouldn’t have had that timestamp.
{u’global’: {u’size’: 154261683, u’ignored’: False, u’name’: u’f/8/file’, u’sequence’: 8355913, u’deleted’: False, u’localFlags’: 0, u’modified’: u’2020-01-09T02:12:32.650551+01:00’, u’invalid’: False, u’version’: [u’DZX4L5C:1’, u’HXCYYKD:2’, u’YK5LG5I:1’],u’modifiedBy’: u’HXCYYKD’, u’noPermissions’: True, u’numBlocks’: 1177, u’type’: 0, u’mustRescan’: False, u’permissions’: u’0644’},u’local’: {u’size’: 154261683, u’ignored’: False, u’name’: u’f/8/file, u’sequence’: 8355913, u’deleted’: False, u’localFlags’: 0, u’modified’: u’2020-01-09T02:12:32.650551+01:00’, u’invalid’: False, u’version’: [u’DZX4L5C:1’, u’HXCYYKD:2’, u’YK5LG5I:1’], u’modifiedBy’: u’HXCYYKD’, u’noPermissions’: True, u’numBlocks’: 1177, u’type’: 0, u’mustRescan’: False, u’permissions’: u’0644’},u’availability’: [{u’fromTemporary’: False, u’id’: u’FPRVNBF-DMT43BA-ONHYOTM-CQ6S33K-C4OXIYF-LVMZXM4-GOM7BCO-XZPVDA7’},{u’fromTemporary’: False, u’id’: u’KPM5ZJN-5ZQKOGY-ISLIFKB-XHDYMX3-2GWBTAJ-HPZFIMQ-FNLFM7C-XZEOXAS’},{u’fromTemporary’: False, u’id’: u’HXCYYKD-S4GEFZU-6547FSB-WDNUMUX-JD55X2T-U5ZURKW-IGJYHKD-ETDDTQZ’}, {u’fromTemporary’: True, u’id’: u’DZX4L5C-DCWT6R4-JV7HRCL-C52O3LZ-EBF7T7C-1C6NGFK-2DJXCJG-3S7LZAI’}]}
That shows that the file has been modified by 3 different devices (which ID is the one where the deletion/scan happens). So if as you stated changes should only happen on one, it looks like there is indeed a problem of the kind I suspected: Either modtimes are incorrectly reported by the FS or there is something on those devices that really changed those files.
The scan device is :
There is no way somebody or something modified these files on other devices unless its syncthing.
We do have cache on nfs enabled but I don’t see how that can cause an issue.
Assuming you are right what is the solution ?
Should I create a bug in github or … ?
Given the file is the same, except for an updated mod time more or less equal to the time of deletion, and Syncthing not doing anything but stating the files, there has to be an issue with the reported mod time. And it seems to be a frequent issue, given it happened on two devices for that single file (I assume not a permanent one, i.e. some deletions do propagate). If you want to be certain, you can run with model,scanner debug facilities enabled on a device where the deletion does not happen to see what exactly is going on.
As to the solution: No clue. If the FS returns changed mtimes we are pretty much screwed I think.
I can but I’m not sure if that would actually help on the solution
Is there any work around or any solution that will be in the next version of syncthing ?
Use a proper filesystem.
so that means that syncthing does not support NFS or any other network file system?
Syncthing doesn’t support any particular filesystem, it just uses Go’s os library. If that reports changed modification times later on, it will consider that as a legitimate change (if it reports an inaccurate mod time when creating the file, that gets abstracted away, but if the mod time changes later, it’s not possible to discern that from a real mod time change).
What that means is you need to investigate why/under which circumstance your nfs FS reports changed mod times, and fix that (or maybe that’s all a red herring, and there is something doing changes on your systems, e.g. to permissions).
It does not “support” or “not support” any particular filesystems. It uses whatever is there, and it misbehaves, you have to deal with it. There are less weird things with real filesystems, and given you asked for a workaround, I provided one.
1 Like
can you please explain me how the timestamps should really work ?
If I add a file to a node it will obviously get the timestamp of the file and add it to the local database but also global ?
the other nodes will save the local timestamp aka the time when the file was replicated so in the end all nodes should have a different timestamp but they save these in the database so it shouldn’t be an issue.
I don’t see how nfs would give wrong timestamps even with caching.
So first of all: The timestamp thing is a theory of mine - it might be entirely wrong. It’s just consistent with the info I have so far:
Initial state:
File exists on all devices, no changes ever happen except on device A and scanning is disabled on all devices.
- You delete file on A and scan.
- Syncthing pulls that file in from remote device.
- Querying
/rest/db/filereveals that remote devices bumped the version of the file (i.e. they saw a change locally), the mod time is approx. equal to when 2. happens and the file is unchanged.
My theory:
The deletion is propagated to remote devices. Before they follow through and delete the file, they check the file on disk. This means stating the file and comparing modtime and size with what is in db. The stated modtime is the time of 2. above, i.e. changed. To prevent data loss Syncthing then doesn’t delete the file, but bumps the version of the file it should delete. This prompts a conflict and eventually recreates the file on A.
This is entirely working as designed. So as there doesn’t seem to be a known obvious reason why this modtime change should happen, my first step would be to check my theory - otherwise you might invest time in finding a solution for a problem that doesn’t exist. To do that:
I don’t know either, but I can e.g. guess at time sync issues (e.g. some problem with ntp changing time between the time of caching and time of access). What I mean is just because something should not do something wrong isn’t a good reason to assume it doesn’t do anything wrong (that applies to Syncthing too - if you provide those logs and it turns out my hypothesis is BS and there’s a bug, I am happy to stand corrected and if possible fix the bug).
Syncthing attempts to sync file modification times, so under normal conditions all devices have the same last modified date.
(This probably has no effect on the posted issue though)