I’m sorry (you are too quick).
Just for that test file a full scan did NOT happen since it was re-touched (this came after the last full scan).
Please hold on before I have re-set-up a new test again. (Quite busy just now unfortunately.)
In the meantime I had a look at the watcher and if the permissions are ignores (as it is for you), modification time only changes are indeed not picked up. So I assume the full scan business was just some confusion, and the issue here is only that those changes aren’t picked up quickly. I filed a PR to fix that: lib/fs: Watch attrib. changes on inotify for mod. time by imsodin · Pull Request #7623 · syncthing/syncthing · GitHub
1 Like
OK, thanks a lot!
I just did a quick test now with ‘ignore permissions’ disabled (i.e. check permissions) and now a change in the mtime is being propagated live. 
The systems are running a full scan just now to clarify if the mtimes if those user files are being updated now as well.
Best regards Matthias
1 Like
The full scan doesn’t update the modification times on the ‘older’ side … while it works ‘live’ (i.e. via inotify).
Can this maybe be corrected by deleting some cache (a ‘really full’ scan)?
Probably my users can live with it, and maybe I can work around it somehow (checking with rsync from time to time?).
But it doesn’t seem to be the correct or intended behaviour. Do you agree?
Best regards Matthias
It’s not clear to me what you did here exactly and it’s hard to guess. What you describe “should be impossible”, i.e. it’s something unexpected - so it’s even more important to know what exactly happens. Please be specific about what exactly you do and what happens, and how that is different from what you expect. Writing it with a standard template like this
Initial setup: Devices A and B, sharing folder F, fully in sync.
- on A do something
- Wait for x and y.
Result: One out of sync item.
Expected: All in sync.
helps a lot, as it forces you to include all the relevant info.
Agreed, mod. times should sync - whether “the behaviour” is correct or intended I cannot say, because I don’t understand the behaviour as explained above 
OK, I’ll try, starting from the situation as it is now:
Two linux servers running Syncthing 1.15.1 on one folder. Subdirectories of that folder are shared via Samba to Windows 10 clients which operate on the files and subsubdirectories.
Last week on both systems ‘ignore permissions’ was set, which led to the behaviour I described (the mtime of unchanged files didn’t get updated on the side where it should be synced to). It didn’t get updated ‘live’ and also not by a full scan!
Now we saw that ‘ignore permissions’ is somehow overreacting (a small bug), and so I disabled it on both sides, which led to the correct behaviour for live-updates. Very good!
Then I initiated a full scan to also update the files (the mtimes) that have been changed last week and where the new mtime didn’t propagate and still is not the same.
But this doesn’t work: the mtime of the files that are ‘newer’ on one side are not being updated to the other side … by the full scan - though they get updated when I touch’em now to an even newer mtime.
We’re talking of identical files: same name, same content, same permission bits; only the mtime is different.
Do you need more information?
Unfortunately I can’t seem to curl-GET the detailed information of such a file, maybe because the path to it contains a ‘!’ (well, it even contains two ‘!’ in the names of two subdirectories).
Do you think mabye the ‘!’ can be the source of the problem?
Best regards Matthias
That’s what’s what is strange to me: The live-updates just kick off a scan, i.e. it essentially does the same thing as a full scan, just on a subset of directories.
The only way I can imagine right now that this happens, is if the full scan somehow newer reaches the file (aborts on a parent), while the live-scan starts below the problematic parent. That should however produce an error visible in the web UI and logs (Scanner (folder ..., ...): *error*). If you don’t get anything like that, please enable walkfs debug logging and start a new scan. Beware, it will produce a ton of logs. Search those logs to see if the problematic file or any parent of it is mentioned.
I think exclamation points are allowed in windows filenames, so shouldn’t be.
Thanks for your answer!
The full scan doesn’t produce errors.
To not get flooded by the logs I decided to first try something different:
- stop both syncthings
- copy the folder ‘wbt’ that contains known ‘problematic’ files to the top of the sync tree (on both sides)
- start both syncthings
- check what happens
Strangely (and since I disabled ‘ignore permissions’) the files in the directory ‘wbt’ produce errors now (permission denied), because they do not belong to the user syncthing runs under - but! the group permissions are OK (rw for the group the syncthing process runs under).
I remember that being the reason why I initially checked ‘ignore permissions’, but is that intended behaviour? Why can’t the process work on those files where the GID fits and the group permissions are OK? I think an expected and in a way ‘correct’ behaviour would be to be able to work on those files without producing errors. Do you agree?
The second strange thing (but which also maybe is an explanation for the initial problem) is that the original folder (i.e. the files in it) from which I made this copy is not detected with permission errors by the full scan.
This supports your theory that the scan doesn’t even reach the directory!
I’ll now be trying a similar thing as above … only copying the folder to a location inside the sync tree where it is not located beneath an ‘!’-containing directory. Just to give it a try.
What is the operation when the permission denied occurs (ideally just copy&past logs, redacting filenames if necessary)?
Ignore permissions shouldn’t matter when scanning: It does the same things regardless of it is set or not, just adding a flag to the (internal) file info object.
You sure? It will obviously only log an error for the top-level directory that failed, not for any of it’s children (as it can’t reach them).
Yes, I’m sure. It didn’t log any error.
Maybe it did once? Does it keep the state in a database?
It looks like it does - because:
I moved the directory somewhere else, then did a full scan, which produces ‘permission denied’. Then I moved it back to the original location, did a full scan again, and now it also produces ‘permíssion denied’ - where it didn’t before!
This looks to me like the files/dirs have been in a database (marked with ‘permission denied’ or ‘not reachable’), and by moving it away they got deleted there, while after being moved back, they now are recognized again (with ‘permission denied’).
A new strange thing now is, that on one side only the topmost directory produces the error, while on the other side there are errors for every file in it.
Permissions on both sides are identical.
Is it correct that the ‘permission denied’ error is one of not-being-able-to-write-on-the-other-side, not one of I-can’t-read-it-here ? This would explain the difference, because syncthing tried to update ‘more’ from one side to other (the new mtimes) and here I get more errors.
Is it intended behaviour that files/dirs are not writeable for the syncthing process even if the group permissions allow it? And if so, why that?
Best regards Matthias
Nope. Scan errors are reset on every scan, and thus also logged again.
Please post actual lines from the log. I still don’t know what the operation is, where permission denied errors occur. And there’s other relevant info to be gained from logs. I am not ignoring your questions willfully, but because I can’t answer them with the info here. And if I guess at the missing info, the answer becomes “that’s not possible” - which is clearly not helpful.
Certainly you’re right (esp. with ‘that’s not possible’). I should’ve posted logs and I will asap.
But for now I had to revert to ‘ignore permissions’ so users’ modifications are sure to travel along.
Hi Simon,
thanks for that!
I patched the source of 1.16.1 to include your fix, then compiled and installed it at the two affected systems, but unfortunately the behaviour seems to be the same as before, i.e. the modification time does not get updated when it’s the only changing parameter of a file.
Best regards Matthias
The “fix” only affected watching for changes. You previously wrote that full scans don’t pick up modification time only changes - no change happened regarding that. And for that I still needs logs because with the info available, I have no idea what is happening:
OK, here you go…
I set up a second process to avoid disturbing the working guys and configured it with STTRACE=walkfs and with ‘ignore-permissions=true’ on both sides.
Now, how can I post the logs as file attachments? Can I?
Edit: Do you want me to post it inline?
You should be able to attach, or https://gist.github.com/ is an option - better not to inline.
Previously you mentioned “permission denied” errors. I’d also be interested in just those errors (as in the full log line containing that error) plus if necessary a short explanation of what you did before that error occurred.
OK, I found it. 
The permission denieds occurred with ‘ignore permission=false’ on files where the group permissions are OK (sufficient for the syncthing process) but the owner is not the owner of the process. Will that still be interesting? syncthing.source.debug.bz2 (67.4 KB) syncthing.target.debug.bz2 (29.9 KB)
Here the first file is from the side where files have been updated (mtime) by the user, and the second file is from the side where the change doesn’t propagate to.
Please check entries reg. files in the directory “!Arbeitsdateien/wbt/xml”.
Best regards Matthias
One of the affected files in wbt/xml for instance is GECC_4_18.xml.
Please let me know if you need anything else.
Best regards Matthias
There’s no relevant permission denied errors - syncthing just doesn’t have permission to write logs to /var/log/syncthing.log:
syncthing.source.debug:May 19 12:16:50 2tgras syncthing[1021182]: [monitor] WARNING: Failed to setup logging to file, proceeding with logging to stdout only: open /var/log/syncthing.debug: permission denied
syncthing.target.debug:May 19 12:16:55 2tsrv syncthing[1150864]: [monitor] WARNING: Failed to setup logging to file, proceeding with logging to stdout only: open /var/log/syncthing.debug: permission denied
Both source and target see GECC_4_18.xml, as in there’s successful LStat calls on them. Other than that the log doesn’t show any activity (changed files or syncing). Lets take a step back: Is the following an accurate description of the problem?
There is a file where only the modification time has changed, contents unchanged, on the source device. Despite triggering a scan by pressing “Rescan” in the web UI on source, the changed mod. time is not reflected on the target device.
If the above is correct, please run the following command (replacing *folder-id* and *path* with actual values of your setup) on both source and target and post the output here, together with the mod. time as observed with independent tools on the fileystem:
syncthing cli debug file *folder-id* *path*
If you encounter an error, first run
syncthing cli config gui debugging set true
Sorry, I’m late.
Yes, your description is accurate.
I can’t seem to be successfull in entering the command you suggested.
I need --gui-address and --gui-apikey for it to work, but the command responds with ‘invalid API key’ despite - of course - using the correct API key.
Best regards Matthias