Syncthing not syncing modification time

Support
1

Situation: two linux servers running syncthing-1.15.1 on a large folder.

When I create a file (empty or not), it gets synced to the other side after a few seconds and the timestamp (mtime) is OK. Date and time on the two systems are the same.

When I touch the file a few minutes later, the new mtime does NOT get synced to the other side, whereas it should according to the syncthing documentation and everything I read about the topic.

What can be the reason for it? Can it be connected to mount options (esp. noatime)? What additional information do you need?

Thanks for suggestions!

2

It should be synced.

I suspect touching the file does not trigger inotify as there is no write on the file, in which case you might need to press the rescan button for syncthing to detect the change in mtime.

3

Thanks for your answer!

The reason for my investigation is that users complained. They saved unchanged files from an application on a Windows client to a samba share on the linux server. I don’t know if it actually re-saves the files or if it only updates the mtime. These files are not being synced, where the update of the modification time is somehow important to the users.

But: even the nightly full scan didn’t resolve it! (Maybe we could live with that.)

Best regards Matthias

4

Remote filesystems such as samba/NFS might not even support inotify etc, so you are fully relying on periodic scans I suspect.

You don’t have to wait for the nightly scan, you can just press the button in the UI, at which point I’d expect the change to propagate.

If they don’t, it could be due to having some weird device/filesystem involved (Some NAS/RPI with exFAT or FAT32 filesystem etc).

5

Thanks again!

inotify works perfectly when a file’s content actually changes (even when the size doesn’t change!), and then of course also the mtime gets updated.

But when only the mtime (or only permissions) are being changed then nothing gets synced.

This is not how it should be as I understand the concept and the documentation of synthing.

There are no weird filesystems involved: ext4 on the servers, and the samba shares offer SMBv3 to Win10 clients.

I’m very happy with syncthing since many years (at another setup), but this really is kind of strange.

Best regards, Matthias

6

Is Syncthing running on the server, i.e. is pointed directly at ext4, or is it running on windows, pointed at a samba share?

First of all we need to establish if the problem even exists if a full scan happens - then whether or not inotify works for mod. time change only is irrelevant. I am not entirely sure from the conversation so far whether that is the case - please clarify.

7

Syncthing is running directly on the server(s) pointed at ext4 (at the moment with ‘user_xattr,acl,noatime’ mount options).

You’re right - I’ll initiate a full sync now and then re-check if the mtime still doesn’t get synced. (Where from yesterday till today it didn’t, and there should’ve been a full sync running last night).

then whether or not inotify works for mod. time change only is irrelevant.

For the users it wouldn’t be irrelevant, but it would be ‘manageable’.

8

Addition: ‘ignore permissions’ is set, so that probably explains why a permission change doesn’t get synced.

9

I didn’t mean whether or not inotify works is irrelevant, but that if a full scan doesn’t pick up the change, notifications are irrelevant because they just trigger a partial scan, that does the same thing as a full scan.

10

OK, understood, thanks.

Now a full scan was triggered via the GUI, first on the side where the files have the new mtime and after that on the side where the mtime didn’t get updated.

Unfortunately neither of those updated the mtime on the target side. :frowning:

11

Just to confirm, we’re talking about files here, correct? We don’t sync mtimes for directories (because there is no consistency on how they are handled between different operating systems).

If it’s files, please get the output of GET /rest/db/file — Syncthing v1 documentation from both sides.

You need to get the API key from the UI, and specify the query parameters (and quote the URL as bash shell doesn’t like & etc).

1 Like
12

Yes, it’s files.

Sorry, I’m somewhat brainstuck: how do I issue the query from bash?

13

For future reference: Once you run v1.16.0 (currently in RC) the command on a system is just syncthing cli debug file *folder-id* *path*. Before that you need to use curl like in the example in the docs (REST API — Syncthing v1 documentation): curl -X POST -H "X-API-Key: abc123" "http://localhost:8384/rest/db/file?folder=*folder-id*&file=*path*".

14

Sorry, it doesn’t seem to work:

“Method not allowed” after that and entering the GUI password:

curl -X POST --basic --user admin -H “X-API-Key: H…myAPIKey…6” “http://localhost:8384/rest/db/file?folder=myFolder-ID&file=t.t”

(The log shows a successfull login.)

15

It should be a GET and not a post.

1 Like
16

I should’ve been more confident in my own judgement with that. :wink:

The ‘source’ side:

{

  "availability": [

    {
      "id": "BPYVG6E-K2LEUYO-NXSYYOS-ZUL2XPG-XGLGMKM-XJBJ6UG-XFMRYBG-STLSNQG",
      "fromTemporary": false
    }
  ],

  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-04-29T15:45:52.968831117+02:00",

    "modifiedBy": "LESOTGJ",
    "mustRescan": false,
    "name": "t.t",
    "noPermissions": true,
    "numBlocks": 1,
    "sequence": 4906949,
    "size": 0,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "LESOTGJ:1619703962"
    ]
  },
  "local": {

    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-04-29T15:45:52.968831117+02:00",
    "modifiedBy": "LESOTGJ",
    "mustRescan": false,
    "name": "t.t",
    "noPermissions": true,
    "numBlocks": 1,
    "sequence": 4906949,
    "size": 0,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "LESOTGJ:1619703962"
    ]
  }
}

The ‘target’ side:

{
  "availability": [

    {
      "id": "LESOTGJ-C2XZRS4-KGQFFVB-N7WPGYG-YVGUGCF-JB3WZDJ-O4E76W6-2ZEYKQ3",
      "fromTemporary": false
    }
  ],
  "global": {

    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-04-29T15:45:52.968831117+02:00",
    "modifiedBy": "LESOTGJ",
    "mustRescan": false,
    "name": "t.t",
    "noPermissions": true,
    "numBlocks": 1,
    "sequence": 4906949,
    "size": 0,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "LESOTGJ:1619703962"
    ]
  },
  "local": {

    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-04-29T15:45:52.968831117+02:00",
    "modifiedBy": "LESOTGJ",
    "mustRescan": false,
    "name": "t.t",
    "noPermissions": true,
    "numBlocks": 1,
    "sequence": 2482353,
    "size": 0,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "LESOTGJ:1619703962"
    ]
  }
}

I hope this can help.

Your efforts are very much welcome!

17

mtime is the same, according to syncthing, so if they are not the same on disk, that means the filesystem did not preserve it whenever syncthing set it.

18

(OK … or Syncthing didn’t propagate it to the filesystem…)

But … the time you see in the output above (“modified”) is the ‘old’ mtime, so Syncthing didn’t catch the mtime after the file has been re-touched on the source side.

And this - please remember - survives a full scan!

Can I provide anything to dive deeper into that issue? A debug log, or part of it?

Best regards and thanks again for your suppport! Matthias

19

There is an abstraction layer in syncthing to compensate for filesystems that don’t preserve mtimes, whereby it:

  1. Sets the mtime
  2. Reads the mtime
  3. If they are not equal, sets a value remapping “this mtime on disk actually means this mtime in the database state because the fs is unable to preserve it”.

Then every time we scan, we replace the mtime on disk with mtime in the database when considering if the file has changed.

I don’t think there is a way to look at the virtual timestamp remappings, as we don’t log it anywhere.

I don’t think syncthing could have failed to propagate the timestamp. The chtimes syscall could have failed, but that would just create the mapping I am talking about, as the time would not align to what we expect it to be.

Failure to set modtimes is not critical in syncthing, so we throw away any errors along the way and use the remapping feature.

If it didn’t create the mapping, during the next scan the time change would have triggered a rescan (as the mtime did not match what we supposedly had in the database), and the time would have been propagated outwards.

You could enable scanner logs, press the rescan button and capture the logs, but I suspect there won’t be anything interesting logged there.

1 Like
20

Maybe the fs respectively walkfs logs would provide some information on what’s going wrong in the filesystem.