Both folders Up to Date, one device remains Syncing 99%

Both folders are “Up to Date” with the same file and folder counts in both global and local states.

Device A shows device B as “Up to Date”.

Device B shows Device A as “Syncing (99% 11.9MB)” for the last day.

Device A has v0.14.44, B has v0.14.40. Both are Linux 64bit.

A rescan of the the folder on Device B had no effect.

The folder on Device A is an NFS mount with a rescan interval of 7200s. FS watcher is not enabled.

The folder on Device B is on a local SSD, uses the built-in inotify FS watcher, and has a rescan interval of 28800s.

Recent Changes/Global Changes do not appear to show anything interesting and list the same events on both ends.

The outputs of rest/db/status?folder=src contain some differences.

Folder status on device A

{
   "globalBytes" : 59652757572,
   "globalDeleted" : 1323305,
   "globalDirectories" : 1470,
   "globalFiles" : 46749,
   "globalSymlinks" : 31,
   "ignorePatterns" : false,
   "inSyncBytes" : 59652757572,
   "inSyncFiles" : 46749,
   "invalid" : "",
   "localBytes" : 59652757572,
   "localDeleted" : 21,
   "localDirectories" : 1470,
   "localFiles" : 46749,
   "localSymlinks" : 31,
   "needBytes" : 0,
   "needDeletes" : 0,
   "needDirectories" : 0,
   "needFiles" : 0,
   "needSymlinks" : 0,
   "pullErrors" : 0,
   "sequence" : 4145745,
   "state" : "idle",
   "stateChanged" : "2018-02-13T11:05:46.426603912-08:00",
   "version" : 4145745
}

Folder status on device B

{
   "globalBytes" : 59652757572,
   "globalDeleted" : 1323304,
   "globalDirectories" : 1470,
   "globalFiles" : 46749,
   "globalSymlinks" : 31,
   "ignorePatterns" : false,
   "inSyncBytes" : 59652757572,
   "inSyncFiles" : 46749,
   "invalid" : "",
   "localBytes" : 59650716903,
   "localDeleted" : 1323301,
   "localDirectories" : 1470,
   "localFiles" : 46749,
   "localSymlinks" : 31,
   "needBytes" : 0,
   "needDeletes" : 0,
   "needDirectories" : 0,
   "needFiles" : 0,
   "needSymlinks" : 0,
   "sequence" : 4145745,
   "state" : "idle",
   "stateChanged" : "2018-02-13T10:41:15.736323204-08:00",
   "version" : 4145745
}

An unrelated question - Should inotify be able to notice filesystem changes initiated on the client side of an NFS mount? I don’t care that inotify on the NFS client is not able to notice fs changes initiated by the NFS server because all changes come from the client. The NFS mount on Device A in the case above is very slow and the folder is relatively large. (Issues with Synology DSM uid’s and NFS uid mapping prevent me from running syncthing on the NAS).

Summary

This text will be hidden

The problem was solved by a reset of the folder index database with POST /rest/system/reset on Device B.

https://docs.syncthing.net/rest/system-reset-post.html

It’s good that a reset worked but it would be helpful to identify the cause through the web ui. This comes up not infrequently. Would it be possible to show why the localBytes values differ when the folders are reported as “Up to Date”?

GET /rest/db/status now shows differences only in localDeleted and stateChanged.

Device A

{
   "globalBytes" : 59651069028,
   "globalDeleted" : 23,
   "globalDirectories" : 1470,
   "globalFiles" : 46748,
   "globalSymlinks" : 31,
   "ignorePatterns" : false,
   "inSyncBytes" : 59651069028,
   "inSyncFiles" : 46748,
   "invalid" : "",
   "localBytes" : 59651069028,
   "localDeleted" : 23,
   "localDirectories" : 1470,
   "localFiles" : 46748,
   "localSymlinks" : 31,
   "needBytes" : 0,
   "needDeletes" : 0,
   "needDirectories" : 0,
   "needFiles" : 0,
   "needSymlinks" : 0,
   "pullErrors" : 0,
   "sequence" : 191063,
   "state" : "idle",
   "stateChanged" : "2018-02-14T11:14:38.088494788-08:00",
   "version" : 191063
}

Device B

{
   "globalBytes" : 59651069028,
   "globalDeleted" : 23,
   "globalDirectories" : 1470,
   "globalFiles" : 46748,
   "globalSymlinks" : 31,
   "ignorePatterns" : false,
   "inSyncBytes" : 59651069028,
   "inSyncFiles" : 46748,
   "invalid" : "",
   "localBytes" : 59651069028,
   "localDeleted" : 0,
   "localDirectories" : 1470,
   "localFiles" : 46748,
   "localSymlinks" : 31,
   "needBytes" : 0,
   "needDeletes" : 0,
   "needDirectories" : 0,
   "needFiles" : 0,
   "needSymlinks" : 0,
   "sequence" : 191063,
   "state" : "idle",
   "stateChanged" : "2018-02-14T11:14:39.609488151-08:00",
   "version" : 191063
}

To update my question about inotify and NFS:

FS-watcher notices filesystem events on the client side of an NFS mount. Creating a file in the NFS mount puts the Syncthing folder into a “Scanning 0%” state for about 30-60s in the case of a large folder (46000 files, 1400 directories). Then the change appears globally. This process appears to be faster for folders containing few files.

I assume that stating the entire nfs filesystem is just pretty slow. Running Syncthing on an NFS mount sounds like a bad idea - why not just run it on the server side?

Because when I bought the Synology server I did not know that their user id’s start at 1024.

Syncthing does not sync uid/gid’s. It writes all files with the uid of the process owner. Synology uid’s start at 1024 for some reason and they refuse to address this issue. Reportedly it is not possible to map an NFSv4 client uid to different uid on the server without a Kerberos server on the network. If I ran Syncthing on the Synology NAS, then I would have to change all the uid’s, set up a Kerberos server, or switch to SMB.

That sounds annoying, but I guess then you need to live with the delay when scanning in that case.

If you have inconsistent file/dir/bytes stats, you (most likely) don’t need to reset the index, recalculating these stats should suffice. You can do so by running Syncthing once with STRECHECKDBEVERY=0 env var set.
As to the cause of it: If you notice any patterns or if there is anything “specific” about your setup, please let us know. We are aware of the problem, but we don’t really have any leads so far.

Thanks. I will try the new STRECHECKDBEVERY env var next time.

I can live with the scanning delay until I find a way to run syncthing on the NAS. FS-watcher working with NFS mounts helps a lot in my case (even though it only notices changes initiated by the NFS client).

One specific thing about my setup is that one node (“Device B”) is running gentoo and syncthing on that node is compiled by portage (emerge net-p2p/syncthing). When it starts, everything looks normal in journalctl except the following messages. These only appear on starting syncthing. /var/tmp/portage/ is a temporary compile-time directory.

[UBDM4] INFO: connections.Service: Failed service '(connections.serviceFunc)(0x8e59e0)' (1.000000 failures of 5.000000), restarting: true, error: runtime error: invalid memory address or nil pointer dereference, stacktrace: goroutine 52 [running]:
github.com/syncthing/syncthing/vendor/github.com/thejerf/suture.(*Supervisor).runService.func1.1(0xc42023c460, 0xcc28c96f00000001)
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/vendor/github.com/thejerf/suture/supervisor.go:448 +0xbc
panic(0xab8580, 0x1110a20)
        /usr/lib64/go/src/runtime/panic.go:491 +0x283
github.com/syncthing/syncthing/vendor/github.com/AudriusButkevicius/kcp-go.(*updateHeap).removeSession(0x1122b60, 0x0)
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/vendor/github.com/AudriusButkevicius/kcp-go/updater.go:65 +0x3b
github.com/syncthing/syncthing/vendor/github.com/AudriusButkevicius/kcp-go.(*UDPSession).Close(0x0, 0x0, 0x0)
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/vendor/github.com/AudriusButkevicius/kcp-go/sess.go:307 +0x4d
github.com/syncthing/syncthing/lib/connections.(*kcpDialer).Dial(0xc4211fcff0, 0xcd0fe6e8ac44841a, 0xde462139c25a175b, 0x3841c14258ebeaf8, 0x83308d667561a49e, 0xc4201d4980, 0x11232c0, 0x5, 0x1, 0xc420014190, ...)
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/lib/connections/kcp_dial.go:48 +0x330
github.com/syncthing/syncthing/lib/connections.(*Service).connect(0xc4214aca50)
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/lib/connections/service.go:403 +0x1200
github.com/syncthing/syncthing/lib/connections.(*Service).(github.com/syncthing/syncthing/lib/connections.connect)-fm()
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/lib/connections/service.go:150 +0x2a
github.com/syncthing/syncthing/lib/connections.serviceFunc.Serve(0xc421678390)
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/lib/connections/structs.go:152 +0x25
github.com/syncthing/syncthing/vendor/github.com/thejerf/suture.(*Supervisor).runService.func1(0xc42023c460, 0xc400000001, 0x10bce80, 0xc421678390)
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/vendor/github.com/thejerf/suture/supervisor.go:454 +0x63
created by github.com/syncthing/syncthing/vendor/github.com/thejerf/suture.(*Supervisor).runService
        /var/tmp/portage/net-p2p/syncthing-0.14.40/work/syncthing-0.14.40/src/github.com/syncthing/syncthing/vendor/github.com/thejerf/suture/supervisor.go:444 +0x5b

This is an old version with bugs, use the latest version.

The startup errors go away with 0.14.44. I’ll try STRECHECKDBEVERY if this comes up again.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.