readdirent: no such file or directory only on > 1.6.0

Tom · June 9, 2020, 10:55am

I did a sudo apt install synthing=1.6.0, coming from 1.4.1.

The webif showed :

For the following folders an error occurred while starting to watch for changes. It will be retried every minute, so the errors might go away soon. If they persist, try to fix the underlying issue and ask for help if you can’t. Support

movies readdirent: no such file or directory

That folder certainly isn’t missing. So, I went back to 1.4.1 , all is well.

Then I went from 1.4.1 to 1.4.2 to 1.5.0 without anything to report. Sync works fine.

Until I went to 1.6.0, and again troubles. I grabbed this from syslog

Jun  8 21:41:29 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:29 INFO: Error while trying to start filesystem watcher for folder movies, trying again in 1m0s: error while traversing /mnt/share/media/video/movies/Some folder name: interrupted system call
Jun  8 21:41:31 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:31 INFO: Error while trying to start filesystem watcher for folder tvshows, trying again in 1m0s: readdirent: interrupted system call
Jun  8 21:41:31 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:31 INFO: Error while trying to start filesystem watcher for folder pictures, trying again in 1m0s: readdirent: no such file or directory
Jun  8 21:41:32 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:32 INFO: Completed initial scan of receiveonly folder "surface_adrian_pictures" (fsqtr-adr4a)
Jun  8 21:41:32 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:32 INFO: Completed initial scan of sendonly folder "3D" (mmaf6-gkdez)
Jun  8 21:41:37 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:37 INFO: Completed initial scan of receiveonly folder "surface_adrian_documents" (t4zjm-sv7sc)
Jun  8 21:41:39 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:39 INFO: Detected 3 NAT services
Jun  8 21:41:47 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:47 INFO: quic://0.0.0.0:22000 detected NAT type: Port restricted NAT
Jun  8 21:41:47 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:47 INFO: quic://0.0.0.0:22000 resolved external address quic://213.219.146.215:22000 (via stun.syncthing.net:3478)
Jun  8 21:41:58 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:41:58 INFO: Joined relay relay://95.102.55.116:22067
Jun  8 21:42:03 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:42:03 INFO: Completed initial scan of receiveonly folder pictures
Jun  8 21:42:16 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:42:16 INFO: Completed initial scan of sendonly folder movies
Jun  8 21:42:30 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:42:30 INFO: Error while trying to start filesystem watcher for folder movies, trying again in 2m0s: readdirent: no such file or directory
Jun  8 21:42:32 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:42:32 INFO: Completed initial scan of sendonly folder tvshows
Jun  8 21:44:30 syncthing syncthing[126]: [JHCJA] 2020/06/08 21:44:30 INFO: Error while trying to start filesystem watcher for folder movies, trying again in 4m0s: readdirent: no such file or directory

I don’t really know where to start logging/troubleshooting for this one?

The folders are a mounted cifs share. The system doing the sharing is an LXC hosted by Proxmox. Syncthing is also an LXC on the same Proxmox.

imsodin · June 9, 2020, 11:15am

Maybe that’s related to go1.14, at least that’s the only potentially relevant change I can think of.

Anyway it’s not actually a problem, actually the contrary: Inotify doesn’t work on cifs shares. I.e. before it didn’t complain, but also didn’t detect any changes (see e.g. https://stackoverflow.com/questions/8124617/getting-file-create-notifications-for-cifs-mount-in-linux).

Tom · June 9, 2020, 11:18am

so, turning of inotify should resolve ?

I don’t really care about inotify in this particular setup anyway. Will test & revert.

Tom · June 9, 2020, 11:38am

well,I went to 1.6.0 and no joy.

I figure both of those are due to :

Jun  9 13:35:04 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:04 INFO: Scanner (folder movies, item "Planet 51 (2009)"): scan: readdirent: interrupted system call
Jun  9 13:35:07 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:07 INFO: Joined relay relay://51.68.174.240:22067
Jun  9 13:35:09 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:09 INFO: quic://0.0.0.0:22000 detected NAT type: Port restricted NAT
Jun  9 13:35:09 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:09 INFO: quic://0.0.0.0:22000 resolved external address quic://213.219.146.215:22000 (via stun.syncthing.net:3478)
Jun  9 13:35:19 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:19 INFO: Completed initial scan of sendreceive folder pictures
Jun  9 13:35:32 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:32 INFO: Completed initial scan of sendonly folder movies
Jun  9 13:35:37 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:37 INFO: Scanner (folder tvshows, item "Jamie- Keep Cooking and Carry On/Season 01"): scan: readdirent: interrupted system call
Jun  9 13:35:39 syncthing syncthing[139]: [JHCJA] 2020/06/09 13:35:39 INFO: Scanner (folder tvshows, item "The Witcher"): scan: readdirent: interrupted system call

It is still in the process of exchanging the indexes, but the errors don’t promise anything good - amirite ?

calmh · June 9, 2020, 11:51am

Yeah that looks like some bad interaction with CIFS/LXC

Tom · June 9, 2020, 11:54am

Is there anything I can do to help out ?

calmh · June 9, 2020, 12:03pm

Can you pinpoint which thing causes the problem? That is, does Syncthing work under proxmox with local folders? Does it work outside proxmox with a folder on cifs? Or some other relevant detail about your system that might be unusual.

Tom · June 9, 2020, 12:17pm

What’s unusual is this :

All those folders reside on the same zfs pool, and are shared by the same LXC. Only those 2 throw errors.

Tom · June 9, 2020, 12:27pm

I clicked Rescan on the failed folders, they briefly showed as Scanning, then went back to Failed.

I refreshed the webif ( F5 ), and all folders go green / Up to Date .

All the while I had a terminal window opened, tailing the log. Absolutely nothing was added to the log during the above manipulations.

Then, 5minutes later the log shows :

 [JHCJA] 2020/06/09 14:28:02 WARNING: Error on folder movies: stat /mnt/share/media/video/movies/.stfolder: interrupted system call

calmh · June 9, 2020, 9:24pm

Yeah the problem seems unpredictable, probably because it is timing related (interrupt happing at an unexpected time). The above narrowing down would still be useful.

AudriusButkevicius · June 10, 2020, 7:26am

There was a change in Go in interrupt handling which we picked up as we upgraded go, so I think your OS misbehaving here.

Malaber · June 10, 2020, 7:49am

I am getting the same problems in the docker container of version 1.6.1 The go version in there should probably be the correct one, so maybe it is not go/os related?

I also have a CIFS share, probably that is causing the problem then?

AudriusButkevicius · June 10, 2020, 8:06am

When I say go version, I mean version of go syncthing was copiled with.

Version of go in the container does not matter at all, as it’s can’t be used hence not needed.

It’s most likely the SMB/CIFS fuse driver most likely.

Tom · June 10, 2020, 8:34am

I’m setting up test installations, but I won’t have feedback soon because creating the index will take >1day for every test installation.

calmh · June 10, 2020, 8:22pm

This problem should occur during initial scanning.

imsodin · June 12, 2020, 7:59am

These kind of errors may become less frequent with go 1.15 again:

Packages os and net now automatically retry system calls that fail with EINTR . Previously this led to spurious failures, which became more common in Go 1.14 with the addition of asynchronous preemption. Now this is handled transparently.
Go 1.15 Release Notes - The Go Programming Language

system · July 12, 2020, 7:59am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.