I apologize if the solution to this is simple, i am NOT an expert at Syncthing OR Rasbian (or this style OS in general).
I was running Syncthing version 1.5.0 off a Raspberry Pi4 running Rasbian at 2 locations. I was using this to Sync data from Site A to Site B (one way, nothing from Site B goes back to Site A. I recently went to update the packages on Site B, and it updated (at that time) to version 1.10 or Syncthing and i started to notice these errors it would say “readdirent: no such file or directory” or sometimes say “lstat /home/pi/ISENAS/Analysis_Archive/Schlumberger/SLB_Odie/SLB_226C_Top_Anys02_120915/1x025_angle.prt.1: interrupted system call” (would change based on file name of course).
At that time i did not have the time to deal with the issue, so i simply pushed a backup of rasbian and the installed packages onto the SD card (before i updated everything) and it worked fine and I moved on, by just selectively updating all packages except Syncthing. Finally today i tried again as i say Syncthing was up to version 1.11.1, however i am still getting these errors.
I could find one thread that was close to my issue readdirent: no such file or directory only on > 1.6.0 however i never really saw a resolution, but it did lead me to read up (at that time a month or so ago) on ‘Go’ and CIFS, however i found no real ‘fix’ so that is why i just reverted to a backup as i knew 1.5 worked.
I was hoping someone might have a better idea or best of all a fix for this issue at this time (as i now have some time to play with it on/off).
Only thing ‘strange’ about this Syncthing setup might be that the directory(s) it is scanning/using (on both RPI4’s) are network mounts to a network share on a fairly old Leonvo NAS device.
The file system is unexpectedly telling Syncthing that this are being interrupted, times out, or don’t exist. This is probably due to a problem there, flaky network, or similar.
I thought that at first too when i had this happen a month or so ago on 1.10, however tried different ether net cables, tried different switches, nothing helped except reverting to 1.5 and it worked great again.
Also i tested using filezilla to connect to the Rpi4 and then copy data from the shares on it that link back to the Lenovo NAS, and it never had any issue moving data that way. It is something Syncthing is doing different in at LEAST very 1.10+ vs version 1.5.
One thing have not tried is deleteing everything and rebuilding syncthing and re-creating the connection between the 2 Syncthing devices. I have tried to avoid doing this as there are so many millions of files it took ~4 days to build the database originally when scanning.
1.8 added handling for case insensitive filesystems, which causes more directory lookups. This isn’t a bug, but probably puts more stress on your system, hence the error occurring (more frequently) in 1.8+.
So, do you still feel this is a network/hardware issue then? I feel fairly strongly that i have eliminated the possibility of it being the network cables themselves and/or the switches, as i have tried multiple patch cables and even moved the pi from it’s normal location to the wiring closet and had it on the same switch as the NAS.
I suppose the issue could be with the NAS hardware itself, just surprised that syncthing did not have issues before if that was the case, and using freefilesync (even with like 50 consecutive files at once) never caused any issues.
Any suggested tests i could do would to help narrow down this issue, would be helpful, especially between the Rpi4 and the NAS itself. I know there is some command line interface on these iomega/lenovo branded NAS(s) but never messed with it in the past but perhaps i can run iperf between the NAS and Rpi4. I will also look at trying the 2nd ethernet port on the NAS and seeing if i can set up bonding between the 2 NAS ports and the ubiquiti switch it goes into for testing purposes and see if that helps.
I realized instead of all the work with the NAS i could simply create a new share pulling off a different device with a network share. Tried this, and it immediately popped up with the “readdirent: no such file or directory” however it is doing the initial scan, due to seeing that ‘working’ i have removed ONE of the original sync folders and i added it back in and i am letting Syncthing Scan it initially again, it also (as soon as i added it) gave the “readdirent: no such file or directory” but is doing the initial scan. Once those complete i will see if it can do the periodic scans, i believe that may be what is making this error pop up, but as for WHY it is occuring i am not sure still.
All I can say is that readdirent: no such file or directory is the OS/filesystem saying “the directory entry you’re looking at doesn’t exist” and that the two of you who experience this problem are apparently both using CIFS. That suggests to me a CIFS problem.
I honestly at this point agree with ya, i may pick up a 2nd Rpi4 just to tinker with this. I am circling back to the info i found a month or so ago, in threads like this.
I will play around and see if i can figure something it, it sounds like this is a program/function called ‘go’ and it is only certain versions. Not sure if this was something Syncthing started using post V1.5 and rasbian contains it or if it is included inside syncthing itself.
I also need to see if i can mount these drives some other method than a CIFS share.
The first link you posted summarises the problem you are seeing and it also suggests how to prevent it, by disabling async preemption via GODEBUG env var.
Other option is to not use CIFS or maybe try moving to a more recent kernel.
Thank you everyone for the help, for now the 4 main folders i care about i have converted to NFS mounts from CIFS and this fixed the issue/error i was getting with CIFS. I will test some of the ‘fixes’ on a CIFS share i will use as a ‘test’ just because i love working on ‘problems’ like this
One quick final question, can syncthing 1.11.1 sync/communicate with syncthing 1.5.0? i unpaused the connection to the remote device today after it looked like it was scanning without errors and i see it is having lots of issues re-establishing the connection to the remote syncthing at the other location.
Yes it can. However there have been lots of bugs fixed since, so it’s generally adviceable to upgrade and if you do see problems, the first step of debugging would be to upgrade.
