I have a problem with “interrupted system call” errors on multiple Android devices. This seems to happen when a large number of small files is being scanned, e.g. when starting Syncthing. I have experienced the issue on three Android devices. All of them run Android 4.4.4 and Syncthing Fork 126.96.36.199. The file system in all cases is the default ext4.
Looking at the issue, the lstat lobotomisation is due to a broken android filesystem/fuse implementation. That lobotomisation is not used by the third-party fs watcher library. Given this hasn’t come up anymore, I assume the underlying error in Android has been fixed sometime after Android 4.4.4. Given that’s ancient and long-time unsupported, I don’t think it’s worth investing any time in fixing this. You’ll just have to disable watching for changes.
Could we have a RC to gather feedback from users on this? I also expect this isn’t important anymore… Android 4.4 devices also became rare and those who have one often lack enough RAM to run Syncthing with a collection of photos and music (just not telling about XY GB repos here).
Well, I can disable watching for changes, but I believe that the problem is something else than that.
I have removed the files on the devices in order to try to sync everything again. On one device, I have replaced Syncthing Fork with the official Syncthing application. The synchronisation completed successfully.
However, on another device, I have tried to synchronise the files again with Syncthing Fork, and after a few hours the application has silently crashed. This is what the log file looks like. It seems that all file operations end with “interrupted system call”.
I have one thing in mind, which may possibly be the culprit and is completely unrelated to Syncthing. All the devices use the same specific I/O scheduler. I will try to change it to a different one, and then see whether there is any difference.
Please make sure that the code has in fact been fixed in newer versions of Android. There may be no reports because the hack is actually working and doing its job .
Thing is the hack is only for one specific operation, and it’s not used by the file system watcher which is the default now. Your problem seems much more pervasive than anything we’ve ever seen on stock android before.
I have used both Syncthing and Syncthing Fork on Android for several months, but this problem has only started to show up very recently when I started to sync thousands of small files. Before, when the number of files was counted in hundreds, everything seemed to work fine.
I have done more testing:
The scan: readdirent: interrupted system call related to File Watcher happens both in Syncthing and Syncthing Fork.
The other interrupted system call errors have so far only occurred in Syncthing Fork, although I am not sure yet whether there is any correlation here, or just a pure luck.
Right now, I have managed to complete the scanning of 22,000+ files on two devices with no errors. I switched to the official Syncthing application in both cases. I also disabled File Watcher.
Changing the I/O scheduler in Android does not make any difference.
On a side note, I have also another issue, where Syncthing seems to crash during a long scanning operation on the device with 512 MB of RAM. I suspect the low amount of memory as the culprit, as there are no crashes on the two other devices with 1 GB and 2 GB of RAM respectively.
Unfortunately, the log file produced by the Syncthing application seems to be overwritten and restarted during the crash, so there is no information there about what exactly has happened. Is it possible to have the Android application keep the old log files, as it is the case with the Windows version of Syncthing (other than running the Linux version directly from terminal)?
I have disabled watching for changes on all three devices, and right now all of them have successfully completed the initial synchronisation.
Two of them are using the official Syncthing application, while the third I have left on Syncthing Fork. I like the Fork mainly due to its ability to start and stop Syncthing on demand, but will probably leave everything in this state for now, just to see whether everything works correctly for a longer period of time.
I am still not sure though whether the issue is limited just to this specific version of Android, as it only manifested itself when trying to sync such a large number of files. There had been no such issues when the number of files was low, and I doubt that there are many other people also trying to sync tens of thousands of files on Android.
I may try to sync the same files on a newer version of Android later and see what happens there.
I have just seen this on the third device running Syncthing Fork with watching for changes disabled.
Thanks for the extensive testing and write up of the results first.
I’ll keep an eye on that, but currently have no clue why the fork’s SyncthingNative should be “more” susceptible to fail with readdirent_interrupted than the official. I guess the wrapper does not cause it.
It would be interesting what Syncthing-Fork>Menu>About shows as max open file handle count on the different Android devices. You would not need to setup the fork to sync anything to get that value. It could be even grabbed via ADB shell.
Yeah, I am also not sure whether the Fork is actually responsible for the errors. However, for the last 2 days there have been no errors on the two devices with the official Syncthing application. The overall configuration is exactly the same - watching for changes disabled with 1h rescan interval.
Unfortunately, the values seem to be unavailable. They are shown as “N/A”.
The errors have also shown up on the two other devices running the official Syncthing application, but it was usually only a single file, and they went away after some time. This probably means that the errors are unrelated to the Android wrapper.
Since the errors seem to go away by themselves, Syncthing in this state is still usable.
Turning watching for changes on makes the errors appear almost immediately. With watching for changes turned off, the errors seem to appear much less frequently.
However, I have also experienced the same errors in Android 7.1 with Syncthing Fork 1.3.4.
In this case, there are only 2200 files, but almost all of them are tiny small files. My guess would be that the errors appear when Syncthing is scanning through a large number of small files very quickly.