"failed to set up inotify handler. please increase inotify limits"

Support Android
1

Background

I use Syncthing on my phone, Arch laptop, and Ubuntu desktop. These devices all share the same exact folders with the same exact options (permissions, rescan interval, etc.). Up until now, syncing files between all my devices has worked properly (albeit a bit too slow and battery-consuming for my taste, especially for folders with many files).

Issue

Recently, two main folders on my Android client have appeared as “Out-of-sync”, despite removing these folders on all my devices and then propagating a fresh copy of them again to them. Only my desktop and my laptop are completely in sync, but my phone’s versions of the 2 folders in question are out of sync, no matter how much I wait. Moreover, I can no longer share new folders with my phone (only between my desktop and laptop); I wasn’t receiving the notifications to share the folders on my phone.

Then, I looked at the Android web GUI and it said this: “Filesystem Watcher Errors — failed to setup inotify handler. Please increase inotify limits. See …”. I went to the specified website and I executed the commands therein on my desktop and even my laptop for good measure (I ran the appropriate Arch command for my laptop), and rebooted, to no avail; I was receiving the same error on my Android client and the folders were still out of sync.

Then I wondered, “Wait, why did I have to run Linux commands to increase the inotify limit when the issue was happening on my phone? Is there an equivalent set of commands for Android?”.

Other Stuff I’ve Tried

I looked up this issue pretty much everywhere: the Github issues page, this forum, and the Arch Wiki. The only advice that I’ve seen for fixing this issue is on the FAQ page, and clearly those commands didn’t work at all. I’m guessing they only apply to Linux machines, not to Android phones.

Screenshot of my phone’s web GUI

20180804_185125
20180804_185125.jpg1080×5928 1020 KB

Final Thoughts

Please help. Also, please include error messages in the official app GUI, not just the web GUI. I didn’t even know the error message would be in the web GUI until I randomly decided to look at it because I assumed the app GUI would tell you about something as important as that.

1 Like
2

Hi, I’ve to look into this when on a computer. Does disabling watch for changes on android solve the out of sync problems? Folder error messages are shown in the app UI since version 0.10.12 I recall file watcher errors are (internally) something different (other json var). Maybe those can be shown too, i’ll have to check that.

3

… This must be a nightmare. I just checked on my phone and max_user_watches is set to 8192 - so this is to be expected. The manual fix needs root and google did not turn up even a single instance of anyone having that problem on android. No idea if there is some way to request more watches without root on Android.

@iusearchbtw If you don’t have root to execute those commands on Android trough adb, the only fix I know is to disable watching for changes. You can disable it for one folder at a time (if there is one huge one, disable it on that) until the error is gone.

1 Like
4

@imsodin Ah, that’s why I didn’t experience this on my phone as I’m using root to sync huge amounts of data without problem. Thanks for helping to investigate this issue.

Should I make a PR to add an explanatory string under the label “Watch for changes” stating “This option should not be enabled on folders containing more than 8.000 objects” if the app detects it is not running under the root account?

1 Like
5

Root by itself doesn’t help, but you need root to increase the max user watches number. And it’s not 8000 items, but directories. I am not sure about an additional explanation - if there is no solution for this then that sounds like a reason to go back to watching directories in the app instead of Syncthing…

6

@imsodin Resurrecting the FolderObserver java thing in the wrapper doesn’t sound good to me because it’s a massive resource hog and would need a better implementation first to be more efficient. For example, it doesn’t have change queueing and just sends every touched file to the scan rest command. If we get into much into trouble with the fswatcher (I really like it since its there on android) we can put it for root users together with running the max limits command and give users withour root the folderobserver… I still need to think about this to come up with a suggestion. Btw how is the command for root users to increase the limits?

7

What works without root is to query max_user_watches so the wrapper could be smart and do whatever makes sense (disable, fallback to external watcher, …) if the number of directories in shared folders exceeds this value.

Increasing limits:
https://docs.syncthing.net/users/faq.html#how-do-i-increase-the-inotify-limit-to-get-my-filesystem-watcher-to-work

1 Like
8

Thanks, i put this on my todo list to make up my mind how we can get the best out of different android and folder use cases.

9

I think we should disable watching on android if we hit that error. I think java watcher would have the same issue.

2 Likes
10

So I’m still a bit confused. According to the web GUI for my phone and for my other devices, my two Out-of-Sync folders (aesthetic and coding) respectively contain 92 and 7551 directories (both global and local). For aesthetic, my phone is missing some files in its local state, but oddly enough coding on my phone has an equal number of files in its local and global state, even though it still says “Out of Sync”. I’m guessing the “Out of Sync” status for coding is because some of the files are different?

More importantly, none of my main folders contain at least 8192 directories; coding comes close, but it only has 7551 directories in total. Is the limit referring to the total number of directories that are being synced across all my folders? Does it also take regular files into account? What’s more confusing is that my laptop and desktop did not receive the “inotify limit” error before I increased the inotify limit, and yet they were syncing perfectly on the same folders.

Meanwhile, after I disabled “Watch for changes” for both coding and aesthetic, the “File Watcher” error did disappear from the phone web GUI, however, the folders still have the “Out of Sync” status. However, I’m now allowed to share new folders with my phone, something I couldn’t do before I disabled “Watch for Changes”.

I’m quite concerned that I might need root to increase the inotify limit on Android. Although, I guess I can unsync some folders that I don’t necessarily need to sync up. For example, I have a coding folder in which I store all my coding projects. Since I have been working on Node projects recently, I have been keeping the node_modules folder, which contains all my needed dependencies for that project. Now I could exclude node_modules from syncing, but that would make my project slightly less portable; I would have to reinstall/sync all the same dependencies on my other work machine manually. There’s also the Android SDK which I don’t necessarily need to sync up, although it would have been convenient to do so. I wanted to keep an exact copy of all my folders on my phone as well so that my phone could be used as a middle man to transfer my work data across my laptop and desktop in case the other work machine is offline.

1 Like
11

8192 is number of file descriptors for the whole process, the application itself probably uses a few hundred file descriptors for connections, database files etc. If you really care you can see ls -lah /proc/<pid of syncthing process>/fd output on the shell, but the problem is not going to go away.

2 Likes
12

I would disable watching for changes on the coding folder and set a suitable regular full scan interval (e.g. 5min) on your Android phone only. This is a local problem, what you do on any other devices is not relevant. That they don’t have the same problem may be due to different default limits.

1 Like
13

So I made some changes, and I’m happy to say that all my syncing issues are gone now.

I did this by creating a .stignore in coding on my phone that excluded android_sdk (my Android sdk folder) and node_modules (the Node dependencies folder). Now, to my knowledge, my laptop and desktop don’t sync to my phone on those folders but they do sync with each other on these folders. That alone eliminated many of the file objects needed to be watched from my phone and ultimately removed the “File Watcher - inotify” error, without having to disable “Watch for Changes” on coding. With regards to my aesthetic folder, apparently that was a completely different issue, since some files weren’t being synced to my phone because I didn’t have permissions to sync them to my phone (even though these were simple jpg, svg files). I removed the offending folders from aesthetic and now my phone doesn’t complain about that anymore.

Just to be absolutely clear though: when I exclude a file/directory in a .stignore on a device, does that mean that it will stop pushing AND pulling changes/additions to that file? In other words, if I exclude a file called something on a device, then would changes to something not propagate to remote devices AND would remote changes to something not modify the local something?

1 Like
14

Correct.

Also note that if you would ever use exclude patterns (prefixed with !), watches will be set up for all directories, also ignored ones.

1 Like
15

Seconded. I’d also expect this as android’s java classes would expose the inotify functionality of the underlying OS to watch for changes.

17

Opened PR with a fix for root users:

Quote: Purpose Fix the issue for root users by temporarily increasing “fs.inotify.max_user_watches” to 128K (instead of 8K).

Related issue: #1208 “Syncthing is running out of inotify watches” Forum discussion: "failed to set up inotify handler. please increase inotify limits"

Testing Verified working running syncthing rooted on Android 7.1.2 on device lg-h815 at commit https://github.com/syncthing/syncthing-android/commit/ef5729d2c8f27d31182f7d61e0666797a3a22870 .

Tests a) Running non-rooted

b) Running rooted

18

@iusearchbtw I guess you don’t have root access on your phone at hand? If you have, feel free to try the debug build at https://build.syncthing.net/repository/download/SyncthingAndroid_Build/24886:id/apk/debug/app-debug.apk and report if it solves the issue. Without root and depending on how you use your data in the sync folders, I can recommend zipping a bunch of files and directories to get below that “8192 directory limit”.

19

Thanks for including the fix! Although, I am not quite sure I want to root my phone. For the time being, I’ll make do trying to unsync unimportant files to my phone.

20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.