How to deal with abysmal I/O performance on Android 10+?

tomasz86 · May 7, 2022, 3:33am

This is not exactly a support request, as I know that the reason why things work as they do is Google and their enforced scoped storage, but I’d like to ask what kind of solutions/hacks could be applied in order to mitigate the problem.

Just to give some context, I’ve got a folder with 11,000 files and it takes 15+ minutes to scan it (without hashing, just rescanning) on a mid-range device from 2019 (Samsung XCover 4S, Android 11). For comparison, this is slower than my old single-core Nexus S from 2010 running Android 4, which was able to go through the same number of files in 5-10 minutes. Another, older yet much faster phone (Samsung Galaxy S5 LTE-A, Android 6), requires just a few seconds to scan the very same folder.

As I understand, the main culprit here is simply the very slow access time caused by all file access forced to go through a Java-based layer, which wasn’t the case in Android 9 and older that allowed direct file access (see https://reddit.com/r/androiddev/comments/kpn68k/android_11_very_slow_file_access_performance).

Is there anything that can be tweaked inside Syncthing itself that could at least mitigate the issue? At the moment, I’ve tried to change the default number of per-folder hashers from 1 to 0 to allow for using all CPU cores, but it hasn’t really made any noticeable difference.

Now, Android provides a way to disable the so called “isolated storage” and enable “legacy storage” though the following commands.

adb shell cmd appops set <app-package-name> android:no_isolated_storage allow
adb shell cmd appops set <app-package-name> android:legacy_storage allow

I’ve used these for a few apps that haven’t implemented proper scoped storage support yet, e.g. Joplin to allow it to use file system sync, and Bromite to allow it to access local HTML files through the file:// URI scheme, as otherwise those applications are unable to access local file paths in Android 10+.

My question is, would using these commands make Syncthing behave as before v1.19, allowing it to access files directly, or would it have no effect in terms of performance? I don’t need external SD card support, so I don’t care if I lose it in the process.

I’d be very thankful for answers to the questions above or any other tips that could help improve the awful performance.

At the moment, any situation, where Syncthing needs to be restarted is a nightmare, as I then need to wait at least 15-20 minutes for all folders to finish scanning, which delays pulling new files from remote devices. Because of this, I’ve now allowed Syncthing to run in all conditions, including battery power, mobile data, and aeroplane mode, as this seems to be the only way to just keep it running all the time without rescanning folders, relying only on the file watcher to detect changes instead (with the full rescan interval set to 1 day or more). This obviously has negative impact on the battery, but I don’t really see any other choice here .

imsodin · May 7, 2022, 9:04am

Can you run these commands without root?

If you do have root, there’s an easy cop-out. I wouldn’t usually recommend it, but this is android, anything goes.

As to whether it will work: I don’t know, and I am pretty sure finding an answer in docs will be harder than just trying it out. I don’t see why it wouldn’t work though, nothing changed in Syncthing, it’s entirely a config in the app (target api) and change in android. So if those command overrides that config, I’d expect it to work as before. Then again it might also be yet another android-ugliness-layer that wraps scoped storage in something that makes it behave like legacy storage

Sorry I can’t be of any more help. Would have been really nice if google took the opportunity of introducing a heavily restricted “all file access” permission to let apps with that permission actually access the filesystem layer.

tomasz86 · May 7, 2022, 9:30am

No problem, I’m thankful for the answer regardless .

The commands do work without root. I’ve tried running them at Syncthing, but I’m not really sure they actually do anything to apps compiled for SDK 30 that support scoped storage already. They seem to be only recommended when using apps compiled for the SDK that don’t support scoped storage yet and thus can’t access any paths outside of their own. The reason why I’m saying this is that I haven’t observed any noticeable performance difference after running them. I don’t think they “override” the config as such, but rather just allow to use the old way of accessing the filesystem if the app is unable to use the new. I’m not an expert though, so I could be wrong here.

I normally like to root all my Android devices, but in this case I’d prefer not to simply because the device is still supported, and it appears that installing updates on rooted Samsung devices requires going through major hoops, as they don’t use the A/B partition layout, so you’ve always got to un-root and re-root when installing each monthly update, which is a no-go for me.

Edit:

I think I’ve found out what problem I’ve encountered specifically. It’s not caused by the scoped storage per se (although related), but it rather seems to be about Android 11 and up using FUSE to handle storage instead of sdcardfs, which was used previously (see https://source.android.com/devices/storage/sdcardfs-deprecate). The problem also appears to be limited to Android 11 (which I’ve got right here), as the slow performance has apparently been dealt with in Android 12 (see https://source.android.com/devices/storage/fuse-passthrough).

If that’s the case, then this would mean that the issue’s not really fixable by tweaking the app, as I’ve also tried to use the older v1.18.3 release, and processing files was still painfully slow there. Disabling FUSE through setprop persist.sys.fflag.override.settings_fuse false seems to be the solution, but the command isn’t permitted to run on my device without root, so no luck with this one, at least for now.

I’ll need some time to think about what to do with this problem, but at the moment I’ve considered three solutions:

Reduce the synced files number to the absolute minimum, so that Syncthing doesn’t need to scan through thousands of them.
Wait for an Android 12 update for the device, but it’s still unclear if and when it’s supposed to get it.
Root the thing and try to disable FUSE, yet the problem is that the phone is still OEM locked, so a factory reset will be required after unlocking the bootloader.

tomasz86 · May 8, 2022, 8:24am

I’d just like to add one more observation. It appears that it takes much, much longer to scan folders that have a lot of files in them instead of many folders with a few files, even though the total number of files scanned is similar.

For example, I’ve got a folder consisting of ~1500 dirs and ~3000 files, i.e. basically two files in each dir. Syncthing scans through it in just about 15-20s. However, the problematic folder mentioned here previously has only 5 dirs and ~12000 files, where ~7000 files are located in a single dir, and ~4000 in yet another dir. At this very moment the scan’s been running for 30+ minutes and still hasn’t finished yet.

Also, it seems that the very first scan after restarting Syncthing is very slow, and then the subsequent ones go much faster. I’d guess this may be due to the files having been cached in memory.

tomasz86 · May 18, 2022, 5:43am

Just a quick update to the situation. I didn’t mention this, but the folder in question was my Joplin notes filesystem sync directory. It contained ~11,000 files split into two folders - one with 7,000 files and one with 4,000 files. As already explained before, it sometimes took as much as 30+ minutes to scan through it.

I’ve since then archived quite a large number of old notes that I no longer need to view on a regular basis (and especially not on the phone), and also reduced the period of keeping note backups from “90” to just “7” days. All this has effectively dramatically decreased the overall number of notes to just ~2,100.

As a result, it now takes Syncthing just ~5s to scan the folder on a fresh run. I still find this quite strange, as proportionally it should take 5-6 times less to scan the folder, and not 360 times less . I’m not complaining, of course, but something seems incredible inefficient in how Android 11 handles folders with a large number of files in them. It’s not the file size either, as in this case, the size has actually increased due to adding a few large attachments to the notes.

All in all, the problem has been worked around on my side for now, but I’d like to let this topic serve as a warning for others who experience similar performance issues in Android 11. I also keep mentioning Android 11 only here, as the I/O performance issues are supposed to have been fixed in Android 12, but I can’t really verify this claim myself.

slimhk45 · June 24, 2024, 12:43am

So the limitation is about the number of files in any given subfolder, or is it about the total number of files in the sync folder?

In the first case, the workaround is to avoid having any subfolder with too many files, while in the second case the workaround consists in setting up multiple sync folders in SyncThing to spread the files across them.

acolomb · June 24, 2024, 4:04am

The former, number of files within any single folder that is part of a Syncthing shared folder.

slimhk45 · September 23, 2024, 6:54am

I think it would be a good idea to add this information to the official Syncthing documentation (i.e. the user must not exceed 1000 files per subfolder on mobile devices).

This information is critical for not loosing potential users of the app.

slimhk45 · October 6, 2024, 2:33pm

Installing SyncThing in Termux doesn’t circumvent this Android limitation?

tomasz86 · October 6, 2024, 4:20pm

I doubt it. I haven’t tested in Termux specifically, but I did either by running a self-compiled Syncthing binary directly from ADB shell or from the command line via an automation app. There was no difference either way.