How to deal with abysmal I/O performance on Android 10+?

Development
1

This is not exactly a support request, as I know that the reason why things work as they do is Google and their enforced scoped storage, but I’d like to ask what kind of solutions/hacks could be applied in order to mitigate the problem.

Just to give some context, I’ve got a folder with 11,000 files and it takes 15+ minutes to scan it (without hashing, just rescanning) on a mid-range device from 2019 (Samsung XCover 4S, Android 11). For comparison, this is slower than my old single-core Nexus S from 2010 running Android 4, which was able to go through the same number of files in 5-10 minutes. Another, older yet much faster phone (Samsung Galaxy S5 LTE-A, Android 6), requires just a few seconds to scan the very same folder.

As I understand, the main culprit here is simply the very slow access time caused by all file access forced to go through a Java-based layer, which wasn’t the case in Android 9 and older that allowed direct file access (see https://reddit.com/r/androiddev/comments/kpn68k/android_11_very_slow_file_access_performance).

Is there anything that can be tweaked inside Syncthing itself that could at least mitigate the issue? At the moment, I’ve tried to change the default number of per-folder hashers from 1 to 0 to allow for using all CPU cores, but it hasn’t really made any noticeable difference.

Now, Android provides a way to disable the so called “isolated storage” and enable “legacy storage” though the following commands.

adb shell cmd appops set <app-package-name> android:no_isolated_storage allow
adb shell cmd appops set <app-package-name> android:legacy_storage allow

I’ve used these for a few apps that haven’t implemented proper scoped storage support yet, e.g. Joplin to allow it to use file system sync, and Bromite to allow it to access local HTML files through the file:// URI scheme, as otherwise those applications are unable to access local file paths in Android 10+.

My question is, would using these commands make Syncthing behave as before v1.19, allowing it to access files directly, or would it have no effect in terms of performance? I don’t need external SD card support, so I don’t care if I lose it in the process.

I’d be very thankful for answers to the questions above or any other tips that could help improve the awful performance.

At the moment, any situation, where Syncthing needs to be restarted is a nightmare, as I then need to wait at least 15-20 minutes for all folders to finish scanning, which delays pulling new files from remote devices. Because of this, I’ve now allowed Syncthing to run in all conditions, including battery power, mobile data, and aeroplane mode, as this seems to be the only way to just keep it running all the time without rescanning folders, relying only on the file watcher to detect changes instead (with the full rescan interval set to 1 day or more). This obviously has negative impact on the battery, but I don’t really see any other choice here :confused:.

1 Like
2

Can you run these commands without root?

If you do have root, there’s an easy cop-out. I wouldn’t usually recommend it, but this is android, anything goes.

As to whether it will work: I don’t know, and I am pretty sure finding an answer in docs will be harder than just trying it out. I don’t see why it wouldn’t work though, nothing changed in Syncthing, it’s entirely a config in the app (target api) and change in android. So if those command overrides that config, I’d expect it to work as before. Then again it might also be yet another android-ugliness-layer that wraps scoped storage in something that makes it behave like legacy storage :stuck_out_tongue:

Sorry I can’t be of any more help. Would have been really nice if google took the opportunity of introducing a heavily restricted “all file access” permission to let apps with that permission actually access the filesystem layer.

1 Like
3

No problem, I’m thankful for the answer regardless :slightly_smiling_face:.

The commands do work without root. I’ve tried running them at Syncthing, but I’m not really sure they actually do anything to apps compiled for SDK 30 that support scoped storage already. They seem to be only recommended when using apps compiled for the SDK that don’t support scoped storage yet and thus can’t access any paths outside of their own. The reason why I’m saying this is that I haven’t observed any noticeable performance difference after running them. I don’t think they “override” the config as such, but rather just allow to use the old way of accessing the filesystem if the app is unable to use the new. I’m not an expert though, so I could be wrong here.

I normally like to root all my Android devices, but in this case I’d prefer not to simply because the device is still supported, and it appears that installing updates on rooted Samsung devices requires going through major hoops, as they don’t use the A/B partition layout, so you’ve always got to un-root and re-root when installing each monthly update, which is a no-go for me.

Edit:

I think I’ve found out what problem I’ve encountered specifically. It’s not caused by the scoped storage per se (although related), but it rather seems to be about Android 11 and up using FUSE to handle storage instead of sdcardfs, which was used previously (see https://source.android.com/devices/storage/sdcardfs-deprecate). The problem also appears to be limited to Android 11 (which I’ve got right here), as the slow performance has apparently been dealt with in Android 12 (see https://source.android.com/devices/storage/fuse-passthrough).

If that’s the case, then this would mean that the issue’s not really fixable by tweaking the app, as I’ve also tried to use the older v1.18.3 release, and processing files was still painfully slow there. Disabling FUSE through setprop persist.sys.fflag.override.settings_fuse false seems to be the solution, but the command isn’t permitted to run on my device without root, so no luck with this one, at least for now.

I’ll need some time to think about what to do with this problem, but at the moment I’ve considered three solutions:

  1. Reduce the synced files number to the absolute minimum, so that Syncthing doesn’t need to scan through thousands of them.
  2. Wait for an Android 12 update for the device, but it’s still unclear if and when it’s supposed to get it.
  3. Root the thing and try to disable FUSE, yet the problem is that the phone is still OEM locked, so a factory reset will be required after unlocking the bootloader.
3 Likes
4

I’d just like to add one more observation. It appears that it takes much, much longer to scan folders that have a lot of files in them instead of many folders with a few files, even though the total number of files scanned is similar.

For example, I’ve got a folder consisting of ~1500 dirs and ~3000 files, i.e. basically two files in each dir. Syncthing scans through it in just about 15-20s. However, the problematic folder mentioned here previously has only 5 dirs and ~12000 files, where ~7000 files are located in a single dir, and ~4000 in yet another dir. At this very moment the scan’s been running for 30+ minutes and still hasn’t finished yet.

image
image1707×301 61.8 KB

Also, it seems that the very first scan after restarting Syncthing is very slow, and then the subsequent ones go much faster. I’d guess this may be due to the files having been cached in memory.

1 Like
5

Just a quick update to the situation. I didn’t mention this, but the folder in question was my Joplin notes filesystem sync directory. It contained ~11,000 files split into two folders - one with 7,000 files and one with 4,000 files. As already explained before, it sometimes took as much as 30+ minutes to scan through it.

I’ve since then archived quite a large number of old notes that I no longer need to view on a regular basis (and especially not on the phone), and also reduced the period of keeping note backups from “90” to just “7” days. All this has effectively dramatically decreased the overall number of notes to just ~2,100.

image
image837×667 59.3 KB

As a result, it now takes Syncthing just ~5s to scan the folder on a fresh run. I still find this quite strange, as proportionally it should take 5-6 times less to scan the folder, and not 360 times less :angry:. I’m not complaining, of course, but something seems incredible inefficient in how Android 11 handles folders with a large number of files in them. It’s not the file size either, as in this case, the size has actually increased due to adding a few large attachments to the notes.

All in all, the problem has been worked around on my side for now, but I’d like to let this topic serve as a warning for others who experience similar performance issues in Android 11. I also keep mentioning Android 11 only here, as the I/O performance issues are supposed to have been fixed in Android 12, but I can’t really verify this claim myself.

3 Likes
6

So the limitation is about the number of files in any given subfolder, or is it about the total number of files in the sync folder?

In the first case, the workaround is to avoid having any subfolder with too many files, while in the second case the workaround consists in setting up multiple sync folders in SyncThing to spread the files across them.

7

The former, number of files within any single folder that is part of a Syncthing shared folder.

2 Likes
8

I think it would be a good idea to add this information to the official Syncthing documentation (i.e. the user must not exceed 1000 files per subfolder on mobile devices).

This information is critical for not loosing potential users of the app.

2 Likes
9

Installing SyncThing in Termux doesn’t circumvent this Android limitation?

10

I doubt it. I haven’t tested in Termux specifically, but I did either by running a self-compiled Syncthing binary directly from ADB shell or from the command line via an automation app. There was no difference either way.

1 Like
11

Did you ever find any other optimizations (like more adb commands) to further improve your speeds?

12

Not really, I just try to avoid syncing folders with Android that contain large numbers of files in them :slightly_frowning_face:.

13

Have you tried Syncthing-Fork?

They’re not using the slow Scoped Storage API (from Storage Access Framework/SAF) since they managed to get approval for usage of the restricted but more performant Shared Storage API (via MediaStore with MANAGE_EXTERNAL_STORAGE permission. They’ve also applied requestLegacyExternalStorage attribute for backward compatibility with Android 10).

(I was going to link more sources but new users can’t do that).

14

That’s not specific to the fork app, same was the case on the official one. And it still uses android’s filesystem abstraction, even while interacting with the filesystem using “normal” syscalls. Those permissions/legacy storage/… flags are just about being allowed to access the filesystem that way and not having to request access to individual paths.

15

And I can also add that the exact same slowness is the case when running just a bare Syncthing binary from the command line, so the wrapper doesn’t really make any difference.

1 Like
16

In that case, did you ever try any other apps for cross platform data sync?

Seeing if some other project have the means of doing it faster could be a starting point in finding the culprit

17

The culprit is well known. Syncthing is not a proper mobile app, never was, never will be. It is essentially a Linux-style daemon, written in Go, with a GUI wrapper. It is not designed to work how a mobile app needs to work and it uses none of the Android-specific APIs to do things, resulting in high power usage and low performance on modern Android versions.

There is no quick fix for this.

3 Likes
18

I suppose there wouldn’t be anything to gain from using Android-specific APIs performance-wise. (Maybe the requirement for the sledge-hammer permission to read/write on the whole shared/external storage could be dropped by using some Android-specific APIs but that’s a different topic.)

That is because the real culprit is the underlying FUSE filesystem which was introduced with Android 11. It was a huge performance regression. I don’t think it matters how the filesystem is accessed. Using some over-engineered Java-based APIs might even make matters worse.

To mitigate the problem, FUSE passthough was implemented for new devices that came with Android 12. My device came with Android 12 and I can say that I/O performance is not abysmal. It is probably still worse than before the switch to FUSE, though. (I could actually compare this with my Android 10 phone which has similar CPU specs and supposedly doesn’t use FUSE yet.)

This problem is comparable to using NTFS under GNU/Linux. I used to use ntfs-3g there which is also a FUSE filesystem. The scan performance of Syncthing was quite bad. When ntfs3 was mature enough I switched to this on kernel-level implemented filesystem and the performance of the Syncthing scan was much faster.

2 Likes
19

Oh thanks for this info - I didn’t know about that fuse passthrough optimization!
Also explains why I don’t have these issues. I have lots of files synced to android, and while it’s slow, it never felt slower than “appropriate” for the platform. I always run custom ROMs, thus pretty up-to-date kernels. And I was often a bit confused about these reports and assumed they were because of rubbish vendor system tweaks or old devices, but had no good/really plausible explanation. Now I do :slight_smile:

As for what a native implementation could do: It’s not about using those APIs, it’s about how you interact with the filesystem. Syncthing is quite heavy on FS interactions, both because it is (intentionally) portable and because usually FS calls are cheap/fast. You could definitely be more efficient if you are only targeting android. Even with the same use-case/operation as syncthing - of course there’s likely also different approaches to UX that are more tailored to mobile. Mainly for the better UX, but I’d expect they’d also be lighter on the filesystem.

1 Like
20

Personally, I cannot really agree. I have one specific folder that contains two subfolders - one with 1000+ files in a single folder, and another one with 3000+ files. Syncing this folder for the very first time on Google Pixel 8 (which came with Android 14) takes 5+ hours.

On the other hand, the same folder syncs in ~5 minutes on an old Galaxy Tab running Android 7.

2 Likes