Syncthing-Fork slow due to Android filesystem abstraction layer

Could you please give a bit of clarification on what this command does?

The adb command is the Android Debug Bridge.

The reason Syncthing worked faster on your old phone and not on your current phone is that the Android operating system has changed.

Syncthing relies on the underlying OS for an IP address. Did you mean configuring Syncthing for a static address of a peer Device? If that’s what you meant, it’s under the Device / Edit / Addresses.

Enable Developer Options on Android. If enabled, then you can issue ADB commands through an USB cable with elevated rights. The command I sent gives Syncthing Fork the permission to use legacy access to the file system which hopefully helps with the access speed.

Same problem. 1 1Gb file takes 1-2 Seconds but 50+ 2-5Mb file takes almost 15-30 Mins.

Apparently, this is an issue for Android 11 or newer but my old setup was:

  • Non-fork app
  • Xiaomi 12T (Android 12-14) to Pixel 1(Android 10)
  • No issue regardless of file size and quantity (Average 1-15Min transfer for 1Gb+ Mixed files)

Now with the fork setup, pretty much same setup but with the issue above.

1 Like

Does the link means that put thousands of files in sdcard/Android/data/com.nutomic.syncthingandroid or other path will slow down the devices?

FUSE performance impact is limited to heavy users of files stored on external shared storage only. External private storage (which includes android/data and android/obb directories) is bypassed by FUSE, while internal storage (such as /data/data, where many apps store data to keep it encrypted and secure) isn’t FUSE mounted.

Scoped storage  |  Android Open Source Project

I will do some expirement with termux on this.

For me, it is still slow in /storage/ABCD-1234/Android/data/com.termux/files . On the other hand, /data/data/com.termux/files is very fast.

Accroding to google’s docs:

https://source.android.google.cn/docs/core/storage/fuse-passthrough

Only the devices which initially released at/after Android 12, can benefit from FUSE passthrough, which might accelerate the access to Android/data/* under sdcard.

Have a good luck on this, or just root your android device and access /mnt/media_rw/ABCD-1234.

I think the problem is that even if you do manage to get fast file access via the data folder, other applications in most cases won’t be able to access them, which limits the actual usefulness of the workaround greatly (e.g. most people need to sync fixed paths like DCIM/Camera, etc.).

Camera is the only large folder which is fixed. Everything else could be solved by the Syncthing app working as a document provider.

There are more :slight_smile:. People seem to often sync folders created in the root path on the internal storage by other apps (e.g. WhatsApp, etc.). Those are also fixed and cannot be moved.

In my case, the main problematic folders containing a large number of files are a) Joplin (a note taking application), which for syncing purposes stores thousands of tiny MD files in a single folder, and b) Radicale with its thousands of tiny ICS files (used with https://forum.syncthing.net/t/how-to-sync-contacts-and-calendars-between-desktop-and-android-without-root/21326).

Not sure if Joplin would be able to utilise the Syncthing app if it acted as a document provider. It wouldn’t help with Radicale for sure, as this one is used via Termux which needs direct access to the folder.

2 Likes

What do you mean by “the Camera is the only large folder which is fixed”?

Do you mean that it is not affected by the FUSE performance bottleneck of 1000+ per single folder, as written in the Android scoped storage documentation?

My use-case is to mirror files from Linux to Android. Could I mirror them to the Camera folder to avoid the bottleneck?

1 Like

I understood it as this: The location of the camera folder can not be changed so we have to live with the limitation and can not make use of app-private storage sharing it via a document provider.

I still don’t know how sharing a whole folder with other apps (not just a single document) is supposed to work via a document provider, though.

2 Likes

Isn’t that what the Storage Access Framework is supposed to provide?

https://developer.android.com/reference/android/content/Intent#ACTION_OPEN_DOCUMENT_TREE

Apps can fully manage documents within the returned directory.

Using the old 1.9 fixed all speed issues on my end. Night and day.

In theory yes. In practice I’m not sure about the usefulness of this. I created a custom document provider for Syncthing Tray on Android. With this entries for Syncthing (one for the private storage on the internal storage and one for the private storage on the SD card) show up in the Android file system dialog triggered using the “intent” you mentioned.

The problem is just that not many relevant apps seem to use such an “intent”. Do you know such apps? The file explorer I’m usually using (Cx File Explorer) doesn’t seem to use that. The same counts for the file browser provided by Samsung. The “Files” app from Google shows the document provider but clicking on it just shows the associated app which is not very helpful. The gallary and music apps from Samsung also don’t seem to allow adding a media directory this way. The VLC app allows entering custom paths for the media directory and doesn’t use the storage access framework. The Samsung camera app just allows selecting between internal storage and SD card but doesn’t offer to select any specifc path.

I was able to test with apps that use Intent  |  API reference  |  Android Developers to select a single document, though. This actually seems to work but it is probably not how I’d access most of the documents I’m syncing with Syncthing.

Maybe there’s also something missing in my document provider which prevents it from being usable in file browsers and other relevant apps.

Otherwise it looks to me like this whole storage access abstraction from Android is kind of a fail. It is there in theory but not actually used by most apps. Most apps simply rely on accessing the storage directly or they use the Android media library to know what songs/images/videos are there. I think the latter also doesn’t really allow storing this kind of data in app-private locations.

I did a little bit more research (before I was just testing with apps that came to mind) and was able to find an open source file browser that supports browsing document providers. I added a note on the Syncthing Tray README. After a brief look it actually seems to work but it probably needs more testing. Maybe this one is even generally better than Cx File Explorer (but I guess bike shedding about file explorers is out of scope here).

My Galaxy Note 10 Ultra 5G and S10 Plus both performed very well with 512GB of data from.what i recall.

Im using an S24 Ultra and havent touched Syncthing for about 6 years, ill give it test on my home 2.5Gbps sym network and let you jnow if “it just works” like it used to.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.