I agree with you @tomasz86 , I have a Pixel 8 whith Android 15 (GrapheneOS), and the slow speed still persists.
I think many people here don’t get that the performance issue is not about the total number of files synced, but about how those files are spread.
This makes any explanation about Android API vs native API non-sensical, why it would be faster to sync 10,000 files spread in 10,000 subfolders, rather than 10,000 files in a single folder?
That’s the question that must be answered.
The most common answer to that specific question is that syncing a file in a folder requires some operations on the file (like reading and writing to it) and some on the folder (e.g., creating the temp file, renaming it, flushing the directory listing to disk). The operations on the file will take the same amount of time always, approximately, regardless of how many “neighbours” it has in the folder. The operations on the folder will take time dependent on how many files are already in it.
For 10 000 files in 10 000 folders this can work out as 10 000 file operations + 10 000 folder operations, each taking one unit of time, so 20 000 units of time.
If the files are all in one directory we have 10 000 file operations and 10 000 folder operations, but now the folder operations take (lets say) 1000 units of time each, so we get 10 000 + 10 000 * 1000 ~= 10 million time units.
Operations in a large directory/folder take longer, and we need to do a lot of them.
Ok, that having all those files in a single folder makes it particularly worse is something I missed. Thanks for the detailed explanation.
In my case I have 10000+ files (~ 50 GiB) but spread over 1000+ folders. This didn’t take hours. In fact it took at most 15 minutes to sync on Android 14. And it re-scans in a matter of seconds (when there are no actual changes).
I still think that using any Java-APIs will help much, though.
So SyncThing is useless for syncing files between desktops and smartphones. One should use a home server (for web apps) or an external SSD (for native apps) to mediate between the two.
I will continue to use Syncthing, but only for syncing between my home server and my desktop (both Linux). And I will use rsync to mirror my desktop files to an external SSD I can read on Android.
Just an update because I found a recent comment made by GrapheneOS developers on the issue tracker of SyncThing for Android:
It’s almost certainly primarily caused by how the app is choosing to do it rather than an inherent issue with Android. Accessing files in shared storage is certainly slower than native filesystem access in the app’s storage. That’s why it would be best for an app like this to sync to their own storage and implement a content provider for accessing it instead of putting it directly into the user’s shared home directory. It’s likely the home directory performance will improve though.
The app accessing their own files is native f2fs filesystem access which has the usual performance of f2fs with SELinux. SELinux adds a bit of overhead, but not an enormous amount. There are multiple ways to access shared storage, some performing better than other methods, but it’s not direct filesystem access. An app using Storage Access Framework to access a huge number of files is certainly going to be a lot slower, while lower level access to shared storage isn’t high overhead.
They’re blaming Android but it’s how they designed their app. They chose to do it this way. They could have synced the data to their app storage by default and provided a storage provider for browsing it in the file manager. If they did that, it would appear as an entry in the file manager’s list of storage providers where the top level home directory appears.
Android focused on trying to get apps to stop misusing the home directory and demanding bulk storage accessed.
The workarounds to provide backwards compatibility with apps depending on bulk storage access led to access to the home directory being slower than before. This app appears to have simply reworked their code to continue functioning with this instead of considering redesigning it to better fit into how Android intends for storage to be used. They could have native filesystem performance if they used the native filesystem access to their own app sandbox directory they have by default.
They just need to start saving files to their own directory and expose their own directory as a storage provider so users can access it from the system file manager / system file picker.
This app is primarily meant to sync a specific directory across computers, right? They should be using a directory in their app sandbox where they have native filesystem access and exposing that to users via a storage providers.
They’re just not doing this in a good way for Android.
It is true that accessing the user’s home folder is slower than native file access, but it shouldn’t be as slow as their app, so something seems wrong with their implementation. It’s also their choice to take that approach instead of the high performance approach. They ended up doing that since they were always doing it that way and the performance gap has always been there but used to be smaller. Security/privacy features slowed down home directory, but not their sandboxed directory.
They’re blaming Android but it’s their choice to put files in a location that has always been slower to access and the gap in performance got much wider. Android wanted developers to stop using the home directory instead of using their sandboxed storage and exposing files to users and other apps via content/storage providers. The performance hit to home directory was a consequence of adding abstraction layers to provide fine grained access control with backwards compatibility.
The reason we were able to implement our Storage Scopes feature in GrapheneOS is because Android abstracted home directory access in a way that enabled us to build that feature on top of what they provide. It was a good change. The problem is that app developers don’t know the platform well and those that do don’t want to rewrite significant parts of their app to work the way it always should have been done. They’re paying for doing things in an invasive way to begin with.
Look, again, Syncthing on Android isn’t really an Android app per se, never mind a properly designed one. It’s a cool hack for running a Linux daemon on the phone, which kinda sorta worked a decade ago when it was made. I was surprised it worked as well as it did then, and I’m entirely unsurprised it works as badly as it does now.
Wrong. That’s the main culprit I see here in other people not understanding the use-cases that users actually want. It’s not Syncthing accessing its own data. Syncthing is a tool that is meant to synchronize data which “belongs” to other apps, which are the ones handling it. Of course one could restrict the app to its own sandbox storage, but in reality that’s not what users want it to do.
They want to sync their photos (from the Camera app), their music collection (which doesn’t “belong” to any particular media player app) and arbitrary files of other apps which don’t store their data natively on some cloud server (often intentionally, because they like to not depend on the cloud). At least for the latter category, it seems unlikely for every other app to implement some workarounds to put their data in Syncthing’s storage provider instead of their own sandbox.
To reason about the app developer’s choices, Syncthing should better be compared to something like Adobe Reader, which is used to display any PDF anywhere on the system, not only inside its own sandbox. That’s why Syncthing needs (permanent, background) access to the other apps’ data locations. A sensible choice at the time of development was for apps to put data in a shared namespace, which provides this kind of interoperability. Sandboxing everything funnels data exchange through specific channels, which are not designed to be as performant as shared storage.
Of course it would be possible to redesign the app around the Storage Access Framework and implement a storage provider. It will limit the use cases to having all synced data aggregated within one sandboxed storage location for Syncthing. And it may very well lead to the functionality having to be reimplemented using some Android-specific SDK instead of re-using the core Syncthing daemon in the background. And there lies the rub, Someone™ needs to spend time on that major development task. Exactly that motivated Someone was not there for the last couple of years, so no adjustments to the fundamental shift of habits on the Android platform were pursued. It’s always simple and easy to blame people for doing something wrong, while the truth is there was simply nobody trying to do it right.
My use-case is to mirror my desktop files to my mobile, so that I always have an up to date, read-only copy of my file (especially my Obsidian vaults for reading my notes on the go, and my gallery of photos that I manage with PhotoPrism on desktop).
People edit local files on desktop devices and read them on mobile devices (which are both phones and glorified multimedia players like iPod).
This use-case is also very common if not more common.
That is to say, I understand that an open-source project might not be able to cover all use-cases due to a lack of human resources
Except we can cover that use-case? You choose yourself where you put your files, you absolutely can put them into the syncthing internal storage dir. For a while with android access restrictions, that was even a recommended (the only?) possible workaround.
A bit more nuanced instead of absolute takes and less blaming would help the discussion quality here. Examples being “useless”, quoting accusatory grapheneos stuff (I use grapheneos, it’s technically great, but their attitude is… something), … Well it would help anywhere really.
If I’m not mistaken, this is essentially what Möbius Sync has been doing on iOS, where basically no access to other apps’ data (or the filesystem itself) is available to begin with. This limits what Syncthing can do extremely heavily though.
As long as the other apps which work on the files allow specifying a custom storage location, and that location allows using the hypothetical Syncthing app’s storage provider. That would of course already increase the number of possible uses that are relevant and requested by users.
Really, setting the sync folder in SyncThing’s private storage will solve the issue then?
I need to try. If so it should be added to the official documentation for my use-case which is quite common.
I apologize if it seemed rude, the message was not from me, and it is indeed “useless” to use SyncThing for Obsidian users if they can’t have a read-only copy of their vaults on Android without waiting one week for each scan.
According to that message there’s no such overlay in place that creates overhead in the private dir. I do not know if that’s true.
Also probably doesn’t cover the other half of what you care about, as acolomb and also the lineageos guys pointed out: You still need to access/expose the data from syncthing’s private storage area. We do not implement that provider mentioned, and many apps might not be able to work with that anyway. And conversely likely apps can’t access syncthing’s private area, either because they don’t allow setting that as a location or possibly because android restricts that anyway. So it might be possible to sync things quickly to android that way, but it’s not clear if that’s going to be useful.
Indeed, I just tried, and Obsidian doesn’t have access to the SyncThing private storage in /Android/data/com.github.catfriend1.syncthingandroid. This private storage needs to be exposed by API.
They just need to start saving files to their own directory and expose their own directory as a storage provider so users can access it from the system file manager / system file picker.
I suppose that’s something I can add to my TODO list for Syncthing Tray on Android. Of course it must still be possible to select any directory because as others mentioned before - Syncthing is not just about syncing its own data.