I’m working on a little experimental iOS app based on Syncthing, with the ideal end goal of allowing on-demand access to files and possibly some sort of Dropbox-like on-demand syncing, integrated with the iOS File Provider mechanism. No promises, just a hobby project for now.
After lots of thinking and one day of actual coding, I have managed to embed the Syncthing code as a Go framework in a bare bones Swift iOS app, and built a very crude UI for adding devices and folders. The current version happily syncs files! (see below). It can also access the global directory tree (by calling model.GlobalDirectoryTree, i.e. not through the file system!) and show a UI using that. I know the code was never designed to be interfaced in this way, but it is working better than expected!
My current plan for getting the on-demand part to work is to somehow prevent the ‘puller’ from actually synchronizing all files, while still fully allowing the app to synchronize the index. Then, when a file is requested, I pull that specific file (or even subset of blocks) from the most suitable peer.
How would I best proceed with implementing this? There does not seem to be a way to add my own folder type in a neat way (‘folderFactories’ is not really extensible). There also doesn’t seem to be a way to programmatically pause file pulling. One idea I had would be to add a BlockPullOrder that simply stalls the puller forever.
Any advice would be appreciated. I would also be willing to contribute the necessary patches back to the Syncthing code to allow programmatic control, I just need to know how to go about implementing it
Cool to see swift bindings can get you that far (and quite quickly). I thought there were some major limitations to what gomobile supports (e.g. types), so wouldn’t have expected our model package to fit that.
Few quick thoughts:
One warning: Syncthing is not a library but an end-user application. Breaking changes in packages can happen at any time. The only somewhat “library-ish” package is lib/syncthing (I know of 1 user ).
Assuming you want to achieve something like on-demand sync with minimal effort (minimal go changes or additional swift code): You basically don’t need most of the existing folder logic. That’s what scans the local filesystem and pulls data from remotes. The index handling the model does you could still keep as is. So maybe just do that: Write your own folder with a factory, with interfaces to update/add local files or pull in remote files from swift.
Also be aware doing the minimal thing will make your device look out-of-sync to all other devices (you don’t have/announce them).f
Yeah, due to gomobile limitations I am actually not exposing syncthing types wholesale to Swift, but rather only a thin wrapper package that exposes only what’s needed for the UI. All logic involving Syncthing is on the Go side and Swift only deals with the iOS-specific bits and UI.
Fully aware! The little Go glue package in between does provide a clean separation point, so tracking minor breaking changes should not be too difficult.
So there appear to be two routes.
Custom folder implementation
This seems like a reasonable idea to try. The first difficulty I’m facing is telling syncthing.App/model.Model to actually use my special folder implementation. (I could of course instantiate it myself, but ideally I’d like to keep using all the existing infrastructure of syncthing.App, as it makes the app stay close to running a full node, and deals with a lot more stuff than just the syncing, i.e. reading config, NAT stuff, etc., which I’d otherwise have to redo myself).
Then after that I would have to duplicate much of folder_receive I guess.
Yes, that’s fine I guess (my client will not really be able to provide blocks to other devices either - unless a specific file is synced).
Add support to model.Model
This appears to be a simpler route.
Ideally the following ‘API’ would exist:
config.FolderConfiguration would gain a setting ‘PullOnDemand’ that, when set, simply prevents the puller from pulling anything by itself (alternatively this could be a value for ‘block pull order: none’).
Something along the lines of func (*m Model).GetFileRange(folder string, peer protocol.DeviceID, file string, version Vector, start int64, end int64) that would allow fetching part of a (specific version of a file). The necessary parameters can be obtained through model.Availability and model.CurrentGlobalFile, correct? (setting end=-1 or similar would fetch the full file. Fetching ranges is nice for streaming media and is also supported by iOS File Provider I think).
In the future it would be nice to be able to do model.PutFile(folder string, file string, contents byte[]) to also write a change to at least one client (seems a bit too complicated for now)
One more thing, which you may or may not already be aware of, is that running in the simulator allows some levels of filesystem access that you won’t get on a real device. It wasn’t clear to me from the initial post how much access you expected to be able to get for Syncthing.
Hm, what limitations are you referring to exactly? The app appears to run fine on a real device. Do note I am using the ios patches from MobiusSync’s fork - perhaps these already include some necessary adjustments.
Yeah so iiuc there are two things you can do on iOS:
Put synchronized files in the app’s own ‘Documents’ folder. If you set a flag in the app’s metadata (Info.plist) this folder becomes viewable through the iOS Files app, and other apps can open and edit these files as well. This is what I have working now (and it happily syncs two ways, but only when you open my app). This is also how MobiusSync works, only it applies some tricks to be able to synchronize in the background.
Offer a ‘File provider extension’ from the app. This is a kind of VFS driver that the system can use to list and request files, which the extension can fetch from e.g. a server. This is what I intend to use for on-demand files. (It also offers a ‘replicated’ mode, in which the system will ask the provider for all files and create ‘dataless’ files. Then, when the user actually wants to open a file, it will fetch the data from the file provider. It also integrates with the iOS UI showing e.g. sync status, so this is pretty nice).
In addition my app could of course use the syncthing index as well as the aforementioned hypothetical ‘GetFile’ method to load file contents from within the app and open up a ‘quick view controller’ to view or ‘share sheet’ to open a file in some other app (this is less optimal because it will not allow for writing changes to the file, does not provide streaming access).
Currently the app is already quite functional - it supports on-demand downloading of files (from within the app, the file provider part is complex) as well as ‘traditional’ full sync of a folder (when the app is opened). I will have to polish the UI a bit to support both modes, support encrypted peers, expose some more configuration options, sync in the background, add a QR code scanner to add peers, show discovered peers, etc.
I also have to think about what to do with my (minimal) patches to Syncthing (they are currently based on the MobiusSync ios branch, so either my patches should be upstreamed there, or upstreamed to syncthing/syncthing, or I keep maintaining my own patch set).
After this I will be able to release a testing version (will have to look into getting an Apple developer account). File provider functionality (on-demand download and sync back of individual files from other apps) will be the next milestone (technically more complicated).
@calmh @ AudriusButkevicius I have now implemented selective and on-demand file syncing using .stignore. Basically upon adding a folder the app writes an .stignore file that looks like this:
*
This appears to have the same effect as stalling the puller: there is nothing to be synced. (I assume the client may also show up on other nodes as ‘Synchronized’ but haven’t checked) I can subsequently still download ignored files using the RequestGlobal method.
For selective syncing, the app writes exclusion rules in the .stignore file like this:
This would cause specific files to be synced while the rest is still ignored. The app would need to do some housekeeping (remove rules that do not match any global files, perhaps offer a way to delete and deselect a file in one action) and show appropriate UI (download file, ‘keep file synced’).
An additional advantage is that the only change needed to the syncthing code for this is the exposure of ‘RequestGlobal’ for on-demand downloading. (The alternative for this would be to call BringToFront after unignoring a file and then waiting for the file to sync, but this would preclude e.g. on-demand streaming of media.). It would also be easier to offer both send-receive and receive-only.
Somewhat ignorant of the details but wouldn’t this approach still “ignore” the file and prevent the file from being transferred back to the cluster if it is modified? Also if the file is changed remotely it won’t be redownloaded.
I think the concept would be if I modify the file and save it, the saved file should be pushed back (assuming it’s not a receive only folder). And if the file is modified remotely, it should be redownloaded.
The ignore file approach should behave better in this regard I think.
Yes, absolutely. This ‘on demand’ download would be suitable for e.g. streaming a large media file (read only, not even storing the blocks) but not for writing. You would re-download each time, to ensure you see remote changes (this is why we use ‘RequestGlobal’ which should download the most recent ‘global’ version).
For any writing, you would need the ignore approach, i.e. ‘select’ the file for syncing first, then let it sync and download the file, and then possibly sync back any changes like regular send-receive folders (unless/until the file is ‘deselected’ for syncing; changes after that would stay local. The app should probably not allow this to happen by deleting a file locally upon deselection).
As this discussion shifts into a bigger discussion about selective sync just want to make sure you’re aware of this lengthy discussion which I think is definitely worth a full read-through.
There’s probably a lot of irrelevant posts but definitely some relevant ones that touch on some of the important issues around selective sync.
I quickly read through the topic as well as Next Gen Ignores: Requirements · Issue #2491 · syncthing/syncthing · GitHub and some of the last comments appear to resemble what I proposed above (i.e. write to .stignore from some tree UI). There appears to be a lot of discussion about also syncing ignore files and the combination of selective sync with generic ‘ignore’ functionality (for e.g. .DS_Store files).
Some issues/concerns to deal with:
One concern could be that .stignore changes are not directly applied (as one commenter notes, what happens when a file that is being pulled is then added to .stignore?). Possibly my app would have to check for this and somehow cancel the pull.
Another issue is the complexity of the ignore patterns. I need to carefully escape paths before adding to .stignore I guess.
What about files/folders that are created anew locally? The app should be able to detect these (at least in the regular syncthing the ‘recent changes’ shows which device added a file) and automatically add these to the local selection.
What about files that are renamed remotely or locally? Ideally the ‘selected’ status would transfer to the new path name. This is a very difficult one to fix I think. Remote rename would make the file disappear, local rename would remove the old one remotely and not sync the new file. Local rename can perhaps be prevented in the UI, or can be supported with some editing of the .stignore file. The issue with remote rename can be simply to accept the behavior, depending on the use case.
Another could be the compatibility between different apps and their way of generating patterns in .stignore. This I see as an implementation detail for my app as .stignore is not supposed to be read by other implementations in my case (it is not synced).
I don’t know what the delta is for current head state and mobius, but making some function public is not rocket science.
I think getting the on-demand bit polished and shipped is probably already valuable.
I can also imagine doing a “bring your own model”, where you do your own state management, request handling etc, at which point doing things via ignores doesn’t need to exist, or you could abuse the “filesystem” abstraction.
Also I have implemented streaming of (ignored!) media files (through RequestGlobal). This is a bit involved, I set up a local HTTP server and then let the iOS media player play a URL from that. The media player will do range requests to the server, and the server will do range requests through RequestGlobal
