Virtual file-system

Hi their,

Was thinking of feature for one of my use case.

A virtual file-system which would only download files when requested. And an option for max redundancy across the nodes.

Then all nodes would have a local copy of the index with all the files. And only download the content when needed, with the redundancy limit of 3 when the 4th host download his copy it would be deleted of host with the oldest atime.

Also the vfs would allow to propagate updates instantly without need for inotify which doesn’t work when their are too many files in a folder.

Does it sound realistic?

2 Likes

It’s certainly not impossible. It needs the “download on demand” feature that is currently missing. With that in place, some sort of virtual file system could be layered on top, to present the illusion of files existing and tell syncthing to get them when needed.

I have been working on a related idea. Syncthing-fuse allows a node to view all files, but only stores a fixed amount of data locally.

In my mind, disk space is cheap, so finding one or more nodes that can hold the full index shouldn’t be a big deal. I wanted a client that could access all those files, but not necessarily store the data for all of them. So, I’m building the client! If you’re interested in helping, let me know!

Selective download and placeholder files discusses similar concepts quite extensively and includes my introduction of syncthing-fuse.

Syncthing-fuse will only support OSX and Linux. There has been some talk about Android supporting a similar feature in syncthing #193 Selective Sync and syncthing-android #81 Selective download.

3 Likes

Have you looked at the Interplanetary File System?

Yup, that’s interesting stuff. However, I’m not really convinced they serve the same purpose (i.e. file synchronization) because it’s not private and you cannot delete files (ever, even when the currently used encryption strategy is broken).

1 Like

I was thinking about this… I like the idea of syncthingfuse. I understood that it was using own implementation of syncthing, which was too much work to maintain. I have weird idea how to greatly simplify syncthing FUSE filesystem. i think it does not need to connect to syncthing network at all.

Imagine following:

  • You mount special FUSE filesystem and let syncthing sync to it.
  • That filesystem does not actualy store the data, only the filenames and shows them in directory listing
  • It also does create virtual .stignore file which matches everything in that shared directory
  • When user wants to open some file, the FUSE filesystem removes that file from .stignore file and calls syncthing API to resync, once it is synced, it can be opened and served to user.
  • When syncthing writes file to this filesystem it is stored in some kind of space limited cache.
  • When evicted from that cache, files are added to .stignore again.
  • When user writes file to this filesystem, it is stored in this cache, but cannot be evicted from this cache unless it is confirmed by syncthing API that it was successfully synced to N nodes.
  • From time to time we need to remove everything from .stignore in order to resync directory listings.

This is just rough idea and for sake of simplicity (and my lack of knowledge about sync process details) i have omited few caveats that will need to be solved for this approach to work correctly.

That way you have FUSE filesystem with relatively few features and you have virtualy no need to know anything about syncthing network. It does not need to sync anything from syncthing p2p network, since syncthing will do that. It stores just enough data to convince syncthing that filesystem is kinda there, while using virtual .stignore to block syncthing from complaining about files being actualy empty.

It would greatly help if there was way for .stignore to specify that you want to sync all files, but not their contents. So we can easily sync directory listings.