Request - File Locking with on demand sync

Feature
1

Nice project you have here. The power and encryption to replace VPN in a more friendly way, is outstanding. I have been playing with many options over the past month, looking for better ways to perform our work.

Syncthing does have shortcomings for my personal use though. Mainly collaboration. Seafile has gone a long way with its seadrive option, but you have to jump through hoops to have a properly encrypted remote system up and running. Syncthing security is pretty much fully configured and functional out of the bag. So the point to this post, is to give syncthing the power to replace seadrive.

As I work with others using shared files, file collisions is not an options as the collisions would be endless. Administration time would be better spent in other areas than fixing collisions. From my point of view it is far easier to have single user file locking for a group shared folder.

  1. One source folder (server versioning) shares its folders to all others (light clients).
  2. Only file/folder structure is sharedā€¦files are shared on demand if in unlocked status.
  3. Downloading files create lock state and prevents write access (ie downloaded).
  4. Saved modified files will change the file version status, upload, release from client and unlock for collaborator use.
  5. Only changed blocks will be uploaded.
  6. The source folder reserves the right to break locks to prevent client domination and issue an ownership rejection notification + reversion of any modification.

So what we have is many-to-one encrypted SMB shares with IMAP behaviour. To keep it peer-peer, any client can download(lock)/upload(unlock) any file and possess the lock token. To enable devices with limited storage to function, only essential needed files are ever downloaded (over fast asynchronous), and only modified blocks are uploaded (over slow asynchronous). The server is the folder owner. So the server and client role are bi-directional, but can also be pure client-server if that model is preferred.

I noted a few others have requested similar features in some ways. So perhaps there could be 4 directory modesā€¦

  1. File Lock + Metadata sync + File sync (manual locking - full/part sync)
  2. File Lock + Metadata sync (auto locking - load on demand)
  3. Lock-less + Metadata sync (conflict files - load on demand)
  4. Lock-less + Metadata sync + File sync (conflict files - full/part sync)

I know this is asking a lot. But if you are going to do something, you may as well do it right and do it wellā€¦like the rest of the program. This should fulfil most if not all usage requirements ? Thus file locks becomes write locks and you can make them user/group dependant too. If you want to extend the permissions system.

Anyway, thanks for reading.

2

What you are describing is not syncthing.

Syncthing is decentralised (in your case itā€™s centralized to some central server), without authority (in your case there is an authority that can revoke locks), cross platform (file locking pretty much does not exist on unixes, so it would have to be windows only).

1 Like
3

What if the metadata was masked with an empty file for the syncthing folder when locked? Could that be done on the linux side?

And if I am not mistaken one side of the link is authoritative anyway ? The fact client server is at the folder level rather than the machine level seems moot to me.

4

Locking as a concept simply does not exist on most filesystems on linux, so I am not sure what you are referring to when you say metadata. Nobody is authoritative in syncthing, everyone is of equal power.

Personally to me it feels youā€™ve shoehorned syncthing in some scenario where it doesnā€™t necessarily fit, and you are now looking for features that would not make sense to 99% of the user base.

Perhaps you should reconsider other tools on the market that might better suit your usecase. From what you described, normal version control system such as git or teamfoundation/perforce (which supports locking) might be a better fit. Or sharepoint, which I suppose has something simillar too.

1 Like
5

The way seafile works it uses metadata for the filename and directory. The files themselves are a seperate entity. So what you exchange is the details (metadata)ā€¦but not the file data. That data is carved up into blocks for on site encryption and fast easy block transfers.

So in an IMAP view metadata = header, data = body. You dont download the body on IMAP unless you really have to.

6

It sounds to me like it would be a better solution to use something like a central Subversion server and have clients check out files from that? It supports all your locking and partial checkout needs. :slight_smile:

7

Thanks Jakob. Iā€™ll dig deeper, but there really isnt anything I have found yet. Do they offer out of the box point to point tunnelling encryption with block level transfer for family and small business ? Dropbox, Onedrive, Googledrive, Sync, Mycloud, Nextcloud, Pcloud, Tonidoid, Alfresco, Seafile, various VPN and cloud collaboration options. All of them have issues in one area or another.

Network file ā€˜managementā€™ software vs Network file ā€˜cloningā€™ software

  1. Point to Point encrypted/authenticated folder/project tunnel
  2. Encrypted Block level file transfer and block storage ā€˜on demandā€™ for small device access.
  3. Encrypted Block level file transfer and block storage for clone device access.
  4. Block size file metadata (index-fragment) network cloning.
  5. Abstracted unlock/key association layer between file blocks and metadata blocks to build unencrypted file-on-demand.
  6. The fully contructed file only ever exists in RAM (and not other storage media) for application handover. Building/encrypting/fragmenting a local file storage is not the bottleneckā€¦block level network transport with on-demand file to block conversion/encryption is. So rather than fragment/convert/encrypt/transport large files as part of the network process, all you do is transport pre-prepared blocks. The fragment/convert/encrypt donkeywork is done at the local storage stage. Hell, shove the fragments in a clustered database. Maybe even create two portsā€¦a no-overhead unencrypted one for the large encrypted file blocks, an encrypted one for small metadata blocks.

I am looking for ā€˜managementā€™ software. When you include features like ignore-file, selective-sync, conflict files, master-file/folder, you are moving from a ā€˜cloningā€™ software to a ā€˜managementā€™ software. You are trying to control what/where/when/if files will be delivered and multi-user versions of one ā€˜masterā€™ file to boot.

There are those who want private P2P cloning with total content control on their mutiple personal devices. There are those who want public P2P cloning to mirror source data at multiple public devices. There are those who want selective P2P cloning to mirror source data at multiple friend devices (with and without full owner control). All of which are at the new file or read only file level for the most part.

Then there are those who want to modify files rather than create files and read. This is the cloud storage side where you might not want to download/sync a 100GB repo on your android but you might want to edit a single file on a train. You might even want your data to just sit on the cloud/server/cluster instead of ā€˜anyā€™ other devices. That might also be a family cloud where multiple members could edit the same file at the same time. The private people that want sole control over file editing and full ownership. The public people who want group editing which means corroboration and conflict control. ā€¦and anywhere in between.

So I wonder why there are 100s of different network software packages (P2P, cloud). IMHO its simply because the existing ones dont meet the demand. They all only offer a half-arsed solution to the problem. Some are badly managed bloatware that suck up all ram, cpu and diskspace and could never be used on a virtual server. Some have sloppy integrity leading to lost files. Some dont offer proper security and authentication or that the enduser can actually understand and implement. Most resort to full file conversion, encryption and transport at the network demand stage. Many are crippled, freemium, clickbait, ransomware where only the exhorbitant full fat version is guaranteed to work and function properly. Many are so deliberately complex and convoluted to make support contracts a fundamental requirement. There is not one simple, afforable, comprehensive, legit network management package for family, small business, non-profit and small charity use. Sharepoint? No comment.

We need affordable, simple, software that enables cross platform, secure file sharing with comprehensive collaboration control. Clearly syncthing is not it and never intends to be. Its still a good package though, for what it does do. Good luck for the future :slight_smile:

8

What a shame, it would be such a great feature. From a user perspective, it seems like a simple yet critical request, while for developers it appears to be a no-go.

with current lockdowns, people are working from home more than ever and the mainstream IT provision is Citrix or VPN to a central server, both terrible! Looks like this would be an incredible opportunity for syncthing to monetize, creating an enterprise version.