I have been using Syncthing on all of my machines and mobile devices for a long time. I honestly couldn’t image live without it these days, so thanks for that
I work in video post production and I’m trying to ascertain the risks in using Syncthing to sync a disk database between a number of machines. The purpose of this is for me to potentially work on the same project as my colleague, sat next to me on another machine. I’m not 100% sure whether the dangers they are warning me of are applicable in this scenario. I am also aware they would rather I use one of their paid solutions in the form of a postgres database.
This is the response I received from one of their staff:
"It’s an issue with concurrency.
If a Disk database is in use, but is being locked or written to by another application, there are no guarantees for sanity of the underlying data. Even if that other application is another DaVinci Resolve in another (or same) machine. This applies for DropBox, mapped drives, local rsync services, SMB shares and pretty much anything else you can imagine doing this.
The PostgreSQL server manages this concurrency well - it reports to the user if a project is already in use (and who is using it), it reports if a project is capable of collaboration and which bits are being used by which collaborators.
Disk databases are not designed for concurrency. If you wanna explicitly close projects in a database, ensure that whatever sync tool you have has finished syncing and access them from a different machine, we don’t expect that to cause issues… bar the risk you assume of the sync tool finishing its job correctly."
So I guess what I want to understand is, how does Syncthing deal with the synchronisation of files that are in use? Are their concerns valid, or is Syncthing a good fit for what I am trying to do?
They are right in essence, but there seems to be two points here:
If the disk database itself is not made for multiple simultanous access, then it doesn’t matter what Syncthing or any other tool does - it seems they are saying that’s the case. I have no clue whether this is true, but it doesn’t matter because:
No matter what kind of file, if you do edit it at the same time on two devices, Syncthing will create conflicts once those changes get scanned and synced. I mean what else would you expect: One common file became two different files with the same name on two devices - anything except creating a conflict copy would result in data loss.
The database supposedly isn’t designed for simultaneous access, however the way Syncthing works means that isn’t a problem as each machine has it’s own copy of the database that it is accessing. These then get compared and overwritten depending on which version is the newer. Is that about right?
Not fully: The first part about multiple access not being a problem with Syncthing is correct. The second part is theoretically possible, but I’d bet practically it isn’t. Syncthing will create a copy of the database file on conflict and yes, if these databases are mergeable, you can compare and merge them, but you’d have to do that manually or script something.
So wait, maybe I didn’t explain myself properly or perhaps I’ve misunderstood? Highly likely
I thought the way Syncthing works, it would just look for the newest version of each file and replace the oldest ones.
I’m a bit confused about how merging might come into play here.
The whole premise of my points is based on your initial statement (emphasis mine):
So potentially you are working on the same database at the same time. If you only ever have a file open on one device and enough time between opening on different devices to syn, there’s indeed no problem, Syncthing will just always sync the newest file on both machines. However as soon as you have it open on both devices, one file might be newer than the other, but both are newer than the last common version. That’s were conflicts and merging comes in.
Yes the purpose is to be able to access the database from two machines at once. (also for clarity, the database isn’t a single file, it’s a full structure of folders and files.)
So I think all I want to try and understand are the possible dangers in what I want to do, as if they outweigh the benefits I will abandon this idea.
They have mentioned the risk of databases being potentially corrupted and un-openable (Their example referred to using Dropbox, but I feel this was potentially more of a versioning issue) but I am acutely aware of their desire to sell an alternative product for sharing databases, so it won’t be in their best interests to help me work out whether or not Syncthing is a valid option for what I want to do
It’s not a risk, it’s a certainty, and not dependent on which synchronization tool you use: By editing/modifying at the same time, you will get conflict files which your program will not know how to handle and unless there is a way to fix those manually and you want to do that (I’d say both are a “no”), it wont work.
