Syncing large disk images

User Stories
1

Hello,

is anybody here who has experience with using Syncthing to sync large disk images?

The scenario that I’d like to establish is:

  • create a sparse disk image of about 300 GB
  • sync this to several machines, not all of which are being used by myself.

One issue I could imagine is, that two people mount the disk image at the same time and that this could result in either data loss or however Syncthing handles sync conflicts. If Syncthing creates a copy of the conflicted file then this would obviously be a big issue with a 300GB file.

What is the best way to handle this requirement? Keeping in mind that there are people on the team who can not and will not want to deal with the technicalities behind file transfer. Currently the team zips and uploads updated files to a third party storage service and then pulls these files down from there and moves them into place by hand. I would still not sync the actual production folder of our files I would just like to sync the transfer folder.

I hope I explained what I’m trying to do.

2

It’ll sync just fine. But:

  • Every change means a rehash (on the sending side) of the full 300 GB which takes time, CPU and RAM even if it’s sparse.

  • Every change means a copy is created and then replaced over the original, on the receiving side. This is a fair amount of disk IO (but sparse, so not necessarily disastrous).

  • Conflicts will be handled as usual, and result in conflict copies synced everywhere.

  • If the file is open for changes, Syncthing will rehash it on each mtime update (as often as it can depending on how long it takes). If the file changes while it’s hashed, which is likely, the hashing will fail and be restarted. If it changes but Syncthing fails to detect that, the hash will succeed but the actual sync will later fail because the contents are unexpected.

In short, you can. But you probably shouldn’t.

3

Thanks for the quick reply.

I’ve just tried it on a 10GB image, mounting it on two machines and editing some text files.

I lost edits. So as you said, this method doesn’t work.

The reason I’d like to use a disk image is, that we have some users in the team who have no discipline at all when it comes to file management. I thought it would be a good idea to tell them to mount an image and check it for changes. But I guess that won’t work.

Perhaps the solution really is to just sync a folder, move the .zipped files to that folder and hope that they don’t expand the archives in place.

Or does anyone have better workflow suggestions?

4

If you have access to set up their Syncthing folders, run syncthing as a different user and don’t give them write access.

Edit: don’t give the usual accounts access. Syncthing will obviously need write access.

5

Actually, if it is that big a problem it may be worth setting up a torrent and RSS feed.

6

Yes, that would be a good idea, except it’s a bit too techy for some of our team members.