SyncThing for syncing a raw block device

colemickens · August 9, 2021, 4:13am

Hi!

I love SyncThing! I would really like to be able to use it to synchronize a block device.

Is this feasible? I tried a symlink to a block device and “Junctions as Dirs” but that didn’t work.

Somewhere a Syncthing contributor mentioned that it should work well for block devices, but I think that meant files that represent block devices.

It seems like all of the chunking and syncing logic should be perfectly re-usable with SyncThing, if only I could tell it to read a symlink to a device and honor it.

Can someone let me know how feasible this is, and/or how hard it would be to add this to SyncThing?

colemickens · August 9, 2021, 5:19am

It seems like maybe one of the changes would need to be in here.

It seems like this is somewhat near the bottom of the scanner’s walking process, where it handles regular files:

github.com

syncthing/syncthing/blob/93a02c677e62b7b4ed3b243676806dc7c96ee1fe/lib/scanner/walk.go#L336

    
      
          			if err = w.handleItem(ctx, ignoredParent, info, toHashChan, finishedChan, skip); err != nil {
          				return err
          			}
          		}
          		ignoredParent = ""
          
          
		return nil
          	}
          }
          
          
func (w *walker) handleItem(ctx context.Context, path string, info fs.FileInfo, toHashChan chan<- protocol.FileInfo, finishedChan chan<- ScanResult, skip error) error {
          	oldPath := path
          	path, err := w.normalizePath(path, info)
          	if err != nil {
          		handleError(ctx, "normalizing path", oldPath, err, finishedChan)
          		return skip
          	}
          
          
	switch {
          	case info.IsSymlink():
          		if err := w.walkSymlink(ctx, path, info, finishedChan); err != nil {

I’ve used rsync --devices /dev/disk/by-partlabel/boot /syncthing/test/boot_node to create another block device node in the /syncthing/test directory and give my user ample permissions to access it.

Any thought so far?

calmh · August 9, 2021, 6:35am

You can probably trick Syncthing into reading the device fairly easily. However it won’t have any useful modification metadata or support notifications so I’m not sure what mechanism you’d use to schedule rescans.

On the receiving end, if you’d like that to be a block device as well, you’d need to implement in place writes as Syncthing currently always uses a temp file that it renames over the destination. This would be a major chunk of work to do properly.

In short, I think you’ll find Syncthing is a bad fit.