Please test in-filesystem copies

The 1.8.0 release candidate has a new feature where it can copy data by asking the file system to do it directly, instead of first reading and then writing the data itself. Per the feature description:

This can be used to optimise copies on network filesystems, improve speed of large copies or clone the data using copy-on-write functionality if the underlying filesystem supports it.

The details of what is supported on which OS and filesystem are somewhat hairy. However, if you’re running Linux with BTRFS or ZFS or (maybe?) ZFS, or ZFS on FreeBSD, or Windows with ReFS, CsvFS or a modern CIFS file share, you might see improved performance and/or reduced space usage.

However this is not well tested on all possible systems, so we’d greatly appreciate some help in seeing what works and what doesn’t.

:warning: :warning: :warning:

This is not without risk. Please do testing on a less critical system, and only with a good backup of your data available. Minimizing that risk for regular users is why we’re asking intrepid ones to do testing. :slight_smile:

:warning: :warning: :warning:

Enabling the feature requires setting the appropriate value for the advanced folder option copyRangeMethod. The documentation is here:

https://docs.syncthing.net/advanced/folder-copyrangemethod.html

2 Likes

This looks very exciting - woop!

Is there a particular way that you would like success/failure to be determined?

Yehaaa thanks, I’ll shoot my data in :+1:

If it doesn’t work, it will error out most likely.

I guess the way to test this is:

  1. Observe no errors
  2. Not lose data
  3. Maybe verify that some blocks were reused between files by checking “total size of data” vs “total size on disk” equivalents.

For network shares I guess verify the speed of 1GB large file before and after or something.

And as far as I understand, savings will happen in two cases:

  • When using versioning, the copy might be copy-on-write so used size is smaller than the whole file
  • When syncing a file and creating a temporary copy of it, blocks will be reused from the existing file

In both cases the amount of read/write I/O should also be less so things might be faster in general.

Or reused from existing file, namely you have file A and make a copy of it as file B, it should reference blocks from file A and not take up any more space.

1 Like

Stupid question - where (in which dialogue) am I supposed to change this value?

it’s not exposed, and only available in the advanced config where you have to supply it the correct text value.

Ok - I know how to access the advanced config, could you share with me the correct text value. Is it something like copyRangeMethod = sendfile ?

The linked help page lists all the available modes in their correct text format.

Hello:

I’m looking at this on a Synology NAS (DSM 6.2.3) running BTRFS - and it’s looking good.

I ran four test passes, duplicating a test file on another device and changing copyRangeMethod on successive passes.

I saw no errors - and looking at my shared folder with btrfs filesystem du '/foo/bar', I can see:

     Total   Exclusive  Set shared  Filename
 119.27MiB   119.27MiB           -  /foo/bar/testfile.dmg
 119.27MiB   119.27MiB           -  /foo/bar/testfile copy.dmg
 119.27MiB       0.00B           -  /foo/bar/testfile copy 2.dmg
 119.27MiB       0.00B           -  /foo/bar/testfile copy 3.dmg
 119.27MiB       0.00B           -  /foo/bar/testfile copy 4.dmg

  • The first copy was generated with copyRangeMethod set to standard;
  • ‘copy 2’ was generated with copyRangeMethod set to sendfile;
  • ‘copy 3’ was generated with copyRangeMethod set to ioctl;
  • ‘copy 4’ was generated with copyRangeMethod set to copy_file_range.

All resultant files produce the same MD5 hash. :joy:

2 Likes

Hello:

And, for the record, copy_file_range, ioctl and sendfile all fail on macOS 10.15.6 Catalina on an APFS destination. (TBH I don’t know if macOS yet supports the necessary commands to CoW-copy a block range.)

Nothing is supported outside of Linux, Windows and Solaris. It’s possible there are APIs we could use on Mac, but they’re not implemented.

has anyone tested how it works on zfs (ZoL)?

I don’t think it has any effect on zfs as zfs does not reuse blocks of the filesystem, it reuses the whole filesystem as you snapshot, so there are no per file optimisations.