I have never used Syncthing before, but I am looking to use it along with btrfs or ZFS to create a home backup system using cheap, low powered ARM boards. I’ve read about people’s experience using Syncthing on Raspberry Pis and one of the biggest problems with low performance systems appears to be the time it takes to generate SHA256 hashes while indexing.
On certain ARMv8 CPUs, SHA256 can be done nearly 100 times faster, but it is hard to find a cheap board that supports the new instructions. The only one I can find is the LeMaker HiKey, but it has no Ethernet or SATA and only USB 2.0.
So my question is this: Do you think I would get better performance with a device with GigE and SATA or USB 3.0, or would the CPU be the bottleneck (in which case it would be better to use a device with slow I/O, but could do SHA256 very quickly)?
My current top pick for a device with faster I/O is an ODROID XU4 (w/ GigE and USB 3.0).
I am going to create a proof of concept system with hardware I already own to do some performance and stability tests, but I was wondering if anyone had experience with this type of setup.
The performance of syncthing depends indeed on a few hardware capabilities. It uses SSL, SHA256 which are cpu/hardware intensive unless hardware offloaded. You would benefit from a good board with native ethernet (compared to shared USB ethernet on raspberries).
As you mention the ARMv8 CPUs indeed support SHA256 hardware offloading instead of CPU calculation but they are not cheap.
Keep in mind you will need approximate 50-100 mbyte of RAM per syncthing instance.
Also syncthing is NOT an backup system, it is a synchronisation tool but can be used in a backup-like way. E.g: sync to a central point and run a backup cronjob (suggestions are borgbackup or restic). When using multiple peers which do bidirectional sync to and from a central server you can get problems and having backup snapshots is not a bad thing then.
You would benefit from a good board with native ethernet (compared to shared USB ethernet on raspberries).
Yeah, I have a few RPis now, but I doubt they would work well (besides their slow CPUs) because the disk bandwidth would be shared with Ethernet (which is only 100Mbps anyway). I’m looking at the ODROID XU4 because it has native gigabit ethernet and benchmarks put it at 3-4 times faster than the RPi2.
As you mention the ARMv8 CPUs indeed support SHA256 hardware offloading instead of CPU calculation but they are not cheap.
Yeah, it seems that most of the ones that support the SHA2 extension are server boards. The HiKey is the only cheap one I could find. Other ARMv8 boards such as the RPi3 do not have the SHA2 extension.
Also syncthing is NOT an backup system
My plan was to sync to multiple servers in different locations that would then perform their own backups, possibly using btrfs or ZFS snapshots. If that doesn’t work (because of memory issues), I could fall back to more traditional backup solutions.
Yeah, you need at least 512Mbyte RAM to run a decent ZFS system. FreeNAS recommends 8GByte or more
The ODROID XU4 has 2GB of RAM so it might work passably. Its not the end of the world if it doesn’t, because there are other ways of doing what I want to do.
Keep in mind filesystem snapshots (btrfs, ZFS) are not real backups. If a disk dies or a sector is fried you have can have holes in your files. Better create a snapshot, mount it, rsync it to another disk or physical location. Also RAID is not a backup, because when the controller dies or it is eating your data you may not notice. A RAID array is still vulnerable to power-surges (when more than the acceptable disks are fried by discharge).
Putting the btrfs / zfs requirement aside - in general you’ll have no problem.
I’m running a group that’s nearly all RPi - with the main ‘master’ source running on an Odroid XU4 (you will get away with a USB3 drive there too).
Obviously don’t expect blazing speeds but it will work. For best results, ensure the RPi are disconnected rarely, and at the time they are reconnected it’s best to leave them so at least overnight.
The Odroid XU4 won’t become corrupted.
Maybe once / twice a year - you’ll decide to wipe the index on an RPi and let it rebuild.
Other tips are:
Spread out folders. Don’t have ‘Photos’ for all Photos, instead have Photos-2016-P1, Photos-2016-P2, etc. and so on for Videos, Audio and whatever data can be meaningfully broken down into discrete folders. This will reduce the hashing & thrashing.
Always quit the Pi gracefully, if you’re making this for someone else - really emphasise this. Plugging a Pi out from the wall suddenly will eventually require a visit from someone with know-how.
To branch out from the Pi, but stay in the same price range - there are the Odroid C2.
You can add an expense here with eMMC cards not only SDs. This makes the C2 practically immune to the SD corruption possible on the Pi, and noticeably faster.
From a media-centre point of view the C2 are 4K ready with HEVC - so beats the Pi squarely on that front.
I’ve been running this setup for over 18 months now.
Very doable, and not as painful as many would suggest.
This is not ARM-based, but I’m running a Intel Braswell based Board with a S-ATA SSD and 8 GB RAM.
The system is running headless with Ubuntu Server x64 and draws about 5 Watts idle. I basically planed this as a Syncthing “hub”, but using it also as a small file server now (Samba, NFS, SFTP etc.). This way I can also share stuff with devices/apps which do not support Syncthing. I thought about using Btrfs - but choose ext4 in the end, since I don’t need/want redundancy on this box.
I decided to build this low power system, after I found that my HP Microserver Gen8 with FreeNAS is nice for backups and all, but consumes IMHO way too much power (about 40 Watts idle). I still have Syncthing installed on the NAS - but using this basically just for backing up the current state of all the Syncthing folders when I power up the NAS.
I can’t answer your question directly, but I can share my experience. I tried Syncthing on both a Beaglebone Black and on a Raspberry Pi 3, with a total archive size of around 500 GB. Neither could handle it. Trying to sync data would cause the little computer to lock up so hard I couldn’t even SSH into it.
The most recently I tried was probably Syncthing v0.13, so maybe it’s a little better now - but I wouldn’t expect much from the Beaglebone or RPi3 on this front.
Personally I handle about 1 Tb on a Cubietruck (Allwinner A20, 2gb Ram), the data used to be on a Rpi 1, though the repos were smaller at the time. It’s mostly static data so only small changes and propagating large amounts takes time - but the connection isn’t very fast anyways.
This runs for well over 2 years now (only A20) without major issues. I can basically do everything with this machine while syncthing is running that I’d do without - and could with the pi as well.
Just don’t expect wonders speed wise. Hashing performance is like 6mb per second, but since my upload speed isn’t much better anyways this works just nicely for me.
So, while this works nicely for me, I wouldn’t necessarily recommend it because of the speeds. Hardware supported hashing definitely would be nice. But so is a dedicated sata port and Ethernet
I am running syncthing on a WRT1200AC router. It has 512MB RAM, an E-SATA port and a USB 3.0 port.
Single thread SHA256 performance is 15 MB/s using crypto/sha256 (15 MB/s using minio/sha256-simd).
my experience with synchting on odroids are equally positive. the 2gb ram and the cpu speed is sufficient for my 500 gb of data, with decent ethernet and disk using usb3. around 100 euro the cpu and 50 euro per disk.
i have three such nodes, which are always on in different locations and serve data to local machines, including serving music to a raspi mpd server. the connections are around 5gb/s nominally.
problems occur only when large (50gb) directories are moved around, otherwise the whole setup is “transparent” to users. users move from location to location and find their desktop at the new location as they left it behind.
for backup there is a separate old cubieboard running rdiff-backup…
i recommend the setup.
I actually had the same thought as you. I ran Syncthing on my ODROID-XU4 for about a year (eventually switching from btrfs to ZFS), which worked well enough, although it was a kind of slow (mostly because it couldn’t handle ZFS that well).
I wanted to try to get better performance, so I recently bought a Rock64 (the 4GB RAM version), but I haven’t had time to get Syncthing running on it yet. The main reasons I like the Rock64 are that it has the most RAM in its price range (important for ZFS), it’s 64 bit (ZFS technically only supports 64 bit systems) and it has hardware accelerated crypto. Not only does it accelerate AES for encryption, but it also does SHA256, which should make Syncthing faster.
Here are the results of cryptsetup benchmark:
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1 254015 iterations per second for 256-bit key
PBKDF2-sha256 474898 iterations per second for 256-bit key
PBKDF2-sha512 248713 iterations per second for 256-bit key
PBKDF2-ripemd160 151178 iterations per second for 256-bit key
PBKDF2-whirlpool 66737 iterations per second for 256-bit key
# Algorithm | Key | Encryption | Decryption
aes-cbc 128b 348.5 MiB/s 435.4 MiB/s
serpent-cbc 128b N/A N/A
twofish-cbc 128b 31.6 MiB/s 34.2 MiB/s
aes-cbc 256b 302.2 MiB/s 397.4 MiB/s
serpent-cbc 256b N/A N/A
twofish-cbc 256b 32.1 MiB/s 34.2 MiB/s
aes-xts 256b 384.4 MiB/s 389.0 MiB/s
serpent-xts 256b N/A N/A
twofish-xts 256b 33.7 MiB/s 34.5 MiB/s
aes-xts 512b 356.5 MiB/s 357.4 MiB/s
serpent-xts 512b N/A N/A
twofish-xts 512b 34.0 MiB/s 34.5 MiB/s
If I have time later, I’ll try to run Syncthing’s benchmark.
The software support for the Rock64 is significantly worse than the ODROID-XU4. Only a few distributions explicitly support the Rock64. For others you will have to compile your own bootloader and kernel for all the hardware to work. The mainline Linux kernel boots fine, but USB does not work. For USB support you have to use the officially supported 4.4 kernel.
Hi all
Does anyone ever tried UDOO X86 Ultra ?
Surely it is expensive !! But it seems a good piece of hardware with 8gb of ram and an mmc memory!
If anyone has experience in running Syncthing in this device, it would be helpful to report some results here
Regards
I got a chance to run Syncthing’s benchmark on the Rock64 today and I got a very impressive result:
INFO: Single thread SHA256 performance is 671 MB/s using minio/sha256-simd (17 MB/s using crypto/sha256).
INFO: Hashing performance with weak hash is 159.14 MB/s
INFO: Hashing performance without weak hash is 493.97 MB/s
I haven’t started using it for real workloads yet, but this result makes me think the Rock64 is going blow all of the older boards out of the water in terms of price/performance ratio for Syncthing.
For comparison, here are the results from a few other of my machines:
Xeon E5-1620 v2:
INFO: Single thread SHA256 performance is 315 MB/s using minio/sha256-simd (210 MB/s using crypto/sha256).
INFO: Hashing performance with weak hash is 265.52 MB/s
INFO: Hashing performance without weak hash is 296.59 MB/s
i7-6700HQ:
INFO: Single thread SHA256 performance is 379 MB/s using minio/sha256-simd (322 MB/s using crypto/sha256).
INFO: Hashing performance with weak hash is 308.00 MB/s
INFO: Hashing performance without weak hash is 363.33 MB/s
Hi, I’m new here.
I also have both Rock64 4GB and Rock64 2GB.
I noticed, that this incredible performance Single thread SHA256 performance is 672 MB/s using minio/sha256-simd (462 MB/s using crypto/sha256) only achieved when using syncthing binaries compiled for aarch64. The armhf variant on the other hand gives very poor results: Single thread SHA256 performance is 18 MB/s using crypto/sha256 (18 MB/s using minio/sha256-simd).
I tested using debian stretch and 0.14.45 release binaries.
