Faster hashing package

Development
1

I’m diggin up this old topic. There are some people busy to speed up SHA256 hashing with more efficient implementations. The guys at minio are creating a alternative to self host a S3 object storage. It is written in golang: https://github.com/minio/sha256-simd. There is also a blake2b: https://github.com/minio/blake2b-simd

2 Likes
2

That’s very interesting, especially for arm!

1 Like
3

Yeah, currently the crypto/sha256 for arm in stdlib is really slow.

4

Right, it’s arm64 only. So no improvement there unfortunately. :disappointed:

5

Still x86 could benefit a 50% performance boost.

6

Yep. I’ve written up a branch with it in for the adventurous.

We need to be a bit careful before merging it, even as a “drop in” replacement, as when we’ve previously experimented with accelerated crypto implementations it’s blown up spectacularly for people with older AMD and Via CPUs (that the authors never tested for) and similar.

It also fails the builds for PPC, so it’s not drop in enough as it is… :confused:

7

Newer Android phones use arm64, so it should be an improvement for them. Although we’d have to handle backwards compatibility somehow.

8

But we don’t compile with -goos arm64 so far, right?

9

I’ve fixed the PPC build, so if anyone wants to test this build and see if it’s faster and/or works on their 386 computers that’d be cool: https://build.syncthing.net/job/syncthing-branch/285/ (linux etc), https://build.syncthing.net/job/syncthing-windows-branch/3/ (windows).

You should see a faster metric in the “hash performance” output that is printed at startup.

10

You mean -goarch? At the moment we only use arm (or 386). But we could actually just distribute an apk with arm64 via Google Play.

11

Ah, yes.

12

Windows x64, Intel Core i5-6200U

0.14.6: INFO: Single thread hash performance is ~243 MB/s

0.14.6+2: INFO: Single thread hash performance is ~318 MB/s

Awesome :slight_smile:

13

So to test the performance on Android, I’d just compile your sha256-simd branch with -goarch=arm64?

14

I won’t see a difference on standard arm, right? Because there isn’t any…

Slightly above 6 MB/s on my Allwinner A20 board

15

Okay I just tried this on my phone, and it seems way too good to be true.

Syncthing-Android 0.8.3: "Single thread hash performance is ~33 MB/s" master with sha256-simd and -goarch=arm64: “Single thread hash performance is ~817 MB/s”

Does that seem reasonable?

Google Play doesn’t give me statistics how many of our users have an arm64 device, but it should be most devices released in the past year.

Edit: I tested on an LG G4.

1 Like
16

I think that’s plausible, amazingly enough. If that cpu had crypto acceleration and it can use them.

17

It’s pretty surprising, my laptop (2016 model) only gets 372 MB/s. Anyway, I’ve opened a wip PR here.

Edit: What exactly does the faster hashing mean for users? Files are added faster and it uses less CPU for that, right? Anything else?

Edit: To be clear, hash rate on my laptop is without the faster hashing stuff.

18

Now if only intel pulled their head out of their arse earlier…

19

That’s what it means, yes. Also lower cpu usage while syncing as block hashes are checked then as well.

2 Likes
20

~817 MB/s? :astonished: Why is an Intel Notebook CPU not capable of this? I guess its already using AES-NI?