Hashing performance degradation when using large blocks

Support
1

I have an interesting issue, probably caused by me not understanding something. I have a 20gb file I want to sync between devices. It is static, and no other process is modifying it.

Now, what I see. When “Use Large Blocks” is not enabled, I get consistent 188.5 MiB/s indexing performance:

2019-02-22 15:43:51 Walk 5rslf-uzcoq [] current progress 1964900352/23638202528 at 188.5 MiB/s (8%)

When I enable “Use Large Blocks”, hashing only shows 3.4 MiB/s:

2019-02-22 15:38:52 Walk 5rslf-uzcoq [] current progress 167772160/23638202528 at 3.4 MiB/s (0%)

On startup, benchmark shows:

2019-02-22 11:14:05 Single thread SHA256 performance is 497 MB/s using minio/sha256-simd (94 MB/s using crypto/sha256).
2019-02-22 11:14:06 Hashing performance is 205.38 MB/s

All other settings for the folder have default values, installed version is 1.0.1. What am I doing wrong? Thank you

Update: I am observing the same behavior with 1.1.0-rc.1

2

Interesting, I wonder if the calculations are based on small block size, doing less blocks per second (as they are bigger).

3

Have you verified it’s not just visual? Do you actually see degraded performance?

4

Yes.

Indexing for a 22gb takes about 2 hours with large blocks. Indexing for the same file without large blocks is 2 minutes. Kind of hard to miss the 60 times difference in time.

Indexing for a 4gb file is 6 minutes (13 MiB/s) vs 30 seconds without large blocks (same 180 MiB/s).

It does look like the larger the file (and so the dynamic block size) - the longer it takes to index.

5

For testing, I have created a folder, not shared with any other server, to ensure there is nothing else (like networking) is involved. The results above come from Synology NAS, but I have consistent results coming from a Linux box as well with crypto/sha256 (so it is not simd library):

2019-02-22 17:13:17 Single thread SHA256 performance is 205 MB/s using crypto/sha256 (201 MB/s using minio/sha256-simd).
2019-02-22 17:13:18 Hashing performance is 154.84 MB/s
6

I have tried setting “Weak Hash Threshold Pct” to 101, there is no difference in performance.

7

Just out of curiosity I dumped a 50gb file. It shows the same hashing performance as 20gb one (around 3.4 MiB/s), and the time is 5 hours. Without large blocks it does ~180 MiB/s and completed in 5 minutes.

Given the default in 1.1 is to always use large blocks, this looks scary.

8

Very interesting. Can you try setting hashers to 1?

Is there any difference between what resources are the bottleneck (CPU vs io) in the two different cases?

Perhaps you could get some profiles?

9

Same issue here.
Scanning a ~30GiB VM Image on an NVMe SSD.
With large block enabled: ~ 6.5 MiB/s; without: 235MiB/s

INFO: syncthing v1.0.1 "Erbium Earthworm" (go1.11.5 linux-amd64) teamcity@build.syncthing.net 2019-01-18 10:34:18 UTC
...
INFO: Hashing performance is 241.76 MB/s
  • Hashers: 1
  • Copiers: 1
  • Weak Hash Threshold Pct: 101

Or another large file on another machine with spinning disk:
With large block enabled: ~ 6.5 MiB/s; without: 175MiB/s

  • Hashers: 0
  • Copiers: 0
  • Weak Hash Threshold Pct: 101

I checked htop on the one with the spinning disk.

  • with large blocks enabled, syncthing maxed out 1 CPU core while scanning that one folder
  • without large blocks, it didn’t (~ 60-80% of 1 core).

Mem usage didn’t really change noticeably.

10

What about io?

11

On Synology:

  • without large blocks: CPU utilization is 30%, disk 40%
  • with large blocks: CPU is 30%, disk 0-3%
12

Yeah I can repro in the tests, looking into.

13

Would I be correct assuming https://github.com/syncthing/syncthing/pull/5556 is the fix?

1 Like
14

Sorry, yes, should have reported back but didn’t.

15

@calmh can we haz rc?

16

Yes you do. And it would actually be helpful if you could raise an issue on github that could then show up in the changelog (or not, then I’ll do it).

17

Huh?

18

The large blocks enabled in the current rc make scanning performance 3mb/s, surely we need to cherry-pick the fix and roll a new rc? Or am I misunderstanding something.

19

Right… Yeah I didn’t think about large blocks being enabled by default in 1.1. Change is safe enough that I’m not worried about the late RC, so yeah, will do. This thing really does need an issue though.

20
3 Likes