Experience with Large Data Sets?

General
1

Hey folks! Congratulations on the 1.0 release!

I tried Syncthing about a year ago with a large data set (~2.5 TB) and my experience was that it didn’t do well with initial sync.

Does anyone have more recent experience with large data sets? Should I give it another try?

Thanks for your feedback!

2

Give it another try :-). Turn large blocks on.

1 Like
3

You haven’t quantified what didn’t work, so it’s hard to say if it changed.

1 Like
4

If memory serves, I believe my initial sync never completed. But it has been a while, and I didn’t take notes…

I’ll read up on if Large Blocks are right for my situation.

5

But I’m really hoping to hear from a user that is currently using Syncthing with a large data set. :wink:

6

Always good to know your stuff, but no need to evaluate yes or no: It’s definitely a yes. It will be enabled for new folders by default in what’s probably to be v1.1.0 (or v1.0.2 if not). The only reason it isn’t enabled everywhere by default is, that all connected folders should have it enabled (there shouldn’t be a mix).

Only 400GB here, but 5% of roughly 40k reporting users have >600GB data, so 2.5TB isn’t that special. You just have to keep in mind it needs to hash all your data, and then compare those hashes with all connected devices on first sync - that will take some patience on your side if you keep monitoring it :slight_smile:

Also don’t try with 1.0.0 please, wait for 1.0.1 (released tomorrow) - it’s worth it :wink:

1 Like