Syncthing scalability / no. of files, folders?

Feature
1

Hi there!

How well does Syncthing scale, not in terms of nodes but in terms of files / sizes?

I have a data network share and would like to sync it with local copies both on my desktop and my notebook.

I’m talking about these dimensions:

103,314,954,729 bytes = 96.2 GiB
        178,100 files
         17,021 folders
            567 kiB average file size

Since Syncthing is hashing every file, that would be a tremendous database! At least 790 kiloblocks times 1 kiB/hash is already more than 700 MiB just for the database.

Do you have any suggestions / alternatives?

Thanks you!

Lucas.

2

It’s more like 48 bytes or so per block. It’s going to be a few tens of megabytes of index database for sure, but not as much as you expect I think.

1 Like
3

Thanks! SHA256 is 32 bytes, so the remaining 16 bytes are a file number / ID? Do you recommend to try it for this amount of data?

4

data.syncthing.net shows stats on something you might care about.

1 Like
5

Thank you very much, this is really convincing!!!

6

Well i am experiencing real life problems with a folder of this size:

  • About 1 million files
  • About 30k folders
  • About 80GB of data

(in this combination)

The problems are mostly OutOfMemory crashes (System has only 1GB Ram + 2GB Swap).

Details here: Syncthing killed repeatedly - no errors

But i guess these limits don’t necessarily apply for stronger systems (esp. with more RAM).

7

My machine is running fine with

122088 Files 4535 Folders ~853 GiB of Data

And it is just an Allwiner A20 with 2GB of RAM (Cubietruck 3)

Speeds are not that impressive though, but enough for me - most important thing is reliability, and this machine has been running almost 24/7 for the past 3-4 years without major issues.

1 Like