Index folder and transfer-on-restart size

Hello there,

New user here. Thanks very much for making Syncthing available! I very much like the configuration experience but have a couple of questions related to indexing please.

The folders that I’m syncing come to 37,516 items, ~864 GiB according to Syncthing’s GUI. My index-v0.11.0.db directory is 1.9G, and every time a node is restarted it seems to result in about 235 MiB being transferred to/from each of the other computers being synced with. Are these sizes/transfers typical for this kind of collection?

If not, is there a safe way to clear Syncthing’s index directory?

As an aside, as a novice I wonder why an index isn’t merely a collection of checksums or similar, which would take up less space/bandwidth. I’m sure there are good reasons; is there somewhere I can go to read on how the index works?

Many thanks.

Index is a collection of checksums, it stores 64 byte hash for every 128kb of data. Which works out at around 60mb of hashes only for the data, plus flags, filenames etc, I guess it sounds about right.

You can try and reduce the index size by making sure folders are in sync, then shutting down all nodes, deleting the index directories, and letting all nodes reindex everything (cpu intensive). This will forget all the deleted files which might help reducing what’s transferred.

The protocol spec is here: https://github.com/syncthing/specs/blob/master/BEPv1.md

@AudriusButkevicius, thanks for that. I’ll try deleting the index directories as you describe, and have a read of the spec and see how much I can understand.

Are you saying that it should be about 60MB for the hashes for the entirity of a 864GB collection of files, and that the rest of the index is other meta data?

Well it’s hard for me to say what’s exactly in the 235MiB. My calculation is actually wrong.

Given 64 bytes per 128 kb of data, we get 864 gb / 128 kb = 6750000 hashes 6750000 hashes * 32 bytes each = 216 mb of hashes.

But there is compression involved by default, hence it’s potentially less.

Edit: we use sha256 not sha512, hence hash size is 32 bytes.

2 Likes

also this will help

delta-only index exchange would be awesome! :smile: Also very Important for Mobile Users. Hope this will be implemented soon :worried: