Is Syncthing v2 with very large folders possible ?

nothing public yet; I have some internal questions unresolved; my internal implementation we use for our 10+M files installation was to remove all the code that corresponds to internal hierarchy fetch support, fine for us because most of users at scale never need it anyway; but for general public release this is just no-go, because GlobalDirectoryTree is used by many of smaller installations.

Implementing with destroying GlobalDirectoryTree is very straightforward and leads to clear and obvious win, that’s just deleting all the (currently very fat) indexes and replacing them all with 4-byte hash index: index IO pressure reduces 10x. But this makes impossible efficient tree traversals. I cannot public WIP with something “lets delete half of the thing for a while”. This all needs further thinking.

When I have something, it will be discussed in this dedicated thread: https://forum.syncthing.net/t/reducing-v2-db-context-and-footprint-yes-or-no

UPD: or at least it needs parent_id impmentation also right now, also a desired thing, but much more complex too.

1 Like