Nice, we know more.
I intentionally set “alphabetic order” assuming that this will prevent high water marks and need for scanning the whole system in order to determine which file is the “oldest” in the scenario “oldest first”. It seems, that regardless of this setting, syncthing scans the whole file set and THEN sorts them in alphabetic order. This is unnecessary obviously, because you can determine alphabetic order just by traversing the directory tree, so scanning, checking for changes and sending to other clients could be done in parallel and not sequentially (as in “oldest” scenario).
As for keeping state, I also assumed that syncthing will use some kind of persistent cache like sqlite, or even memory mapped file. This would prevent the need for memory allocations for a whole file set. Using sqlite or mmf allows for block access and of course cleaner architecture (and possibility of external analytic tools for cache/scan progress checking).
Thanks for suggestion with scan interval -1, I’ll check this later, because syncthing is also a processor hog, and I need some of my processing power for other tasks
As it seems I encountered “algorithmic” and “implementation” problems with state management. The other problem I reported was a situation where syncthing couldn’t sync a large file > than 50GB. This probably resulted in memory overflow and pagefile quota alert which hanged OS.
Is there any way for me to help you test or something? I’m developer, but sorry, the Go language and its tooling is not my area.