Continuing the discussion from Syncthing killed repeatedly - no errors :
Would it make sense to use a disk-spilling queue in both scanner and puller? Scanner is a bit more problematic as we store a file info (without block though), not just file name. It seems pretty doable to use an adjusted version of the index sorter for this. It’s not the nicest solution considering the disk io discussions going on, but better than OOM crashes and short of walking the folders twice, I don’t see a possibility to have progress updates without intermediately storing files to be processed. I’d definitely propose to use some heuristic criterion for spilling, not a fixed max size, to prevent spilling on systems that can take the memory spike.
Do we have any upcoming changes to fix this?
I have my biggest folder constantly paused, being unable to sync it due to its huge number of files and resulting OOM crashes while scanning / syncing.