Prioritize small tasks while transferring

ajp_anton · July 12, 2024, 9:47pm

I’ve noticed that if I’m already transferring a large file, small tasks get postponed until it finishes. Most annoying being deleting files.

I would like large transfers (or data transfers in general) to take a back seat and dynamically prioritize deleting (or moving) files. Even prioritizing small files before large ones would be nice to have happen on the fly, pausing a large file’s transfer in order to transfer newly added smaller files, assuming the large file isn’t almost done already.

Would this be a feature request, or can I already achieve this somehow?

mraneri · July 13, 2024, 5:55am

You can choose to transfer files in size order or random order. I don’t know what happens if a large file is underway and additional small files are found to be transferred.

bt90 · July 13, 2024, 6:06am

Docs to the rescue!

https://docs.syncthing.net/users/config.html#config-option-folder.order

Note that the scanned files are sent in batches and the sorting is applied only to the already discovered files. This means the sync might start with a 1 GB file even if there is 1 KB file available on the source device until the 1 KB becomes known to the pulling device.

tomasz86 · July 13, 2024, 6:07am

Yes, and it’s already on GitHub.

github.com/syncthing/syncthing

Reprioritize sync queue on new incoming index updates

opened 04:18PM - 14 Jul 22 UTC

calmh

enhancement

Currently the queue order ("smallest first", "newest first", etc) is rather best…-effort as the queue is built at one time, and then processed (which can take a long time). During processing any new index updates are effectively ignored until the previously built queue is completed. It would be nice if we could count on a system pulling a small file fairly soon after announcement, if it is configured for smallest-first. I propose that the pulling loop periodically check whether the need situation has changed and if so re-queue everything. This could happen at convenient points (like after completing a file and at least x time has passed since last check) and if we should prioritize we set some flag to stop the operation after currently processing files have completed, effectively ending the pulling loop before it was really done. This is low impact. However, it's not sufficient by itself, because we might be processing a 2 TiB file that will take a week to complete, and this 1 KiB file we should sync just came up. I propose also adding a max sync duration config, unset by default, which could be set to something like an hour or so. That then becomes the deadline for the top level context of the sync operation for a file, so after an hour (if still ongoing) the file will fail and be retried. This is obviously inefficient in the general case, so again default to off, but might make sense in some setups...

At the moment, you can only do it manually, e.g. by pausing and unpausing the related folder.

mraneri · July 13, 2024, 6:13am

I’m also not sure prioritizing deletes is a good idea as deleted files might actually be renames if the scanning is still ongoing on the remote side.

Although I have had a situation where syncing stopped because the disk was at the 1% limit because 1TB of files had been deleted but that update hadn’t happened yet. But that’s another problem.