Assume unchanged on startup

Hello there,

this is a feature suggestion: It would be great to have an option like “assume files unchanges on startup”.

In my case i have to restart my computer often and i have a large amount of folders and data.

So currently every time i restart my computer:

  1. no files are ever changed while syncthing is off
  2. after startup syncthing re-scanns all folders & data
  • which takes long
  • and uses power
  • and strains the HDD
  • and makes my PC slow
  • and due to #1 is completely unnecessary

Thus could you add an option: “assume data unchanged on startup”?

It might be a general option, or per folder option.

Thanks in advance! And thanks for your great work with synthing. I love it :heart:

Happy holidays!

2 Likes

Newer versions of syncthing has the option to linierize scans (maxScans or something like that), as in, scan only a single folder at a time, in which case if nothing actually changed (nothing needs to be scanned and we only need to check metadata), the scans should happen much faster as there is no disk thrashing.

Thanks for your reply.

Which version? i have v0.14.54, Windows (64 bit)

Where is the option? Exact name? I couldn’t find it.

Thanks!

I recall this will come with 0.14.55?

Ok, but even scanning meta data takes 1 hour in my case since i have so many files.
Most folders complete pretty fast, but one is extremely big:
969.230 files 26.035 dirs ~139 GiB

So my feature request still remains: :slight_smile:

Assume nothing changed on startup.

It will scan sooner or later anyway, so I don’t think this will happen as a feature. Scanning if nothing has changed is a cheap operation and should not take an hour, it takes less than a minute for millions of files, and the fact that it takes an hour means things have changed or something else is wrong.

Either here in the forum or on github there recently was the proposal to save the time of the last rescan, meaning no initial scan unless the next scan is already in the past.

4 Likes

This sounds like a great idea!

Here is my scenario:

  • Slow PC
  • Run by my mom
  • Syncthing running on startup
  • She turns on the PC
  • It gets really slow and runs hot
  • To remedy it she restarts the PC
  • Syncthing restarts scanning all files - cycle restarts from the beginning

With this option like @imsodin said it would simply skip the scan at startup and just run normally - as if it had already scanned all folders - ready to sync.

Then it would be like this:

  • Mommy restarts PC
  • Syncthing starts up, but does not do much
  • Mommy is happy because she can use her PC
  • I am happy because Syncthing does its thing :slight_smile:
  • And you can be happy, since you made us happy :wink:

Wouldn’t that be great!? :smile:

This would only be a thing if you’re running without fs notifications, though. If notifications are enabled we might have missed some while we were not running, and should do a full scan on startup regardless.

Hello Jakob,

that’s exactly what i am asking for. I don’t mind if i missed a few changes for the benefit of system stability and usability and not over-using the hardware too much.

Eventually each folder will be rescanned and changes will be picked up anyway.

But it would drastically improve my mentioned scenarios.

BTW most of my folders change seldomly. And i can keep track of improtant changes myself and trigger a manual rescan if needed at any time.

Greetings Fred;

1 Like

Scanning milion files under minute? … Try to have storage on classical rotating disc on Microsoft Storage Spaces volume … Its not under minute :smiley: This feature request is valid. Mostly for receive-only folders, where scan interval is set to zero (never, unles manualy)

Yeah, although it shouldn’t take an hour either :wink:.

The hard data I have is that it takes around 30 seconds to scan 50,000 files. This is on an old HDD on low end hardware running Windows 10 (with Windows Defender enabled). Judging by this, it would take 10 minutes to scan 1,000,000 files.

Of course, we’re talking about rescanning with no changes here, and not the initial scan.

From testing untrusting device last evening:

69(!) minutes startup-scanning (disk thrashing very loudly), 1.2mio files - for NOTHING. Because there is nothing changed. Because nothing can be changed here… Sounds like solid adept to optimization for me.

  • device is Win10, core i5 @ 3.3Ghz, 24G ram … With two 8TB hdd in two-way mirror in ms storage spaces. That storage is not build for speed, just for resiliency. I don’t want to lose that data, you know :wink:

Sounds excessive, but I’ve got no experience with storage spaces to know how they potentially can affect I/O performance.

At the moment, the only thing you can really do is to either keep the computer with Syncthing running all the time, or if you have to shut down, use sleep or hibernation only. There’s an option in the Advanced Configuration called Restart On Wakeup. If you disable it, Syncthing won’t restart itself when resuming the OS from those states.

It will scan sooner or later anyway, so I don’t think this will happen as a feature. Scanning if nothing has changed is a cheap operation and should not take an hour, it takes less than a minute for millions of files, and the fact that it takes an hour means things have changed or something else is wrong.

OK, something else is wrong then. Check my previous post - 69 minutes to check 1.2mio files, without changes to anything. I would say, even if it scan sooner or later anyway (or not at all in my case, if not needed) this OP idea is legit and worth to add as option.

As @tomasz86 wrote, running 24/7 on some UPS can solve this. Sometimes we (or win) need restart, or there is another reason to stop/start syncthing. This option would save day! … I am aware, we can set MaxFolderConcurentLimit to 1, to avoid disk thrashing, but this also limit sync(transferring) functionality too, which is big no-no for me, as I want all folders make progress regardless if previous finished.

Yep, I have the exact same problem with this setting (actually called maxFolderConcurrency). This was discussed on the forums quite a few times already, but essentially, with limited bandwidth, there’s no need to limit syncing concurrency, however you may still want to limit scanning concurrency, especially with HDDs.

Personally, I’ve been looking into completely reverting the commit https://github.com/syncthing/syncthing/pull/6290, at least for my own use, so that I can have the good old MaxConcurrentScans back :sweat_smile:.

Edit:

Reverting the whole commit is too complicated, because things have changed too much since then, but I think that I’ve just managed to hack around this by removing the following part of the code.

With it gone, the whole FolderSyncWaiting state doesn’t trigger, leaving only FolderScanWaiting in place, effectively making maxFolderConcurrency limit scanning only.

I still think this is the better idea.

3 Likes

Wouldn’t it be nice to have something like this enabled by default on Receive Only/Encrypted folders, but not necessarily on the other types? This is of course assuming that the option is going to be configurable.

It could be default on every folder.

1 Like

Respectively every folder with watching for changes disabled.

3 Likes