Syncthing and defragging, also missing files

General
#1

Something that catches my eye occasionally, more so tonight then previously, when I defrag, is Syncthing treating the moved files as modified?

In theory it shouldn’t however a defrag that’s been running has resulted in many jobs thinking something has changed on the RO side. Many of the remote devices are showing as items changed (could this be the 95% / x B differences that keeps cropping up recently on RO devices?).

For example, a pst file that’s 10 years old (along with around 280Gb of data) has decided to now redownload in a folder that I know was pretty much up to date. There is no reason why that file would redownload as it’s part of a deep archive that no one knows exists and absolutely isn’t hanging off an Outlook .

I checked the SO side in case something hit the folder, eg, a crypto virus, but all seems ok.

I have shutdown Syncthing to let the defrag finish and then will resume when done.

Thought it worthy of reporting (in general, not support)

Windows 10 both sides

#2

No, as far as I understand defrag is a pure disk operation, moving blocks around to reclaim space, it should not have any effect on filesystem metadata

#3

I agree, defrag shouldn’t do anything and maybe i’m looking at reasons why something went wrong tonight and happen to be defragging.

I can’t find anything in the logs, only thing that’s changed was RC4, but here’s what i’m seeing

SO

image

RO

image

So a few folders on the RO side have been created, and when I look in them, a huge chunk of data has been removed. Nothings gone into the ‘file versioning’, it’s literally ‘gone’

image

almost echos I lost all my desktop icons which I replied with a plausible answer. Now i’m not so sure.

I have enabled fs logging to see if anything changes over night

#4

here’s the difference…

SO

image
image524×711 92.9 KB

RO

image
image592×554 5.19 KB

So this folder seemed to have suffered the most data loss and is redownloading everything again

#5

The RO screenshot shows those blue arrows, the files seem to have been compressed. Could that - if true - cause a metadata change or is the Windows’ compressed attribute not relevant to Syncthing’s scanner?

#6

The compressed attribute is not something we look at, and I assume compression should be otherwise transparent.

Given local state << global state, it seems Syncthing is at least aware the files are missing, and given the out of sync items is struggling to fix it?

#7

Files across the folders are coming in…

image

Just the fact they disappeared is concerning. Only difference between the SO and RO at the time they dissappeared was SO was rc4, RO was rc2.

I documented this in case others report a similar issue.

#8

Are you using the built-in defragmentation tool or a 3rd party solution? The former should not cause any issues for sure, but with the latter there is always a possibility that the software is doing some shenanigans while defragmenting the files.

#9

both. However I was using a 3rd party to recover free space, however I have used this for more years than I can remember, plus, this was specific to a group of Windows folders within one St folder

#10
  1. I’m puzzled why anyone should be defragging in Windows10,isn’t this something the O/S does automagically. (Not before time Microsoft!)
  2. Since defragging involves moving blocks around wouldn’t it be sensible to pause Syncthing while this was going on? Expecting it to cope with such disk activity is asking a lot.
#11

  1. W10’s own defragger often doesn’t kick in when needed, only when it decides. Whilst 3rd party defrags all use the windows API, there’s times where I want a little more control, usually to consolidate the free space.

  2. I tend to only manually defrag when there is next to no activity, else Syncthing suffers with slow activity

I do find that Syncthing creates a lot of fragmentation so the scanning tends to suffer. I run large drives, 5 / 8 / 14Tb so I have to be on top of the optimization.

#12

Just for the record, it does not happen “automagically”, but rather there is a scheduled task that runs only when idle, and only when there are mechanical HDDs present in the system. “Idle” in Windows 10 usually means a state after at least 4 minutes of inactivity or, in other words, lack of any user input (e.g. mouse, keyboard, stylus, etc.) and no video being played.

I personally disable the built-in task and do not bother defragmenting hard drives anymore, not because there is no benefit, but simply due to the fact that it takes way too much time. The only ones that I would still bother to do it are 10k or 15k rpm SAS drives.

I have not used this myself, but have you perhaps tried setting blockPullOrder to inOrder (see https://docs.syncthing.net/users/config.html)?

#13

I didn’t know about the block pull order, so have updated the config and will give that a go. im still defragging - 24 hours in and still at 31%!! it’s the Tb sized files that drag the scan speed down when they get fragmented.

#14

Sorry, my attempt at humour obviously misfired. “Automagically” = “Scheduled Task”. I’d be a little worried if Windows tried to defrag anything other than a mechanical HDD!

1 Like
#15
  1. I’ve heard others say this about W10s defragger but it’s not something that’s ever troubled me. I can see that with your very large HDDs the scheduled task might not be suitable.
  2. Have you ever tried pausing Syncthing while defragging to see if the problem with moved files being treated as modified disappears? Or with such large drives would this put Syncthing out of action for an unacceptably long time?
#16

I run defrag with Syncthing running and without. Key thing is the IO, if it’s busy, then I will wait as there’s no point is making extra work with excessive seeking.

#17

It is all right :wink:. I just wanted to add some details regarding the defragmentation process’s run conditions. Also, just for completeness, Windows Vista and older will in fact defragment SSDs too, so there the automatic task should definitely be disabled (unless there are no SSDs in the system).