Get logs of file writes?


First, thanks for Syncthing, I use it daily as my personal cloud replacement with an always on server. And it is just great! :grin:

Recently I noticed large writes to disk on my server even when there should be only very little transferred.

I use it mainly for small documents/files etc. so I was surprised to see larger then 30Gbs of writes to disk (measured with iotop) in 24hours.

The installation runs on proxmox host → ubuntu server guest → docker install. I first suspected problems with discard not working properly, but I did not find anything. Also atime=off + noatime did not change it.

So I enabled fs logs on Syncthing, but it it only shows what is written, not how much. I suspect that one of the shared folders has a outlook .pst file which is quite big, maybe causing those writes? Even when nothing much is send over the network, due to chunking and hashing, it might write the whole file to disk every time?

Does anybody have suggestions how I could further look into this?

Thank you!

Every synced file is first written as a copy, with blocks transferred or copied locally, then moved in place. So if you have a 5 GB file and change 1 KB, you’ll see 1 KB transferred and 5 GB written.

Hi @calmh

Thank you for answer.

I guess that is a relieve and the “issue” is found.

Just out of curiosity – I don’t know much about how files actually work – why don’t you just write 1KB to the file, if only 1KB has changed?

I am sure there is a reason you are not doing that. :grin:

Because its not atomic.

A rename is atomic.

Ah thanks, I understand.

Write to temp file until finished, then rename if no conflict. Needs to be atomic to reliable prevent conflicts. Kind of no way around that I guess.

Yeah not really. 1 KiB is an extreme case, but consider instead if it were a few hundred megs taking quite a while to transfer and write. What if we crash halfway through? Then your file is trash and we don’t know how to recover it.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.