Question about copiers.

[irve]

Would the increase in the number of copiers be a suitable reaction to the following problem?

Last week our 200+GB synced folder stopped its work and while Network IO was low, the system itself had a high CPU load. This caused a lot of confusion and I spent a day investigating it.

What I suspect: a user moved a directory containing around 20000 small files into another place which caused the block-tracker to be so overwhelmed by requests that it stopped the regular sync while it sorted out the change.

While I recreated the directory from the most intact copy whilst throwing out the problematic folder I would have liked to understand how I could avoid such problems in the future. So: was the amount of copiers the problem and would it be okay to put 4 copiers to work under such circumstances?

A related topic which I didn’t want to resurrect:

What is the difference between copier, puller, and hasher

This is a two part question. 1) - What are the responsibilities of the copier, puller, and hashers? I can assume the hasher is just busy getting file hashes, and the copier is busy doing io, but what about the puller? 2) - What would be an ideal starting setup in general and also in number of copiers, pullers, and hashers for a application that will only consist of a local syncthing instance, and an instance of syncthing in a docker container on the host machine. So just a one to one two w…

[calmh (Jakob Borg)]

I doubt the number of copiers would have much impact. I also doubt the root cause of the issue. We fixed a couple of things in the last releases, mostly around big files, but could still be related. Moving a directory with 20k files in it shouldn’t take that long or overwhelm it, I’d be surprised if it took more than a minute or so. Assuming we’re not talking about a phone or something.

[irve]

Thanks. I guess the index was broken somehow, then.

If we ran into the same issue I’ll try to get some logs.