I finally looked at the log. There’s indeed a whole lot of filesystem calls originating from osutil.IsDeleted calls in the first phase of scanning and after WithBlocksHash was called. That plus calls on files which have the same name and live at different paths points at findRename. And that’s inefficient in this case: It will check all these files whenever one of the files is scanned, not caching the fact that it existed when it was checked the last time.
2 Likes
Thank you very much for your work on this. I’ll have a look and see if I can compile with the pull request incorporated. Otherwise, I may have to wait for the next version to come out. I’ll keep you posted.
I’ve managed to compile and test with the change and it has definitely rectified the problem. It would take a couple of weeks to make it to 40% and now it’s up to 65% in about 15 minutes.
Because I’m running it in a docker container, I ended up compiling outside of docker and then mapping the compiled binary into the docker container. Seems to have done the job.
Thank you very much for your help on this rather unique issue. I think we can mark this one as solved.
2 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.