Very long initial scan

How can I distinguish it?

Database has ~620MB (du -sh in .config/syncthing/index-v0.14.0.db), this should fit in memory. If it is really one of theese two options, I would guess metadata lookup (perhaps something related to encryption?). Can I check it and compare between both machines in some way?

Can you just move the directory somewhere where itā€™s not encrypted and symlink it?

It has 176 GB and I want to have it only encrypted on new machine. So, yes, but I would prefer another way how to check it ā€¦

Iā€™d Google reasonable performance troubleshooting steps for your OS and narrow down what the issue is.

1 Like

Will try it, but is a fresh install to a new computer.

Can I enable some more detailed syncthing logs, to see, what it does?

Yes, start by using the -verbose flag. There are various trace options, but the output is almost impossible to interpret without looking at the code.

Just as a reference:

I use LUKS full disk encryption for my system (NVMe SSD) and data (WD Red) on my home server with an i3-6100T. My initial scan with a bunch of folders and 53GiB in total only takes a few seconds.

So encryption itself shouldnā€™t result in your problems.

Will try it with -vebose flag. But I realized one strange thing: it had finished initial scan after 42mins. After that, I tried to do a rescan of the biggest folder (176GB from 289GB total) and it was nearly instant. Shouldnā€™t it be the same task?

Update> I have a suspicion, that it somewhat thinks, that files did change and rehashes them ā€¦

Hmmm. Tried to stop syncthing with systemctl and start it again. Initial scan was in seconds and rescan as well. Then I tried to stop it, logout, umount encrypted home (as another user), log back it, start syncthing ā€¦ still fine, several seconds. But after complete system restart, the big long initial scan is back. It does this horrific long initial scan only after system restart.

So, filesystem cache. Your disks are slow. Once the metadata is cached, itā€™s fast.

Sounds reasonably. But both machines have similar disks and if that one on a new machine is ten times slower, I would notice it besides slow syncthing scan, I guess.

I can try to empty filesystem cache manually and do rescan.

Update> I realized, that I tried to umount home before second syncthing start, and Iā€™am not sure, if content of filesystem cache can survive umount and remount of filesystem?

Itā€™s possible that itā€™s the encryption layer that takes time to populate the kernel caches. You can stop syncthing, run free && sync && echo 3 > /proc/sys/vm/drop_caches && free as root and start it again.

After dropping caches, the problem is back. So it will probably be some problem with encryption layer & file metadata gathering. Will try to run simple ls-lR on affected directory, it should be similar. But I cannot believe that encryption layer may cause ten times slowdown :frowning:

Update> Hmm, it seems, that ls -lR is slow too. I donā€™t have time to wait for it right now, will inspect this later ā€¦

There is a bug report for Ubuntu, which may be related:

So Iā€™am bothering at wrong place :wink: Thank you all for your help!

And I filled a new bug report:

https://bugs.launchpad.net/ecryptfs/+bug/1737510

That older one is very old and may be not the exactly same problem.

Thatā€™s a poor bug report as it talks about too many different pieces of software.

You have a reproducer with ls, you should just used that, as I expect the author will tell you itā€™s possibly due syncthing, and I donā€™t expect him to install syncthing to try and reproduce it. Where as he already has ls.

Also, I am pretty sure the original issue does talk exactly about this problem, and the author doesnā€™t seem to care, so I suspect raising a second issue talking about the same issue wonā€™t get you far.

You are right, but author doesnā€™t care about ā€œslow lsā€. He may care about fact, that this bug makes Syncthing (and probably other sync things) useless with encrypted folder.

Iā€™am obviously inexperienced in bug reporintg, what should I do now?

I think youā€™ve done your thing. From the original bug report;

This is probably because stat() has to decrypt every single file.

If thatā€™s true itā€™s a horrible, horrible, implementation of an encrypted filesystem and thereā€™s not much you can do except migrate to better alternatives.

1 Like

You are right. If this is true, then it will not be fixed is reasonable time frame and maybe it is the reason, why author ā€˜donā€™t careā€™.

Iā€™ll abandon it and look for some alternative. Somebody mentioned LUKS here ā€¦

Perhaps it is worth of mentioning somewhere in Syncthing documentation?

Maybe in the ecryptfs documentation. Given that this affects ls equally I donā€™t think itā€™s something we should spend energy on.