% du -sh .config/syncthing/index-v0.14.0.db
2.2G .config/syncthing/index-v0.14.0.db
% ls -1 .config/syncthing/index-v0.14.0.db | wc -l
938
debian sid, 1.1.4~ds1-4
2Gb index-v0.14.0.db
after syncthing -reset-database index-v0.14.0.db is 32Mb
-reset-database deletes the db so it’s sort of expected that is has gone down. Did it stay like that after scanning all folders?
How many files/folders are we talking about here?
How many devices connected?
Did they (all of them) reconnect to you since resetting the database? If yes, what’s the db size then?
What does the database look like in terms of size on other devices?
Did it stay like that after scanning all folders?
before -reset-database I pressed Rescan All but nothing has changed.
How many files/folders are we talking about here?
6 synced folders
How many devices connected?
There only 2 devices pc and phone.
Did they (all of them) reconnect to you since resetting the database?
Yes!
If yes, what’s the db size then?
32Mb.
What does the database look like in terms of size on other devices?
The other device is only android, I don’t know how to check it there.
Did you rescan after the db reset?
Do you still have a copy of the old database?
What number of files/folders inside those 6 folders?
Did you rescan after the db reset?
Yes
Do you still have a copy of the old database?
No ):
Presumably the other side should have the same database in terms of size if it was not reset.
This could have been caused if you had a large number of files that were later deleted, as items are never removed from the index.
Other than that, it’s hard to say or understand given the evidence is gone.
What number of files/folders inside those 6 folders?
4500 files in 363 folders takes 34.05Gb total
/data/data/com.nutomic.syncthingandroid/files/index-v0.14.0.db is 25Mb
Right, see if you can get your hands on a db, as now the evidence is gone, so hard to understand the cause in order to fix anything.
I am seeing not only large index directories, but also an enormous number of files in my index. Here is a raspberry pi running (only) syncthing, with over 2.4 million files in the index folder. I had another pi with over 3 million:
pi@pijeffoffice:~/.config/syncthing $ du -h index-v0.14.0.db
12G index-v0.14.0.db
pi@pijeffoffice:~/.config/syncthing $ ls -fl index-v0.14.0.db | wc -l
2457856
pi@pijeffoffice:~/.config/syncthing $ syncthing --version
syncthing v1.3.1 “Fermium Flea” (go1.13.3 linux-arm) deb@build.syncthing.net 2019-10-07 11:30:25 UTC
Before I do a -reset-database, is there anything you would find useful?
You probably run a 32bit system. This is a problem that came up on these systems with the new db setting introduced in 1.3.0. Update to 1.3.1, where these have been reverted for 32bit and the problem will go away.
OK, so I should go ahead and do the -reset-database, and it should clean it up and not recur now that this pi is on 1.3.1? thanks for the quick response!
I am not sure, but I believe you don’t even need to do the -reset-database. Doesn’t hurt to try: Run 1.3.1 and wait some time. If the amount of files in the db doesn’t decrease, do -reset-database.
FYI, I let it run all night, and it seems to be stalled at around 164,000 files and 3.3GB consumed, which still seems very high:
pi@pijeffoffice:~ $ date ; ls -lf ~/.config/syncthing/index-v0.14.0.db/ | wc -l
Wed Nov 6 08:08:56 EST 2019
164085
pi@pijeffoffice:~ $ date ; ls -lf ~/.config/syncthing/index-v0.14.0.db/ | wc -l
Wed Nov 6 10:45:54 EST 2019
164084
pi@pijeffoffice:~ $ du -h ~/.config/syncthing/index-v0.14.0.db/
3.3G /home/pi/.config/syncthing/index-v0.14.0.db/
I’ll leave it until tonight, but if it is not greatly reduced, looks like I’ll be doing a -reset-database in the end anyway…
Someone else had the same problem and it “worked” without -reset-deltas (“worked” as maybe the reduction of files by one order of magnitude was enough for that user). Maybe the db only fixes files that contain something, that it needs to write to/read from. So before doing the full -reset-database, you could first try -reset-deltas which also results in lots of db activity (but no hashing).
There is also a secret environment variable, STDEBUG_CompactEverything=1 which will force compacting the database on startup. Count on this taking a while, during which nothing appears to happen.
Tried the STDEBUG_CompactEverything=1 and it made no practical difference (still 164074 files, 2.9GB in the index directory). This still seems excessive since my main server with a significant superset of the files on this node has only 30 files and 2.0GB in the index directory. There are tons of very small .ldb files (~200 bytes - 3KB) in there. The only other significant (non .ldb) files are a 21MB MANIFEST and a 1.1MB log file.
Going to try -reset-deltas now.
Unfortunately, -reset-deltas does not seem to have made much difference when set as an environment variable. Still over 163k files in the index directory. Seems to be working for now, so will ignore this for a while and revisit if see any practical issues… Hopefully it will somehow work its way down over time.