Many small LDB files with database tuning

Syncthing v1.3.0 has introduced a database tuning enhancement. This is not a fast machine (dual core and spinning HDD) so I’m used to slow scan times - but after 5 hours Syncthing still has not finished scanning.

I noticed that there are 1000s of very small .ldb files (~ 300 bytes) and every second 20 or more are created. I don’t know how this .ldb thing works and if these files are just temporary.

The folder that is currently scanned has lots of small files, maybe this is the reason? Or is it just slow during first update because all .ldb are converted to bigger file structure?

There is no difference for small databases, and no conversion at startup, it just affects future writes. If it’s using the large database parameters it says something about it at startup. Does it?

This sounds bizarre regardless…

ok, thanks, I will restart Syncthing.
Yes, it says INFO: Using large-database tuning

That’s very strange, and literally the opposite of the intended effect of that tuning. You should see fewer, larger files, created less often.

jb@kvin:~/L/A/S/index-v0.14.0.db $ ls -l
total 478664
-rw-r--r--  1 jb  staff   2933107 Oct  1 07:58 118717.ldb
-rw-r--r--  1 jb  staff   1729770 Oct  1 14:18 118718.log
-rw-r--r--  1 jb  staff  67180822 Oct  1 07:58 118720.ldb
-rw-r--r--  1 jb  staff  67158257 Oct  1 07:58 118721.ldb
-rw-r--r--  1 jb  staff  67196757 Oct  1 07:58 118722.ldb
-rw-r--r--  1 jb  staff  36853641 Oct  1 07:58 118723.ldb
-rw-r--r--  1 jb  staff        16 Oct  1 07:58 CURRENT
-rw-r--r--  1 jb  staff        16 Oct  1 07:58 CURRENT.bak
-rw-r--r--  1 jb  staff        45 Aug 19 19:13 LARGE
-rw-r--r--  1 jb  staff         0 Dec 26  2016 LOCK
-rw-r--r--  1 jb  staff     69962 Oct  1 07:58 LOG
-rw-r--r--  1 jb  staff   1048730 Aug 19 13:36 LOG.old
-rw-r--r--  1 jb  staff      1708 Oct  1 07:58 MANIFEST-118719
jb@kvin:~/L/A/S/index-v0.14.0.db $

Paste the last 50 or so lines of LOG?

That was my understanding and there are also a few larger .ldb with todays date. It seems the restart has removed the small .ldb, there are now 11k instead of 300k but the problem persists, the small .ldb are still created like horny :rabbit2:

LOG output

14:27:23.587139 table@build created L3@1055833 N·1 S·350B "\x00\x00\x00..txt,v428376476":"\x00\x00\x00..txt,v428376476"
14:27:23.637457 table@build created L3@1055834 N·1 S·353B "\x00\x00\x00..txt,v398374788":"\x00\x00\x00..txt,v398374788"
14:27:23.687916 table@build created L3@1055835 N·1 S·355B "\x00\x00\x00..txt,v426707640":"\x00\x00\x00..txt,v426707640"
14:27:23.771664 table@build created L3@1055836 N·1 S·349B "\x00\x00\x00..txt,v424815321":"\x00\x00\x00..txt,v424815321"
14:27:23.838674 table@build created L3@1055837 N·1 S·351B "\x00\x00\x00..txt,v297520510":"\x00\x00\x00..txt,v297520510"
14:27:23.905662 table@build created L3@1055838 N·1 S·363B "\x00\x00\x00..txt,v455091780":"\x00\x00\x00..txt,v455091780"
14:27:23.956026 table@build created L3@1055839 N·1 S·345B "\x00\x00\x00..txt,v297715104":"\x00\x00\x00..txt,v297715104"
14:27:24.006424 table@build created L3@1055840 N·1 S·345B "\x00\x00\x00..txt,v297998622":"\x00\x00\x00..txt,v297998622"
14:27:24.056793 table@build created L3@1055841 N·1 S·345B "\x00\x00\x00..txt,v297908978":"\x00\x00\x00..txt,v297908978"
14:27:24.115488 table@build created L3@1055842 N·1 S·349B "\x00\x00\x00..txt,v297722261":"\x00\x00\x00..txt,v297722261"
14:27:24.174131 table@build created L3@1055843 N·1 S·347B "\x00\x00\x00..txt,v297520706":"\x00\x00\x00..txt,v297520706"
14:27:24.232794 table@build created L3@1055844 N·1 S·370B "\x00\x00\x00..txt,v331766050":"\x00\x00\x00..txt,v331766050"
14:27:24.291559 table@build created L3@1055845 N·1 S·345B "\x00\x00\x00..txt,v297790218":"\x00\x00\x00..txt,v297790218"
14:27:24.341843 table@build created L3@1055846 N·1 S·349B "\x00\x00\x00..txt,v428364846":"\x00\x00\x00..txt,v428364846"
14:27:24.393367 table@build created L3@1055847 N·1 S·346B "\x00\x00\x00..txt,v297805910":"\x00\x00\x00..txt,v297805910"
14:27:24.452125 table@build created L3@1055848 N·1 S·350B "\x00\x00\x00..txt,v428362914":"\x00\x00\x00..txt,v428362914"
14:27:24.502454 table@build created L3@1055849 N·1 S·345B "\x00\x00\x00..txt,v297831307":"\x00\x00\x00..txt,v297831307"
14:27:24.561145 table@build created L3@1055850 N·1 S·349B "\x00\x00\x00..txt,v297669543":"\x00\x00\x00..txt,v297669543"
14:27:24.620008 table@build created L3@1055851 N·1 S·341B "\x00\x00\x00..txt,v297919018":"\x00\x00\x00..txt,v297919018"
14:27:24.678530 table@build created L3@1055852 N·1 S·341B "\x00\x00\x00..txt,v297740926":"\x00\x00\x00..txt,v297740926"

every second ~15 new lines like above are appended to LOG

The currently scanning folder has ~1.5 million files (.txt, .php, .gz) and 61 GB total size = average file size of 40 KB.

Can you test this somehow?

So it’s creating a file for every KV item. What platform is this? Any magic env vars set? (All of the database stuff can be overridden that way, but the variables aren’t documented and you shouldn’t use them…)

By the way I’d stop Syncthing, force it to the “small” setting to see if that stops the lunacy. As it’s going, it’s going to toast your database totally.

Linux 32 bit, no env vars.
I will give it one hour and if this does not stop I will revert to “small” manually.

I have no idea what’s going on. The 32 bit thing is an outlier and could be a cause of some overflow, but 32 bit Mac works for me at least and the numbers in the config struct aren’t that large. Maybe there is something else in the leveldb layer that’s broken for large-db && 32bit…

There is one thing you could try if you like. If you start up with STDEBUG_CompactEverything=1 it will, before starting to do other stuff, do a full compaction pass of the database. If the large db parameters are somehow broken, doing this with the small setting should repair things down to normal state. If there’s something else odd going on this may get it to clean up things to the large-db expected state. It may take some time and your only progress indicator will be by looking in LOG.

Thanks for testing. Maybe there is a good ol’ integer somewhere?
Can you also try with a large folder with 1.5 m files?

I run now with env STDEBUG_CompactEverything=1 ./syncthing and watch LOG
maybe this is interesting:

15:41:58.891189 table@remove removed @1044533
15:41:58.894188 table@remove removed @1044534
15:41:58.901725 table@remove removed @1044535
15:42:02.355143 table@build created L2@1091300 N·453118 S·64MiB "\x00\x00\x00..URL,v713928887":"\x00\x00\x00...gz,v705541461"
15:42:07.280520 table@build created L2@1091301 N·923488 S·64MiB "\x00\x00\x00...gz,v705542640":"\x00\x00\x00.._DE,v715853083"
15:42:11.796808 table@build created L2@1091302 N·1655252 S·49MiB "\x00\x00\x00..son,v716229099":"\x00\x00\x00..son,v695439092"
15:42:11.797043 version@stat F·[0 9 14 882] S·-1593157276B[0B 234MiB 737MiB 1GiB] Sc·[0.00 2.34 0.74 0.16]
15:42:11.880176 table@compaction committed F-1 S-14MiB Ke·0 D·904940 T·13.315443816s
15:42:11.880250 table@compaction limiting F·14 -> F·2
15:42:11.880664 table@compaction L2·2 -> L3·565 S·1GiB Q·721026823
15:42:11.893969 table@remove removed @1091289
15:42:11.898831 table@remove removed @1091290
15:42:11.926856 table@remove removed @1044551
15:42:11.958522 table@remove removed @1044553
15:42:12.028744 table@build created L3@1091303 N·1 S·265B "\x00\x00\x00..URL,v713928887":"\x00\x00\x00..URL,v713928887"
15:42:12.081599 table@build created L3@1091304 N·1 S·261B "\x00\x00\x00..URL,v713928889":"\x00\x00\x00..URL,v713928889"
15:42:12.140295 table@build created L3@1091305 N·1 S·220B "\x00\x00\x00..URL,v713928891":"\x00\x00\x00..URL,v713928891"
15:42:12.199001 table@build created L3@1091306 N·1 S·173B "\x00\x00\x00..txt,v713935896":"\x00\x00\x00..txt,v713935896"
15:42:12.257880 table@build created L3@1091307 N·1 S·221B "\x00\x00\x00..txt,v713934980":"\x00\x00\x00..txt,v713934980"

after this section the small files continue…

These are fine:

But:

Negative size looks bad. I wonder if there’s a 32 bit int in leveldb, and your database looks like it’s just a hair larger than 2 GiB…

And this is broken (I think):

(No need for an L3 file to be that small.) So I think this is probably not related to the new tuning, rather some internal brokenness.

Consider using a 64 bit build…?

size of all ldb currently is ~2.8 GB, afaik before update it was around the same size.
Startup is still ongoing :face_with_raised_eyebrow:

on my long todo list, until then I’m happy about other ideas and suggestions.

What I don’t understand is what changed in this version that could cause this. Startup in the past (and initial scan after update) has always been slow but never this long. Files also have not changed since yesterday.

I also don’t know. Perhaps it’s something with the new parameters plus large database plus 32 bits that causes an overflow somewhere.

If your hardware is actually just 32 bit then clearly you have no choice. Otherwise all you need is a 64 bit libc I think and you should be good to use the 64 bit version.

You make it sound so easy but I’m not a linux pro.

Yesterday at night I restarted again (without env var) to see what happens. According to startup log scan took hours (expected because of many small files), but the creation of the byte sized .ldb continued through the night. In the morning I found 500k (!) files in the index-db folder.
I also noticed that the file creation continues although all folders are scanned and Syncthing is idle!
Next try is with manual “small” setting today, I strongly hope this works.

result: thankfully “small” setting works :relieved:
1.2k .ldb files (larger than just a few bytes), total db size back to ~2.7 GB, scan time also a significantly faster. And no .ldb files are created once scanning has finished

Same thing happened on my raspberry pi (those things run 32 bit versions). Database was over 2GB autoset for large tuning. When I stopped db was 7GB and more than 300000 ldb files. On my other device that syncs same folder, but runs 64 bit version the db folder is 2.2GB and only 500 ldb files

@MikusR thanks for reporting. I think many 32bit devices could be filling up disk space right now. It’s a slow process because of high I/O of byte sized files.
Can you please try setting the field “Database Tuning” (advanced settings) to small and see if this it fixes it for you too.

I did that and yes that fixes it.

1 Like