8k+ ldb files, crashing in browser and scanning issues.

[drsprogrammer]

Hello Syncthing community,

I’ve been using Syncthing for ~4 months now, my filesystem contains ~10TB of data. Syncthing was working great, but has become extremely sluggish and unreliable as of late. My servers should be more than capable of handling anything Syncthing throws at it – 16 core / 32 thread xeon, 32gb RAM, OS on SSD, data on 15TB RAID-5.

1. The browser client crashes all the time. I’m not entirely sure why, but it appears to be correlated to the amount of data Syncthing is managing. It will slowly start consuming large amounts of RAM (up to 4gb for one window) and then crash and free up the RAM. Not sure if there’s a memory leak or what’s happening.
2. Eventually Syncthing will completely crash and continue to crash shortly after subsequent load attempts. This appears to be correlated to the number of LDB files I have. I’ve tried removing the DB files and re-scanning from scratch. What I noticed is that Syncthing generates a massive number of LDB (*.ldb) files. My server has upwords of 8,000 (was 15,000 before DB reset) that have persisted immediately after the 10TB of data was scanned. This amounts to about 16-30gb of LDB files.
3. Both locations I’m syncing are experiencing the same problems even though they are on different operating systems (one latest Ubuntu, other Win-10). Both have crashing issues in the browser and an extremely large number of LDB files.
4. Probably unrelated, but possibly not, scanning seems to be bugged. I tend to pause the folders as to minimize read/write head movement during my Scans. However, when I unpause it appears to resume my current scan and start a new scan, effectively creating two instances of scan against the same folder at the same time. I’ve noticed that the JavaScript will switch between each scan in the progress bar. To resolve this, I pause all folders, restart Syncthing then unpause them one at a time.

Syncthing has been a wonderful tool in the past and was a solid snyc application up until about 8TB of data where we saw it start to struggle.

I checked my .config files for errors and there doesn’t seem to be anything reporting outside of a couple of notices regarding UTF-8 encoded file names.

Any help on this would be greatly appreciated.

Thanks,

Dave

[calmh (Jakob Borg)]

Any logs from these crashes, and what version of Syncthing exactly?

There is a known issue around folders being paused or restarted in the middle of a scan - the scan is not interrupted, and restarting / unpausing the folder will then start another scan. You’re probably better off increasing the scan interval than manually twiddling them.

The database grows during updates (scanning, syncing) and compacts after a while. Perhaps this operation is taking some time due to the size. The minimum expected database size is something like 2.5 GB per 10 TB of data, per device, plus overhead for file metadata (name, permissions etc) which depends on how many files and folders you have, plus whatever internal overhead the database format has.

[drsprogrammer]

Hello Jakob,

I have a 77mb log in the index folder that appears to have logged the build process. (called LOG) But I think that just logs my indexing not sure that would be of any help. But outside in the .config/syncthing there’s no logs there – it’s my understanding that’s where they should be if it recorded an issue, I also looked in /var/log for anything, no luck there.

Also, I’m on v0.14.26, Linux (64 bit) “Dysprosium Dragonfly”.

Anything else I could provide that might help?

[calmh (Jakob Borg)]

The log is printed to standard output, so whether that is actually saved anywhere depends on how you installed. Typically, on Linux, systemd would “manage” the logs. On Windows I guess something like Synctrayzor takes care of it most of the time. If Syncthing crashed, there should be a panic-somethingsomething.log file in the config directory, though.

[drsprogrammer]

Hello Jakob,

It only took about a week after the DB rebuild for it to start crashing again. First the browser client started crashing (“Aw Snap”) in Google Chrome – this doesn’t produce a log file.

Now a couple days later the service itself is completely broken and it’s producing a log file every minute. Restarting SyncThing doesn’t help, it just continues to produce log files. I have 6,932 .ldb files. Below is the latest log file before I killed SyncThing processes (they all seems to be very similar and there are now hundreds of them).

I noticed the error had to do with corruption on a data-block. I checked my smartctl report for the SSD SyncThing lives on and it passed with no issues or errors.

Any help would be greatly appreciated!!! Thanks again for looking into this,

Dave

---

Panic at 2017-04-21T10:02:50-04:00 panic: leveldb/table: corruption on data-block (pos=1968858): checksum mismatch, want=0xf1d6bb3a got=0x1b8baf07 [file=423423.ldb]

goroutine 1 [running]: github.com/syncthing/syncthing/lib/db.(*Instance).checkGlobals(0xc42084ecc0, 0xc4207c28d0, 0xb, 0x10, 0xc4201f3db8) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/db/leveldb_dbinstance.go:600 +0x7b1 github.com/syncthing/syncthing/lib/db.NewFileSet(0xc420157834, 0xb, 0xc42084ecc0, 0xc42057ba40) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/db/set.go:125 +0x225 github.com/syncthing/syncthing/lib/model.(*Model).addFolderLocked(0xc4206a43c0, 0xc420157834, 0xb, 0xc420157858, 0x8, 0xc4204b7320, 0x14, 0x0, 0xc4201a8200, 0x2, …) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/model.go:313 +0xe5 github.com/syncthing/syncthing/lib/model.(*Model).AddFolder(0xc4206a43c0, 0xc420157834, 0xb, 0xc420157858, 0x8, 0xc4204b7320, 0x14, 0x0, 0xc4201a8200, 0x2, …) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/model.go:307 +0x94 main.syncthingMain(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb6e1b4, 0x1, 0x0, 0x0, …) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:820 +0x2cf1 main.main() /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:459 +0x5ac

goroutine 62 [select]: github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc4200c0000) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x5d9 created by github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.openDB /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x583

goroutine 7 [chan receive]: github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go.(*Emitter).emitTask(0x109dbc8) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go/emitter.go:37 +0x99 created by github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go.(*Emitter).init /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go/emitter.go:32 +0x6c

goroutine 8 [select]: github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go.(*updateHeap).updateTask(0x10a01e0) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go/updater.go:83 +0x4d7 created by github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go.init.4 /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/xtaci/kcp-go/updater.go:13 +0xb5

goroutine 9 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:116 +0x104 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.1 /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 17 [chan receive]: main.trackCPUUsage() /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/gui_unix.go:24 +0x103 created by main.init.3 /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/gui_unix.go:17 +0x35

goroutine 18 [select, locked to thread]: runtime.gopark(0xd13428, 0x0, 0xb71224, 0x6, 0x18, 0x2) /usr/local/go/src/runtime/proc.go:271 +0x13a runtime.selectgoImpl(0xc4201d0f50, 0x0, 0x18) /usr/local/go/src/runtime/select.go:423 +0x1364 runtime.selectgo(0xc4201d0f50) /usr/local/go/src/runtime/select.go:238 +0x1c runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:434 +0x2dd runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 19 [chan receive]: main.setupSignalHandling.func1(0xc4201aa420) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:974 +0x40 created by main.setupSignalHandling /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:976 +0xbc

goroutine 20 [chan receive]: main.setupSignalHandling.func2(0xc4201aa5a0) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:984 +0x40 created by main.setupSignalHandling /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:986 +0x181

goroutine 21 [select]: github.com/syncthing/syncthing/vendor/github.com/thejerf/suture.(*Supervisor).Serve(0xc4201e40e0) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/thejerf/suture/supervisor.go:338 +0xf65 created by github.com/syncthing/syncthing/vendor/github.com/thejerf/suture.(*Supervisor).ServeBackground /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/thejerf/suture/supervisor.go:297 +0x43

goroutine 22 [chan receive]: github.com/syncthing/syncthing/lib/events.(*bufferedSubscription).pollingLoop(0xc420196460) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/events/events.go:318 +0x83 created by github.com/syncthing/syncthing/lib/events.NewBufferedSubscription /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/events/events.go:313 +0x1ba

goroutine 23 [chan receive]: github.com/syncthing/syncthing/lib/events.(*bufferedSubscription).pollingLoop(0xc4201964b0) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/events/events.go:318 +0x83 created by github.com/syncthing/syncthing/lib/events.NewBufferedSubscription /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/events/events.go:313 +0x1ba

goroutine 65 [select]: github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc4201e4000) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x23f created by github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x19c

goroutine 63 [select]: github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc4200c0000) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x249 created by github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.openDB /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x5a5

goroutine 64 [select]: github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc4200c0000) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x638 created by github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.openDB /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x722

goroutine 81 [select]: github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc4200c0000) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x242 created by github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb.openDB /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x744

goroutine 13 [select]: github.com/syncthing/syncthing/lib/model.(*ProgressEmitter).Serve(0xc420852fa0) /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/progressemitter.go:56 +0x559 created by github.com/syncthing/syncthing/lib/model.NewModel /home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/model.go:163 +0x980

[AudriusButkevicius]

You either have bad ram, cpu, disk or power cuts or have some magic caching filesystem or filesystem driver that corrupts writes, there is no other explanation to this.