Hi there! I’ve just started using Syncthing about a week ago. Since two days ago, every time I come in the office the next morning, I find that Syncthing on our “main server” has stalled. The GUI loads up slow all the shares show Unknown and all the other machines are disconnected.
Just some background on my setup.
- Syncthing version on 0.14.7
- Syncing about 5 machines
- Roughly 110k files and 1TB data
- “Server” is a Hackintosh. All files stored on software mirror RAID.
- 4 machines on OSX 10.10
- 1 machine (the “server”) on OSX 10.6
- Syncing over LAN as well as Internet
- Port forwarding done
I’d love to provide more information but I have no idea where to look or where to pull the information out from. Any help would be appreciated!
This sounds like it’s deadlocking for some reason. Easiest way to debug it is to run it manually and capture the output, as I’m not sure how you’re starting it today.
Stop Syncthing from running using your current method
Open a new terminal, run
syncthing -no-restart. Leave it like that.
When this happens again, open another terminal and run
pkill -QUIT syncthing. It will exit with lots of trace data.
Paste that data somewhere, here will do in a pinch.
I think I’m seeing the same issue. I’m running on Ubuntu 14.04 systems. I’ll try what you mentioned here to generate the trace data.
You can also set the environment variable
STDEADLOCKTIMEOUT=600 and otherwise run it as usual. It should then panic when detecting a deadlock (after ten minutes), producing a nice panic file with all the relevant data in the configuration directory.
So, it finally stalled, but when I run pkill I get a -bash: pkill: command not found - it turns out 10.7 doesn’t have pkill yet.
I did a regular kill but it did not exit with any trace data. I started syncthing again just to let stuff sync, so it’ll be awhile before I get another chance to kill it.
Any alternative ideas to pkill would be welcome!
I managed to figure out how to set the STDEADLOCKTIMEOUT=600. I’ll upload the panic file when I have it.
you need to
kill -QUIT not just kill, that should produce the same effect.
It finally died sometime today. I suspect when one of the remote points came online. These were the last few lines. Are these the lines you guys need to troubleshoot or is there a panic file somewhere that I need to upload?
panic: deadlock detected
goroutine 42 [running]:
github.com/syncthing/syncthing/lib/model.deadlockDetect.func1(0x8bb2c97000, 0x8420f00208, 0x420fce200)
created by github.com/syncthing/syncthing/lib/model.deadlockDetect
Check if there is a panic file in syncthing’s home dir with more content or more content in the crash in general.
I’ve looked at the following locations, but I don’t see anything panic file or any file that looks like a log of a panic.
- ~/Library/Application Support/Syncthing
I should mention that this was the app running with STDEADLOCKTIMEOUT=600. I don’t know if that affects the existence of a panic log.
syncthing -help and
Also make sure syncthing is not running via service manager.
That output, but we need all of it. There should be lots.
You do know go 1.7 defaults dumping only the routine panicing right?
No, but I do now. You’d think that might have been mentioned somewhere in the release notes… If you knew, why didn’t you fix it already? This renders our panic traces useless. fixing…
So I’ve read it, thought it will be good for most cases, apart from the deadlock detection, and never got to fixing that case… Or raising a ticket…
Can you raise a bug with this.
Which category does that belong to? Development?
Bugs in github do not have categories I think.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.