panic: deadlock detected at fmut

Samuel_Bruyere · August 13, 2018, 11:43am

Hi, My syncthing server crash every day

Panic file here :

[5FPON] 12:47:17 INFO: syncthing v0.14.50-rc.1 "Dysprosium Dragonfly" (go1.10.3 windows-amd64) teamcity@build.syncthing.net 2018-08-06 19:44:53 UTC
[5FPON] 12:47:17 INFO: My ID: /deleted for confidentiality/
[5FPON] 12:47:18 INFO: Single thread SHA256 performance is 255 MB/s using minio/sha256-simd (173 MB/s using crypto/sha256).
[5FPON] 12:47:18 INFO: Hashing performance is 220.56 MB/s
[5FPON] 12:47:24 INFO: Starting deadlock detector with 20m0s timeout
[5FPON] 12:47:24 INFO: Ready to synchronize " /deleted for confidentiality/ (sendreceive)
[5FPON] 12:47:24 INFO: Overall send rate is unlimited, receive rate is unlimited
[5FPON] 12:47:24 INFO: Rate limits do not apply to LAN connections
[5FPON] 12:47:24 INFO: Using discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[5FPON] 12:47:24 INFO: Using discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[5FPON] 12:47:24 INFO: Using discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW

 /deleted for confidentiality/

[5FPON] 13:00:56 INFO: Connection to /deleted for confidentiality/ /tcp-server closed: reading length: read tcp  /deleted for confidentiality/:22000-> /deleted for confidentiality/:51724: wsarecv: Une tentative de connexion a échoué car le parti connecté n’a pas répondu convenablement au-delà d’une certaine durée ou une connexion établie a échoué car l’hôte de connexion n’a pas répondu.
Panic at 2018-08-13T13:12:24+02:00
panic: deadlock detected at fmut

goroutine 81 [running]:
github.com/syncthing/syncthing/lib/model.(*deadlockDetector).Watch.func1(0xc04214e680, 0x132e2f0, 0xc042208860, 0xb608ad, 0x4)
	C:/BuildAgent/work/src/github.com/syncthing/syncthing/lib/model/util.go:57 +0x3ad
created by github.com/syncthing/syncthing/lib/model.(*deadlockDetector).Watch
	C:/BuildAgent/work/src/github.com/syncthing/syncthing/lib/model/util.go:33 +0xbe

imsodin · August 13, 2018, 12:11pm

There should be more lines like the goroutine 81 [running:] and following ones - please post them. They contain the relevant information (and nothing confidential).

Samuel_Bruyere · August 13, 2018, 2:05pm

panic-20180813-131224.log (189.7 KB)

LOg in attachement.

Regards. Sam

imsodin · August 13, 2018, 3:08pm

I just see a whole lot of routines trying to acquire fmut, but not a single one that has it (I might have just missed it, but I don’t think so). Maybe it’s a false positive, but even then, shouldn’t there be a routine holding it? I might be blind @AudriusButkevicius @calmh

calmh · August 13, 2018, 3:55pm

Yeah I don’t see it either. One possibility is that someone held the lock and then (silently, recover()ed) panic:ed…

Samuel_Bruyere · August 13, 2018, 3:57pm

If that can help, I’ve the interface but all share with unknow status and update/download to 0 Kb/s

Thanks Sam

calmh · August 13, 2018, 4:02pm

Can you try this build:

https://build.syncthing.net/viewLog.html?buildId=25315&buildTypeId=Syncthing_BuildWindows&tab=artifacts

It just adds warning level logging for some things that might otherwise sneak silently past

imsodin · August 13, 2018, 4:16pm

This, so much:

calmh · August 13, 2018, 6:03pm

I have that fixed, it just won’t be adopted upstream and I’ve been hesitant to do the fork.

imsodin · August 13, 2018, 6:06pm

What, for real? I mean assuming it’s a non-default option in Spec I see no down-side. Given that it doesn’t change much, I’d totally be in favor of forking if they really don’t want it.

calmh · August 13, 2018, 6:09pm

https://github.com/calmh/suture/commit/fbfdc4ee9f367f14eb901906d906777378d03834

It’s a boolean and an if.

Issue literally says I can live with forking it, so I guess should just get my ass in gear.

Samuel_Bruyere · August 14, 2018, 7:07am

HI all, I’ve download syncthing from calmh and trying.

This morning, syncthing is down :

calmh · August 14, 2018, 7:09am

It wasn’t expected to fix the problem, but to give a better panic trace when it comes.

Samuel_Bruyere · August 14, 2018, 7:11am

Panda kill syncthing , looking to exclusion parameters

Samuel_Bruyere · August 14, 2018, 7:33am

Syncthing starting, waiting for result.

(email to panda support for false detection)

Tks Sam

Samuel_Bruyere · August 14, 2018, 8:26am

Syncthig create panic log , but is wotking : panic-20180814-100515.log (1.1 MB)

calmh · August 14, 2018, 8:38am

This one is different.

fatal error: out of memory

You should, first of all, use the 64 bit version unless you absolutely cannot for some reason. (If you can’t run the 64 bit version we probably can’t do much more here.)

imsodin · August 14, 2018, 8:52am

Note to self: Always as very first measure check the platform and ask about AV if it’s windows - do not spend a second on the problem before that’s cleared!

Btw: I am not complaining about the report, I am annoyed by me not thinking of AV even though it’s the enemy no1 on windows - and felt like venting

Samuel_Bruyere · August 14, 2018, 9:39am

HI, sorry for the mistake.

My local computer have Panda and killed file when download , not the server. Server is on Windows 2012 R2 no AV no Fw.

(download & run 64 bits syncthing) Tks Sam

Samuel_Bruyere · August 14, 2018, 8:51pm

panic log here : panic-20180814-212134.log (197.5 KB)