Since upgrading Syncthing to V0.12 I have a number of servers that are experiencing page allocation failures quite frequently. The V0.13 series has not improved the situation, and it may have even made it worse.
I understand that these messages are more informational than serious warnings, but I’d like to raise this issues anyhow. There ought to be some guidance on how to prevent this.
The server that has the problem the worst is as follows:
- HP Proliant DL360p G8 E5-2609V2 , 8GB RAM
- Debian Wheezy
- Linux backup-co 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u2 x86_64 GNU/Linux
- Syncthing V0.13.10
I’ve already tried setting vm.vfs_cache_pressure=500 with no improvement.
I’ve created a heap profile file using STHEAPPROFILE if that will help with diagnosis: https://gist.github.com/wsgcsysadmin/d83e88197b2fca9ce3cb5ca3072ec6a2
$ free
total used free shared buffers cached
Mem: 8158712 8031372 127340 0 1082240 5145716
-/+ buffers/cache: 1803416 6355296
Swap: 7812604 2908 7809696
$ dmesg|grep order
[87824.596300] swapper/0: page allocation failure: order:3, mode:0x20
[87856.503314] syncthing: page allocation failure: order:3, mode:0x20
[87862.130270] swapper/0: page allocation failure: order:4, mode:0x20
[87917.136316] syncthing: page allocation failure: order:3, mode:0x20
[87926.352179] syncthing: page allocation failure: order:3, mode:0x20
[87972.902113] swapper/0: page allocation failure: order:4, mode:0x20
[87991.637362] swapper/0: page allocation failure: order:3, mode:0x20
[88043.779886] swapper/0: page allocation failure: order:4, mode:0x20
[88145.498062] syncthing: page allocation failure: order:3, mode:0x20
[88176.452043] swapper/0: page allocation failure: order:4, mode:0x20
[88232.373329] swapper/0: page allocation failure: order:4, mode:0x20
[88396.402128] swapper/0: page allocation failure: order:4, mode:0x20
[88418.135491] syncthing: page allocation failure: order:3, mode:0x20
[88475.634021] syncthing: page allocation failure: order:3, mode:0x20
[88518.636470] syncthing: page allocation failure: order:3, mode:0x20
[88530.557792] syncthing: page allocation failure: order:4, mode:0x20
[88535.123246] syncthing: page allocation failure: order:2, mode:0x20
[88549.576884] swapper/0: page allocation failure: order:4, mode:0x20
[88767.139566] syncthing: page allocation failure: order:4, mode:0x20
Full dmesg output: https://gist.github.com/wsgcsysadmin/70efa3a92422e06e61977c5a2553dc90
/proc/slabinfo from the exact moment a page allocation failure happened: https://gist.github.com/wsgcsysadmin/fc80ff3f1c065802960bb689fbb081c4
$ curl … https://backup-co:8384/rest/svc/report {“announce”:{“defaultServersDNS”:1,“defaultServersIP”:0,“globalEnabled”:true,“localEnabled”:true,“otherServers”:0},“deviceUses”:{“compressAlways”:0,“compressMetadata”:8,“compressNever”:0,“customCertName”:0,“dynamicAddr”:3,“introducer”:0,“staticAddr”:5},“folderMaxFiles”:168939,“folderMaxMiB”:760158,“folderUses”:{“autoNormalize”:9,“externalVersioning”:0,“ignoreDelete”:0,“ignorePerms”:0,“readonly”:0,“simpleVersioning”:3,“staggeredVersioning”:0,“trashcanVersioning”:0},“longVersion”:“syncthing v0.13.10 “Copper Cockroach” (go1.6.2 linux-amd64) unknown-user@lx-buildslave.syncthing.net 2016-07-03 11:29:32 UTC”,“memorySize”:7967,“memoryUsageMiB”:814,“numCPU”:4,“numDevices”:8,“numFolders”:9,“platform”:“linux-amd64”,“relays”:{“defaultServers”:1,“enabled”:true,“otherServers”:0},“rescanIntvs”:[600,3600,3600,3600,3600,3600,3600,7200,7200],“sha256Perf”:138.26,“totFiles”:554903,“totMiB”:1040560,“uniqueID”:“xfchdkiw”,“upgradeAllowedAuto”:false,“upgradeAllowedManual”:false,“urVersion”:2,“usesRateLimit”:true,“version”:“v0.13.10”}