Page allocation failures. Syncthing causing memory fragmentation?

Since upgrading Syncthing to V0.12 I have a number of servers that are experiencing page allocation failures quite frequently. The V0.13 series has not improved the situation, and it may have even made it worse.

I understand that these messages are more informational than serious warnings, but I’d like to raise this issues anyhow. There ought to be some guidance on how to prevent this.

The server that has the problem the worst is as follows:

  • HP Proliant DL360p G8 E5-2609V2 , 8GB RAM
  • Debian Wheezy
  • Linux backup-co 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u2 x86_64 GNU/Linux
  • Syncthing V0.13.10

I’ve already tried setting vm.vfs_cache_pressure=500 with no improvement.

I’ve created a heap profile file using STHEAPPROFILE if that will help with diagnosis: https://gist.github.com/wsgcsysadmin/d83e88197b2fca9ce3cb5ca3072ec6a2

$ free
             total       used       free     shared    buffers     cached
Mem:       8158712    8031372     127340          0    1082240    5145716
-/+ buffers/cache:    1803416    6355296
Swap:      7812604       2908    7809696


$ dmesg|grep order
[87824.596300] swapper/0: page allocation failure: order:3, mode:0x20
[87856.503314] syncthing: page allocation failure: order:3, mode:0x20
[87862.130270] swapper/0: page allocation failure: order:4, mode:0x20
[87917.136316] syncthing: page allocation failure: order:3, mode:0x20
[87926.352179] syncthing: page allocation failure: order:3, mode:0x20
[87972.902113] swapper/0: page allocation failure: order:4, mode:0x20
[87991.637362] swapper/0: page allocation failure: order:3, mode:0x20
[88043.779886] swapper/0: page allocation failure: order:4, mode:0x20
[88145.498062] syncthing: page allocation failure: order:3, mode:0x20
[88176.452043] swapper/0: page allocation failure: order:4, mode:0x20
[88232.373329] swapper/0: page allocation failure: order:4, mode:0x20
[88396.402128] swapper/0: page allocation failure: order:4, mode:0x20
[88418.135491] syncthing: page allocation failure: order:3, mode:0x20
[88475.634021] syncthing: page allocation failure: order:3, mode:0x20
[88518.636470] syncthing: page allocation failure: order:3, mode:0x20
[88530.557792] syncthing: page allocation failure: order:4, mode:0x20
[88535.123246] syncthing: page allocation failure: order:2, mode:0x20
[88549.576884] swapper/0: page allocation failure: order:4, mode:0x20
[88767.139566] syncthing: page allocation failure: order:4, mode:0x20

Full dmesg output: https://gist.github.com/wsgcsysadmin/70efa3a92422e06e61977c5a2553dc90

/proc/slabinfo from the exact moment a page allocation failure happened: https://gist.github.com/wsgcsysadmin/fc80ff3f1c065802960bb689fbb081c4

$ curl … https://backup-co:8384/rest/svc/report {“announce”:{“defaultServersDNS”:1,“defaultServersIP”:0,“globalEnabled”:true,“localEnabled”:true,“otherServers”:0},“deviceUses”:{“compressAlways”:0,“compressMetadata”:8,“compressNever”:0,“customCertName”:0,“dynamicAddr”:3,“introducer”:0,“staticAddr”:5},“folderMaxFiles”:168939,“folderMaxMiB”:760158,“folderUses”:{“autoNormalize”:9,“externalVersioning”:0,“ignoreDelete”:0,“ignorePerms”:0,“readonly”:0,“simpleVersioning”:3,“staggeredVersioning”:0,“trashcanVersioning”:0},“longVersion”:“syncthing v0.13.10 “Copper Cockroach” (go1.6.2 linux-amd64) unknown-user@lx-buildslave.syncthing.net 2016-07-03 11:29:32 UTC”,“memorySize”:7967,“memoryUsageMiB”:814,“numCPU”:4,“numDevices”:8,“numFolders”:9,“platform”:“linux-amd64”,“relays”:{“defaultServers”:1,“enabled”:true,“otherServers”:0},“rescanIntvs”:[600,3600,3600,3600,3600,3600,3600,7200,7200],“sha256Perf”:138.26,“totFiles”:554903,“totMiB”:1040560,“uniqueID”:“xfchdkiw”,“upgradeAllowedAuto”:false,“upgradeAllowedManual”:false,“urVersion”:2,“usesRateLimit”:true,“version”:“v0.13.10”}

That’s above my pay grade on Linux memory internals so I can’t say much right off the bat. Does Syncthing also crash at that point due to not being able to allocate memory? While googling on it there is a thread on the Go issue tracker about a similar issue, but closed way back.

In any case we do not manage memory manually as this is abstracted away by Go, so the right place to report, reproduce and fix this issue (whatever it is) would be GitHub.com/golang/go. It may be that Syncthing is the only longer running Go process on your systems.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.