v2.1.5: memory climbs to the cgroup limit and gets OOM-killed every 1–2 days (Docker, ~430k files)

Support
1

Hi,

I’m running Syncthing in Docker on a Synology NAS and it keeps hitting its memory limit. The kernel OOM killer takes out the syncthing process, the built-in monitor restarts it, and Docker never notices (container stays Up, RestartCount=0).

Environment

  • Syncthing v2.1.5 “Hafnium Hornet” (go1.27.1 linux-amd64), official image syncthing/syncthing:latest
  • Host: Synology DSM 7.4.1 (build 90080), kernel 4.4.302+, Ryzen R1600 (2c/4t), 32 GB RAM
  • Container: mem_limit / memswap_limit = 2 GiB (now 4 GiB), GOMEMLIMIT=1600MiB (now 3200MiB)
  • One remote device (macOS, v2.1.5), LAN only
  • index-v2 database: ~956 MB

Folders (7, all send-receive except one)

Folder Files Size
A 271,824 243 GB
B 90,436 98 GB
C 63,213 9.4 GB
D 5,188 5.7 GB
E 654 0.8 GB
F 246 2.6 GB
G 99 5.7 MB

~431k files / ~359 GB total. fs watcher enabled on all folders; rescan interval 120 s on two of them (NFS-backed, no inotify), 3600 s on the others.

What happens

Kernel log each time:

syncthing invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
mem_cgroup_oom_synchronize+0x2c2/0x2f0
Killed process 4189 (syncthing) total-vm:3812372kB, anon-rss:2073532kB, file-rss:0kB

Syncthing log right after:

INF Syncthing exited (error="signal: killed" log.pkg=main)

Nothing unusual is logged in the minutes before any of the kills. Anonymous RSS sits right at the limit every time.

The memory is not in the Go heap. Snapshot taken 18 h after the last restart, with the process close to the 4 GiB limit:

VmRSS            3,732,192 kB     (VmSwap 73,480 kB)
cgroup total_rss 3,813,568,512    total_cache 385,630,208
runtime Sys        227,264,824
HeapInuse          110,272,512    HeapIdle 97,280,000   HeapReleased 94,445,568
goroutines         142

So roughly 3.5 GB of anonymous memory lives outside the Go runtime, which is why GOMEMLIMIT has no effect. My guess is the SQLite index (the binary is static, so the SQLite allocator isn’t the Go heap), but I haven’t confirmed it.

/proc/<pid>/smaps of the syncthing process, 3 h after the last restart (RSS 1.96 GiB, HeapInuse 66 MB), anonymous mappings only:

mapping size    count   RSS
>= 64 MB            4   108 MB    <- the Go arenas
4-64 MB           108   412 MB
1-4 MB            602   843 MB
< 1 MB          3,981   633 MB
file-backed total        16 MB

~4,700 small anonymous mappings hold ~1.9 GB while the Go arenas hold ~108 MB. That looks like an allocator that mmaps on its own, outside the Go runtime. If the release binary uses the pure-Go SQLite (modernc), is its libc allocator the one growing here?

Kills

At 2 GiB, 10 kills in 9 days (UTC):

2026-09-19 15:40   2026-09-23 03:35   2026-09-23 14:40   2026-09-23 20:26
2026-09-24 06:31   2026-09-25 09:49   2026-09-25 12:53   2026-09-28 01:21
2026-09-28 09:53   2026-09-28 12:27

At 4 GiB (since 2026-09-28 13:31 UTC), 2 kills so far:

2026-09-28 17:43   2026-09-29 12:24

The second one came 15 minutes after the snapshot below (RSS 3.73 of 4 GiB). The cgroup’s memory.max_usage_in_bytes reads 4,298,772,480 — right at the limit.

Questions

  1. Is ~3.5 GB of non-Go memory expected for ~430k files with the v2 SQLite index (total DB size ~1 GB)? Is there a setting that bounds the SQLite cache / memory?
  2. If not, what would help you most? Attached: the snapshot above plus a heap profile from STPROFILER (/debug/pprof/heap) taken at the same moment, and the full smaps capture.

Thanks!

2

This seems unexpected. I have about half the amount of files, though more data, but my Syncthing uses 22 MiB RSS. You’re probably right about the non-Go allocations being SQLite. SQLite cache memory is capped at 2 MiB per connection, with four persistent connections and a maximum under busy times of 16 or 4*NumCPU, whichever is higher. You’d have to have a lot of CPUs, and reason for Syncthing to use a lot of database connections, for that to be cache memory though…

If the release binary uses the pure-Go SQLite (modernc), is its libc allocator the one growing here?

I don’t know, but our binaries (and hence the container) use the embedded C SQLite.

3

Thanks — that corrects my guess: the binary does contain go-sqlite3, so it’s the embedded C SQLite, statically linked.

Some numbers against your cache math: v2.1 here keeps 8 databases (main + one per folder), 23 open DB handles right now, 15 threads, NumCPU=4. Even at the burst maximum that’s ~256 MiB of cache, nowhere near 3.5 GB.

New data point: a third kill at 4 GiB this morning. Growth isn’t linear; it comes in bursts: RSS went 3186 → 3729 MiB in under 3 minutes, then the process was killed 20 min later.

Something I hadn’t noticed: the remote device reaches this container over two paths (outbound to its LAN IP, classified as WAN because of Docker bridge networking, and inbound via the bridge gateway), and they keep replacing each other. That’s 112 connections in 42 h, with “Abandoning old index handler in favour of new connection” among them. Could repeated index-handler restarts allocate in SQLite/cgo without releasing it? I’ll try to remove the dual path and report whether the bursts stop.

4

Follow-up on the network_mode: host test, now 9 days in.

What changed: the container runs with host networking since 2026-09-30 07:35 UTC. The Mac now connects once, as a LAN tcp-client; the dual path (WAN-classified outbound + inbound via the bridge gateway) is gone. Since then: 0 connections via the bridge, 0 “Abandoning old index handler” from the Mac. Two other changes in the meantime, for completeness: a second remote device (a laptop on Wi-Fi) joined on 2026-10-02, and the image moved to v2.1.6.

Result: the first 48 h were clean, and the process stayed under 3 GiB for ~6 days (my watcher captures smaps at 3 GiB and never fired). But it was killed again at the 4 GiB limit on 2026-10-08 04:07 UTC, ~19 h after a container restart, with no connection activity at all in the 2 h 20 before (three stable connections to the Mac, nothing logged but UPnP warnings). So the dual path was a side issue, not the cause.

Right before the kill:

03:57:55 UTC   RSS 3315 MiB   31,044 anonymous mappings
03:58:18 UTC   RSS 3621 MiB   33,110 anonymous mappings   (+306 MiB, +2,066 mappings in 23 s)
04:07:04 UTC   killed
Go heap profile at 03:58:18: inuse_space 36.9 MiB

So ~99 % of the resident memory is outside the Go heap. It’s almost entirely anonymous rw-p mappings, mostly 64 KiB to 320 KiB each (64 KiB multiples), ~3.5 GiB in total, ~45 % of them fully resident. Interesting difference from the 2026-09-30 kill: then it was 6,919 mappings for 3.7 GiB, now 33k smaller ones, so it’s not one allocation pattern repeated.

Given the binary embeds the C SQLite, my questions:

  1. Which allocator does the C side use in the release binary? Thousands of 64 KiB-multiple anonymous mappings appearing in seconds looks like an mmap-per-allocation malloc rather than SQLite’s page cache.
  2. Is there a way to see SQLite’s own accounting at runtime (sqlite3_memory_used / sqlite3_status), e.g. via a debug endpoint or STTRACE facility, so I can tell whether this memory is SQLite’s or something else’s before the next kill?

I have the smaps captures (both kills) and the Go heap profiles if useful.

Thanks!

5

Did you try 2.1.7-rc.1 which has a possibly related fix?