Syncthing deletes config.xml ?

Support
1

Hi,

This happened to me twice. I use syncthing for a few weeks, then I notice one of my nodes are not synchronizing anything. When I log in to the node’s syncthing, I find it completely empty: syncthing is “reset” to stock config.xml, no UI authentication, no folders. The node also appear “offline” to other nodes.

When this happened the first time, I thought it might be some mistake. Here are the logs:

root@REDACTED:~# cat /tmp/syncthing.log
[start] 2024/06/24 23:52:02 INFO: syncthing v1.27.1 "Gold Grasshopper" (go1.22.2 linux-arm64) openwrt@openwrt 2024-05-11 17:33:28 UTC [noupgrade, stnoupgrade]
[start] 2024/06/24 23:52:02 INFO: Generating ECDSA key and certificate for syncthing...
[start] 2024/06/24 23:52:02 INFO: We will skip creation of a default folder on first start
[start] 2024/06/24 23:52:02 INFO: Default config saved. Edit /path/.syncthing/config.xml to taste (with Syncthing stopped) or use the GUI
[start] 2024/06/24 23:52:02 INFO: Archiving a copy of old config file format at: /path/.syncthing/config.xml.v0
[F7O6R] 2024/06/24 23:52:02 INFO: My ID: F7O6REZ-JABQKBG-WF4R7B5-VVLOF6X-447NGW7-XQU54BB-SAUEGVP-YVDVYQQ
[F7O6R] 2024/06/24 23:52:03 INFO: Single thread SHA256 performance is 1021 MB/s using minio/sha256-simd (713 MB/s using crypto/sha256).
[F7O6R] 2024/06/24 23:52:03 INFO: Hashing performance is 253.47 MB/s
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 1...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 2...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 3...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 5...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 6...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 7...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 9...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 10...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 11...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 13...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 14...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 16...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 17...
[F7O6R] 2024/06/24 23:52:03 INFO: Running database migration 19...
[F7O6R] 2024/06/24 23:52:03 INFO: Compacting database after migration...
[F7O6R] 2024/06/24 23:52:03 INFO: Overall send rate is unlimited, receive rate is unlimited
[F7O6R] 2024/06/24 23:52:03 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[F7O6R] 2024/06/24 23:52:03 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[F7O6R] 2024/06/24 23:52:03 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[F7O6R] 2024/06/24 23:52:03 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[F7O6R] 2024/06/24 23:52:03 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[F7O6R] 2024/06/24 23:52:03 INFO: TCP listener ([::]:22000) starting
[F7O6R] 2024/06/24 23:52:03 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[F7O6R] 2024/06/24 23:52:03 INFO: QUIC listener ([::]:22000) starting
[F7O6R] 2024/06/24 23:52:03 INFO: Loading HTTPS certificate: open /path/.syncthing/https-cert.pem: no such file or directory
[F7O6R] 2024/06/24 23:52:03 INFO: Creating new HTTPS certificate
[F7O6R] 2024/06/24 23:52:03 INFO: Detected 0 NAT services
[F7O6R] 2024/06/24 23:52:03 INFO: GUI and API listening on [::]:8384
[F7O6R] 2024/06/24 23:52:03 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[F7O6R] 2024/06/24 23:52:03 INFO: My name is "REDACTED"
[F7O6R] 2024/06/24 23:52:03 WARNING: Syncthing should not run as a privileged or system user. Please consider using a normal user account.
[F7O6R] 2024/06/24 23:52:23 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
[F7O6R] 2024/06/24 23:52:23 INFO: listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: read udp [::1]:50613->[::1]:53: i/o timeout
[F7O6R] 2024/06/24 23:52:23 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[F7O6R] 2024/06/25 00:47:23 INFO: Joined relay relay://51.15.175.80:22067
[F7O6R] 2024/06/25 00:47:23 INFO: Joined relay relay://51.15.175.80:22067
[F7O6R] 2024/06/25 00:51:26 INFO: quic://0.0.0.0:22000 detected NAT type: Symmetric UDP firewall
[F7O6R] 2024/06/25 00:51:26 INFO: quic://0.0.0.0:22000 resolved external address quic://REDACTED:22000 (via stun.syncthing.net:3478)
[F7O6R] 2024/06/25 18:47:15 INFO: quic://0.0.0.0:22000 resolved external address quic://REDACTED:22000 (via stun.syncthing.net:3478)
[F7O6R] 2024/06/25 18:47:24 INFO: Joined relay relay://5.196.64.99:22067
[F7O6R] 2024/06/26 11:29:49 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
[F7O6R] 2024/06/26 11:29:49 INFO: QUIC listener ([::]:22000) shutting down
[F7O6R] 2024/06/26 11:29:50 INFO: TCP listener ([::]:22000) shutting down
[F7O6R] 2024/06/26 11:29:50 INFO: Exiting
[start] 2024/06/26 11:31:52 INFO: syncthing v1.27.1 "Gold Grasshopper" (go1.22.2 linux-arm64) openwrt@openwrt 2024-05-11 17:33:28 UTC [noupgrade, stnoupgrade]
[KPAHX] 2024/06/26 11:31:52 INFO: My ID: KPAHX6Y-7CUSAVN-WP2DUKY-VPZ42XX-4QKVRUB-54MA55M-4LEH4RW-47R3TAK
[KPAHX] 2024/06/26 11:31:53 INFO: Single thread SHA256 performance is 1045 MB/s using minio/sha256-simd (719 MB/s using crypto/sha256).
[KPAHX] 2024/06/26 11:31:53 INFO: Hashing performance is 254.41 MB/s
[KPAHX] 2024/06/26 11:31:53 INFO: Cleaning metadata for dropped folder "wqdnt-rapyr"
[KPAHX] 2024/06/26 11:32:06 INFO: Overall send rate is unlimited, receive rate is unlimited
[KPAHX] 2024/06/26 11:32:06 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[KPAHX] 2024/06/26 11:32:06 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[KPAHX] 2024/06/26 11:32:06 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[KPAHX] 2024/06/26 11:32:06 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[KPAHX] 2024/06/26 11:32:06 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[KPAHX] 2024/06/26 11:32:06 INFO: TCP listener ([::]:22000) starting
[KPAHX] 2024/06/26 11:32:06 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[KPAHX] 2024/06/26 11:32:06 INFO: QUIC listener ([::]:22000) starting
[KPAHX] 2024/06/26 11:32:06 INFO: GUI and API listening on [::]:8384
[KPAHX] 2024/06/26 11:32:06 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[KPAHX] 2024/06/26 11:32:06 INFO: My name is "REDACTED"
[KPAHX] 2024/06/26 11:32:06 INFO: Device F7O6REZ-JABQKBG-WF4R7B5-VVLOF6X-447NGW7-XQU54BB-SAUEGVP-YVDVYQQ is "REDACTED" at [dynamic]
[KPAHX] 2024/06/26 11:32:06 WARNING: Syncthing should not run as a privileged or system user. Please consider using a normal user account.
[KPAHX] 2024/06/26 11:32:16 INFO: quic://0.0.0.0:22000 detected NAT type: Symmetric UDP firewall
[KPAHX] 2024/06/26 11:32:16 INFO: quic://0.0.0.0:22000 resolved external address quic://REDACTED:22000 (via stun.syncthing.net:3478)
[KPAHX] 2024/06/26 11:32:16 INFO: Detected 1 NAT service
[KPAHX] 2024/06/26 11:33:05 INFO: Joined relay relay://129.151.224.182:22067

One line stood out

[start] 2024/06/24 23:52:02 INFO: Archiving a copy of old config file format at: /path/.syncthing/config.xml.v0

I didn’t find config.xml.v0. Was it deleted as well ?

syncthing version: syncthing v1.27.1 "Gold Grasshopper" (go1.22.2 linux-arm64) openwrt@openwrt 2024-05-11 17:33:28 UTC [noupgrade]

NOTE: when this happened previously, I was using v1.27.7 as far as I remember, downloaded from Github releases

2

I’m pretty sure Syncthing didn’t delete it’s config, database, certificates and all. I would suspect confusion about what the config/data dir is supposed to be, perhaps running as a different user.

3

Yeah I am skeptical about Syncthing doing it, but looking at the mtime for /path/.syncthing

root@host:/path/.syncthing# ls -al
drwx------    3 root     root          4096 Jun 26 13:32 .
drwxr-xr-x    9 root     root          4096 May 19 23:01 ..
-rw-r--r--    1 root     root           794 May 19 23:01 cert.pem
-rw-------    1 root     root          6529 Jun 26 13:19 config.xml
-rw-------    1 root     root           594 Jun 26 13:32 csrftokens.txt
-rw-r--r--    1 root     root           794 May 19 23:01 https-cert.pem
-rw-------    1 root     root           288 May 19 23:01 https-key.pem
drwxr-xr-x    2 root     root         12288 Jun 26 13:53 index-v0.14.0.db
-rw-------    1 root     root           288 May 19 23:01 key.pem
root@host:/path/.syncthing# du -sh index-v0.14.0.db/
201.0M  index-v0.14.0.db/
root@host:/path/.syncthing#

it appears config.xml was modified today. Database looks updated and ok. I am running Syncthing on my openWRT box and I barely touch it. Here’s /etc/config/syncthing if it may help

config syncthing 'syncthing'
        option enabled '1'
        option gui_address 'http://0.0.0.0:8384'
        option home '/path/.syncthing/'
        option nice '19'
        option macprocs '2'
        option user 'root'

        option logfile '/tmp/syncthing.log'
        option log_max_old_files 7
        option log_max_size 1048576

        option _no_default_folder '1'

        # # Extra settings
        # list _ 'verbose'

I am running Syncthing as root because I sync my NFS directory and Syncthing user needs permissions to sync file ownership

4

Okay, I just checked my other OpenWRT device and the same thing happened! I have a third device/node (running Debian) that is running ok.

Do you know where can I start debugging this ? Syncthing on both OpenWRT devices have their “home” path on mounted USB drives. On startup, I think USB drives might be available after a while. Can that be the cause ? I am clueless and frustrated

5

Unfortunately the redacted log entries don’t provide enough detail, but now that you mention a USB drive, yes…

Depending on if Syncthing is set to autostart, there definitely could be a race condition between when the USB drive is mounted and when Syncthing is started.

If Syncthing starts before your USB drive is mounted, you could’ve ended up with two sets of /path/.syncthing/config.xml.

Let’s assume that your USB drive’s mount point is /path/, where an example /etc/fstab entry might look like the following:

/dev/sdb1	/path	vfat	defaults 0 0

If your USB drive is properly mounted, the first time Syncthing is launched it creates a subdirectory plua a number of files including its XML config file, example output from the tree command:

/path/
└── .syncthing
    └── config.xml

2 directories, 1 file

But if the next time your router is restarted Syncthing launches before your USB drive is mounted, /path/ will be empty, resulting in Syncthing creating a brand new default configuration that looks almost exactly like the one on your USB drive but with different time stamps:

/path/
└── .syncthing
    └── config.xml

2 directories, 1 file

However, as soon as your USB is mounted, it’ll temporarily overlay/hide whatever is under /path/ – all the while Syncthing is still running. Because one or more of Syncthing’s files are already open, they’ll still be in use effectively behind-the-scenes while the files on your USB drive remain untouched (until you restart Syncthing without rebooting the router).

The quickest solution is to resolve the catch-22 with mounting your USB drive and starting Syncthing, which must occur in that specific order to meet your end goal.

6

Thanks for the detailed reply. I believe it’s some sort of a race condition as per your thoughts, because my devices restart often and this doesn’t happen for any restart. Syncthing is set to auto-start with old-school init scripts, which is suitable for low-power devices like routers.

AFAIK there is no dependency between init scripts, but I will check it and see. If I can get Syncthing to wait for USB mounts and this happened again, I will probably reply here again for any guidance. Thanks!

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.