Syncthing on SQLite -- help test!

calmh · April 13, 2025, 1:17pm

marbens · April 13, 2025, 4:53pm

Does a new database layer (or any of the changes in the v2 branch) even warrant a major version bump?

According to Versions & Releases: Major, Minor, or Patch:

We decide the version number for a new release based on the following criteria:

Is the new version protocol incompatible with the previous one, so that they cannot connect to each other or otherwise can’t sync files for some reason? That’s a new major version. (This hasn’t happened yet.)

Are there changes in the REST API so that integrations or wrappers need changes, or did the database schema or configuration change so that a downgrade might be problematic? That’s a new minor version.

If there are no specific concerns as above, it’s a new patch version.

AFAIK you can sync v2 devices with v1 devices, so it’s seems to me like it shouldn’t be a new major version, or the docs are out of date.

Nummer378 · April 13, 2025, 6:55pm

Please see this post for a reasoning on why this is v2:

Syncthing SQLite rollout

However, I think we should release Syncthing-with-SQLite as Syncthing 2.0. The reason for this is more to do with deployment and building than it bringing amazing new user-facing features. To be clear, I do expect it to be more stable, more correct, and easier to debug, however I expect the rollout to be bumpy.

It’s a new way to build, with a new compiler in the mix. It’s an entirely new dependency for a critical part of the system. There’s a significant migration at startup, which takes up a significant amount of disk space and which can leave Syncthing in a bad state if it fails. We’ll probably break all the distribution packagers, and the Debian people will freak out about the bundled SQLite dependency and need ugly hacks to use dynamic linking to their packages etc.

All of this points to it not being a great idea to just roll it out to all users without saying anything, hence the 2.0. The new major version is significant in that the current auto upgrade code will not upgrade to it without prompting.

tomasz86 · April 17, 2025, 7:07am

I’ve got another successful migration story. This is one of my ultimate tests - Intel Atom Z3740D paired with 2 GB of RAM and slow eMMC storage. It runs Windows 10 x86 (with x64 not supported due to 32-bit UEFI). Previously, when trying to migrate it, I experienced a lot of crashes due to running out of memory.

This time, everything has gone very smoothly. The migration took just 1 minute. The device doesn’t sync much, so this is probably expected, but still, last time it was much slower.

[start] 2025/04/17 08:48:26 INFO: syncthing v2.0.0-beta.9.dev.16.g6254b066-tomasz86-v2.0.0-beta.9 "Hafnium Hornet" (go1.24.2 windows-386) tomasz86@tomasz86 2025-04-17 06:41:30 UTC [noupgrade, stnoupgrade]
[start] 2025/04/17 08:48:26 INFO: Migrating old-style database to SQLite; this may take a while...
[start] 2025/04/17 08:48:27 INFO: Migrated folder 5eief-x9lna; 26 files and 0k blocks in 0s, 225.6 files/s
[start] 2025/04/17 08:48:37 INFO: Migrating folder abwgi-qj5iw... (12000 files and 0k blocks in 10s, 1198.5 files/s)
[start] 2025/04/17 08:48:47 INFO: Migrating folder abwgi-qj5iw... (24000 files and 0k blocks in 20s, 1198.7 files/s)
[start] 2025/04/17 08:48:48 INFO: Migrated folder abwgi-qj5iw; 25950 files and 0k blocks in 21s, 1187.9 files/s
[start] 2025/04/17 08:49:01 INFO: Migrated folder axfiw-stvqt; 7792 files and 19k blocks in 12s, 626.5 files/s
[start] 2025/04/17 08:49:07 INFO: Migrated folder hqzau-rtdrn; 4002 files and 4k blocks in 6s, 636.2 files/s
[start] 2025/04/17 08:49:10 INFO: Migrated folder j2kgm-h9kws; 329 files and 14k blocks in 3s, 107.8 files/s
[start] 2025/04/17 08:49:10 INFO: Migrated folder mgieg-z3mzc; 80 files and 0k blocks in 0s, 418.3 files/s
[start] 2025/04/17 08:49:15 INFO: Migrated folder ofxx9-nqyuz; 4978 files and 0k blocks in 4s, 1144.3 files/s
[start] 2025/04/17 08:49:15 INFO: Migrated folder p4qoa-r9bwd; 63 files and 0k blocks in 0s, 546.6 files/s
[start] 2025/04/17 08:49:15 INFO: Migrated folder w9h6p-vpccd; 21 files and 0k blocks in 0s, 286.7 files/s
[start] 2025/04/17 08:49:18 INFO: Migrated folder wqpwv-jpr97; 965 files and 5k blocks in 2s, 342.2 files/s
[start] 2025/04/17 08:49:31 INFO: Migrating folder xicj7-mw4ht... (5000 files and 12k blocks in 12s, 388.1 files/s)
[start] 2025/04/17 08:49:36 INFO: Migrated folder xicj7-mw4ht; 8212 files and 16k blocks in 18s, 449.7 files/s
[start] 2025/04/17 08:49:36 INFO: Migrating virtual mtimes...
[start] 2025/04/17 08:49:36 INFO: Migration complete, 52418 files and 62k blocks in 1m9s

What is more important is that the RAM usage was only 100 MB at the maximum the whole time, which prevented Syncthing from running into any out of memory issues .

On a side note, I also want to confirm that I still experience the issue described in https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/196, where folders with many files sync extra slowly when downloading files for the first time. The problem happens universally regardless of the hardware.

For the time being, on the device in question, I “worked around” it by manually copying the folder in advance, so that both sides had all the files. This way, Syncthing just scanned and compared the two, marking both sides “Up to Date”, without having to actually sync anything.