Syncthing on SQLite -- help test!

Development
181

I’ve observed that refreshing the Web GUI seems significantly slower with the new SQLite database in comparison to the old LevelDB database.

What I mean is that if I refresh the page in the browser, with the old database, it reloads nearly instantly. However, with the new database, it takes significantly more time to reload everything, especially if there are many folders involved.

Is this something that can be expected considering the differences between SQLite and LevelDB? I can see the slowness regardless of the hardware, even on a very fast machine, however it seems more pronounced when the storage is on the slow side.

182

I wouldn’t expect it, no.

183

Could you refresh the UI a few times while capturing a profile?

184

Hmm, I’ll try try doing some actual testing then, and if I can reproduce the GUI slowness reliably, I will definitely come back with some profiles.

In the meantime, I’ve also observed that for some reason it takes extremely long time to rename directories and sync changes after doing so. For example, I’ve got a three device (instance) setup. Two of them run Syncthing v2 (beta5) and the last one runs Syncthing v1. On the first of the v2 instances, I renamed a directory containing 97,019 files and 16,374 directories. The rename synced to the last instance running Syncthing v1 in about an hour.

The problem is that when it comes to the other instance running Syncthing v2, after the same amount of time, the sync is still in progress, and it is moving extremely slowly.

This is what the current state looks like:

image
image702×591 58.2 KB

The number of “Out of Sync Items” does keep decreasing, but it seems to happen in bursts, decreasing only by a few hundred items per minute. At this pace, it looks like the process is going to take days to finish :confused:.

This is all tested on very fast hardware, so there’re no other bottlenecks in place.

Edit 1

The number of “Out of Sync Items” has decreased by 20,000 from 157,000 to 137,000 in 3 hours time, so if the estimation is correct, the sync is going to take ~20 hours to finish. This is just a test set of data, so I’m cancelling the whole thing now. I’ll try to reproduce the issue with a smaller set, possibly without any private filenames, so that I can share the logs too.

Edit 2

Not sure if this is the culprit, but I’ve found out that with the new database, it takes much longer to process file deletions.

  1. Create an new directory with 100,000 empty files.
  2. Let Syncthing add them to the database. This seems to take the same amount of time when both using SQLite and LevelDB.
  3. Delete the newly created directory.
  4. With LevelDB, it took 1m 35s to process the deletion.
  5. With SQLite, the same process took 3m 40s.
2 Likes
185

Casual question, does Syncthing compact the database in the background or is there a manual way to do it? Just asking due to the db now being 26.2Gb

Or does it not matter?

186

We’re invoking incremental vacuum as part of our maintenance routine. This should shrink the DB file.

187

I think I can share a positive migration experience with the current beta v9. I’ve successfully migrated the following device with 49 folders:

image
image700×233 17.4 KB

In comparison to my pervious tests, when the SQLite database was still a single file instead of the current per-folder databases, this migration was extremely quick.

[start] 2025/04/12 20:20:06 INFO: syncthing v2.0.0-beta.9.dev.15.ga86d7366-tomasz86-v2.0.0-beta.9 "Hafnium Hornet" (go1.24.2 windows-amd64) tomasz86@tomasz86 2025-04-12 18:11:05 UTC [noupgrade, stnoupgrade]
[start] 2025/04/12 20:20:06 INFO: Migrating old-style database to SQLite; this may take a while...
[start] 2025/04/12 20:20:06 INFO: Migrated folder 5eief-x9lna; 26 files and 0k blocks in 0s, 2395.7 files/s
[start] 2025/04/12 20:20:07 INFO: Migrated folder 74ex6-feapt; 3533 files and 3k blocks in 0s, 13060.6 files/s
[start] 2025/04/12 20:20:07 INFO: Migrated folder 7oakc-ziezc; 19 files and 0k blocks in 0s, 5235.6 files/s
[start] 2025/04/12 20:20:07 INFO: Migrated folder 7vafc-tydfh; 3893 files and 14k blocks in 0s, 6474.4 files/s
[start] 2025/04/12 20:20:09 INFO: Migrated folder 9pzqk-mllzf; 9738 files and 42k blocks in 1s, 7853.2 files/s
[start] 2025/04/12 20:20:12 INFO: Migrated folder abwgi-qj5iw; 25961 files and 41k blocks in 3s, 8389.4 files/s
[start] 2025/04/12 20:20:12 INFO: Migrated folder bzacn-hx9xg; 16 files and 0k blocks in 0s, 3434.0 files/s
[start] 2025/04/12 20:20:12 INFO: Migrated folder c5vov-haafs; 21 files and 0k blocks in 0s, 4494.6 files/s
[start] 2025/04/12 20:20:13 INFO: Migrated folder capi4-h3srq; 7511 files and 97k blocks in 1s, 4572.5 files/s
[start] 2025/04/12 20:20:13 INFO: Migrated folder cyc5g-wwqoc; 672 files and 12k blocks in 0s, 4591.6 files/s
[start] 2025/04/12 20:20:13 INFO: Migrated folder djmvr-prmdj; 111 files and 3k blocks in 0s, 2968.2 files/s
[start] 2025/04/12 20:20:13 INFO: Migrated folder dwalb-k6yyf; 0 files and 0k blocks in 0s, 0.0 files/s
[start] 2025/04/12 20:20:14 INFO: Migrated folder eaz9j-66wjd; 1358 files and 2k blocks in 0s, 7667.8 files/s
[start] 2025/04/12 20:20:17 INFO: Migrated folder egdxf-hwss5; 21511 files and 40k blocks in 3s, 7060.6 files/s
[start] 2025/04/12 20:20:17 INFO: Migrated folder euxso-mejvb; 17 files and 0k blocks in 0s, 2176.5 files/s
[start] 2025/04/12 20:20:20 INFO: Migrated folder flpvr-njdfi; 21073 files and 119k blocks in 3s, 5948.5 files/s
[start] 2025/04/12 20:20:21 INFO: Migrated folder gjfnh-vvfnn; 1692 files and 13k blocks in 0s, 5830.8 files/s
[start] 2025/04/12 20:20:22 INFO: Migrated folder gjhfl-4nzzc; 7664 files and 32k blocks in 1s, 7060.4 files/s
[start] 2025/04/12 20:20:22 INFO: Migrated folder hgedz-qtsau; 4944 files and 9k blocks in 0s, 8267.6 files/s
[start] 2025/04/12 20:20:23 INFO: Migrated folder hqzau-rtdrn; 4040 files and 4k blocks in 0s, 8607.3 files/s
[start] 2025/04/12 20:20:23 INFO: Migrated folder huxxk-ypuaf; 11 files and 0k blocks in 0s, 2131.4 files/s
[start] 2025/04/12 20:20:23 INFO: Migrated folder iy3dq-jyqvv; 1263 files and 4k blocks in 0s, 8015.8 files/s
[start] 2025/04/12 20:20:23 INFO: Migrated folder j2kgm-h9kws; 330 files and 14k blocks in 0s, 3810.1 files/s
[start] 2025/04/12 20:20:24 INFO: Migrated folder jvtg5-hymk6; 7918 files and 21k blocks in 1s, 7724.1 files/s
[start] 2025/04/12 20:20:24 INFO: Migrated folder k5mqc-dtdww; 16 files and 0k blocks in 0s, 3832.5 files/s
[start] 2025/04/12 20:20:25 INFO: Migrated folder kazza-hew2c; 5919 files and 19k blocks in 0s, 7466.3 files/s
[start] 2025/04/12 20:20:35 INFO: Migrating folder llyqx-ugzel... (83000 files and 85k blocks in 10s, 8243.0 files/s)
[start] 2025/04/12 20:20:40 INFO: Migrated folder llyqx-ugzel; 120775 files and 154k blocks in 15s, 7837.1 files/s
[start] 2025/04/12 20:20:46 INFO: Migrated folder m65ep-rgntd; 36824 files and 255k blocks in 5s, 6662.9 files/s
[start] 2025/04/12 20:20:46 INFO: Migrated folder mfk32-snfdv; 247 files and 4k blocks in 0s, 3793.6 files/s
[start] 2025/04/12 20:20:46 INFO: Migrated folder mgieg-z3mzc; 80 files and 0k blocks in 0s, 5745.3 files/s
[start] 2025/04/12 20:20:46 INFO: Migrated folder mutyx-cwgvy; 3142 files and 26k blocks in 0s, 6548.6 files/s
[start] 2025/04/12 20:20:47 INFO: Migrated folder o2gvm-qojrk; 5423 files and 58k blocks in 1s, 5184.8 files/s
[start] 2025/04/12 20:20:47 INFO: Migrated folder o6kpx-zrftj; 125 files and 0k blocks in 0s, 6716.2 files/s
[start] 2025/04/12 20:20:49 INFO: Migrated folder ofxx9-nqyuz; 7353 files and 260k blocks in 2s, 3600.7 files/s
[start] 2025/04/12 20:20:52 INFO: Migrated folder orxwk-ztxia; 18711 files and 54k blocks in 2s, 6923.4 files/s
[start] 2025/04/12 20:20:52 INFO: Migrated folder p4qoa-r9bwd; 64 files and 0k blocks in 0s, 6836.8 files/s
[start] 2025/04/12 20:20:54 INFO: Migrated folder pxvg7-zlmhu; 15506 files and 21k blocks in 1s, 8214.7 files/s
[start] 2025/04/12 20:20:55 INFO: Migrated folder q3wg6-ukkft; 8008 files and 38k blocks in 1s, 5999.8 files/s
[start] 2025/04/12 20:20:57 INFO: Migrated folder qojqz-thphx; 12560 files and 73k blocks in 2s, 5892.2 files/s
[start] 2025/04/12 20:20:57 INFO: Migrated folder qxhqo-pnwhz; 0 files and 0k blocks in 0s, 0.0 files/s
[start] 2025/04/12 20:21:02 INFO: Migrated folder rpruk-ctygm; 38822 files and 71k blocks in 4s, 7849.0 files/s
[start] 2025/04/12 20:21:04 INFO: Migrated folder tcato-lqtcw; 11025 files and 120k blocks in 2s, 5340.5 files/s
[start] 2025/04/12 20:21:04 INFO: Migrated folder tjfqa-5t6or; 16 files and 0k blocks in 0s, 3852.0 files/s
[start] 2025/04/12 20:21:06 INFO: Migrated folder ut4ck-ffdny; 14828 files and 38k blocks in 2s, 7404.7 files/s
[start] 2025/04/12 20:21:06 INFO: Migrated folder v9jvo-zma9v; 84 files and 0k blocks in 0s, 5800.8 files/s
[start] 2025/04/12 20:21:06 INFO: Migrated folder w9h6p-vpccd; 21 files and 0k blocks in 0s, 4519.5 files/s
[start] 2025/04/12 20:21:07 INFO: Migrated folder wqpwv-jpr97; 972 files and 31k blocks in 0s, 4370.6 files/s
[start] 2025/04/12 20:21:07 INFO: Migrated folder wxenr-xezkt; 16 files and 0k blocks in 0s, 2584.6 files/s
[start] 2025/04/12 20:21:07 INFO: Migrated folder xkuuf-xyskf; 0 files and 0k blocks in 0s, 0.0 files/s
[start] 2025/04/12 20:21:09 INFO: Migrated folder xmptn-sxvx5; 7654 files and 176k blocks in 1s, 3869.8 files/s
[start] 2025/04/12 20:21:09 INFO: Migrated folder yjjea-2q5p5; 627 files and 12k blocks in 0s, 4123.8 files/s
[start] 2025/04/12 20:21:19 INFO: Migrating folder ynb9x-ccysv... (65000 files and 195k blocks in 10s, 6407.9 files/s)
[start] 2025/04/12 20:21:19 INFO: Migrated folder ynb9x-ccysv; 67840 files and 208k blocks in 10s, 6375.0 files/s
[start] 2025/04/12 20:21:19 INFO: Migrating virtual mtimes...
[start] 2025/04/12 20:21:20 INFO: Migration complete, 499980 files and 2104k blocks in 1m13s

Granted, this is Ryzen 5700G with decent SATA SSD storage, however, previously on the very same hardware, it took way longer to migrate a similarly sized Syncthing instance.

Also, I’m not seeing the slow GUI issue that I experienced previously, however I still need to test the current iteration of the new database on slower devices to be sure.

I’ve got just one question about these lines:

Migrated folder dwalb-k6yyf; 0 files and 0k blocks in 0s, 0.0 files/s
Migrated folder qxhqo-pnwhz; 0 files and 0k blocks in 0s, 0.0 files/s
Migrated folder xkuuf-xyskf; 0 files and 0k blocks in 0s, 0.0 files/s

I’ve double checked, but these folders don’t exist in Syncthing on the device anymore. I guess the deleted folders are still present in the database? Is this something normal?

4 Likes
188

Looks like index entries for the folders were present but no files for them. No issue.

Very nice.

1 Like
189

I’ve finished yet another very successful migration, and with this one, it’s possible to directly compare the difference (see https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/99 for the previous test).

The hardware is weak (Intel Core M-5Y10c), and last time, the migration took 2 hours. This time, with beta v9, the migration took less than 13 minutes.

[start] 2025/04/13 00:37:38 INFO: syncthing v2.0.0-beta.9.dev.15.ga86d7366-tomasz86-v2.0.0-beta.9 "Hafnium Hornet" (go1.24.2 windows-amd64) tomasz86@tomasz86 2025-04-12 18:11:05 UTC [noupgrade, stnoupgrade]
[start] 2025/04/13 00:37:39 INFO: Migrating old-style database to SQLite; this may take a while...
[start] 2025/04/13 00:37:39 INFO: Migrated folder 5eief-x9lna; 26 files and 0k blocks in 0s, 567.5 files/s
[start] 2025/04/13 00:37:44 INFO: Migrated folder 74ex6-feapt; 3525 files and 3k blocks in 4s, 709.0 files/s
[start] 2025/04/13 00:37:57 INFO: Migrating folder 9pzqk-mllzf... (5000 files and 15k blocks in 13s, 381.6 files/s)
[start] 2025/04/13 00:38:10 INFO: Migrating folder 9pzqk-mllzf... (9000 files and 39k blocks in 26s, 343.9 files/s)
[start] 2025/04/13 00:38:14 INFO: Migrated folder 9pzqk-mllzf; 9650 files and 41k blocks in 29s, 323.9 files/s
[start] 2025/04/13 00:38:24 INFO: Migrating folder abwgi-qj5iw... (7000 files and 12k blocks in 10s, 696.1 files/s)
[start] 2025/04/13 00:38:36 INFO: Migrating folder abwgi-qj5iw... (14000 files and 25k blocks in 22s, 629.9 files/s)
[start] 2025/04/13 00:38:46 INFO: Migrating folder abwgi-qj5iw... (19000 files and 31k blocks in 32s, 586.9 files/s)
[start] 2025/04/13 00:39:00 INFO: Migrating folder abwgi-qj5iw... (24000 files and 38k blocks in 45s, 523.4 files/s)
[start] 2025/04/13 00:39:13 INFO: Migrated folder abwgi-qj5iw; 25950 files and 41k blocks in 58s, 440.2 files/s
[start] 2025/04/13 00:39:28 INFO: Migrating folder capi4-h3srq... (2000 files and 8k blocks in 15s, 131.0 files/s)
[start] 2025/04/13 00:40:05 INFO: Migrating folder capi4-h3srq... (4000 files and 29k blocks in 51s, 77.3 files/s)
[start] 2025/04/13 00:40:26 INFO: Migrating folder capi4-h3srq... (5000 files and 48k blocks in 1m13s, 68.5 files/s)
[start] 2025/04/13 00:40:37 INFO: Migrating folder capi4-h3srq... (7000 files and 64k blocks in 1m23s, 83.8 files/s)
[start] 2025/04/13 00:40:37 INFO: Migrated folder capi4-h3srq; 7511 files and 97k blocks in 1m24s, 89.0 files/s
[start] 2025/04/13 00:40:59 INFO: Migrating folder ddfwz-if24v... (2000 files and 197k blocks in 21s, 92.5 files/s)
[start] 2025/04/13 00:41:10 INFO: Migrating folder ddfwz-if24v... (3000 files and 339k blocks in 32s, 93.0 files/s)
[start] 2025/04/13 00:41:29 INFO: Migrating folder ddfwz-if24v... (4000 files and 422k blocks in 51s, 77.2 files/s)
[start] 2025/04/13 00:41:39 INFO: Migrating folder ddfwz-if24v... (10000 files and 422k blocks in 1m1s, 161.4 files/s)
[start] 2025/04/13 00:41:41 INFO: Migrated folder ddfwz-if24v; 11003 files and 422k blocks in 1m3s, 174.3 files/s
[start] 2025/04/13 00:41:41 INFO: Migrated folder djmvr-prmdj; 99 files and 3k blocks in 0s, 139.6 files/s
[start] 2025/04/13 00:41:52 INFO: Migrating folder flpvr-njdfi... (10000 files and 42k blocks in 10s, 972.1 files/s)
[start] 2025/04/13 00:42:04 INFO: Migrating folder flpvr-njdfi... (15000 files and 50k blocks in 22s, 658.4 files/s)
[start] 2025/04/13 00:42:17 INFO: Migrating folder flpvr-njdfi... (20000 files and 65k blocks in 35s, 560.4 files/s)
[start] 2025/04/13 00:42:21 INFO: Migrated folder flpvr-njdfi; 21073 files and 119k blocks in 39s, 529.8 files/s
[start] 2025/04/13 00:42:28 INFO: Migrated folder gjhfl-4nzzc; 7664 files and 32k blocks in 6s, 1132.1 files/s
[start] 2025/04/13 00:42:30 INFO: Migrated folder hgedz-qtsau; 4946 files and 9k blocks in 2s, 2065.9 files/s
[start] 2025/04/13 00:42:30 INFO: Migrated folder huxxk-ypuaf; 11 files and 0k blocks in 0s, 845.1 files/s
[start] 2025/04/13 00:42:31 INFO: Migrated folder j2kgm-h9kws; 329 files and 14k blocks in 0s, 1056.5 files/s
[start] 2025/04/13 00:42:37 INFO: Migrated folder jvtg5-hymk6; 7452 files and 20k blocks in 6s, 1125.6 files/s
[start] 2025/04/13 00:42:48 INFO: Migrating folder llyqx-ugzel... (21000 files and 17k blocks in 10s, 1950.6 files/s)
[start] 2025/04/13 00:42:58 INFO: Migrating folder llyqx-ugzel... (29000 files and 25k blocks in 21s, 1368.8 files/s)
[start] 2025/04/13 00:43:10 INFO: Migrating folder llyqx-ugzel... (35000 files and 34k blocks in 32s, 1078.2 files/s)
[start] 2025/04/13 00:43:21 INFO: Migrating folder llyqx-ugzel... (41000 files and 39k blocks in 43s, 934.0 files/s)
[start] 2025/04/13 00:43:35 INFO: Migrating folder llyqx-ugzel... (46000 files and 46k blocks in 57s, 798.4 files/s)
[start] 2025/04/13 00:43:48 INFO: Migrating folder llyqx-ugzel... (50000 files and 52k blocks in 1m11s, 704.2 files/s)
[start] 2025/04/13 00:43:59 INFO: Migrating folder llyqx-ugzel... (52000 files and 55k blocks in 1m21s, 634.8 files/s)
[start] 2025/04/13 00:44:10 INFO: Migrating folder llyqx-ugzel... (54000 files and 57k blocks in 1m33s, 580.0 files/s)
[start] 2025/04/13 00:44:22 INFO: Migrating folder llyqx-ugzel... (57000 files and 60k blocks in 1m45s, 541.2 files/s)
[start] 2025/04/13 00:44:33 INFO: Migrating folder llyqx-ugzel... (60000 files and 66k blocks in 1m56s, 516.4 files/s)
[start] 2025/04/13 00:44:44 INFO: Migrating folder llyqx-ugzel... (63000 files and 85k blocks in 2m6s, 497.5 files/s)
[start] 2025/04/13 00:44:56 INFO: Migrating folder llyqx-ugzel... (67000 files and 89k blocks in 2m18s, 483.0 files/s)
[start] 2025/04/13 00:45:11 INFO: Migrating folder llyqx-ugzel... (72000 files and 95k blocks in 2m33s, 467.9 files/s)
[start] 2025/04/13 00:45:21 INFO: Migrating folder llyqx-ugzel... (80000 files and 101k blocks in 2m43s, 487.8 files/s)
[start] 2025/04/13 00:45:36 INFO: Migrating folder llyqx-ugzel... (83000 files and 104k blocks in 2m58s, 464.5 files/s)
[start] 2025/04/13 00:45:47 INFO: Migrating folder llyqx-ugzel... (85000 files and 106k blocks in 3m9s, 447.8 files/s)
[start] 2025/04/13 00:45:57 INFO: Migrating folder llyqx-ugzel... (88000 files and 109k blocks in 3m20s, 439.5 files/s)
[start] 2025/04/13 00:46:08 INFO: Migrating folder llyqx-ugzel... (95000 files and 113k blocks in 3m31s, 450.1 files/s)
[start] 2025/04/13 00:46:21 INFO: Migrating folder llyqx-ugzel... (99000 files and 117k blocks in 3m43s, 442.2 files/s)
[start] 2025/04/13 00:46:36 INFO: Migrating folder llyqx-ugzel... (103000 files and 121k blocks in 3m59s, 430.8 files/s)
[start] 2025/04/13 00:46:47 INFO: Migrating folder llyqx-ugzel... (108000 files and 126k blocks in 4m9s, 432.7 files/s)
[start] 2025/04/13 00:47:02 INFO: Migrating folder llyqx-ugzel... (111000 files and 132k blocks in 4m24s, 419.4 files/s)
[start] 2025/04/13 00:47:14 INFO: Migrating folder llyqx-ugzel... (112000 files and 133k blocks in 4m36s, 404.4 files/s)
[start] 2025/04/13 00:47:33 INFO: Migrating folder llyqx-ugzel... (113000 files and 136k blocks in 4m55s, 381.9 files/s)
[start] 2025/04/13 00:47:36 INFO: Migrated folder llyqx-ugzel; 114395 files and 137k blocks in 4m58s, 382.8 files/s
[start] 2025/04/13 00:47:47 INFO: Migrating folder m65ep-rgntd... (4000 files and 9k blocks in 10s, 375.9 files/s)
[start] 2025/04/13 00:47:58 INFO: Migrating folder m65ep-rgntd... (13000 files and 25k blocks in 21s, 594.4 files/s)
[start] 2025/04/13 00:48:10 INFO: Migrating folder m65ep-rgntd... (24000 files and 43k blocks in 33s, 709.6 files/s)
[start] 2025/04/13 00:48:22 INFO: Migrating folder m65ep-rgntd... (31000 files and 53k blocks in 46s, 670.6 files/s)
[start] 2025/04/13 00:48:33 INFO: Migrating folder m65ep-rgntd... (36000 files and 97k blocks in 56s, 633.9 files/s)
[start] 2025/04/13 00:48:39 INFO: Migrated folder m65ep-rgntd; 36710 files and 186k blocks in 1m2s, 585.1 files/s
[start] 2025/04/13 00:48:39 INFO: Migrated folder mfk32-snfdv; 247 files and 4k blocks in 0s, 1215.5 files/s
[start] 2025/04/13 00:48:39 INFO: Migrated folder mgieg-z3mzc; 80 files and 0k blocks in 0s, 617.2 files/s
[start] 2025/04/13 00:48:49 INFO: Migrating folder mheku-2gahx... (22000 files and 42k blocks in 10s, 2178.8 files/s)
[start] 2025/04/13 00:48:53 INFO: Migrated folder mheku-2gahx; 37898 files and 55k blocks in 13s, 2793.9 files/s
[start] 2025/04/13 00:48:56 INFO: Migrated folder o2gvm-qojrk; 4886 files and 58k blocks in 3s, 1547.3 files/s
[start] 2025/04/13 00:48:56 INFO: Migrated folder o6kpx-zrftj; 132 files and 0k blocks in 0s, 2327.4 files/s
[start] 2025/04/13 00:49:02 INFO: Migrated folder ofxx9-nqyuz; 4977 files and 236k blocks in 6s, 794.6 files/s
[start] 2025/04/13 00:49:02 INFO: Migrated folder p4qoa-r9bwd; 64 files and 0k blocks in 0s, 2282.6 files/s
[start] 2025/04/13 00:49:12 INFO: Migrated folder qojqz-thphx; 12800 files and 75k blocks in 9s, 1295.4 files/s
[start] 2025/04/13 00:49:18 INFO: Migrated folder tcato-lqtcw; 11025 files and 120k blocks in 5s, 1865.5 files/s
[start] 2025/04/13 00:49:24 INFO: Migrated folder ut4ck-ffdny; 14349 files and 38k blocks in 6s, 2298.1 files/s
[start] 2025/04/13 00:49:24 INFO: Migrated folder w9h6p-vpccd; 21 files and 0k blocks in 0s, 1083.7 files/s
[start] 2025/04/13 00:49:25 INFO: Migrated folder wqpwv-jpr97; 965 files and 31k blocks in 0s, 1318.1 files/s
[start] 2025/04/13 00:49:35 INFO: Migrated folder xmptn-sxvx5; 7654 files and 176k blocks in 10s, 724.9 files/s
[start] 2025/04/13 00:49:49 INFO: Migrating folder ynb9x-ccysv... (14000 files and 103k blocks in 13s, 1020.2 files/s)
[start] 2025/04/13 00:50:02 INFO: Migrating folder ynb9x-ccysv... (39000 files and 141k blocks in 26s, 1455.0 files/s)
[start] 2025/04/13 00:50:12 INFO: Migrating folder ynb9x-ccysv... (66000 files and 196k blocks in 36s, 1792.9 files/s)
[start] 2025/04/13 00:50:13 INFO: Migrated folder ynb9x-ccysv; 67805 files and 208k blocks in 37s, 1801.2 files/s
[start] 2025/04/13 00:50:16 INFO: Migrated folder zu2la-qszfr; 10501 files and 5k blocks in 3s, 3368.0 files/s
[start] 2025/04/13 00:50:16 INFO: Migrating virtual mtimes...
[start] 2025/04/13 00:50:17 INFO: Migration complete, 423748 files and 2143k blocks in 12m37s

On top of that, previously, the index exchange phase went on for 24+ hours with no end on the horizon, making Syncthing completely unusable. With the current beta, the index exchange also took just about 13 minutes.

I’ve paid attention to the RAM usage as well, which grew up to about 800 MB. Previously, it went up as high as 5 GB.

7 Likes
190

I’m experiencing a problem that began when trying to revert local changes in a Receive Only folder. The case was similar to what I already experienced before in https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/184. Basically, it was taking forever to re-download files that have been deleted locally.

After many hours, I’ve tried to pause the folder and also pause the remote device, but Syncthing seems to be completely stuck. The GUI appears responsive on the outside, however trying to make any changes leads to an endless “Syncthing is saving changes state”.

I’ve also noticed that Syncthing has now taken 10 GB of RAM. The CPU is also doing something, but I’m not sure what, and the CPU load seems to be steady at same level, all the time.

I’m attaching CPU and heap profiles, and also a full goroutine stack dump.

  • cpu.pprof (32.7 KB)
  • heap.pprof (301.8 KB)
  • goroutine.txt (108.9 KB)

The Syncthing binary has been built from https://github.com/tomasz1986/syncthing/tree/tomasz86/releases/tomasz86-v2.0.0-beta.9.

I’d very thankful if you could have a look at the them and see if there is anything obvious in there. Otherwise, I will likely be forced to kill and restart the program :slightly_frowning_face:.

2 Likes
191

Thanks, I’ll check it out, though it may be a little while as I’m getting on a plane later today and will be in transit for a bit over a day and then on vacation. :palm_tree: I’m hoping for the 2.0 stuff to marinate in the meantime so this is awesome keep it coming :pray:

1 Like
192

Sure, I hope you have a great time :slightly_smiling_face:.

I’ve started the migration with less important devices/instances, so it’s not that huge of a problem. This one, I could actually keep in this state if it was helpful later for further troubleshooting. Does it matter or should I just kill the process and launch Syncthing again?

2 Likes
193

If you can take a support bundle before killing it, that’s all we need. It’s essentially what you already had plus some logs and config for correlation, no need to publish it publicly if you don’t want.

1 Like
194

I’ve just taken and saved it. I would need to obfuscate device names and paths if I was to share some of them, but I can do it if they are necessary. I’ll keep the bundle file on the disk, so just let me know if something else is needed from it.

2 Likes
195

The goroutine profile in pprof format would be nice. The CPU profile above has the things doing leveldb gets, so I think that one is from something else as well.

1 Like
196

You’re right about the profiles, there are two instances of Syncthing running, and I’ve taken them from the wrong one :flushed:. Only the goroutine dump was correct.

I’m attaching the proper files below, taken from the support bundle. I’ve also added the few other files from the bundle that don’t require obfuscation.

Edit:

Just to show the main issue:

image
image696×628 56.7 KB

This is in real time, so basically 2 files are downloaded per minute :confused:. With this in mind, it’s actually much faster to just remove the whole folder and download everything from scratch, because there is no such slowdown when actually syncing. It seems just when local changes are reverted in this “Preparing for Sync” phase that is super slow. I was wrong, it’s also very slow when synchronising the folder from scratch. I’m going to try to reproduce this in a clean environment. I think I can also reproduce the crazy RAM usage, as after leaving the folder in this state alone for a while, it’s again at 12+ GB.

Just in case, I’m uploading yet another set of support bundle files. The previous one was after breaking the folder reverting changes and the connection. This one is still from when the folder is in the very slow “Preparing for Sync” state with extremely high RAM usage.

3 Likes
197

Could someone compile the code to a windows amd x64. I have no idea how to do it so allows me to also try out beta 9. Specifically interested if the first indexing run through is quicker. Last one took 3 days

198

I think you can just use beta v8. The only difference is that beta v9 has a fix for the certificate reset issue (see Comparing v2.0.0-beta.8...v2.0.0-beta.9 · syncthing/syncthing · GitHub).

1 Like
199

Or download the Github Action artifacts (bottom of the page):

200

My bad, I see I forgot to publish beta.9, sorry. I’ll push a new one with the latest merges (nothing new in v2 but from main), coming up shortly.

1 Like