Syncthing 2.0.0-beta.11

Catfriend1 · April 3, 2025, 1:49pm

I’ve adjusted to let the wrapper handle it. SyncthingNative itself generates the default folder just fine as “basic” in config.xml.

Thanks for pointing it out. Which releases of v1.x are/will be affected by this, too? I would then merge the v2.x wrapper change to my main branch for v1.x accordingly.

calmh · April 3, 2025, 3:24pm

That is not supposed to be a change in how the config or API works. @pixelspark can we make sure this is not breaking? If we allowed an empty filesystem type previously (which I considered but the code diff looked like we didn’t…), then we should probably continue doing so.

terry · April 3, 2025, 3:51pm

Just an FYI, for beta 4, I blew away the index files to replicate the large wal file that appeared under beta3. it’s still growing…

and im not yet finished scanning the folders…

I also know I have a lot of folders and handle very large files so maybe the large wal is to be expected. I am most definitely not a typical Syncthing user!!!

pixelspark · April 3, 2025, 6:07pm

Just tested this; when no <filesystemType> is given in a <folder>, the default should be ‘basic’ but instead it reads an empty string somehow. I did add the necessary default:"basic" annotations to these configuration fields so I’m not sure why it is doing this.

I missed this issue because the folder defaults actually have the filesystemType set to basic, so the default folder gets created with an explicit filesystemType set in config.

~~Will attempt a fix and submit a PR.~~

@calmh: fix(config): properly apply defaults when reading folder configuration by pixelspark · Pull Request #10034 · syncthing/syncthing · GitHub

calmh · April 3, 2025, 6:18pm

Yeah there’s a lot going on there. I think you’re a bit of an outlier of a worst case scenario unfortunately, with a shitload of folders all being added from scratch at the same time, lots and lots of writes and reads. Can’t say that I know what would be the solution here other than saying that massive installations doing massive amounts of things apparently can result in a massive WAL, and you could set down folder concurrency to reduce the impact at initialisation time, possibly…

terry · April 3, 2025, 7:07pm

I think there’s something else going on. It’s getting bloaty…

but, yesterday, after I restarted beta3, not all the folders had scanned fully yet the wal size was smaller and index2 was around 16Gb in the morning. It’s as if the initial scan on an empty database is looping and filling out? (wild speculation), but a database that’s already in use seems to handle the indexing a whole lot better.

terry · April 4, 2025, 1:59pm

Update. I have left it running over night, but I think it’s stalling internally. It claims it’s been running for 8h 26m, but it’s around 12 hours. The indexing is still running, but the log updates are very slow now

[RTF25] 2025/04/04 14:40:38.744576 db_update.go:596: DEBUG: checkpoint at 344786 returned 0 32916 32916

[RTF25] 2025/04/04 14:41:29.393857 db_update.go:596: DEBUG: checkpoint at 279499 returned 0 37042 37042

[RTF25] 2025/04/04 14:42:55.367700 db_update.go:596: DEBUG: checkpoint at 262961 returned 0 35411 35411

at memory usage is high

I asked it to create a support package, but nothing is happening. However there is disk activity

Index files are

So i’m going to do a restart but will pause everything and start with a blank db in case it’s the volume of folders that’s the cause

bt90 · April 4, 2025, 2:21pm

At least checkpointing and WAL size are looking good now.

How many files and datasize are we talking about in total?

terry · April 4, 2025, 4:25pm

Probably in excess of 6M files spread over 6 drives. File sizes up to 1.5Tb, but the vast majority of files are sub 20Mb

bt90 · April 4, 2025, 5:07pm

For a single one? How much overall? I’d just like to get feeling of what the DB size is in relation to the dataset.

terry · April 4, 2025, 5:14pm

I have not yet successfully fully indexed under v2, The largest I have seen index-v2 so far is 14Gb. index-v1 is/was 11.2Gb so v2 is going to be much larger.

calmh · April 4, 2025, 5:48pm

calmh · April 4, 2025, 6:04pm

I appreciate all the testing you guys are doing, even if some if it is on outlier cases. It makes Syncthing better for us all.

That said, I get the impression everyone is testing on some sort of setting-up-from-scratch-with-lots-of-folders scenario. It would be good to see some testing of the actual migration path as well, as that will be critical for 99% of normal users…

@terry with your massive setup it would be super interesting to let it run a migration on your existing v1 database for example.

bt90 · April 4, 2025, 6:14pm

@calmh the latest workaround regarding smaller read transactions collects all file infos in a single transaction. I’ve red your comment in the code but I’m still not sure why we shouldn’t do it in batches? At least for large setups like the one @terry is rocking, that might contribute to the high RAM usage.

calmh · April 4, 2025, 6:21pm

It certainly will, yes. Large sync (pull) operations have always been heavy though, since they (now and previously) keep all the files to sync in memory. The reasons for not doing it in batches, more detailed, are,

The GUI allows you to reprioritise files within the queue, which depends on them being in the queue to begin with. (I’m thinking that feature can go, though, to be replaced with a better, more generalised prioritisation of files within an iteration.)
Rename detection can only run on the files that are in the batch. If a rename is split across two batches it would degenerate into separate operations, either copy+delete or delete+pull depending on what the configured pull order results in. (Or at least that’s how it works now, possibly something smarter could be rigged with direct queries to the database for each file using the blocklist hash)
We’d need to let a batch finish and commit to database before starting the next one, which means we lose out on concurrency (e.g., if we’re normally pulling 8 files concurrently, we might spend a while waiting for the last file with 7 “slots” idle, before starting the next iteration)

Possibly only the last point is one that is “inherent” to batching and needs to remain unless we make it very complicated, and maybe it’s not a very large blocker. This is one of those things that could legimitately be configurable too, the batch size that is.

Previously we also depended on having all the files in the queue in order to be able to sort it by various criteria, that at least now happens in the database layer instead.

terry · April 4, 2025, 6:28pm

on it!!

239 · April 4, 2025, 6:52pm

Do the Folder Path suggestions still work for you?

calmh · April 4, 2025, 6:57pm

They do not. They also do not work on the current main branch, so it was broken somewhere non-v2. Maybe the new filesystem type stuff… Yeah. Got a fix coming.

Fixed on main.

239 · April 4, 2025, 8:39pm

I’m trying to reproduce strange zombie files and folders after deleting them on the same side. I think it’s similiar to the duplication issue I found by testing simple rename operations.

WTXPS3T (xfs) renames a file, it is renamed on 5WDSW75 (exfat) but the file gets duplicated, old and new on both sides. The other way around, no problem. I’m not sure what I should look for:

WTXPS3T

2025-04-04 21:43:44 walker/bdayt-7zxpk@0xc0003c2420 Walk [aa] Matcher/[]@0xc0003bc000
2025-04-04 21:43:44 walker/bdayt-7zxpk@0xc0003c2420 checking: File{Name:"aa", Sequence:0, Permissions:0644, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795824} {5WDSW75 1743795689}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:34.992003173 +0200 CEST}
2025-04-04 21:43:44 walker/bdayt-7zxpk@0xc0003c2420 rescan: File{Name:"aa", Sequence:306649, Permissions:0644, ModTime:2025-04-04 21:41:29.12 +0200 CEST, Version:{[{WTXPS3T 1743795824} {5WDSW75 1743795689}]}, VersionHash:, Length:0, Deleted:true, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:1970-01-01 01:00:00 +0100 CET}
2025-04-04 21:43:44 walker/bdayt-7zxpk@0xc0003c2420 to hash: aa File{Name:"aa", Sequence:0, Permissions:0644, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795824} {5WDSW75 1743795689}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:34.992003173 +0200 CEST}
2025-04-04 21:43:44 walker/bdayt-7zxpk@0xc0003c2420 real to hash: aa
2025-04-04 21:43:44 started hashing: File{Name:"aa", Sequence:0, Permissions:0644, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795824} {5WDSW75 1743795689}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:34.992003173 +0200 CEST}
2025-04-04 21:43:45 completed hashing: File{Name:"aa", Sequence:0, Permissions:0644, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795824} {5WDSW75 1743795689}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:1423, BlocksHash:16c7012a7d74d053aace2a39262cfad8bd6aa0d47a0ae2fcaec6ed2fdfb3b6eb, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:34.992003173 +0200 CEST}
2025-04-04 21:43:45 walker/bdayt-7zxpk@0xc0003c2420: Walk bdayt-7zxpk [aa] current progress 745961903/745961904 at 0.0 MiB/s (99%)
2025-04-04 21:43:45 walker/bdayt-7zxpk@0xc0003c2420 Walk progress done bdayt-7zxpk [aa] Matcher/[]@0xc0003bc000
2025-04-04 21:43:45 walker/bdayt-7zxpk@0xc0003c2420 Walk [b] Matcher/[]@0xc0003bc000
2025-04-04 21:43:45 open: open /home/239/xyz/.syncthing.b.tmp: no such file or directory

5WDSW75

2025-04-04 21:43:22 walker/bdayt-7zxpk@0xc000abc210 Walk [b] Matcher/[]@0xc0001361b0
2025-04-04 21:43:22 walker/bdayt-7zxpk@0xc000abc210 checking: File{Name:"b", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{5WDSW75 1743795802}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:12 +0200 CEST}
2025-04-04 21:43:22 walker/bdayt-7zxpk@0xc000abc210 rescan: File{Name:"b", Sequence:322813, Permissions:0755, ModTime:2025-04-04 21:41:29.12 +0200 CEST, Version:{[{5WDSW75 1743795802}]}, VersionHash:, Length:0, Deleted:true, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:32:58.67 +0200 CEST}
2025-04-04 21:43:22 walker/bdayt-7zxpk@0xc000abc210 to hash: b File{Name:"b", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{5WDSW75 1743795802}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:12 +0200 CEST}
2025-04-04 21:43:22 walker/bdayt-7zxpk@0xc000abc210 real to hash: b
2025-04-04 21:43:22 started hashing: File{Name:"b", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{5WDSW75 1743795802}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:12 +0200 CEST}
2025-04-04 21:43:23 completed hashing: File{Name:"b", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{5WDSW75 1743795802}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:1423, BlocksHash:16c7012a7d74d053aace2a39262cfad8bd6aa0d47a0ae2fcaec6ed2fdfb3b6eb, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:43:12 +0200 CEST}
2025-04-04 21:43:23 walker/bdayt-7zxpk@0xc000abc210: Walk bdayt-7zxpk [b] current progress 745961903/745961904 at 0.0 MiB/s (99%)
2025-04-04 21:43:23 walker/bdayt-7zxpk@0xc000abc210 Walk progress done bdayt-7zxpk [b] Matcher/[]@0xc0001361b0
2025-04-04 21:43:23 walker/bdayt-7zxpk@0xc000abc2c0 Walk [a] Matcher/[]@0xc0001361b0
2025-04-04 21:43:45 open: open /run/media/239/931/xyz/.syncthing.aa.tmp: no such file or directory
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc840 Walk [a] Matcher/[]@0xc0001361b0
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc840 checking: File{Name:"a", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795272} {5WDSW75 1743795851}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:44:01.15 +0200 CEST}
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc840 rescan: File{Name:"a", Sequence:322819, Permissions:0644, ModTime:2025-04-04 21:43:23.12 +0200 CEST, Version:{[{WTXPS3T 1743795272} {5WDSW75 1743795851}]}, VersionHash:, Length:0, Deleted:true, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:34:36.52 +0200 CEST}
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc840 to hash: a File{Name:"a", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795272} {5WDSW75 1743795851}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:44:01.15 +0200 CEST}
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc840 real to hash: a
2025-04-04 21:44:11 started hashing: File{Name:"a", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795272} {5WDSW75 1743795851}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:0, BlocksHash:, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:44:01.15 +0200 CEST}
2025-04-04 21:44:11 completed hashing: File{Name:"a", Sequence:0, Permissions:0755, ModTime:2025-04-04 21:28:38.12 +0200 CEST, Version:{[{WTXPS3T 1743795272} {5WDSW75 1743795851}]}, VersionHash:, Length:745961903, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:true, BlockSize:524288, NumBlocks:1423, BlocksHash:16c7012a7d74d053aace2a39262cfad8bd6aa0d47a0ae2fcaec6ed2fdfb3b6eb, Platform:{<nil> <nil> <nil> <nil> <nil> <nil>}, InodeChangeTime:2025-04-04 21:44:01.15 +0200 CEST}
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc840: Walk bdayt-7zxpk [a] current progress 745961903/745961904 at 0.0 MiB/s (99%)
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc840 Walk progress done bdayt-7zxpk [a] Matcher/[]@0xc0001361b0
2025-04-04 21:44:11 walker/bdayt-7zxpk@0xc0000dc8f0 Walk [aa] Matcher/[]@0xc0001361b0

calmh · April 4, 2025, 8:45pm

Is this new to v2?