I saw an upgrade was available to Syncthing 2.0 and started the process. It’s been running for 27 hours on two of the machines in my cluster and seems to be slowly making progress ever since.
I have a fairly large (huge?) amount of data (~40-50TB) in the 15 folders which are across the 3 machines in my Syncthing cluster.
I sorted through the logs and so far it seems to have completed 6 of those folders. But I’m beginning to have some questions:
Is there anywhere I can read up on this migration to understand what’s taking so long?
What exactly does the filesrate in the logs mean? Should I be worried that in the recent log entries filesrate has dropped into the single digits?
What happens if the script gets interrupted?
Is there anything that I can do to speed it up?
My Syncthing is running on my TrueNAS machine and obtained from its app catalog. The hardware is has a Xeon D-2123 CPU, 32 GB of RAM and 2 pools: a 4x1TB SATA SSD pool and a 7x20TB HDD pool. The Syncthing config and database are stored on the pool of SSDs. The folders being synched by Syncthing exist on both pools.
From what I’ve seen, there doesn’t seem to be any limitations with regards to hardware utilization, the CPU, RAM, and disk utilization for the machine all seem to be pretty normal
I primarily use Syncthing to make sure that my offsite NAS is up-to-date. I upgraded Syncthing on both machines at about the same time, each machine has been running upgrade for this whole time.
I have a third NAS which also is running Syncthing, but only 1 folder is set up on that machine. It’s upgrade compelted in moments and it’s currently functional.
If you have any suggestions or information you think might be helpful, I’d appreciate it!
Here’s the output of the upgrade. I trimmed out a bunch of the entries in the middle to fit under the character count:
I can only say that with this amount of data, the migration process will likely take days (if not more).
I think the exact number of files isn’t that important, because this is about the number of blocks in the database, e.g. you can have a folder with a single 100 GB file, which will take longer time to migrate, and a folder with 100,000 very small files, which will migrate much faster.
The file rate is unfortunately precisely what you might think it is – migrated files per second at the current point in time. Inserts get slower as the database gets larger, because of index rewrites and whatnot, SQLite internals. Two and a half files per second is obviously abysmal and will take ages, assuming you have more than say 274000 files in that folder… I’m going to run a couple of benchmarks to see if this seems reasonable or not.
Yeah, the hundreds of millions of blocks is what is hurting you here primarily I think, because they go into the same database. That probably needs some sort of sharding to behave reasonably, which we don’t do.
Right now, manual sharding into smaller folders might be the only practical advice. (Or, obviously, sticking with v1 for the moment, unfortunate as that is.)
If the files are unevenly sized, it might be a little bit of a hump as well. I notice that in the beginning you have 2706551 blocks in 48000 files (avg 56 blocks per file) while in the last output it’s up to 137002961 blocks in 273000 files (avg 501 blocks per file). If it’s currently processing files with 1000 blocks each then that file rate of 2.8 is still something like 2800 database inserts per second which is maybe not great for your case but not molasses either. If some remaining files are smaller it could pick up speed.
That makes sense! This particular folder would definitely will have files of different sizes in it. There’s quite a bunch of video footage from cameras that might explain the differences.
I uploaded the the entire log to pastebin. Do you think it’s worrisome at all that there’s a distinct downward trend in the filesrate?
From your description, it doesn’t sound like there would be much benefit to wiping everything out and starting over from scratch. Am I understanding that correctly?
There will be no benefit from starting over, for sure. There will always be a downward trend, because inserts get slower the more data there already is in the database (because items are ordered by primary key and pages need to be rewritten to make space, as I understand it, simplified). The question is if there can ever be an upwards trend again, and it’s at least possible if you have files of varying size.
I’m having a similar problem to that of briancmoses… I’m currently running TrueNAS Community version 25.04.2.1. Syncthing was updated earlier today to version 2.0 and has been in Deploying mode for about 12 hours. Where briancmoses is able to view several lines of his logs, I am only able to see the standard 2 lines…
2025-08-13 13:57:55.918506+00:002025-08-13 16:57:55 INF Migrating old-style database to SQLite; this may take a while… (log.pkg=syncthing)
Once I was made aware that this is a migration, I then realized that it could take quite some time, due to my having a very large photo archive (I’m a retired photojournalist).
Nothing else has popped up in the logs. Is this normal or problematic? Should I be seeing some lines of progression?
I’m very much a newbie to this and any help would be greatly appreciated.
In this example, the filesrate value decreased from 400 to 40 within ten minutes, and it will continue to approach zero until the expected runtime is months or years.
When this happened, iostat was reporting a disk utilization rate of 25% and vmstat was reporting a memory utilization rate of 50%, so Syncthing 2.0 should/does have enough system capacity to quickly finish the conversion.
I also tried setting sync=disabled according to an sqlite troubleshooting tutorial, but it had no effect.
Is there anything you can recommend that I try? I’m down to filesrate=1.4097565115211683 on the remaining machine.
I am perfectly happy to let it keep chugging along. But I’m a tiny bit worried about the remaining folders it hasn’t even attempted to migrate though. The folder it’s on is the second largest but I have an even larger folder that hasn’t even started yet.
I’m willing to be a guinea pig, though.
Is it just me, or do most of these reports all seem to have TrueNAS SCALE (or Community Edition) in common?