# Syncthing on SQLite -- help test!
**URL:** https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981
**Category:** Development
**Created:** [March 13, 2025, 9:26am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981 "2025-03-13T09:26:51Z")
**Posts on this page:** 20
**Page:** 4
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 18, 2025, 8:18am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/61 "2025-03-18T08:18:27Z")
Zero testing on it from my side. You’re absolutely right of course, and I’ve put in some effort on building with zig, which unlocked a couple more native combinations. Probably we should not ship binaries with a driver we don’t do any testing on, and I’m half minded to even not ship binaries for the modernc combination either, instead leaving binary production for the respective os/distribution packagers…
Looking at the usage reporting stats for users running _our_ binaries currently, we’d have 0.056% on modernc and 0.008% wazero. I think the FreeBSD and OpenBSD users have native packaging, and illumos/Solaris folk are used to fend for themselves (speaking as a former one).
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 18, 2025, 9:14am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/62 "2025-03-18T09:14:48Z")
> **[Release SQLite Preview 6 · calmh/syncthing](https://github.com/calmh/syncthing/releases/tag/v2.0.0-beta.6)**
>
> Now with zig, for great justice
---
### Author:  [@tomasz86](https://forum.syncthing.net/u/tomasz86)
#### Post date: [March 18, 2025, 9:32am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/63 "2025-03-18T09:32:58Z")
> [@calmh](#):
>
> Yeah then you get a build with the [modernc.org](http://modernc.org) translation of SQLite, which may be fine but a bit slower and not what we’ve tested as much here.
So I’ve tested the two, and yes, your build took 5m 12s while mine took 6m 23s to scan the same folder.
> [@Martchus](#):
>
> It could be that cgo is needed when compiling for Android anyway - I never tried it without.
The compiler throws an error about it immediately if you try to build without `CGO_ENABLED` 😉. This has always been the case with Android.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 18, 2025, 9:43am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/64 "2025-03-18T09:43:42Z")
I guess we should at least have the modernc option in the code as it is probably a pain to build on Windows without it. Even if we don’t ship that combo, having it available to reduce friction for someone who just wants to contribute something non-database related and build locally seems reasonable.
---
### Author:  [@Martchus](https://forum.syncthing.net/u/Martchus)
#### Post date: [March 18, 2025, 10:22am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/65 "2025-03-18T10:22:09Z")
That makes sense. I’m wondering whether it would make sense if Syncthing could display what database backend it uses then - just so one is aware of the limitation. This info might also help when users report bugs.
Note that thanks to MSYS2 is really isn’t a big deal (and definitely not a pain) to install a C/C++ compiler on Windows, though. You can then even get Go and many other packages for additional programming languages from MSYS2 and thus install/update your whole devel setup from one source - which is pretty neat.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 18, 2025, 10:23am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/66 "2025-03-18T10:23:40Z")
It’s in the “about” dialog and `--version` output, the odd builds get a build tag `ncruces-sqlite` or `modernc-sqlite`.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 18, 2025, 11:02am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/67 "2025-03-18T11:02:35Z")
So who’s excited about reviewing the PR
(On the plus side, that’s 4000 fewer lines of database layer 🙂 and there is still a certain amount of the old stuff left for migration purposes.)
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 18, 2025, 1:54pm UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/68 "2025-03-18T13:54:34Z")
> **[Release SQLite preview 7 · calmh/syncthing](https://github.com/calmh/syncthing/releases/tag/v2.0.0-beta.7)**
>
> ⚠️ Database name changed to index-v2.db -- no automatic migration ⚠️
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 20, 2025, 9:54am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/69 "2025-03-20T09:54:07Z")
> **[Release SQLite preview 8 · calmh/syncthing](https://github.com/calmh/syncthing/releases/tag/v2.0.0-beta.8)**
>
> Wrapping things up a bit
---
### Author:  [@tomasz86](https://forum.syncthing.net/u/tomasz86)
#### Post date: [March 20, 2025, 1:08pm UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/70 "2025-03-20T13:08:35Z")
Just one more observation about the new database. When a folder finishes scanning for the first time, the progress hangs at `100%` for about a minute or so. Only after that, the scanning process actually completes. What is it doing during that time exactly? If there is still something to do, then maybe it would be better to have the progress stay something like `99%` instead?
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 20, 2025, 4:25pm UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/71 "2025-03-20T16:25:00Z")
I believe that would be the check for deleted files, which has always been there at that step. Is it markedly slower than before?
---
### Author:  [@tomasz86](https://forum.syncthing.net/u/tomasz86)
#### Post date: [March 20, 2025, 4:41pm UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/72 "2025-03-20T16:41:04Z")
Just tested and they’re about the same 😅, so I guess it’s unrelated to the database format.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 20, 2025, 4:49pm UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/73 "2025-03-20T16:49:57Z")
Cool, yeah, should be. That phase is typically limited by stat:ing each file in the folder. I think that may be a bit slower on Windows than other OSes, for reasons.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 21, 2025, 4:42pm UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/74 "2025-03-21T16:42:22Z")
> **[Release SQLite preview 9 · calmh/syncthing](https://github.com/calmh/syncthing/releases/tag/v2.0.0-beta.9)**
>
> In which the cleanups continued and I think we're pretty much done here
Happy Friday y’all
---
### Author:  [@tomasz86](https://forum.syncthing.net/u/tomasz86)
#### Post date: [March 22, 2025, 12:31am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/75 "2025-03-22T00:31:51Z")
Just a comment, but if I use NTFS compression on the new database, it compresses by as much as 50% (e.g. from 120 MB to 60 MB). For the record, the old database barely compresses.
---
### Author:  [@bt90](https://forum.syncthing.net/u/bt90)
#### Post date: [March 22, 2025, 6:53am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/76 "2025-03-22T06:53:37Z")
@tomasz86 could you run `VACUUM` on the DB while syncthing is stopped?
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [March 22, 2025, 7:09am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/77 "2025-03-22T07:09:21Z")
I think that’s an observation on the sqlite file format more than on anything we do with it, really. I don’t find it super surprising since the data is indexed multiple ways. I get about a 50% reduction when gzipping the entire database file, so it’s impressive to get about the same from fs compression. Possibly since your database is comparatively quite small there’s a lot of easily compressible metadata. My server database is 4.3 GB which gzips down to 2.6 GB.
---
### Author:  [@tomasz86](https://forum.syncthing.net/u/tomasz86)
#### Post date: [March 22, 2025, 8:12am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/78 "2025-03-22T08:12:01Z")
The `VACUUM` command has reduced the database size to mere 35 MB 🙂.
---
### Author:  [@bt90](https://forum.syncthing.net/u/bt90)
#### Post date: [March 22, 2025, 8:43am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/79 "2025-03-22T08:43:42Z")
That was what I feared. Fragmentation means that a lot of pages are mostly empty. As empty space is the easiest thing to compress, we can see a similar reduction in size for FS and the more advanced gzip compression.
The hard part is to automate this in our maintenance routine. As `VACUUM` is about the most expensive operation sqlite offers, we’d need to quickly evaluate the fragmentation level and only do this after hitting a certain threshold.
---
### Author:  [@bt90](https://forum.syncthing.net/u/bt90)
#### Post date: [March 22, 2025, 9:07am UTC](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981/80 "2025-03-22T09:07:12Z")
[The DBSTAT Virtual Table](https://www.sqlite.org/dbstat.html) might be useful to check the fragmentation
[Previous page](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981.md?page=3)
[Next page](https://forum.syncthing.net/t/syncthing-on-sqlite-help-test/23981.md?page=5)