I use Thunderbird with local folders (some of which are a few hundred MB) and Thunderbird keeps an index in a global-messages-db.sqlite file (300MB).
More generally, does anyone else use Thunderbird and syncthing?
Please check https://forum.syncthing.net/search?q=thunderbird first, because there have been quite a few topics on this already.
In short, syncing a whole Thunderbird profile is a no-go, unless you only want to do it for backup purposes. For one, syncing databases is only safe if they are closed, meaning that you may end up with broken files if you do it while Thunderbird is running (and don’t wait for the application to exit and only then sync the closed database files). Secondly, if you want to sync local e-mails themselves, then I think it’s much more efficient if you switch to using maildir inside Thunderbird (which is a per-account option), because then instead of huge mbox files you only have to deal with single eml files.
However, the main problem I’ve experienced is that even if you do sync your local e-mail like that, Thunderbird itself isn’t capable of automatically refreshing the e-mail tree after the data has changed on the disk. This means that even if you do sync any changes to another device, on that device you will still have to either use the repair folder function inside Thunderbird or remove the relevant folder’s msf index file in order for the newly synced messages to actually appear inside the application.
Just for the record, I do sync my Local Folders directory as below, i.e. store type is maildir and the folder itself I actually store in a different location outside the Thunderbird profile path.
However, I still need to either apply the repair folder / delete msf index workaround described above or remember not to run Thunderbird on multiple devices at the same time. If you always close Thunderbird on one device before opening it on the other, then this Local Folders sync will work even without the workaround. I’d also advise to add *.msf files to ignore patterns, because they contain column-related settings, and those you may need to customise differently depending on your screen size, resolution, etc.
@tomasz86, thanks for the response. I did read about this issue, including in the forum, and you confirmed what I feared – I was hoping things might’ve changed recently or I misunderstood.
I’ve wanted maildir in TB for over a decade now (really) but it still sounds like it’s not properly supported. I’ve wasted too much time over the years trying to get it to work.
I’d probably have to move to running a local IMAP folder or something like that, but don’t want that level of complication. Looks like I’ll keep up with unison since that’s what I’ve been using.
This is probably because Thunderbird’s maildir isn’t the proper maildir (see https://wiki.mozilla.org/Thunderbird/Maildir) 
. When it comes to Thunderbird specifically, changing the store type to maildir really only means that e-mails are stored in individual eml files instead of being lumped together inside giant mbox files. Nevertheless, this does make it more suitable for possible synchronisation (and less prone to corruption, etc.).
I use Syncthing to sync my Thunderbird profile (2.8GB, mbox) between two laptops with a Raspberry Pi4 in between. The Pi runs 24/7 and its Thunderbird folder is type ‘Receive Encrypted’.
Heeding tomasz’s warning, after closing Thundbird, I always ensure laptop A is fully synced to the Pi. Before opening Thunderbird on laptop B, I wait for the folder to be fully up-to-date.
With care over several years, this arrangement allows the laptops to function independently whether on the same LAN or different, distant locations, even using mobile data connection when needed.
@tomasz86 has already covered all of the important bits, so I’ll just share some thoughts regarding support/reliability of Thunderbird’s Maildir mail store format…
Although Thunderbird 12 was the first version to support a pluggable mail store, it wasn’t until version 38 that a read-only drop down menu showing the configured mail store format was even visible. As of the latest version 115, it’s still read-only – i.e., cannot switch an existing email account between storage formats (when the default mail store is changed in the preferences, only new accounts are affected). It makes migrating from Mbox to Maildir possibly time consuming, but it’s still well worth it.
At my employer, one of our teams of 10+ users has been using Thunderbird’s Maildir mail store format since late summer of 2015 without losing a single message to issues with the storage format.
For the team, email is the single most important application they rely on. They’ve got some pretty large mail folders – biggest one so far is over 25GB containing 2+ million messages. (I think some of it’s borderline digital equivalent of hoarding, but I’m just there to support what they need to do their work even if it complicates storage management. 
)
Using Maildir format made access over network shares, rsync, Syncthing and backups much more efficient and faster. It also made enabling inline compression in btrs more practical.
(With Mbox, there are filesystem considerations and older 32-bit versions of Thunderbird only supported up to 4GB per mail folder. In Thundebird’s config editor, there’s the advanced setting mailnews.allowMboxOver4GB – by default it’s set to false even for 64-bit Thunderbird because a user might copy their profile to a FAT32 or VFAT formatted storage device such as a USB flash drive.)
With regards to Thunderbird’s global-messages-db.sqlite index file, it’s not an index to each message in a mail folder, but rather a search index. If it’s damaged or lost, no emails are corrupted and Thunderbird will just repair/rebuild it as needed.
Because the search index isn’t constantly being updated, most of the time the database is simply open for reading. Generally speaking, copying a SQLite database that’s already open is safe. You’ll lose whatever data that hasn’t been flushed from RAM, but otherwise the database will be consistent, even more so if the underlying filesystem supports copy-on-write (COW).
Of course, a caveat is that Thunderbird on your destination Syncthing device shouldn’t be in use while syncing, but even that won’t corrupt the database because Syncthing assembles the updated database file first as a temporary file before renaming it as the final step. A running Thunderbird will be reading/writing to the old database that will eventually be released and deleted by the OS. When Thunderbird tries to reopen the database, it’ll be looking at the new one.
@gadget, thanks for that, especially the reminder that the global-messages DB can always be reindexed. Perhaps I’ll have time in the future to try maildir again – I’ll have to see if it’ll challenge apple FS at all to have that many more files.
I haven’t delved much into APFS internals, but I do know that it uses B-trees for organizing files. Btrfs also uses B-trees, which I’ve found to be way more suitable when there are a lot of files in a volume.
At work, the 10+ member team I mentioned earlier has its own dedicated file server (running Linux). The btrfs-formatted /home volume contains over 3 million files, most of it from Thunderbird’s Maildir mail store. Directory listings, ncdu, and other disk activity is quick and responsive due to the combination of SSDs and btrfs. Although 3+ million files is a lot, it doesn’t hold a candle to the primary NAS at work currently holding 50+ million files (most are less than 250KB each).
One downside to Maildir will depend on the typical size of each message. Modern HDDs and SSDs use 4096-byte sectors, so a filesystem block usually can’t be smaller than that (btrfs and some other filesystems support “suballocation”). So there could potentially be lots of slack storage space lost to blocks that are only partially filled. However, like btrfs, APFS also supports compression which can help compensate for the lost space.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
