I fear that my question might be rather related to Thunderbird than to Syncthing, but I figured that maybe somebody on the forums already ran into the same issue and has a useful hint.
My issue is as following: I recently set up Syncthing on my Raspberry Pi and several Linux PCs in the local network. I would like to use it to back up all the user data from the PCs to the Raspi. So I configured each PC’s home folder (one PC’s as a test for the moment, to be more precise) to be synced with a folder on the Raspi’s hard disk. The syncing goes only one way (the PC’s only send, the Raspi only receives data).
After fiddling with the .stignore file a bit things are going quite well so far with one exception: Whenever Thunderbird receives a new message, the whole Inbox.msf file (which is currently at around 1GB), along with some other files is changed and thus completely resent to the Raspi by Syncthing.
This is a pity since it creates a lot of local network traffic and produces high CPU load on the Raspi. So I was thinking if there is any way to reduce the traffic. Say, diff the old and new .msf file and sync only the .diff and then attach the diff to the outdated .msf file on the Raspi? Or somehow split the .msf file in two parts (old mails in one .msf, this year’s or month’s mails in another .msf) without interfering with Thunderbird’s way of reading the file? Or anything else which I didn’t think of?
I’d very much recommend against doing that from personal experience: Thunderbird changes and locks various files and if you are like me, you have your email client open pretty much all the time the pc is running. This means the backup client will almost never see a consistent state and thus backups are also not very useful. I’d recommend direct backups on your pc or doing backups on your pie getting messages from the email server.
Are you sure the whole file is transferred: Parts of the file that haven’t changed aren’t actually transferred but copied locally from the old file (unless of course thunderbird completely mangled the file on write, but I’d consider that weird as in very inefficient).
Thanks for the input, didn’t realize that! Is there a way to find out if the backups are broken without restoring them? Possibly Syncthing gives some error messages in such a case?
No, I’m not, I just saw Syncthing reporting that a 1GB .msf file is being synced and assumed that this means that the whole file is being transferred over network. But if there is a mechanism which basically does exactly what I’d like it to do, that is, only transfer the parts that have changed, then the initial problem seems to be settled.
Not really, it might list failed items (reasons like file changed before it could by synced or such) but in general it’s a problem outside of Syncthing. If there is enough time between stopping thunderbird and stopping syncthing (shutting down), you might get a good sync. There’s just no guarantees and it is definitely not an optimal solution - I’d really recommend looking into directly backing up from the mail-server or backing up locally on the pc and then syncing to the rpi or a pre-startup/shutdown rsync task…
See here for earlier discussion about syncing Thunderbird profiles.
Why not reduce your inbox size by creating stable folders? Inbox is then quicker and easier to manage. ST/Thunderbird combo works great for me across several devices.
@imsodin: I see. The problem with backing up the mail server is that the mail account is a very old one registered with a provider who offers no IMAP and only 12MB of storage, that is, all the mails are downloaded by Thunderbird every time it checks for mail and then deleted from the server. Maybe they increased the available storage already so one could configure Thunderbird to keep the downloaded mails on the server, but I’d need to check.
Or maybe fetch POP3 emails and store them in a local IMAP server? I also would not recommend to sync Thunderbird files - just use IMAP directly (and backup IMAP accounts)
@Photon I have a Thunderbird profile of current and archived emails and attachment of about 1.8GB. The profile is shared across three laptops using Syncthing. To be successful and without creating conflicting files, you must close T/B on one device and allow time to sync before opening on another. My use case allows this to happen. I backup the profile (and other directories) using Borg every 24 hours. I share the profile and the Borg backup across each device and a RaspberryPI 3B, running 24/7, using Syncthing. Works for me!
I haven’t tried @uok’s solution but I might, sounds good too.
