Best practises for whole home directory backup?

My current backup solution (déjà dup) has been flaky lately, so I want to try using syncthing, which I’ve been using on a smaller scale, for backup of my entire /home directory. I’d like to discuss best practises for doing this, and have a few questions.

I guess as a first question, is this a reasonable use case for syncthing?

So for the basic setup, I’ll set my laptop as the master and enable versioning. I’d like versions to be stored on my remote host (a raspberry pi), but not on my laptop. I’m assuming this is what will happen?

Here are a few things I am unsure of:

files/directories to ignore

I’ll .stignore some directories like Steam’s app database, but are there any others I should avoid? Particularly, I’m wondering about if files that change often or rapidly might cause an sorts of problems (or high CPU/network load?) If I’m downloading, say, a Ubuntu system image, will syncthing wait until the file is closed by my downloader before it syncs, or will it be constantly working throughout the download?

How about files in .config/syncthing, like the .idx files? If Syncghing is indexing its own index files, might I run into infinite loops on changes or anything like that?

nested repositories

I have a few repos already set up in various subdirectories of my .home; will this cause any problems? If I blacklist these repos with a .stignore, will that .stignore be applied to the repos themselves, making them stop syncing too? Or does a .stignore actually need to be in the given repo for it to apply?

Are there other concerns I should be thinking about?<

Thanks!

For a simple backup solution why not use a simple rsync over ssh/nfs/smb/ftp or something like rsnapshot while on local network with a small bash script ?

syncthing is really useful if you need to connect multiple nodes and repositories.

By default, it will check for change in your repositories every 60’ – avoiding repositories like ~/Downloads may be a good idea.

In my case, I use syncthing to synchronize some specific folders I really care (mainly ~/Documents & ~/Pictures) btw 3 nodes. If you use it to synchronize your all home folder, it might never be idle …

As my own backup solution, one of my 3 nodes implement a rsnapshot solution to a local mount disk (daily, weekly and monthly full backup). I don’t want to use versioning because I find it hard to restore in case of problem (every folder have its own .stversions/*), I prefer to backup to another drive (in case of disk failure) and synching is still in alpha …

So for the basic setup, I’ll set my laptop as the master and enable versioning. I’d like versions to be stored on my remote host (a raspberry pi), but not on my laptop. I’m assuming this is what will happen?

No, I think versioning need to be enable on your Rasbperry Pi (where you actually want them to be stored).

I’d say there are two issues to consider. The first is just the size of things. Syncthing has some scalability issues with really large repositories; if your home directory is hundreds and hundreds of gigs, you may want to hold off on this idea. The second is the nested repositories. Things might get confused. Lets assume a structure

/home/jb/
   photos/
   downloads/
   otherstuff/

where photos is already a repository, and you want to sync the entire /home/jb now. The part of syncthing that manages /home/jb does not know about photos already being a repo, so if a photo is changed on another node it will be synced to the photos dir, detected as a change and published again on /home/jb. This is probably inefficient. If you have the same setup on the other side, I think things will work but it’s not tested - you might find some scenario that lands you in an update loop…

However, you can stick an .stignore files with the line photos in it in /home/jb. The /home/jb repo will then ignore the photos dir, and the photos repo will not know about the ignore file (because it’s not in there).

1 Like

In case anyone is still looking for suggestions on how to do this let me share my setup:

I use syncthing with syncthing-inotify to continuously sync my /home/plouj to a fileserver. The snapshotting/backup stuff happens periodically on the fileserver (via ZFS).

I’ve had to setup the following ignore rules to avoid synchronizing frequently changing temporary files:

.xsession-errors
.config/syncthing
.config/google-chrome/**Cookies-journal
.config/google-chrome/**.log
.config/google-chrome/**.ldb
.mozilla/**.sqlite-wal
.mozilla/**.sqlite-shm
**.sqlite-wal
**.sqlite-shm
**/.stversions
.cache

Plouj,

Did you configure this as a local sync, between two folders on the same computer? (Do you have the fileserver mapped/mounted locally? Or is Syncthing running remotely on the fileserver too?).

If so, how would you add these rules to apply recursively–to all sub directories?

Elika, my working setup (in debian) is as follows: I have a folder ./Sync/home-dir which is synced across all my devices, and I make symlinks from particular items in ./Sync/home-dir to the corresponding items in /home/luke.

Saves you the trouble of setting up the .stignore files and more importantly, for example in midnight commander, you immediately see which files and folders are synced and which are local (because symlinks get a ~ in front of their names)

1 Like

So far so good with my 600GB homedir on OSX. Added some ignores for Firefox’s caches, otherwise it seems to be working alright. Actually doing this with two laptop-homedirs to a ‘central’ FreeBSD server, ZFS snapshots for revisions.

Syncthing certainly does eat a lot of CPU in doing this, and takes a very long time scan, but I’ve set it to scan every 10ish hours.