Questions regarding database size

Hi. I’m new at the forums, so hi everyone :slight_smile:

I have a couple of questions regarding the database size. My scenario is this:

I have a relative with a laptop that has a 500 GB ssd drive (main and only drive), and she travels all around the world because of her job. The info she carries in her laptop is very important to her, so she needed a backup solution. I’ve done it before with Syncthing with great results, so I used it to keep a backup of a specific folder (which is the one she uses for, well, almost everything. Almost all of her info is located inside that folder), and also to backup the user folder (the laptop has Windows 10 Pro, so that way it backs up the content of the Desktop, Documents, etc.).

Both folders are configured as send only folders, and the device is linked to my Linux Mint Server (which has plenty of hard drive space. Folders here are receive only, and every night those folders are copied automatically to a second drive in my server, so if one day Syncthing messes up something, I’d already have a backup of the backup). So I’m using Syncthing as a backup solution in this case (I know it’s not designed for that, but I’ve using it this way with great results for some time).

The thing is that I didn’t know about the minimum free disk space setting, and what impact it has in all this. Well, my relative tends to keep her hard drive full or almost full (2 GB of free space I would say is a good estimation of how much free space she usually has), and I started to notice that Syncthing stopped syncing frequently (all folders appear as up to date in my server, but the device is trying to sync, but can’t). Whenever she frees some space, Syncthing catches up and syncs without issues. I understand that this is not a bug, and it’s the expected behaviour.

My questions are:

  • In this type of scenario (send only folders, 500 GB hard drive, almost no free space), what minimum free space disk would you recommend for the database? I don’t know if the default 1% is too much for this use case (that would be 5 GB, which is too much for this to work. She rarely has more free space than that), but I don’t want to mess things up with a lower value that could make Syncthing to not start at all. I don’t know what could be a reasonable value for this setting in this case.

  • The same question, but regarding the value for every folder. Should I set roughly the same minimum free space than what I chose for the database?

  • I get that the database is needed on both ends and that it has to have a minimum of disk space reserved for it, but in the case of folders set as send only (which wouldn’t have the problem of receiving files that can get corrupted because of lack of free space), is it a problem if I set that value to zero (or a very small size)? I repeat, this would only be for send only folders in a back up scenario. The receiving end (server) has 1% for both settings (database and folders), and plenty of free disk space.

Thanks in advance!

PS: Sorry for my English.

Just make sure to not run out of space and you’ll be fine. What threshold to use for that is a personal choice. If you want some absolute minimum I’d maybe start with 2x the database size.

1 Like

My general recommendation would be to install a larger SSD. Being this severly limited in space can trigger all sorts of weird behavior with apps (e.g app trying to write to temp folder, but can’t). For SSD it also means that the disk is limited in the amount of write optimizations (wear leveling or whatever) it can do (unless the disk is overprovisioned), reducing both disk performance and lifespan. Manufacturers often recommend leaving somewhere between 20-25% of free space for “maximum performance”.

With a larger disk I believe most of the worries will simply go away and fiddling with disk free settings will hopefully no longer be neccessary. I believe in the long run this can dodge many bullets.

1 Like

Thanks for the quick replies.

Honestly, she’s not that organized, and I can assure you that even if I install a 2 TB disk, it will be full in less than a week. I don’t want to mess with how she manages her storage, and if the ssd spanlife is reduced or if she starts having weird program behaviour because of lack of free space, that’s something that she has to take care of (and she knows that when that happens, she has to free some space). In other words, I want to help her with the backup solution, but I’m not going to be her mother XD. She already knows that she has to have more free space on a regular basis, so that’s her responsibility. I just want to make sure that, under those circumstances, Syncthing works even with low free disk space, without corrupting files or the database. So I’m trying to find a reasonable value for that setting that could also be considered safe.

The other thing is that she’s in the other side of the word right now, and with the pandemic I don’t think I’m going to see her again in a while (so no chance to change the hard drive by myself. And again, that new hard drive would be full in like a week, that’s for sure).

To sum up, my worries are setting a too low value for the database and get corruption issues or Syncthing not starting at all (and also settting a too high value that makes Syncthing to not sync for days until she frees some space), and also if setting a very low value (or zero) for a send only folder has any impact at all or not (I would think no, since there’s no chance that that folder could receive corrupted files because of low disk space. After all, it’s a send only folder).

Thanks again.

2 Likes

Syncthing is exactly made for that (well at least 2/3 maintainers do basically the same). It’s just terminology: You don’t have a backup of the backup on the server: You have a synced copy of the data on the server and backups of that :wink:

There’s different settings for folder and db free space. For Syncthing itself to stay operational you only need to set the db free space appropriately and it will stop everything when that is reached (except sync deletions).

And for the SSD free space/trim issues: Just shrink the partition xD

3 Likes

For this kind of a special case, I would consider creating a separate partition just to store the database. If necessary, the partition may be even hidden, i.e. remove the drive letter and access it by the UNC path (see https://forum.syncthing.net/t/handling-a-changing-drive-letter-windows/16624), so that the user cannot mess with it.

1 Like

Thanks again for your reply. Yeah, you’re right, I didn’t use the terminology properly (again, sorry for my English :grimacing:). I set the server that way, so it always has a “mirror” of the other device’s folders, but also a daily copy of those files in another drive for safety purposes (I use Rsync for that). Idk, she could delete everything by mistake, a buggy build of Syncthing may corrupt some files someday, etc, so I feel more safe having a copy of the synced files elsewhere.

Regarding the db free space (I’m aware there is a setting for the db and one for every folder :slight_smile: ), that’s my main question. I may be wrong, but I understand that setting a value that’s too low could cause issues with the db. But if I set it at a value of, like, 2 GB, I’m pretty sure that Syncthing will not be syncing that often (in other words, and idk why, she manages and apparently likes to have 2 GB or less of free disk space, no matter what :man_facepalming: ).

What would you think would be a reasonable and safe value for the minimum free disk space setting for the db, considering it’s a 500 GB drive (and like 400 GB of it are located in the synced folder), and that the disk will almost always be full? 1 - 3 GB of free space is what she’s used too (again, idk why). And like I said, even with a 10 TB disk, I can assure you it will have 1 - 3 GB of free space probably in less than a month. That’s her problem (well, apparently for her, it isn’t), I’m just figuring out the backing up solution :laughing:

I know it’s kind of a tricky question, but that’s my situation :laughing:. I just want to make sure Syncthing keeps syncing even if she has like 2 GB of free space, without setting a value that’s too low either (I understand that by doing so there could be db issues). Is a setting of 1 GB reasonable in this scenario? I really don’t know how big a Syncthing db can get if it has to sync an entire 500 GB disk.

Btw, what about the setting at folder level? I mean, I know its purpose, but can you set it to zero (or to a very low value) if it’s a send only folder? Those folders won’t be receiving any files (so no chance of corruption if she runs out of disk space), so I would think you can. But clearly I’m not an expert, so I prefer to ask.

Good idea, haven’t thought of that. But sadly, it’s not a solution in my case, since I would have to do it myself, and she’s literally on the other side of the planet right now (and she barely knows how to use Office).

Anyway, the question still remains, which is the proper size for that partition. What would be the lowest size you would use in my case, that you consider to be safe? So you don’t eat too much of her disk space creating an unnecessarily big partition, but also not too small so the db can get corrupted or something like that, and considering that the folders to be synced (send only on her side) are roughly 400 GB of files. I really don’t know how big a Syncthing db can get in that scenario, not even a clue.

Thanks for your reply. For example, my PC Syncthing db (that has nothing to do with the server and her laptop, just an example) is roughly 500 Mb now. If I was on her side and using this PC (or I was the one that, without any reason, likes to permanently have 1 - 3 GB of free disk space regardless of the disk size), would you say that 1 GB would be the lowest setting that you consider to be safe then (db, not folder)?

In this particular case, the folders to be synced are two, and they are always going to be like 400 GB of files. I trust that most of the time she’s going to have at least more than 1 GB of free space left (that’s a risky bet). 1 GB would be enough so the db size never gets to fill that space? I really don’t have a clue how big a Syncthing db can get, so I wouldn’t even know if a setting of 1 GB is too much or too less.

Btw, I really appreciate (I mean it) the advices about creating another partition for the db or to just free more disk space, but sadly those advices can’t help me with this. For one thing, she knows how to use Office, and that’s about it (so creating another partition is Nasa stuff for her), and for another, freeing disk space… well, more than 5 GB of free disk space for more than 5 minutes, I don’t think that will ever happen :man_facepalming: . That fight is already lost.

I didn’t mean to criticize, quite the contrary: I wanted to confirm that what you are doing is very much an intended use-case for Syncthing :wink:

That doesn’t leave you with much choice then, does it? Sounds like anything above 1GB wont fly, so maybe something like 500-1000MB.

You can - from https://docs.syncthing.net/users/config.html:

Set to zero to disable.

You would probably need some kind of a remote desktop solution to do it yourself. This is what I do, and I take care of some devices located on the other side of the globe. However, if this is impossible (for any reason), and the only way is to give oral instructions to the user or such, then under no circumstances I would try touching any partition-related or other highly dangerous system settings.

Still, if I were to do it myself, then I would probably check the current size of the DB, and then give it least 5x more space just to be extra safe. Of course, this is under assumption that you have no plans to add mass amounts of new data to Syncthing later (which as a result will likely enlarge the DB quite a lot).