I tried to see if this topic has been addressed before, but couldn’t find it. I apologize in advance if I missed it.
I have a NAS running Syncthing to support several remote computers. In general, each remote computer syncs more than one folder. Right now, I can only enable compression on a per computer basis. Since the type of files in their folders tend to vary by folder, it would be nice if compression could be enabled on a per folder basis rather than a per computer basis.
Are there any plans to enable compression on a per folder basis?
I can absolutely see this as a practical feature if that’s what you’re asking.
A folder full of compressible text files…
And a folder full of already compressed image files.
Enabling compression on the image folder is a waste of compute resources. Of course it still works. But.
Disabling compression on a folder of text files is also a waste of reaoirces (network bandwidth.). Of course it still works, but.
I generally disable compression on nearby hosts, and enable compression on distant hosts (like >3000 miles distance) but per folder compression does have practical uses.
No, I’m really asking if you experienced a situation in which this was an actual problem that needed addressing, or just imagined that there could be such a situation. It looks like the latter.
Text files are of course compressible; they also tend to be quite small. Large data tends to be mostly compressed already, with some exceptions (e.g., disk images).
Syncthing’s compression settings offer an easy trade-off between network usage and CPU usage. If you care very much about the lowest amount of network usage, because you are on satellite broadband for example, enable compression and spend a few extra CPU cycles. If you’re syncing large amount of non-compressed data like VM images, maybe enable compression there as well; the worst that will happen is that some CPU work is wasted on non-compressible data. In most other cases the default (compress index metadata, but don’t bother with data) is the better choice.
(Very rarely would I suggest disabling compression altogether.)
We have machine generated data (text) files that are highly compressible and range from a few hundred kB each to a few MB each. These typically compress 90%. We also have automated vision images that are already compressed.
We send these files literally around the world… we leave the compression on because the bandwidth is a bigger limitation than the compute.
We didn’t imagine this situation. This is our actual situation. Of course because the software can’t be configured as the OP is asking we have to make a compromise. Anyway I think the original request is valid. Even if you don’t want it to be valid.
Yeah, but the question @calmh asked is: “What’s stopping you from always having compression enabled?”. It’s not about whether compression is (sometimes) worthwhile or not, it’s whether having it perfectly fine-tuneable for every folder that’s necessary. The compression syncthing uses (LZ4) is fairly light on CPU, so there’s a chance that you can just flip the switch to “on” for all devices and just be happy.
For my particular use case I’ve got thousands of JPEG images that are individually updated as frequently as every minute. The image files are commingled with a small number of database and text files that are synced between multiple hosts at different datacenters.
Syncthing is configured to compress all data because I found LZ4’s detection of incompressible data is efficient enough that the CPU load for similar sized text and image files were within a few percentage points or less of each other.
In addition to lower network bandwidth usage, an unexpected result was that the average CPU load actually dropped slightly – enough to be noticeable in the performance graphs – after enabling compression of all data instead of just metadata.
LZ4 supports multi-core CPUs, plus newer Intel and AMD CPUs include hardware accelerated support for decompression and compression.
