Feature Proposal: Smart/Adaptative Resource Usage

Lean and targeted are great qualities but the goal of the syncthing fundation is: to encourage usage of Syncthing.

Anything that helps should be considered, tested, etc.

I like the idea of quick/dirty prototypes.

Actually I think the try-icon that I use to pause would be a good place to put the hack.

1 Like

Little advice from me: Use the word should with caution. When interacting with volunteers, maybe don’t use it at all.

Have a go. As Audrius said, even if it doesn’t go into Syncthing itself, you can make a wrapper. If it gets adopted by lots of users, you get the kudos, and maybe it will also get fully integrated.

2 Likes

Sorry I dont even know whay I use.

Who implemented this:

https://docs.syncthing.net/users/contrib.html. If it isn’t there, please search for it and add it there - thanks.

Did not mean to offend: I guess “it is worth considering, testing, etc”… is what I meant.

As it stands, one of syncthing’s primary principles is “continuous”. Whilst I think that automated pausing/restricting could be useful for some users, I agree with Audrius that it would be bloat for the core project.

However, I would be surprised if nobody else found use from a well-written and documented contribution to make this happen.

1 Like

Syncthing is generally a background service; those use as much resources as they need to complete the mission, and no more. Trying to react somehow to surrounding system activity is a fools errand as we don’t know the cause or intentions of the system/user. As mentioned we can and do lower our priority to indicate that the process isn’t time critical. We also act slightly different on Mac/Windows (using fewer CPU threads) as it’s reasonable to expect those systems to be interactive and more negatively affected by high CPU or I/O usage.

More generally I try to consider “what if everyone did this” as one rule of thumb when thinking about sociable behavior. If everyone stopped processing and idled when the load increased then 1) nobody would get anything done and 2) the system as a whole would become inscrutable and oddly oscillating. We should not do that.

6 Likes

Windows uses three different priorities, i.e. CPU, Memory, and I/O. Syncthing by default starts with its CPU priority set to “below normal”, but does not seem to have any special settings for the other two. I am not sure whether adjusting the Memory priority would have any benefits. On the other hand, a lower I/O priority does help when a mechanical hard drive is used both for data storage, and the OS and program files. It does not really matter on an SSD though.

From a user perspective, Windows only allows to easily change CPU priorities, either with command line or Task Manager. When it comes to the other two, there are external utilities like Process Hacker (open source) or Process Lasso (closed source) capable of adjusting them.

My personal opinion is that Syncthing seems to be optimised mainly for typical desktop machines. This can be seen in its default usage of all CPU cores for hashing, etc. These settings do not play very well with low end hardware (e.g. Intel Atom or slow ARM processors). It also does drain battery pretty heavily on mobile (including laptops). I have recently been using “GOMAXPROCS=1” on such devices, simply because otherwise using them with a lot of synchronisation taking place in the background is very difficult. This does not help with the battery drain though.

All in all, it is probably impossible to have one universal setting working in all cases. This dilemma is not limited to Syncthing though, as Dropbox with all its money and programming man power, etc. is also an extreme resource hog and battery drainer, almost impossible to use on slow hardware (like the aforementioned Intel Atom or similar). It also does not offer any configuration options to change such a state, unlike Syncthing :slight_smile:.

1 Like

Hi:

For any use they are, my feelings on this are:

  • I would love to be able to set a folder priority, so that some folders can always be synced when they have changes pending, at the expense of other folders’ sync operations;

  • As an easy-to-implement partial solution, I would love to have a ‘Pause for 1-hour/4-hours/8-hours’ option in the ‘Pause All’ button;

  • On my NAS units (i.e. spinning disks, low power CPU, lots of files), I always set ‘GOMAXPROCS=2’ on a 4-core unit. Even though Syncthing is set to low priority, I find it can make other services on the NAS extremely sluggish otherwise. However I don’t know if it’s actually CPU contention, or just lowering the CPU limit also limits the disk I/O usage, and that’s what’s actually responsible for the sluggishness;

  • On my ‘office’ workstations (MacBook Pros with SSDs), I never have any issue with Syncthing taking too many resources, and I never have to pause it;

  • On my media production workstations (Mac Pros with SSD OS drives, spinning disks for media due to data capacity requirements), I find that Syncthing does interfere with the responsiveness (and real-time production capability) of the system, and I have to pause Syncthing whilst working on some projects. I suspect this is disk I/O contention rather than CPU or RAM bottlenecking.

3 Likes

I never intended to open a heated debate about what syncthing is, is not, should be… etc.

I just have a factual problem that other users may also be experimenting, and that I would like to solve.

I read all the arguments against Adaptative Resource Usage and I agree with most of it. Thanks for the many comments with potential fixes in the OS, API, etc.

I also realize that Adaptative Resource Usage is somehow a reality, i.e.:

  • Mobile syncthing optionally allows to disable Mobile Networks because it is sometimes a expensive resource.
  • Android will shut syncthing on low battery because last 20% battery is expensive as a resource.
  • Last % of HD space…
  • Users manually pausing/unpausing…

etc. I don’t think the brainstorming around this should upset the community, and certainly it is not my intention to do so.

4 Likes

This morning I monitored a bit closer the synthing daemon and peaked 167% CPU (I don’t know if this is normal or not). Then I tried to tweak the process priority by renicing it to 19.

I left syncthing on during our ZOOM daily huddle and experienced no issues. I am going to keep testing along those lines.

I understand now that continuous is a primary principle, I was not aware of its relative importance.

My problem is that I stop syncthing during high load and then forget to put it back on. Auto Pause for me is not a goal in itself, but rather a mitigating measure for a short period of time.

For example the suggestion:

This would actually be a good workaround. But you should see it as a way to maintain continuity under adverse situation, I mean. I solve my problem while ensuring that syncthing will continue later.

I think it could be a start.

@imsodin @calmh are there statistics of the usage of the pause feature? I could not see anything on data.syncthing.net

At this point I am not sure if this is an issue that affect just a few of us, or many others, if my CPU usage is normal or there is something with wrong with my system, etc.

If there are records that could definitely help.

I think this could be a helpful GUI addition and also improve sync frequency because users don’t forget to resume. If user clicks “pause all” there is a popup selection with

  • permanent
  • 1 hour
  • 4 hours
  • 1 day

That should be enough choices for a typical work day.
A label is displayed e.g. “paused until 11:45”. After timer ends all folders resume.

I could not find an existing issue on Github (but search is not so good), I can create it if you like.

1 Like

What is your use case? Do you pause yourself often? do you forget?

Debate is healthy! It’s how we understand the subtleties of the system and how we use it.

There’s been speculation here by several people that their problem is CPU load, sometimes when they themselves said they “think” it’s the case, so it’s not necessarily a factual problem until empirical testing has been done. It should be easy for affected users to run tests to confirm that they’re not barking up the wrong tree.

One idea that calmh reminded me of is limiting syncthing’s CPU affinity - if you give it only one CPU to work with, it can never peg your CPU however hard it tries. Along with the lower CPU priority, even if your system is busy with foreground work, syncthing’s CPU consumption will be minimised.

I have been trying this kind of thing today. A lot of the feedback is that the System should take care of this kind of thing, so I started by looking at Systemd, as it is running my syncthing.

I could not manage to get Systemd to setup niceness of the process, but I did manage to renice it from command line as a test, and the result seems promissing.

I will try next to check the CPU affinity.

In the .service unit file, in the [Service] section you can use:

CPUAffinity=0
Nice=5
IOSchedulingPriority=5

Sets the process to only use CPU core 0, make it 5 nicer (-20 is the least nice, 19 is the most nice) and gives it 5 I/O priority (0 is highest, 7 is lowest). There’s also IOSchedulingClass but I fear if you give syncthing the lower class, it may never sync if there’s a lot of data to sync :slight_smile:

More details available in man systemd.exec, SCHEDULING section.

When you do those check please also check what the niceness is without changing anything - if everything works it should already be lower than default. If it isn’t there might be a bug.

1 Like

In my system the process is staring with Niceness=11, which is lower priority than default of 0.

Trying Nice=19 in Systemd did not change it. Not sure why yet. I will check with @ProactiveServices settings.