System resource monitoring for devices running Syncthing

This is a place to discuss solution to monitor system resource usage.

As a start I’ll quote from a post by n1trux at https://github.com/syncthing/syncthing/issues/6512#issuecomment-612873235:

There are countless awesome and FOSS monitoring tools out there.

@tomasz1986 which tools did you try? How did they not work properly? Is this just about Android? Apparently in Android 7.0 you can only monitor the CPU statistics of the current process. This however could be implemented in the Android GUI and not into the Syncthing core.

Hi,

I’d like to chime in, now after I’ve read the github issue. The first time I saw the “remove CPU / RAM usage from Web UI” commit I also had to swallow my first anger down. But reading through what it really did and thinking it over a second time, it made sense and I saw the reason behind it. It was often discussed in the past on this forum that the CPU/RAM diagnostic values were not really accurate to what really happens when Syncthing is running.

For example, some time ago, I tried to implement a better RAM usage diagnostic in Syncthing-Android and failed. Android doesn’t give “real” values of free RAM because it has its fancy app-2-cache machanisms in place and already returns “modified” RAM values if asked by Java code. The only rough info I could get was from Syncthing’s REST API. And the value is still there, so I don’t understand the common outcry. Perhaps we should deliver a batch script on the forum which queries and shows Syncthing’s RAM usage if someone really thinks it helps him debugging an issue. I doubt that it just boils down to RAM usage. The past on the forums showed me there’s so much more, for example, enabling debug facilities, taking heap profiles and downloading a support bundle is more useful than just saying “it doesn’t sync but I’ve got high RAM usage”.

I pretty well understand the core maintainer’s point in ceasing a somehow “unfinished” feature which consists of functionality everyone of us has got in Task Manager, htop, Android app OSMonitor or alike.

In conclusion, I just miss the CPU usage a little bit. But on Android, it should be do-able with a PR to the wrapper.

Please don’t argue and enjoy syncing :heart: .

P.S.: What’s a Syncthing CPU usage percentage worth if it stays at 5% and your anti virus (Windows) takes up one total core (e.g. 25%) and causes delay.

Ref.: https://github.com/Catfriend1/syncthing-android/pull/176

Example: Query RAM usage from Syncthing’s REST API using Windows PowerShell

function Format-Size() {
    [CmdletBinding()]
    param(
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [double]$SizeInBytes
    )
    switch ([math]::Max($SizeInBytes, 0)) {
        {$_ -ge 1PB} {"{0:N2} PB" -f ($SizeInBytes / 1PB); break}
        {$_ -ge 1TB} {"{0:N2} TB" -f ($SizeInBytes / 1TB); break}
        {$_ -ge 1GB} {"{0:N2} GB" -f ($SizeInBytes / 1GB); break}
        {$_ -ge 1MB} {"{0:N2} MB" -f ($SizeInBytes / 1MB); break}
        {$_ -ge 1KB} {"{0:N2} KB" -f ($SizeInBytes / 1KB); break}
        default {"$SizeInBytes Bytes"}
    }
}

$hostname = "localhost"
$port = "8384"
$apikey = "YOURAPIKEY"

# Query REST API
$headers = @{ "X-API-Key" = $apikey }
$result = Invoke-WebRequest -uri "http://${hostname}:${port}/rest/system/status" -Headers $headers

# Display raw result
$jsonObject = $result.content | ConvertFrom-Json
# $jsonObject

# Display RAM usage
$jsonObject.sys | Format-Size
# Example output: 74,52 MB

#
# Query Web UI
#
# $user = "USERNAME"
# $pass = "PASSWORD"
# 
# Encode the string to the RFC2045-MIME variant of Base64
# $pair = "${user}:${pass}"
# $bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
# $base64 = [System.Convert]::ToBase64String($bytes)
# $basicAuthValue = "Basic $base64"
# $headers = @{ Authorization = $basicAuthValue }
# Invoke-WebRequest -uri "http://${hostname}:${port}/" -Headers $headers
1 Like

I really like Syncthing, and I already expressed my opinion in the GitHub issue, so I will try to keep it short here, but for me this is really all about remote access and remote administration.

Syncthing with its Web GUI makes it very easy to control everything from one place. In my case, this one place is just a simple Web browser on a Windows desktop PC. I use the Web GUI to have a remote view on all of my devices and check whether everything has been synchronising correctly. Having the CPU and RAM statistics there provides useful information, accessing which would otherwise require using specific tools for each platform, which in the case of Android is convoluted, due to the nature of the OS.

Yes, you can use non-GUI shell commands like adb shell top or adb shell ps to monitor resources in Android. The problem is that ADB itself requires a physical connection with the device to be useful. It does work over WiFi, but in this case it needs to be enabled every time in Developer Options manually, which makes it unusuable for any type of remote access, unless you play around with automation, scripts, etc. which is a different story in itself.

I hope that it is OK to revive and update this thread. Since the removal of the CPU and RAM statistics from the GUI I have been reverting the commit, possibly adapting the changes to the current release.

I though that I could upload my patch here, just in case someone else wants to have these statistics in their self-compiled version of Syncthing.

This is the patch for Syncthing v1.12.0, which you can apply as follows.

git am < Revert-gui-lib-api-Remove-CPU-RAM-measurements-fixes-v1.12.0.patch

Revert-gui-lib-api-Remove-CPU-RAM-measurements-fixes-v1.12.0.patch (14.7 KB)

From 63c34f195ac2224e02b19a251f152a003049b87b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Tomasz=20Wilczy=C5=84ski?= <twilczynski@naver.com>
Date: Tue, 1 Dec 2020 22:11:30 +0900
Subject: [PATCH] Revert "gui, lib/api: Remove CPU & RAM measurements (fixes
 #6249) (#6393)"
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This reverts commit c7d6a6d78042876d9a27b13df2ae45783dca7c37.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
---
 gui/default/index.html            |  8 ++++
 lib/api/api.go                    | 14 ++++--
 lib/api/api_test.go               |  7 +--
 lib/api/mocked_cpuusage_test.go   | 13 ++++++
 lib/rc/rc.go                      |  1 +
 lib/syncthing/cpuusage.go         | 59 +++++++++++++++++++++++
 lib/syncthing/cpuusage_solaris.go | 78 +++++++++++++++++++++++++++++++
 lib/syncthing/cpuusage_unix.go    | 18 +++++++
 lib/syncthing/cpuusage_windows.go | 27 +++++++++++
 lib/syncthing/syncthing.go        |  5 +-
 10 files changed, 223 insertions(+), 7 deletions(-)
 create mode 100644 lib/api/mocked_cpuusage_test.go
 create mode 100644 lib/syncthing/cpuusage.go
 create mode 100644 lib/syncthing/cpuusage_solaris.go
 create mode 100644 lib/syncthing/cpuusage_unix.go
 create mode 100644 lib/syncthing/cpuusage_windows.go

diff --git a/gui/default/index.html b/gui/default/index.html
index fe282d48..5aa6feea 100644
--- a/gui/default/index.html
+++ b/gui/default/index.html
@@ -640,6 +640,14 @@
                           </span>
                       </td>
                     </tr>
+                    <tr>
+                      <th><span class="fas fa-fw fa-microchip"></span>&nbsp;<span translate>RAM Utilization</span></th>
+                      <td class="text-right">{{system.sys | binary}}B</td>
+                    </tr>
+                    <tr>
+                      <th><span class="fas fa-fw fa-tachometer-alt"></span>&nbsp;<span translate>CPU Utilization</span></th>
+                      <td class="text-right">{{system.cpuPercent | alwaysNumber | percent}}</td>
+                    </tr>
                     <tr>
                       <th><span class="fas fa-fw fa-sitemap"></span>&nbsp;<span translate>Listeners</span></th>
                       <td class="text-right">
diff --git a/lib/api/api.go b/lib/api/api.go
index e857344e..fc859f9d 100644
--- a/lib/api/api.go
+++ b/lib/api/api.go
@@ -82,6 +82,7 @@ type service struct {
 	connectionsService   connections.Service
 	fss                  model.FolderSummaryService
 	urService            *ur.Service
+	cpu                  Rater
 	contr                Controller
 	noUpgrade            bool
 	tlsDefaultCommonName string
@@ -95,6 +96,10 @@ type service struct {
 	systemLog logger.Recorder
 }
 
+type Rater interface {
+	Rate() float64
+}
+
 type Controller interface {
 	ExitUpgrading()
 	Restart()
@@ -107,7 +112,7 @@ type Service interface {
 	WaitForStart() error
 }
 
-func New(id protocol.DeviceID, cfg config.Wrapper, assetDir, tlsDefaultCommonName string, m model.Model, defaultSub, diskSub events.BufferedSubscription, evLogger events.Logger, discoverer discover.Manager, connectionsService connections.Service, urService *ur.Service, fss model.FolderSummaryService, errors, systemLog logger.Recorder, contr Controller, noUpgrade bool) Service {
+func New(id protocol.DeviceID, cfg config.Wrapper, assetDir, tlsDefaultCommonName string, m model.Model, defaultSub, diskSub events.BufferedSubscription, evLogger events.Logger, discoverer discover.Manager, connectionsService connections.Service, urService *ur.Service, fss model.FolderSummaryService, errors, systemLog logger.Recorder, cpu Rater, contr Controller, noUpgrade bool) Service {
 	s := &service{
 		id:      id,
 		cfg:     cfg,
@@ -125,6 +130,7 @@ func New(id protocol.DeviceID, cfg config.Wrapper, assetDir, tlsDefaultCommonNam
 		urService:            urService,
 		guiErrors:            errors,
 		systemLog:            systemLog,
+		cpu:                  cpu,
 		contr:                contr,
 		noUpgrade:            noUpgrade,
 		tlsDefaultCommonName: tlsDefaultCommonName,
@@ -314,7 +320,7 @@ func (s *service) serve(ctx context.Context) {
 	configBuilder.registerGUI("/rest/config/gui")
 
 	// Deprecated config endpoints
-	configBuilder.registerConfigDeprecated("/rest/system/config") // POST instead of PUT
+	configBuilder.registerConfig("/rest/system/config") // POST instead of PUT
 	configBuilder.registerConfigInsync("/rest/system/config/insync")
 
 	// Debug endpoints, not for general use
@@ -916,7 +922,9 @@ func (s *service) getSystemStatus(w http.ResponseWriter, r *http.Request) {
 
 	res["connectionServiceStatus"] = s.connectionsService.ListenerStatus()
 	res["lastDialStatus"] = s.connectionsService.ConnectionStatus()
-	res["cpuPercent"] = 0 // deprecated from API
+	// cpuUsage.Rate() is in milliseconds per second, so dividing by ten
+	// gives us percent
+	res["cpuPercent"] = s.cpu.Rate() / 10 / float64(runtime.NumCPU())
 	res["pathSeparator"] = string(filepath.Separator)
 	res["urVersionMax"] = ur.Version
 	res["uptime"] = s.urService.UptimeS()
diff --git a/lib/api/api_test.go b/lib/api/api_test.go
index de7e3a80..16265fec 100644
--- a/lib/api/api_test.go
+++ b/lib/api/api_test.go
@@ -113,7 +113,7 @@ func TestStopAfterBrokenConfig(t *testing.T) {
 	}
 	w := config.Wrap("/dev/null", cfg, events.NoopLogger)
 
-	srv := New(protocol.LocalDeviceID, w, "", "syncthing", nil, nil, nil, events.NoopLogger, nil, nil, nil, nil, nil, nil, nil, false).(*service)
+	srv := New(protocol.LocalDeviceID, w, "", "syncthing", nil, nil, nil, events.NoopLogger, nil, nil, nil, nil, nil, nil, nil, nil, false).(*service)
 	defer os.Remove(token)
 	srv.started = make(chan string)
 
@@ -590,11 +590,12 @@ func startHTTP(cfg config.Wrapper) (string, *suture.Supervisor, error) {
 	connections := new(mockedConnections)
 	errorLog := new(mockedLoggerRecorder)
 	systemLog := new(mockedLoggerRecorder)
+	cpu := new(mockedCPUService)
 	addrChan := make(chan string)
 
 	// Instantiate the API service
 	urService := ur.New(cfg, m, connections, false)
-	svc := New(protocol.LocalDeviceID, cfg, assetDir, "syncthing", m, eventSub, diskEventSub, events.NoopLogger, discoverer, connections, urService, &mockedFolderSummaryService{}, errorLog, systemLog, nil, false).(*service)
+	svc := New(protocol.LocalDeviceID, cfg, assetDir, "syncthing", m, eventSub, diskEventSub, events.NoopLogger, discoverer, connections, urService, &mockedFolderSummaryService{}, errorLog, systemLog, cpu, nil, false).(*service)
 	defer os.Remove(token)
 	svc.started = addrChan
 
@@ -1093,7 +1094,7 @@ func TestEventMasks(t *testing.T) {
 	cfg := new(mockedConfig)
 	defSub := new(mockedEventSub)
 	diskSub := new(mockedEventSub)
-	svc := New(protocol.LocalDeviceID, cfg, "", "syncthing", nil, defSub, diskSub, events.NoopLogger, nil, nil, nil, nil, nil, nil, nil, false).(*service)
+	svc := New(protocol.LocalDeviceID, cfg, "", "syncthing", nil, defSub, diskSub, events.NoopLogger, nil, nil, nil, nil, nil, nil, nil, nil, false).(*service)
 	defer os.Remove(token)
 
 	if mask := svc.getEventMask(""); mask != DefaultEventMask {
diff --git a/lib/api/mocked_cpuusage_test.go b/lib/api/mocked_cpuusage_test.go
new file mode 100644
index 00000000..c89a1dba
--- /dev/null
+++ b/lib/api/mocked_cpuusage_test.go
@@ -0,0 +1,13 @@
+// Copyright (C) 2017 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package api
+
+type mockedCPUService struct{}
+
+func (*mockedCPUService) Rate() float64 {
+	return 42
+}
diff --git a/lib/rc/rc.go b/lib/rc/rc.go
index f21d0e31..f953ac85 100644
--- a/lib/rc/rc.go
+++ b/lib/rc/rc.go
@@ -618,6 +618,7 @@ func (p *Process) Connections() (map[string]ConnectionStats, error) {
 
 type SystemStatus struct {
 	Alloc         int64
+	CPUPercent    float64
 	Goroutines    int
 	MyID          protocol.DeviceID
 	PathSeparator string
diff --git a/lib/syncthing/cpuusage.go b/lib/syncthing/cpuusage.go
new file mode 100644
index 00000000..695859c9
--- /dev/null
+++ b/lib/syncthing/cpuusage.go
@@ -0,0 +1,59 @@
+// Copyright (C) 2017 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package syncthing
+
+import (
+	"math"
+	"time"
+
+	metrics "github.com/rcrowley/go-metrics"
+)
+
+const cpuTickRate = 5 * time.Second
+
+type cpuService struct {
+	avg  metrics.EWMA
+	stop chan struct{}
+}
+
+func newCPUService() *cpuService {
+	return &cpuService{
+		// 10 second average. Magic alpha value comes from looking at EWMA package
+		// definitions of EWMA1, EWMA5. The tick rate *must* be five seconds (hard
+		// coded in the EWMA package).
+		avg:  metrics.NewEWMA(1 - math.Exp(-float64(cpuTickRate)/float64(time.Second)/10.0)),
+		stop: make(chan struct{}),
+	}
+}
+
+func (s *cpuService) Serve() {
+	// Initialize prevUsage to an actual value returned by cpuUsage
+	// instead of zero, because at least Windows returns a huge negative
+	// number here that then slowly increments...
+	prevUsage := cpuUsage()
+	ticker := time.NewTicker(cpuTickRate)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ticker.C:
+			curUsage := cpuUsage()
+			s.avg.Update(int64((curUsage - prevUsage) / time.Millisecond))
+			prevUsage = curUsage
+			s.avg.Tick()
+		case <-s.stop:
+			return
+		}
+	}
+}
+
+func (s *cpuService) Stop() {
+	close(s.stop)
+}
+
+func (s *cpuService) Rate() float64 {
+	return s.avg.Rate()
+}
diff --git a/lib/syncthing/cpuusage_solaris.go b/lib/syncthing/cpuusage_solaris.go
new file mode 100644
index 00000000..74a4e280
--- /dev/null
+++ b/lib/syncthing/cpuusage_solaris.go
@@ -0,0 +1,78 @@
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+//+build solaris
+
+package syncthing
+
+import (
+	"encoding/binary"
+	"fmt"
+	"os"
+	"time"
+)
+
+type id_t int32
+type ulong_t uint32
+
+type timestruc_t struct {
+	Tv_sec  int64
+	Tv_nsec int64
+}
+
+func (tv timestruc_t) Nano() int64 {
+	return tv.Tv_sec*1e9 + tv.Tv_nsec
+}
+
+type prusage_t struct {
+	Pr_lwpid    id_t        /* lwp id.  0: process or defunct */
+	Pr_count    int32       /* number of contributing lwps */
+	Pr_tstamp   timestruc_t /* real time stamp, time of read() */
+	Pr_create   timestruc_t /* process/lwp creation time stamp */
+	Pr_term     timestruc_t /* process/lwp termination time stamp */
+	Pr_rtime    timestruc_t /* total lwp real (elapsed) time */
+	Pr_utime    timestruc_t /* user level CPU time */
+	Pr_stime    timestruc_t /* system call CPU time */
+	Pr_ttime    timestruc_t /* other system trap CPU time */
+	Pr_tftime   timestruc_t /* text page fault sleep time */
+	Pr_dftime   timestruc_t /* data page fault sleep time */
+	Pr_kftime   timestruc_t /* kernel page fault sleep time */
+	Pr_ltime    timestruc_t /* user lock wait sleep time */
+	Pr_slptime  timestruc_t /* all other sleep time */
+	Pr_wtime    timestruc_t /* wait-cpu (latency) time */
+	Pr_stoptime timestruc_t /* stopped time */
+	Pr_minf     ulong_t     /* minor page faults */
+	Pr_majf     ulong_t     /* major page faults */
+	Pr_nswap    ulong_t     /* swaps */
+	Pr_inblk    ulong_t     /* input blocks */
+	Pr_oublk    ulong_t     /* output blocks */
+	Pr_msnd     ulong_t     /* messages sent */
+	Pr_mrcv     ulong_t     /* messages received */
+	Pr_sigs     ulong_t     /* signals received */
+	Pr_vctx     ulong_t     /* voluntary context switches */
+	Pr_ictx     ulong_t     /* involuntary context switches */
+	Pr_sysc     ulong_t     /* system calls */
+	Pr_ioch     ulong_t     /* chars read and written */
+
+}
+
+var procFile = fmt.Sprintf("/proc/%d/usage", os.Getpid())
+
+func cpuUsage() time.Duration {
+	fd, err := os.Open(procFile)
+	if err != nil {
+		return 0
+	}
+
+	var rusage prusage_t
+	err = binary.Read(fd, binary.LittleEndian, rusage)
+	fd.Close()
+	if err != nil {
+		return 0
+	}
+
+	return time.Duration(rusage.Pr_utime.Nano() + rusage.Pr_stime.Nano())
+}
diff --git a/lib/syncthing/cpuusage_unix.go b/lib/syncthing/cpuusage_unix.go
new file mode 100644
index 00000000..2b149abb
--- /dev/null
+++ b/lib/syncthing/cpuusage_unix.go
@@ -0,0 +1,18 @@
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+//+build !windows,!solaris
+
+package syncthing
+
+import "syscall"
+import "time"
+
+func cpuUsage() time.Duration {
+	var rusage syscall.Rusage
+	syscall.Getrusage(syscall.RUSAGE_SELF, &rusage)
+	return time.Duration(rusage.Utime.Nano() + rusage.Stime.Nano())
+}
diff --git a/lib/syncthing/cpuusage_windows.go b/lib/syncthing/cpuusage_windows.go
new file mode 100644
index 00000000..6627b8af
--- /dev/null
+++ b/lib/syncthing/cpuusage_windows.go
@@ -0,0 +1,27 @@
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+//+build windows
+
+package syncthing
+
+import "syscall"
+import "time"
+
+func cpuUsage() time.Duration {
+	handle, err := syscall.GetCurrentProcess()
+	if err != nil {
+		return 0
+	}
+	defer syscall.CloseHandle(handle)
+
+	var ctime, etime, ktime, utime syscall.Filetime
+	if err := syscall.GetProcessTimes(handle, &ctime, &etime, &ktime, &utime); err != nil {
+		return 0
+	}
+
+	return time.Duration(ktime.Nanoseconds() + utime.Nanoseconds())
+}
diff --git a/lib/syncthing/syncthing.go b/lib/syncthing/syncthing.go
index e9c5acd2..63f651ed 100644
--- a/lib/syncthing/syncthing.go
+++ b/lib/syncthing/syncthing.go
@@ -409,10 +409,13 @@ func (a *App) setupGUI(m model.Model, defaultSub, diskSub events.BufferedSubscri
 		l.Warnln("Insecure admin access is enabled.")
 	}
 
+	cpu := newCPUService()
+	a.mainService.Add(cpu)
+
 	summaryService := model.NewFolderSummaryService(a.cfg, m, a.myID, a.evLogger)
 	a.mainService.Add(summaryService)
 
-	apiSvc := api.New(a.myID, a.cfg, a.opts.AssetDir, tlsDefaultCommonName, m, defaultSub, diskSub, a.evLogger, discoverer, connectionsService, urService, summaryService, errors, systemLog, &controller{a}, a.opts.NoUpgrade)
+	apiSvc := api.New(a.myID, a.cfg, a.opts.AssetDir, tlsDefaultCommonName, m, defaultSub, diskSub, a.evLogger, discoverer, connectionsService, urService, summaryService, errors, systemLog, cpu, &controller{a}, a.opts.NoUpgrade)
 	a.mainService.Add(apiSvc)
 
 	if err := apiSvc.WaitForStart(); err != nil {

I have updated the patch for Syncthing v1.12.1. A few more changes are required to make it work with each new version, but as for now, everything seems to work as it used to in the past. I have also added the CPU/RAM counters to the untrusted UI.

Patch: Revert-gui-lib-api-Remove-CPU-RAM-measurements-fixes-v1.12.1.patch (15.7 KB) Commit: https://github.com/tomasz1986/syncthing/commit/643350b