Hello All,
I am experimenting with Syncthing in Kubernetes. I found a couple GitHub gists that run syncthing behind a DIY Nginx rev proxy Pod. I’m adapting those instructions but using the ingress-nginx ingress controller rather than a DIY Nginx pod. I believe everything is correctly set up and I can access the UI in two Kubernetes clusters and sync files between the two Kubernetes clusters. I’m running this image: docker.io/syncthing/syncthing:1.28.
While things appear to function, I’m experiencing something others have reported - which is that the syncthing web UI recurringly pops up a mesage that appears and then goes away all on its own:
Connection Error Syncthing seems to be down, or there is a problem with your Internet connection. Retrying…
If I tail the logs of the ingress controller there is a recurring set of these entries that do not coincide with this browser pop up:
10.112.25.11 - - [29/Oct/2024:14:30:35 +0000] "GET /syncthing/rest/system/connections HTTP/1.1" 200 886 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 600 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 886 0.001 200 aedd5cc03b592c5a98a54052e41dea69
10.112.25.11 - - [29/Oct/2024:14:30:35 +0000] "GET /syncthing/rest/system/error HTTP/1.1" 200 21 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 594 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 21 0.001 200 af3ff628efe2b5be153535178678626a
10.112.25.11 - - [29/Oct/2024:14:30:35 +0000] "GET /syncthing/rest/system/discovery HTTP/1.1" 200 341 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 598 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 341 0.001 200 f9121b00e13980d6c46dffed59a9b0df
10.112.25.11 - - [29/Oct/2024:14:30:35 +0000] "GET /syncthing/rest/system/status HTTP/1.1" 200 2743 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 595 0.002 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 2743 0.002 200 b4ab0489cc6dd4707b9dd3b799355712
Then there is this recurring set of log entries that does coincide with the popup:
10.112.25.11 - - [29/Oct/2024:14:29:21 +0000] "GET /syncthing/rest/events?since=59 HTTP/1.1" 200 0 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 597 60.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 0 60.000 200 2e192f9d926c38c840f758fb213d3400
2024/10/29 14:29:21 [error] 303#303: *291133 upstream timed out (110: Connection timed out) while reading upstream, client: 10.112.25.11, server: _, request: "GET /syncthing/rest/events?since=59 HTTP/1.1", upstream: "http://10.42.0.9:8384/rest/events?since=59", host: "redactedhost", referrer: "http://redactedhost/syncthing/"
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/events?limit=1 HTTP/1.1" 200 254 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 596 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 254 0.001 200 17045ed03563ead3549e19d6d4d57ee3
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/stats/device HTTP/1.1" 200 301 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 594 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 301 0.000 200 468326f2747b7767555420dbec7c108e
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/stats/folder HTTP/1.1" 200 349 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 594 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 349 0.001 200 6a5cb52a498e111e8671c9af87cacb4f
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/events/disk?limit=25 HTTP/1.1" 200 706 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 602 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 706 0.000 200 8fe776d8467e181dc18eab7e290c3c20
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/system/discovery HTTP/1.1" 200 341 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 598 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 341 0.001 200 5a85f8f1b84e1dccb742ffe13a88d2a9
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/config/insync HTTP/1.1" 200 27 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 595 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 27 0.000 200 8472dc76d38ab4d0ba1a5677b9dddc78
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/config HTTP/1.1" 200 9533 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 588 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 9519 0.000 200 031130ab15e8f86c25018bdbf0615257
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/cluster/pending/devices HTTP/1.1" 200 3 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 605 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 3 0.001 200 6ed9b7e799754d7b693267be7bf0093a
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/system/status HTTP/1.1" 200 2743 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 595 0.002 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 2743 0.002 200 632b31adba3e35eb096828f2ce9c52e0
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/cluster/pending/folders HTTP/1.1" 200 3 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 605 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 3 0.001 200 299416f083b56ffcfe23f3317cb3706a
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/system/connections HTTP/1.1" 200 886 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 600 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 886 0.001 200 8b56847f9f9efb04b6280e0118305325
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/themes.json HTTP/1.1" 200 74 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 588 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 74 0.000 200 7d7f7c11f3c8cb31453e8d9d78a630d4
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/db/completion?device=6MWMZHF-SFIRZRB-YSSOLPG-F32ZETY-QQUTJTC-HCBQWAT-6OHIKX6-3Y2HAAN&folder=nvgf4-4jtaz HTTP/1.1" 200 163 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 685 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 163 0.000 200 470f5f20067862ed83fa7336e6aae642
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/db/status?folder=default HTTP/1.1" 200 865 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 606 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 865 0.001 200 7fdc4933fe5d482d0eeda93b3e512b7b
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/db/status?folder=nvgf4-4jtaz HTTP/1.1" 200 941 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 610 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 941 0.001 200 034f55d07d1d0ddccc84ea58230125ea
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/system/version HTTP/1.1" 200 455 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 596 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 455 0.000 200 eec0455a8f12c72250f0f18cafcda173
10.112.25.11 - - [29/Oct/2024:14:29:22 +0000] "GET /syncthing/rest/system/upgrade HTTP/1.1" 501 20 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 596 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 20 0.000 501 de0ac4c00e3691e09f85853514f6a84f
10.112.25.11 - - [29/Oct/2024:14:29:23 +0000] "GET /syncthing/rest/svc/report HTTP/1.1" 200 1680 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 592 1.310 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 1680 1.310 200 e3db01d35292406eb60ff9e58d2b9c78
10.112.25.11 - - [29/Oct/2024:14:29:25 +0000] "GET /syncthing/rest/stats/folder HTTP/1.1" 200 349 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 594 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 349 0.001 200 c472bb538b8739df6159bf88645cc8fc
10.112.25.11 - - [29/Oct/2024:14:29:25 +0000] "GET /syncthing/rest/system/error HTTP/1.1" 200 21 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 594 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 21 0.000 200 1dd464787e41684a444280257be04be8
10.112.25.11 - - [29/Oct/2024:14:29:25 +0000] "GET /syncthing/rest/system/connections HTTP/1.1" 200 886 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 600 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 886 0.000 200 6ae4f45b830fe8c7ed3c58dc5a843a61
10.112.25.11 - - [29/Oct/2024:14:29:25 +0000] "GET /syncthing/rest/system/discovery HTTP/1.1" 200 341 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 598 0.000 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 341 0.000 200 4efc67dce7d8a18fc5ac72a912073aff
10.112.25.11 - - [29/Oct/2024:14:29:25 +0000] "GET /syncthing/rest/system/status HTTP/1.1" 200 2743 "http://redactedhost/syncthing/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 595 0.001 [syncthing-web-ui-web-ui] [] 10.42.0.9:8384 2743 0.002 200 3d03c7da0c1c5700f042a83ebc9de258```
The only error I’m seeing in that second set is the 501 on GET /syncthing/rest/system/upgrade. Could that be causing it? The browser is Chrome 126.0.6478.182 (Official Build) (64-bit)
(There are no logs coming from the syncthing pod)
Any ideas or help is appreciated. I’m not sure it’s an actual problem but if this works for us we would want to run it in production and so I want to make sure this behavior is not indicative of some issue with how I’ve installed syncthing that could rear its head later.
Thanks for any help