"connect: permission denied" v1.6-rc5

Support
1

Cluster of three servers, A, B, & C.

  • A is master/send only, Centos 7, syncthing 1.5.0 Linux x64
  • B is receive/send, FreeBSD 11.2 release p15, syncthing 1.5.0 FreeBSD x64, UFS
  • C is receive/send, FreeBSD 11.2 release p15, syncthing 1.5.0 FreeBSD x64, ZFS

These have been running beautifully for close to 14 months.

  1. B was suddenly throwing out “too many open files” errors, increasing files limits and restart/reboot wasn’t making any difference.

  2. Upgraded B to Syncthing 1.6.0-rc5 to see if that would help.

  3. Suddenly receiving “connect: permission denied” on B for connections to A & C.

  4. Upgrading A & C to 1.6.0-rc5 did not change the error message (restarted all three Syncthing servers). All are running as an unprivileged user, not root.

I’ve checked obvious things like DNS resolution consistency from all three viewpoints, firewall rules (but nothing had changed in the firewall rules when this started).

I’ve since seen it happening briefly between A & B when resuming one folder pair after hitting “Pause All” when seeing “too many open files” errors on B (with ZFS).

Possibly related or not, I’m seeing a new-to-me message under discovery errors: global@https://discovery-v4.syncthing.net/v2/: Post "https://discovery-v4.syncthing.net/v2/": dial tcp: lookup discovery-v4.syncthing.net on 208.67.220.220:53: dial udp 208.67.220.220:53: socket: too many open files

(This does appear to disappear after a few minutes.)

There are quite a few “panic-date-time.reported.log” files in the syncthing config directory, but they are all within about a 2 hour time frame from several weeks ago.

Looking for suggestions and/or troubleshooting ideas.

2

Is your problem. There is plenty of content on the internet explaining how to remedy this. Worst case you might have to disable file watching as it seems your os does not support recursive watches.

3

Thanks for the confirmation.

I did try all the options I found for resolving that first, to no result.

I will try disabling file watching. Did something in v1.5.0 break this, because file watching has been working fine since I first started using Syncthing.

4

So far disabling file watcher for all folders on B and C has made no change in the “connect: permission denied” message for the connection between B & C.

I staggered the intervals for all the folders so they don’t all get rescanned at the same time.

No messages about too many open files.

5

This sounds like a local firewall if anything. Though I’ve never seen this variant on a Unix, might be a FreeBSD specific thing.

6

Definitely firewall – its working now after updating firewall rules.

What is confusing is that the firewall rules haven’t changed in over a year, yet suddenly Syncthing started showing this error. O/S upgrades/updates don’t seem to be at fault here either.

I’ll work with the firewall rules some more.

Thanks for the sanity checks.

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.