Were is a spurious node RJR2G2A coming from?

Support
1

Were is a spurious node RJR2G2A coming from? In shows on none of my GUI panels and doesn’t match the list of devices processed at boot up time. I noticed this error because the warning pops up on the syncthing GUI for me to read and click through.

The log below is extracted from 10.0.0.123 (node 3T7HJJ7) as it boots up syncthing. I stripped all but the first 6 characters of node ids for privacy. And I replaced a real remote IP with “a.b.c.d” for privacy. Notice 6 nodes are processed at boot up time and none of them are RJR2G2A. This matches the booting computer and the 5 nodes defined on its syncthing GUI. Then the message at 48.671 tries to connect to RJR2G2A. Where is this node ID coming from? It is not associated with any of my ID numbers on the network, and does not show in the logs of any other nodes on the network.

As an aside, I’m unsure what “did not negotiate bep” means, but that is strange anyhow, because 10.0.0.123 is the computer booting up so why is it calling itself a peer and trying to negotiate something with itself?

[3T7HJ] 2019/08/15 08:26:32.370029 INFO: Device BXBD6EM- is "Offsite Repository" at [tcp://a.b.c.d:22000]
[3T7HJ] 2019/08/15 08:26:32.370198 INFO: Device C2PVFR6- is "Dell-E6440" at [dynamic]
[3T7HJ] 2019/08/15 08:26:32.370300 INFO: Device OCLV7CX- is "bmmbp" at [dynamic]
[3T7HJ] 2019/08/15 08:26:32.370427 INFO: Device O2YGBV4- is "inspiron-linux" at [dynamic]
[3T7HJ] 2019/08/15 08:26:32.370511 INFO: Device 2YNUA3L- is "Elite-Win10" at [dynamic]
[3T7HJ] 2019/08/15 08:26:32.370598 INFO: Device 3T7HJJ7- is "raspberrypi" at [dynamic]

[3T7HJ] 2019/08/15 08:26:32.386921 INFO: GUI and API listening on [::]:8384
[3T7HJ] 2019/08/15 08:26:32.387041 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/

[3T7HJ] 2019/08/15 08:26:33.673739 INFO: Peer at 10.0.0.123:45372-a.b.c.d:22000/tcp-client did not negotiate bep/1.0
[3T7HJ] 2019/08/15 08:26:48.671929 WARNING: Connecting to RJR2G2A- (a.b.c.d:22000): the remote device speaks an unknown (newer?) version of the protocol
[3T7HJ] 2019/08/15 08:26:48.673648 INFO: Established secure connection to C2PVFR6- at 10.0.0.123:48014-10.0.0.115:22000/tcp-client (TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384)
[3T7HJ] 2019/08/15 08:26:48.679449 INFO: Established secure connection to BXBD6EM- at 10.0.0.123:22150-a.b.c.d:43832/tcp-server (TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305)
[3T7HJ] 2019/08/15 08:26:49.455972 INFO: Established secure connection to OCLV7CX- at 10.0.0.123:54666-10.0.0.107:22152/tcp-client (TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305)
2

did not negotiate bep means that the other device has not performed any matching ALPN in the TLS handshake. Combined with the message the remote device speaks an unknown (newer?) version of the protocol it (probably) means that the other side is not a syncthing device. On the other side of the connection is someone who speaks TLS (a webbrowser or webserver for example) but does not understand Syncthings Block Exchange Protocol and is therefore not a syncthing device. That is also the reason why the device id is unknown to you.

From the logs it appears your local device initiated the connection and ended up connecting to something that understands TLS but not syncthing.

Do you know the ip address a.b.c.d? Is it an internal/local or public ip address? Apparently you are connected to another device using that IP so there is a syncthing instance running on the system, but probably under a different port? It’s interesting that the “wrong connection” used port 22000 - this is the usual syncthing port and normally not used for something else. The “correct” connection was later made in the reverse direction - the other side connected to your local instance.

Without detailed information about what the involved networks look like I can only guess, but it seems like this was a spurious issue with your network setup, that some TLS service from another world (possibly https server) was accidentally reached under a ip/port pair which was supposed to reach one of your syncthing instances.

If this does not happen again, it’s pretty irrelevant. If it does happen again it would probably indicate issues in your network setup. I do not believe that this was a TLS-speaking port scan (those can trigger the same messages), as the connection was outgoing and not incoming from the internet.

Edit: I just saw that you’ve manually configured syncthing to connect to a.b.c.d:22000, which is the ‘problematic’ endpoint. Check what is listening on that ip and port.

PS: Just had some thought: Maybe you’ve mistakenly configured the syncthing service on IP a.b.c.d to have it’s GUI listen address on 22000? I guess that would trigger exactly this behavior. If that is not the case, again, check your network and listen addresses/ports. Something is probably misconfigured (or it was just a network glitch?).

2 Likes
3

Numer378,

Sorry for any confusion. The a.b.c.d node is another node of mine at an offsite remote location on the other side of my LAN modem/router/gateway. It’s been peacefully running fine for over a year as a mirror site.

In the post, I just put the letters because I didn’t want to publish my real remote IP address. a.b.c.d is a legitimate Syncthing node with a known ID# that is ~not~ the RJR2G2A number, which is why I do not understand the Warning which associates the RJR number with my known IP address. I check the site’s ID# by logging into its GUI remotely over the internet and using the GUI to see its ID#. It is BXBD6EM “Offsite Repository”, not RJR2GA.

Notice a proper connection is established with BXBD6EM at the a.b.c.d IP addres. The RJR2G2A is spurious. It’s not clear why my 10.0.0.123 even tried to connect with something exhibiting an ID RJR2G2A. Is this what would happen if someone was trying to hack into my syncthing pool of nodes?

4

Numer378, your last paragraph…

I have the a.b.c.d GUI listening on 0.0.0.0:8384. But did you mean GUI or syncthing traffice port?

Isn’t 22000 the normal Syncthing port for sync traffic? Docs say, “you should set up a port forward for port 22000/TCP , or the port set in the Sync Protocol Listen Address setting.” My listening address is “default”.

Wouldn’t it be normal that Syncthing would naturally (default) try to connect on 22000? It’s just that the ID# RJR2G2A found there is spurious. I don’t think anything else would be talking syncthing protocols on port 22000 by accident.

Even if there ~were~ something other than syncthing at 22000 on the remote site, how did something other than syncthing come up with a fully-qualified ID number trying to connect? I thought any node that wanted to be my peer had to have approval to be part of my pool.

5

I keep getting an orange/yellow error message popping up on the GUI of the 10.0.0.123 node (which lives behind my NAT modem/router/gateway and is the only node that talks to the remote node elsewhere on the internet).

Something keeps trying to connect to me. I don’t see a way to attach a screenshot to a post, so here’s the text.

" 2019-08-15 18:16:21: Connecting to RJR2G2A-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx (162.xxx.xxx.151:22000): the remote device speaks an unknown (newer?) version of the protocol"

This RJR2G2A node is not configured in the 10.0.0.123 node list of devices, so I don’t know where it’s getting the desire to connect to RJR2G2A.

6

“Connecting to” is a misnomer, it applies also to incoming connections.

7

Would it make sense to replace that log text with Connecting with, that is probably less confusing?

Or (if at that place in the code this knowledge is available), use connection to and connection from ?

8

Sure, or “talking to” or something. The distinction in from/to doesn’t really exist or make sense in the quic case or (possibly upcoming) holepunched tcp either.

9

It’s indeed very curious that for the same address tcp://a.b.c.d:22000 there is a successful, expected connection Device BXBD6EM- is "Offsite Repository" at [tcp://a.b.c.d:22000] I got that wrong, that’s got nothing to do with connection, that’s the startup message.
and a totally spurious connection attempt:

Peer at 10.0.0.123:45372-a.b.c.d:22000/tcp-client did not negotiate bep/1.0
Connecting to RJR2G2A- (a.b.c.d:22000): the remote device speaks an unknown (newer?) version of the protocol

I think an easy, but unsatisfactory solution would be to change the logging: Don’t warn about incompatibilities, if the connection is anyway useless, i.e. not for a device we care for.

@increa Did this problem come up recently? And what version do you run on 3T7HJ and BXBD6EM?

10

Summary of my point of view:

  1. You’ve configured some local syncthing instance to always connect to Offsite Repository at [tcp://a.b.c.d:22000]

  2. Because you’ve configured this, syncthing immediatly connects to this endpoint on startup, expecting to reach Offsite Repository (device ID BXBD6EM).

  3. The TCP connection is successfully established, then a TLS handshake follows, which is also sucessfull but has anomalies (no BEP negotiation, which indicates the other peer is not syncthing).

  4. Node ID's are derived from the TLS certificate. We expected to connect to a peer with a TLS certificate BXBD6EM but instead we connected to someone holding a certificate RJR2G2A. Again, this indicates the other side is someone else.

  5. Since the certificates do not match, syncthing would now terminate the connection - this node is not authorized. However, the syncthing protocol defines that connections are dropped only after exchanging HELLO messages. Since the other device doesn’t speak the protocol, there’s never a HELLO message received. This causes syncthing to terminate the connection a bit awkwardly with the “unknown protocol version” message. The connection is dropped anyway.

  6. Afterwards, the real Offsite Repository connects to your local syncthing instance (incoming connection from the local point of view) and everything is fine.

Summary of summary:

The outgoing connection to Offsite Repository ends up somewhere were it should not end up and terminates with an ugly message. Since this is outgoing this is most likely not an attack. Offsite Repository can establish a connection to your local syncthing instance perfectly fine (in the reverse-direction).

Another possible cause: Maybe some port forwardings are incorrect (do you use port forwards)? Maybe your router or whatever does the forwarding goes crazy? There are quite a few routers which go crazy when port mappings do unusual things, e.g multiple devices attempting to map the same local port.

Also, what is the reason for the static IP and port? Global discovery should find this info automatically.

11
12

Imsodin,

The two nodes you ask about are both raspberry pi computers running ARM versions of Syncthing. The local 3T7HJ is v0.14.54. The remote BXBD6M is v1.1.1. Nothing has changed here for months, and the error messages started this week.

Locally, 3T7HJ aggregates data from one syncthing peer group, encrypts it,and makes the encrypted data available to BXBD6M, which is the remote site.

For the encrypted folder, only the two raspberry pies are configured into the syncthing peer pool. There never has been a 3rd peer let alone a peer with the ID RJR2G2A.

That’s what’s confusing is something is making up a syncthing ID# and pushing it at me on the normal syncthing data port 22000.

I thought maybe the remote computer was somehow running a duplicate copy of syncthing but looking at the processes indicates only one copy running.

13

Numer378,

Well… Your point #1 about configuration challenges me. There are only two syncthing instances running, and I can look at the GUI and logs and nothing indicates the phantom ID# is configured by me. Calmh clarified the error message text; this could be a connection initiated outbound or inbound. Seems the rest of your numbered points clarify what is happening, but not why.

Your summary summary says it’s outgoing, but I now understand it could be either.

About port forwards, when two nodes talk, do they both use the same port number (normally 22000)? Do they have to both use the same port number, or can it be different ports in the two directions?

14

I never said that you configure a connection to RJR2G2A, but you have configured node 3T7HJ to connect to BXBD6EM using the endpoint [tcp://a.b.c.d:22000].

(We can see this, because the log has this message):

[3T7HJ] 2019/08/15 08:26:32.370029 INFO: Device BXBD6EM- is "Offsite Repository" at [tcp://a.b.c.d:22000]

(If it was not statically configured, it would’ve been “dynamic” instead of tcp://…).

Syncthings performs this as you request, but the connection ends up at RJR2G2A instead of BXBD6EM. Due to the nature things work, it can’t know the device ID beforehand -> Syncthing performs a TLS handshake first and then looks at “who is there”. Your config says “I want to always connect to tcp://a.b.c.d:22000 and expect to see BXBD6EM”, which syncthing does as requested, but then sees that there’s an unexpected peer (RJR2G2A) at the other end.

Yes, this exact message does not explicitly say if a connection is inbound or outbound, but there is other data available which proof that this is an outgoing connection:

  1. The connection to the unknown peer is immediatly established after startup. That likely too fast for an incoming connection, those would only come in a littlebit later (after global/local discovery was made).

  2. You explicitly configured syncthing to perform an outgoing connection to this endpoint.

  3. The log also says 10.0.0.123:45372-a.b.c.d:22000/tcp-client -> /tcp-client means that this instance is the TCP client, ergo this was an outgoing connection (otherwise we would be server). This statement may no longer hold in the future (if TCP punchthrough is implemented), but for now client is always outgoing, server is incoming.

Again, syncthing ID’s are derived from the certificate. Every TLS computer has a syncthing device ID, even https://google.com has a syncthing device ID, because it has a TLS certificate.

Syncthing always accepts connection from anyone (There are some exceptions, but we can ignore these for now), but drops them after a quick handshake if the other node is not authorized (= ID is not known). Same thing in the other direction, syncthing will connect to everyone if it believes that on the other side is a known peer. Whether this is true will only be revealed after the TLS handshake.

Edit: But yeah you’re right, it’s really confusing why there is someone else on that port/IP pair.

To understand this we need to talk a little about TCP and how syncthing uses TCP:

First, TCP differentiates between server and client. In strict TCP logic, you can either setup a TCP server or a TCP client on a given port. You cannot do both at the same port. A TCP server can support only incoming connections, but multiple at the same time. A TCP client can only perform one outgoing connection.

In Syncthing, you can configure the port for the TCP server. That is by default 22000. This is the port used for incoming connections (i.e connections that go to the server). However, syncthing is also capable to perform outgoing connections to other syncthing computers. For these, syncthing creates TCP clients as needed (one TCP client for each outgoing connection) and the port number used is random/choosen by OS kernel.

So syncthing connections normally look like this:

Outgoing:

TCP-client @ 1.1.1.1 port 552323 [random] | connects to | TCP-server @ 2.2.2.2 port 22000

Incoming:

TCP-server @ 1.1.1.1 port 22000 | accepts connection from | TCP-client @ 2.2.2.2 port 43215 [random]

Syncthing allows these to combine freely, e.g any direction is fine -> whatever works first is used. Syncthing doesn’t care about who is server or client, but TCP requires the usage of these concepts.

So the general answer to your question is: There are different ports for both directions, one port is static and the other random.

Question: What other things are inside the network where Offsite Repository is in? It is possible that something inside the network is interfering with the incoming connection to Offsite Repository. Some SSL/TLS scanner maybe? Apparently the issue is only in one direction, in the other it works fine.

Sadly, I don’t know what your network is doing.

1 Like
15

Nummer378, your description details are important to clarify the operational architecture of Syncthing. The sentence,I want to always connect to tcp://a.b.c.d:22000 and expect to see BXBD6EM clicked in my head. Although I specify BXBD6EM in the configuration, that information is not used in any way. Rather my syncthing simply goes there and tries a connect to the port and sees what is there and then responds.

Eventually the exchange described above does happen and BXBd6EM works fine. However, the prior spurious initiation of RJR (unknown from which side) is still unexplained.

You pointed out that “tcp-client” means my local aggregator 3T8 is initiating the spurious connect to RJR. This is also supported by the remote site port 22000 on RJR. This is also supported by the fact that the attempt quickly happens after the local node boots up.

For reasons, I cannot remember, on my local aggregator, I explicitly specified port 22000 as the destination server port when connecting to BXB. I wonder if my explicit declaration of the default standard somehow is tripping up syncthing.

So far I have:

Currently, local node 3T7:45372 reaches out to remote BXB:22000. Something responds but is not interpretable by syncthing.

Later, remote BXB:43832 opens a link to 3T7:22150 and it works fine.

The two nodes BXB (remote) and 3T7 (local) point at each other. What causes one side or the other to be the first to open the link? Maybe if I can get the remote side to try first, then the bogus RJR connect attempt will never happen.

I will poke around and see if I can figure out what else is on the remote site, possibly intercepting port 22000.

Also, it shouldn’t matter, but I’ll delete the explicit declaration of BXB:22000 and let the default option accomplish the same thing. Maybe it’s confusing syncthing in some way. I was thinking explicitly naming would keep me off the syncthing servers so the link would work even if the servers were off line.

Ya’ know… maybe related, but I’ve also noticed that sync progress has stopped between 3T7 and BXB. Remote BXB thinks it needs 258 files (84 MB). Remote BXB (only folder) is send-receive and 3T7 has the folder designated send-only.

16

A solution I think!

First, I paused all the syncing and did an rsync to directly update the remote directory because there were some deletions at the remote site that kept trying to propagate back to my send-only aggregator.

But for the big fix, Numer378’s thoughts led me the right way. At the remote site, the modem/router forwarded port 8384 in to the computer port 8384. But… oops… the router also forwarded 22000 to port 8384.

So I think the spurious reply happened when my local site tried to connect to remote port 22000 and the GUI console responded. If anybody has a moment to play, try setting one of your sites to map port 22000 into port 8384 and see if you get a spurious node of the same ID#.

1 Like
17

Sounds plausible. The ID won’t be the same though, because the HTTPS cert/key is generated per-device on first start, just like the “regular” cert/key.

18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.