Disconnected at work

Support
1

I have 3 machines: A. 1 Mac at home with the latest Mac OS X. B. 1 Mac at work with the latest Mac OS X. C. 1 Windows 7 machine at work.

B and C (at work) can see each other, connect, and sync as advertised.

A seems to know about B and C, and B and C seem to know about A, but A is always disconnected on B and C. B and C are always disconnected on A.

UPnP seems to be working on my home machine, because my router shows a UPnP rule to forward 22000 to A. Just in case, I added a manual rule for the same port forward and also a 21025 UDP rule. No change.

At work I have no control over the network.

I assume that is the problem and that the network here will not ever let whatever needs to come in and out.

I note that at home, A reports that it is looking for B and C both on the same IP:port, and it seems to be the right public IP for my work machines. From what I recall about IPs and ports, though, that cannot work, right? Coming into a network with 2 machines both running syncthing from outside the network must require different ports for the two machines or the packets can’t get routed, no?

So I’m not surprised that my home computer can’t connect to the work ones. But I have no idea why my work computers cannot connect to the home one.

I know they use different connection methods, but Bittorrent Sync worked between all 3 machines (except 2.0 is a piece of garbage that kept crashing and leaving files out of sync and unsyncable requiring manual correction, hence my investigation of syncthing).

Anyway, does anyone have a link to the basics of what I can do to troubleshoot?

This is what STTRACE=beacon ./syncthing produces:

[monitor] 2015/06/25 11:39:41.782182 monitor.go:94: INFO: Starting syncthing [SMJ3B] 2015/06/25 11:39:42.248113 main.go:473: INFO: syncthing v0.11.10 (go1.4.2 darwin-amd64 default) unknown-user@syncthing-builder 2015-06-21 09:45:54 UTC [SMJ3B] 2015/06/25 11:39:42.248317 main.go:474: INFO: My ID: SMJ3BZF-Y5BK5GC-CB3GOTZ-TL2ELTT-PMJWICR-6QP6REW-NWWWGL2-YMU2DQX [SMJ3B] 2015/06/25 11:39:42.261954 main.go:755: INFO: Database block cache capacity 32768 KiB [SMJ3B] 2015/06/25 11:39:42.300925 main.go:631: OK: Ready to synchronize IC_notes (read-write) [SMJ3B] 2015/06/25 11:39:42.301072 main.go:810: INFO: Starting web GUI on http://127.0.0.1:8384/ [SMJ3B] 2015/06/25 11:39:42.323165 rwfolder.go:295: INFO: Completed initial scan (rw) of folder IC_notes [SMJ3B] 2015/06/25 11:39:42.728699 main.go:886: INFO: Starting local discovery announcements [SMJ3B] 2015/06/25 11:39:42.729058 broadcast.go:74: DEBUG: broadcastWriter@0xc208128100 starting [SMJ3B] 2015/06/25 11:39:42.729224 broadcast.go:163: DEBUG: broadcastReader@0xc2081280e0 starting [SMJ3B] 2015/06/25 11:39:42.730329 main.go:891: INFO: Starting global discovery announcements [SMJ3B] 2015/06/25 11:39:42.730648 broadcast.go:114: DEBUG: addresses: [192.168.191.255] [SMJ3B] 2015/06/25 11:39:42.730819 broadcast.go:140: DEBUG: sent 56 bytes to 192.168.191.255:21025 [SMJ3B] 2015/06/25 11:39:42.731126 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%lo0]:21026 [SMJ3B] 2015/06/25 11:39:42.731351 multicast.go:61: DEBUG: write udp6: can’t assign requested address on write to {ff32::5222 21026 en0} [SMJ3B] 2015/06/25 11:39:42.731541 multicast.go:61: DEBUG: write udp6: can’t assign requested address on write to {ff32::5222 21026 bridge0} [SMJ3B] 2015/06/25 11:39:42.731734 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%en2]:21026 [SMJ3B] 2015/06/25 11:39:42.731919 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%utun1]:21026 [SMJ3B] 2015/06/25 11:39:42.731965 broadcast.go:192: DEBUG: recv 56 bytes from 192.168.191.105:63292 [SMJ3B] 2015/06/25 11:39:42.732251 main.go:677: INFO: Device F2VAP6H-ELGRGNV-I7VG7N5-QH574CF-47F4F3J-PZU2LHL-NZTYMRQ-NE5YVAB is “HHLWS100” at [dynamic] [SMJ3B] 2015/06/25 11:39:42.732408 main.go:677: INFO: Device SMJ3BZF-Y5BK5GC-CB3GOTZ-TL2ELTT-PMJWICR-6QP6REW-NWWWGL2-YMU2DQX is “Trevors-MBA.local” at [dynamic] [SMJ3B] 2015/06/25 11:39:42.732570 main.go:677: INFO: Device ZU7XBOE-XON2VZ2-YXH4HJN-NDISWOZ-L4TWTYW-FJCVSBQ-V7KYQQG-FMA7QQU is “Mac Mini” at [dynamic] [SMJ3B] 2015/06/25 11:39:42.732780 usage_report.go:154: INFO: Starting usage reporting [SMJ3B] 2015/06/25 11:39:42.744091 gui.go:219: INFO: API listening on 127.0.0.1:8384 [SMJ3B] 2015/06/25 11:40:04.272512 broadcast.go:192: DEBUG: recv 56 bytes from 192.168.191.104:60832 [SMJ3B] 2015/06/25 11:40:04.273310 broadcast.go:114: DEBUG: addresses: [192.168.191.255] [SMJ3B] 2015/06/25 11:40:04.273433 broadcast.go:140: DEBUG: sent 56 bytes to 192.168.191.255:21025 [SMJ3B] 2015/06/25 11:40:04.273470 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%lo0]:21026 [SMJ3B] 2015/06/25 11:40:04.273527 broadcast.go:192: DEBUG: recv 56 bytes from 192.168.191.105:63292 [SMJ3B] 2015/06/25 11:40:04.273740 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%en2]:21026 [SMJ3B] 2015/06/25 11:40:04.274215 multicast.go:61: DEBUG: write udp6: can’t assign requested address on write to {ff32::5222 21026 en0} [SMJ3B] 2015/06/25 11:40:04.274252 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%utun1]:21026 [SMJ3B] 2015/06/25 11:40:04.274323 multicast.go:61: DEBUG: write udp6: can’t assign requested address on write to {ff32::5222 21026 bridge0} [SMJ3B] 2015/06/25 11:40:10.382359 connections.go:176: INFO: Established secure connection to F2VAP6H-ELGRGNV-I7VG7N5-QH574CF-47F4F3J-PZU2LHL-NZTYMRQ-NE5YVAB at 192.168.191.105:22000-192.168.191.104:60305 [SMJ3B] 2015/06/25 11:40:10.392766 model.go:626: INFO: Device F2VAP6H-ELGRGNV-I7VG7N5-QH574CF-47F4F3J-PZU2LHL-NZTYMRQ-NE5YVAB client is “syncthing v0.11.10” [SMJ3B] 2015/06/25 11:40:10.392860 model.go:631: INFO: Device F2VAP6H-ELGRGNV-I7VG7N5-QH574CF-47F4F3J-PZU2LHL-NZTYMRQ-NE5YVAB name is “HHLWS100” [SMJ3B] 2015/06/25 11:40:12.731481 broadcast.go:114: DEBUG: addresses: [192.168.191.255] [SMJ3B] 2015/06/25 11:40:12.731716 multicast.go:61: DEBUG: write udp6: can’t assign requested address on write to {ff32::5222 21026 en0} [SMJ3B] 2015/06/25 11:40:12.731753 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%lo0]:21026 [SMJ3B] 2015/06/25 11:40:12.731778 broadcast.go:140: DEBUG: sent 56 bytes to 192.168.191.255:21025 [SMJ3B] 2015/06/25 11:40:12.731849 multicast.go:61: DEBUG: write udp6: can’t assign requested address on write to {ff32::5222 21026 bridge0} [SMJ3B] 2015/06/25 11:40:12.731872 broadcast.go:192: DEBUG: recv 56 bytes from 192.168.191.105:63292 [SMJ3B] 2015/06/25 11:40:12.731903 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%utun1]:21026 [SMJ3B] 2015/06/25 11:40:12.731977 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%en2]:21026 [SMJ3B] 2015/06/25 11:40:34.272615 broadcast.go:192: DEBUG: recv 56 bytes from 192.168.191.104:60832 [SMJ3B] 2015/06/25 11:40:42.734094 broadcast.go:114: DEBUG: addresses: [192.168.191.255] [SMJ3B] 2015/06/25 11:40:42.734331 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%en2]:21026 [SMJ3B] 2015/06/25 11:40:42.734385 broadcast.go:192: DEBUG: recv 56 bytes from 192.168.191.105:63292 [SMJ3B] 2015/06/25 11:40:42.734593 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%utun1]:21026 [SMJ3B] 2015/06/25 11:40:42.734691 multicast.go:63: DEBUG: sent 56 bytes to [ff32::5222%lo0]:21026

Then the last 10 or so lines reproduce forever.

Where do I start?

Or am I screwed if I have no control over my work network?

EDIT: global discovery is on on all 3 machines. B and C read 1/2 for global discovery.

2

So beacon is only todo with local discovery, hence it won’t have anything useful, plus it’s not clear which device the log is from.

What about global discovery on A?

If you run nc -l 22000 on A, and echo test>/dev/tcp/<public ip>/22000 on one of the work machines, does the message arrive, proving that the port forwarding is correctly setup?

As that’s usually the culprit.

3

I’m not sure if I understand the instructions correctly but this is what I did:

On machine A (the one at home) I ran this:

Mac-mini:~ work$ nc -l 22000

I do NOT have syncthing running there right now, because I assumed I just wanted nc to listen to the port.

On machine B (one of the machines at work), I ran this:

Trevors-MBA:~ tbs$ echo test>/dev/tcp/162.249.163.206/22000

Nothing happened on either machine. nc is still apparently listening on my home machine (at least it is showing no messages).

Maybe I got the instructions wrong.

4

Does echo hang? And are you using bash as your shell?

You can try echo test | nc <ip> <port>

5

Using bash I think. Whatever the default Mac OS X shell is.

Echo does not hang. It just accepts the command and returns me to the prompt.

There is no /dev/tcp/ directory on my Mac though.

Your second post makes me think I misunderstood your first post. Are the Echo and nc commands supposed to be running on the SAME machine?

6

No they are not. There is a bash built-in which redirects stuff sent to /dev/tcp/<ip>/<port> to the relevant address and port (atleast on linux).

Given that gives strange results, I suggested to use nc to send the echo message instead.

Basically, what I am trying to do, is to prove that the port forwarding works. The fact that bash returns immediately (probably with a non-zero exit code) to me implies that it’s actually getting a connection refused, meaning that something’s wrong with the port forwarding/firewall.

7

I got home and discovered that my firewall (little snitch) was waiting for me to tell it that nc was allowed to receive connections.

I told it yes, and then was able to test (sort of) that nc works. It only works if I send a packet on port 22000 to localhost but that’s almost certainly because the router my fiber optic internet provider users (zhone) has a bizarre bug where you CANNOT send packets out to the internet and back in through the public IP of my machine. This means that you cannot view the web site I host on my netbook at home from home. I have asked and they refused to admit it’s a bug but said zhone refuses to change/fix the behaviour.

I cannot ssh into my work machine (because I have no control over the network there so can’t set port forwarding on their router) so I cannot test further until tomorrow when I’m back in the office.

global discovery is turned on the home machine (machine A in my original description) and the web interface reports global discovery 1/2, whatever that means.

Both machines B and C are still disconnected though.

8

Just a note: I haven’t tried remotely to connect to my home machine yet, but I did use shields up at https://www.grc.com/x/ne.dll?bh0bkyd2 and it reports that 22000 is open on my machine.

9

But, from work, the echo test does not get through to my home machine.

So the packets are getting stopped somewhere. Hmm.

Thoughts about how to figure out the problem?

10

I mean, can you access anything on your home machine that you are forwarding, just to prove that forwarding works? Because it looks to me that it doesn’t. Best suggestion I can give you is tweaking the router settings.

11

Port forwards work for everything else. That’s how I’m checking things remotely–I ssh into a machine (not the one in question), and hop from that one to the Mac in question. So I have port 22 forwarded successfully. I also have port 80 forwarded to the same machine (again, not the Mac I’m trying to get into).

Just to confirm, I just set up another random port, ran a custom ssh server on machine A (the Mac) listening on that port, and forwarded that port to the Mac in question (machine A). I am now able to successfully log into machine A (the Mac) from my work machine (machine B) on that random port #.

So forwarding works fine, it seems.

Bizarrely, now nc has started receiving messages on port 22000. The only change I made was to change my manual port forward for port 22000 from TCP only to TCP+UPD.

Syncthing has not started seeing my home machine yet though.

Syncthing on the home machine (machine A) is reporting though that UPnP automatically mapped external port 12867 to local port 22000.

12

If you have manual port mapping setup, disable UPnP, as that uses a random port by given by the router, and if you have a hard-coded address, that will obviously not work (though should work via discovery).

You can probably kill syncthing, run nc over the port which Syncthing aquired the UPnP mapping, and see if the nc test passes to verify that UPnP works correctly.

13

That works. At work:

Trevors-MBA:~ tbs$ echo stuff>/dev/tcp/162.249.163.206/12867 Trevors-MBA:~ tbs$

meanwhile, at home…

[QENPR] 10:28:06 INFO: New UPnP port mapping: external port 12867 to local port 22000. ^C[monitor] 11:00:03 INFO: Signal 2 received; exiting Mac-mini:syncthing-macosx-amd64-v0.11.10 work$ nc -l 12867 stuff Mac-mini:syncthing-macosx-amd64-v0.11.10 work$

14

Right, so UPnP works. I guess the next step is to check what address gets advertised to the announce server.

Try setting STTRACE=discovery env var at home, and see what self-lookup returns.

You can also set the env var on the work machine, and see what the lookup for the home device ID returns on that so make sure that its showing the right port and address.

Edit: In the meanwhile, you might want to try disabling UPnP at home and see if it manages to connect given the protocol port is mapped on the router.

You can also hardcode in the address for the device via the web UI on one of the work machines (remove dynamic for now), and perhaps set STTRACE=net to see what it connects to.

15

um…

I have no idea what just happened.

I had written a lot about UPnP maybe not working after all. I had manually entered a forward rule for the random port that syncthing was choosing. I removed it. The machine was rejecting connections on that port.

I was going to give up.

Then I re-ran syncthing on the home machine. Then I sent messages to it via nc, and saw that syncthing was receiving them (I got this each time:

[QENPR] 2015/06/26 11:21:26.576196 connections.go:235: INFO: TLS handshake: tls: oversized record received with length 29797)

Then I tried putting in a manual IP:port in syncthing’s web interface here at work for the machine at home.

A NEW machine popped up in syncthing’s interface here at work, asking to connect. It had a different device ID than the one I had already entered for my home machine.

Maybe I had the wrong ID entered somehow? Dunno.

Then I said OK, and it identified itself as Mac-Mini.local (which would be the right name), so I deleted the older entry I had (the one that wouldn’t connect).

Then restarted.

There were STILL 3 machines listed as peers to my work machine (there should only be 2).

I deleted the one identified only by a machine ID.

Now all 3 machines are synched.

Maybe I have been hacked by someone watching this thread (but I can’t imagine that). Maybe I had the wrong machine ID entered before. Maybe it’s all random.

I’m at a loss.

16

But thank you for your patience!

17