'Client.Timeout exceeded while awaiting headers' error on Android w/ self-hosted discovery server

Support Android
1

Getting a ‘Client.Timeout exceeded while awaiting headers’ error from my self-hosted discovery server on my Android phone while the phone is on a VPN. Same VPN works fine from my laptop with Syncthing and my discovery server. Discovery server is reverse-proxied behind Nginx (both in Docker containers). The discovery server otherwise works great with all our phones and computers. This has been going on for awhile, but I unfortunately don’t remember how long :frowning: Just didn’t have time to look into it…

Discovery server was built from Master about 6 days ago, so should be current. Syncthing Android is v0.13.2 (most current available).

Android Syncthing log (with STTRACE=net,discover): http://sprunge.us/hWjc

Android log: http://sprunge.us/hjRc

Let me know if I can provide any other information.

Thanks, Scott

1 Like
2

I suggest you start by checking nginx logs.

3

So after spending some time watching/digging through the nginx logs:

  1. The POST requests for all the clients look something like

    discovery.domain.name ip.add.res.sxx - - [10/Jun/2016:18:58:11 -0700] "POST /v2/ HTTP/1.1" 204 0 "-" "Go-http-client/1.1"

  2. When the phone is on Wifi but NOT on the VPN, I can see its POST requests to the discovery server about every 30 minutes, just like all the other devices. The GET requests for its status by all other devices return a code 200 (successful).

  3. When the phone is on Wifi and on the VPN there are NO POST requests visible from the phone. In addition, after the 30 min elapses without a POST, the discovery server starts returning a 404 to the other clients GET requesting status. I’m assuming this means that the phone gets dropped from the database since it’s not checking in regularly.

  4. Laptop on the same wifi network and on a VPN connection works fine with all GET/POSTs visible in the logs. It’s the privateinternetaccess VPN so both devices have different exit IP addresses.

Ideas? Thanks!

Scott

Edit: just to clarify…I see 3/3 for ‘discovery’ when not on VPN on the phone and 2/3 when the VPN is on.

4

So it probably tries to send the traffic over the vpn. Debug the phone vpn client.

5

Sorry man, I’m confused… I would expect all traffic from the phone to go through the VPN connection and out to the internet and then the discovery server. The global discovery requests are just more HTTP/S traffic, right? Why would the VPN be “eating” outgoing http requests from Syncthing while passing http requests from everything else?

Forgive me if I’m being dense here :slight_smile:

Scott

6

Well it clearly is, given the are no post or get requests from the phone when it’s on VPN. Perhaps it’s a DNS issue when on VPN?

7

I tried a different VPN client (OpenVPN Android version instead of the default PIA client). Same result.

I’m still confused as to how it can be a VPN issue when every other internet connected app on the phone works normally over the VPN? DNS is the same for everything…it’s not a per app setting that I’m aware of.

Was there a Syncthing-Android change between .12 and .13? It used to work correctly.

What else can I do to help troubleshoot this? I’m running out of ideas :frowning:

Scott

Edit: pasting the discovery server URL string with a device ID into Firefox returns the correct response when I’m on the VPN connection.

8

Try replacing the hostname of the server with an IP address. There was a change sometime in 0.12 which fixes DNS on android in general.

9

(reposting this edit here since a reply appeared after the edit).

Pasting the discovery server URL string (non-numeric) with a device ID into Firefox returns the correct response when I’m on the VPN connection.

I’ll try changing to a numeric IP later… Gotta get some sleep…

Scott

10

Can’t try a numeric IP easily since I’m using a subdomain along with multiple services. I’ll try to setup a separate discovery sever and VPN on some DO instances for testing.

@Nutomic - any ideas here?

11

Well, crap. Apparently this has something to do with the connection at work I was using last night. I’m never on the VPN at home, so didn’t test it until I came home. Of course…works fine over the VPN through my home wifi. Guess I’ll have to go test it on other connections, then figure out what’s different about being on the VPN over work wifi that works with (apparently) everything but Syncthing discovery :confused:

1 Like
12

There is definitely an issue…tested on several other Wifi networks and saw the same results as above. I also switched from my self-hosted discovery server to the ‘default’ discovery servers and saw the same timeout. I’m guessing it’s probably working at home because I’m on a fast connection.

I’m wondering if the connection times out when the discovery server is queried just because of the slightly slower VPN connection.

I see the timeout is set at requestTimeout = 5 * time.Second in lib/discover/global.go. Would there be a downside to raising that by a bit (10*time.Second?).

I can attempt to compile an APK to test with that change if you think that’s a reasonable change without any negative side effects for other use cases.

Thanks, Scott

13

If it solves your problem, sure, but five seconds is a lot for a single small HTTP request.

14

Did you actually try numeric IP as I suggested?

15

I spun up a DO instance and tried, but it doesn’t seem easily possible (that I’m aware of) like it used to be with the old discovery server. I used the SSL certs generated by syncthing -generate=. and entered the discovery server IP address into Syncthing as https://xxx.xxx.xxx.xxx/v2/id=?<server id> but Syncthing threw a certificate validation error about ‘not containing any IP SANs’. I’m assuming this is due to the numeric IP. Tried using the -http flag without the certs, but that also threw errors because I don’t think it’s supposed to be used ‘bare’ like that without the proxy in front of it.

I also tried re-compiling the APK, increasing the requestTimeout to 10 and 20 seconds but that didn’t help either :confused:

You’re probably right that it’s some kind of DNS lookup issue, probably taking longer than it should over the VPN. It’s doesn’t seem to be a VPN client problem…tried a couple of different clients. Doesn’t seem to be a PIA issue…spun up my own openvpn server and saw the same results. I’m 80% sure this worked a few months ago, but I can’t easily prove it :frowning:

Guess I’m out for now unless you guys have any other suggestions or someone smarter than me has the same issue and manages to figure a workaround.

Thanks, Scott

16

So we do very nasty hacks for dns resolution on android, which might break under vpn. If you have the ID pinned, it shouldn’t complain about SSL certificates, but I am too drunk and too far from a computer to verify. It might an insecure flag, but best is to check the source.

17

Good for you being drunk and not at a computer! (I’m going to a bachelor party tomorrow) :grinning:

Yeah, it looks like you patched the go source as part of the DNS resolution fix. It works inside a VPN…sometimes (ok, in one place so far: home). I’m happy to be guinea pig if you have other ideas to test out…

I haven’t looked at the source yet, but when I tried the -http flag, I still had the ID pinned in the URL.

Scott

18

Don’t use -http on the discovery server unless you’re fronting it with a proxy like Nginx and that proxy sends the expected headers and so on. Discovery over actual http isn’t supported and that flag is for proxied setups only. What you want to do in the minimal, most insecure case is:

  • Run the discovery server with a self signed certificate for whatever name (but not a numeric IP - those are magic in TLS certificates), such as one that Syncthing has generated.
  • Let Syncthing use it using https://192.0.2.42/?insecure to disable the certificate verification (or using the proper ?id=...).
19

Ah, thanks for the hint for using the ‘?insecure’ flag!

I set the Syncthing global discovery server URL to:

https://numeric.ip.xxx.xxx:8443/v2/?insecure

However…using the numeric IP address for the discovery server on my cell phone behind the VPN still garnered the ‘Client.Timeout exceeded’ error message. Worked fine from laptop behind VPN on the same Wifi network (i.e. I can see the POST and GET messages from the laptop in the discovery server output).

Any other ideas?

Thanks! Scott

20

Having the same issue using OpenVPN client on Android.