Rg-listener: [#######-IAD] origin timeout: dial tcp IP.ADDRESS:443: i/o timeout

[GoCruz]

Hey Team, having some issues with railgun. Maybe you all have a quick fix for the issue?

I’m running railgun on a separate server from my web server.

Running Wordpress on all my domains.

When I try to access my site with railgun enabled I get Error 524 each time.

But when I disable railgun the web site works just fine.

Did some digging in the logs and found this error:

rg-listener: [#######-IAD] origin timeout: dial tcp IP.ADDRESS:443: i/o timeout

I tried to change the lan.timeout value in the /etc/railgun/railgun.conf file but it was not there.

So I added it to the bottom of the file and set the lan.timeout value from 30 to 60. Restarted the railgun service but still the same problem.

I also increased the memcached.timeout in the railgun.conf file from 100 to 1000… Restarted the railgun service but still the same problem.

Any help is greatly appreciated.

Cheers!

[sandro]

Is that IP address a Cloudflare one or your own? I’d suspect some firewall setting which prevents the connection.

[GoCruz]

Hey thanks for the response!

Both servers are on the same local network.

And sorry if I forgot to mention that it was working before for 2 months…

And then out of the blue my websites started having issues; just acting weird…

Then some websites would pop on and offline at random…

The first thing I noticed was that I could not get to my Wordpress Admin page (wp-admin). It would report that as offline…

Then it would report the entire website as offline… just weird.

Then when I traced down the logs I saw the error I mentioned in the first post.

But when I saw the error, I went through and troubleshot for an entire day… and when we finally disabled railgun, all the websites worked great…

I thought increasing the railgun timeouts would work but it did not. And that setting was not even part of the railgun.conf.

We recently updated the entire railgun OS and apps. Rebooted but still same problem.

The websites work ok when you turn railgun on but as soon as you visit the Wordpress admin login page… it timeout and knocks the entire site offline.

We will troubleshoot the network and ensure nothing is blocking connectivity between railgun and the webserver.

Thanks again; any help is greatly appreciated.

Cheers!

[sandro]

Which IP address does it mention?

[GoCruz]

That’s the web server’s Public IP address.

[sandro]

All right, so you are connecting from one server to the other. That most likely is some network filter then. Maybe even the fact that you are using a public IP address (assuming there is also a private one).

[GoCruz]

Yes, we have pub and priv IPs. Is there a way to force all communications from railgun to webserver to run over private vs public IP?

And we will check all the ACLs on and report back too. Thank you!

[sandro]

Simply specify the private address, assuming you can reach it from your server.

[GoCruz]

Thanks a million Sandro!

We specified the web server’s private IP address in the /etc/railgun/railgun-nat.conf configuration file.

As: default = web.server.private.ip.address

Restarted the Railgun service and enabled 1 website and it worked great!

We got the railgun headers (cf-railgun & cf-ray) in the websites headers.

We also enabled debug logging to see the traffic in /var/log/messages and we were able to see the traffic passing successfully for the one domain.

It is now setup to work as such:

Web server private IP <–> Railgun Private IP
Railgun Public IP <–> Cloudflare Public Subnets over TCP/2408

However, we could only enable railgun for 3 out of 5 domains…

2 of them are not showing any railgun headers on the website… and when you hit the “test” railgun button it fails…

But i guess I’ll open a different thread to figure that out…

Thanks again for all the help!

[MarkMeyer]

Forget about the test button. When I started using railgun, it told me that it doesn’t work, but it did. Support told me that I should check the headers instead.

What does your nat.conf look like?
Are you able to curl the private IPs from your Railgun host? Alternatively run a tcptraceroute on Port 443.
Are you using haproxy or nginx as (reverse) proxy?

[GoCruz]

Hey Brother; I appreciate the response, thank you.

The only thing in the railgun-nat.conf file is the following:

default = web.server.private.ip.address

And curl works when pulling from the web server’s private IP from the railgun server.

We are not using any proxies.

Thanks again for the assist.

[MarkMeyer]

That’s strange. Are your websites hosted on different servers? Or is there any rate limiting configured?

This is definitely something on network level.

Regarding

@GoCruz

for 3 out of 5 domains…

Add every single domain to your railgun-nat.conf and comment out the default. See If this helps.