So I have a super weird one, and I’ve reproduced this fairly consistently so I dont think it’s anything obvious.
I have a forum on subdomain that uses apache and Letsencrypt running perfectly fine using Cloudflare to as just the DNS provider. I want to use some of the CF firewall features to protect form spam/abuse, so I turn on the orange cloud. It works perfectly, I check it form multiple sources and it’s running fine using the Commodo cert etc.
Now here’s where it gets weird, 24hrs later all requests to that subdomain return a 404 The Apache logs make no sense to me, they look like normal 404’s, it should just work. So I turn off the CF orange cloud, clear my DNS etc. And its still 404’ing!!! It requires an Apache restart to fix the issue! I’ve done this about 3-4 times now having exactly the same results. Do you folks have any ideas? I’m pretty sure its a bad config on my server side, but why does it break after 24 hours and not straight away?
Even if you clear your DNS, your DNS server can have their own cache; When you turn off Cloudflare’s cloud service, make sure the TTL remains very low (minutes) to avoid this possibility. The TTL changes from “not in your control” when managed by Cloudflare, to any number of choices when you don’t.
If all your origins are secure (valid TLS), make sure that under SSL/TLS app, the SSL mode is “Full (strict)” - to make sure that Cloudflare always contacts you the same way - over TLS (in Flexible they may drop to plain HTTP in case of transient errors; perhaps your TLS and non-TLS config for the Name-based VirtualHost in Apache are not the same).
Finally, I have not used Apache in years (as I find Nginx to be much better in almost any imaginable way) - I do recall it has an error_log; I don’t remember if it’s like Nginx, that in case of a 404, it tells you what it tried do (i.e. failed to open file /path/to/file) or not. You could try to check that log to get some more hints - 404 from the access log is the end result, not the cause.
Lastly I’ll mention (could be related, could be not) that you may have a daily Apache reload due to logrotate replacing the logs and asking Apache to close/reopen the logs after rotation, by sending it a signal.
It’s not due to the SSL/TLS settings. There’s no connection error or redirect loops, etc.
The Apache logs dont give me any insights. Why would this break with zero serverside changes? It’s so strange.
Nope. I dont have a cron or setting that restarts Apache.
To be clear, its not exactly 24hrs later, it 404’s at some point after I swap to CF orange cloud, after its been successfully served form Cloudflares servers. It’s so strange.
DNS being resolved is correct = you get Cloudflare IP, right?
I did not say that there are connection errors (that you can see). I said maybe between Cloudflare and your origin. Those connections that happen in the background when you browse through Cloudflare’s service. And at that time, Cloudflare (if not set to use SSL mode “Full (strict)”) can bounce between TLS and non-TLS, which may have a different configuration in your Apache (two different VirtualHosts). So I am asking - if you have proper TLS setup - do you use “Full (strict)”? If not, you should. It would be interesting to see if the issue persists even in that configuration.
Yes, I get the CF IP when the orange cloud is on and after some reasonable TTL, the server IP if I disable the orange cloud. It’s working as expected.
The connection errors I was referring to was between CF and the server. The connection between CF and my server is set to Strict, I’ve also tried it on Flexible for a limited time when investigating this issue. But I dont believe this is the issue due to lack or errors on the server or CF side and no redirect loop.
None of these issue would explain why it impacts the unproxied server to serve 404’s on that subdomain AFTER CF is switched off and the server IP is being served by DNS and being requested…I’m more interested in the why than fixing it at this stage
So my direction was alternating between HTTP and HTTPS.
Anyway, what are those files? static assets? dynamic?
Maybe something crazy like mod_security?
If it was plain HTTP, I would tell you to sniff the traffic while it’s returning 404, so we can take a closer look at what the server sees exactly (as browser tools may not show exactly what’s on the wire…)
I’m assuming the virtual hosts are defined by IP addresses and not DNS names, and only ServerName is used to differentiate between the various NameVirtualHosts… (I’m trying to think of things that would change on Apache’s side even though your config did not change)
