500 internal server errors due to high traffic - not (vastly) improved by Cloudflare?

Hi! Our site https://www.vondt.net/ has seen a big increase in international traffic and is experiencing several 500 internal server errors due to the bandwidth strain. There is around 1.7K hourly unique visitors. We are on “the best dedicated server” on One.com, but they have not gotten back to us with the specs of that server - but we fear it isn’t “all that”. Personally I wouldn’t think 1.7K/hr would be enough to bring down a server? But I fear we might be on “shared server” - even though they say it is ‘dedicated’.

As of today we had Cloudflare Pro activated w/ settings, and unfortunately is just isn’t doing the trick. Even though the analytics shows it is reporting taking away a lot of bandwidth usage from origin server (74% according to CF analytics). But it is still very unstable and it is generating a lot of 500 internal server errors.

Does anyone in here have any ideas on how to improve the situation? The One.com support hasn’t been super helpful to say the least, and only told us to throttle bots (which we have done through Wordfence Premium).

Also, does anyone here use W3 Total Cache with Cloudflare? We have had trouble having both services at the same time - as if we active W3TC along with CF the page doesn’t load. Weird, but perhaps there are some settings we must have gotten wrong. Currently the W3TC plug-in is deactivated.

Hope some of you might have some knowledge around this and suggestions towards a good solution.

All the best,
A

This post was flagged by the community and is temporarily hidden.

Thank you for the quick reply. :slight_smile: We are aware that 500 internal server errors are from the origin. But wouldnt saving a reported 74% bandwidth be supposed to reduce server load and pave the way for a functioning server again? Would you recommend running a Cache-plugin like W3TC along with Cloudflare? Thanks in advance for your reply. Appreciate it.

As mentioned Error establishing database connection is a common cause of HTTPS error 500 (Internal Server Error)

Also, could you provide a screenshot of the error?

This looks like a site that’s primarily for a local audience. It’s…ummm…Swedish?

You might try a firewall rule like this to cut back on unwanted traffic. It will challenge international visitors, but let good search engine bots through.

(ip.geoip.country ne "SE" and not cf.client.bot)

Plan B would be to use APO to cache your pages. It’s $5/month if you are on a free plan, but included in paid plans.

https://developers.cloudflare.com/automatic-platform-optimization/

1 Like

Not necessarily.

If you cause all the static assets like images, css, and js to be cached in Cloudflare you will significantly reduce the bandwidth your server consumes, and probably the total requests per second. But static assets are very cheap to deliver from a regular server. Dynamic content, database lookups etc. are far more expensive to deal with. So you might have reduced the overall number of bits being passed by your origin, but the expensive calls might not be adequately cached.

If I was in your shoes, I’d check the server logs to see what it was actually serving. If you still see large amounts of requests for things that should be cached in Cloudflare, just make sure that there is an appropriate Cache-Control header in place, and those should stop being an issue. I sometimes see things like random number query strings that serve no purpose, and those can be removed in various ways, including with Cloudflare transform rules. You might also look at things like the database slow query logs, to see if there is a potentially poorly performing query.

You already have a varnish tier in front of your server (at least that’s what it looks like to me), so you probably don’t need another HTTP cache.

You might check the setting of your Tiered Cache in Cloudflare, as that should reduce the number of requests to the origin as your user base becomes more global.

3 Likes

Hi Sdayman! Thank you for your answer. Appreciate it!

  1. The site uses GT Translate Premium which creates an abundance of translated pages of a higher-ish quality (which have generated high amounts of traffic from around the world, including India, South Africa, eastern Europe and US). You can see the languages at the bottom right of the page. Primary traffic comes from Norway though.

I was thinking that particularly the distant requests would take up a lot of bandwidth and wreck havoc on the page, but upon installing CF theis situation was not improved unfortunately.

  1. In terms of Plan B. Forgot to mention I am on the Cloudflare Pro w/ APO activated. Still the same fault applies. Meaning the host server is at fault, right? Thinking of that - we first bought APO using the free version, but afterwards upgraded to Pro version - does this leave us at a APO 5 dollars/month plan nullifying its use? (Just thought about that now actually)

  2. Between 12am-2am our traffic was at it lowest, barely 1K unique visitors an hour - which did not result in any “Internal Server Error”. Certainly another clue that its just server overloading? (Which one.com refuses to admit…)

  3. This morning we have activated W3 Total Cache Pro again to see if we can reduce the problem w/ Internal Server Errors. Btw, the Pro version does not seem to add a lot of benefits - except stripping you for 99 dollars a year (pls correct me if I am wrong).

  4. @sdayman : Bots - We are rate limiting / throttling bots quite aggressively (except for Google bots who are whitelisted) using Wordfence Premium.

Thanks again for a good reply though! Appreciate it.

1 Like

Thanks for your reply, @Michael. Very interesting to hear what you are saying about the static versus dynamic requests.

  1. Checking the server logs. We are, unfortunately, hosted with One .com which keeps the server logs etc close at heart. Meaning we don’t really have access to them - and get crappy “first line support” answers when we ask. Like “throttle the bots visiting your site”.

  2. Cache control header: We are using the standard settings from W3 Total Cache for both Browser caching and CSS caching. Is this what you mean?

  3. Tiered Argo Service - This was disabled. I activated it now. Is this free with the Pro plan, or will I get subjected to a lot of “hidden costs” based on amount of GB used? Do you guys have this activated? @michael : Could this take an substantial amount of load away from the origin server?

  4. Status codes for
    “Status codes for requests:
    200 OK: 691.76k
    301 Moved Permanently: 25.08k
    302 Found: 7.9k
    500 Internal Server Error: 6.71k
    499 Client Closed Request: 5.11k”

As you can see we are serving up a lot of bad errors…

  1. Back to APO… I cant seem to find the page for APO anymore after we upgraded to Pro. Is this because it is automatically implemented?

  2. The questions keep on coming… One com claims they have me on a dedicated server, but W3 Total Cache Pro only allows me to use options for Shared Servers. Is this sth one can check somewhere?

Thanks again for being super helpful! Really, really appreciate it!

Argo Tiered Caching is now free for all plans, so no worries.

It should be under Speed tab - Optimization tab in Cloudflare dashboard. Is APO disappeared from the list?

My bad, @erictung ! Found it now. Thanks for pointing it out to me! Much appreciated! On the theme of 1) Speed Opti - would you recommend activating the “Image resizing” even though we are using a third party plug-in for this (Imagify pro)? Also, we have the “Polish” setting deactivated, but perhaps this could be beneficial as well? Or am I getting this wrong… in that Imagify does not serve the same purpose as these two settings?

  1. @michael : Btw, applying the Tiered Caching and meddling with some of the W3TC setings seem to have - fingers crossed - hit the spot. Just ran a speed test which showed a great improvement on the “without Cloudfare”-part. Even though with Cloudflare was still 30% faster. :slight_smile:

  2. Uptime Robot has not reported down time since 7.43 AM (wohoo… it was down and up like a yoyo yesterday…) after making the changes.

Man, this community is so helpful! :smiley: Thanks everyone. Much obliged.

1 Like

A follow up question: The W3 Total Cache plug-in has a ‘Cloudflare extension’. Do you see any benefits of running Cloudflare through this extension versus just managing it here through Cloudflare com?

Thanks in advance for you answers.

Hi again! Hope you all had a good weekend. In regards to my follow up questions - is it better if I ask these as separate questions type “W3 Total Cache Plug-in + Cloudflare Extension”. Or is it OK if I “bump” this post?

Have a great day. :slight_smile:

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.