I´m still getting 520 errors across 3 different servers and dozens of sites… and only when cloduflare is turned on. I can´t believe it is taking weeks to fix this
Judged too early. I just got the 520 error again. But I have the feeling that it does happen less quickly. But I hope to get more clarity soon. Because this has been going on for a week.
I also confirm, it happens to me exactly like you. Cloudflare Chile server
it is happening to me on an old server and also on a totally new one, freshly set up at the weekend with directodmin / Centos 7.
I broke my head trying to find some fault in the server. But no error log.
I had to disable cloudflare and the sites were back to normal.
It is something in cloudflare, not in the servers since it is also happening in accounts that have been with cloudflare for months.
I find it really shocking that there is no response at all. I’m going to dedicate another email to it, and I really hope CloudFlare takes action as soon as possible, because this has been going on for a week and a half. One time less often, and the other time continuously.
This is unacceptable!
I experienced this issue few minutes ago when I went with the approach:
- If I made 20-30s “middle-click” mouse, to open the same URL
https://www.treinenweb.nl/
in a new tab, eventually I got 520 erros (are they returned from the origin web server due to your web server and with combination of PHP cannot handle that much requests?)
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0
colo=OTP
http=http/3
loc=HR
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off
If that happens, it would mean that on the origin for example if WordPress is running without “page cache” version of the published post which could be delivered to the visitor as a “.html”, meaning each request is handled directly (loading post data from database, etc.), instead of “hey new visitor, let me have a look in cache … I have a copy of this post in html, here you go” - just an pure example.
Not saying this is the thing, but, just being curious.
Hi there @clay.aar,
Apologies for the delay in response regarding your issue!
I took a look at our logs for your zone on Cloudflare and can confirm that you’ve been intermittently been served multiple 520 errors.
I tried reproducing the error by sending cURL requests from the exact Cloudflare colo/server that recently logged the errors but consistently got successful 200s.
I then tried from Chrome, and after 20 clicks, I saw the 520.
Here’s the output:
curl -svo /dev/null/ 'https://www.treinenweb.nl/nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29' \
-H 'authority: www.treinenweb.nl' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'upgrade-insecure-requests: 1' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'sec-fetch-site: same-origin' \
-H 'sec-fetch-mode: navigate' \
-H 'sec-fetch-dest: document' \
-H 'referer: https://www.treinenweb.nl/nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29' \
-H 'accept-language: en-US,en;q=0.9,es;q=0.8' \
-H 'cookie: _ga=GA1.3.919061867.1628095714; _gid=GA1.3.1080516417.1628095714; __gads=ID=96d348671111e30e-225af75bccc90092:T=1628095714:RT=1628095714:S=ALNI_Ma9Din8PCFuUhDOwZZhDaJVfG99TA; _hjid=ee527ef6-a816-42e2-a39f-79f8179dbbf2; _hjFirstSeen=1; _hjAbsoluteSessionInProgress=0; _gat=1; cf_ob_info=520:67994a30ceb10c0f:DFW; cf_use_ob=443' \
--compressed
* Trying 2606:4700:3033::6815:503d...
* TCP_NODELAY set
* Connected to www.treinenweb.nl (2606:4700:3033::6815:503d) port 443 (#0)
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb2e8010a00)
> GET /nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29 HTTP/2
> Host: www.treinenweb.nl
> Accept-Encoding: deflate, gzip
> authority: www.treinenweb.nl
> pragma: no-cache
> cache-control: no-cache
> sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
> sec-ch-ua-mobile: ?0
> upgrade-insecure-requests: 1
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
> sec-fetch-site: same-origin
> sec-fetch-mode: navigate
> sec-fetch-dest: document
> referer: https://www.treinenweb.nl/nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29
> accept-language: en-US,en;q=0.9,es;q=0.8
> cookie: _ga=GA1.3.919061867.1628095714; _gid=GA1.3.1080516417.1628095714; __gads=ID=96d348671111e30e-225af75bccc90092:T=1628095714:RT=1628095714:S=ALNI_Ma9Din8PCFuUhDOwZZhDaJVfG99TA; _hjid=ee527ef6-a816-42e2-a39f-79f8179dbbf2; _hjFirstSeen=1; _hjAbsoluteSessionInProgress=0; _gat=1; cf_ob_info=520:67994a30ceb10c0f:DFW; cf_use_ob=443
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 520
< date: Wed, 04 Aug 2021 16:52:29 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: cf_use_ob=0; path=/; expires=Wed, 04-Aug-21 16:52:59 GMT
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< cf-ray: 67994be5da140e52-DFW
< server: cloudflare
What is interesting here is that if I remove the cookies, the request completes with no issues:
curl -svo /dev/null/ 'https://www.treinenweb.nl/nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29' \
-H 'authority: www.treinenweb.nl' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'upgrade-insecure-requests: 1' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'sec-fetch-site: same-origin' \
-H 'sec-fetch-mode: navigate' \
-H 'sec-fetch-dest: document' \
-H 'referer: https://www.treinenweb.nl/nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29' \
-H 'accept-language: en-US,en;q=0.9,es;q=0.8' \
--compressed
* Trying 2606:4700:3033::6815:503d...
* TCP_NODELAY set
* Connected to www.treinenweb.nl (2606:4700:3033::6815:503d) port 443 (#0)
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe987010a00)
> GET /nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29 HTTP/2
> Host: www.treinenweb.nl
> Accept-Encoding: deflate, gzip
> authority: www.treinenweb.nl
> pragma: no-cache
> cache-control: no-cache
> sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
> sec-ch-ua-mobile: ?0
> upgrade-insecure-requests: 1
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
> sec-fetch-site: same-origin
> sec-fetch-mode: navigate
> sec-fetch-dest: document
> referer: https://www.treinenweb.nl/nieuws/9082/ns-in-een-jaar-tijd-bijna-2-500-meldingen-via-whatsapp-en-sms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Treinenweb+%28Treinenweb.nl+-+Nieuws%29
> accept-language: en-US,en;q=0.9,es;q=0.8
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Wed, 04 Aug 2021 16:53:36 GMT
< content-type: text/html; charset=iso-8859-1
< vary: Accept-Encoding,User-Agent
< cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=HZSTSASbc2aloefN13K3LB2P3pBg6juraHDZJfU1qqPhZHO76R0VAb4RcMfn8Dq2wgHspJQVOENjHzXAu%2BXRiXioufgWt3H7V1I6Ump3HXRiWaL0TjZ9K9FlR6N50KQ1e3oq4KoYmOA6D1wF%2FvS21A%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 67994d834ee70f4a-DFW
Note, I am hitting the DFW colo, so definitely not isolated to a specific region. Can you take a look at the cookies being sent in the requests and try and determine what the issue may be here? It could be the headers exceeding 32 KB. You may be able to locate the 520 I was able to reproduce based on the RAYID I received: 67994be5da140e52
Let us know if you need further assistance!
Cheers,
Blas
Got it again, via VPN Albania, colo=OTP
:
fl=50f55
h=www.treinenweb.nl
ip=80.246.28.14
ts=1628097658.502
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
colo=OTP
http=http/3
loc=AL
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off
Ray IDs:
67997429ea32ad00
6799742c0b84ad00
679974300e9bad00
679974321825ad00
Again, via Germany, colo=PRG
:
fl=31f79
h=www.treinenweb.nl
ip=216.131.114.111
ts=1628097937.241
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
colo=PRG
http=http/3
loc=DE
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off
Ray ID: 67997bfcddd94113
And more, your colo=CDG
, Paris, via France:
fl=19f523
h=www.treinenweb.nl
ip=185.147.212.76
ts=1628098431.261
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36
colo=CDG
http=http/3
loc=FR
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off
Ray IDs:
6799883ba83540d5
6799883ebe2040d5
6799884038db40d5
Hello Blas,
Thank you for your willing assistance.
The problem has been going on for a week and a half now and I haven’t made any changes to my site. I also looked at the cookies on my site, and in my case this comes out to 500 bytes. So well within limits. Most cookies are even from third parties.
Do you have an idea what it could be? I suspect something is wrong with CloudFlare.
Update 23:09:
Tonight is the prize battle with the 520 errors…
I seem to be getting more errors than usual. I have also already deleted my cookies, although a large part comes from external parties (trackings). The cookies are also well within the limits. yet the 520 keeps returning regularly. It seems that there is something wrong with the Cloudflare cache, as it often happens with new pages that I create in my CMS system (not Wordpress), as well as on sites that I have previously requested.
When I surf in the test environment of my site (same source, same hostingplatform and configuration) , which does not run on CF, I don’t get strange errors anywhere, and everything is loaded neatly. Also there are no errors in the server logs.
I’m convinced it’s clearly an issue within Cloudflare.
I’m curious about Blas’s findings.
Maybe because we have “bombarded” it back few hours with the requests to see what is going on?
Hmm …
Maybe? I don’t know.
I’ve been monitoring the PHP-process on my server until the devilish 520 error, but there was no increase in load or memory there either or outage. I’m not saying it 100%, but I’m going for 99% that it’s not my server, but Cloudflare.
I’ve tried to turn of the Development-mode to turn off the cache. But that seems to make the situation worse. I got the message more often, with a CF ‘cache bar’ at the top of the site more often than a 520 that I got as often as before.
Development mode is now out again.
That wasn’t a solution either.
Just an addition.
It’s not the first time I get these errors. See also this topic from June '19
Then CF staffer Cloonan said:
"The team found an issue with a peering partner. Traffic returned from that partner path wasn’t making it back to Cloudflare and resulted in the error. We’ve made routing changes and advised the partner of the issue so that they can address. "
Maybe this is a trace to to the problem. But I can assure that no outgoing firewall is running on my server, so maybe in a path from my serverhosting (VIP Internet, Provider.nl) to Cloudflare?
That particular thread you linked to was a very specific issue causing all requests to fail with specific providers. I’m pretty sure that’s not the case here because of how the issue surfaces.
I asked @1blas to take a look at this and I believe their answer points to the issue being outside of Cloudflare. While it is Possible for a 520 to be caused by Cloudflare, it’s very unlikely.
Based on this and the reply to me, I don’t believe this is a Cloudflare issue, but must be caused by something on your origin’s side. This can be very difficult to debug and I understand your frustration with it, but I’m not really sure how else we can help here.
As I mentioned before, this cannot be coming from my server. I’ve been researching for hours and haven’t found anything.
- The test environment, which is separate from Cloudflare, on the same hostingplatform, same hosting-configuration, same codebase, same server works fine.
- No errors in my (server)log
- No outages of services, such as PHP/http on my server
- No overload on the server
- No outbound firewall blocking CloudFlare IPs. Anyway, that’s different because it happens randomly, and I don’t have a load balancer.
- It happened spontaneously, without changing anything on the server
- Cookies are well within the limits of 32 Kb, no less than +/- 550 bytes.
And it happens randomly, and some times of the day it seems to happen more often than others.
I hope it can still be seriously looked into, because I really suspect there is a routing issue, just like last time. Maybe it’s one of many peering partner servers that has problems?
I hope there will be a solution soon. Otherwise I will be forced to end the collaboration with CF, and that would be a shame.
I don’t understand how a routing issue could be reproducible with a large number of page reloads and then resolved when the cookies are cleared.
Your issue is nothing like the one you are referring to, though.
There is nothing more the community can assist you with here, if you still believe it to be a Cloudflare issue please take it to your support ticket and explain the issue. Make sure you provide two HAR files - one with Cloudflare enabled on your website, and the other with Cloudflare temporarily disabled.
As a troubleshooting step, try disabling Always Online
which can be found under Caching/Configuration. Let us know if that makes a difference.
Cheers,
I don’t know if the cookies are the problem. But maybe there is a routing issue in one of the many server, what the route from my site to cloudflare goes through.
No idea. I made a guess.
Tomorrow i will checking this.
Now i go to sleep. it;s too late now
To be continued…
This Allways Online function is supposed to give a static cached page when the origin webserver is onreachable? In my case i don’t see a cached page with this mode on (it was on). How come?
Any idea?
When i turn on the Development-mode, i can see the static cached pages.
Now i’m going to turn off this Allway’s onlinefunction. Let’s hope this work!