Random TCP handshake problems / HTTP 522

Hi,
Since 24th Nov we’re experiencing random 522 at customers’ pages. I’ve found that some CF servers doesn’t want to talk with us when they get SYN-ACK send from synproxy, it seems like the SYN-ACK is rejected somewhere. I see that in DCs located in Warsaw and Frankfurt. When I pass traffic form CF beyond the synproxy everything is ok. It’s not a subject of specific prefix or address. One address generally works, but once upon a time it doesn’t establish TCP session for a few minutes. MTRs doesn’t show any packet loss. tcpdump and traceroute outputs below.

Warsaw:
13:21:06.538049 IP 141.101.96.37.29036 > 77.79.221.187.80: Flags [S], seq 2677923046, win 64240, options [mss 1460,sackOK,TS val 2161900510 ecr 0,nop,wscale 13], length 0
13:21:06.538120 IP 77.79.221.187.80 > 141.101.96.37.29036: Flags [S.], seq 654843810, ack 2677923047, win 0, options [mss 1460,sackOK,TS val 2026397533 ecr 2161900510,nop,wscale 11], length 0
13:21:08.586068 IP 141.101.96.37.29036 > 77.79.221.187.80: Flags [S], seq 2677923046, win 64240, options [mss 1460,sackOK,TS val 2161902558 ecr 0,nop,wscale 13], length 0
13:21:08.586118 IP 77.79.221.187.80 > 141.101.96.37.29036: Flags [S.], seq 654843810, ack 2677923047, win 0, options [mss 1460,sackOK,TS val 2026399581 ecr 2161902558,nop,wscale 11], length 0
13:21:14.740469 IP 173.245.61.140.63754 > 77.79.221.187.80: Flags [S], seq 1470154039, win 64240, options [mss 1460,sackOK,TS val 836285320 ecr 0,nop,wscale 13], length 0
13:21:14.740525 IP 77.79.221.187.80 > 173.245.61.140.63754: Flags [S.], seq 1425432, ack 1470154040, win 0, options [mss 1460,sackOK,TS val 2026405725 ecr 836285320,nop,wscale 11], length 0
13:21:14.913054 IP 173.245.61.140.63754 > 77.79.221.187.80: Flags [.], ack 1, win 8, options [nop,nop,TS val 836285493 ecr 2026405725], length 0
13:21:14.913148 IP 77.79.221.187.80 > 173.245.61.140.63754: Flags [.], ack 1, win 21, options [nop,nop,TS val 2026405725 ecr 836285493], length 0
13:21:15.085708 IP 173.245.61.140.63754 > 77.79.221.187.80: Flags [P.], seq 1:210, ack 1, win 8, options [nop,nop,TS val 836285665 ecr 2026405725], length 209: HTTP: GET / HTTP/1.1

traceroute to 141.101.96.37 (141.101.96.37), 30 hops max, 60 byte packets
 1  ip-254.net-77-79-221-0.eco.atman.pl (77.79.221.254)  0.483 ms  0.487 ms  0.480 ms
 2  rev-85.232.252.89.atman.pl (85.232.252.89)  0.678 ms  0.732 ms  0.661 ms
 3  cloudflare.thinx.pl (212.91.0.28)  1.784 ms * *
 4  198.41.160.185 (198.41.160.185)  24.813 ms * *
 5  198.41.160.236 (198.41.160.236)  37.877 ms  37.862 ms  37.879 ms
 6  141.101.96.37 (141.101.96.37)  32.315 ms  32.285 ms  32.343 ms

traceroute to 173.245.61.140 (173.245.61.140), 30 hops max, 60 byte packets
 1  ip-254.net-77-79-221-0.eco.atman.pl (77.79.221.254)  0.589 ms  0.569 ms  0.567 ms
 2  rev-85.232.252.89.atman.pl (85.232.252.89)  0.708 ms  0.733 ms  0.681 ms
 3  ae5.4031.r1.isp-r3.isp.atman.pl (217.17.33.54)  0.615 ms  1.447 ms  1.435 ms
 4  war-b4-link.ip.twelve99.net (62.115.37.54)  0.903 ms  0.951 ms  0.909 ms
 5  hbg-bb3-link.ip.twelve99.net (62.115.112.58)  14.889 ms  14.832 ms *
 6  ldn-bb2-link.ip.twelve99.net (62.115.122.161)  27.377 ms  27.214 ms *
 7  nyk-bb1-link.ip.twelve99.net (62.115.112.244)  99.247 ms  95.671 ms  99.234 ms
 8  palo-b24-link.ip.twelve99.net (62.115.122.36)  167.824 ms  167.544 ms palo-b24-link.ip.twelve99.net (62.115.118.121)  163.662 ms
 9  cloudflare-ic-322348.ip.twelve99-cust.net (62.115.50.51)  164.570 ms  164.558 ms  169.006 ms
10  198.41.132.89 (198.41.132.89)  167.881 ms  168.218 ms  168.180 ms
11  198.41.132.77 (198.41.132.77)  168.046 ms  167.725 ms  172.970 ms
12  173.245.61.140 (173.245.61.140)  167.540 ms *  167.488 ms
Frankfurt:
16:45:06.527659 IP 162.158.102.141.59676 > 23.88.122.133.443: Flags [S], seq 2717989287, win 64240, options [mss 1460,sackOK,TS val 1110808232 ecr 0,nop,wscale 13], length 0
16:45:06.527698 IP 23.88.122.133.443 > 162.158.102.141.59676: Flags [S.], seq 2141045124, ack 2717989288, win 0, options [mss 1460,sackOK,TS val 4082193501 ecr 1110808232,nop,wscale 7], length 0
16:45:07.587294 IP 162.158.102.141.59676 > 23.88.122.133.443: Flags [S], seq 2717989287, win 64240, options [mss 1460,sackOK,TS val 1110809292 ecr 0,nop,wscale 13], length 0
16:45:07.587334 IP 23.88.122.133.443 > 162.158.102.141.59676: Flags [S.], seq 2141045124, ack 2717989288, win 0, options [mss 1460,sackOK,TS val 4082194589 ecr 1110809292,nop,wscale 7], length 0
16:45:09.636298 IP 162.158.102.141.59676 > 23.88.122.133.443: Flags [S], seq 2717989287, win 64240, options [mss 1460,sackOK,TS val 1110811341 ecr 0,nop,wscale 13], length 0
16:45:09.636339 IP 23.88.122.133.443 > 162.158.102.141.59676: Flags [S.], seq 2141045124, ack 2717989288, win 0, options [mss 1460,sackOK,TS val 4082196637 ecr 1110811341,nop,wscale 7], length 0
16:45:13.667341 IP 162.158.102.141.59676 > 23.88.122.133.443: Flags [S], seq 2717989287, win 64240, options [mss 1460,sackOK,TS val 1110815372 ecr 0,nop,wscale 13], length 0
16:45:13.667391 IP 23.88.122.133.443 > 162.158.102.141.59676: Flags [S.], seq 2141045124, ack 2717989288, win 0, options [mss 1460,sackOK,TS val 4082200669 ecr 1110815372,nop,wscale 7], length 0

traceroute to 162.158.102.141 (162.158.102.141), 30 hops max, 60 byte packets
 1  172.31.1.1 (172.31.1.1)  2.679 ms  2.675 ms  2.680 ms
 2  22787.your-cloud.host (162.55.117.113)  0.284 ms  0.277 ms  0.271 ms
 3  * * *
 4  213.239.235.73 (213.239.235.73)  1.160 ms 213.239.235.77 (213.239.235.77)  1.128 ms 213.239.235.73 (213.239.235.73)  1.566 ms
 5  * * *
 6  core22.fsn1.hetzner.com (213.239.227.214)  1.080 ms core21.fsn1.hetzner.com (213.239.227.194)  0.747 ms  0.720 ms
 7  core5.fra.hetzner.com (213.239.224.94)  5.445 ms 213-239-224-114.clients.your-server.de (213.239.224.114)  4.883 ms hos-tr4.ex3k1.dc4.fsn1.hetzner.com (213.239.224.98)  4.818 ms
 8  core8.fra.hetzner.com (213.239.245.126)  5.247 ms  5.313 ms core8.fra.hetzner.com (213.239.224.217)  5.194 ms
 9  cloudflare.1-ix.net (185.1.254.19)  28.836 ms * *
10  162.158.102.141 (162.158.102.141)  28.436 ms  29.240 ms  28.471 ms

We are experiencing the same problems that occurred right around that time too.
I live chatted a support engineer, claimed there was an issue with my network/server. I spent countless hours realizing that it is not an issue on my part… This happened suddenly and when I take off Cloudflare proxy everything works okay.

I have a support ticket that has been opened for 10 days now with no response breaching SLA time. I am very frustrated in this matter… here is a screenshot of my analytics logs - maybe you could find some correlation between yours too?

@bcrivelli, do you have a synproxy enabled on your network / servers? Since I disabled it for CF networks the problems disappeared. However I have still synproxy enabled on my test environment and last errors was on 8th Dec.

@dominik.dusik I still have it enabled for production, however on a test app I disabled Cloudflare proxy completely - no issues since I did that for that specific app. Also based on those analytics logs seems like my last error was on Dec. 8/9th too. There must be something Cloudflare is not telling us that is happening on their backend. (Also my logs show there were no types of errors prior to this event for past 30 days and beyond.)

@bcrivelli, by synproxy I mean a TCP feature on your backend servers, or on network devices in front of them, not any setting on CF.

@dominik.dusik Whoops sorry for confusion. Your point about the SYN proxy is a game-changer. I’ve just had a look at our firewall setup and, guess what, our firewall rules are set to ‘synproxy’ state. I hadn’t considered this before.

So, I’m going to switch our rules back to the default ‘keep’ state and see how things pan out. If this does the trick like it did for you, it’ll be a huge relief. Also, our similar timelines for errors are intriguing. Makes me wonder if something changed on Cloudflare’s side recently. I’ll keep an eye on things and update you on how it goes. Thanks for pointing me in the right direction!

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.