Hello Cloudflare Community,
I would like to share the following situation I started to see after making some changes to my web application environment. But before, let me describe a bit the environment itself:
- I am using Cloudflare as the DNS provider to get the Public IP of my web application and also to proxy the incoming traffic.
- This public IP belongs to a HAProxy instance which is being used for SSL termination (it manages the SSL certificates issued by
certbot) and for load balancing the incoming requests. - HAProxy redirects the incoming request to the final web server, where the web application runs, through HTTP.
- In the corresponding domain created in Cloudflare, the SSL/TLS encryption mode is Full (strict)
The above scenario has been working without problems over the last few years. Now I have to set some additional HTTP headers for security reasons, and I set them up at HAProxy level (X-Frame-Options and X-Content-Type-Options):
frontend https-http-in
bind 0.0.0.0:80
bind 0.0.0.0:443 ssl crt $PATH_TO_CERT_FILE alpn h2,http/1.1
...
http-response set-header X-Frame-Options: DENY
http-response set-header X-Content-Type-Options: nosniff
...
After applying the above two additions, from time to time I see that an HTTPS request (specifically, an API call sent from a client on my side, through Python-urllib) is being received twice in the web server. In my specific case, it causes an issue because the first request is replied to by HTTP 200 (expected result), but the second one, is received after 400ms approx. is replied by an expected HTTP error 407, because this API call triggers a transaction (write operation) into a database, and the second HTTP request (from now on, duplicated HTTP request) tries to write the same object to the database, which is not allowed and thus replying with HTTP error 407. In the end, the client receives the HTTP error 407.
One interesting thing is that the above behavior is not seen consistently. For example, if I try to run the same API call from the same client, and without modifying the above setup, maybe it happens after 100-200 attempts. If I remove the above additional HTTP headers, I don’t see this issue happening (tested hundreds of times).
Doing a further analysis by capturing packet traces on the client side (encrypted traffic), on the HAProxy side, and web server side (unencrypted traffic), I see that when the issue happens, the duplicate HTTP request comes from two different IPs from Cloudflare (based on the value from X-Forwarded-For HTTP header in both the original and duplicated request and the information I get from whois). The packet captures show me that, on the client side, there’s always the number of TLS sessions when the issue happens and when not, meaning that the HTTPS request is not duplicated on the client side.
Another interesting thing I tried is to not proxying the traffic through the Cloudflare network, and use it only as DNS. After doing it, and preserving the additional HTTP headers I set in HAProxy, I have not seen the issue again (I did more than 200 attempts without going through the Cloudflare network).
I tried to search for similar issues on the web and the Cloudflare community, but I was not able to find anything relevant. The only similar topic I found is this one, but it did not help me so much to understand what’s happening.
Has anyone ever faced the above situation? Does anyone have some insights to understand the root cause of this behavior and, if possible, share possible solutions or things to check?
Thank you so much in advance! Any helpful comment is very welcome.