I am able to access https://unjobs.org/organizations/ctbto using any browser, no problem.
But when I try to WebClient.DownloadData() within my web application, I get 403. I think it is not an origin fault because I see this in the HttpWebResponse:
Server => "cloudflare"
I don’t expect you to solve my problem; however, I would like to know what I should ask the site owner? I’ve read about “WAF rules”, is that something he controls?
I regret that I do not know what any of those terms mean, nor do I have any idea how their Cloudflare settings are configured. The site belongs to the customer of the web application I maintain, and I need to access a page of that site with my C# code. All I get is 403 and I am hoping to know the best way to ask the client’s site admin when I finally contact him/her. What’s killing me is that requests from any browser on any client machine get a valid response; I’ve traced the headers and they are only sending four headers:
GET /offices/ctbto_vie HTTP/1.0
Host: unjobs.org
Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01
Accept-Language: en
User-Agent: Lynx/2.8.8dev.15 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/1.0.1e-fips
which I’ve emulated and it still fails, so I’m at a loss.
. . . but those are the headers which work in a browser. My code has tried both with and without those headers (and others, especially User-Agent) and always fails.