C# HttpClient blocked by CloudFlare when fetching a certain image

Greetings,
I’ve got some C# code that uses an HttpClient to scrape certain information off web pages. (Don’t worry; it’s all totally legit.)
Now, there’s one particular site that appears to be protected by Cloudflare, and while I can happily download the image using Postman, when I use an HttpClient in my code to download the exact same image, I get the following response:

	<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>


<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script><!--<![endif]-->



</head>
<body>
  <div id="cf-wrapper">
	<div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
	<div id="cf-error-details" class="cf-error-details-wrapper">
	  <div class="cf-wrapper cf-header cf-error-overview">
		<h1 data-translate="block_headline">Sorry, you have been blocked</h1>
		<h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> edweek.org</h2>
	  </div><!-- /.header -->

	  <div class="cf-section cf-highlight">
		<div class="cf-wrapper">
		  <div class="cf-screenshot-container cf-screenshot-full">
			
			  <span class="cf-no-screenshot error"></span>
			
		  </div>
		</div>
	  </div><!-- /.captcha-container -->

	  <div class="cf-section cf-wrapper">
		<div class="cf-columns two">
		  <div class="cf-column">
			<h2 data-translate="blocked_why_headline">Why have I been blocked?</h2>

			<p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>
		  </div>

		  <div class="cf-column">
			<h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2>

			<p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>
		  </div>
		</div>
	  </div><!-- /.section -->

	  <div class="cf-error-footer cf-wrapper">
  <p>
	<span class="cf-footer-item">Cloudflare Ray ID: <strong>46ece24d180e220a</strong></span>
	<span class="cf-footer-separator">&bull;</span>
	<span class="cf-footer-item"><span>Your IP</span>: 192.152.22.2</span>
	<span class="cf-footer-separator">&bull;</span>
	<span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
	
	<span class="cf-footer-separator">&bull;</span>
	<span class="cf-footer-item">
	  <select id="lang-selector">
		<option value="">Select a Language</option>
		<option value="en">English</option>
		<option value="es">Español</option>
	  </select>
	</span>
	
  </p>
</div><!-- /.error-footer -->


	</div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="text/javascript">
  window._cf_translation = {};
  window._cf_translation.locale = 'en';
  window._cf_translation.blobs = {};
</script>

</body>
</html>

How can I work around this?

Probably a challenge, firewall rule or other fancy stuff.

What’s legit on scraping foreign sites? If they want you to use their stuff on your own pages they’d provide embed code or an API.

You should contact the site owner to get around this kind of error.

1 Like

From the error I’d contact the website owner. They can help you determine what rule they might have in place which is blocking your tool.

-OG

This topic was automatically closed after 31 days. New replies are no longer allowed.