Amazon Price scraping - No Like

[MagnusN]

Hi

Amazon is price scraping our site and we like to block them. We run PRO level and the setting Super Bot Fight Mode, but this does not help.

Any tips how to solve it?

Best Magnus

[fritex]

Greetings,

I am sorry to hear you have an bad experience with Amazon scraping

Nevertheless, thank you for asking. I believe if you are using Cloudflare for your domain, there are multiple ways how could we detect and block “bad guys”

@MagnusN

PRO

@MagnusN

tips

Great, awesome! Pro plan offers “Web Application Firewall” (Managed WAF Rules), “Cloudflare Managed Ruleset” and “Package: OWASP ModSecurity Core Rule Set” which we can enable with a single click and configure per demand.

Despite of this, Pro plan has got an option to “Configure Super Bot Fight Mode” to challenge “Definitely automated” bots.

Furthermore, you can use Firewall Rules to block Amazon AS numbers - there are multiple as far as I remember - and even more like old HTTP/1.0 requests, or block some known “bad” user-agents, crawlers, etc.

Under the Firewall Setings, you can set the Security Level, Challenge Passage, enable Browser Integrity Check and few others.

You could also track your Firewall Events and check for any suspicious or being challenged or blocked, that way you can tune-up and adapt your security settings as you need.

Using Cloudflare Scrape Shield, you can protect written e-mail addresses in your HTML content of a webpage by enabling Email Address Obfuscation for example.

I am sharing my settings at the below screenshots from which you can compare with yours and analyze :

In terms of Amazon, in my case and my example I am blocking the three “Amazon” ASN numbers with Firewall → IP Access Rules as follows:

[MagnusN]

Wow, Thanks a lot. We try it. Cheers Magnus

[MagnusN]

Hi
And thanks again. But sorry to say the settings did not work. Amazon is still scraping the prices.
Do you know any other ways to stop Amazon price scraping?
Best Magnus

[michael]

@MagnusN

Amazon is still scraping the prices.

Can you see the requests in your web server logs? If so, can you share one of the log entries?

[MagnusN]

Hi and thanks

I will comeback with it.

Best Magnus

[MagnusN]

Hi

We having a lot of promblem with Amazon Price scraping and have done changes and blocked IP.

Is there any one that know how to solve it.

Best Magnus

[erictung]

There are some discussions regarding on how to deal with bots:

Bot traffic has not stopped even after subscribed to Pro version

Hello My site is getting bots traffic (mostly page views, it is 5 6 times the total users). It is coming from 1 of December and it is badly affecting my site ranking since the bot traffic started coming to my site. Maybe some of my competitors doing this. First, there was a total of 2.04k page views and 1.44k users per day because at that time when there was no bots traffic. You can check the current page views on the screenshot I attached. It has a total of 6.02k page views and 1.5k users. You…

Finding the right Bot Management Approach

We have an ongoing issue with bots that are not attacking, but interacting with our website. It’s not DDOS nor are they trying injections or anything else. Essentially an IP hits the site, loads a page, and then clicks on ads, for a couple of dozen pages on one day, then a hundred the next (sometimes a third day, but no more), and then we never see that IP address again. This will happen from 10 or 15 IPs on a daily basis. The only issue is that it messes up our ad counts. Never really a perform…

CloudFlare Still Lets in A HUGE Amount of Bot Traffic

Other than putting my wordpress site into “under attack mode”, I’ve pretty much enabled all of the security features possible within Cloudflare. I’ve added a firewall against XMLRPC attacks with Cloudflare (good), and the Hotlinking blocker. But otherwise, I had to installed a Proxy/VPN blocker on my wordpress today, and holy sh*t… so much bot traffic still getting through Cloudflare (just in the last 3 hours): [image] And no way am I doing custom firewalls for every single IPv4 on cloudfla…

[jnperamo]

Yeah, given how complex nowadays bots are, it’s doubtful that any non-enterprise solution can keep up with detections right off the bat.
Amazon is an excellent example of an arms races fight. Amazon dedicates millions to keeping their scrapers up to date to serve the lowest prices across the web; it’s a bit unfair to expect any $20-200 solution to be bulletproof against that.

My best bet to counter Amazon (without prior experience in doing so) would be to block their ASN completely; however, it wouldn’t surprise me if they used more than one ASN to scrape prices.