Price Scraping

[help46]

What is the name of the domain?

euroimportpneumatici.com

What is the issue you’re encountering

Sitest, scraping prices

What steps have you taken to resolve the issue?

Activated Super Bot Fight mode.

Was the site working with SSL prior to adding it to Cloudflare?

Yes

What is the current SSL/TLS setting?

Full

[fritex]

@help46

Activated Super Bot Fight mode.

Make sure to enable Block AI Bots as well, if not already

Consider checking your Security → Events for past and further next days.
Determine if the IPs and ASNs are rotating or not, therefrom make sure to block some ASNs as well using IP Access Rules.

Create an IP Access rule · Cloudflare Web Application Firewall (WAF) docs

Nevertheless, consider blocking some of the known “bad user-agents”, “crawlers” or “bad ASNs” using below posts:

Help Needed Please To Exempt Google Bots From Firewall Rules

There’s a list of ASN belongs to hosting providers:

Top 50 user agents to block

Here’s a list from the perishablepress.com 7G .htaccess firewall: (360Spider|acapbot|acoonbot|ahrefs|alexibot|asterias|attackbot|backdorbot|becomebot|binlar|blackwidow|blekkobot|blexbot|blowfish|bullseye|bunnys|butterfly|careerbot|casper|checkpriv|cheesebot|cherrypick|chinaclaw|choppy|clshttp|cmsworld|copernic|copyrightcheck|cosmos|crescent|cy_cho|datacha|demon|diavol|discobot|dittospyder|dotbot|dotnetdotcom|dumbot|emailcollector|emailsiphon|emailwolf|exabot|extract|eyenetie|feedfinder|flaming|…

Top 50 user agents to block

Good list, thanks. I have deployed that but removed python and demon (those seem to block some RSS feedreaders, YMMV). What I also have in place is this: (http.user_agent contains “SemrushBot”) or (http.user_agent contains “AhrefsBot”) or (http.user_agent contains “DotBot”) or (http.user_agent contains “WhatCMS”) or (http.user_agent contains “Rogerbot”) or (http.user_agent contains “trendictionbot”) or (http.user_agent contains “BLEXBot”) or (http.user_agent contains “linkfluence”) or (http.us…