Request for new field in Firewall Rules to filter out abusive users and increase AdSense revenues

I want to limit the most abusive users on my site. Therefore, I am asking for a new field in the firewall page rule field to monitor excessive usage (pageviews) by bots or human scrappers.

The average legitimate visitor on my site views 3 pages per session and visit 1 to 5 times a month. However, my 5 most abusive users (not search engines) visit >1,000 pageviews a month, and the top “user” has 14,000 pageviews/month (despite turning on super bot fighting mode).

Here’s an example of the new field I propose
Captcha Challenge users (who are not known bots) with over 100 pageviews a week.

Any of the following fields would address my needs. An option to use any of the 4 would be ideal:

  1. Pageviews/day
  2. Pageviews/Week
  3. Pageviews/Month
  4. Pageviews/session - This alone is not as useful because some bots create a new session for each request. Pageviews/session + Sessions/week or month would handle all of the use cases for #1, 2 & 3.

Why am I asking for this?
Google Adsense provides better ads (higher revenue ads) to websites that limit robot traffic and only show legitimate users. Adsense recently released a YouTube video confirming this. I am sure the percentage of legitimate users is also a determining factor in other search-related algorithms (organic search, Adwords, etc.).

Adsense RPM (revenue per 1,000 pageviews) increased significantly when I enabled super bot fighting mode. So if I can limit other illegitimate/abusive users who get through super bot fighting mode, I will generate more revenue and reduce the demands on my server.

I am sure this would be useful for many other Cloudflare users as well.

That’s Rate Limiting.

But what you’re asking is for Cloudflare to log every visit for up to a month, then for every single request, check that list to see if they’ve hit your rate limit, then challenge it. That’s computationally impossible while still maintaining any level of performance.

There’s a reason that Rate Limiting tops out at 1 minute on lower plans. Even Enterprise plans top out at one hour.

4 Likes

Rate limiting is currently capped at 10 minutes on my business plan. Most of these users/bots are smart enough to get around that limitation. With 1440 minutes in a day, they can still scrape ~1,000 pages per day if I set the rate limiting rule to 10 pageviews in 10 minutes.

I have tested rate limiting out with the 10 page views in 10 minutes and had some issues:

  1. I catch non united states traffic currently filtered out in my Firewall Rules.
  2. I caught mostly normal users. Even though the average visitor views 3 pages per visit, with thousands of visitors a day, many users view over 10 in 10 minutes.

I would be content with 24 hours (#1 in my initial request). But, 72 hours would be better. Based on the Firewall overview report, it appears that you already track users for 72 hours.

The added benefit of adding pageviews to firewall rules, is that I could add multiple criteria (known bots, user agent, path, country, etc). Then, I could do more advanced searches.

For example, some known bots (like ahrefs or grapeshot) are not bots I want on my site. I could run a rule that says, JS challenge all known bots who are not Google, Bing, Yahoo and have over 100 pageviews in 72 hours. This would make it easy for me (or other users) to challenge and possibly filter out unwanted known bots.

I could also block users (than are not known bots) with over 100 pageviews in 72 hours a then show a CF custom page blocking message that enabled legitimate user to request access if they were blocked.

Initially, I didn’t think about the storage and computational demands a week or month tracking data would place on Cloudflare. But, if you already track visitors for 72 hours, that would still be much more helpful.

Here’s a video with a more methodical approach. If you don’t mind some curse words, the strategy is solid. It’s a good way to block out unwanted traffic.

I agree that this is a good methodology, I’ve had some success after playing with it for an hour and will continue to use it. But, I don’t have the standard DDOS attacks. I have users who scrape my data (slowly if necessary) and they make frequent changes. 10 minutes in the rate-limiting is not enough to stop them or flag them… As I pointed out in the 10 pageviews for 10 minutes example above.

That guy in the video you shared spends every day looking at that log and tries to piss off the attackers so they make themselves known. I don’t have that kind of time to commit to checking every bad actor and then add their IP or asn to a list of rules. He also claims 5 levels of protection because on small attacks, “Cloudflare DDoS protection does not kick in”. The firewall rule field (filtering pageviews per 24 or 72 hours) I suggested would enable me to challenge users based on usage (over usage) and ignore good bots. Then, I can Challenge these users without the time commitment. The captcha alone will discourage 99% of these users. And, that’s the beauty of Cloudflare. Unlike CloudFront, Cloudflare saves a great deal of time and is usable by low-level/occasional programmers (like myself) so we can handle basic tasks on our own. I think this would be a big-time saver and you already collect the data.

It is much easier to challenge a user viewing far more than a normal number of pageviews than to expect Cloudflare users to figure out all these intricate details. That time savings seems like a good justification for the change.