Internal IP limiting broken and some pages not going through CF proxy

[alexr451]

Hello CF Community,

I’m new to Cloudflare, I signed up due to bots using up server resources and running up API costs, but the problems seem to be getting worse. I had internal rate limiting by IP address (5 calls/day) to use certain public APIs, but I’m realizing since all traffic now comes from CF’s IP, this creates 2 potential problems. Bots and people can easily make more than 5 calls due to the variety of IPs CF sends traffic from, and conversely some genuine people might get blocked if CF has already sent other traffic from the same IP more than 5 times that day.

I’m on the Pro plan which blocks “More advanced bots” and tested the bot blocking by trying to crawl my own site, it blocks direct CURL request but if I send requests through proxy crawling services, those seem to get through just fine. And I’m definitely still seeing bot traffic in the logs. So it hasn’t solved my advanced bot problem… but created other problems with my internal IP controls.

It seems I need to be on the Enterprise plan with “custom pricing” to enable the True-IP header. I’m a small business and can’t afford over $2,400 per year.

Also, I have a large public DB/directory, the logs from that are showing that direct traffic is still coming in from non-CF IPs, I’m very confused by this… it’s not a on a subdomain or anything either, just part of the regular website. Why is this traffic not being proxied like other parts of the site are?

I haven’t done anything in the rules section… I’m not very familiar with it yet. So it the whole site should be proxied at this point right?

Also, I’m not seeing the average load times improve.

I’m feeling a little demoralized at this point. I just paused CF. but maybe there are some solutions I’m not understanding, in particular with the IP control problem on APIs?

So it would be great if anyone had feedback or suggestions. Thanks!

[epic.network]

Welcome to the Cloudflare Community.

You don’t need an expensive plan to obtain your visitor’s actual IPs.

Restoring original visitor IPs · Cloudflare Support docs

[alexr451]

Great thank you for the advice! I wasn’t able to install the mods on my server but the $_SERVER['HTTP_CF_CONNECTING_IP'] seems to work great forwarding the IP correctly. I also realized that ‘$_SERVER[‘HTTP_X_FORWARDED_FOR’]’ would forward it was well. Which my directory pages where logging the IP from ‘$_SERVER[‘HTTP_X_FORWARDED_FOR’]’ as default when ‘$_SERVER[‘HTTP_CLIENT_IP’]’ didn’t come through. So I found out why it appeared they weren’t being proxied.

So this leaves multiple birds deceased with a singular stone.

But I am realizing that the Pro option still let’s through a fair amount of advanced bots. I’m seeing some well know malicious IPs making it through. So I’ll probably need to implement more server side measures, but that’s okay.

So thanks again.