Hello CF Community,
I’m new to Cloudflare, I signed up due to bots using up server resources and running up API costs, but the problems seem to be getting worse. I had internal rate limiting by IP address (5 calls/day) to use certain public APIs, but I’m realizing since all traffic now comes from CF’s IP, this creates 2 potential problems. Bots and people can easily make more than 5 calls due to the variety of IPs CF sends traffic from, and conversely some genuine people might get blocked if CF has already sent other traffic from the same IP more than 5 times that day.
I’m on the Pro plan which blocks “More advanced bots” and tested the bot blocking by trying to crawl my own site, it blocks direct CURL request but if I send requests through proxy crawling services, those seem to get through just fine. And I’m definitely still seeing bot traffic in the logs. So it hasn’t solved my advanced bot problem… but created other problems with my internal IP controls.
It seems I need to be on the Enterprise plan with “custom pricing” to enable the True-IP header. I’m a small business and can’t afford over $2,400 per year.
Also, I have a large public DB/directory, the logs from that are showing that direct traffic is still coming in from non-CF IPs, I’m very confused by this… it’s not a on a subdomain or anything either, just part of the regular website. Why is this traffic not being proxied like other parts of the site are?
I haven’t done anything in the rules section… I’m not very familiar with it yet. So it the whole site should be proxied at this point right?
Also, I’m not seeing the average load times improve.
I’m feeling a little demoralized at this point. I just paused CF. but maybe there are some solutions I’m not understanding, in particular with the IP control problem on APIs?
So it would be great if anyone had feedback or suggestions. Thanks!