DNS rule for BOT

Feedback Feature Request Submitting & Feedback
1

Hello,
Bing, Google,etc… are performing too many requests on my site. Those requests increase drastically the servers load and affect the performance of the website for regular visitors.

So ideally I would like to be able to define a DNS rule that route all those requests to 1 dedicated IP (server).

So a rule that will look something like: if [request user-agent is BOT ] forward request to server: [someIP]

Thank you

2

This can’t really work at the DNS level because most commonly DNS servers do not receive requests from HTTP clients (except in the case of DoH, which is not used by all) so the DNS server will not know the “User-Agent” at all at the point of DNS resolution.

That said, assuming your DNS record is proxied through Cloudflare, you can use Origin Rules to conditionally direct traffic to a different place:

In that case, we will send traffic to a different origin chosen by you based on the User-Agent, or any other HTTP request characteristic you define in your rule.

1 Like
3

Thanks for your feedback.

I checked the origin rules. From what I can see it only allows to change the port number.

Ideally I would like to forward (proxy) the request to a different server.

Any idea ?

Thanks, Dan.

4

So the Origin Rules for overriding DNS are Enterprise features. You can use custom rules as part of a Load Balancer to do this instead:

In that case, you’d configure a Load Balancer for your hostname with at least 2 pools and then use a custom rule to steer traffic to the other pool based on the user-agent or bot fields.

1 Like
5

This feature requires a payment.
I have a PRO account. Is there a solution that comes without additional fees for PRO users?

I provide a free service and I can’t afford any additional cost.
Thanks, Dan.

6

Depending on your # of requests Workers could do this for free by setting the resolveOverride property:

That would require all of your traffic to your hostname to go via the worker and in Javascript you would write the logic to direct requests to a different origin via resolveOverride.