Need to skip the cloudflare block in AI bot crawl setting

Website, Application, Performance
1

Hi Team,

I have planned to add the Tawk.to AI bot to our site. There is a setting to allow crawling and data extraction. However, we encountered a blocking issue with Cloudflare.

Is it possible to unblock the site firewall using the hostname?

I set the hostname to “tawk.to”, but it is not working.

How can we resolve this? Please help.

Thanks,
Arun

3

Hi there,

You can use Custom Rules to skip security features - Configure a custom rule with the Skip action · Cloudflare Web Application Firewall (WAF) docs

You would need to understand which security feature is triggering the captcha challenge - so the way to do that would be to look at Firewall Events - Security Events — Paid plans · Cloudflare Web Application Firewall (WAF) docs

You can add filters on the events to search for the rayID/sourceIP that you see on that captcha page and from there you can see the service that generated the challenge - and add it to your skip rule.

Hope this helps!

4

Hi Damian,

Thanks for the response…

Is that possible to skip the firewall rule by hostname?

if you say yes, kindly advise us to add hostname as “tawk.to ai assitant chatbot”,

Thanks,
Arun

5

If you can elaborate on skip by hostname and what exactly you are seeing and expecting to happen?

A hostname does not make a request to your website, a request comes from an IP address - there is no reverse DNS resolution to lookup a client IP’s reverse lookup hostname and then be able to skip based on a client hostname, if that is what being asked.

A better way to do this would be look for other signals that highlight this client - (Eg. does it use a specific user-agent, source IPs? ) - you could create a skip rule on these.

Are you the owner of tawk.to and looking to have folks not be blocked by while connecting to your hostname?

6

Hi Damian,

Thanks for the response…

I am not the owner tawk.to.

i am developer of one site. we planned to integraed the tawk.to in our site.

As per you suggestion we need to get souce ip and user agent of tawk.to? right?

kindly confirm…

Thanks,
Arun