Secret links being picked up by bots

Hi there,

Been using Cloudflare for a number of months now and really happy with how it all works.

I’ve been keeping an eye on the firewall section, and noticed some strange behaviour. Nothing wrong with Cloudflare, but just something I’m curious about.

We have a system which generates a unique page link, which we email to a client to view. The link itself would not be possible to guess, and links are not shared between clients. Since we’re based in the UK, we have set “JS Challenge” to all countries outside the UK.

We see our UK users clicking on these links, and they’re not challenged or impacted in any way.

However, in a small number of cases, around a similar time that a UK user views one of the links, there will also be a JS Challenge triggered in the firewall for the same link, outside the UK (often somewhere like France, India or the USA). These challenges don’t make it to our origin, which makes me think they’re definitely a bot. In some cases the IP addresses look clean, in other cases they have a small number of complaints listed in abuseipdb.com

So I guess my question is, how are bots getting these links, given that they are very much random and difficult to guess, and the foreign visits kind of coincide with when a UK visitor goes to the link. Could it be some kind of email scanning service that the client employs? Or perhaps their computer has malware, which makes a note of any links they click on and sends it back to a bot to investigate?

I’m not looking for any difinitve answers, but just some clues as to whether there could be legitimate reasons for this happening.

Thanks

Could be the same as if you send a link via Instagram or Facebook. Their crawlers will crawl the link to display the content of the link as preview. Just like here in the forum. Discours will also call the link and crawl the content of it to display it here.

Maybe their Email Systems or company settings are doing so aswell to prevent dangerous links? If you know the IPs which the requests came from you can simply check who these IPs belong to.

Hi M4rt1n

I’m inclined to think it is some kind of email scanning service. It doesn’t happen often and certainly not for every link visited. The IP’s in question usually belong to some kind of hosting company or datacentre, for example OVH, Leaseweb, Clouvider, etc.

I know from IONOS they have Email-Antivirus services which are scanning links too. So maybe other services does have this too.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.