Anyone know how do i detect GTmetrix agent on workers?

comilla · January 11, 2021, 8:09am

I am trying to detect GTmetrix agent but cant get it working on workers. Here is the code how i am trying. Thanks!

const html = `

Hello World

This markup was generated by a Cloudflare Worker.

`;

async function handleRequest(request) {
let userAgent = request.headers.get(‘User-Agent’) || ‘’
if (userAgent.includes(“GTmetrix”)) {

return new Response(html, {
headers: {
“content-type”: “text/html;charset=UTF-8”
}
})
}

else {

return fetch(request)
}

}

addEventListener(“fetch”, event => {

return event.respondWith(handleRequest(event.request));
})

arunesh90 · January 11, 2021, 9:58am

From what I’m seeing it does not appear that GTmetrix always has GTmetrix in it’s user-agent.
Dumping the headers shows that the user-agent did not include GTmetrix when I had pointed it at my Worker.

As far I’m aware there’s no other way to detect whether GTmetrix is running on your site

comilla · January 11, 2021, 12:25pm

On firewall rules it works it shows the expression like this

(http.user_agent contains “GTmetrix”)

Not sure how to do implement on worker

M4rt1n · January 11, 2021, 12:25pm

Just tested against my own website and logged the requestheader.

Looks like this:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36 GTmetrix

So actually I can see the trailing GTmetrix and you could identify it by this string.

arunesh90 · January 11, 2021, 12:29pm

That’s odd, because I’m not getting that user-agent on my Worker and also not when using a RequestBin

comilla · January 11, 2021, 12:29pm

I am trying to detect by “GTmetrix” but it seems not working on worker, on firewall it works

(http.user_agent contains “GTmetrix”)

M4rt1n · January 11, 2021, 12:49pm

I have not tested it via workers apparently.

I have tested it against a normal URL on my Server.

Logged in at GTmetrix:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36 GTmetrix

Anonym GTmetrix:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36 GTmetrix

It definitely shows up with the “Gtmetrix” in my logs (serverside). But I actually dont know how to detect it via Workers, or why it does not show up there. Maybe it gets cut off there?

But you’re right. When I check what GTmetrix shows me as “request Header” it is not part of the user-agent:

On my server it shows up anyway as I catch the raw header and save it to a DB

As the syntax is:
User-Agent: <product> / <product-version> <comment>
(Source: https://www.geeksforgeeks.org/http-headers-user-agent/)

Maybe the comment is “GTmetrix” and is getting cut of by CloudFlare? I cant tell sorry. Maybe the OP should create a ticket to ask CloudFlares support if there is a way to detect/fetch the raw full user-agent?

But the approach of the OP seems to be right. As this example uses the same method:

  const userAgent = request.headers.get("User-Agent") || ""
  if (userAgent.includes("bot")) {
    return new Response("Block User Agent containing bot", { status: 403 })
  }

chandan.khatri · January 11, 2021, 2:53pm

Hi,

I had a use case to detect certain bots and serve response from cache instead of making a call to origin.

We wrote the following method to use a regex pattern to determine the agent. Better to make the regext pattern configurable than hardcode it in the code.

export async function checkIfBot(request) {

const patternConfig = await getCache(“BOT_PATTERN_REGEX”, BrowsePagesCommonKV)
if (patternConfig) {
botPattern = patternConfig
}

const re = new RegExp(botPattern, 'i')
const userAgent = request.headers.get('User-Agent')

let url = new URL(request.url)
console.log(url.href)
let isBotRequest = false
if (re.test(userAgent) || url.href.includes(“cache-test”)) {
console.log(‘the user agent is a crawler!’)
isBotRequest = true
}

return isBotRequest

}