Hi,
I’m trying to analyze data in the Firewall Events, but I cannot find the Referer anywhere. As seen in the screenshot below, it seems the Referer is NOT shown anywhere in Firewall Events.
I believe Referer should definitely be included in Firewall Events, especially since you can add Rules based on Referer.
Thank you.
What subscription level are you using? I know in the enterprise level you can get the referrer and more with the logs sent out to your preferred bucket. We do this at my employer and then have Splunk grab them from the bucket. Lots more data in those logs than the limited views you will see in the Firewall Events.
Prior to this, I used to query the API with the RayID to return the full log data. See Cloudflare API v4 Documentation.
I have a Pro plan only.
I think if you can create rules using the Referer, you should be able to see it in the Firewall Events. Because otherwise, it makes it impossible to analyze the traffic by Referer.
Agreed. Few months I also raised this to Cloudflare Support as a feature request, but sadly the Referer field is still not available yet.
I just opened a support ticket as well. Maybe if enough people request it, they will decide to add it.
I seriously think it should be there by default, and NOT be an optional feature you can get by buying a bigger plan. You simply cannot provide half of an important feature like this. If you enable adding rules based on Referer, you should also add the Referer field to the Firewall Events for all plans that have access to it.
Agreed. With the “Hotlink Protection” feature under “Scrape Shield”, I think most people would like to know why the requests were blocked by Hotlink Protection, i.e. who stole their image links.
Exactly.
I also use the Referer when analyzing traffic to block bad bots. Most of the bad bots come with a blank Referer, and this is easy to detect by using a Referer rule. But some of them seem to spoof the Referer, and those are practically impossible to detect / block without knowing what Referer they use. I can check Cloudflare Analytics and figure out what the Top Referers are and see if I can make a correlation between Analytics Requests and Firewall Events, but it pretty much stops there.
I really want to use Super Bot Fight Mode (the more I write that name, the more I think they should change it), but right now, it’s blocking too many good bots, like Facebook crawlers for instance. I also realized it blocks curl requests from my own web server. So until those things are fixed, I need to continue using Firewall Rules to block bad bots. But without having access to the Referer, it makes the task much harder.
You can query this information directly from GraphQL:
query
{
viewer
{
zones(filter: { zoneTag: “$zone_id”})
{
firewallEventsAdaptive(filter: {datetime_gt: “2021-03-28T10:15:05Z”,
datetime_lt: “2021-03-28T11:23:05Z” },
limit: 2,
orderBy: [datetime_DESC, rayName_DESC])
{
action
datetime
rayName
clientRequestHTTPHost
userAgent
ruleId
clientRefererHost
clientRefererPath
clientRefererQuery
}
}
}
}
+1 would love to see referrer info in firewall events GUI dashboard. Though @ncano is correct it’s available in Firewall GraphQL API too
example me filtering Firewall GraphQL for past 24hrs for a specific IP and filter only for block requests and limit output to 2 entries for specific hostname = domain.com
as you can see referrer info is there too
./cf-analytics-graphql.sh ip-hrs 24 198.144.149.253 block 2 domain.com
{ "query":
"query {
viewer {
zones(filter: {zoneTag: $zoneTag}) {
firewallEventsAdaptiveGroups(
limit: $limit,
filter: $filter,
orderBy: [datetime_ASC]
) {
dimensions {
action
botScore
botScoreSrcName
source
datetime
clientIP
clientAsn
clientCountryName
edgeColoName
clientRequestHTTPProtocol
clientRequestHTTPHost
clientRequestPath
clientRequestQuery
clientRequestScheme
clientRequestHTTPMethodName
clientRefererHost
clientRefererPath
clientRefererQuery
clientRefererScheme
edgeResponseStatus
clientASNDescription
userAgent
kind
originResponseStatus
ruleId
rayName
}
}
}
}
}",
"variables": {
"zoneTag": "zoneid",
"limit": 2,
"filter": {
"clientIP": "198.144.149.253",
"action": "block",
"clientRequestHTTPHost": "domain.com",
"datetime_geq": "2021-03-27T11:36:39Z",
"datetime_leq": "2021-03-28T11:36:39Z"
}
}
}
------------------------------------------------------------------
Cloudflare Firewall
------------------------------------------------------------------
since: 2021-03-27T11:36:39Z
until: 2021-03-28T11:36:39Z
------------------------------------------------------------------
2 Firewall Events for Request IP: 198.144.149.253
------------------------------------------------------------------
198.144.149.253 636857d7bdd14004 403 1xHeuristics block 7040 NETMINDERS CA YYZ 2021-03-27T11:40:18Z domain.com GET HTTP/1.0 /login/
198.144.149.253 63692692ad403ff1 403 1xHeuristics block 7040 NETMINDERS CA YYZ 2021-03-27T14:01:26Z domain.com GET HTTP/1.0 /login/
------------------------------------------------------------------
{
"results": [
{
"action": "block",
"botScore": 1,
"botScoreSrcName": "Heuristics",
"clientASNDescription": "NETMINDERS",
"clientAsn": "7040",
"clientCountryName": "CA",
"clientIP": "198.144.149.253",
"clientRefererHost": "domain.com",
"clientRefererPath": "/login/",
"clientRefererQuery": "",
"clientRefererScheme": "https",
"clientRequestHTTPHost": "domain.com",
"clientRequestHTTPMethodName": "GET",
"clientRequestHTTPProtocol": "HTTP/1.0",
"clientRequestPath": "/login/",
"clientRequestQuery": "",
"clientRequestScheme": "https",
"datetime": "2021-03-27T11:40:18Z",
"edgeColoName": "YYZ",
"edgeResponseStatus": 403,
"kind": "firewall",
"originResponseStatus": 0,
"rayName": "636857d7bdd14004",
"ruleId": "bd706145258349c686ddb32b94dxxxxx",
"source": "firewallrules",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 Edg/83.0.478.37"
},
{
"action": "block",
"botScore": 1,
"botScoreSrcName": "Heuristics",
"clientASNDescription": "NETMINDERS",
"clientAsn": "7040",
"clientCountryName": "CA",
"clientIP": "198.144.149.253",
"clientRefererHost": "domain.com",
"clientRefererPath": "/login/",
"clientRefererQuery": "",
"clientRefererScheme": "https",
"clientRequestHTTPHost": "domain.com",
"clientRequestHTTPMethodName": "GET",
"clientRequestHTTPProtocol": "HTTP/1.0",
"clientRequestPath": "/login/",
"clientRequestQuery": "",
"clientRequestScheme": "https",
"datetime": "2021-03-27T14:01:26Z",
"edgeColoName": "YYZ",
"edgeResponseStatus": 403,
"kind": "firewall",
"originResponseStatus": 0,
"rayName": "63692692ad403ff1",
"ruleId": "bd706145258349c686ddb32b94dxxxxx",
"source": "firewallrules",
"userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
}
]
}
I agree with this. Even though it’s not as common nowadays, it was a common technique for some attackers to inject iframes that launched DDoS attacks, you could block these more easily with clear analytics.
Totally agree with you on that.
We need to have access to the Referer in the Firewall Events.