CloudFlare LogPush Log Format

Getting Started
#1

Hi community,

I’ve rather new to CloudFlare. I’m working as a SIEM Engineer has recently came across the opportunity to work with CloudFlare LogPush.

My client had recently pushed the CloudFlare logs to S3 buckets, and from the S3 buckets, the SIEM will read and normalize the logs.

Client had 4 domains, and all 4 domains has been configured to LogPush to this S3 buckets. Out of 4 domains, one of the domains logs format is different from other 3. Due to this difference, the SIEM customer parser unable to recognize this logs format and failed to normalize the domain’s log.

The differences are as the screenshot shown:

CF_Log_Differences
CF_Log_Differences1713×915 168 KB

As above shown, the main differences is the log field is not sorted in alphabet order.
I’ve check with the client, they mentioned that when enabling the LogPush, there’s no option to ensure sorting is inplace.

So the ask: Is there any option in LogPush that ensure all the field is sorted in alphabet order?

#2

What’s the SIEM tool you are using?

#3

ArcSight. As it doesn’t support out of the box, we’ve written custom parser that can only recognize log format in alphabet order sorting.

#4

Perhaps you need a way to sort the log fields after the logs have been pushed to S3 buckets.
You may use Lambda function to run the sorting code once there’s a new object uploaded to S3 bucket (S3 triggers).

Else, contact Cloudflare Support and see whether they can do something from their side.

#5

By the way, is your SIEM directly connected to the S3 buckets and listen for new objects?
If not, how’s your SIEM picks up new log files?

#6

not really direct connect. A combination of SQS listening on new object available in S3 bucket, and python script to copy those object to another file storage for processing.

#7

What’s weird is, there are 4 domains in the CloudFlare gateway: abc.com, xyz.com, jkl.com and stu.com
Only abc.com is having this random field sort issue. The rest of the domains LogPush files is having perfectly normal sorting of alphabet order (Descending).

#8

Can you modify the existing Python script to sort the JSON object first, then copy them to another file storage for processing?

#9

Anyway, since your client is Cloudflare Enterprise customer (only Enterprise customer has Logpush function), they can open a support ticket by writing an email to [email protected] regarding the issue.

PS: I’m not a Cloudflare employee. I’m just looking around and helping people in this community.

#10

This topic was automatically closed after 31 days. New replies are no longer allowed.