Any service recommendation who can save log data cheaply?


#1

I want to create worker that will send log data, so in time of ddos attack I will have better understanding of the attack and how to block it using cf Firewall.

the problem is in ddos attack I can get a lot of rps so I need 3th solution for that I found https://logz.io/ is anyone know any other cheap solution? I saw about aws kinesis but for it to be economical good it needs to get data at 25kb batches, since I don’t think we have any option to batch data in cf workers(like sending put request once in 1000 requests with all the log data)

would love for recommendation


#2

Since I run my application on AWS Beanstalk, my solution is to have a CloudWatch alarm like below:

image

With a trigger that sends a message to SNS. A Lambda function is then subscribed to that SNS topic, and if that alarm was triggered the Lambda function will use the CF API to turn on Under Attack Mode.

This is a very custom solution, requiring that your app reports metrics to CloudWatch which might not be possible. I recommend looking into the many different providers of automatic data point/log monitoring and see which one would work best for you.


What happens to bulk requests when Rate Limiting is OFF?
#3

thanks that interesting…
I know my questions is not 100% clear, I am looking for service who I can send info on each request hitting my cf worker, so in the end I will have a log file of all the requests hitting my worker… I cant use my servers for that because in big ddos attack they will fail


#4

I use https://logdna.com/ but only for error handling.


#5

You actually can do the type of batching you’re describing. The global memory of your Worker is persisted between invocations. You can buffer some number of log lines in an array, and then flush them with an asynchronous request when you have collected 25kb.


#6

Thanks looking good

hi thanks didnt know about it, so to batch them by 25kb will be anywhere near 50-60 requests, I can assume that the memory will persist for that many requests?

other related question, does it legal to use the worker to create something I can get in higher plans? like what I really want here is getting something like Access to Raw Logs


#7

Yes, your Worker will generally only be evicted occasionally. Many only get restarted once a week when we deploy a new runtime.

And yes, you can build anything you like in a Worker, there’s no issue if it’s an alternative to a Cloudflare feature.