I want to create worker that will send log data, so in time of ddos attack I will have better understanding of the attack and how to block it using cf Firewall.
the problem is in ddos attack I can get a lot of rps so I need 3th solution for that I found https://logz.io/ is anyone know any other cheap solution? I saw about aws kinesis but for it to be economical good it needs to get data at 25kb batches, since I don’t think we have any option to batch data in cf workers(like sending put request once in 1000 requests with all the log data)
Since I run my application on AWS Beanstalk, my solution is to have a CloudWatch alarm like below:
With a trigger that sends a message to SNS. A Lambda function is then subscribed to that SNS topic, and if that alarm was triggered the Lambda function will use the CF API to turn on Under Attack Mode.
This is a very custom solution, requiring that your app reports metrics to CloudWatch which might not be possible. I recommend looking into the many different providers of automatic data point/log monitoring and see which one would work best for you.
thanks that interesting…
I know my questions is not 100% clear, I am looking for service who I can send info on each request hitting my cf worker, so in the end I will have a log file of all the requests hitting my worker… I cant use my servers for that because in big ddos attack they will fail
You actually can do the type of batching you’re describing. The global memory of your Worker is persisted between invocations. You can buffer some number of log lines in an array, and then flush them with an asynchronous request when you have collected 25kb.
hi thanks didnt know about it, so to batch them by 25kb will be anywhere near 50-60 requests, I can assume that the memory will persist for that many requests?
other related question, does it legal to use the worker to create something I can get in higher plans? like what I really want here is getting something like Access to Raw Logs
You can use XpoLog https://www.xplg.com/.
Send your events directly to XpoLog from your appliances using syslog protocol (It has built-in integration for syslog). XpoLog will manage all your syslog data. Then you will be able to monitor, analyze, search, visualize, etc.
XpoLog also has an app for Syslog data and others.
hope I helped.
Thank you Mr Spammer, I will now use your amazing service that you recommended so discretely with your first single post in this forum - even though applicances and syslogs are not possible in workers…
Hi, I’m working on an api proxy app for Cloudflare that can stream the logs to AWS kinesis. Once the logs are in Kinesis we can batch the logs over to Logz.io. The app supports writing logs straight to logz.io, but they don’t really like writing separate log-messages when the volumes get high… Let me know if this is something you would be interested in testing out and I can send you a worker file.
It looks like a really nice service!
I don’t have provide logging backend, but try to integrate with any existing log-services out there. Think pushing to logdna shouldn’t be a problem.
We do provide a few extra things that the logdna worker isn’t providing, such as duration/ttfb, tracking of aborted requests, downloaded bytes and logging of the request body. Let me know if it’s something that could be interesting.
@boybet2 I think you’ve seen Logflare but would love to get you using it. Getting your logs into BigQuery is going to be orders of magnitude cheaper than logdna or any Elastic based solution. If you use Logflare we provide the alerting and stuff on top of that and can shuffle your logs to your own BigQuery table for you so you automatically keep all your data for however long you want.
</pitch>
Even if you don’t use Logflare you should get your logs into BigQuery as it’s much better for long term storage and analysis than anything else I’ve seen.
Hi look really I really want to use it but few things make me postpone the transition I am already in aws so it much easier for me to store them in s3 than adding another service to the stack
but I do have a few questions:
do you store\have access to the logs?(sorry for being rude but since most free services sell the data I need to make sure)
any option for different theme in the log viewer page? instead of black-green do sometime like black-gray-white
is there option to customize the default line text in the log viewer page?
do you have any usage limit? my main goal is to log millions of events in time of ddos attack and I would like to clarify this line in the Terms of Service(in normal times I have only around 1.5gb uncompressed logs a month):
Member shall pay associated fees for Services based on the product used
Yes, we have access to your logs as we need to write to that table and other things, but we plan on offering paid plans here soon with generous limits. Like 13 million events/month for $20 a month. And we charge you just to shuffle the data to BigQuery, you pay the underlying backend cost (which with BQ is pretty negligible as long as you keep your queries in line). Absolutely refuse to sell your data to 3rd parties.
Yes, definitely plans for themes / customizing the log stream view.
With the Cloudflare app you can choose from like 15 fields and pick the order, but then we give you everything about every request (except the body) in the metadata payload. So, request/response headers, cf data, status code, and origin_time.
No real usage limits right now until we have the paid plans in place. I have a rate limit of 3000 requests per minute per source to keep the service from going down but I can increase that for you if you need.
Yes, that line is in there for our paid plans (coming soon).
Setting up BG and GCP I’ve found is way more user friendly than AWS so it’s pretty straight forward, plus we do all of the table setup for you once you have BigQuery enabled.
But yeah I totally get the need to stay only on AWS. Ultimately you will be able to “bring your own backend” so we can talk about that if you like. I think Athena is really the AWS answer to BigQuery but you also get Data Studio with BigQuery which is super straight forward (I’ve never played with AWS BI tool though).
Oh and workers are free for both of us since it’s a Cloudflare app that initiates them
I got attacked by “script kid” who manage to send me around 40m request in hour, so in that case your service will just limit me down
but even if I could live with that, its even barely enough for peak times in normal usage
I don’t sure I really understand what limiting you?
don’t you just send the data from the worker directly into bigquery?
I dont really sure I understand, does they stored anywhere but my bigquery table? if yes are you removing them after finish doing whatever you do with them?
anyway i guess my best option is just to wait for you paid plans
Like I said that’s adjustable. We don’t rate limit your website, we rate limit the amount of logs you can send us. We don’t alter your traffic at all we just pull out the request/response data and send that to our logging endpoint.
And we don’t go directly to bigquery because we want to give you a UI to stream data into, do some alerting if you want/etc. Like I use it to monitor our signups and text me each time someone signs up.
We also manage the BigQuery schema for you, basically making it schemaless as you never know exactly what request/response headers you’re going to get. To do that we have to pipe it through Logflare.
We keep that last 100 messages in memory, and any buffered messages that still need to get written to BigQuery but the buffer stays at zero or close to it.
