Are there bandwith logs to see requests?

bandwidth
cache
#1

I’ve been getting few spikes like this over the last few days. It usually takes out my server. I have no idea what it is, or whats causing it. As far as I can see, nothing special is happening at these times, and traffic just before hand, is a little on the low side.

Is there any place where I can pull a log for this time period, to see what the requests are?

I’m trying to figure out if there are IPs that I need to block, if its a small ddos, if its some hacker, or if its something legit that my server just isn’t handling well.

This bandwidth chart is so far the only thing I’ve seen that corresponds with the server outages I’m experiencing. Any additional information would be greatly helpful.

#2

First-party Cloudflare logs are only available on the Enterprise plan, but you could try installing this Cloudflare app (forum) and seeing logs in there, or you could write your own Worker script.

Just curious, what’s the median amount of “requests” your site has? The logflare server might not be running on powerful enough hardware to take something like 100 requests/second if your site is that heavy on traffic. cc @chasers

#3

Thanks for the reply. I thought that might be the case, but was hoping otherwise. I don’t normally need them. But since this has been reoccurring the last few days, I thought it might help identify the problem.

I’ll check out the cloudflare app that you linked.

By cloudflare analytics, it looks like we hover around 50 requests per second. Heavy days are probably just below 100.

#4

LOL was all set to do a load test and show you, but yeah we can handle the load. Great thing about Elixir / Phoenix…

Now the spikes will be harder for you to debug with our current feature set as we don’t keep things around for too long. I’m happy to work with you more closely and keep a lot more events for you for now. PM me if you’re interested.

I’m about to start on notifications for events. One thing I’m interested in for myself is a notification if we get over x requests per second so I can log in and check it out.

1 Like
#5

And there we’re actually handling double that volume because I’m logging to logflare with a worker also… turned that off temporarily so we should be able to handle any spike you’re seeing just in case.

#6

@fpanko has it happened on the same hour every day?

#7

No, its more of an anomaly. That has just happened to have happened a few times in a few days. Its actually quite unusual.

I’ve seen this spike at different times the last few days. It started Sunday, and happened a few times since. But almost never before. Traffic was pretty normal before and afterwards, and I wasn’t seeing anything too unusual in the server logs.

As far as logflare, what’s captured looks like it would be pretty helpful. It would just be a matter of being able to download, or copy the log, a few minutes after we see the problem. I’m really trying to find out what was so unusual about those minutes. (Usually its just a minute or two. This one, and another last night, just lasted substantially longer.)

#8

Well a couple other things I have on my todo list … archive to S3 or a capable database (e.g. Big Query is a great fit for log type stuff) or Elastic, etc.

Big Query or elastic would let you query easily. S3 logs would probably be split into files of 5 minutes of logs each. So you could pretty easily read those by hand if you knew roughly the minute of hour. Big Query is interesting though because to store the data it’s pretty much equal to S3 but then you only pay for when you want to query stuff.

Just throwing some ideas out there.

#9

@fpanko we now have average requests per second:

And this option … not yet functional but should be tomorrow. Basically it will trap logs that cause a significant increase in volume for you. It seems like you spike around 3x your average, at least what I’m seeing right now. So I’m thinking 5x will do it but we can set it to something higher if needed.

I haven’t noticed a spike like you’ve seen yet, at least I don’t think. But after this is done tomorrow we can let it run for the weekend and see if it catches anything.

1 Like
#10

@fpanko alright now if you go here: https://logflare.app/sources/142/edit

And then choose a source as your overflow, it will copy events there when you see spikes. Currently set at 10x your average. I think this will catch what you’re looking for.

Next time you see one of those lmk if anything shows up in there.

1 Like
#11

@chasers This is pretty awesome. Set up easily. Thanks for the super fast turn-around. It wasn’t expected, bur is definitely appreciated.

10x sounds like it might. 5x would for sure. Looking at this last one, we went from about 1gb to 6.6gb.

I’m currently under the impression that these spikes are the result of infrastructure changes at cloudflare. My uncached bandwidth went down to 0 after this. Unless their reporting is off, I suspect they spidered me pretty hard a few times to build up the cache. I appreciate the site being a bit faster than before. This was just a massive headache for a few days.

If I get another one of these, I’ll take a look at the logs and let you know. I’m curious if this was just temporary.

1 Like
#12

Cloudflare doesn’t fetch to add items to the cache unless it’s requested from someone at the edge.

#13

And now we have max events / second also to help us all qa that feature. Looks like you’re almost at 5x already (note avg was reset when I deployed this but it was around 50 I think).

#14

@fpanko we have email alerts now. You can set that up on your overflow source. See: Check out my new Cloudflare app!

closed #15

This topic was automatically closed after 30 days. New replies are no longer allowed.