Enable CF to read and analyze web server logs from the backend server


I use the basic CF paid plan. Recently I had a security issue that a client, probably attacker, requested queries that my security CF config did not block, so the requests reached my backend server and caused the server’s CPU to spike badly for a rather long time.

I went to my CF dashboard but the non-security logs are limited in time range stored by CF and cannot filter by Ray ID (the CF given unique request ID), so I was left much in the dark here. I had to go to the web server’s logs and try to analyze them, without much success, as it was hard to correlated the CPU spike with root cause requests.

I understand CF’s limitation of storing logs in general and non-blocked request in particular, so I suggest to create a kind of a “log bridge”, that CF will get access to the relevant web server logs at the backend server (or from another storage location configured by the customer), instruct the customer how to enable adding the Ray ID value to be logged in the web server logs - and then, when the customers will do analysis at the CF dashboard, for non-blocked activity, CF will connect to the backend logs and query them in real time, I guess based on the Ray ID values as the correlation index.

This way CF will be a mirror of the web server’s logs, at the CF dashboard.

Another, more intrusive option, is to have CF develop a software agent application, to be installed on the backend operating system, to do the above but also gather data (stored locally at the backend server) like CPU value and which process consumed it (say, only the top 5 ones, to make it lite) - to automatically analyzed such issues and point as directly as possibly - which request(s) caused which issue.

This will help us much better understand what happened, what bypassed our CF security measures, hence how to harden our security at CF and possibly on the backend server.