We are a software company that is hosting web software in the cloud. Every instance is behind Cloudflare, this is around 3800 instances atm. So im guessing the issue is too much data.
We have implemented this solution: https://developers.cloudflare.com/logs/analytics-integrations/elastic/
Cloudflare logs are being pushed to an Amazone S3 bucket. With a local logstash installation we pick them up and forward them to Elastic cloud. I can see all the data in the elastic cloud under Discover. We have added all the data from the message part into separate json fields to make it readable and searchable.
So far so good. We have imported the dashboard.json from githhub: https://github.com/Cloudflare/Cloudflare-elastic/releases/tag/v0.1
And this will give errors failing shards and timeouts.
When I remove the scripted fields the board will load and show me some data. A day later when more logs have been collected it won’t show any data anymore because of failing shards.
I have tested with 10 up to 200 shards but no difference. The shards keep on failing.
Can anyone confirm that we are just dealing with too much data for these dashboards or is there an option for us to tweak it?