We have a small service which provides content with the data we have.
someone can rent a package to create a blog using the content we provide, they can also do a custom domain.
We implemented a lot of rules to limit bad crawls and scrapers, but we just realized that there is a problem between Cloudflare and our service. This problem occurs when someone applies a custom domain with a domain using dns cloudflare, then he uses the cloudflare page rule and enables cache everything with ttl edge cache a month.
someone with malicious intent and wants to download all our content will use this to bypass our bad bot rules.
My question is, can I tell cloudflare not to cache all the pages in this user’s domain using the response header from our service?
This problem has given us a headache for the last few weeks.
I’ve tried to use headers
cache-control: no-cache, no-store, max-age=0
but cloudflare doesn’t respect that at all when the user enables cache everything.