Something has changed in cloudflare which has broken my website

Hi,

around October 10th something has changed in cloudflare which has broken a functionality on my website, more specifically on a drupal 7 website batch/backgroud processes stopped working. I’m sure it’s cloudflare because if I pause it my website works normally, also I’m sure I have not changed anything on my website in that period of time.

I tried to disable all functionalities of cloudflare one by one to find out which one may cause the problem, without success. Only pausing cloudflare on the website fixes the problem.

Can anyone help with any hint on how to further troubleshoot this problem?
Is there a change history for cloudflare services where I can check which changes where made around that time?

Thank you.

Assuming this background process is making requests through a Cloudflare proxied domain, things suddenly stopping are often caused by expiry of an origin SSL certificate.

Other than that, can you check Security…Events… to see if your background process has started to be regarded as a bot and is being challenged or blocked. If so, you can use a WAF rule to skip that check for the host, URI or other pattern to match the request.

Hi sjr and thank you for your reply,
ssl certificate expires in two months, also I don’t see any log entry in the events page when I try to run a batch and it doesn’t work, I have already a WAF rule to allow traffic from my server ip. Only pausing cloudflare fixes the problem.

1 Like

Can you run the back end task manually or look at its logs and see if it outputs any errors, text or other information? It should be reporting why it can’t perform or connect.

Hi and thank you again,
unfortunately the only log entry I can find in drupal is about background processes being released from the queue since they never started. Also php error log doesn’t help at all.
kind regards

There’s a mention here that rocket loader can break stuff so try disabling that if it’s enabled.

I don’t use Drupal so in the absense of any information from the application or logs, just have to hope someone that’s had the same problem can offer something.

Hi and thank you for your help,
disabling rocket loader didn’t help, I also allowed all cloudflares ip in .htaccess without any improvement.

From ssh console I used curl to load the url which shoud fire the batch process: https://www.[hidden].com/batch?op=start&id=63092

and I get the following http status:

HTTP/2 301
date: Tue, 24 Oct 2023 16:34:45 GMT
location: http://www.[hidden].com/batch?op=start
cache-control: max-age=3600
expires: Tue, 24 Oct 2023 17:34:45 GMT
report-to: {“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report/v3?s=jnY%2BAD%2BKLcgkrCcjEqTzTpk53sz2IQqN0ySLDPgiOO%2F%2F5y3HwSanyo8XeJdcEBufZn1zVC%2F1oxGi5RrdN4ibhEp07Au%2B2LjJNo2nKRXASSbD9REDwZsEmUQUQFBaU3hLMWWTfPnD7t4%3D”}],“group”:“cf-nel”,“max_age”:604800}
nel: {“success_fraction”:0,“report_to”:“cf-nel”,“max_age”:604800}
server: cloudflare
cf-ray: 81b3a10b1a06670a-AMS

it seems cloudflare redirects the request removing an essential part of the query string (&id=63092)???

caching level is set to standard and I already had a rule to bypass caching for requests from server own ip.

Note that the redirect is going from https to http as well which, depending on your Cloudflare settings, will redirect back to https and likely end up in a loop, or just not get an answer. That could be the problem, need to find the root cause (origin settings, Cloudflare settings for SSL/TLS or HTTPS, etc).

I see one site in your account, cloudflare is not paused and the site loads fine. Are you still encountering an issue @massitrc?


for folks downtheline…

Audit Log - https://dash.cloudflare.com/?to=/:account/audit-log
Cloudflare Status
https://www.cloudflarestatus.com/
&
https://community.cloudflare.com/c/cloudflarestatus/55

Hi cloonan and thank you for your help,
I confirm I have one website which loads fine but since two weeks ago some administrative functionalitites have stopped working (batch/background processes), I confirm this functionality works fine if I pause cloudflare. I’ve tried to disable all enabled cloudflare features one by one but couldn’t find which one is responsible. No events are logged, no errors on my drupal error logs or php error log. It may have somethig to do with the behaviour described here: Something has changed in cloudflare which has broken my website - #8 by massitrc

At the moment I don’t have any other clue but I will post again in case of news.
thank you.

Can you enter that into Trace? It will show you how the Cloudflare features handle that request as it passes through and if you have any rogue rules that are causing the behaviour.

Hi and thank for helping,
here is the results of trace:

‘{“trace”:[{“step_name”:“http_request_sanitize”,“type”:“phase”,“matched”:false,“public_name”:“URL Normalization”,“trace”:[{“step_name”:“eea9e26ca80b47a6896bcf5775d2ad56”,“type”:“ruleset”,“matched”:false,“description”:“ruleset for controlling url normalization”,“name”:“Entrypoint for url normalization ruleset”,“kind”:“zone”}],“zoneName”:“[hidden].com”},{“step_name”:“explicit_cache_control”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“bic”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“rocket”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“wan_host”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“cache_lvl”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“minify”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“sec_lvl”,“type”:“page_rule”,“matched”:true,“zoneName”:“[hidden].com”},{“step_name”:“http_config_settings”,“type”:“phase”,“matched”:false,“public_name”:“Configuration Rules”,“trace”:[{“step_name”:“54d9373f357b41ebbe7b98aa0154bd98”,“type”:“ruleset”,“matched”:false,“name”:“default”,“kind”:“zone”}],“zoneName”:“[hidden].com”},{“step_name”:“file_upload_scan”,“type”:“product”,“matched”:false,“zoneName”:“[hidden].com”},{“step_name”:“http_request_firewall_custom”,“type”:“phase”,“matched”:false,“public_name”:“WAF Custom Rules”,“trace”:[{“step_name”:“896eef5d6430470094a400281b911c72”,“type”:“ruleset”,“matched”:false,“name”:“default”,“kind”:“zone”}],“zoneName”:“[hidden].com”},{“step_name”:“api_shield”,“type”:“product”,“matched”:false,“zoneName”:“[hidden].com”},{“step_name”:“http_request_firewall_managed”,“type”:“phase”,“matched”:false,“public_name”:“WAF Managed Rules”,“trace”:[{“step_name”:“77454fe2d30c4220b5701f6fdfb893ba”,“type”:“ruleset”,“matched”:false,“description”:“Created by the Cloudflare security team, this ruleset is designed to provide protection for free zones”,“name”:“Cloudflare Managed Log4J Ruleset”,“kind”:“managed”}],“zoneName”:“[hidden].com”},{“step_name”:“request_managed_headers”,“type”:“product”,“matched”:false,“managed_headers”:[{“id”:“add_client_certificate_headers”,“enabled”:false},{“id”:“add_visitor_location_headers”,“enabled”:false},{“id”:“remove_visitor_ip_headers”,“enabled”:false}],“zoneName”:“[hidden].com”},{“step_name”:“http_request_cache_settings”,“type”:“phase”,“matched”:true,“public_name”:“Cache Rules”,“trace”:[{“step_name”:“8726448a5cde4a66ac738ff2c229808e”,“type”:“ruleset”,“matched”:true,“name”:“default”,“kind”:“zone”,“trace”:[{“step_name”:“718bf6d7e4984de5900e9c0173e51021”,“type”:“rule”,“matched”:true,“action_parameter”:{“cache”:false},“expression”:“(http.request.uri.path eq "/batch") or (http.request.uri.path eq "/cron.php") or (http.request.uri.path eq "/update.php") or (starts_with(http.request.uri.path, "/admin")) or (http.request.uri.path eq "/authorize.php")”,“description”:“drupal services”,“action”:“set_cache_settings”}]}],“zoneName”:“[hidden].com”},{“step_name”:“response_managed_headers”,“type”:“product”,“matched”:false,“managed_headers”:[{“id”:“remove_x-powered-by_header”,“enabled”:false},{“id”:“add_security_headers”,“enabled”:false}],“zoneName”:“[hidden].com”}],“status_code”:301}’

It seems this issue is not new anyway: https://www.drupal.org/project/background_process/issues/2512696

1 Like

It shows there are 8 page rules, can you click the arrow to expand that and see what is happening inside the page rules.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.