Getting 403 when trying to make insecure REST from a worker code

So
Somehow when accessing to an IP address I get the 403 (over both 443 or 80)
I gave it some random DNS name (which mismatches with the certificate) and I do get 526
When accessing over HTTP - it works as expected

I manage the API as well and control its Firewall - it should allow traffic over 443 and 80 from anywhere

The 403 with the IP address is definitely comes from Cloudflare as I don’t see this traffic in my API
When I log the response object from the Worker code I see -
“resp: {"webSocket":null,"url":"https://","redirected":false,"ok":false,"headers":{},"statusText":"Forbidden","status":403,"bodyUsed":false,"body":{"locked":false}}”