Ports 2053 and 2083 fail when JS Challenge

#1

Hi, I am new to cloudflare. We had a DDOS attack and now I am testing it.
Everything works fine. BUT, we use 2 ports for login and other things.
I changed them to reflect the ports that cloudflare seems to support. Namely 2053 and 2083.

However, when someone is in a country that triggers a js challenge. The following happens:
You get the cloudflare window checking the browser.
Then he ends on the login page and when clicking login receives a cors violation.
Server side, yes, it only accepts origin calls coming from our domain. Which works great, except when js challenge is triggered.

I am completely lost as to how I should even approach this. Just adding allow origin * is not an option.