Azure Conditional Access Trusted Locations

I have setup ZTNA and it is working well for https and ssh access. We have conditional access polices through Azure as this is our id provider. Which states that we are not able to login/reset users unless they are in the trusted IP range, which is our office IPs (4 ips) When connected with OpenVPN this works fine and it shows that we are part of the internal network. With ZTNA it doesn’t seem to work this way so would like to know if there is some config to allow this? Or if you have another recommendation.