We are using Plesk for mail and webhosting on the same server. If I proxy the main domain (so not the mail.domain) Outlook is unable to connect. If I disable the proxy on the main it is able to connect.
This problem is just with Outlook, using the webmail or other mail software like Spark works.
Welcome to the Cloudflare Community. 
This sounds like Outlook is configured to use the proxied apex domain instead of the mail hostname. Have you looked at the account server settings in your Outlook to see if that is the case?
If I force Outlook to use mail.domain. It won’t connect to the server, authentication failed.
You will need to work with your Plesk host to resolve the server authentication issue with your mail hostname. Any hostnames used for services other than HTTP or HTTPS need to be set to 
DNS Only.
Moving email to a separate server from your webhost is another good option since it avoids this an other problems altogether.