GSuite custom URLs only work with Cloudflare (proxied) when loaded via Google’s servers over http, ie. when both the browser and Cloudflare connect over non-SSL connection. This requires the url be http:// as well as your zone not having “always use HTTPS” turned on. If either of these are false Cloudflare will error.
Note that GSuite’s custom URLs don’t do anything other than provide a convenient redirect to a page that limits the account switcher to emails on your domain.
Since it’s just a redirect, my solution is to leave the “Custom URLs” part of the admin console at their default values, and either have a page rule or Cloudflare Worker active on that subdomain to perform the redirect.
A worker for that would look like this:
let to_url = "https://mail.google.com/a/EXAMPLE.COM";
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
return new Response('', {status: 301, headers: {'location': to_url}})
}
(create the worker, save and deploy, then add a route on your zone with the route set to mail.example.com. Also make sure a DNS record exists for the subdomain mail as an A record with the value set to 192.0.2.1)