Problem with TLS on secured websocket to websocket server in AWS instance

I registered a subdomain to an amazon ec2 public DNS(of windows instance), and there is a websocket server to echo any messages.
I faced in some problems with it.

  1. I could get success to get message back with ip address, but couldn’t get success with subdomain, of course without certificates installed.
    ws://xx.xx.xx.xx:xx:myport worked.
    ws://ec2-xx-xx-xx-xx.compute.amazonaws.com:myport worked also.
    ws://sub.mydomain.com:myport didn’t work.

  2. I ran webserver with certificate to access tls applied websites.(especially windows, C#, SuperSocket library) and it is configured with SSL certificate.(from Sectigo).
    But an exception is occured on serverside.
    Here is a screenshot.

Please help me if anyone could face in these kind of problems, or anyone knows why this can be happened.
Thanks