Ah ok, see this article for in-depth analysis: Community Tip - Fixing Error 525: SSL handshake failed
Since that is an SSL port, you do need to set up TLS and have an actual SSL certificate on your server. IIRC Flexible SSL mode doesn’t affect how SSL works on the other ports.
If you use an AWS load balancer or API gateway, you can get a valid certificate for free in the certificates manager (however, if you are not using an ELB, API gateway, etc. I would not recommend you start using them just for this).
If you’re using a plain ec2/lightsail instance, you can get a free certificate from LetsEncrypt, or use an Origin Certificate from Cloudflare