The reality today is that clients do guess, it does sometimes take minutes to fail, and they really do send credentials to unexpected places. I don’t like it any more than you do, but this is already reality.
Mail clients are a great example, in a worst-case scenario clients often try the bare domain, various prefixes (mail., imap, imap4, smtp., pop, pop3, etc) each on multiple ports (25, 110, 143, 465, 587, 993, 995, and I’ve seen some that hit 2525 and others), potentially on each of IPv4 and IPv6. If the domain has a wildcard DNS record pointing to an IP that drops packets (rather than sending a connection failure packet) then each of these takes one timeout to fail. The default timeout is typically going to be 60 seconds for a TCP connection and 300 seconds for each command if a TCP session is accepted and the server decides to be slow.
While you could make multiple attempts in parallel, typically most mail clients don’t.
Yes, this is a worst case scenario, if the connections get a proper TCP RST or ICMP response then no timeout are needed, often there is only IPv4, and hostnames that don’t exist can be quickly skipped. These are not always things that are under my control.
Clients also do end up sending credentials: Consider the above example, my mail server might be
secretmail.example.com, but my wildcard DNS record might point to a webserver hosted by a third party who happens to have their own SMTP/POP3/IMAP servers. In this case the client will get a connection and will then attempt to authenticate. It fails, but the damage is done: the credentials have been transmitted across the wire.
It is a lot easier to convince client authors to stop guessing when they get real answers from SRV records, but it will likely never happen that clients stop guessing in the absence of autoconfiguration hints simply because creating a negative first-try onboarding experience results in a high user abandon rate, and increases technical support load for those users who do stick around. Some protocols have a high adoption rate for SRV records and therefore default to “no record = no service autoconfiguration”, but for legacy protocols that predate SRV records autoconfiguration guesswork is the best fallback.
I haven’t seen any suggestion as to why valid records should be refused.