What is the name of the domain?
The Hive Workshop is the largest Warcraft III community since 2004 with over 100k users, centered around map making, modeling, custom games, programming & art.
What is the error number?
ERR_ECH_FALLBACK_CERTIFICATE_INVALID
What is the error message?
ERR_ECH_FALLBACK_CERTIFICATE_INVALID
What is the issue you’re encountering
Downloading something goes to upload DOT hiveworkshop DOT com
and people get an error. I must use sub domain not-proxied for uploads to the site as they’re bigger than 100 MB.
What steps have you taken to resolve the issue?
Disabling QUIC, disabling TLS 1.3 (as suggested in other threads) but no dice. Still 24 hours later people are reporting errors when attempting to start downloads.
Was the site working with SSL prior to adding it to Cloudflare?
Yes
What is the current SSL/TLS setting?
Full
What are the steps to reproduce the issue?
Click Maps, click Download on anything, get error screen.
Screenshot of the error
It does not happen for everyone though. It’s 1-2 reports per day of 100s.
sjr
August 29, 2024, 6:47am
3
As the subdomain is not proxied and is set to “DNS only”, requests are not passing through Cloudflare so any problem is coming from your origin server.
Not so fast. I am getting the exact same problem as this:
Except it’s long ago and disabling TLS 1.3 did not work.
The certificate for the subdomain is not invalid, check for yourself. From what I understand, the problem stems from higher security on the root domain and then lower (still HTTPS) on the subdomain.
Here is someone else with this exact problem:
opened 09:29AM - 12 Nov 22 UTC
type: feature
### Your Feature Request
ECH (Encrypted client hello) is a developing specifi… cation for encrypting the original client hello in an HTTP1.1/HTTP2 context.
The main purpose is making SNI sniffing impossible by middle boxes and other such adversarial systems. The practical implementation is not dissimilar to how SSL certificates trust is established, by using certain new DNS records (+ DNS-over-HTTPS/TLS) as source of public keys involved (instead of a few CAs).
I had a short chat on the topic with @wlallemand at HAProxyConf, and he was aware of it and of the PoC referenced below. He hinted at it maybe being less relevant than before due to QUIC bringing encryption all the way through, but QUIC reaching the same level of usage as HTTP1.1/2 will take years. Especially when it still relies on Alt-Svc response headers at the moment, and while one will soon be able to advertise QUIC at the DNS level directly (see https://datatracker.ietf.org/doc/draft-ietf-dnsop-svcb-https/) this is also going to take a hot minute to be widely available, so I'm still quite interested in ECH in general (and hopefully I'm not alone in that).
Some relevant references/notes:
- IETF draft: https://datatracker.ietf.org/doc/draft-ietf-tls-esni/
- Cloudflare article on the topic: https://blog.cloudflare.com/encrypted-client-hello/ and public rollout announcement: https://blog.cloudflare.com/announcing-encrypted-client-hello/
- PoC implementation for HAProxy (and other popular webservers/load-balancer software): https://defo.ie/ (and specifically https://github.com/sftcd/openssl/blob/ECH-draft-13a/esnistuff/haproxy.md)
- Browser tracking bug for Chromium: https://bugs.chromium.org/p/chromium/issues/detail?id=1091403 and feature now shipping to stable in version 117: https://chromestatus.com/feature/6196703843581952
- Browser tracking bug for Firefox: ~https://bugzilla.mozilla.org/show_bug.cgi?id=1590863~ https://bugzilla.mozilla.org/show_bug.cgi?id=1725938
- No tracking bug for Webkit since it's more constrained in scope (as a rendering engine rather than a fully-fledged browser per se, see https://github.com/WebKit/standards-positions/issues/46)
More specifically for HAProxy, the work done by the DEfO PoC people has progressed quite a bit on the OpenSSL side:
- ECH tracking issue https://github.com/openssl/openssl/issues/7482
- Which itself is waiting on HPKE support (issue https://github.com/openssl/openssl/issues/14748) but the relevant PR for that issue is getting to its final stages so hopefully getting close 🤞 (the PR author is the same person that did the early exploration of ECH on defo.ie)
This is still somewhat early days (need HPKE merged, ECH to go from draft to RFC, and OpenSSL to adopt ECH) but I thought I'd raise this issue to have it in the tracker.
### What are you trying to do?
Use ECH with HAProxy
### Output of `haproxy -vv`
```plain
HAProxy version 2.7-dev8-7941ead+mangadex-cd2a7ce 2022-11-01T14:10+00:00 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.4.143-1-pve #1 SMP PVE 5.4.143-1 (Tue, 28 Sep 2021 09:10:37 +0200) x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -ggdb3 -gdwarf-4 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment -DMAX_SESS_STKCTR=5
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_STATIC_PCRE2=1 USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1
DEBUG = -DDEBUG_MEMORY_POOLS -DDEBUG_STRICT
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE +STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT +QUIC +PROMEX -MEMORY_PROFILING +SHM_OPEN
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1q+quic-mangadex-cd2a7ce 1 Nov 2022
Running on OpenSSL version : OpenSSL 1.1.1q+quic-mangadex-cd2a7ce 1 Nov 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 14.0.6
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
```
I don’t know a ton about this, but I see CF is giving my subdomain an ech thing in dig
:
root@localhost:~# dig +short https upload.hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=95.217.146.41 ech=AEX+DQBBggAgACBXw8l9AXjPoyTZnJZRkVPwX0X1tUfzxFsxtfs2P/iEegAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA=
root@localhost:~# dig +short https www.hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=104.21.24.123,172.67.218.199 ech=AEX+DQBBggAgACBXw8l9AXjPoyTZnJZRkVPwX0X1tUfzxFsxtfs2P/iEegAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3035::6815:187b,2606:4700:3037::ac43:dac7
sjr
August 29, 2024, 8:11am
7
I can see you have set the domain to max TLSv1.2, but www
still supports TLSv1.3 so I assume you’ve set TLSv1.2 using a Page Rule or Configuration rule?
If so, can you try disabling TLSv1.3 globally on this page…
https://dash.cloudflare.com/?to=/:account/:zone/ssl-tls/edge-certificates
Otherwise it probably needs looking at by Cloudflare.
I have already tried disabling TLS v1.3 but the problem remained.
After looking more at the latest link from the haproxy ticket I found the following:
In my configuration I have proxy enabled for hiveworkshop.com
, www.hiveworkshop.com
but disabled for upload.hiveworkshop.com
.
When I do dig
I get this:
# dig +short https hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=104.21.24.123,172.67.218.199 ech=AEX+DQBB1QAgACCCe94ieZ7pFhg+8SQKfPYVTiwfqsDTLoUpA9DxUQ+rUgAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3035::6815:187b,2606:4700:3037::ac43:dac7
# dig +short https www.hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=104.21.24.123,172.67.218.199 ech=AEX+DQBBggAgACBXw8l9AXjPoyTZnJZRkVPwX0X1tUfzxFsxtfs2P/iEegAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3035::6815:187b,2606:4700:3037::ac43:dac7
# dig +short https upload.hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=95.217.146.41 ech=AEX+DQBBggAgACBXw8l9AXjPoyTZnJZRkVPwX0X1tUfzxFsxtfs2P/iEegAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA=
The ech=AEX...
for the unproxied upload.hiveworkshop.com
is wrong because it is not proxied and I don’t have ECH on my nginx server.
NOW. Someone in the ticket said:
Most likely they do it on @ if it is set to proxy mode, and then it applies to subdomains? Because on a zone of mine I do indeed not see it yet. Tho rte.ie might be on a non-free CF plan and they’re delaying the rollout for those. idk.
So I disabled proxy on hiveworkshop.com
but kept it on for www.hiveworkshop.com
.
Now the ech=
disappeared from upload.hiveworkshop.com
.
I think this is what I want. But I also think it might be a mistake on CF’s part to enable ech=
on subdomains that are not proxied.
I will run with this for a while to see if that solves the issues I am having.
Thanks @user3138 - We are going to rollback ECH. I think you have identified a corner case.
The HTTPS record SHOULD NOT be returned for the upload domain.
I assume you are gray clouding upload.
to an Orange Clouded root domain hiveworkshop . com
My quick hunch is CNAME flattening is causing this issue. We will investigate with the DNS team to fix. The ECH record should only be returned for proxied records.
The TLS 1.3 disable button would fix this as well, but we will investigate the root cause so will back this out.
Thanks for reporting and apologies.
Matt
4 Likes
Wow thank you for the response!
No problem at all and apologies again.
Matt
mm0109
August 29, 2024, 2:06pm
13
Adding to this in case anyone else is having this problem.
I was having the exact same symptoms (requests randomly failing, although most would work) and I found that disabling TLS 1.3 worked for me.
My web app sends API calls to a subdomain that is not proxied via Cloudflare, although the DNS is hosted on Cloudflare.
The specific troubleshooting steps I took were:
Chrome developer tools showed error: ERR_QUIC_PROTOCOL_ERROR.QUIC_NETWORK_IDLE_TIMEOUT
Disabled Experimental QUIC protocol via chrome://flags/#enable-quic
Re-tried request, which showed error: ERR_ECH_FALLBACK_CERTIFICATE_INVALID
Disabled TLS 1.3 via Cloudflare SSL/TLS settings
Tried again around 12 hours later.
I don’t seem to be getting any further errors, but I will update this post if I do.
system
Closed
August 31, 2024, 2:07pm
14
This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.