CF Trace + Aioquic succeeds but Workers + Aioquic fails after GREASE

Developers Workers
#1

Hello!

I’m very interested in utilizing workers to replace my backend. I need the speed of QUIC to power my software, so since Workers supports HTTP3 it seems like an excellent pairing. However despite being able to reach my worker over H3 in Chrome, I’m having problems reaching it in Aioquic (Python’s only async QUIC+H3 library). This is puzzling because I’m able to reach Cloudflare trace with aioquic with no issues, but trying to reach my worker will hang the program. I don’t have a QLog, but I have output from Aioquic.

Below is from my application requesting /cdn-cgi/trace.

ProtocolNegotiated(alpn_protocol=‘h3’)
HandshakeCompleted(alpn_protocol=‘h3’, early_data_accepted=False, session_resumed=False)
ConnectionIdIssued(connection_id=b’gA\x1b\xc9\x83\xfbA\x02’)
StreamDataReceived(data=b’\x00\x04\x15\x06\x80\x01\x00\x00\xc2\xe0\xf25\xb8\xee y\xef)\xc7\x89\x08P\x8a\x8a’, end_stream=False, stream_id=3)
StreamDataReceived(data=b’\x02’, end_stream=False, stream_id=7)
StreamDataReceived(data=b’\x03’, end_stream=False, stream_id=11)
StreamDataReceived(data=b’\xea=\x0bvMW\xa5AGREASE is the word’, end_stream=True, stream_id=15)
StreamDataReceived(data=b’\xe4:\xf41\xc7\x8fb\x0b\x00\xc8Z\xf1\x1a\xea_\x0f\x89\x12GREASE is the word\[email protected]\x00\x00\xd9V\x96\xdf=\xbfJ\x05\xa55\x11*\x08\x02\n\x81z\xe3O\xdc\x10\x14\xc5\xa3\x7f\xf5\xe3_M\x87%\x07\xb6Ih\x1d\x85-$\xabX?_\x8fq\xf2\x8aRF\x02>\xf0\x80A:\xb5\xfc<\xbf_S\x84\xbf\x83O?\xfd-/\x9a\xcdaQ\x96\xdf=\xbfJ\x00e\x1dJ\x05\[email protected]\xa0\x01p\x00\xb8\x00\xa9\x8bF\xff\xe7\[email protected]\xb4fl=\nh=www.cloudflare.com\nip=...\nts=1634237360.769\nvisit_scheme=https\nuag=aioquic/0.9.15\ncolo=DFW\nhttp=http/3\nloc=US\ntls=TLSv1.3\nsni=plaintext\nwarp=off\ngateway=off\n\x00\x00’, end_stream=True, stream_id=0)
Response received for GET /cdn-cgi/trace : 0 bytes in 0.0 s (0.000 Mbps)
ConnectionTerminated(error_code=<QuicErrorCode.NO_ERROR: 0>, frame_type=None, reason_phrase=’’)

Below is from my application trying to reach my worker.

ProtocolNegotiated(alpn_protocol=‘h3’)
HandshakeCompleted(alpn_protocol=‘h3’, early_data_accepted=False, session_resumed=False)
ConnectionIdIssued(connection_id=b’;\xc9\xdb\xb4%\x1ee\xd2’)
StreamDataReceived(data=b’\x00\x04\x15\x06\x80\x01\x00\x00\xdf\x89\x1e\x11\x06|\xea!\xcd\x05\t\x8f$\x17\x1f\xbe’, end_stream=False, stream_id=3)
StreamDataReceived(data=b’\x02’, end_stream=False, stream_id=7)
StreamDataReceived(data=b’\x03’, end_stream=False, stream_id=11)
StreamDataReceived(data=b’\xc3\x91\x06|k\x9f\xe6KGREASE is the word’, end_stream=True, stream_id=15)

I am using aioquic 0.9.15 and this is a test using the standard example http3 client.
Thank you for the read!

#2

I’m sitting here with Wireshark and I think that the version of Quiche that Workers uses may differ from the rest of the Cloudflare stack. I’ve tried a few things to try and narrow it down. I copied 1 for 1 all of Chrome’s headers that were sent to my worker in it’s successful request into my aioquic request, to no avail. Comparing packets from Chrome and aioquic, the only difference I’m noticing is that Chrome never sends a SCID and the PKN in the Initial Packet is 1 versus aioquic’s 0. Since Cloudflare gets as far as responding to everything except for what the worker is responsible for, I’d like to think aioquic has a slightly different behavior from say Chrome or Firefox and it’s behavior that whatever build of Quiche that Workers is running on is expecting. Does anyone have an input or questions?

#3

To check if this was an issue with workers or aioquic’s implementation, I requested a Cloudflare owned workers site (https://silentspacemarine.com/’) with both Firefox (so that I had a truly first time negotiation of it connecting) and aioquic. They look identical, the only difference is that Workers never responds to the first stream, stream 0. Seeing as how /cgi-bin/trace has no cryptographic or protocol issues with responding to aioquic, I’m assuming this is a regression in Quiche or some other in-house software. Could anyone else try communicating with workers with aioquic and see if they get the same issue? Maybe some Cloudflare employees, as I think this is an internal issue?

#4

Hi @Ancillary,

Thanks for the report. There seems to be something up here that needs some further investigation.

#5

Much appreciated! Let me know if there’s any more info I can provide.