Nginx http3 closes connection with big html page

I got a problem while testing an nginx server patched with Quiche implementation of HTTP/3 with curl: when I try to send multiple consecutive request for a small html page (~1kb), nginx responds correctly

    [email protected]:~# ./curl/src/curl https://192.168.19.128?[1-5] -Ik --http3

[1/5]: https://192.168.19.128?1 --> <stdout>
--_curl_--https://192.168.19.128?1
HTTP/3 200
server: nginx/1.16.1
date: Mon, 25 Nov 2019 13:44:21 GMT
content-type: text/html
content-length: 924
last-modified: Mon, 25 Nov 2019 12:07:59 GMT
etag: "5ddbc41f-39c"
alt-svc: h3-23=":443"; ma=86400
accept-ranges: bytes


[2/5]: https://192.168.19.128?2 --> <stdout>
--_curl_--https://192.168.19.128?2
HTTP/3 200
server: nginx/1.16.1
date: Mon, 25 Nov 2019 13:44:21 GMT
content-type: text/html
content-length: 924
last-modified: Mon, 25 Nov 2019 12:07:59 GMT
etag: "5ddbc41f-39c"
alt-svc: h3-23=":443"; ma=86400
accept-ranges: bytes


[3/5]: https://192.168.19.128?3 --> <stdout>
--_curl_--https://192.168.19.128?3
HTTP/3 200
server: nginx/1.16.1
date: Mon, 25 Nov 2019 13:44:21 GMT
content-type: text/html
content-length: 924
last-modified: Mon, 25 Nov 2019 12:07:59 GMT
etag: "5ddbc41f-39c"
alt-svc: h3-23=":443"; ma=86400
accept-ranges: bytes


[4/5]: https://192.168.19.128?4 --> <stdout>
--_curl_--https://192.168.19.128?4
HTTP/3 200
server: nginx/1.16.1
date: Mon, 25 Nov 2019 13:44:21 GMT
content-type: text/html
content-length: 924
last-modified: Mon, 25 Nov 2019 12:07:59 GMT
etag: "5ddbc41f-39c"
alt-svc: h3-23=":443"; ma=86400
accept-ranges: bytes


[5/5]: https://192.168.19.128?5 --> <stdout>
--_curl_--https://192.168.19.128?5
HTTP/3 200
server: nginx/1.16.1
date: Mon, 25 Nov 2019 13:44:21 GMT
content-type: text/html
content-length: 924
last-modified: Mon, 25 Nov 2019 12:07:59 GMT
etag: "5ddbc41f-39c"
alt-svc: h3-23=":443"; ma=86400
accept-ranges: bytes

If I try to make a single request to a medium/big html file, nginx respond correctly again, but when I try to make multiple consecutive request to a medium/big html page (>=30kb), nginx stop responding after an arbitrary number of requests (2-5 requests normally). Here’s an example made of 10 requests to the https://cloudflare-quic.com html page (which I downloaded on my server):

   [email protected]:~# ./curl/src/curl -Ik https://192.168.19.128/cloudflare.html?[1-10] --http3 -v

[1/10]: https://192.168.19.128/cloudflare.html?1 --> <stdout>
--_curl_--https://192.168.19.128/cloudflare.html?1
*   Trying 192.168.19.128:443...
* Sent QUIC client Initial, ALPN: h3-23
* h3 [:method: HEAD]
* h3 [:path: /cloudflare.html?1]
* h3 [:scheme: https]
* h3 [:authority: 192.168.19.128]
* h3 [user-agent: curl/7.67.0-DEV]
* h3 [accept: */*]
* Using HTTP/3 Stream ID: 0 (easy handle 0x5614ee569460)
> HEAD /cloudflare.html?1 HTTP/3
> Host: 192.168.19.128
> user-agent: curl/7.67.0-DEV
> accept: */*
>
< HTTP/3 200
HTTP/3 200
< server: nginx/1.16.1
server: nginx/1.16.1
< date: Mon, 25 Nov 2019 13:53:43 GMT
date: Mon, 25 Nov 2019 13:53:43 GMT
< content-type: text/html
content-type: text/html
< content-length: 106072
content-length: 106072
< vary: Accept-Encoding
vary: Accept-Encoding
< etag: "5ddbdc21-19e58"
etag: "5ddbdc21-19e58"
< alt-svc: h3-23=":443"; ma=86400
alt-svc: h3-23=":443"; ma=86400
< accept-ranges: bytes
accept-ranges: bytes

<
* Excess found: excess = 27523 url = /cloudflare.html (zero-length body)
* Connection #0 to host 192.168.19.128 left intact

[2/10]: https://192.168.19.128/cloudflare.html?2 --> <stdout>
--_curl_--https://192.168.19.128/cloudflare.html?2
* Found bundle for host 192.168.19.128: 0x5614ee56db00 [can multiplex]
* Re-using existing connection! (#0) with host 192.168.19.128
* Connected to 192.168.19.128 (192.168.19.128) port 443 (#0)
* h3 [:method: HEAD]
* h3 [:path: /cloudflare.html?2]
* h3 [:scheme: https]
* h3 [:authority: 192.168.19.128]
* h3 [user-agent: curl/7.67.0-DEV]
* h3 [accept: */*]
* Using HTTP/3 Stream ID: 4 (easy handle 0x5614ee56b2b0)
> HEAD /cloudflare.html?2 HTTP/3
> Host: 192.168.19.128
> user-agent: curl/7.67.0-DEV
> accept: */*
>
* Got h3 for stream 0, expects 4
* Got h3 for stream 0, expects 4
* Got h3 for stream 0, expects 4
* Got h3 for stream 0, expects 4
[...]

It stucks on this screen repeating “Got h3 for stream 0, expects 4”. Also I noticed that, when testing on smaller pages, that the smallest the file the bigger is the number of requests fullfilled before stop responding and start printing the error “Got h3 for stream x, expecting y”, whith the relation that y=x+4.
Also the access.log and the error.log are clean, meaning that it could maybe be some king of parameter missing in the server configuration, but I’m not sure about it.
Does anyone have an idea of what the problem could be?

My config

nginx version:

nginx version: nginx/1.16.1
built by gcc 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)
built with OpenSSL 1.1.0 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: 
--prefix=/root/nginx-1.16.1 
--with-http_ssl_module 
--with-http_v2_module 
--with-http_v3_module 
--with-openssl=../quiche/deps/boringssl 
--with-quiche=../quiche

nginx.conf:

user root;
# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

# only log critical errors
error_log logs/error.log crit;
error_log  logs/error.log debug;
error_log  logs/error.log  notice;
error_log  logs/error.log  info;

# provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = worker_connections * worker_processes
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;

    # optimized to serve many clients with each thread, essential for linux -- for testing environment
    use epoll;

    # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
    multi_accept on;
}

http {
    # cache informations about FDs, frequently accessed files
    # can boost performance, but you need to test those values
    open_file_cache max=200000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    # to boost I/O on HDD we can disable access logs
    access_log on;

    # copies data between one FD and other from within the kernel
    # faster than read() + write()
    sendfile on;

    # send headers in one piece, it is better than sending them one by one
    tcp_nopush on;

    # don't buffer data sent, good for small data bursts in real time
    tcp_nodelay on;

    # reduce the data that needs to be sent over network -- for testing environment
    gzip on;
    # gzip_static on;
    gzip_min_length 10240;
    gzip_comp_level 1;
    gzip_vary on;
    gzip_disable msie6;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types
        # text/html is always compressed by HttpGzipModule
        text/css
        text/javascript
        text/xml
        text/plain
        text/x-component
        application/javascript
        application/x-javascript
        application/json
        application/xml
        application/rss+xml
        application/atom+xml
        font/truetype
        font/opentype
        application/vnd.ms-fontobject
        image/svg+xml;

    # allow the server to close connection on non responding client, this will free up memory
    reset_timedout_connection on;

    # request timed out -- default 60
    client_body_timeout 10;

    # if client stop responding, free up memory -- default 60
    send_timeout 2;

    # server will close connection after this time -- default 75
    keepalive_timeout 30;

    # number of requests client can make over keep-alive -- for testing environment
    keepalive_requests 100000;
    
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
  ########################################################
	########################################################  
        server {


        access_log  logs/access.log  main;
     	  sendfile on;
    	  tcp_nopush on;
    	  tcp_nodelay on;
    	  keepalive_timeout 65;
    	  types_hash_max_size 2048;
    	  # server_tokens off;
        gzip  on;
      
		    # Enable QUIC and HTTP/3.
        listen 443 quic reuseport;

        # Enable HTTP/2 (optional).
        listen 443 ssl http2;

        ssl_certificate      certificate.pem;
        ssl_certificate_key  key.pem;

        # Enable all TLS versions (TLSv1.3 is required for QUIC).
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        
        # Add Alt-Svc header to negotiate HTTP/3.
        add_header alt-svc 'h3-23=":443"; ma=86400';
 
        listen       80;
        server_name  localhost;

        location / {
            root   html;
            index  index.html index.htm;
        }
 
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        } 
        ###Limits the maximum number of concurrent HTTP/3 streams in a connection.
        http3_max_concurrent_streams 256;
        
        ###Limits the maximum number of requests that can be served on a single HTTP/3 connection, 
        ###after which the next client request will lead to connection closing and the need of establishing a new connection.
        http3_max_requests 20000;
        
        ###Limits the maximum size of the entire request header list after QPACK decompression.
        http3_max_header_size 100000k;
        
        ###Sets the per-connection incoming flow control limit.
        http3_initial_max_data 2000000m;
        
        ###Sets the per-stream incoming flow control limit.
        http3_initial_max_stream_data 1000000m;
        
        ###Sets the timeout of inactivity after which the connection is closed.
        http3_idle_timeout 1500000m;
    }
 ########################################################
	########################################################
}

Curl version

curl 7.67.0-DEV (x86_64-pc-linux-gnu) libcurl/7.67.0-DEV BoringSSL zlib/1.2.11 nghttp2/1.39.2 quiche/0.1.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets

Thats a question best for https://webmasters.stackexchange.com/questions/tagged/nginx or https://www.nginx.com/resources/wiki/community/

actually Nginx HTTP/3 patch lives in CF Quiche repo at https://github.com/cloudflare/quiche/issues so probably best to log the issue there :slight_smile:

nginx issues https://github.com/cloudflare/quiche/issues?q=nginx+sort%3Aupdated-desc