HTTP/2 Adaptive Window sometimes triggers ENHANCE_YOUR_CALM

[blyxxyz]

Downloading https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin tends to fail midway through if rustls and http2_adaptive_window are used.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
 let url = "https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin".parse()?;
 let https = hyper_rustls::HttpsConnector::with_native_roots();
 let client: hyper::Client<_, hyper::Body> = hyper::Client::builder()
 .http2_adaptive_window(true)
 .build(https);
 let mut response = client.get(url).await?;
 hyper::body::to_bytes(response.body_mut()).await?;
 Ok(())
}

[dependencies]
hyper = { version = "0.14.7", features = ["http2"] }
hyper-rustls = "0.22.1"
tokio = { version = "1.5.0", features = ["macros", "rt-multi-thread"] }

Outputs:

Error: hyper::Error(Body, Error { kind: Io(Kind(BrokenPipe)) })

It fails consistently on my Debian machine. It fails inconsistently on @austinbutler's macOS and Arch Linux machines, see ducaale/xh#135.

I can't reproduce it when using reqwest's native-tls feature, so it's possible the real problem is somewhere in hyper-rustls or rustls itself.

I don't know if http2_adaptive_window is really vital.

[seanmonstar]

Hm, interesting! I don't think the native-tls backend uses ALPN, so it may be using the HTTP/1 protocol then.

If you turn on the tracing logs, does it reveal anything interesting shortly before the error occurs? Particularly the trace/debug logs for h2...

[sfackler]

native-tls does support ALPN now, though I'm not sure if the hyper wrapper uses it.

[seanmonstar]

Good clarification, but I don't think reqwest currently is setup to use it.

[blyxxyz]

Aha:

 TRACE h2::codec::framed_read > read.bytes=17 
 TRACE h2::codec::framed_read > FramedRead::decode_frame; offset=17
 TRACE h2::codec::framed_read > decoding frame from 17B 
 TRACE h2::codec::framed_read > frame.kind=GoAway 
 DEBUG h2::codec::framed_read > received frame=GoAway { error_code: ENHANCE_YOUR_CALM, last_stream_id: StreamId(1) } 
 TRACE h2::proto::connection > recv GOAWAY frame=GoAway { error_code: ENHANCE_YOUR_CALM, last_stream_id: StreamId(1) } 

(here's a more complete log)

RFC 7540:

• ENHANCE_YOUR_CALM (0xb): The endpoint detected that its peer is exhibiting a behavior that might be generating excessive load.

Maybe Cloudflare doesn't like something h2 is doing?

[seanmonstar]

Does it happen if you keep the adaptive window disabled? Usually I see that goaway code when receiving more pings than it likes (the adaptive code uses ping ttl to understand latency).

[blyxxyz]

No, I can't get it to happen without the adaptive window.

I have seven logged runs. Five of them crash after 101 pings (and 100 pongs), which does suggest a limit on pings.

One of them succeeds after 160 pings and 160 pongs, and one more crashes after 118 pings and 117 pongs.

[seanmonstar]

This is great info! So, to unblock yourself immediately, you can disable the adaptive window, and consider just setting a high constant window since you probably want to download the big file as quickly as possible.

I would like to be able to improve the BDP detector to not trigger issues like this, but I'm not sure what the best way to do so is. Unfortunately there isn't really a common way in HTTP/2 to send some sort of warning before just killing the connection (maybe some specific PING payload (b"stahpeet" XD)). It partly looks like just a max limit of 100, but then some of your runs suggest a higher limit, so I wonder if it decays over time (or is it similar to the gRPC ping strike thing, where it's upset at two PINGs without a DATA frame in-between...

[blyxxyz]

Thank you! It's great to know a reliable solution, I was worried it would be something less deterministic.

[seanmonstar]

I wanted to note that as of the latest version of hyper, the adaptive window feature has received some changes that should greatly reduce the number of pings, and make this less likely to get throttled. It also happens to improve benchmarks a bit, better than the default window, so I'm interested in eventually enabling this by default. But did you want to try your code again with the adaptive window enabled?

[blyxxyz]

Thank you, I think that fixed it! Even a 850MB file manages to download to the end. (The earlier test file was 125MB.)

We'll turn it back on, and if it's broken in other cases a user will hopefully let us know.