h3: add support for streaming HEADERS with new sequence #2153

LPardue · 2025-09-09T03:16:20Z

Calls to send_request(), send_response(), send_response_with_priority(),
and send_additional_headers() can fail with a StreamBlocked
error indicating lack of underlying transport capacity. When this
occurs, applications are expected to retry the operation when
the stream is later reported as writable.

However, certain conditions could mean that sufficient capacity
might never be made available, effectively permenantly blocking
header sends. The root cause of this problem was the choice to enforce
that a HEADERS frame is always sent (buffered into a quiche stream)
in whole.

This change adds new functions related to stream management and header
sending to support a streaming send design. These will produce a HEADERS
frame that can be sent in whole or in part, depending on available
capacity. When a frame is only partly sent, applications are notified and
can resume sending using the new continue_partial_headers()
method, once the stream is writable.

The new functions unify how clients and servers send headers.
For a client, the expected sequence is now something like:

Decide to initate a request
reserve_request_stream() - reserves a stream if limits allow
stream_headers() - inititates streaming of a HEADERS frame
continue_partial_headers() - if stream_headers() returned Error::PartialHeader

For a server, the expected sequence is now something like:

Receive a request via the poll() function and decide to respond
stream_priority() - set the stream's sending priority per RFC 9218
stream_headers() - inititates streaming of a HEADERS frame
continue_partial_headers() - if stream_headers return Error::PartialHeader

While headers are being streamed, other operations that
would cause an HTTP/3 frame to be sent on the stream are
prevented. HEADERS frames must be sent completely before
other operations are successful.

Applications do not need to manage the partial HEADERS
buffer, this is dealt with inside quiche.

Co-authored-by: Gregor Maier [email protected]

Calls to send_request(), send_response(), send_response_with_priority(), and send_additional_headers() can fail with a StreamBlocked error indicating lack of underlying transport capacity. When this occurs, applications are expected to retry the operation when the stream is later reported as writable. However, certain conditions could mean that sufficient capacity might never be made available, effectively permenantly blocking header sends. The root cause of this problem was the choice to enforce that a HEADERS frame is always sent (buffered into a quiche stream) in whole. This change adds new functions related to stream management and header sending to support a streaming send design. These will produce a HEADERS frame that can be sent in whole or in part, depending on available capacity. When a frame is only partly sent, applications are notified and can resume sending using the new continue_partial_headers() method, once the stream is writable. The new functions unify how clients and servers send headers. For a client, the expected sequence is now something like: 0. Decide to initate a request 1. reserve_request_stream() - reserves a stream if limits allow 2. stream_headers() - inititates streaming of a HEADERS frame 3. continue_partial_headers() - if stream_headers() returned Error::PartialHeader For a server, the expected sequence is now something like: 0. Receive a request via the poll() function and decide to respond 1. stream_priority() - set the stream's sending priority per RFC 9218 3. stream_headers() - inititates streaming of a HEADERS frame 4. continue_partial_headers() - if stream_headers return Error::PartialHeader While headers are being streamed, other operations that would cause an HTTP/3 frame to be sent on the stream are prevented. HEADERS frames must be sent completely before other operations are successful. Applications do not need to manage the partial HEADERS buffer, this is dealt with inside quiche. Co-authored-by: Gregor Maier <[email protected]>

LPardue · 2025-09-09T12:47:22Z

Still needs a bit more testing / hardening but is ready for initial review

gregor-cf

LGTM. Had a couple of smaller questions and comments but I'm really liking this new API.

gregor-cf · 2025-09-09T15:54:59Z

quiche/src/h3/ffi.rs

    }
 }

+#[no_mangle]


I think you're missing the reserve_request_stream implementation.

gregor-cf · 2025-09-09T15:58:19Z

tokio-quiche/src/http3/driver/client.rs

        // TODO: retry the request if the error is not fatal
-        let stream_id = driver.conn_mut()?.send_request(
+        let stream_id = driver.conn_mut()?.reserve_request_stream(qconn)?;
+        driver.conn_mut()?.stream_headers(


I would leave tokio-quiche as-is for now. It can continue to use the old functions until we update all of it to use the new ones.

gregor-cf · 2025-09-10T20:44:41Z

quiche/src/h3/mod.rs

            Error::MessageError => WireErrorCode::MessageError as u64,
            Error::ConnectError => WireErrorCode::ConnectError as u64,
            Error::VersionFallback => WireErrorCode::VersionFallback as u64,
+            Error::PartialHeader => 0x1000,


nit: do you want to use a larger number here to avoid possible future conflicts?

gregor-cf · 2025-09-10T20:50:57Z

quiche/src/h3/mod.rs

+        // If we received a GOAWAY from the peer, MUST NOT initiate new
+        // requests.
+        if self.peer_goaway_id.is_some() {
+            return Err(Error::StreamCreationError);


Should we use a different error for this to distinguish it from the self.is_server check about? The "old" send_request used FrameUnexpected

gregor-cf · 2025-09-10T21:59:02Z

quiche/src/h3/mod.rs

+        conn: &mut crate::Connection<F>, stream_id: u64, priority: &Priority,
+    ) -> Result<()> {
+        // Clamp and shift urgency into quiche-priority space
+        let urgency = priority


nit: should we check if the stream is in self.streams?

gregor-cf · 2025-09-10T22:24:42Z

quiche/src/h3/mod.rs

-            Ok(false) => return Err(Error::StreamBlocked),
+        let payload_len = self
+            .qpack_encoder
+            .encode(headers, &mut frame_payload)


I think that and the above change will make the compiler happy :-)

Suggested change

.encode(headers, &mut frame_payload)

.encode(headers, frame_payload)

gregor-cf · 2025-09-10T22:27:01Z

quiche/src/h3/mod.rs

+    ///     struct.Connection.html#method.continue_partial_headers
    /// [`FrameUnexpected`]: enum.Error.html#variant.FrameUnexpected
-    /// [`StreamBlocked`]: enum.Error.html#variant.StreamBlocked
+    /// [`PartialHeader`]: enum.Error.html#variant.PartialHeader


leave the StreamBlocked and remove the link continue_partial_headers()

gregor-cf · 2025-09-10T22:41:07Z

quiche/src/h3/mod.rs

+        self.send_headers_to_quiche(
+            conn,
+            stream_id,
+            is_trailer_section,


I don't think you need to pass is_trailer_section to send_headers_to_quiche(). You can directly call stream.mark_trailers_sent() here (and you don't need trailers_in_flight).

If I remember the H3 spec correctly, after this point the only the valid thing for the user todo on this stream is to call continue_partial_headers() so I don't think we need to distinguish trailers_in_flight from trailers_sent

gregor-cf · 2025-09-10T22:47:37Z

quiche/src/h3/mod.rs

+    /// reported as writable again.
+    ///
+    /// [`PartialHeader`]: enum.Error.html#variant.PartialHeaders
+    /// [`Done`]: enum.Error.html#variant.Done


I don't think we ever return Done

gregor-cf · 2025-09-10T22:50:59Z

quiche/src/h3/mod.rs

+    /// [`PartialHeader`]: enum.Error.html#variant.PartialHeader
+    pub fn stream_headers<T: NameValue, F: BufFactory>(
+        &mut self, conn: &mut super::Connection<F>, stream_id: u64,
+        headers: &[T], is_trailer_section: bool, fin: bool,


is it possible / legal to call this with is_trailer_section == true and fin == false? Should we check for it?

antoniovicente · 2025-10-08T20:34:59Z

apps/src/common.rs


 impl HttpConn for Http3Conn {
+    fn retry_partial_headers(&mut self, conn: &mut quiche::Connection) {
+        for req in self.reqs.iter_mut().filter(|r| {


Possible efficiency concern: this filter seems to go through the full list of active streams even if none of them is in the HeadersSendStatus::InProgess state.

Is that a problem?

Related question: is Http3Conn expected to be used in high performance applications?

antoniovicente · 2025-10-08T20:40:14Z

apps/src/common.rs

    ) {
-        let mut reqs_done = 0;
+        // First retry partial headers
+        self.retry_partial_headers(conn);


Should this code proceed to the 2nd loop in cases where continue_partial_headers failed in a retryable way?

I would have expected that at most 1 stream be in the HeadersSendStatus::InProgess state, but it seems like this code allows for multiple streams to move to the HeadersSendStatus::InProgess state

antoniovicente · 2025-10-08T20:41:43Z

quiche/include/quiche.h

+
+// Continues sending headers on the given stream.
+int quiche_h3_continue_partial_headers(quiche_h3_conn *conn,
+                            quiche_conn *quic_conn, uint64_t stream_id);


weird indentation

antoniovicente · 2025-10-08T20:42:00Z

quiche/include/quiche.h


+// Initiates streaming of a new HTTP/3 HEADERS frame.
+int quiche_h3_stream_headers(quiche_h3_conn *conn,
+                            quiche_conn *quic_conn, uint64_t stream_id,


weird indentation; 1 space missing on this line and next 2.

antoniovicente · 2025-10-08T20:52:42Z

quiche/src/h3/mod.rs

+    /// [`PartialHeader`]: enum.Error.html#variant.PartialHeaders
+    /// [`Done`]: enum.Error.html#variant.Done
+    pub fn continue_partial_headers(
+        &mut self, conn: &mut super::Connection, stream_id: u64,


Can we avoid this stream_id argument? And limit the number of streams that can be in the partial header sent state to 1?

I think we need to allow multiple streams to be in the partial header state. Otherwise stream_headers() would need to return an error if there's already a stream in a partial state and the application would need to deal with retries.

antoniovicente · 2025-10-08T22:26:34Z

quiche/src/h3/mod.rs

+    ///
+    /// [`PartialHeader`]: enum.Error.html#variant.PartialHeaders
+    /// [`Done`]: enum.Error.html#variant.Done
+    pub fn continue_partial_headers(


Do we need a check to make sure that send_body is not called on a stream that is waiting for calls to continue_partial_headers?

LPardue requested a review from a team as a code owner September 9, 2025 03:16

LPardue force-pushed the lucas/streaming-h3-header-sends-new-api-take-2 branch from e710ee1 to 62513ef Compare September 9, 2025 03:18

This was referenced Sep 9, 2025

h3: add support for streaming HEADERS sends #2147

Closed

h3: add support for streaming HEADERS sends with new functions #2148

Closed

gregor-cf reviewed Sep 10, 2025

View reviewed changes

gregor-cf mentioned this pull request Sep 17, 2025

tokio-quiche: retry client request and better client error handling #2139

Closed

antoniovicente reviewed Oct 8, 2025

View reviewed changes

	.encode(headers, &mut frame_payload)
	.encode(headers, frame_payload)

Uh oh!

h3: add support for streaming HEADERS with new sequence #2153

Are you sure you want to change the base?

h3: add support for streaming HEADERS with new sequence #2153

Conversation

LPardue commented Sep 9, 2025

Uh oh!

LPardue commented Sep 9, 2025

Uh oh!

gregor-cf left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants