CONNECT-UDP [QUIC] Stream reset: reset reason: protocol error, response details: http3.invalid_header_field

[rishabh78]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
[email protected] where the issue will be triaged appropriately.

Title: One line description
I have client which sends CONNECT-UDP traffic through an Envoy proxy, which forwards it to an upstream UDP server.

Description:

What issue is being seen?
I am seeing following error on envoy stream reset: reset reason: protocol error, response details: http3.invalid_header_field

Envoy configuration

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: UDP
        address: 127.0.0.1
        port_value: 10001
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: certs/servercert.pem
                private_key:
                  filename: certs/serverkey.pem
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  connect_matcher:
                    {}
                route:
                  cluster: cluster_0
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: cluster_0
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http3_protocol_options:
            allow_extended_connect: true
    load_assignment:
      cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10002
    transport_socket:
      name: envoy.transport_sockets.quic
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
        upstream_tls_context:
          sni: localhost

UDP server

# udp_server.py
import socket

def start_udp_server():
    server_address = ('127.0.0.1', 10002)
    buffer_size = 1024

    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as server_socket:
        server_socket.bind(server_address)
        print(f"UDP server is listening on {server_address[0]}:{server_address[1]}")
        
        while True:
            data, client_address = server_socket.recvfrom(buffer_size)
            print(f"Received {data} from {client_address}")
            response = b"Response from UDP server"
            server_socket.sendto(response, client_address)

if __name__ == "__main__":
    start_udp_server()

Client

# connect_udp_client.py
import asyncio
from aioquic.asyncio import connect
from aioquic.quic.configuration import QuicConfiguration
from aioquic.h3.connection import H3_ALPN, H3Connection
from aioquic.h3.events import HeadersReceived, DataReceived, H3Event
from aioquic.asyncio.protocol import QuicConnectionProtocol

class CustomQuicClient(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.h3_connection = H3Connection(self._quic)
        self.events = []

    def quic_event_received(self, event):
        for http_event in self.h3_connection.handle_event(event):
            self.events.append(http_event)

async def main():
    # Configure QUIC client
    configuration = QuicConfiguration(is_client=True)
    configuration.alpn_protocols = H3_ALPN
    configuration.verify_mode = False  # Disable verification for local testing
    configuration.server_name = "localhost"  # Set the appropriate SNI

    async with connect("127.0.0.1", 10001, configuration=configuration, create_protocol=CustomQuicClient) as protocol:
        quic = protocol._quic
        h3 = protocol.h3_connection

        # Create a new stream
        stream_id = quic.get_next_available_stream_id()

        # Send CONNECT-UDP request headers
        target_host = "127.0.0.1"
        target_port = 10002
        h3.send_headers(
            stream_id=stream_id,
            headers=[
                (b":method", b"CONNECT-UDP"),
                (b":authority", f"{target_host}:{target_port}".encode()),
            ],
        )

        # Send some data over the UDP connection
        message = b"Hello from CONNECT-UDP client"
        h3.send_data(stream_id, message, end_stream=False)

        # Handle incoming events
        await asyncio.sleep(1)
        for event in protocol.events:
            if isinstance(event, HeadersReceived):
                print(f"Received headers: {event.headers}")
            elif isinstance(event, DataReceived):
                print(f"Received data: {event.data.decode()}")

if __name__ == "__main__":
    asyncio.run(main())

Envoy logs

[2024-11-14 15:14:07.360][2386822][debug][http] [source/common/http/conn_manager_impl.cc:385] [Tags: "ConnectionId":"15930477111956885085"] new stream
[2024-11-14 15:14:07.360][2386822][debug][http] [source/common/http/conn_manager_impl.cc:1894] [Tags: "ConnectionId":"15930477111956885085","StreamId":"18077200307027120781"] stream reset: reset reason: protocol error, response details: http3.invalid_header_field
[2024-11-14 15:14:08.256][2386793][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-14 15:14:13.259][2386793][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-14 15:14:18.264][2386793][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-14 15:14:18.454][2386817][debug][http] [source/common/http/conn_manager_impl.cc:385] [Tags: "ConnectionId":"11738211879808972429"] new stream
[2024-11-14 15:14:18.454][2386817][debug][http] [source/common/http/conn_manager_impl.cc:1894] [Tags: "ConnectionId":"11738211879808972429","StreamId":"5847963984936782337"] stream reset: reset reason: protocol error, response details: http3.invalid_header_field

[rishabh78]

@DavidSchinazi

[yanavlasov]

Adding @danzh2010 for any insights

[danzh2010]

The request has some invalid HTTP header. Can you share the requests you are sending?

[rishabh78]

'''
h3 = protocol.h3_connection

    # Create a new stream
    stream_id = quic.get_next_available_stream_id()

    # Send CONNECT-UDP request headers
    target_host = "127.0.0.1"
    target_port = 10002
    h3.send_headers(
        stream_id=stream_id,
        headers=[
            (b":method", b"CONNECT-UDP"),
            (b":authority", f"{target_host}:{target_port}".encode()),
        ],
    )

'''

[danzh2010]

''' h3 = protocol.h3_connection

    # Create a new stream
    stream_id = quic.get_next_available_stream_id()

    # Send CONNECT-UDP request headers
    target_host = "127.0.0.1"
    target_port = 10002
    h3.send_headers(
        stream_id=stream_id,
        headers=[
            (b":method", b"CONNECT-UDP"),
            (b":authority", f"{target_host}:{target_port}".encode()),
        ],
    )

'''

@DavidSchinazi Is this how CONNECT-UDP header should be like?

[rishabh78]

I am using this clinet

# connect_udp_client.py
import asyncio
from aioquic.asyncio import connect
from aioquic.quic.configuration import QuicConfiguration
from aioquic.h3.connection import H3_ALPN, H3Connection
from aioquic.h3.events import HeadersReceived, DataReceived, H3Event
from aioquic.asyncio.protocol import QuicConnectionProtocol

class CustomQuicClient(QuicConnectionProtocol):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.h3_connection = H3Connection(self._quic)
        self.events = []

    def quic_event_received(self, event):
        for http_event in self.h3_connection.handle_event(event):
            self.events.append(http_event)

async def main():
    # Configure QUIC client
    configuration = QuicConfiguration(is_client=True)
    configuration.alpn_protocols = H3_ALPN
    configuration.verify_mode = False  # Disable verification for local testing
    configuration.server_name = "localhost"  # Set the appropriate SNI

    async with connect("127.0.0.1", 10001, configuration=configuration, create_protocol=CustomQuicClient) as protocol:
        quic = protocol._quic
        h3 = protocol.h3_connection

        # Create a new stream
        stream_id = quic.get_next_available_stream_id()

        # Send CONNECT-UDP request headers
        target_host = "127.0.0.1"
        target_port = 10002
        h3.send_headers(
            stream_id=stream_id,
            headers=[
                (b":method", b"CONNECT-UDP"),
                (b":authority", f"{target_host}:{target_port}".encode()),
            ],
        )

        # Send some data over the UDP connection
        message = b"Hello from CONNECT-UDP client"
        h3.send_data(stream_id, message, end_stream=False)

        # Handle incoming events
        await asyncio.sleep(1)
        for event in protocol.events:
            if isinstance(event, HeadersReceived):
                print(f"Received headers: {event.headers}")
            elif isinstance(event, DataReceived):
                print(f"Received data: {event.data.decode()}")

if __name__ == "__main__":
    asyncio.run(main())

Please feel free to suggest any other library / option of testing this envoy config @DavidSchinazi @danzh2010 . Can I warp CONNECT-UDP with http3 curl ?

[rishabh78]

@danzh2010 @DavidSchinazi I tried using masque_client too

./bazel-bin/quiche/masque_client --disable_certificate_verification 127.0.0.1:10001  127.0.0.1:10002
E1116 01:03:07.662913  453146 masque_client.cc:128] Failed to connect. Error: QUIC_NETWORK_IDLE_TIMEOUT
E1116 01:03:07.663169  453146 masque_client_tools.cc:116] Failed to prepare MasqueEncapsulatedClient for 127.0.0.1:10002

envoy error logs

[2024-11-16 01:03:03.655][453085][info][quic] [external/com_github_google_quiche/quiche/quic/core/tls_server_handshaker.cc:982] No hostname indicated in SNI
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:393] [Tags: "ConnectionId":"14877056478440268800"] new stream
[2024-11-16 01:03:03.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:165] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] Received headers: { :method=CONNECT, :protocol=connect-udp, :scheme=https, :authority=127.0.0.1:10001, :path=/.well-known/masque/udp//0/, }.
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1183] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] request headers complete (end_stream=false):
':method', 'GET'
':scheme', 'https'
':authority', '127.0.0.1:10001'
':path', '/.well-known/masque/udp//0/'
'upgrade', 'connect-udp'
'connection', 'upgrade'

[2024-11-16 01:03:03.662][453085][warning][misc] [source/common/http/header_utility.cc:383] CONNECT-UDP request with a malformed URI template in the path /.well-known/masque/udp//0/
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/filter_manager.cc:1084] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] Sending local reply with details invalid_path
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1878] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] encoding headers via codec (end_stream=false):
':status', '404'
'content-length', '37'
'content-type', 'text/plain'
'date', 'Sat, 16 Nov 2024 01:03:03 GMT'
'server', 'envoy'

[2024-11-16 01:03:03.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:55] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] encodeHeaders (end_stream=false) ':status', '404'
'content-length', '37'
'content-type', 'text/plain'
'date', 'Sat, 16 Nov 2024 01:03:03 GMT'
'server', 'envoy'
.
[2024-11-16 01:03:03.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_stream.cc:14] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] encodeData (end_stream=true) of 37 bytes.
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1993] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] Codec completed encoding stream.
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:257] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] doEndStream() resetting stream
[2024-11-16 01:03:03.662][453085][debug][http] [source/common/http/conn_manager_impl.cc:1950] [Tags: "ConnectionId":"14877056478440268800","StreamId":"17777896718110749399"] stream reset: reset reason: local reset, response details: -
[2024-11-16 01:03:04.972][453078][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-16 01:03:07.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:325] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] received STOP_SENDING with reset code=6
[2024-11-16 01:03:07.662][453085][debug][quic_stream] [source/common/quic/envoy_quic_server_stream.cc:352] [Tags: "ConnectionId":"14877056478440268800","StreamId":"0"] received RESET_STREAM with reset code=6
[2024-11-16 01:03:09.973][453078][debug][main] [source/server/server.cc:237] flushing stats
[2024-11-16 01:03:14.974][453078][debug][main] [source/server/server.cc

[DavidSchinazi]

@rishabh78 CONNECT-UDP uses different headers than what CONNECT uses.

The headers generated by your python client are incorrect:

  headers=[
                (b":method", b"CONNECT-UDP"),
                (b":authority", f"{target_host}:{target_port}".encode()),
            ],

An example of valid headers is:

HEADERS
:method = CONNECT
:protocol = connect-udp
:scheme = https
:path = /.well-known/masque/udp/192.0.2.6/443/
:authority = example.org
capsule-protocol = ?1

This is documented in RFC 9298.

However, masque_client should work. That said, you've found a bug in masque_client. I'll have a fix for that shortly, but in the meantime please add https:// to your second URL parameter to work around the bug. Example:

./bazel-bin/quiche/masque_client --disable_certificate_verification 127.0.0.1:10001 https://127.0.0.1:10002

Additionally, note that you can pass in --stderrthreshold=0 and --v=2 to enable logging in masque_client. Example:

./bazel-bin/quiche/masque_client --stderrthreshold=0 --v=2 --disable_certificate_verification 127.0.0.1:10001 https://127.0.0.1:10002

[rishabh78]

Hey @DavidSchinazi ,
The masque_client forces me to pass upd server address but what if my usptream udp cluster has multiple endpoint for load balancing example

cluster_name: cluster_0
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10002
       - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10003
      - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10004

How will this work ? I want envoy to do load balancing between multiple udp upstreams servers , but in ./bazel-bin/quiche/masque_client --disable_certificate_verification 127.0.0.1:10001 https://127.0.0.1:10002

we are specifying https://127.0.0.1:10002

[DavidSchinazi]

You can't load balance in this way. CONNECT-UDP works similar to CONNECT in that the client tells the proxy what target is it trying to reach. For example:

Client -- Proxy -- Target

Target in this case is a host+port, or in other words an application somewhere with an open listening UDP socket.

Separately, you can load balance at the HTTP layer. In that world, you can have the proxy send the HTTP request to another HTTP server. For example:

Client -- Proxy -- Server1
               \-- Server2

You can combine those two features, by having the proxy load balance the CONNECT-UDP request to another proxy. For example:

Client -- MainEnvoy -- UpstreamEnvoy1 -- Target
                   \-- UpStreamEnvoy2 --/

In that scenario, the client (in this case masque_client) just knows about the Target (in your example https://127.0.0.1:10002/). And then the MainEnvoy config just knows about UpstreamEnvoy1 and UpstreamEnvoy2.

[rishabh78]

@DavidSchinazi the upstreams are udp backends

Just for clarification

Client:
Sends a CONNECT-UDP request to Main Proxy (HTTP/3, port 10001).

Main Proxy:
Handles CONNECT-UDP request and forwards tunneled UDP packets to Proxy (UDP load balancer on port 20001).

Proxy (UDP Load Balancer):
Load balances UDP packets across multiple upstream servers.

UDP Servers:
Respond to the UDP packets and send the responses back through the chain.

./bazel-bin/quiche/masque_client masque://127.0.0.1:10001 127.0.0.1:20001 --disable_certificate_verification

Do you mean this @DavidSchinazi

[DavidSchinazi]

I'm sorry, but I'm not sure I understand what you mean by Proxy (UDP Load Balancer). Is this a pure UDP load balancer, or is this an HTTP/3 proxy? Please make a diagram with all the ports included.

[rishabh78]

@DavidSchinazi

Main proxy

static_resources:
  listeners:
  - name: http3_listener
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10001  # Main Proxy's HTTP/3 port
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: certs/servercert.pem
                private_key:
                  filename: certs/serverkey.pem
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: udp_proxy
              domains: ["*"]
              routes:
              - match:
                  connect_matcher: {}
                route:
                  cluster: udp_proxy_cluster
                upgrade_configs:
                - upgrade_type: CONNECT-UDP
          http_filters:
          - name: envoy.filters.http.router
          http3_protocol_options:
            allow_extended_connect: true
          upgrade_configs:
          - upgrade_type: CONNECT-UDP
  clusters:
  - name: udp_proxy_cluster
    connect_timeout: 1s
    type: STRICT_DNS
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http3_protocol_options:
            allow_extended_connect: true
    load_assignment:
      cluster_name: udp_proxy_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 20001

Proxy (LB )

static_resources:
  listeners:
  - name: udp_listener
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 20001  # UDP Proxy listening port
    udp_listener_config:
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - filters:
      - name: envoy.filters.udp_listener.udp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.udp.udp_proxy.v3.UdpProxyConfig
          stat_prefix: udp_proxy
          cluster: udp_cluster

  clusters:
  - name: udp_cluster
    connect_timeout: 1s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: udp_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10002
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 10003

./bazel-bin/quiche/masque_client masque://127.0.0.1:10001 udp://127.0.0.1:20001 --disable_certificate_verification

[DavidSchinazi]

Thanks for the diagram. What you're describing isn't possible with CONNECT-UDP, because the client controls the target IP and port, not the load balancer. What's the motivation behind your design choices? What are you trying to do?

[rishabh78]

Hey @DavidSchinazi ,
I have clients which are using HTTP3 to send request to UDP backends. I am planning to tunnel the request using envoy and was thinking to use CONNECT_UDP . The only problem with CONNECT_UDP is how do I load balance the request among multiple udp upstream .

In the above diagram,
Client sends a CONNECT-UDP request to Main Proxy (HTTP/3, port 10001) which handles CONNECT-UDP request and forwards tunneled UDP packets to Proxy which load balance the request and forward it to a udp usptream

Client
. Do you have any suggestion for this ?

[DavidSchinazi]

What protocol are you running over UDP between the Client and the UDP Servers? Is it HTTP/3 or something else?

[rishabh78]

HTTP/3

[DavidSchinazi]

In that case you don't need to use connect-udp. If you're comfortable with the proxies having access to the encrypted data then you can have the client send the requests to the main proxy and have them load balanced individually.

[rishabh78]

@DavidSchinazi Envoy doesn't directly support HTTP to UDP bridging. Do I need to write a custom filter for this ?

The proxy listens for HTTP/3 traffic and terminates the QUIC connection. How will It extracts the HTTP payload and forwards it as raw UDP packets to the second proxy / upstream ?

For this case, do I need to write some custom filter ?

[DavidSchinazi]

I'm sorry, I don't understand what you're trying to do.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.