An alert with no obvious owner
We turned HTTP/3 on for every hostname on our edge in September 2026. From 26 September the nginx error log had a new line in it, at the alert level, the second-highest nginx has. It came at most once per connection (340 of the 340 we checked), and across the fleet it ran at 8 to 35 a day.
The line is not ours alone. nginx issue #888, opened in September 2025, reports the same message from several operators, with no answer from the core team yet. Turning off quic_gso did not make it go away for the people reporting it.
What a visitor saw depended on what was on the connection. An audio stream or a page load re-requested within one to three seconds, usually over HTTP/2. A viewer of a live video stream sat on a frozen picture for 10 to 61 seconds; the alert marks the end of that freeze, not its start. A browser that hits this a few times stops trying HTTP/3 on that site for hours, so the cost compounds quietly. The count is also a lower bound: a path that drops large packets without sending any ICMP leaves no log line at all.
The timing pointed somewhere else first. Shortly before, we had changed how our edges run QUIC (persistent keys and eBPF packet routing, described at the end). The alert turned out to be older than that change and appeared on an edge that runs without eBPF routing, so we set the rollout aside and read the send path.
What the log says, and what 90 means
# nginx error log, one line per broken connection
[alert] 1234#1234: *5678 sendmsg() failed (90: Message too long) while sending frames, client: 203.0.113.7
# 90 is EMSGSIZE on Linux
$ grep -w EMSGSIZE /usr/include/asm-generic/errno.h
#define EMSGSIZE 90 /* Message too long */
How a datagram becomes too big for its own socket
QUIC starts every path at 1200 bytes, the size RFC 9000 guarantees a path must carry. nginx then runs datagram path MTU discovery (RFC 8899): it sends larger probe packets, and each acknowledged probe raises path->mtu, up to the client's max_udp_payload_size. On a clean 1500-byte path that is 1472 bytes of UDP payload. Larger datagrams carry the same bytes in fewer packets, which is why discovery is worth doing.
QUIC sockets are opened with IP_PMTUDISC_DO: the Don't Fragment bit is set and the kernel never fragments. A router whose next link is narrower drops a packet that does not fit and returns an ICMP "fragmentation needed" (ICMPv6 "packet too big") with its own MTU. The kernel stores that MTU for the destination address, for ten minutes by default (net.ipv4.route.mtu_expires, 600 seconds). From then on, sendmsg() with a larger datagram to that address fails at once with EMSGSIZE. The packet never leaves the host.
So there are two path MTUs: nginx's, which goes up when a probe is acknowledged, and the kernel's, which goes down whenever an ICMP message for that address arrives, from any router on the path, from the client's own modem, or occasionally with a wrong value. When the kernel's number falls below nginx's, the next full-size datagram is refused.
nginx already expected this for probes: they are sent with EMSGSIZE ignored. A regular datagram had no such branch. ngx_quic_create_datagrams() returned an error, the connection was closed with ngx_quic_close_connection(NGX_ERROR), the client received a CONNECTION_CLOSE with INTERNAL_ERROR and every stream on it was cut. A recoverable condition, "this path is narrower than we thought", was handled like a programming error. The same code is in the current nginx mainline.
What the network was actually sending us
Before writing a fix we wanted to know where the ICMP came from. We captured "fragmentation needed" messages for ten minutes on two edge servers, read-only, and deleted the raw capture afterwards because it contains client addresses.
There were 25 messages in ten minutes, every one of them for one of our own QUIC datagrams of 1424 to 1500 bytes; most were 1490 to 1500 bytes, the top of the probe range. The MTUs they reported were 1492 (PPPoE, the most common), 1420 (tunnels), 1485, 1480, 1460, 1456, 1440 and 1380. On one 1420-byte path the probes went 1453, then 1434, then 1424, a binary search refused at every step.
Who sent them was the surprise. 18 of the 25 came from routers along the path. 7, about a quarter, came from the client's own address: the home router or CPE doing PPPoE or NAT, telling us its own link is narrower. One of those reported "MTU 1440" for a 1212-byte datagram, an MTU larger than the packet it refused. The kernel cached 1440 anyway.
That settled the cause. Probes teach the kernel a per-client path MTU for ten minutes; nginx's own value can be above it; the next full-size data datagram gets EMSGSIZE. It also showed what a fix could not do: a path that reports less than 1228 bytes (a 1200-byte QUIC payload plus IP and UDP headers) cannot carry QUIC at all, and no client-side MTU logic changes that. None of the 25 was that low.
Seeing the ICMP and the kernel's cached path MTU yourself
# ICMP "fragmentation needed" (type 3, code 4) arriving at an edge
tcpdump -ni any 'icmp[icmptype] == 3 and icmp[icmpcode] == 4'
# the IPv6 equivalent, "packet too big" (ICMPv6 type 2)
tcpdump -ni any 'icmp6 and ip6[40] == 2'
# what the kernel now believes about that client, and for how long
$ ip route get 203.0.113.7
203.0.113.7 via 198.51.100.1 dev eth0 src 198.51.100.20 uid 0
cache expires 583sec mtu 1492
Reproducing it in three network namespaces
A bug that depends on somebody else's router is hard to fix against production, so we built the path ourselves: three Linux network namespaces (client, router, server) joined by veth pairs, a 40 ms round trip and 20 Mbit/s towards the client. The server ran our production nginx build. The client was aioquic, advertising a max_udp_payload_size of 1472, downloading an 8 MB file over HTTP/3. In the middle of the transfer the router's link towards the client was narrowed, with or without ICMP.
The cases that matter:
1500 → 1492 or 1400, ICMP delivered. Without the fix the connection was closed after about 1 MB (999 KB) with exactly one alert, every time. With the fix the 8 MB arrived complete, the checksum matched and the longest pause in the transfer was 0.06 seconds.
A clean 1500-byte path. Path MTU discovery behaved exactly as before.
A 1220-byte path, smaller than a 1200-byte QUIC payload needs. Both builds failed once, logged once and stopped, with no loop and no CPU spent spinning.
The same path with ICMP dropped (a black hole). Without the fix the transfer hung indefinitely, with no log at all. That case is not EMSGSIZE, since the kernel never learns anything, and upstream has a separate change for it, #1901 (PMTU black hole detection). Our own build carries a small black-hole fallback of its own; upstream we left that case to #1901, and the two changes complement each other.
All runs together came to 52, on both builds we deploy, with the same results.
A condensed version of the lab topology
# three namespaces: client <-> router <-> server
ip netns add cli; ip netns add rtr; ip netns add srv
ip link add c0 netns cli type veth peer name r0 netns rtr
ip link add s0 netns srv type veth peer name r1 netns rtr
ip -n cli addr add 10.0.1.2/24 dev c0; ip -n rtr addr add 10.0.1.1/24 dev r0
ip -n srv addr add 10.0.2.2/24 dev s0; ip -n rtr addr add 10.0.2.1/24 dev r1
for ns in cli rtr srv; do ip -n $ns link set lo up; done
ip -n cli link set c0 up; ip -n rtr link set r0 up
ip -n rtr link set r1 up; ip -n srv link set s0 up
ip -n cli route add default via 10.0.1.1
ip -n srv route add default via 10.0.2.1
ip netns exec rtr sysctl -qw net.ipv4.ip_forward=1
# 40 ms round trip, 20 Mbit/s towards the client
ip netns exec rtr tc qdisc add dev r0 root netem delay 20ms rate 20mbit
ip netns exec rtr tc qdisc add dev r1 root netem delay 20ms
# start nginx in srv and an 8 MB HTTP/3 download in cli, then
# narrow the client-side link: the router answers with ICMP
ip -n rtr link set r0 mtu 1400
# black-hole variant (no ICMP at all) -- the case #1901 covers
ip netns exec rtr iptables -A OUTPUT -p icmp --icmp-type fragmentation-needed -j DROP
The fix: take the send back, rebuild, search again
The useful thing about EMSGSIZE is that it is synchronous: the kernel refused the datagram before it left the host. nginx already has the machinery for a send that did not happen, because it handles EAGAIN the same way. ngx_quic_revert_send() puts the frames back and rewinds the packet numbers, and the loop builds them again.
The first version, which has run on our edges since 13 November 2026, did the minimum: on EMSGSIZE for a datagram above 1200 bytes, revert the send, set the path MTU to 1200, stop discovery on that path and rebuild the same frames in 1200-byte datagrams. At 1200 the error stays fatal. QUIC cannot work on a path that refuses its minimum size, and keeping it fatal means there is no path to a retry loop.
In review on the pull request, an operator who had been running his own fix since October, GitHub user stathis, argued that a connection should not stay at 1200 for the rest of its life: lower the ceiling to the refused size and keep discovering. He was right, and the revision now in #1984 lowers the path's maximum MTU to the refused size and restarts discovery below it. In the lab the datagram size settles 0.25 to 0.7 seconds after the ICMP at 1463 bytes on a 1492 link, 1370 on a 1400 link, 1338 over IPv6 on a 1400 link, instead of 1200. A second drop later in the same transfer (1400, then 1300) lowers the cap again and it settles at 1263. A port change after the drop keeps the discovered size. The refused size never exceeds the current MTU, so the maximum only goes down.
The change builds with -Werror with and without --with-debug, and the nginx test suite for HTTP/3 and QUIC passes, 402 tests, the same as master.
The core of the change (from nginx/nginx#1984)
/* ngx_quic_create_datagrams(), src/event/quic/ngx_event_quic_output.c */
n = ngx_quic_send(c, dst, len, path->sockaddr, path->socklen);
err = ngx_socket_errno;
if (n == NGX_ERROR) {
if (err == NGX_EMSGSIZE && len > NGX_QUIC_MIN_INITIAL_SIZE) {
/* send the same frames again in smaller datagrams */
c->write->error = 0;
ngx_quic_revert_send(c, preserved_pnum);
ngx_quic_path_mtu_rejected(c, path, len);
continue;
}
return NGX_ERROR; /* a refused 1200-byte datagram is still fatal */
}
/* ngx_quic_path_mtu_rejected(), src/event/quic/ngx_event_quic_migration.c */
path->max_mtu = size; /* never probe the refused size again */
path->mtu = NGX_QUIC_MIN_INITIAL_SIZE; /* 1200 */
if (path->state != NGX_QUIC_PATH_VALIDATING) {
ngx_quic_discover_path_mtu(c, path); /* search again, below the refused size */
}
GSO, half-sent batches and nonces
Our edges run with quic_gso off, so the first commit covered only ngx_quic_create_datagrams(). With quic_gso on, most data goes out in batches built by ngx_quic_create_segments() and handed to the kernel with UDP_SEGMENT, and the same EMSGSIZE still closed the connection there. stathis runs GSO everywhere, and that branch was where his alerts came from.
A batch is harder than a single datagram, because the kernel may already have sent part of it before it refused the rest; we saw that over IPv6. Rewinding the packet numbers, as the single-datagram path does, would then send two different packets under the same number, and in QUIC the packet number is part of the AEAD nonce. His commit skips the numbers of the refused batch instead of reusing them, and sends the frames again in new packets. In our lab no packet number went out twice and the skipped ones were never reused.
There was one more trap. Before Linux 6.14 (upstream commit 235174b2bed8, backported to 6.13.3, 6.12.14, 6.6.78, 6.1.129, 5.15.179, 5.10.235 and 5.4.291), the kernel returns EINVAL, not EMSGSIZE, when the GSO segment size exceeds the path MTU. The third commit treats EINVAL like EMSGSIZE in the GSO branch, for segments above 1200 bytes only. Our lab kernel already had the newer behaviour, so we tested it with an LD_PRELOAD shim that turns those EMSGSIZE errors into EINVAL: without the commit the connection closed (IPv4 8/8, IPv6 2/2), with it every run completed.
stathis opened his commit as a separate pull request on top of ours. We tested it, proposed carrying it in #1984 with his authorship, and he agreed and closed his own. The pull request now holds three commits, so all of #888 is in one place.
What changed in production
We rolled the first version out to a single edge server, checked it, and took it to the rest of the fleet the same evening, 13 November 2026. On the canary, CPU per gigabit of traffic stayed where it was.
Before the patch the alert ran at 8 to 35 a day across the fleet. In the first two days after the rollout there were two. Both were the case we chose to leave fatal: a datagram of 1200 bytes or less refused, on a path that cannot carry QUIC's minimum size. Every other EMSGSIZE now costs a connection one rebuilt flight of packets instead of the connection itself.
The upstream pull request, nginx/nginx#1984, was opened on 14 November 2026 and closes #888. As of 20 November it is open and waiting for review by the nginx team, with the contributor licence agreement signed by both authors. If #1901 lands first, the restart logic in #1984 already follows the same steps, so the two fit together.
Related: HTTP/3 connections that went silent after a NAT rebinding
The rollout we first suspected had its own story. We had a report of HTTP/3 connections that stopped answering after 10 to 15 seconds of idle time: no error and no reset, just silence until the client gave up.
Two things were behind it. First, without quic_host_key, every reload gives nginx a new random key for stateless resets. A client whose connection ends up at a worker that does not know it gets a reset it cannot validate, so it waits for its idle timeout instead, a minute or two in Safari. In the lab, with the CONNECTION_CLOSE lost after a reload, 8 of 8 clients hung until the timeout; with a persistent key 8 of 8 received a valid stateless reset in 3 ms or less and reconnected at once.
Second, NAT rebinding. A home router that has seen no traffic for a few seconds may give the flow a new source port. With reuseport, the kernel picks a socket by hashing the 4-tuple, so the new port can land on a different worker, one that does not know the connection. We reproduced it on live edges with 15 seconds of idle time and a port change: 5 of 6 and 6 of 6 connections hung until the 40-second timeout, while 6 of 6 on an unchanged port were fine. A persistent key alone does not fix this (7 of 8 still hung in the lab). quic_bpf does: an eBPF program steers each packet to the worker that owns its connection ID. With both in place the same test passed 16 of 16 and 8 of 8.
One warning if you run quic_bpf: do not swap the nginx binary in place with USR2. After our first binary upgrade that way, about 40% of new HTTP/3 handshakes timed out, because packets were reaching workers that did not own the connection. A reload did not fix it; a clean restart did. Our upgrade tooling now refuses USR2 when quic_bpf is on.
The two directives, as our edges run them
# main context: route each QUIC packet to the worker that owns the
# connection, by connection ID (Linux 5.7 or later)
quic_bpf on;
http {
# one key across reloads and workers, so stateless resets stay valid
# (generate once: openssl rand -out /etc/nginx/quic_host.key 32)
quic_host_key /etc/nginx/quic_host.key;
server {
listen 443 quic reuseport;
listen 443 ssl;
# ...
}
}
How to set it up
Step-by-step guides in the help centre:
Frequently asked questions
My nginx error log shows "sendmsg() failed (90: Message too long)". What can I do today?
The fix is in nginx/nginx pull request 1984, which applies to current mainline; until it is merged you can carry it as a patch in your own build. Turning off quic_gso does not help. nginx has no directive that caps the QUIC datagram size, and the kernel settings that would stop path MTU discovery (ip_no_pmtu_disc, mtu_expires) also change TCP, so we did not touch them.
Is this the same problem as PMTU black holes?
No. In a black hole the large packets are dropped and no ICMP comes back, so the kernel never learns anything and the sender sees only loss; nginx pull request 1901 handles that. Here the ICMP does arrive, the kernel refuses the next large datagram with EMSGSIZE, and nginx used to close the connection. The two fixes complement each other.
Why not send 1200-byte datagrams all the time?
Because larger datagrams carry the same data in fewer packets, which costs less per byte on both ends. That is why the fix goes back to 1200 only for the rebuilt packets and then searches again below the refused size, settling at 1370 bytes on a 1400-byte link in our lab.
Where do the ICMP messages come from?
In our ten-minute capture, 18 of 25 came from routers along the path and 7 from the client's own address, typically a home router doing PPPoE (MTU 1492) or a tunnel (MTU 1420). One reported an MTU larger than the packet it refused, and the kernel cached it anyway.
Are sites on CDN.com.tr affected?
Our edges have run the patched build since 13 November 2026. HTTP/3 connections on a path that narrows now continue at a smaller datagram size instead of being closed.