IPv4 endpoints on links with MTU < 1280 can't be accessed

[zajdee]

Hello,

this might not be an issue with Jool, rather a description of an issue we're seeing, with the hope that there's an option we've missed.

There are hosts in the IPv4 Internet that apparently reside on links with a very low MTU. When an IPv6-only host accesses those hosts and attempts to send a large packet, the router in front of the remote host, or the remote host, responds with ICMPv4 unreachable - need to frag. This packet gets delivered to Jool, and Jool translates it to an ICMPv6 packet too big, mtu message. So far so good and expected behaviour.

The problem here is that the IPv4 MTU is below IPv6 minimal MTU, hence the TCP connection stalls and eventually times out and breaks.

13:20:45.176062 IP 139.162.188.91 > 192.0.2.1: ICMP 139.162.188.91 unreachable - need to frag (mtu 905), length 136
13:20:45.176067 IP6 fd00:64::8ba2:bc5b > 2001:db8:dead:0:d250:99ff:fec1:f6e4: ICMP6, packet too big, mtu 1280, length 156
13:20:45.176079 IP 139.162.188.91 > 192.0.2.1: ICMP 139.162.188.91 unreachable - need to frag (mtu 905), length 136
13:20:45.176081 IP6 fd00:64::8ba2:bc5b > 2001:db8:dead:0:d250:99ff:fec1:f6e4: ICMP6, packet too big, mtu 1280, length 156
13:20:45.176181 IP 139.162.188.91 > 192.0.2.1: ICMP 139.162.188.91 unreachable - need to frag (mtu 905), length 136
13:20:45.176186 IP6 fd00:64::8ba2:bc5b > 2001:db8:dead:0:d250:99ff:fec1:f6e4: ICMP6, packet too big, mtu 1280, length 156
13:20:45.176388 IP 139.162.188.91 > 192.0.2.1: ICMP 139.162.188.91 unreachable - need to frag (mtu 905), length 136
13:20:45.176394 IP6 fd00:64::8ba2:bc5b > 2001:db8:dead:0:d250:99ff:fec1:f6e4: ICMP6, packet too big, mtu 1280, length 156
13:20:45.176406 IP 139.162.188.91 > 192.0.2.1: ICMP 139.162.188.91 unreachable - need to frag (mtu 905), length 136
13:20:45.176407 IP6 fd00:64::8ba2:bc5b > 2001:db8:dead:0:d250:99ff:fec1:f6e4: ICMP6, packet too big, mtu 1280, length 156
13:20:45.176723 IP 139.162.188.91 > 192.0.2.1: ICMP 139.162.188.91 unreachable - need to frag (mtu 905), length 136
13:20:45.176724 IP6 fd00:64::8ba2:bc5b > 2001:db8:dead:0:d250:99ff:fec1:f6e4: ICMP6, packet too big, mtu 1280, length 156
13:20:45.176976 IP 139.162.188.91 > 192.0.2.1: ICMP 139.162.188.91 unreachable - need to frag (mtu 905), length 136
13:20:45.176976 IP6 fd00:64::8ba2:bc5b > 2001:db8:dead:0:d250:99ff:fec1:f6e4: ICMP6, packet too big, mtu 1280, length 156

(fd00:64::/96 is my NAT64 testing prefix.)

We found the issue when running the ICMP blackhole check, the issue can be easily reproduced with just two commands executed from behind the NAT64:

perl -e "print 'packettoolongyaithuji6reeNab4XahChaeRah1diej4' x 180" > payload.bin
curl --connect-to '::[fd00:64::139.162.188.91]'  -v -s http://icmpcheck.popcount.org/icmp --data @payload.bin

I plan to test some other NAT64 implementations to see if they behave differently, but this looks like a fundamental protocol issue that cannot be easily solved.
At this moment, Jool and AWS' NAT64 implementations were tested, and both exhibit the same issue: which, again, seems to lead to a conclusion that this is a (NAT64) protocol issue rather than Jool implementation problem.

It seems I can work around the issue by clamping MSS on the TCP SYN/ACK packets received from the remote side (note that I run Jool in a namespace to which to_jool routes to, so this is happening before Jool sees the packet):

iptables -A FORWARD -p tcp -s 139.162.188.91 -o to_jool --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 865

The remote side by default advertises MSS 1460, so it's possible it's (intentionally, in this case) connected like this:

server <-> link with MTU 1500 <-> router <-> link with MTU 906 -> router <-> internet link with MTU 1500

[ydahhrk]

Well, this is what RFC 7915 has to say about this:

o  When translating an ICMPv4 "Fragmentation Needed" packet, the
   indicated MTU in the resulting ICMPv6 "Packet Too Big" will never
   be set to a value lower than 1280.  This ensures that the IPv6
   nodes will never have to encounter or handle Path MTU values lower
   than the minimum IPv6 link MTU of 1280.  See Section 4.2.

o  When the resulting IPv4 packet is smaller than or equal to 1260
   bytes, the translator MUST send the packet with a cleared Don't
   Fragment bit.  Otherwise, the packet MUST be sent with the Don't
   Fragment bit set.  See Section 5.1.

This approach allows Path MTU Discovery to operate end-to-end for
paths whose MTU are not smaller than the minimum IPv6 MTU of 1280
(which corresponds to an MTU of 1260 in the IPv4 domain).  On paths
that have IPv4 links with MTU < 1260, the IPv4 router(s) connected to
those links will fragment the packets in accordance with Section 2.3
of [RFC791].

So the strategy is "Jool should disable DF, so the low MTU router will fragment."

If you can confirm Jool is disabling DF, then the question becomes "why is the IPv4 router not fragmenting?"

If DF is enabled, then this is a Jool bug.

[ydahhrk]

Expected packet flow:

1. IPv6 client writes packet sized 1500 (probably).
2. Jool translates that to IPv4 packet sized 1480 (plus 20 bytes for IPv4 header, -40 bytes for IPv6 header). Because of the second RFC rule, DF will be enabled.
3. Router returns ICMPv4 Fragmentation Needed containing MTU 905.
4. Jool translates that to ICMPv6 Packet Too Big containing MTU 1280. (Should be 885, but can't be lower than 1280 because of first RFC rule.)
5. IPv6 client adjusts packet, sends packet sized 1280.
6. Jool translates that to IPv4 packet sized 1260 (+ 20 - 40), with DF disabled (because of second RFC rule).
7. IPv4 router fragments.