🐛 BUG: Nebula inappropriately prioritises IPv6 even when not available, resulting in failed connections, unnecessary relay use

[Clancy508]

What version of nebula are you using? (nebula -version)

1.9.6

What operating system are you using?

Fedora Linux

Describe the Bug

Nebula seems to fail to maintain direct connections using IPv4 if IPv6 appears available despite specifying an IPv4 subnet range in preferred ranges in the config file.

My setup is using Nebula to connect multiple virtual servers and some clients together, for convenience and security reasons the clients always use Nebula to communicate with the servers regardless of if they're locally connectable or roaming. The servers are behind a stateful firewall that blocks inbound and some outbound connections, server nodes punch out to allow Nebula connections. There are multiple lighthouse nodes, all of which can be directly connected without any firewall blocking access from server or client nodes (one on the LAN for HA purposes, 2 external nodes that all other nodes can reach).

On initial connection while running locally I get a direct connection with good performance and low latency, however after a period of time the connection seems to fail over to a relayed connection with ping times showing a latency double that of my ping to my external Lighthouse and a sudden performance degradation. This seems to take anywhere from seconds to multiple minutes. The only log messages I can see are msg="Failed to write outgoing packet" error="listener is IPv4, but writing to IPv6 remote", over and over. I'm not really sure why it's trying to use IPv6 given that all my nodes are configured to listen on IPv4 addresses only. Adding preferred_ranges: ["10.0.1.0/24"] (my LAN subnet) didn't help, both the server and client still attempt, and fail, to connect to IPv6 and then seemingly fail over onto a relayed connection instead of the known working IPv4 direct connection.

(Note I'm expecting this is to do with Nebula trying IPv6 connections when it isn't configured to listen for them because that's what's turning up in the logs, but I could be wrong and it could be that something else is forcing Nebula to switch to relayed connections. I can't really think of anything else though, if it was just difficulty punching out through the firewall why would it work directly at the start?)

Logs from affected hosts

The only error messages in the logs are one of the 2 below
level=error msg="Failed to send handshake message" error="listener is IPv4, but writing to IPv6 remote" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2853929362 localIndex=2853929362 remoteIndex=0 udpAddr="[<IPv6 address here>]:47473" vpnIp=y.y.y.y

level=error msg="Failed to write outgoing packet" certName=<node> error="listener is IPv4, but writing to IPv6 remote" localIndex=2500263824 remoteIndex=450316016 udpAddr="[<IPv6 address here>]:47473" vpnIp=y.y.y.y

Config files from affected hosts

I've snipped out irrelevant parts for brevity, I've also redacted specific IP addresses

static_host_map:
  "x.x.x.x": ["<public IPv4 address>:4242"]
  "x.x.x.x": ["<public IPv4 address>:4242"]
  "x.x.x.x": ["<LAN IPv4 address>:4242"]

lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - "x.x.x.x"
    - "x.x.x.x"
    - "x.x.x.x"

  local_allow_list:
    "10.89.0.0/16": false # Don't advertise Podman network addresses

listen:
  host: 0.0.0.0
  port: 0
  send_recv_error: private

punchy:
  punch: true
(only difference is here - server has respond: true set as well)

cipher: chachapoly

preferred_ranges: ["10.0.1.0/24"]

relay:
  relays:
    - x.x.x.x
    - x.x.x.x
    - x.x.x.x
  am_relay: false
  use_relays: true

[JackDoan]

There's a couple questions here, so I'm gonna try to pull them out and address them one-by-one:

The only log messages I can see are msg="Failed to write outgoing packet" error="listener is IPv4, but writing to IPv6 remote", over and over. I'm not really sure why it's trying to use IPv6 given that all my nodes are configured to listen on IPv4 addresses only.

Yeah, this is kind of an unpleasant quirk. If you want to forbid Nebula from trying to use IPv6, you can use lighthouse.remote_allow_list to forbid a node from trying to do a handshake to an IPv6 destination. (Should this be the default behavior, if we aren't bound to an IPv6 socket? Maybe! Today it is not though.)

Note I'm expecting this is to do with Nebula trying IPv6 connections when it isn't configured to listen for them because that's what's turning up in the logs, but I could be wrong and it could be that something else is forcing Nebula to switch to relayed connections. I can't really think of anything else though, if it was just difficulty punching out through the firewall why would it work directly at the start?

If you are switching to a relayed connection, that means, for whatever reason, the tunnel between the two nodes was abandoned, and when Nebula went looking for (or made) a new one, the relayed connection came back faster. When choosing which path to use, Nebula uses the one that completes the handshake the quickest, even if it's not as direct. This heuristic usually results in a desirable outcome, but not always.

The servers are behind a stateful firewall that blocks inbound and some outbound connections, server nodes punch out to allow Nebula connections.

Is it possible that your firewall is losing the UDP conntrack entry for your servers? Needing to re-punch could be why the relay path is getting selected. Decreasing punchy.delay and punchy.respond_delay might help, or, if you control the firewall rules, extending your conntrack timeouts or opening up some ports might do the trick?

Finally, if it works for your requirements/runtime environment, you could try going dual stack and binding to IPv6 as well. Less NAT is always nice :)

[Clancy508]

There's a couple questions here, so I'm gonna try to pull them out and address them one-by-one:

Yeah, sorry about that, and thanks for picking through it, it became a bit of a mess trying to figure out what I was seeing haha

(Should this be the default behavior, if we aren't bound to an IPv6 socket? Maybe! Today it is not though.)

I do think so, it doesn't make a lot of sense to advertise an address that isn't actually connectable. I think this was part of the problem I was seeing

Finally, if it works for your requirements/runtime environment, you could try going dual stack and binding to IPv6 as well. Less NAT is always nice :)

Poking and prodding around a lot seems to have mostly fixed it, the firewall isn't NAT as such (it's a hypervisor firewall but each VM still gets a direct IP on the network, just filtered), there was a misconfiguration though which was making punching not work so well, and there was a double NAT elsewhere I was able to get rid of which seems to have helped quite a bit. What I can't figure out though is why the connection would work smoothly for a brief moment before degrading, the worst case it should have been doing would have been relaying via the local lighthouse which is on the same LAN instead of the remote ones, but it would always wind up using the remote ones and wind up with massively degraded performance and some packet loss while trying to switch over. I'm not really sure how to reproduce it though and it's probably a quirk of my setup outside of the IPv6 when not bound issue, which it looks like you're already working on haha. Thanks again!