Skip to content

BGP is not functional when Wireguard is enabled #11596

@muvaf

Description

@muvaf

We're using Calico in BGP mode with no encapsulation and wireguard enabled so that we can have virtual-kubelet nodes become part of the pod network. All the nodes have public IPs only with no VPC/VLAN.

While the inter-pod communication works, we continuously see the following in calico-node pods:

bird: Netlink: Network is unreachable
bird: Netlink: Network is unreachable
bird: ...

It seems like even though wireguard is enabled, Calico is trying to add node's public IPs as next hops instead of wireguard IPs and netlink returns error because it requires next hops to be on-link. Now I'm a bit vague on the requirements of what can constitute as next hop but gpt 5.2 says it must be on the same L2/L3 underlay and with public IP setups like ours, the only other on-link IP is the router gateway so that's why it fails.

Expected Behavior

When Wireguard is enabled, use wireguard IPs as node IPs in BGP as well.

Current Behavior

Even though wireguard is enabled, BGP uses node IP as next hops in routes.

Possible Solution

Use wireguard IPs as node IPs in BGP as well.

Steps to Reproduce (for bugs)

This is how we set it up on k0s:

# helm upgrade --install calico projectcalico/tigera-operator --version v3.31.3 --namespace tigera-operator --create-namespace -f calico-values.yaml
# kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true,"wireguardEnabledV6":false}}'
installation:
  cni:
    type: Calico
  calicoNetwork:
    bgp: Enabled
    serviceCIDR: "10.96.0.0/12"
    ipPools:
      - name: default
        cidr: 10.244.0.0/16
        encapsulation: None
kubeletVolumePluginPath: "/var/lib/k0s/kubelet"

BGP setup:

# kubectl annotate node eu-hel1-1-2362016 projectcalico.org/RouteReflectorClusterID=10.255.255.1
# kubectl label node eu-hel1-1-2362016 route-reflector=true
# kubectl annotate node eu-hel1-3-2585842 projectcalico.org/RouteReflectorClusterID=10.255.255.1
# kubectl label node eu-hel1-3-2585842 route-reflector=true
# kubectl annotate node eu-hel1-4-2585843 projectcalico.org/RouteReflectorClusterID=10.255.255.1
# kubectl label node eu-hel1-4-2585843 route-reflector=true
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  nodeToNodeMeshEnabled: false
  asNumber: 64512
  serviceClusterIPs:
    - cidr: "10.96.0.0/12"
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: route-reflector-mesh
spec:
  nodeSelector: "has(route-reflector)"
  peerSelector: "has(route-reflector)"
---
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
  name: peer-with-route-reflectors
spec:
  # Local nodes that will peer with this peer
  nodeSelector: "!has(route-reflector)"

  # Selecting the remote peer to peer with local nodes.
  peerSelector: "has(route-reflector)"

Context

The end goal is that we have non-kubernetes nodes that are able to participate in BGP and we have this setup working in another cluster where all are on the same physical network and without wireguard. So the point of wireguard here is to provide that L3 but it seems it's not used for BGP so we're not able to reach our goal.

We'll try to manage Wireguard external to Calico and make Calico use wg0 as its sole interface and report back here but ideally, we'd prefer Calico take care of Wireguard coordination.

Your Environment

  • Calico version: 3.13.1
  • Calico dataplane (bpf, nftables, iptables, windows etc.): iptables
  • Orchestrator version (e.g. kubernetes, openshift, etc.): k0s 1.32.4
  • Operating System and version: Ubuntu 24.04.3
  • Link to your project (optional): Proprietary

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions