-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
We're using Calico in BGP mode with no encapsulation and wireguard enabled so that we can have virtual-kubelet nodes become part of the pod network. All the nodes have public IPs only with no VPC/VLAN.
While the inter-pod communication works, we continuously see the following in calico-node pods:
bird: Netlink: Network is unreachable
bird: Netlink: Network is unreachable
bird: ...
It seems like even though wireguard is enabled, Calico is trying to add node's public IPs as next hops instead of wireguard IPs and netlink returns error because it requires next hops to be on-link. Now I'm a bit vague on the requirements of what can constitute as next hop but gpt 5.2 says it must be on the same L2/L3 underlay and with public IP setups like ours, the only other on-link IP is the router gateway so that's why it fails.
Expected Behavior
When Wireguard is enabled, use wireguard IPs as node IPs in BGP as well.
Current Behavior
Even though wireguard is enabled, BGP uses node IP as next hops in routes.
Possible Solution
Use wireguard IPs as node IPs in BGP as well.
Steps to Reproduce (for bugs)
This is how we set it up on k0s:
# helm upgrade --install calico projectcalico/tigera-operator --version v3.31.3 --namespace tigera-operator --create-namespace -f calico-values.yaml
# kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true,"wireguardEnabledV6":false}}'
installation:
cni:
type: Calico
calicoNetwork:
bgp: Enabled
serviceCIDR: "10.96.0.0/12"
ipPools:
- name: default
cidr: 10.244.0.0/16
encapsulation: None
kubeletVolumePluginPath: "/var/lib/k0s/kubelet"BGP setup:
# kubectl annotate node eu-hel1-1-2362016 projectcalico.org/RouteReflectorClusterID=10.255.255.1
# kubectl label node eu-hel1-1-2362016 route-reflector=true
# kubectl annotate node eu-hel1-3-2585842 projectcalico.org/RouteReflectorClusterID=10.255.255.1
# kubectl label node eu-hel1-3-2585842 route-reflector=true
# kubectl annotate node eu-hel1-4-2585843 projectcalico.org/RouteReflectorClusterID=10.255.255.1
# kubectl label node eu-hel1-4-2585843 route-reflector=true
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
nodeToNodeMeshEnabled: false
asNumber: 64512
serviceClusterIPs:
- cidr: "10.96.0.0/12"
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: route-reflector-mesh
spec:
nodeSelector: "has(route-reflector)"
peerSelector: "has(route-reflector)"
---
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
name: peer-with-route-reflectors
spec:
# Local nodes that will peer with this peer
nodeSelector: "!has(route-reflector)"
# Selecting the remote peer to peer with local nodes.
peerSelector: "has(route-reflector)"Context
The end goal is that we have non-kubernetes nodes that are able to participate in BGP and we have this setup working in another cluster where all are on the same physical network and without wireguard. So the point of wireguard here is to provide that L3 but it seems it's not used for BGP so we're not able to reach our goal.
We'll try to manage Wireguard external to Calico and make Calico use wg0 as its sole interface and report back here but ideally, we'd prefer Calico take care of Wireguard coordination.
Your Environment
- Calico version: 3.13.1
- Calico dataplane (bpf, nftables, iptables, windows etc.): iptables
- Orchestrator version (e.g. kubernetes, openshift, etc.): k0s 1.32.4
- Operating System and version: Ubuntu 24.04.3
- Link to your project (optional): Proprietary