(Work in progress) E2EE setup across CG-NAT using Wireguard, radvd and keepalived

Created at: Feb 23, 2025 / Last updated at: Mar 17, 2025

This site describes attempt to create a provider, network and environment agnostic site-to-site tunnel through which an end-to-end encrypted Wireguard tunnel can be routed.

There’s a version of Pro Custodibus that uses a wireguard tunnel between every hop in a hub and spoke configuration. This version has a similar idea, but instead of using wireguard tunnels for each hop (as it is quite costly due to re-encryption) it uses a combination of ULA advertisements and static routes instead.

High Level Overview

The problem

I have many systems at multiple locations (including mobile devices) which I would like to be able to interconnect, but facing the following issues:

At some locations I only have a mobile connection behind CG-NAT
At some locations I have only IPv4
At some locations I do have a public routable IPv6 but it’s (at least officially) not static
At some locations I regularly switch between providers through SIM switching or similar

Conceptual overview

There are multiple sites which have a standard internet router, possibly behind CG-NAT. On each site there are servers which get their LAN IP address via DHCP as usual (it does not matter whether its IPv4, IPv6 or both).

Physical Networks

A /48 ULA range is chosen for which an edge router is placed on each site that advertises a unique /64 ULA subnet on the LAN, for example:

ULA Range: <ULA Prefix>::/48
Site A: <ULA Prefix>:A::/64
Site B: <ULA Prefix>:B::/64

The advertisement causes other machines on the same link (e.g. the servers) to assign themselves a stable IPv6 in the /64 range via SLAAC. As such, each server is going to have a unique ULA address across all sites.

The edge router also advertises itself as router for the entire /48 subnet, such that all traffic that is not on link is sent to it.

In between the edge routers there is a relay. Traffic destined for a range outside the /64 handled by the edge router is forwarded to the relay which then forwards it to the edge router (if any) that handles the destination range. To avoid issues with CG-NAT or similar, there are Wireguard tunnels between the edge routers and the relay with a PersistentKeepalive configured so that bi-directional communication is possible. The IP used within the tunnel is irrelevant, but it should be a /128 ULA address for each device, with the ULA address being outside the chosen /48 range.

Site-To-Site VPN

Once connectivity between the sites has been established, the servers can address each other via their ULA IP addresses and use them to build a wireguard mesh, where the endpoint for each peer is the respective ULA IP address. Inside the tunnels each server uses another /128 ULA address within a separate range.

With this configuration, servers do not need to know about the backing routing using the edge routers and relay. This allows an edge router to be portable and easily replaceable, without the need of additional configuration on the servers.

Wireguard Mesh

Inter-Site communication

Assume a source server at site A wants to communicate with a destination server at site B.

Server A configures a wireguard interface which contains the destination server B as peer, with the public/endpoint IP being server B’s ULA address. When a message is sent, it is encrypted using the public key of the destination server and routed to the edge router, which forwards it to the relay Due to the static routing of the ULA range, traffic is routed from the source server to the edge router at site A. The edge router in turn encapsulates the encrypted message into another wireguard tunnel to the relay. The relay then re-encapsulates the packet to send it to the edge router at site B. The edge router at site B decrypts it and sends it to the destination server. The destination server can then decrypt the message using its key.

Intra-Site communication

When two servers at the same site want to communicate with each other, they will do so directly thanks to neighbor discovery .

Configuration

The sections below describe the basic configuration for the VPN to work. Any up to date Linux-based OS should work for the edge routers and the relay, the configuration was only tested on an Ubuntu-24.04 minimal server, however.

Prerequisites

For this configuration to work, the following is required:

A server on each site’s link used to advertise an ULA range
An internet-addressable server to handle relaying traffic between sites, addressable via IPv4 and IPv6
A DNS A and AAAA entry pointing to the relay server

In addition, the design requires a total of 3 ULA ranges

A /48 Mesh ULA range, used for addressing and routing between the servers
A unique /128 ULA address for each server, outside the /48 range of (1), used within the wireguard tunnel mesh
A unique /128 ULA address for each edge router and relay, outside the /48 range of (1), used within the routing wireguard tunnels

Note that GUA addresses can be used as well (if any are available), however, LLA addresses should be avoided as they are not routable and require interfaces to be supplied for communication since they aren’t necessarily unique and have to be used with a target interface.

VPN Relay

The relay can be a cheap VPS somewhere on the internet. Optimally, it would be geographically close / in-between the sites. It should be reachable via a public, static IPv4 and IPv6 address and have DNS A/AAAA entries assigned to it (this will be important for the edge routers later on).

Wireguard routing tunnel configuration

First of all, wireguard-tools should to be installed. This gives access to wg-quick which is very convenient for setting up the interface.

sudo apt install wireguard-tools

Then generate the private/public key pair:

# Run the following as root
umask 077
wg genkey > /etc/wireguard/wgvpnr01-privatekey
wg pubkey < /etc/wireguard/wgvpnr01-privatekey > /etc/wireguard/wgvpnr01-publickey

Next, a wireguard interface needs to be configured.

# /etc/wireguard/wgvpnr01.conf
 
[Interface]
Address = <Routing Tunnel ULA>/128
ListenPort = <Listen Port>
PrivateKey = <Private Key>
 
# Enable IP forwarding while the interface is up
PreUp = sysctl -w net.ipv6.conf.all.forwarding=1
PostDown = sysctl -w net.ipv6.conf.all.forwarding=0
 
# Routing tunnel to edge router at site A
[Peer]
PublicKey = <Public Key>
AllowedIPs = <Peer Routing Tunnel ULA>/128,<Site ULA Prefix>::/64
 
# Routing tunnel to edge router at site B
[Peer]
PublicKey = <public key of site B edge router>
AllowedIPs = <Peer Routing Tunnel ULA>/128,<Site ULA Prefix>::/64
 
# ...

Enable the wireguard interface on startup (note that if it was manually started the interface must be taken down first).

sudo systemctl enable --now wg-quick@wgvpnr01.service

Firewall configuration

The firewall configuration below configures the following rules:

Input: Accept incoming traffic on the wireguard port and ICMP
Forward: Allow forward across the wireguard interface if the destination is in the ULA /48 range and the target port is in the wireguard port range configured on the servers

# /etc/nftables.conf
flush ruleset
 
# The interface through which forwarding between sites is allowed
# i.e. the wireguard interface configured in the previous step
define vpn_routing_interface = "wgvpnr01"
 
# The port the interface listens on
define vpn_routing_wireguard_port = <Port>
 
# The ULA range that is used to address devices across sites
define vpn_routing_full_range = <ULA Prefix>::/48
 
# The port range on which devices are listening for wireguard connections
define vpn_routing_dport_range = 58000-60000
 
table inet filter {
    chain input {
        # Default to drop
        type filter hook input priority filter; policy drop;
 
        # Allow established and related connections
        ct state established,related accept
 
        # Allow loopback traffic
        iif lo accept
 
        # Reject traceroute for 30 hops (this allows clients to see the hop IP)
        udp dport { 33434-33474 } reject
 
        # Allow incoming traffic on the routing port
        udp dport $vpn_routing_wireguard_port accept
 
        # Allow ICMP
        icmp type { echo-request, echo-reply, destination-unreachable, time-exceeded } accept
        icmpv6 type { echo-request, echo-reply, destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
 
        log prefix "Dropped Input: "
        drop
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
 
        # Allow established and related connections
        ct state vmap { invalid : drop, established : accept, related : accept }
 
        # Allow forward of ICMPv6 and establishing wireguard connections across relay network
        iifname $vpn_routing_interface oifname $vpn_routing_interface ip6 daddr $vpn_routing_full_range udp dport $vpn_routing_dport_range accept
        iifname $vpn_routing_interface oifname $vpn_routing_interface ip6 daddr $vpn_routing_full_range icmpv6 type { echo-request, echo-reply, destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
 
        log prefix "Rejected Forward: "
        reject with icmpx type host-unreachable
    }
    chain output {
        type filter hook output priority filter;
    }
}

Start the firewall:

systemctl enable --now nftables

At this point, the relay is ready to accept connections from the edge routers and to forward messages between the two sites to the specified wireguard port range.

VPN Edge Router

The edge router can be any device on a site’s network. It uses wireguard for communication with the relay(s), radvd for advertising the site’s subnet and keepalived preventing edge router conflicts:

sudo apt install wireguard-tools radvd keepalived

The configuration in the following sections assumes that the edge router is part of the <ULA Prefix>::/48 network and is responsible for advertising <Site Prefix>::/64.

Internet-facing interface configuration

The default interface should simply use DHCP to get an address. This allows the router to be portable and agnostic to the network it is connected to (as long as the network uses DHCP).

# /etc/systemd/network/<pri>-<interface>.network
 
[Match]
Name=<interface name>
 
[Network]
DHCP=yes
IPv6AcceptRA=true

VRRP configuration

To avoid issues with multiple edge routers for the same site prefix being on the same link, only one of the edge router is assigned the router IP. This is done using VRRP/keepalived. The VRRP IP must be the same for the routers that advertise the same subnet.

Using VRRP is optional, of course, but it makes things easier in case fallback routers are configured.

# /etc/keepalived/keepalived.conf
 
vrrp_sync_group G1 {
    group {
        wg_vpn_edge_router
    }
}
 
vrrp_instance wg_vpn_edge_router {
    # Configure the interface to set the IP on
    interface <lan interface name>
 
    # Configure the router IP, should be the same across all routers
    # that advertise the same subnet
    virtual_router_id <router id>
 
    # Configure the priority, the higher the more important
    priority 100
 
    # The interval to send VRRP advertisements
    advert_int 1.0
 
    # The IP addresses to assign, it should be in the ULA range advertised
    # by the router (e.g. <Site ULA Prefix>::1/128)
    virtual_ipaddress {
        <VRRP IP>/128
    }
 
    # Disable preemption
    nopreemt
 
    # Set the GARP delay for publishing the MAC address
    garp_master_delay 1
}

Enable it on system startup:

sudo systemctl enable --now keepalived

radvd configuration

Radvd is used to advertise the site’s /64 ULA prefix as well as the supported routes.

# /etc/radvd.conf
 
# Replace the <interface name> with the interface which should be used
# to advertise the subnet, i.e. the link to which the servers are hooked up to
interface <lan interface name> {
    # Send advertisements
    AdvSendAdvert on;
 
    # Advertise every 10 seconds
    MinRtrAdvInterval 10;
 
    # Advertise at least once every 30 seconds
    MaxRtrAdvInterval 30;
 
    # This line is important, setting the lifetime to 0 prevents it from
    # advertising itself as default gateway
    AdvDefaultLifetime 0;
 
    # Consider a device reachable if available within the last 10 minutes
    AdvReachableTime 600000;
 
    # Advertise the VRRP IP as source address (same as the VRRP configuration)
    AdvRASrcAddress {
        <VRRP IP>/128;
    }
 
    # Choose a /64 prefix under the ULA /48 for the current site, here site A
    prefix <Site Prefix>::/64 {
        # The /64 is on link
        AdvOnLink on;
 
        # Enable SLAAC
        AdvAutonomous on;
 
        # Advertise the router address
        AdvRouterAddr on;
    };
 
    # Add the entire /48 to RIO to advertise routing to the subnet being supported
    route <ULA Prefix>::/48 {
        AdvRoutePreference high;
    };
};

Enable radvd on startup:

sudo systemctl enable --now radvd

Wireguard routing tunnel configuration

This configuration establishes the connection to the relay server.

# /etc/wireguard/wgvpnr01.conf
 
# Use the address configured as peer on the relay
[Interface]
Address = <Routing Tunnel ULA>/128
ListenPort = <Listen Port>
PrivateKey = <Private key>
MTU = <Main interface MTU - 80>
 
# Enable IP forwarding while the interface is up
PreUp = sysctl -w net.ipv6.conf.all.forwarding=1
PreDown = sysctl -w net.ipv6.conf.all.forwarding=0
 
# Add the relay as peer
# AllowedIPs should be the same as the one configured on the relay's wireguard interface.
#
# The Endpoint should be a DNS entry so that it will work no matter whether
# the internet connection is IPv4 or IPv6.
#
# The PersistentKeepalive is to keep the connection alive behind CG-NAT. A value lower than
# 30 should be OK for most ISP's
[Peer]
PublicKey = <Public Key>
AllowedIPs = <Routing Tunnel ULA>/128,<ULA prefix>::/48
Endpoint = <Relay Domain>:<Port>
PersistentKeepalive = 25

sudo systemctl enable --now wg-quick@wgvpnr01.service

Note: wg-quick loads the configuration on start and shutdown. If stopping the service fails, manual removal of the interface might be necessary with wg-quick down <interface name>

nftables configuration

# /etc/nftables.conf
flush ruleset
 
# The interface that shares the link with the servers
define lan_interface = "<LAN interface>"
 
# The interface through which the forwarding from/to the relay is allowed
# i.e. the wireguard interface configured in the previous step
define vpn_routing_interface = wgvpnr01
 
# The port the interface listens on
define vpn_routing_wireguard_port = <listen port>
 
# The ULA range that is used to address devices across sites
define vpn_routing_full_range = <ULA Prefix>::/48
 
# The ULA range that is used to address devices on-site
define vpn_routing_site_range = <Gateway Prefix>::/48
 
table inet filter {
    chain input {
        # Default to drop
        type filter hook input priority filter; policy drop;
 
        # Allow established and related connections
        ct state established,related accept
 
        # Allow loopback traffic
        iif lo accept
 
        # Reject traceroute for 30 hops (this allows clients to see the hop IP)
        udp dport { 33434-33474 } reject
 
        # Allow wireguard
        udp dport $vpn_routing_wireguard_port accept
 
        # Allow VRRP
        iif $lan_interface ip protocol vrrp accept
        iif $lan_interface ip6 nexthdr vrrp accept
 
        # Allow ICMP
        icmp type { echo-request, echo-reply, destination-unreachable, time-exceeded } accept
        icmpv6 type { echo-request, echo-reply, destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
 
        # Allow established and related connections
        ct state vmap { invalid : drop, established : accept, related : accept }
 
        # Allow forwarding to relay via wireguard interface for other subnets but not the one managed by this roter
        iifname $lan_interface oifname $vpn_routing_interface ip6 daddr $vpn_routing_site_range reject with icmpv6 type port-unreachable
        iifname $lan_interface oifname $vpn_routing_interface ip6 daddr $vpn_routing_full_range accept
        iifname $vpn_routing_interface oifname $lan_interface ip6 daddr $vpn_routing_site_range accept
 
        reject with icmpx type host-unreachable
    }
    chain output {
        type filter hook output priority filter;
    }
}

Start the firewall:

systemctl enable --now nftables

Once this step has been completed, communication should be possible between the router and the relay. Once another router has been configured, the communication should work site-to-site.

Servers

As long as at least one of the routers is on the same link as the server, the latter should auto-configure its IP address via SLAAC after a few seconds.

Note that any IPv6 privacy extension should be disabled to make the IP stable, meaning that SLAAC should always request the same IP address for the respective edge router. With privacy extensions, the IP address is partially randomized, in which case the servers IP address can not be configured as Wireguard endpoint on other servers (since the “public IP” keeps changing). Alternatively, the systemd configuration below can be used to configure a the IP address for any router advertisement.

In any case, after a few seconds the server should map an IP address in the <ULA Prefix>:A::/64 range as global dynamic noprefixroute (see ip -6 a) and the edge router and the server should be able to communicate with each other. As SLAAC configures a noprefixroute the route must be added manually.

# /etc/systemd/network.d/<pri>-<interface>.conf
[Match]
Name=<interface name>
 
[Network]
DHCP=no
 
# Accept router advertisements
IPv6AcceptRA=true
 
# See https://man7.org/linux/man-pages/man5/systemd.network.5.html#[IPV6ACCEPTRA]_SECTION_OPTIONS
#
# Token should be anything that is stable (see systemd documentation for details)
# static expects the "full" part after the prefix, i.e. should start with ::
#
# In addition, optionally, create allow/deny list for advertised prefixes
# via PrefixAllowList / PrefixDenyList
[IPv6AcceptRA]
UseRedirect=true
Token=static:<'Host' Part>

Wireguard mesh configuration

Once more than one server is set up, they can establish wireguard tunnel in-between them.

Here, the MTU should be lowered by more than configured automatically by wireguard as the chance of traffic flowing across the router -> relay tunnel is quite high. As such, the MTU should be the MTU of that outer tunnel minus 80.

The Server Tunnel ULA below is a unique ULA address for communication inside the tunnel. The address should be outside the VPN’s /48 range.

# /etc/wireguard/wgvpn01.conf
 
[Interface]
Address = <Server Tunnel ULA>/128
ListenPort = <Listen Port>
PrivateKey = <Private key>
MTU = <Routing Tunnel MTU - 80>
 
# Add all other servers as peers
[Peer]
PublicKey = <Public Key>
AllowedIPs = <Server Tunnel ULA>/128
Endpoint = <ULA address>:<Wireguard Port>

And register it as service:

sudo systemctl enable --now wg-quick@wgvpn01.service

Worklog

24.02.2025: Extended the concept with some thoughts on each component and a note on third party solutions. Set up a small PoC (to be documented)
27.02.2025: Added configuration notes
05.03.2025: Extended configurations
09.03.2025: Added VRRP
17.03.2025: Updated relay configurations, renamed gateways to edge routers
17.03.2025: Revised individual configurations & concept documentations

Matteias Collet

(Work in progress) E2EE setup across CG-NAT using Wireguard, radvd and keepalived

The problem

Conceptual overview

Inter-Site communication

Intra-Site communication

Configuration

Prerequisites

VPN Relay

Wireguard routing tunnel configuration

Firewall configuration

VPN Edge Router

Internet-facing interface configuration

VRRP configuration

radvd configuration

Wireguard routing tunnel configuration

nftables configuration

Servers

Wireguard mesh configuration

Worklog