Control plane

Routing & security

The overlay is the interesting part: dynamic routing that runs inside encrypted tunnels, so adding a site is a matter of one adjacency, not a spreadsheet of static routes. Underneath it, a firewall built for line rate.

WireGuard

The overlay: hub-and-spoke, meshed by routing

Every site holds a single WireGuard tunnel to Magikarp on wg1 (UDP 51820, MTU 1440). The hub carries the whole 172.16.20.0/24 transit range; each spoke gets a /32 loopback that doubles as its OSPF router-ID.

Residential spokes have no stable public IP, so they dial the hub by dynamic-DNS hostname and hold the tunnel open with PersistentKeepalive. The Linode VPS anchors the mesh with a real static endpoint.

The multicast trick: WireGuard is point-to-point and won’t carry multicast on its own — but OSPF hellos are multicast. Adding 224.0.0.0/24 to each peer’s AllowedIPs (exactly once) lets the hellos ride the tunnel, and the adjacency forms. Duplicate it across peers and routing quietly breaks.

# Magikarp — /etc/wireguard/wg1.conf (keys redacted)
[Interface]
ListenPort = 51820
MTU        = 1440
Address    = 10.10.10.1/32, 172.16.20.1/24
Table      = off

[Peer] # Jeffrotik (Dad)
Endpoint   = hha0a8kxk9w.sn.mynetname.net:51820
AllowedIPs = 10.10.20.1/32,192.168.5.0/24,192.168.30.0/24,
             172.16.20.3/32,10.10.90.0/24,224.0.0.0/24
PersistentKeepalive = 25

[Peer] # Lindarotik (Mom)
Endpoint   = hhm0aedy210.sn.mynetname.net:51820
AllowedIPs = 10.10.40.1/32,172.16.20.2/32,192.168.25.0/24,
             224.0.0.0/24
PersistentKeepalive = 25

[Peer] # Melodyrotik (Sister)
Endpoint   = hmn0b3fhz4x.sn.mynetname.net:51820
AllowedIPs = 10.10.30.1/32,172.16.20.4/32,192.168.6.0/24,
             224.0.0.0/24
PersistentKeepalive = 25

OSPF

BIRD2 on the hub, RouterOS on the spokes

The hub runs BIRD2 speaking OSPFv2 in area 0.0.0.0. The wg1 interface is declared point-to-multipoint with each spoke listed as an eligible neighbour; the LAN and HPBX interfaces are injected as stub networks so remote sites can reach them.

The single most important line is the export filter. Without scoping what OSPF is allowed to push into the kernel table, an OSPF-learned default route can overwrite the real WAN default — and the whole site loses its way out. Here it only exports connected and static routes, and the kernel default stays put.

# Magikarp — /etc/bird.conf (excerpt)
router id 10.10.10.1;

protocol ospf v2 ospf_wg {
  ipv4 {
    import all;
    export where source = RTS_DEVICE
                 || source = RTS_STATIC;
  };
  area 0.0.0.0 {
    interface "wg1" {
      type pointomultipoint;
      cost 10; hello 10; dead 40;
      neighbors {
        172.16.20.2 eligible;  # Lindarotik
        172.16.20.3 eligible;  # Jeffrotik
        172.16.20.4 eligible;  # Melodyrotik 
        172.16.20.5 eligible;  # Linode
      };
    };
    interface "enp16s0f1np1"    { stub yes; };
    interface "enp16s0f1np1.77" { stub yes; };
  };
}

RouterOS gotcha: MikroTik does not do ordered DNS failover — all configured resolvers race at once. The spokes use static FWD entries to steer the internal .fishtank domain deterministically while public DNS handles everything else.

Firewall

nftables, built for 10G

The firewall is inet-family nftables with a flowtable: once a connection is established, its packets are offloaded at ingress and skip the forward chain entirely, so forwarding runs at hardware speed. That’s also its one catch — offloaded flows are invisible to anything hooking the forward path, so specific traffic (VLAN 77 SIP) needs deliberate flowtable exemptions to avoid one-way audio.

Guest VLAN 20 is boxed in: WAN only, explicitly rejected to LAN, HPBX, WG and management. The HPBX VLAN accepts SIP only from Socket’s and Telnyx’s published source ranges. WAN SSH is rate-limited into a dynamic set, and every RFC1918 / bogon source is dropped on the way in.

# Magikarp — /etc/nftables.conf (excerpts)
flowtable ft {
  hook ingress priority 0;
  devices = { enp16s0f0np0, enp16s0f1np1,
              enp16s0f1np1.77, enp16s0f1np1.20, wg1 };
}

chain forward {
  type filter hook forward priority 0; policy drop;
  ct state established,related flow add @ft accept
  # Guest VLAN 20 → WAN only, isolated
  iifname $VLAN20_IF oifname $WAN_IF accept
  iifname $VLAN20_IF oifname { $LAN_IF,$VLAN77_IF,$WG_IF } reject
  # Carrier SIP → HPBX only
  iifname $WAN_IF oifname $VLAN77_IF ip saddr $SOCKET_SIP accept
}

Defence in depth

Security highlights

Flowtable offload

Established flows are offloaded at ingress (enp16s0f0np0, enp16s0f1np1, both VLAN sub-ifs, wg1), skipping the forward chain for line-rate forwarding.

RFC1918 / bogon anti-spoof

A full bogon set (this-net, CGNAT, TEST-NETs, multicast, reserved) dropped on WAN ingress, egress and in the forward path.

SIP source pinning

Only Socket and Telnyx published SIP ranges may reach the HPBX VLAN from the WAN. SIP ALG stays off everywhere — NAT is handled at the carrier SBC.

SSH rate-limit + whitelist

WAN SSH is rate-limited (4/min, burst 10) with a dynamic timeout set; management DNATs (Pleco :22, Winbox :8291) are whitelisted to known carrier ranges.

Guest isolation

VLAN 20 can reach the WAN only — reject to LAN, HPBX, WG and MGMT.

XDP drop-plane

hellgate drops bogons/abusers before skb allocation, with central guard rails that refuse to blackhole SSH or the WireGuard port.

RouterOS hardening

Spokes auto-blacklist unauthorised management hits for 5 h, run a detect-ddos chain, and drop DNS scans + blacklisted sources at the raw level.

Hard-won

Principles this network taught

WireGuard + OSPF is the right call at this scale. IPsec would need GRE/IPIP to get multicast dynamic routing back — more overhead, no throughput win, because residential WAN is the real bottleneck, not the crypto.
Flowtable offload creates monitoring blind spots. Offloaded flows skip the forward chain, so NFQUEUE-based tooling needs flowtable exemptions (e.g. VLAN 77 SIP to avoid one-way audio).
SIP ALG stays off. The hosted PBX handles NAT at its SBC edge; the Linux conntrack SIP helper would only interfere. MikroTik SIP service-port is disabled everywhere too.
Hardware crypto offload is moot at residential speeds. ChaCha20-Poly1305 on Zen 3 AVX2 is not the bottleneck, and GPU WireGuard offload is architecturally impossible anyway.
XDP is the right tool for early drop. Bogon/DDoS dropping before skb allocation skips the whole netfilter path.
RouterOS does not do ordered DNS failover — all servers race. Use static FWD entries to steer internal domains deterministically.