Network Topology

Physical and logical topology of both sites: the ISP-to-host data path, Netgear 10 Gb core and access switch hierarchy, WireGuard inter-site link, and UniFi WAN-transit branch.

Both sites share an identical physical spine: ISP feeds an OPNsense firewall VM on a Supermicro E200 edge host, which uplinks to a Netgear 10 Gb core switch that distributes to Proxmox servers, storage, and a UniFi WAN-transit branch for existing users. A WireGuard tunnel between the two OPNsense VMs joins the sites, routing 10.10.0.0/16 ↔ 10.20.0.0/16 with no L2 stretch.

Canonical Data Path

Both sites follow this spine:

ISP → OPNsense → 10Gb Core Switch → Proxmox / K8s / Storage / PBS
                                  → UniFi Router WAN → Existing Wi-Fi/users

OPNsense (sa-fw-01 at Site A, sb-fw-01 at Site B) runs as a VM pinned to the local sa-edge-01 / sb-edge-01 host. It owns WAN routing, VLAN gateways, firewall policy, DHCP, and the WireGuard tunnel.

OPNsense VM stays pinned to its local E200 — no HA migration

sa-fw-01 is pinned to sa-edge-01; sb-fw-01 is pinned to sb-edge-01. HA migration of either firewall VM must not be enabled.

Site A Topology

Site A supernet: 10.10.0.0/16. Proxmox cluster: sa-pve.

ISP/ONT
  |
sa-fw-01 (OPNsense VM on sa-edge-01 — SYS-E200-8D)
  |
sa-sw-01 (Netgear XS716T — 10 Gb core)
  |-- sa-stor-01   ZFS / PBS-A / DNS / monitoring / databases
  |-- sa-cmp-01    Proxmox worker / CI
  |-- sa-cmp-02    Proxmox worker / GPU / AI
  |
  |-- sa-sw-02 (access / IPMI)
  |     |-- sa-edge-01 mgmt + IPMI
  |     |-- sa-stor-01 mgmt + IPMI
  |
  |-- sa-sw-03 (access / AP / PoE++)
  |     |-- sa-cmp-01 mgmt
  |     |-- sa-cmp-02 mgmt
  |     |-- sa-ap-01 (UniFi U7 Pro XGS — port 3, PoE++)
  |
  |-- UniFi Gateway Max WAN (VLAN 253, 10.10.253.2)
        |
        UniFi → existing Wi-Fi / users

Site B Topology

Site B supernet: 10.20.0.0/16. Proxmox cluster: sb-pve.

ISP/ONT
  |
sb-fw-01 (OPNsense VM on sb-edge-01 — SYS-E200-8D)
  |
sb-sw-01 (Netgear XS748T — 10 Gb core)
  |-- sb-cmp-01    Ceph MON/MGR, K8s control-plane
  |-- sb-cmp-02    Ceph MON/MGR, K8s control-plane
  |-- sb-cmp-03    Ceph OSD, K8s worker
  |-- sb-cmp-04    Ceph OSD, K8s worker
  |-- sb-cmp-05    Ceph OSD, K8s worker
  |
  |-- sb-sw-02 (UniFi USW 24 PoE — access / IPMI / AP)
  |     |-- All nodes: IPMI (VLAN 10)
  |
  |-- USG Pro WAN (VLAN 253, 10.20.253.2)
        |
        UniFi → existing Wi-Fi / users

Switch Hierarchy

Each site runs a two-tier switch stack. The Netgear core handles all infrastructure VLANs; access switches carry only management, client-side, and IPMI VLANs.

Switch	Site	Model	Role
`sa-sw-01`	Site A	Netgear XS716T	10 Gb L2 core; no routing, no DHCP
`sa-sw-02`	Site A	—	Access / IPMI
`sa-sw-03`	Site A	—	Access / AP / PoE++
`sb-sw-01`	Site B	Netgear XS748T	10 Gb L2 core; no routing, no DHCP
`sb-sw-02`	Site B	UniFi USW 24 PoE	Access / IPMI / AP

Access switches (sa-sw-02, sa-sw-03, sb-sw-02) carry only VLANs 10, 20, 100, 110, and 120. VLANs 25, 30, 40, 50, 60, 65, 70, 80, and 90 are restricted to the core switch.

At Site A, sa-sw-02 and sa-sw-03 also carry Proxmox Management (VLAN 20) and IPMI (VLAN 10) to free sa-sw-01 ports for dedicated Corosync links (VLAN 25). See Site A Port Map for the full per-port assignment.

WireGuard Inter-Site Link

The two OPNsense VMs connect across the internet via WireGuard. A dedicated /24 transit prefix is used for the tunnel endpoints.

Endpoint	WireGuard address	Role
`sa-fw-01`	`10.255.0.1`	WireGuard peer A
`sb-fw-01`	`10.255.0.2`	WireGuard peer B

Transit prefix: 10.255.0.0/24. Routes exchanged: 10.10.0.0/16 ↔ 10.20.0.0/16.

No stretched L2 between sites

Inter-site connectivity is routed only, over WireGuard. Never bridge L2 between sites.

UniFi WAN-Transit Branch

UniFi routers sit downstream of OPNsense on VLAN 253 (UniFi WAN transit). OPNsense is the gateway; the UniFi WAN port receives a static address in the transit subnet.

Site	OPNsense GW	UniFi WAN	Transit subnet
Site A	`10.10.253.1`	`10.10.253.2`	`10.10.253.0/24`
Site B	`10.20.253.1`	`10.20.253.2`	`10.20.253.0/24`

Double-NAT is intentional

Existing UniFi routers remain in place so current Wi-Fi and user devices continue to work. The resulting double-NAT (OPNsense → UniFi → clients) is a deliberate architecture trade-off, not a misconfiguration.

Proxmox and OpenShift must not sit behind UniFi

All infrastructure VMs — Proxmox, OpenShift, Proxmox Backup Server (PBS), DNS — must connect to the OPNsense-side network, never behind the UniFi router.

Interactive topology map

A richer interactive topology with per-port wiring and faceplate views is planned. Source HTML widgets already exist in vault/interactive/ (port-wiring-guide.html, switch-faceplate-wiring.html). A rendered interactive version will appear in the Interactive Tools section once that section is built.

WireGuard — tunnel configuration and inter-site routing
Site A Port Map — per-port VLAN and cabling details for Site A
Interactive Topology Map — rendered interactive version of the topology