How It All Fits Together
End-to-end narrative overview of the AORXI homelab: two Proxmox sites, OPNsense edge routing, 10 Gb core switching, WireGuard VPN, VLAN segmentation, storage strategy, and how the build phases connect all the pieces.
Two independent Proxmox sites, each fronted by an OPNsense firewall on a Supermicro E200 and a Netgear 10 Gb core switch, connected by a WireGuard routed VPN — this page traces the path from ISP to Kubernetes and explains how the sites, roles, VLANs, storage, and build phases relate to each other. For deep references, follow the links in each section.
The Two-Site Model
The project builds a two-site private cloud:
- Site A (
10.10.0.0/16) — management, ZFS storage, Proxmox Backup Server (PBS), DNS, and monitoring. - Site B (
10.20.0.0/16) — Ceph block storage, Kubernetes/OpenShift, and distributed compute.
Each site follows the same traffic-flow pattern:
ISP/ONT
│
OPNsense VM (sa-fw-01 / sb-fw-01) on Supermicro E200
│
Netgear 10 Gb core switch (sa-sw-01 / sb-sw-01)
├── Proxmox compute and storage hosts
└── UniFi router WAN port → existing Wi-Fi / users (double-NAT, intentional)Inter-site: a WireGuard tunnel between OPNsense-A (10.255.0.1) and OPNsense-B (10.255.0.2) routes 10.10.0.0/16 ↔ 10.20.0.0/16. No L2 bridge crosses the tunnel — routing only.
No stretched clusters or L2
One Proxmox cluster per site: sa-pve at Site A, sb-pve at Site B. Never stretch a Proxmox cluster across WireGuard/WAN. Inter-site connectivity is routed only — L2 is never bridged between sites.
UniFi is retained — double-NAT is intentional
Existing UniFi routers remain in place so current Wi-Fi and user devices continue to work. UniFi sits behind OPNsense (double-NAT). This is a deliberate trade-off, not a misconfiguration.
See Architecture Overview and Two-Site Model for the full topology.
Site Roles
Site A — Management and Storage
Site A hosts the management plane and primary storage.
| Host | Hardware | Primary Role |
|---|---|---|
sa-edge-01 | Supermicro SYS-E200-8D | Proxmox + OPNsense VM (sa-fw-01) |
sa-cmp-01 | ThinkPad P51 | Proxmox worker / CI |
sa-cmp-02 | ThinkPad P52 | Proxmox worker / GPU / AI |
sa-stor-01 | Supermicro 5049A-T | ZFS mirror, PBS-A, DNS, monitoring, databases |
Switches: sa-sw-01 (Netgear XS716T, 10 Gb core), sa-sw-02 and sa-sw-03 (access / IPMI / AP). The UniFi Gateway Max is bootstrap/fallback only.
Site B — Ceph, Kubernetes, and Compute
Site B runs the Ceph storage cluster and Kubernetes/OpenShift workloads.
| Host | Hardware | Primary Role |
|---|---|---|
sb-edge-01 | Supermicro SYS-E200-8D | Proxmox + OPNsense VM (sb-fw-01) |
sb-cmp-01 | SYS-5019D-4C-FN8TP | Ceph MON/MGR, K8s control-plane |
sb-cmp-02 | SYS-5019D-4C-FN8TP | Ceph MON/MGR, K8s control-plane |
sb-cmp-03 | SYS-5018D-FN4T | Ceph OSD, K8s worker |
sb-cmp-04 | SYS-5018D-FN4T | Ceph OSD, K8s worker |
sb-cmp-05 | SYS-5018D-FN4T | Ceph OSD, K8s worker |
Switch: sb-sw-01 (Netgear XS748T, 10 Gb core). sb-sw-02 (UniFi USW 24 PoE) handles access/IPMI/AP. The USG Pro is bootstrap/fallback only.
No stretched Ceph
Site B Ceph stays local. Use PBS replication for cross-site disaster recovery — never stretch Ceph across sites.
See Site Roles and Per-Site Inventory for per-host hardware detail.
OPNsense Edge Layer
OPNsense runs as a VM on each site's E200 node, pinned to that host (no HA migration). It owns WAN routing, NAT, VLAN gateways, DHCP, firewall policy, and the WireGuard site-to-site tunnel.
OPNsense owns infrastructure routing
Proxmox and OpenShift must connect to the OPNsense-side network. Never place Proxmox or OpenShift behind a UniFi router.
OPNsense VM stays pinned to its local E200
sa-fw-01 is pinned to sa-edge-01; sb-fw-01 is pinned to sb-edge-01. HA migration of the firewall VM must not be enabled.
The E200 nodes join their local Proxmox cluster but run only lightweight workloads — DNS helper, monitoring agent, small utility VMs — so OPNsense always has CPU and memory headroom. See Migration Phases for how OPNsense is deployed into place.
VLAN Segmentation
Fifteen VLANs carry separate traffic classes. VLAN IDs are identical at both sites; only the third octet of the subnet differs (10.10.x.0 at Site A, 10.20.x.0 at Site B). The key infrastructure VLANs:
| VLAN | Canonical Name | Site A | Site B |
|---|---|---|---|
| 10 | Network Mgmt / IPMI | 10.10.10.0/24 | 10.20.10.0/24 |
| 20 | Proxmox Management | 10.10.20.0/24 | 10.20.20.0/24 |
| 25 | Corosync heartbeat | 10.10.25.0/24 | 10.20.25.0/24 |
| 30 | VM Services | 10.10.30.0/24 | 10.20.30.0/24 |
| 60 | Storage / Ceph public | 10.10.60.0/24 | 10.20.60.0/24 |
| 65 | Ceph cluster | reserved | 10.20.65.0/24 |
| 90 | Backup / Replication | 10.10.90.0/24 | 10.20.90.0/24 |
VLAN 25 (Corosync heartbeat) and VLAN 65 (Ceph cluster) carry no gateway — they are isolated, per-site L2 fabrics. Each VLAN's /24 follows a fixed address-block convention: .1 is the OPNsense gateway, .2–.9 network infrastructure, .10–.39 physical host interfaces, .40–.49 pinned service VMs, .200–.254 VIPs and MetalLB pools.
The full 15-VLAN table including K8s nodes (VLAN 40), K8s LB / VIPs (VLAN 50), DMZ, monitoring, IoT, guest Wi-Fi, and UniFi WAN transit is in VLAN Reference. Address-block conventions are in IP Addressing.
Storage and Backups
Storage strategy is split cleanly by site role.
Site A — sa-stor-01 runs ZFS mirror vdevs (Samsung SM863 1.92 TB enterprise SSDs). Proxmox Backup Server A (PBS-A) runs on this host with a management interface on VLAN 30 (10.10.30.20) and a backup-data interface on VLAN 90 (10.10.90.40). DNS, monitoring, and core databases also run here.
Site B — Ceph spans all five compute nodes (sb-cmp-01 through sb-cmp-05), targeting 20–30 OSDs with replication size 3. Ceph public network uses VLAN 60; OSD replication uses VLAN 65 on dedicated per-node onboard 10 Gb links. PBS-B mirrors PBS-A's layout (VLAN 30 mgmt 10.20.30.20, VLAN 90 backup-data 10.20.90.40).
Cross-site: PBS-A and PBS-B replicate to each other over the WireGuard tunnel on VLAN 90. Neither site depends on the other for local backup operation.
See Proxmox Clusters, Site A ZFS, Site B Ceph, and PBS Backups.
Build Phases
The system is built one site at a time, starting flat (no VLANs) and introducing VLANs only after OPNsense is running and every node has its final management IP.
| Phase | Description |
|---|---|
| 1 | Hardware preparation — drives, NICs, RAM, BIOS/IPMI |
| 2 | Bootstrap Proxmox — flat network, no clustering |
| 3 | OPNsense VM — WAN, VLAN trunk, gateways, DHCP |
| 4 | Switch VLANs — Netgear trunks, server and UniFi uplinks |
| 5 | Move Proxmox to final VLAN 20 IPs (10.x.20.x) |
| 6 | Create Proxmox clusters (sa-pve on sa-stor-01; sb-pve on sb-cmp-01) |
| 7 | WireGuard site-to-site VPN (10.255.0.0/24) |
| 8 | WAN migration — ISP WAN moves from UniFi to OPNsense |
| 9 | ZFS, Ceph, PBS, Kubernetes/OpenShift |
Site A bootstraps on 192.168.1.0/24 (the existing UniFi LAN). Site B uses 192.168.16.0/24. Do not create a Proxmox cluster until every node in the site has its final 10.x.20.x IP and /etc/hosts is consistent.
See Build Phases for phase runbooks, success criteria, and current state.
Related Pages
- Architecture Overview — hard rules, topology diagram, E200 workload guidance
- Two-Site Model — detailed model description
- Site Roles — per-site hardware and workload breakdown
- VLAN Reference — full VLAN table with subnets and gateways
- IP Addressing — supernets, host-octet convention, WireGuard plan
- Migration Phases — OPNsense deployment sequence
- Proxmox Clusters — cluster strategy and rules
- Build Phases — full phase runbooks and current progress