Architecture Overview
Two-site private cloud architecture: goals, site roles, topology, and the hard rules that must not be violated.
Two independent Proxmox sites connected by a WireGuard routed VPN — never a stretched cluster, never stretched L2. Site A (10.10.0.0/16) owns management and storage; Site B (10.20.0.0/16) owns Ceph, Kubernetes, and distributed compute. OPNsense at each site handles WAN, VLANs, DHCP, and the WireGuard tunnel.
Primary Goal
Build a two-site private cloud using Proxmox VE on all server hardware, OPNsense at each site as the main WAN router/firewall, WireGuard for site-to-site routing, Netgear 10 Gb switches as core/data switching, and UniFi retained for existing Wi-Fi, APs, and user devices. The long-term target adds Ceph block storage and Kubernetes/OpenShift on Proxmox VMs.
High-Level Topology
Site A Site B
ISP/ONT ISP/ONT
| |
OPNsense (sa-fw-01 on sa-edge-01) OPNsense (sb-fw-01 on sb-edge-01)
| |
sa-sw-01 (Netgear XS716T, 10 Gb core) sb-sw-01 (Netgear XS748T, 10 Gb core)
|-- Proxmox / ZFS / PBS / DNS |-- Proxmox / Ceph / K8s
|-- UniFi (existing users/Wi-Fi) |-- UniFi (existing users/Wi-Fi)
Inter-site: OPNsense A <—WireGuard—> OPNsense B
Routes: 10.10.0.0/16 <-> 10.20.0.0/16 (no L2 stretch)Site Roles
Site A — Management & Storage
Site A runs management, DNS, monitoring, ZFS storage, and Proxmox Backup Server.
| Host | Hardware | Primary Role |
|---|---|---|
sa-edge-01 | Supermicro SYS-E200-8D | Proxmox + OPNsense VM (sa-fw-01) |
sa-cmp-01 | ThinkPad P51 | Proxmox worker / CI (Intel X550-T2) |
sa-cmp-02 | ThinkPad P52 | Proxmox worker / GPU / AI (Intel X550-T2) |
sa-stor-01 | Supermicro 5049A-T | ZFS mirror, PBS-A, DNS, monitoring, databases |
Cluster name: sa-pve. Switches: sa-sw-01 (10 Gb core), sa-sw-02, sa-sw-03 (access/IPMI/AP). UniFi Gateway Max is bootstrap/fallback only.
Site B — Ceph, Kubernetes & Compute
Site B runs Ceph, Kubernetes/OpenShift, and high-core worker nodes.
| Host | Hardware | Primary Role |
|---|---|---|
sb-edge-01 | Supermicro SYS-E200-8D | Proxmox + OPNsense VM (sb-fw-01) |
sb-cmp-01 | SYS-5019D-4C-FN8TP | Ceph MON/MGR, K8s control-plane |
sb-cmp-02 | SYS-5019D-4C-FN8TP | Ceph MON/MGR, K8s control-plane |
sb-cmp-03 | SYS-5018D-FN4T | Ceph OSD, K8s worker |
sb-cmp-04 | SYS-5018D-FN4T | Ceph OSD, K8s worker |
sb-cmp-05 | SYS-5018D-FN4T | Ceph OSD, K8s worker |
Cluster name: sb-pve. Switches: sb-sw-01 (10 Gb core), sb-sw-02 (UniFi USW 24 PoE). USG Pro is bootstrap/fallback only.
Hard Architecture Rules
The rules below are locked. Do not suggest alternatives that violate them.
One cluster per site — no cross-WAN Proxmox clusters
One Proxmox cluster per site: sa-pve at Site A, sb-pve at Site B. Never stretch a Proxmox cluster across WAN or WireGuard. Corosync requires LAN-like latency; a WAN/VPN cluster risks quorum instability.
No stretched Ceph
Site B Ceph stays local. Do not stretch Ceph across sites. Use Proxmox Backup Server replication for cross-site disaster recovery.
OPNsense owns infrastructure routing — Proxmox/OpenShift must not sit behind UniFi
OPNsense handles WAN, VLAN gateways, firewall policy, DHCP, and WireGuard at each site. Proxmox and OpenShift must connect to the OPNsense-side network, not behind UniFi.
OPNsense VM stays pinned to its local E200 — no HA migration
sa-fw-01 is pinned to sa-edge-01; sb-fw-01 is pinned to sb-edge-01. HA migration of the firewall VM must not be enabled.
Never expose IPMI to the internet
All Supermicro IPMI interfaces stay on VLAN 10 (Network Mgmt / IPMI), isolated from WAN. Never route IPMI to the internet.
No stretched L2 between sites
Inter-site connectivity is routed only (WireGuard 10.255.0.0/24). Never bridge L2 across sites.
UniFi is retained for continuity — double-NAT is intentional
Existing UniFi routers remain in place so current Wi-Fi and user devices continue to work. UniFi sits behind OPNsense (double-NAT). This is a deliberate trade-off, not a misconfiguration.
E200 Workload Guidance
The SYS-E200-8D nodes (sa-edge-01, sb-edge-01) join their local Proxmox cluster but host only lightweight workloads to avoid starving OPNsense.
Suitable E200 workloads
OPNsense VM, DNS helper, UniFi controller, WireGuard helper, small reverse proxy, monitoring agent.
Avoid on E200
Heavy databases, Ceph OSDs, storage-heavy VMs, heavy Kubernetes workers, or any workload that competes with OPNsense for CPU or memory.
Build Sequence
Bring up one site at a time, starting flat (no VLANs), then introduce VLANs after each node has its final management IP and the OPNsense VM is running. Do not cluster Proxmox until every node in the site has its final 10.x.20.x IP and /etc/hosts is correct. See Build Phases for the full sequence.
Related Pages
- Two-Site Model — detailed model description
- Site Roles — per-site hardware and workload breakdown
- IP Addressing — supernets, bootstrap networks, VLAN plan
- VLAN Reference — full VLAN table with subnets and gateways
Design Principles & Hard Rules
Non-negotiable architecture rules and guiding principles every operator must understand before making any change to this homelab.
The Two-Site Model
Two independent Proxmox clusters connected by WireGuard: the rationale for splitting across two physical sites, the per-site topology pattern, and the hard rules governing inter-site connectivity.