Compute & Storage
Proxmox cluster layout, Site A ZFS storage, Site B Ceph cluster, and PBS cross-site backup replication.
Compute and storage split by site role: Site A anchors management workloads with ZFS on sa-stor-01; Site B runs Ceph across five nodes for distributed block storage and Kubernetes. Two independent Proxmox clusters — sa-pve and sb-pve — anchor each site. Proxmox Backup Server (PBS) at each site replicates cross-site for disaster recovery.
Proxmox Clusters
One cluster per site — no cross-WAN Proxmox clusters
One Proxmox cluster per site: sa-pve at Site A, sb-pve at Site B. Never stretch a Proxmox cluster across WAN or WireGuard. Corosync requires LAN-like latency; a WAN/VPN cluster risks quorum instability and split-brain.
| Site | Cluster | Nodes | Create on |
|---|---|---|---|
| Site A | sa-pve | sa-edge-01, sa-cmp-01, sa-cmp-02, sa-stor-01 | sa-stor-01 |
| Site B | sb-pve | sb-edge-01, sb-cmp-01, sb-cmp-02, sb-cmp-03, sb-cmp-04, sb-cmp-05 | sb-cmp-01 |
Cluster only after final IPs and /etc/hosts
Do not create or join a cluster until every node in the site has its final 10.x.20.x Proxmox Management IP and /etc/hosts is correct. Never join nodes across sites. Never run Corosync over WireGuard or WAN.
See Proxmox Clusters for the full creation and join procedure.
Site A — ZFS Storage
Site A storage runs on sa-stor-01 (Supermicro 5049A-T), which hosts ZFS mirror vdevs, PBS-A, DNS VMs, monitoring, and core databases. ZFS is chosen over Ceph here because Site A is management-focused: simpler administration, strong data integrity, and native snapshots for PBS storage pools.
| Detail | Value |
|---|---|
| Host | sa-stor-01 |
| Drives | 8–12 × Samsung SM863 1.92 TB SSDs |
| Layout | ZFS mirror vdevs |
| Services hosted | PBS-A, sa-dns-01/sa-dns-02, monitoring, databases |
See Site A ZFS for pool layout, vdev configuration, and PBS dataset setup.
Site B — Ceph Storage
No stretched Ceph
Site B Ceph stays local. Do not stretch Ceph across sites. Use PBS replication for cross-site disaster recovery instead.
Site B runs a Ceph cluster across all five compute nodes, providing distributed block storage for Kubernetes workloads and VM disks. Ceph public traffic runs on VLAN 60 (Storage / Ceph public, 10.20.60.0/24); cluster replication traffic runs on VLAN 65 (Ceph cluster, 10.20.65.0/24, no gateway).
| Host | Hardware | Ceph Role |
|---|---|---|
sb-cmp-01 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-02 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-03 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-04 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-05 | SYS-5018D-FN4T | OSD, K8s worker |
Plan: 4–6 enterprise 1.92 TB SSDs per node; replication size 3. Ceph is provisioned via the Proxmox ceph-tentacle no-subscription repo (Ceph release: tentacle, decided 2026-06-28).
See Site B Ceph for OSD layout, CRUSH map, and Kubernetes RBD integration.
Backup & Replication
PBS runs at each site as a dedicated service VM. Each instance backs up its local Proxmox cluster, and the two instances replicate to each other for cross-site disaster recovery.
| Instance | Proxmox mgmt IP | Backup-data IP |
|---|---|---|
| PBS-A (Site A) | 10.10.30.20 | 10.10.90.40 |
| PBS-B (Site B) | 10.20.30.20 | 10.20.90.40 |
Cross-site replication is required
Do not rely on one site for all backups. PBS cross-site replication is a design requirement — Site A and Site B each hold backup copies of the other.
See PBS Backups for backup job configuration, datastore layout, and cross-site replication setup.
Related Pages
- Proxmox Clusters — cluster creation, join order, Corosync requirements
- Site A ZFS — pool layout and PBS-A storage configuration
- Site B Ceph — OSD plan, CRUSH map, K8s integration
- PBS Backups — backup jobs and cross-site replication
OPNsense / UniFi / Proxmox Handoff
How OPNsense fronts the demoted UniFi routers at each site: VLAN 253 transit wiring, the double-NAT model for user devices, Proxmox network zone placement, firewall zone rules, and optional cross-zone routing.
Proxmox Clusters
Proxmox clustering plan for both sites: cluster names, node membership, the IP and /etc/hosts prerequisites, dedicated Corosync on VLAN 25, and the phased build order.