Compute & Storage
Proxmox cluster layout, Site A ZFS storage, Site B Ceph cluster, and PBS cross-site backup replication.
Compute and storage split by site role: Site A anchors management workloads with ZFS on sa-stor-01; Site B runs Ceph across five nodes for distributed block storage and Kubernetes. Two independent Proxmox clusters: sa-pve and sb-pve: anchor each site. Proxmox Backup Server (PBS) at each site replicates cross-site for disaster recovery.
Proxmox Clusters
One Cluster per Site: No Cross-WAN Proxmox Clusters
One Proxmox cluster per site: sa-pve at Site A, sb-pve at Site B. Never stretch a Proxmox cluster across WAN or WireGuard. Corosync requires LAN-like latency; a WAN/VPN cluster risks quorum instability and split-brain.
| Site | Cluster | Nodes | Create on |
|---|---|---|---|
| Site A | sa-pve | sa-edge-01, sa-cmp-01, sa-cmp-02, sa-stor-01 | sa-stor-01 |
| Site B | sb-pve | sb-edge-01, sb-cmp-01, sb-cmp-02, sb-cmp-03, sb-cmp-04, sb-cmp-05 | sb-cmp-01 |
Cluster Only After Final IPs and /etc/hosts
Do not create or join a cluster until every node in the site has its final 10.x.20.x Proxmox Management IP and /etc/hosts is correct. Never join nodes across sites. Never run Corosync over WireGuard or WAN.
See Proxmox Clusters for the full creation and join procedure.
Site A: ZFS Storage
Site A storage runs on sa-stor-01 (Supermicro 5049A-T), which hosts ZFS mirror vdevs, PBS-A, DNS VMs, monitoring, and core databases. ZFS is chosen over Ceph here because Site A is management-focused: simpler administration, strong data integrity, and native snapshots for PBS storage pools.
| Detail | Value |
|---|---|
| Host | sa-stor-01 |
| Drives | 8-12 × Samsung SM863 1.92 TB SSDs |
| Layout | ZFS mirror vdevs |
| Services hosted | PBS-A, sa-dns-01/sa-dns-02, monitoring, databases |
See Site A ZFS for pool layout, vdev configuration, and PBS dataset setup.
Site B: Ceph Storage
No Stretched Ceph
Site B Ceph stays local. Do not stretch Ceph across sites. Use PBS replication for cross-site disaster recovery instead.
Site B runs Ceph across all five compute nodes. Ceph public/client traffic uses VLAN 60 and cluster replication uses VLAN 65; both are L2-only with no SVI or default gateway. Consumers attach directly to VLAN 60. PBS cross-site synchronization uses routed VLAN 90 instead.
| Host | Hardware | Ceph Role |
|---|---|---|
sb-cmp-01 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-02 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-03 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-04 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-05 | SYS-5018D-FN4T | OSD, K8s worker |
Plan: 4-6 enterprise 1.92 TB SSDs per node; replication size 3. Ceph is provisioned via the Proxmox ceph-tentacle no-subscription repo (Ceph release: tentacle, decided 2026-06-28).
See Site B Ceph for OSD layout, CRUSH map, and Kubernetes RBD integration.
Backup & Replication
PBS runs at each site as a dedicated service VM. Each instance backs up its local Proxmox cluster, and the two instances replicate to each other for cross-site disaster recovery.
| Instance | Proxmox mgmt IP | Backup-data IP |
|---|---|---|
| PBS-A (Site A) | 10.10.30.20 | 10.10.90.40 |
| PBS-B (Site B) | 10.20.30.20 | 10.20.90.40 |
Cross-Site Replication Is Required
Do not rely on one site for all backups. PBS cross-site replication is a design requirement: Site A and Site B each hold backup copies of the other.
See PBS Backups for backup job configuration, datastore layout, and cross-site replication setup.
Related Pages
- Proxmox Clusters: cluster creation, join order, Corosync requirements
- Site A ZFS: pool layout and PBS-A storage configuration
- Site B Ceph: OSD plan, CRUSH map, K8s integration
- PBS Backups: backup jobs and cross-site replication
OPNsense / UniFi Handoff
Site A's direct OPNsense-to-UniFi Layer-2 trunk and Site B's unchanged downstream-USG VLAN 253 handoff.
Proxmox Clusters
Proxmox clustering plan for both sites: cluster names, node membership, the IP and /etc/hosts prerequisites, dedicated Corosync on VLAN 25, and the phased build order.