Site A ZFS Storage
ZFS mirror storage plan for sa-stor-01 (Supermicro 5049A-T): drive layout, vdev structure, network configuration, and the service VMs the pool serves.
Site A ZFS storage runs on sa-stor-01, a Supermicro 5049A-T (X11SPA-T board), using Samsung SM863 1.92 TB enterprise SSDs in mirror vdevs to serve Proxmox Backup Server (PBS-A), DNS, monitoring, databases, and core infrastructure VMs.
Host Overview
sa-stor-01 is the dedicated storage and infrastructure host at Site A and the node on which the sa-pve cluster is created. It holds the only ZFS pool in the site.
| Attribute | Value |
|---|---|
| Hostname | sa-stor-01 |
| Hardware | Supermicro 5049A-T (X11SPA-T board) |
| Proxmox Mgmt — VLAN 20 | 10.10.20.20 |
| IPMI — VLAN 10 | 10.10.10.20 |
| Corosync heartbeat — VLAN 25 | 10.10.25.20 |
| Storage / Ceph public — VLAN 60 | 10.10.60.20 |
| Backup / Replication — VLAN 90 | 10.10.90.20 |
ZFS Drive and Vdev Plan
The ZFS pool uses Samsung SM863 1.92 TB enterprise SSDs arranged in mirror vdevs. The final drive count is a range, not yet confirmed at build time.
Drive count tentative — 8 to 12 drives
The plan calls for 8–12 × Samsung SM863 1.92 TB SSDs. The exact number determines how many mirror vdevs are created. Do not use 1.92 TB enterprise SSDs as boot drives — sa-stor-01 already has a dedicated boot drive (512 GB M.2).
| Drive Count | Vdev Layout |
|---|---|
| 8 | 4 × mirror vdev |
| 12 | 6 × mirror vdev |
Any even number in the 8–12 range is valid. Each mirror vdev tolerates single-drive failure independently. Usable capacity equals half of total raw capacity.
Drive selection rationale
Samsung SM863 drives are enterprise SATA SSDs with high write endurance, well-suited to the mixed read/write workload of PBS-A and VM image storage.
NIC Configuration
The X11SPA-T board provides one onboard 1 GbE (Intel i210, igb) and one onboard 10GBASE-T (Aquantia AQC107, atlantic). An existing XL710/X710-class quad-port 10GBASE-T add-in card handles all production 10 G traffic.
| Interface | Driver | VLAN(s) | Role |
|---|---|---|---|
| Onboard 1 GbE — red RJ45 | igb (Intel i210) | 20 | Proxmox management (vmbr0) |
| Onboard 10GBASE-T | atlantic (AQC107) | — | Unused / non-critical only |
| XL710 quad-port 10GBASE-T — port 1 | — | 25 30 | Corosync heartbeat + VM Services (tagged) |
| XL710 quad-port 10GBASE-T — ports 2–4 | — | 60 90 | Storage and Backup (see Site A Port Map) |
| Dedicated IPMI LAN | — | 10 | BMC / IPMI only |
Do not use the AQC107 for critical traffic
The atlantic driver is historically unstable under Linux and Proxmox. The onboard 10GBASE-T (AQC107) is reserved for non-critical use or left unused. Proxmox management and Corosync must not run on this interface.
Corosync runs tagged on XL710 port 1 alongside VLAN 30 (VM Services) rather than a dedicated link — the AQC107 instability rules out the onboard 10 G as a Corosync alternative, and sa-stor-01 has only a single onboard 1 GbE port (used for management).
Storage Network
ZFS storage and backup traffic use two dedicated VLANs with jumbo frames enabled.
| VLAN | Name | Subnet | MTU | Purpose |
|---|---|---|---|---|
| 60 | Storage / Ceph public | 10.10.60.0/24 | 9000 | ZFS data traffic from PBS-A and VMs |
| 90 | Backup / Replication | 10.10.90.0/24 | 9000 | PBS-A backup jobs; cross-site PBS replication |
Jumbo frames on storage VLANs only
MTU 9000 applies exclusively to VLANs 60 and 90. All other VLANs remain at MTU 1500. Corosync (VLAN 25) must never use jumbo frames.
Both VLANs carry a gateway (OPNsense .1). VLAN 90 supports cross-site PBS replication over WireGuard; the replication traffic exits sa-fw-01 and reaches Site B's PBS-B instance at 10.20.90.40.
Service VMs
sa-stor-01 hosts the following service VMs on its local ZFS pool.
| VM | Management IP (VLAN 30) | Data / Backup IP | Role |
|---|---|---|---|
| PBS-A | 10.10.30.20 | 10.10.90.40 (VLAN 90) | Proxmox Backup Server for Site A |
sa-dns-01 | 10.10.30.10 | — | Primary Technitium DNS |
sa-dns-02 | 10.10.30.11 | — | Secondary Technitium DNS |
Monitoring and databases
Site A monitoring and database VMs also run on sa-stor-01. Specific VM names and IPs are allocated when those services are deployed; they are not yet confirmed in the current build plan.
PBS-A is the authoritative local backup target for the sa-pve cluster. Cross-site replication copies PBS data to PBS-B at Site B for disaster recovery. See PBS Backups for replication configuration.
Related Pages
- Proxmox Clusters —
sa-pvecluster creation and join order;sa-stor-01as the bootstrap node - PBS Backups — PBS-A and PBS-B configuration and cross-site replication
- VLAN Reference — full VLAN table including VLANs 60 and 90
- Site A Port Map — per-port XL710 VLAN assignments on
sa-stor-01
Proxmox Clusters
Proxmox clustering plan for both sites: cluster names, node membership, the IP and /etc/hosts prerequisites, dedicated Corosync on VLAN 25, and the phased build order.
Site B Ceph Storage
Site B Ceph plan: five-node cluster across FN8TP and FN4T hardware, replication size 3, public (VLAN 60) and cluster (VLAN 65) network split, Micron drive recommendations, and Proxmox pveceph deployment.