Site B Ceph Storage
Site B Ceph plan: five-node cluster across FN8TP and FN4T hardware, replication size 3, public (VLAN 60) and cluster (VLAN 65) network split, Micron drive recommendations, and Proxmox pveceph deployment.
Site B runs a Ceph cluster distributed across all five compute nodes, providing block storage for Kubernetes and VM workloads. This page covers node roles, OSD sizing, network layout, drive recommendations, and deployment via pveceph.
No stretched Ceph
Site B Ceph stays local to Site B. Never stretch Ceph across the WireGuard link to Site A. Cross-site disaster recovery is handled by Proxmox Backup Server (PBS) replication — not Ceph replication.
Node Roles
All five Site B nodes participate in the Ceph cluster. The two FN8TP nodes (sb-cmp-01, sb-cmp-02) run Ceph MON and MGR processes alongside Kubernetes control-plane workloads. The three FN4T nodes (sb-cmp-03 through sb-cmp-05) serve as high-core OSD and Kubernetes worker nodes. Additional MON/MGR instances may be deployed on FN4T nodes as appropriate.
| Node | Hardware | Ceph Role |
|---|---|---|
sb-cmp-01 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-02 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-03 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-04 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-05 | SYS-5018D-FN4T | OSD, K8s worker |
OSD Plan
Each node targets 4–6 enterprise 1.92 TB SSDs as OSDs, for a planned cluster total of 20–30 OSDs. Replication size 3 is the fixed policy: every object is written to three OSDs on three distinct nodes.
Tentative — OSD counts not yet finalized
The per-node OSD count (4–6) and cluster total (20–30) are planning targets. Exact counts depend on drive procurement and per-chassis slot availability. Confirm slot counts before ordering drives.
Boot drive policy
Always use a 512 GB M.2 drive for the Proxmox OS. Never consume a 1.92 TB enterprise SSD as a boot device.
Drive Recommendations
| Use Case | Drive | Rationale |
|---|---|---|
| Write-heavy OSDs (databases, high-ingest workloads) | Micron 5200 MAX | High endurance; write-optimized |
| VM disk images and Kubernetes persistent volumes | Micron 5300 Pro | Balanced read/write for general storage |
Ceph Networks
Ceph uses a two-network split: a public network for client I/O and OSD heartbeats, and a separate cluster network for OSD-to-OSD replication traffic. VLAN 65 carries no gateway by design — replication traffic must not leave the local site.
| Network | VLAN | Subnet | GW | NIC |
|---|---|---|---|---|
| Storage / Ceph public | 60 | 10.20.60.0/24 | GW | X710-T4 port 3 |
| Ceph cluster | 65 | 10.20.65.0/24 | no GW | onboard 10GBASE-T |
No gateway on VLAN 65
VLAN 65 (Ceph cluster) has no OPNsense gateway. Do not assign a default gateway to any host interface on this VLAN. Ceph replication traffic must remain local and must never traverse WireGuard.
Per-Node Network IPs
| Node | Ceph public 10.20.60.x | Ceph cluster 10.20.65.x |
|---|---|---|
sb-cmp-01 | 10.20.60.20 | 10.20.65.20 |
sb-cmp-02 | 10.20.60.21 | 10.20.65.21 |
sb-cmp-03 | 10.20.60.30 | 10.20.65.30 |
sb-cmp-04 | 10.20.60.31 | 10.20.65.31 |
sb-cmp-05 | 10.20.60.32 | 10.20.65.32 |
Deployment
Ceph is deployed and managed through Proxmox via pveceph. Do not install Ceph packages manually outside the Proxmox tool chain.
Ceph Release and Repository
The cluster runs Ceph Tentacle (not Squid). The Ansible baseline role configures each node with the Proxmox ceph-tentacle no-subscription repository at download.proxmox.com/debian/ceph-tentacle (suite: trixie) and removes the Proxmox enterprise source file entirely.
Package version management
Proxmox owns the Ceph package versions through this repository. The Ansible packages role blacklists the full Ceph stack (ceph.*, radosgw.*, librados.*, librbd.*, libcephfs.*, librgw.*, python3-ceph*) from unattended-upgrades so that security auto-patching never desynchronizes OSD and MON versions across nodes. Ceph upgrades are a manual, staged Proxmox operation.
Deploy Sequence
Complete these steps after the sb-pve cluster is formed and all nodes have their final 10.20.20.x Proxmox Management IPs.
- Confirm all five nodes are joined to
sb-pveand reachable on VLAN 20. - Verify VLAN 60 and VLAN 65 interfaces are up and correctly addressed on all nodes.
- Initialize Ceph on
sb-cmp-01, specifying the public network (10.20.60.0/24) and cluster network (10.20.65.0/24). - Add Ceph MON on all five nodes; bring up MGR on
sb-cmp-01andsb-cmp-02first. - Create OSDs per node using the designated enterprise SSDs.
- Create pools for VM disk images and Kubernetes persistent volumes.
- Verify cluster health (
ceph -s) after each node is added.
Cross-Site Disaster Recovery
Ceph replication does not cross the WireGuard link. PBS replication from PBS-B (backup-data 10.20.90.40) to PBS-A (backup-data 10.10.90.40) provides cross-site disaster recovery for VM and container data. See PBS & Backups for the replication configuration.
Related Pages
- Proxmox Clusters — cluster creation and node joining order
- PBS & Backups — cross-site backup replication via PBS
- VLAN Reference — VLAN 60 and VLAN 65 subnets and gateway assignments
- Site B Port Map — per-node NIC and switch wiring for Ceph interfaces
Site A ZFS Storage
ZFS mirror storage plan for sa-stor-01 (Supermicro 5049A-T): drive layout, vdev structure, network configuration, and the service VMs the pool serves.
PBS Backups & Replication
Proxmox Backup Server instances, IPs, VLAN 90 allocation, and the cross-site replication strategy for disaster recovery.