Site B Ceph Storage
Site B Ceph plan: five-node cluster across FN8TP and FN4T hardware, replication size 3, public (VLAN 60) and cluster (VLAN 65) network split, Micron drive recommendations, and Proxmox pveceph deployment.
Site B runs Ceph across all five compute nodes. The cluster supplies block storage for Kubernetes and VM workloads, with separate public and cluster networks. Deployment uses pveceph.
No Stretched Ceph
Site B Ceph stays local to Site B. Never stretch Ceph across the WireGuard link to Site A. Cross-site disaster recovery uses Proxmox Backup Server (PBS) replication rather than Ceph replication.
Node Roles
All five Site B nodes participate in the Ceph cluster. The two FN8TP nodes (sb-cmp-01, sb-cmp-02) run Ceph MON and MGR processes alongside Kubernetes control-plane workloads. The three FN4T nodes (sb-cmp-03 through sb-cmp-05) serve as high-core OSD and Kubernetes worker nodes. Additional MON/MGR instances may be deployed on FN4T nodes as appropriate.
| Node | Hardware | Ceph Role |
|---|---|---|
sb-cmp-01 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-02 | SYS-5019D-4C-FN8TP | MON, MGR, OSD |
sb-cmp-03 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-04 | SYS-5018D-FN4T | OSD, K8s worker |
sb-cmp-05 | SYS-5018D-FN4T | OSD, K8s worker |
OSD Plan
Each node targets 4-6 enterprise 1.92 TB SSDs as OSDs, for a planned cluster total of 20-30 OSDs. Replication size 3 is the fixed policy: every object is written to three OSDs on three distinct nodes.
Tentative: OSD Counts Not Yet Finalized
The per-node OSD count (4-6) and cluster total (20-30) are planning targets. Exact counts depend on drive procurement and per-chassis slot availability. Confirm slot counts before ordering drives.
Boot Drive Policy
Always use a 512 GB M.2 drive for the Proxmox OS. Never consume a 1.92 TB enterprise SSD as a boot device.
Drive Recommendations
| Use Case | Drive | Rationale |
|---|---|---|
| Write-heavy OSDs (databases, high-ingest workloads) | Micron 5200 MAX | High endurance; write-optimized |
| VM disk images and Kubernetes persistent volumes | Micron 5300 Pro | Balanced read/write for general storage |
Ceph Networks
Ceph uses a two-network split: a public network for client I/O and OSD heartbeats, and a separate cluster network for OSD-to-OSD replication. Both are direct, site-local L2 networks with no default gateway. This follows the Ceph network configuration reference.
| Network | VLAN | Subnet | GW | NIC |
|---|---|---|---|---|
| Storage / Ceph public | 60 | 10.20.60.0/24 | no GW | X710-T4 port 3 |
| Ceph cluster | 65 | 10.20.65.0/24 | no GW | onboard 10GBASE-T |
No Gateway on VLAN 60 or 65
Do not create an SVI, default gateway, or OPNsense interface/static route for either storage network. Ceph clients attach directly to VLAN 60; OSD replication stays on VLAN 65. Never route or bridge 60↔65.
Routing-Necessity Matrix
| Traffic | Network | Routing model |
|---|---|---|
| Ceph client I/O and public daemons | VLAN 60 | Direct attachment; no gateway |
| OSD replication, backfill, recovery | VLAN 65 | Direct attachment; no gateway |
| PBS cross-site synchronization | VLAN 90 | Routed through OPNsense/WireGuard |
Future Kubernetes or VM storage consumers receive a secondary VLAN 60 interface. Routing VLAN 60 is a new architecture decision, not a convenience change.
Per-Node Network IPs
| Node | Ceph public 10.20.60.x | Ceph cluster 10.20.65.x |
|---|---|---|
sb-cmp-01 | 10.20.60.20 | 10.20.65.20 |
sb-cmp-02 | 10.20.60.21 | 10.20.65.21 |
sb-cmp-03 | 10.20.60.30 | 10.20.65.30 |
sb-cmp-04 | 10.20.60.31 | 10.20.65.31 |
sb-cmp-05 | 10.20.60.32 | 10.20.65.32 |
Deployment
Ceph is deployed and managed through Proxmox via pveceph. Do not install Ceph packages manually outside the Proxmox tool chain.
Ceph Release and Repository
The cluster runs Ceph Tentacle (not Squid). The Ansible baseline role configures each node with the Proxmox ceph-tentacle no-subscription repository at download.proxmox.com/debian/ceph-tentacle (suite: trixie) and removes the Proxmox enterprise source file entirely.
Package Version Management
Proxmox owns the Ceph package versions through this repository. The Ansible packages role blacklists the full Ceph stack (ceph.*, radosgw.*, librados.*, librbd.*, libcephfs.*, librgw.*, python3-ceph*) from unattended-upgrades so that security auto-patching never desynchronizes OSD and MON versions across nodes. Ceph upgrades are a manual, staged Proxmox operation.
Deploy Sequence
Complete these steps after the sb-pve cluster is formed and all nodes have their final 10.20.20.x Proxmox Management IPs.
- Confirm all five nodes are joined to
sb-pveand reachable on VLAN 20. - Verify VLAN 60 and VLAN 65 interfaces are up and correctly addressed on all nodes.
- Initialize Ceph on
sb-cmp-01, specifying the public network (10.20.60.0/24) and cluster network (10.20.65.0/24). - Add Ceph MON on all five nodes; bring up MGR on
sb-cmp-01andsb-cmp-02first. - Create OSDs per node using the designated enterprise SSDs.
- Create pools for VM disk images and Kubernetes persistent volumes.
- Verify cluster health (
ceph -s) after each node is added.
Cross-Site Disaster Recovery
Ceph replication does not cross the WireGuard link. PBS replication from PBS-B (backup-data 10.20.90.40) to PBS-A (backup-data 10.10.90.40) provides cross-site disaster recovery for VM and container data. See PBS & Backups for the replication configuration.
Related Pages
- Proxmox Clusters: cluster creation and node joining order
- PBS & Backups: cross-site backup replication via PBS
- VLAN Reference: VLAN 60/65 local-only policy and VLAN 90 routed backup path
- Site B Port Map: per-node NIC and switch wiring for Ceph interfaces
Site A ZFS Storage
ZFS mirror storage plan for sa-stor-01 (Supermicro 5049A-T): drive layout, vdev structure, network configuration, and the service VMs the pool serves.
PBS Backups & Replication
Proxmox Backup Server instances, IPs, VLAN 90 allocation, and the cross-site replication strategy for disaster recovery.