Architecture

Architecture Overview

A guided mental model of the two-site private cloud: current state, site responsibilities, control boundaries, and the best path through the architecture documentation.

AORXI is two independent private-cloud sites joined by routed WireGuard. Site A (10.10.0.0/16) provides management, durable services, ZFS, and backups; Site B (10.20.0.0/16) provides Ceph, Kubernetes/OpenShift, and worker-heavy compute. OPNsense owns each site edge, while every switch remains Layer 2.

Read the Diagrams as Architecture State, Not a Live Monitor

Site A still uses its Gateway Max and XG6 bootstrap island. Its approved three-switch target fabric is on hand but not yet production. Site B is parked. Architecture pages distinguish the current bootstrap, approved Site A target, retained Site B design, and future inter-site services.

The System in One Minute

PHYSICAL + LOGICAL MAP · TWO FAILURE DOMAINS

Two Sites. One Routed System.

Select or focus a node. Its routed links brighten while the inspector explains the physical role and logical boundary.

Site A fabric · target

Wire language

  • WAN edge
  • WireGuard route
  • Firewall to core
  • Platform path
  • Client branch
ROUTED TRANSIT

WireGuard

2 active links

WireGuard joins the two OPNsense firewalls while preserving each site as an independent Layer 2, quorum, and storage boundary.

  • sa-fw-01 10.255.0.1
  • sb-fw-01 10.255.0.2
  • No L2, Corosync, or Ceph-cluster stretch
Follow connection
QuestionSite ASite B
What is the site for?Management, ZFS, PBS-A, DNS primary, monitoring, core servicesCeph, Kubernetes/OpenShift, distributed compute, PBS-B
Who routes?sa-fw-01 on sa-edge-01sb-fw-01 on sb-edge-01
What switches packets?sa-sw-01/02/03 UniFi L2 RSTP fabricsb-sw-01 Netgear L2 core plus sb-sw-02 access
What is the local cluster?sa-pve — four nodessb-pve — six nodes
What survives as legacy equipment?Gateway Max/XG6 only for bootstrap and rollbackUSG Pro user branch behind OPNsense on VLAN 253
What crosses the sites?Routed, explicitly allowed service traffic over WireGuardRouted, explicitly allowed service traffic over WireGuard

Ownership Chain

Each layer has one job and one clear owner:

LayerOwnerResponsibilityExplicit non-responsibility
WAN and policyOPNsenseWAN, NAT, WireGuard, routed gateways, DHCP, DMZ, stateful zone policyDoes not provide storage or cluster quorum
Local transportSite-local L2 switchesVLAN carriage, access ports, trunks, link redundancyNo gateway, DHCP, SVI, or cross-site extension
VirtualizationOne Proxmox cluster per siteLocal VM lifecycle and site-local Corosync quorumNo cross-WAN cluster membership
StorageSite A ZFS; Site B CephLocal durable data and workload storageCeph never spans the WAN
Disaster recoveryPBS-A and PBS-BBackup retention and planned cross-site replicationNot synchronous cross-site storage
Platform servicesDNS, OpenBao, UOS, Kubernetes/OpenShiftNamed application and control-plane functionsNo hidden replacement for the routing boundary

Read the Section in Order

1 · The Two-Site Model

Understand why the sites are independent, which boundaries stay local, and why their designs are deliberately asymmetric.

2 · Site Roles & Systems

Learn what each site is responsible for and where shared services, hosts, and workload classes belong.

3 · Physical & Logical Topology

Follow representative packet paths, network planes, and the failure behavior of the routed design.

4 · Full System Map

Search every named host, switch, edge VM, service VM, and logical platform; inspect IPs, networks, placement, and connections.

Invariants That Shape Every Page

Local Failure Domains Stay Local

One Proxmox cluster per site. No stretched Ceph, Layer 2, Corosync, or OpenBao Raft. WireGuard is routed transit only.

OPNsense Is the Stateful Boundary

The firewall VM at each site stays pinned to its local E200. At Site A, OPNsense owns every routed .1, WAN, NAT, DHCP, WireGuard, DMZ, and inter-VLAN policy; the three target switches remain Layer 2 only.

Management Access Stays Private

IPMI never faces the internet. Network management lives on VLAN 10 and is reached only through controlled management paths.

Follow the Detail from Here

Architecture explains ownership and relationships. Exact implementation data lives in the domain sections:

Networking

VLANs, gateways, addressing conventions, DNS, and WireGuard routes.

Switching

Site-specific port maps, trunks, RSTP behavior, and switch operations.

Compute & Storage

Proxmox clusters, Site A ZFS, Site B Ceph, and PBS replication.

Current Build State

What is already built, what is staged, and which gate comes next.

Operational Architecture

These pages turn the system model into operating boundaries: what must start first, which flows are allowed, what recovery can promise, where capacity is constrained, and how failures propagate.

Service Dependencies & Startup Order

Trace runtime and configuration dependencies, boot precedence, degraded modes, and recovery order.

Security Trust Zones & Permitted Flows

See the target default-deny zone model, allowed flow classes, unrouted planes, and service-specific port anchors.

Disaster Recovery Objectives

Review recovery mechanisms, technical ownership, implementation state, and the RPO/RTO decisions that remain open.

Capacity, Power, Thermal & Expansion

Compare known compute, storage, NIC, port, and PoE envelopes with the electrical and thermal measurements still required.

Blast Radius & Recovery

Map infrastructure failures to affected capabilities, surviving functions, recovery paths, and unresolved exposure.

Design Principles & Hard Rules

Non-negotiable architecture rules and guiding principles every operator must understand before making any change to this homelab.

The Two-Site Model

Why AORXI uses two independent site-local failure domains, what the sites share, and why Site A and Site B intentionally use different network and storage designs.