Platform Services

Secrets: OpenBao

Two independent OpenBao instances (sa-bao-01, sb-bao-01) on the edge E200s: secret tiers, cross-site transit auto-unseal, AppRole consumers, and the break-glass fallback.

Each site runs one OpenBao (Vault OSS fork) instance on its edge E200. A small Go binary in a roughly 2 GB VM fits the E200 workload boundary. This placement was decided on 2026-07-02. The sites do not share a stretched Raft cluster: OpenBao has no cross-site replication, and a two-node Raft cluster over WireGuard cannot maintain quorum.

Topology

Site ASite B
VMsa-bao-01 on sa-edge-01sb-bao-01 on sb-edge-01
Final address10.10.30.40/24 (VLAN 30)10.20.30.40/24 (VLAN 30)
Specs2 vCPU / 2 GB / 20 GB, Ubuntu noblesame
Storagesingle-node integrated Raftsame
Version2.5.4 (pinned tarball, sha256-verified)same

Cross-site disaster recovery uses scheduled bao operator raft snapshot save, shipped to the other site over the backup path (VLAN 90, PBS pattern): each site's secrets are restorable at the other.

Seal / Unseal

Each instance auto-unseals via the other site's transit engine over the WireGuard tunnel: sa-bao-01 points at sb-bao-01 and vice versa. Because the tunnel is the unseal path, Pulumi declares OPNsense order=1,up=90,down=180 before bao order=2,up=60,down=180; Site A UOS follows at order=3,up=0,down=180. Proxmox applies up after the current VM starts, so the firewall's 90-second value is the delay before bao.

Cold-Start Deadlock: Documented Break-Glass

If both sites are down at once, neither can auto-unseal. To recover, seal-migrate one instance back to Shamir with a temporary seal stanza swap and the recovery keys. Unseal it manually, let the other instance auto-unseal against it, and then migrate back. Both sites' recovery keys live in the password manager; this is the documented recovery path.

Interim: Site A Only (2026-07-02)

Site B is non-operational, so sa-bao-01 runs standalone with Shamir manual unseal (3 of 5 keys after the rare VM reboot). This is the first bootstrap step, not temporary work. Transit auto-unseal follows when Site B comes online. The site-b Pulumi stack stays parked at enabled: false.

Secret Tiers

TierStoreContents
0: bootstraprepo-root .env.local (gitignored)Proxmox creds, PULUMI_CONFIG_PASSPHRASE, ANSIBLE_VAULT_PASSWORD: everything needed to (re)build bao itself
1: runtimeOpenBao KV v2 (per site)app/service credentials, API keys, cert material

Tier 0 can never migrate into bao: bao cannot hold the secrets that build bao.

Consumers (bao-first, Executed 2026-07-01)

The KV v2 mount is homelab, with paths opnsense/site-a, unifi/site-a, and pulumi/proxmox; Site B mirrors under */site-b on sb-bao-01 later. Access is per-consumer read-only via AppRole (opnsense-config, unifi-config, pulumi-provision), with role_id/secret_id in root .env.local.

  • Ansible secret variables in opnsense/config and unifi/config use KV v2 lookups. When bao is unreachable or sealed, they fall back to the encrypted vault-credentials.yml files. Those files remain as the documented break-glass path and are resynchronized with make openbao-config-seed.
  • Pulumi make targets overlay PROXMOX_VE_* from homelab/pulumi/proxmox and fall back to .env.local on failure. Stack encryption stays on the local passphrase provider because a bao-backed provider cannot work while bao is sealed, and aorxi-openbao cannot depend on the service it builds.
  • Unsetting BAO_ADDR forces pure fallback mode everywhere.

Still Deferred

Transit engines and seal migration (needs Site B), TLS via Let's Encrypt DNS-01 (sa-bao-01.core.aorxi.io; never expose port 8200 beyond the local segment while TLS is off), the Raft snapshot timer with cross-site shipping, and the api_addr correction at the VLAN 30 cutover. Init and recovery-key handling stay a manual runbook.

  • Initial Site Bootstrap: where bao provisioning and the bao-first migration sit in the build order
  • DNS VMs: the internal zone that will name the bao endpoints
  • Decisions Log: dated entries for the 2026-07-01/02 secrets decisions