Site Replication and Disaster Recovery (DR)

Cockpit provides hypervisor-level disaster recovery through active-passive virtual machine replication. This enables workloads running at a primary datacenter (Site A) to replicate block storage devices asynchronously to a secondary disaster recovery site (Site B) over a WAN or secondary network.

---

1. Architectural Design

Replication is coordinated by Cockpit but executes directly peer-to-peer between physical virtualization hosts (Vapor daemons) to optimize network throughput and reduce management plane bottlenecking.

• Data Transfer Protocol: Files and incremental block delta patches are streamed over secure TLS using the TUS (Resumable Upload) Protocol (PATCH HTTP requests).
• Asynchronous Storage Snapshots: Uses QCOW2 backing chains to transfer blocks without disrupting active virtual machines.

---

2. Replication Mechanism

Replication runs in a two-stage process using snapshot layers:

Phase 1: Baseline Sync (Full Copy)

1. Cockpit instructs the Source Vapor host to create a full backup.
2. The hypervisor freezes the guest filesystem temporarily (via the QEMU Guest Agent) and runs qemu-img convert to stream the disk volume.
3. Cockpit registers the upload session on the Target Vapor host.
4. The Source Vapor streams the base .qcow2 file block-by-block using the TUS protocol.
5. The Target Vapor registers the baseline disk image.

Phase 2: Incremental Sync (RPO-driven)

1. At the configured Recovery Point Objective (RPO) interval (e.g., 15 minutes), Cockpit triggers an incremental sync.
2. The Source Vapor locks QEMU, creates a new overlay file, and runs qemu-img convert referencing the parent snapshot:

   qemu-img convert -O qcow2 -o backing_file=parent.qcow2,backing_fmt=qcow2 source.qcow2 delta.qcow2

3. Source Vapor streams only the delta.qcow2 patch file via TUS patch requests to the target host.
4. Target Vapor writes the delta block patch into the replication backing chain, creating a Point-in-Time (PIT) recovery state.

---

3. Configuration Steps

3.1 Link Site Pairs

Before enrolling workloads, administrators must pair the Cockpit sites:

1. Log in to the target (DR) Cockpit instance and extract the active authorization token (from cockpit:auth in local storage).
2. Log in to the primary Cockpit instance, navigate to Configure -> Site Pairs, and click Link Site Pair.
3. Input the remote site name, endpoint API URL (e.g., https://192.168.122.128:7771/api/v1), token, and optional TLS certificates.

3.2 Enroll Virtual Machines

1. Select the target VM in the inventory tree and navigate to the Replication tab.
2. Configure the replication policy:
   • Target Site: Select the paired DR site.
   • Target Datastore: Select the storage pool at the DR site.
   • RPO (Recovery Point Objective): Select synchronization intervals (15 Minutes, 30 Minutes, 1 Hour, 4 Hours, or 24 Hours).
   • PIT Snapshots: The maximum number of historical checkpoints to retain on the DR site (Multiple Point-in-Time count).
   • Data Compression: Option to compress the disk stream during transit to optimize WAN bandwidth.
3. Click Enable Replication to trigger the baseline synchronization.

---

4. Disaster Recovery Workflows

Administrators initiate recovery and validation procedures from the ACTIONS ▾ menu of the VM's Replication status panel:

4.1 Test Failover (Disaster Recovery Drills)

To verify recovery viability without disrupting the production VM or stopping replication:

1. Select Test Failover and click Start Drill.
2. Backend Action: Cockpit clones the target replica storage from the latest PIT snapshot, creates a temporary VM instance, and boots it inside an isolated host network (bubble network).
3. Verify the VM's guest OS services via the console.
4. Select Stop Test Failover to power off the sandbox VM and delete the temporary clones from the DR storage pool.

4.2 Planned Migration (Graceful Switchover)

Used to permanently relocate workloads during scheduled maintenance:

1. Select Planned Migration and click Start Migration.
2. Backend Action:
   • The primary VM is gracefully powered off.
   • A final catch-up replication sync executes to transfer any remaining dirty blocks.
   • The VM registration is deleted from the primary host, and the VM is registered and booted on the DR host.

4.3 Emergency Failover (Unplanned Outage)

Used when the primary datacenter is offline due to disaster:

1. Select Emergency Failover and click Emergency Failover.
2. Backend Action: Cockpit skips primary VM shutdown and final synchronization. It registers the VM on the DR host immediately using the last successfully synced PIT snapshot and powers the VM on.