Host Synchronization and Cluster Management

Cockpit coordinates multiple physical Vapor host servers. It maintains a synchronized registry of their virtual machines, virtual networks, and storage pools in a centralized database, enabling unified management from a single control pane.

---

1. Host Claiming and High Availability

In high availability (HA) environments running multiple Cockpit instances, coordination is required to ensure that only a single Cockpit server communicates with a given physical Vapor host at any given time. This prevents race conditions and conflicting commands (such as duplicate power operations).

PostgreSQL Transactional Advisory Locks

Cockpit prevents concurrent host management conflicts using PostgreSQL transactional advisory locks. The locking mechanism coordinates host assignment dynamically:

1. Discovery (Sweeping): A background daemon (the Collector Service) in Cockpit regularly polls the database for registered and active hypervisor hosts.
2. Lock Acquisition: For each discovered host, the Cockpit instance attempts to acquire an advisory lock associated with the host's record in the PostgreSQL database.
3. Connection Establishment: Upon successfully acquiring the advisory lock, the Cockpit instance establishes a secure WebSocket connection to the Vapor host agent and updates the host's status to connected in the system state.
4. Failover Execution: If the active Cockpit instance encounters network degradation or crashes, the advisory lock is automatically released. A standby Cockpit instance will immediately acquire the lock, take over the WebSocket connection, and assume management of the host without service interruption.

---

2. State Synchronization Pipeline

Once connected to a Vapor host, Cockpit synchronizes its central database state with the real-time configuration of the physical hypervisor. This synchronization utilizes a two-phase pipeline:

Phase 1: Full State Snapshot (Get Sync Snapshot)
         Downloads the complete inventory of virtual machines,
         networks, and storage pools.
                        │
                        ▼
Phase 2: Real-time Event Streaming (Get Sync Events + Sync Cursor)
         Receives incremental state updates and monitors events via WebSocket.

Phase A: Full State Snapshot

During initial connection or after recovering from a prolonged network partition, Cockpit performs a full reconciliation. It requests the complete configuration catalog from the Vapor host agent. This snapshot populates the central database with the active state of all virtual machines, virtual networks, and storage pools.

Phase B: Incremental Event Streaming and Cursor Tracking

To minimize network overhead, Cockpit transitions to an incremental synchronization mode after processing the initial snapshot. It listens to a real-time WebSocket stream of events (e.g., vm_started, network_created, storage_pool_updated) emitted by the host.

• Sequence ID Tracking (sync_cursor): Each event contains a monotonically increasing sequence number. Cockpit stores the last successfully processed sequence number as a sync_cursor in the database.
• Network Partition Recovery: If the connection drops briefly, Cockpit does not request a full snapshot upon reconnection. Instead, it sends the stored sync_cursor to the host agent, requesting all events starting after that sequence number. This ensures sequential event delivery and prevents duplicate processing or missed metrics.
• Full Fallback: If the Cockpit instance remains disconnected for a duration that exceeds the host's event-log retention window, the synchronization pipeline automatically falls back to Phase A to perform a fresh full state reconciliation.