Host Synchronization & Cluster Management

Cockpit coordinates multiple physical Vapor host servers. It does this by keeping a copy of their virtual machines, virtual networks, and storage pools in its own central database, ensuring you can manage them all from a single dashboard.

---

1. How Cockpit Claims a Host (Advisory Locks)

If you run multiple Cockpit instances for high availability (to prevent downtime), we must ensure that only one Cockpit server talks to a physical Vapor host at a time. Otherwise, two Cockpit servers might send conflicting commands (like both trying to turn on the same VM).

• 💡 Analogy: The Library Sign-out Card Imagine a single physical Vapor host is like a book in a library. To avoid multiple people writing notes in the book at the same time, we use a checkout card (a PostgreSQL transactional advisory lock). Only the Cockpit server whose name is stamped on the card is allowed to manage the host.

The Claiming Process:

1. Sweeping: A background scheduler (the Collector Service) in Cockpit regularly checks the database for active servers.
2. Locking: For each server, the Cockpit instance attempts to write its name on the database checkout card using an advisory lock.
3. Connecting: If successful, it "claims" the server, opens a secure live communication line (WebSocket), and marks the host status as connected in your dashboard.
4. Failing Over: If that Cockpit server loses its internet connection or crashes, it releases the lock automatically. Another standby Cockpit server immediately checks out the card, takes over the connection, and keeps the server online.

---

2. How Cockpit Syncs Data (State Synchronization)

Once connected to a server, Cockpit needs to keep its dashboard data perfectly up-to-date with what is happening on the physical machine. It does this using a two-part synchronization process:

Step 1: Download Full Snapshot (Get Sync Snapshot)
      "Download the whole catalog of virtual machines & networks"
                               │
                               ▼
Step 2: Stream Live Updates (Get Sync Events + Sync Cursor)
      "Only send updates (like a VM starting or stopping)"

Part A: Full Snapshot (Downloading the Movie)

When a server first connects or starts up after a long disconnect, Cockpit requests a complete configuration overview.

• 💡 Analogy: Think of this like downloading a full directory catalog. It lists every virtual machine, virtual network switch, and storage drive currently active on the physical server.

Part B: Incremental Event Stream (Streaming Live Changes)

After the initial snapshot is saved, Cockpit stops requesting full catalogs to save network bandwidth. Instead, it listens to a live feed of changes (events) as they happen in real-time.

• 💡 Analogy: The Video Bookmark (sync_cursor) Every event (like vm_started or network_created) has a sequence number. Cockpit stores the last processed sequence number as a bookmark (sync_cursor) in the database.
  • If the network drops for 10 seconds, Cockpit doesn't need to download the full snapshot catalog again.
  • When the connection returns, it looks at its bookmark and says: "I stopped at event number 105. Please send me everything starting from 106." This prevents any missed metrics or double-processed events.
  • If Cockpit was offline for a very long time and the event history is cleared, it will automatically fall back to downloading a fresh full snapshot.