Troubleshooting Guide

This guide details diagnostics and remediation procedures for common operational issues encountered in Cockpit deployments.

---

1. Diagnostics: Host Node Status is "Disconnected"

Symptom: A physical hypervisor node running Vapor displays as disconnected in the Cockpit console, and performance graphs for that node show no updates.

Remediation Workflow:

1. Network Connectivity Verification: Check that the Cockpit service host can route to the Vapor host. Verify that TCP ports 7770 (API interface) and 7770 (event streams) are open and reachable from the Cockpit server.
2. Daemon Status Check: Log into the affected hypervisor host and query the status of the local Vapor service:

   systemctl status vapor.service

   If the service is stopped, restart it: systemctl start vapor.service.
3. API Token Rotation: Cockpit authenticates with Vapor using API tokens. If the token is invalid or has expired, access will be refused. Generate a new API token from the Vapor local configuration and update the host registration details in Cockpit.

---

2. Diagnostics: TLS Certificate Errors

Symptom: Operations fail with SSL/TLS handshake warnings, or live migrations fail to initialize between hypervisors.

Remediation Workflow:

1. Verify Hypervisor Virtualization Status: Connect to the hypervisor and ensure the virtualization daemon is active:

   systemctl status libvirtd

2. Re-initialize TLS Credentials: You can force Cockpit to re-generate libvirt certificates on the node by triggering the TLS configuration endpoint:

   # Renewal request sent to the local Vapor daemon
   curl -sk -X POST https://localhost:7770/api/v1/system/tls/libvirt/setup

3. Establish Trust Chains: Ensure both physical host servers are active and registered within the same Cockpit inventory scope to allow Cockpit to distribute and synchronize CA certificates.

---

3. Diagnostics: Host Connection Flipping (Advisory Lock Conflict)

Symptom: A physical host's connection state in the Cockpit console switches rapidly between connected and disconnected.

Remediation Workflow:

1. Identify Lock Holders: This behavior occurs when multiple active Cockpit instances attempt to manage the same physical host simultaneously. Query the PostgreSQL database to identify current advisory lock sessions:

   -- Query active transactional advisory locks
   SELECT pid, classid, objid FROM pg_locks WHERE locktype = 'advisory';

2. Re-synchronize Management Daemons:
   • Stop the Cockpit service processes on all active instances.
   • Start the Cockpit service on the primary node and wait for it to establish connections to the host inventory.
   • Start the Cockpit service on secondary nodes. The secondary nodes will detect the active advisory locks and remain in standby mode.