O2CB Cluster Management

Oracle Cluster File System v2 (OCFS2) is a general-purpose, shared-disk clustered filesystem designed for active/active concurrent read and write operations across multiple virtualization hosts. In Vapor, the cluster infrastructure and distributed lock management are handled by the O2CB (OCFS2 Cluster Base) stack.

O2CB enables multiple Vapor nodes to concurrently mount the same SAN LUN, iSCSI target, or Multipath NVMe-oF device, allowing virtual machines and clustered storage pools to run simultaneously across your infrastructure without data corruption.

---

Heartbeat Modes

O2CB monitors host health and cluster quorum through two heartbeat mechanisms:

Heartbeat Mode	Description	Recommended For
Local Heartbeat (Default)	Heartbeat metadata is written directly to a reserved block on each individual mounted OCFS2 filesystem. Simpler setup with no dedicated devices required.	Simple or single-storage deployments with a few shared datastores.
Global Heartbeat	Heartbeat runs against dedicated, independent heartbeat region block devices as a persistent background service before datastores are mounted.	Fault-tolerant multi-node clusters, enterprise SAN/iSCSI environments, or setups with multiple shared datastores.

---

Migrating from Local to Global Heartbeat

Global heartbeat separates cluster health monitoring from individual datastore I/O. If an individual datastore experiences transient I/O latency or pauses, it will not trigger a node fencing event (kernel panic or reboot) as long as the dedicated heartbeat quorum remains healthy.

Prerequisites & Architecture

To make global heartbeat truly fault-tolerant, you should configure 3 shared heartbeat region block devices mapped to all cluster nodes, ideally backed by independent storage providers or a distributed storage pool (e.g., Region 1 on Storage Target A, Region 2 on Storage Target B, Region 3 on Storage Target C):

• Shared Storage Mapping: Every host in the cluster must attach and map all 3 heartbeat block devices. Each node writes its heartbeat to its assigned slot across all 3 regions and reads the other nodes' heartbeats from them.
• Quorum & Fault-Tolerance: O2CB uses a majority quorum rule. With 3 heartbeat regions across independent storage targets, the failure of any single storage server or network path still preserves 2/3 active quorum, preventing cluster fencing.
• Device Sizing: Dedicated heartbeat devices only need 500 MB to 1 GiB each. Large disks are unnecessary.
• Multipath / Transport: Map the 3 heartbeat block devices to all cluster nodes over redundant network interfaces (e.g., multipath iSCSI, SAN fabric, or NVMe-oF).
• Device Availability: Ensure the block devices are persistently discovered and mapped on each host upon boot.

---

Step-by-Step Migration Guide

Follow these steps sequentially to convert an active cluster from Local to Global heartbeat mode:

[Step 1: Quiesce VMs] ──► [Step 2: Enter Maintenance] ──► [Step 3: Set Global Mode]
                                                                    │
[Step 6: Exit Maint]  ◄── [Step 5: Update Superblocks] ◄── [Step 4: Add 3 Heartbeats]

Step 1: Quiesce Workloads / VMs

Before modifying cluster-wide heartbeat parameters, all active I/O on the shared datastores must be stopped:

1. Live-migrate running virtual machines to storage pools outside the target cluster, or cleanly shut down the VMs.
2. Ensure no processes or containers hold open file handles to the OCFS2 mount points.

Step 2: Enter Maintenance Mode on All Nodes

All nodes in the cluster must unmount the shared datastores and unregister from the O2CB kernel stack before the heartbeat mode can be changed:

1. In the Vapor Web Console, go to Storage > O2CB Cluster.
2. Click Enter Maintenance on each node.
3. Vapor verifies active locks, safely unmounts all active OCFS2 filesystems, stops local heartbeats, and unregisters the cluster from live kernel memory.

Step 3: Switch Heartbeat Mode to Global

1. On any cluster node, open the cluster actions menu (⋮) in the top header and click Set Heartbeat Mode (global).
2. In the confirmation dialog, confirm changing the mode from local to global.
3. Vapor updates /etc/ocfs2/cluster.conf with heartbeat_mode = global and replicates the change across cluster members.

Step 4: Add 3 Heartbeat Region Devices

1. In the Heartbeat Regions card, click Add Heartbeat.
2. Enter the block device path for your first shared heartbeat LUN (e.g., /dev/mapper/mpath-hb1 or /dev/sdb).
3. Click Create.
   • If the block device is raw and unformatted ($\le 1\text{ GiB}$), Vapor automatically initializes it with mkfs.ocfs2 --global-heartbeat and extracts its region UUID.
4. Repeat this step for Region 2 and Region 3.

Step 5: Update On-Disk Cluster Stack Metadata on Existing Datastores

When an OCFS2 volume is originally formatted under local heartbeat mode, its on-disk superblock is stamped with local. If you attempt to mount a local-mode volume while the running O2CB kernel stack is in global mode, the mount will fail with: mount.ocfs2: Heartbeat mode is invalid while trying to join the group

To update existing datastores to match the new global heartbeat stack:

1. Identify the block device or WWN path of your shared datastore (e.g., /dev/disk/by-id/wwn-0x60000...).
2. Inspect the current on-disk vs. running cluster stack:

   sudo o2cluster -o /dev/disk/by-id/<wwn-id>
   # Outputs: o2cb,<cluster-name>,local

3. Update the on-disk superblock while the filesystem is unmounted:

   sudo tunefs.ocfs2 --update-cluster-stack /dev/disk/by-id/<wwn-id>

   (Confirm with y when prompted).
4. Verify that the on-disk stack now matches:

   sudo o2cluster -o /dev/disk/by-id/<wwn-id>
   # Outputs: o2cb,<cluster-name>,global

   (Note: Since this updates the shared disk's superblock, performing this command on one node automatically updates it for all cluster members).

Step 6: Start Heartbeat & Exit Maintenance Mode

1. In the Vapor Web Console, click Sync to Kernel to register the new cluster configuration and nodes.
2. Select Actions (⋮) > Start Heartbeat to start the 3 global heartbeat regions in the kernel.
3. Click Exit Maintenance on each node.
4. Vapor starts the storage pools, remounts the OCFS2 shared filesystems, and brings all virtualization storage pools back online.
5. Power on or migrate virtual machines back to the shared storage pools.

---

Managing O2CB in the Vapor Web Console

In the Vapor web console, navigate to Storage > O2CB Cluster to manage and monitor O2CB cluster state.

1. Cluster Status & Actions

The top card provides real-time status and operational controls:

• Cluster Identity: Displays the effective cluster name, heartbeat mode, total configured node count, and active heartbeat regions.
• Status Badges:
  • Online (Green): The cluster is registered and active in kernel memory.
  • Offline (Gray): The cluster is stopped or unregistered.
  • Not in Kernel (Amber): The cluster exists in configuration but has not been registered in the kernel.
• Primary Actions:
  • Register Cluster / Sync to Kernel: Automatically parses cluster.conf and synchronizes all nodes into running kernel memory.
  • Enter Maintenance / Exit Maintenance: Safely unmounts active filesystems and unregisters the node from the cluster before host servicing.
• Actions Menu (⋮):
  • Start Heartbeat / Stop Heartbeat: Controls global heartbeat runtime (global mode only).
  • Set Heartbeat Mode: Switch between local and global heartbeat modes.
  • Unregister Cluster: Unloads cluster objects from kernel memory.
  • Delete Cluster: Removes the cluster definition from /etc/ocfs2/cluster.conf.

---

2. Nodes Management

The Nodes section allows administrators to view, add, and remove cluster members:

+---------------------------------------------------------------------------------------------------+
| Node Name    Number    IP Address       Port     Status                          Actions          |
+---------------------------------------------------------------------------------------------------+
| reg2-1       1         10.99.16.20      7777     [Local] [✓ In Kernel]          —                |
| reg2-2       2         10.99.16.21      7777     [✓ In Kernel]                   [Remove]         |
| reg2-3       3         10.99.16.22      7777     [✓ In Kernel]                   [Remove]         |
| reg2-4       4         10.99.16.250     7777     [⚠ Not in Kernel]               [Remove]         |
+---------------------------------------------------------------------------------------------------+

Node Status Indicators:

• Local Badge: Indicates that this node corresponds to the current local Vapor host.
• ✓ In Kernel (Green Badge): Confirms the node is actively recognized in running Linux kernel memory.
• ⚠ Not in Kernel (Amber Badge): Indicates the node is present in /etc/ocfs2/cluster.conf but missing from live kernel memory. Use Sync to Kernel to register it online.

Adding a Node:

1. Under Nodes, expand the Add Node form.
2. Enter the Node Name (e.g. n04 or host-4), Node Number (slot index), IP Address, and Port (default 7777).
3. Click Add Node.
4. Vapor automatically saves the node to /etc/ocfs2/cluster.conf and immediately executes online registration to sync the node into kernel memory without downtime.

Removing a Node:

• Click Remove next to a node to delete its entry from the configuration file.

---

3. Heartbeat Regions (Global Mode)

When global heartbeat mode is enabled, the Heartbeat Regions section allows administrators to:

• Add block device paths (e.g. /dev/mapper/mpatha or /dev/sdb) to participate in cluster heartbeating.
• Remove obsolete or decommissioned heartbeat devices.

---

Maintenance Mode Workflow

Before performing disruptive storage network maintenance (such as switch firmware upgrades, multipath reconfiguration, or adapter replacements), place the host into Maintenance Mode:

1. Click Enter Maintenance in the cluster header.
2. Vapor performs a safety preflight check:
   • Identifies all mounted OCFS2 filesystems.
   • Verifies whether virtual machine virtual disks or active workloads are using the storage.
3. Unmounts all active OCFS2 storage pools cleanly.
4. Stops heartbeat services and unregisters the node from the O2CB stack.
5. Once physical work is completed, click Exit Maintenance to re-register the cluster, start heartbeating, and remount shared datastores.

Never force unmount while VMs are active

Taking a node down abruptly without unmounting active OCFS2 filesystems will trigger O2CB self-fencing (kernel panic or immediate reboot) to prevent split-brain storage corruption.

---

REST API Reference

Vapor exposes a full REST API for O2CB cluster management:

Method	Endpoint	Description
GET	/storage/o2cb/clusters	List all configured O2CB clusters.
POST	/storage/o2cb/clusters	Create a new O2CB cluster definition.
GET	/storage/o2cb/clusters/{name}	Get detailed cluster configuration and nodes.
DELETE	/storage/o2cb/clusters/{name}	Delete an O2CB cluster configuration.
GET	/storage/o2cb/clusters/{name}/status	Query live kernel status of the cluster.
POST	/storage/o2cb/clusters/{name}/register	Dynamically register cluster and sync nodes into kernel.
POST	/storage/o2cb/clusters/{name}/unregister	Unregister cluster from live kernel memory.
GET	/storage/o2cb/clusters/{name}/nodes	List all configured member nodes with kernel status.
POST	/storage/o2cb/clusters/{name}/nodes	Add a new member node and auto-register online.
DELETE	/storage/o2cb/clusters/{name}/nodes/{node}	Remove a member node.
GET	/storage/o2cb/clusters/{name}/heartbeats	List all configured heartbeat regions.
POST	/storage/o2cb/clusters/{name}/heartbeats	Add a heartbeat region device.
DELETE	/storage/o2cb/clusters/{name}/heartbeats/{device}	Remove a heartbeat region device.
PUT	/storage/o2cb/clusters/{name}/heartbeat-mode	Set heartbeat mode (local or global).
GET	/storage/o2cb/clusters/{name}/maintenance/preflight	Perform preflight check before entering maintenance.
POST	/storage/o2cb/clusters/{name}/maintenance/enter	Safely enter O2CB maintenance mode.
POST	/storage/o2cb/clusters/{name}/maintenance/exit	Exit maintenance mode and remount datastores.

---

CLI Reference

Administrators with SSH or console access can also inspect O2CB status using standard CLI tools:

# Check cluster status in kernel memory
sudo o2cb cluster-status <cluster-name>

# Register configuration into running kernel
sudo o2cb register-cluster <cluster-name>

# Unregister cluster from kernel
sudo o2cb unregister-cluster <cluster-name>

# Inspect active nodes in kernel configfs
ls -la /sys/kernel/config/cluster/<cluster-name>/node/

# Start and stop global heartbeat service
sudo o2cb start-heartbeat <cluster-name>
sudo o2cb stop-heartbeat <cluster-name>

# Inspect on-disk cluster stack vs running stack
sudo o2cluster -o /dev/disk/by-id/<device>
sudo o2cluster -r

# Update on-disk filesystem cluster stack to match running cluster
sudo tunefs.ocfs2 --update-cluster-stack /dev/disk/by-id/<device>

---

Troubleshooting & Best Practices

1. Resolving "Not in Kernel" Node Discrepancies

• Cause: A node was added to /etc/ocfs2/cluster.conf directly or while the cluster stack was offline.
• Solution: Click the Sync to Kernel button in the Vapor UI or run sudo o2cb register-cluster <cluster-name> via CLI.

2. Network Firewall Requirements

• All participating cluster nodes must be able to communicate bidirectionally over TCP port 7777. Ensure firewall rules allow traffic between all host storage interfaces:

  sudo ufw allow 7777/tcp
  # or firewalld:
  sudo firewall-cmd --permanent --add-port=7777/tcp && sudo firewall-cmd --reload

3. Heartbeat Mode Mismatch on Mount

• Symptom: mount.ocfs2: Heartbeat mode is invalid while trying to join the group
• Cause: The running cluster is in global heartbeat mode, but the filesystem on disk is stamped with local mode.
• Solution: Run sudo tunefs.ocfs2 --update-cluster-stack /dev/disk/by-id/<device> while the volume is unmounted.

4. Fencing and Quorum Rules

• OCFS2 uses a majority quorum algorithm. If communication is lost between nodes, minority partitions will automatically self-fence (reboot) to protect data integrity. Ensure redundant network links (Bonds / Multipath) are configured on storage interfaces.