O2CB Cluster Management

Oracle Cluster File System v2 (OCFS2) is a general-purpose, shared-disk clustered filesystem designed for active/active concurrent read and write operations across multiple virtualization hosts. In Vapor, the cluster infrastructure and distributed lock management are handled by the O2CB (OCFS2 Cluster Base) stack.

O2CB enables multiple Vapor nodes to concurrently mount the same SAN LUN, iSCSI target, or Multipath NVMe-oF device, allowing virtual machines and clustered storage pools to run simultaneously across your infrastructure without data corruption.

---

Architecture: Configuration vs. Live Kernel State

O2CB operates on two distinct layers that administrators must understand:

+-------------------------------------------------------------+
|                 Vapor Web Console / REST API                |
+-------------------------------------------------------------+
                              |
        +---------------------+---------------------+
        |                                           |
        v                                           v
+-------------------------------+   +-------------------------------+
|  Static Configuration File    |   |    Running Linux Kernel       |
|    /etc/ocfs2/cluster.conf    |   |   /sys/kernel/configfs/...    |
| (Defines cluster, nodes, IPs) |   | (Live active cluster objects) |
+-------------------------------+   +-------------------------------+
        |                                           ^
        +---- Dynamic Online Registration (o2cb) ---+

1. Static Configuration (/etc/ocfs2/cluster.conf):
   • The on-disk configuration file specifying the cluster name, node slots, node names, IP addresses, ports (default 7777), and global heartbeat regions.
2. Live Kernel Memory (/sys/kernel/config/cluster/<cluster>/):
   • The dynamic configfs kernel filesystem representing active cluster objects in memory. For a node to participate in cluster communications and DLM locking, its definition must exist in kernel memory.
3. Dynamic Online Registration:
   • Running o2cb register-cluster <cluster> parses cluster.conf and registers newly added nodes dynamically into running kernel memory without requiring filesystems to be unmounted or services restarted.

---

Heartbeat Modes

O2CB monitors host health through two heartbeat mechanisms:

Heartbeat Mode	Description	Recommended For
Local Heartbeat (Default)	Heartbeat metadata is written directly to a reserved block on each individual mounted OCFS2 filesystem. Simpler setup with no dedicated devices required.	Most standard deployments with a few shared datastores.
Global Heartbeat	Heartbeat runs against dedicated, independent heartbeat region block devices as a persistent background service before datastores are mounted.	Large-scale clusters, numerous shared LUNs, or enterprise SAN environments.

---

Managing O2CB in the Vapor Web Console

In the Vapor web console, navigate to Storage > O2CB Cluster to manage and monitor O2CB cluster state.

1. Cluster Status & Actions

The top card provides real-time status and operational controls:

• Cluster Identity: Displays the effective cluster name, heartbeat mode, total configured node count, and active heartbeat regions.
• Status Badges:
  • Online (Green): The cluster is registered and active in kernel memory.
  • Offline (Gray): The cluster is stopped or unregistered.
  • Not in Kernel (Amber): The cluster exists in configuration but has not been registered in the kernel.
• Primary Actions:
  • Register Cluster / Sync to Kernel: Automatically parses cluster.conf and synchronizes all nodes into running kernel memory.
  • Enter Maintenance / Exit Maintenance: Safely unmounts active filesystems and unregisters the node from the cluster before host servicing.
• Actions Menu (⋮):
  • Start Heartbeat / Stop Heartbeat: Controls global heartbeat runtime (global mode only).
  • Set Heartbeat Mode: Switch between local and global heartbeat modes.
  • Unregister Cluster: Unloads cluster objects from kernel memory.
  • Delete Cluster: Removes the cluster definition from /etc/ocfs2/cluster.conf.

---

2. Nodes Management

The Nodes section allows administrators to view, add, and remove cluster members:

+---------------------------------------------------------------------------------------------------+
| Node Name    Number    IP Address       Port     Status                          Actions          |
+---------------------------------------------------------------------------------------------------+
| reg2-1       1         10.99.16.20      7777     [Local] [✓ In Kernel]          —                |
| reg2-2       2         10.99.16.21      7777     [✓ In Kernel]                   [Remove]         |
| reg2-3       3         10.99.16.22      7777     [✓ In Kernel]                   [Remove]         |
| reg2-4       4         10.99.16.250     7777     [⚠ Not in Kernel]               [Remove]         |
+---------------------------------------------------------------------------------------------------+

Node Status Indicators:

• Local Badge: Indicates that this node corresponds to the current local Vapor host.
• ✓ In Kernel (Green Badge): Confirms the node is actively recognized in running Linux kernel memory.
• ⚠ Not in Kernel (Amber Badge): Indicates the node is present in /etc/ocfs2/cluster.conf but missing from live kernel memory. Use Sync to Kernel to register it online.

Adding a Node:

1. Under Nodes, expand the Add Node form.
2. Enter the Node Name (e.g. n04 or host-4), Node Number (slot index), IP Address, and Port (default 7777).
3. Click Add Node.
4. Vapor automatically saves the node to /etc/ocfs2/cluster.conf and immediately executes online registration to sync the node into kernel memory without downtime.

Removing a Node:

• Click Remove next to a node to delete its entry from the configuration file.

---

3. Heartbeat Regions (Global Mode)

When global heartbeat mode is enabled, the Heartbeat Regions section allows administrators to:

• Add block device paths (e.g. /dev/mapper/mpatha or /dev/sdb) to participate in cluster heartbeating.
• Remove obsolete or decommissioned heartbeat devices.

---

Maintenance Mode Workflow

Before performing disruptive storage network maintenance (such as switch firmware upgrades, multipath reconfiguration, or adapter replacements), place the host into Maintenance Mode:

1. Click Enter Maintenance in the cluster header.
2. Vapor performs a safety preflight check:
   • Identifies all mounted OCFS2 filesystems.
   • Verifies whether virtual machine virtual disks or active workloads are using the storage.
3. Unmounts all active OCFS2 storage pools cleanly.
4. Stops heartbeat services and unregisters the node from the O2CB stack.
5. Once physical work is completed, click Exit Maintenance to re-register the cluster, start heartbeating, and remount shared datastores.

Never force unmount while VMs are active

Taking a node down abruptly without unmounting active OCFS2 filesystems will trigger O2CB self-fencing (kernel panic or immediate reboot) to prevent split-brain storage corruption.

---

REST API Reference

Vapor exposes a full REST API for O2CB cluster management:

Method	Endpoint	Description
GET	/storage/o2cb/clusters	List all configured O2CB clusters.
POST	/storage/o2cb/clusters	Create a new O2CB cluster definition.
GET	/storage/o2cb/clusters/{name}	Get detailed cluster configuration and nodes.
DELETE	/storage/o2cb/clusters/{name}	Delete an O2CB cluster configuration.
GET	/storage/o2cb/clusters/{name}/status	Query live kernel status of the cluster.
POST	/storage/o2cb/clusters/{name}/register	Dynamically register cluster and sync nodes into kernel.
POST	/storage/o2cb/clusters/{name}/unregister	Unregister cluster from live kernel memory.
GET	/storage/o2cb/clusters/{name}/nodes	List all configured member nodes with kernel status.
POST	/storage/o2cb/clusters/{name}/nodes	Add a new member node and auto-register online.
DELETE	/storage/o2cb/clusters/{name}/nodes/{node}	Remove a member node.
GET	/storage/o2cb/clusters/{name}/heartbeats	List all configured heartbeat regions.
POST	/storage/o2cb/clusters/{name}/heartbeats	Add a heartbeat region device.
DELETE	/storage/o2cb/clusters/{name}/heartbeats/{device}	Remove a heartbeat region device.
PUT	/storage/o2cb/clusters/{name}/heartbeat-mode	Set heartbeat mode (local or global).
GET	/storage/o2cb/clusters/{name}/maintenance/preflight	Perform preflight check before entering maintenance.
POST	/storage/o2cb/clusters/{name}/maintenance/enter	Safely enter O2CB maintenance mode.
POST	/storage/o2cb/clusters/{name}/maintenance/exit	Exit maintenance mode and remount datastores.

---

CLI Reference

Administrators with SSH or console access can also inspect O2CB status using standard CLI tools:

# Check cluster status in kernel memory
sudo o2cb cluster-status <cluster-name>

# Register configuration into running kernel
sudo o2cb register-cluster <cluster-name>

# Unregister cluster from kernel
sudo o2cb unregister-cluster <cluster-name>

# Inspect active nodes in kernel configfs
ls -la /sys/kernel/config/cluster/<cluster-name>/node/

# Start and stop global heartbeat service
sudo o2cb start-heartbeat <cluster-name>
sudo o2cb stop-heartbeat <cluster-name>

---

Troubleshooting & Best Practices

1. Resolving "Not in Kernel" Node Discrepancies

• Cause: A node was added to /etc/ocfs2/cluster.conf directly or while the cluster stack was offline.
• Solution: Click the Sync to Kernel button in the Vapor UI or run sudo o2cb register-cluster <cluster-name> via CLI.

2. Network Firewall Requirements

• All participating cluster nodes must be able to communicate bidirectionally over TCP port 7777. Ensure firewall rules allow traffic between all host storage interfaces:

  sudo ufw allow 7777/tcp
  # or firewalld:
  sudo firewall-cmd --permanent --add-port=7777/tcp && sudo firewall-cmd --reload

3. Fencing and Quorum Rules

• OCFS2 uses a majority quorum algorithm. If communication is lost between nodes, minority partitions will automatically self-fence (reboot) to protect data integrity. Ensure redundant network links (Bonds / Multipath) are configured on storage interfaces.