Kubernetes Cluster Auto-Provisioner: End-User & Administrator Guide

The Kubernetes Cluster Auto-Provisioner in Cockpit provides a native, streamlined mechanism to deploy, scale, and maintain Kubernetes clusters on top of virtualized compute and network resources. By leveraging a multi-VM architecture, lightweight k3s distributions, and automated orchestration, Cockpit abstracts the complexity of bootstrapping clusters while offering enterprise-grade customization for networking, storage, and scheduling.

---

1. Architectural Overview

Cockpit's auto-provisioner orchestrates VMs on physical hypervisors using a structured background pipeline. The system deploys control plane and worker nodes using a pre-configured OS golden image, customized on boot via cloud-init and managed post-boot using the QEMU Guest Agent.

flowchart TD
    subgraph Cockpit Management Plane
        UI[Cockpit UI Dashboard] -->|REST API / WebSockets| API[Cockpit API Engine]
        API -->|Task Runner| Tasks[Background Provisioning Task]
        DB[(PostgreSQL DB - GORM)] <-->|Cluster & Node States| API
    end

    subgraph Datastores & Storage
        DS_Shared[Shared Datastore - NFS] -->|Immediate VM Cloning| Hypervisors
        DS_Local[Local Datastore] -->|Fallback Template Caching| Hypervisors
    end

    subgraph Hypervisors & VMs
        Hypervisors -->|Clone & Deploy| VM_CP[Control Plane VM]
        Hypervisors -->|Clone & Deploy| VM_W1[Worker Node 1 VM]
        Hypervisors -->|Clone & Deploy| VM_W2[Worker Node 2 VM]
    end

    subgraph Kubernetes Cluster
        VM_CP -->|Bootstrap k3s Server| K8S_CP[Control Plane Node]
        VM_W1 -->|Join Cluster via Token| K8S_CP
        VM_W2 -->|Join Cluster via Token| K8S_CP
    end

The Cockpit backend manages all interactions with hypervisors, monitors VM status, reads configuration state from VMs via the QEMU Guest Agent, and executes cluster bootstrapping.

---

2. Infrastructure Prerequisites

Before utilizing the auto-provisioner, administrators must ensure that the virtualized environment and base templates meet the requirements outlined below.

2.1 Golden OS Image Template

To support rapid VM cloning, a standardized "Golden Image" is required.

• Base Operating System: Ubuntu 24.04 LTS (qcow2 format).
• QEMU Guest Agent: Must be preinstalled (apt-get install -y qemu-guest-agent) and set to enable at boot. The guest agent allows Cockpit to:
  • Dynamically discover VM IP addresses allocated via DHCP.
  • Retrieve authentication tokens (such as the k3s join token) securely.
  • Retrieve the generated kubeconfig without exposing SSH keys or setting up network tunnels.
• Cloud-Init: Must be enabled and configured to read metadata passed by the libvirt hypervisor.
• Auto-Growing Root Partition: The root partition must be configured to automatically resize and grow to fit the allocated disk size on first boot (utilizing growpart or similar cloud-init utilities).

2.2 Datastore Storage Pools

The template must be stored in one of the following datastores:

1. Shared Storage Pools (Recommended): Storing the golden image template on a shared datastore (e.g., NFS or clustered OCFS2) allows all physical hypervisors in the Cockpit cluster to clone and boot control plane/worker VMs immediately.
2. Local Storage Pools (Fallback Caching): If nodes are provisioned on local hypervisor storage pools, the Cockpit backend checks if the golden image file exists locally in the target host's storage pool. If it is missing, Cockpit automatically caches (copies) the template file from the primary Cockpit store to that host's local storage pool before commencing the VM clone operation.

---

3. Provisioning Wizard Step-by-Step

Administrators can launch the provisioning flow directly from the Cockpit UI:

1. Right-click the Cluster node in the left-hand inventory tree.
2. Select Kubernetes -> New Cluster in the context menu.
3. The Kubernetes Provisioning Wizard will open, guiding you through five steps.

Step	Section Name	Configurable Fields	Key Architectural Rules / Actions
1	Identity & Version	• Cluster Name • Description • Target Network • K3s Version (e.g., v1.29.2+k3s1) • CNI Network Plugin	• CNI options include Flannel (Default), Calico, and Cilium eBPF. • Target Network assigns the VMs to a virtual network bridge.
2	Control Plane Config	• CPU vCores • Memory (GB) • Storage Pool • Disk size (GB) • Node Count (1 vs 3) • Placement Strategy	• Node Count options: Single Control Plane (1 Node) or High Availability (3 Nodes). • Placement Options:   1. Auto-Distribute: Enforces anti-affinity rules using Cockpit's DRS engine, ensuring the 3 HA Master VMs are scheduled on three separate physical hosts.   2. Manual Override: Dropdowns appear to explicitly assign nodes to specific hypervisors.
3	Worker Node Pools	• Worker Count (default: 3) • CPU vCores per worker • Memory (GB) per worker • Storage Pool • Disk size (GB) • Placement Strategy	• Placement Options:   1. Auto-Distribute: Automatically spreads worker VMs across physical hosts to balance CPU/Memory utilization.   2. Manual Override: Explicitly assigns worker VMs to specific hosts (useful for node-specific hardware like GPUs).
4	Credentials & CSI Tiers	• SSH Public Key (Textarea) • CSI Storage Tier Options	• SSH Key: Authorizes root/admin access to VM nodes for troubleshooting. • CSI Storage Tiers (Multi-select options detailed in Section 5):   - Tier 1: Host Path (local-path-provisioner) [Enabled by default].   - Tier 2: NFS Shared Storage [Disabled by default].   - Tier 3: Replicated Block Storage (Longhorn) [Disabled by default].
5	Review & Provision	• Summary Dashboard	• Displays configuration, resource footprints, and physical host mapping. • Clicking Provision registers a long-running Task in Cockpit and closes the wizard.

---

4. Container Network Interface (CNI) Configurations

The choice of CNI determines the pod network routing, security policies, and performance characteristics.

CNI Plugin	Default Option	Routing Technology	Security / Network Policies	Primary Use Cases
Flannel	Yes	VXLAN / Host-gw encapsulation	None	Lightweight, local dev, single-node or low-overhead environments.
Calico	No	IP-in-IP / BGP routing	Full Kubernetes NetworkPolicies	Multi-tenant environments requiring secure isolation policies.
Cilium	No	Linux eBPF (direct routing)	Advanced L3-L7 policy enforcement & Hubble telemetry	High-performance production environments with high throughput.

CNI Bootstrapping Process

1. Flannel (Default): Cockpit configures the control plane VM with standard k3s installation flags:

   curl -sfL https://get.k3s.io | sh -s - server --disable traefik --write-kubeconfig-mode 644

2. Calico & Cilium (Custom CNIs):
   • To prevent conflicting routing rules, Cockpit boots the control plane VM using flags that disable the built-in Flannel and network policy components:

     curl -sfL https://get.k3s.io | sh -s - server --disable traefik --flannel-backend=none --disable-network-policy --write-kubeconfig-mode 644

   • The cluster initially boots in a "Network Unavailable" state. Once the API server is reachable, the Cockpit backend applies the custom Calico or Cilium manifests (Helm chart deployments or YAML resources) directly to bootstrap pod networking before worker VMs attempt to register.

---

5. Container Storage Interface (CSI) Options

Cockpit implements a multi-tier CSI selection interface allowing persistent volumes (PVs) to map back to physical hypervisor storage pools or shared infrastructure.

                  ┌────────────────────────────────────────┐
                  │          CSI Storage Options           │
                  └───────────────────┬────────────────────┘
                                      │
         ┌────────────────────────────┼────────────────────────────┐
         ▼                            ▼                            ▼
   [Tier 1: Host Path]        [Tier 2: NFS Shared]        [Tier 3: Longhorn Block]
   • Built-in local-path      • NFS CSI Driver            • Longhorn aggregates
   • Pod data saved to VM     • Connects to NFS DS         local VM disks
     root disk (host storage) • Supports ReadWriteMany     • Replicated, high-availability

5.1 Tier 1: Host Path (Enabled by Default)

• Underlying Engine: Uses the k3s built-in local-path-provisioner.
• Data Flow: PV data is written to a designated local path on the worker VM's root disk (/opt/local-path-provisioner). Since the VM's disk (.qcow2) resides on Cockpit's physical host storage pool, the storage is ultimately backed by the host's physical disk.
• Pros/Cons: High performance, zero setup. However, volumes are non-portable. If a Pod is rescheduled to a different worker VM, it cannot access the data.

5.2 Tier 2: NFS Shared Storage (Optional)

• Underlying Engine: Deploys the Kubernetes NFS CSI Driver.
• Configuration Options:
  • Use Existing NFS Datastore: Select from NFS storage pools already configured in Cockpit. Server IP and Export path details are retrieved automatically from Cockpit's database.
  • Use External NFS Server: Manually specify the NFS server IP/hostname, mount path, and credentials.
• Backend Pre-Flight Check: Before provisioning begins, the Cockpit task runner attempts a TCP connection on port 2049 (NFS port) to the specified NFS server IP from the selected virtual network bridge. If unreachable, the task halts and alerts the administrator.
• Pros/Cons: Supports ReadWriteMany (RWX) PVs. Pods can migrate dynamically across worker nodes and physical hypervisors while retaining access to the same volumes.

5.3 Tier 3: Replicated Storage - Longhorn (Optional)

• Underlying Engine: Deploys the Longhorn distributed block storage operator inside the cluster.
• Configuration: The administrator selects the target Cockpit host datastore (e.g., fast-ssd-pool) to house the worker VMs' disk files.
• Data Flow: Longhorn aggregates the local disk space of all worker VMs (which reside physically on the selected high-speed host datastore) and provides replicated block storage.
• Pros/Cons: Fully redundant, highly available storage (ReadWriteOnce). Ideal for stateful databases. If a physical host or worker VM goes down, volumes remain accessible on other nodes.

---

6. Accessing the Cluster: Kubeconfig Setup

Once cluster status shifts to active, the administrative kubeconfig becomes available for download.

6.1 Dynamic Endpoint Rewrite

When a user requests the kubeconfig via GET /api/v1/kubernetes/clusters/:id/kubeconfig, the Cockpit backend executes the following post-processing pipeline:

1. Reads the raw generated file from /etc/rancher/k3s/k3s.yaml on the Control Plane VM using the QEMU Guest Agent.
2. The default file points to the local loopback address: server: https://127.0.0.1:6443.
3. Cockpit dynamically rewrites 127.0.0.1 (or localhost) with the control plane VM's external physical DHCP IP address (e.g. server: https://192.168.122.155:6443).
4. Serves the payload as a downloadable attachment:
   • Content-Type: application/x-yaml
   • Filename: kubeconfig-{cluster_name}.yaml

6.2 Step-by-Step Connection Instructions

To connect to your new cluster using the kubectl CLI from your local system:

1. Download the Kubeconfig: Click Download Kubeconfig from the cluster details page in the Cockpit UI.

2. Move and Restrict Permissions: Move the downloaded file to your local machine's config folder and restrict access permissions (required by kubectl):

   mkdir -p ~/.kube
   mv ~/Downloads/kubeconfig-k8s-prod.yaml ~/.kube/config-k8s-prod
   chmod 600 ~/.kube/config-k8s-prod

3. Set the KUBECONFIG Environment Variable: Point your active terminal to the newly downloaded configuration:

   export KUBECONFIG=~/.kube/config-k8s-prod

   TIP

   To make this configuration persistent, append the export statement to your shell profile file (e.g., ~/.bashrc or ~/.zshrc).

4. Verify Connectivity: Test the connection by querying the API server and checking node availability:

   kubectl cluster-info
   kubectl get nodes -o wide

---

7. Scaling and Modifying Worker Pools

Cockpit allows users to dynamically scale the worker node pool up or down without rebuilding the cluster or control plane.

                          Scale Pool Request Received
                                      │
                                      ▼
                           Calculate Node Delta
                                      │
                   ┌──────────────────┴──────────────────┐
                   ▼ (Delta > 0)                         ▼ (Delta < 0)
              [Scale Up]                            [Scale Down]
                   │                                     │
         Fetch Join Token from CP VM                     │
         via QEMU Guest Agent                            │
                   │                                     │
         Clone new Worker VMs from                       │
         OS Golden Template                              │
                   │                                     │
         Inject Cloud-Init Join Command                  │
         using CP IP and Join Token                      │
                   │                                     │
         Boot VMs & Wait for DHCP                      Drain Node Gracefully
         IP Discovery                                  (kubectl drain)
                   │                                     │
         Wait for K8s Node Status                      Delete Worker VMs
         to show "Ready"                                 │
                   │                                     │
         Save Nodes to GORM DB                         Update DB Records
                   │                                     │
                   └──────────────────┬──────────────────┘
                                      ▼
                             Task Completed

7.1 Scaling Up

1. Delta Calculation: The backend compares the requested count with the current desired_count configured in the K8sNodePool database.
2. Token Collection: Cockpit retrieves the current cluster joining secret by reading /var/lib/rancher/k3s/server/node-token from the Control Plane VM using the QEMU Guest Agent.
3. VM Cloning: The task runner clones additional worker VM instances from the OS template.
4. Cloud-Init Setup: The new VM user-data is populated with the join configuration:

   #cloud-config
   runcmd:
     - systemctl enable --now qemu-guest-agent
     - curl -sfL https://get.k3s.io | K3S_URL=https://<CONTROL_PLANE_IP>:6443 K3S_TOKEN=<JOIN_TOKEN> sh -

5. Boot & Registration: Once booted, the VMs obtain DHCP IPs, which are discovered via the QEMU Guest Agent. Cockpit monitors the Kubernetes cluster API until the new nodes appear and report Ready.

7.2 Scaling Down

1. Node Selection: Cockpit selects the excess worker VMs (preferring nodes with the lowest uptime or highest index).
2. Graceful Draining: Cockpit executes a cluster drain command (equivalent to kubectl drain <node-name> --delete-emptydir-data --ignore-daemonsets --force) via the control plane to migrate workloads.
3. Tear Down: Once drained, Cockpit stops and deletes the target virtual machines, frees up hypervisor resources, and deletes their respective records from the K8sNode database table.

---

8. Dynamic Version Selection & Upgrades

Cockpit implements zero-downtime upgrades for Kubernetes clusters using Rancher's System Upgrade Controller operator.

8.1 Dynamic Version Selection

Rather than relying on a hardcoded version list, the Cockpit UI dynamically queries the backend endpoint GET /api/v1/kubernetes/k3s-versions.

• Dynamic Retrieval: The backend queries the GitHub Releases API to retrieve active, stable k3s tags.
• Fallback List: If the hypervisor plane is air-gapped or GitHub APIs are rate-limited, Cockpit falls back to a curated embedded list of K3s releases starting from v1.36.1+k3s1 down to v1.31.0+k3s1.

8.2 System Upgrade Controller Setup

During the initial cluster bootstrap, Cockpit installs the system-upgrade-controller deployment in the cluster. This controller watches for custom Plan resources to drive upgrades.

8.3 The Upgrade Pipeline

When an administrator selects a newer k3s version in the UI (e.g. upgrading from v1.28.7+k3s1 to v1.29.2+k3s1):

1. Triggering the Upgrade: Cockpit sends a POST /api/v1/kubernetes/clusters/:id/upgrade request.
2. Plan Deployment: The Cockpit backend applies a custom Plan manifest to the cluster targeting the new version.

   apiVersion: upgrade.cattle.io/v1
   kind: Plan
   metadata:
     name: k3s-server
     namespace: system-upgrade
   spec:
     concurrency: 1
     version: v1.29.2+k3s1
     nodeSelector:
       matchExpressions:
         - key: node-role.kubernetes.io/master
           operator: In
           values:
             - "true"
     serviceAccountName: system-upgrade
     upgrade:
       image: rancher/k3s-upgrade

3. Sequential Execution:
   • The System Upgrade Controller selects one node at a time (beginning with the control plane nodes, then worker nodes).
   • It drains the workloads off the target node.
   • It updates the k3s binary and configuration files inside the VM.
   • It restarts the service, monitors health check statuses, and uncordons the node.
   • Once the node returns to a Ready state, the controller proceeds to the next node in the sequence.
4. Monitoring: The Cockpit backend queries the status of the Plan custom resources and displays a rolling progress bar in the Task panel.

---

9. Troubleshooting & Common Issues

9.1 VM Stuck in "Creating" or "Booting" State

• Likely Cause: The OS golden image template is missing from the target host's local datastore, or network copy speed is slow.
• Resolution: Check the Cockpit tasks logs for image copying events. If copying fails, verify that host-to-host replication permissions and network bridges are active.

9.2 Nodes Booted but Cannot Join Cluster (CNI Unavailable)

• Likely Cause: Custom CNI installation manifests (Calico/Cilium) failed to deploy on the control plane, preventing network initialization.
• Resolution: Access the Control Plane VM's console or terminal and verify that the API server is up. Ensure the VM has internet access to retrieve custom CNI images if utilizing external registries.

9.3 QEMU Guest Agent Unreachable

• Likely Cause: The golden image did not have the QEMU Guest Agent daemon started, or the daemon is crashing on startup.
• Resolution: Open the VM Console via Cockpit's noVNC panel, log in, and ensure the agent service is active:

  systemctl status qemu-guest-agent

  If not installed, install it manually and reboot:

  sudo apt-get install -y qemu-guest-agent && sudo systemctl enable --now qemu-guest-agent