🖥️ Proxmox Cluster Update Runbook

Description:
Step‑by‑step instructions to safely update a three‑node Proxmox cluster using Ceph storage.
This runbook ensures node‑by‑node updates, VM evacuation, Ceph health validation, and post‑update checks.

🧭 Prerequisites

🧩 Cluster Overview

The cluster consists of:

Update order:

  1. pve‑2
  2. pve‑1
  3. pve‑0

Only update one node at a time.

🚦 Step 1 — Log In to the Node

SSH into the node you are updating:

ssh root@pve-2

(Replace with the appropriate node.)

🚚 Step 2 — Migrate Virtual Machines Off the Node

Migrate all VMs to another node (example: migrate to pve‑0):

for vm in $(sudo qm list | awk '{print $1}' | tail -n +2); do
    sudo qm migrate "$vm" pve-0 --online
done

Verify VM migration

📦 Step 3 — Update Package List

sudo apt-get update

⬆️ Step 4 — Upgrade Installed Packages

sudo apt-get dist-upgrade

🔁 Step 5 — Reboot the Node

sudo reboot

Wait for the node to return, then verify cluster membership:

pvecm status

Expected:

🔍 Post‑Update Validation

Perform these checks before returning VMs to the node.

🧪 Step 6 — Validate Proxmox Services

Check core services

systemctl status pvedaemon.service pveproxy.service pvestatd.service

All should be active (running).

Check cluster filesystem

systemctl status pve-cluster

Should be active (running) with no corruption warnings.

🐙 Ceph Validation

Ceph must remain healthy throughout the update process.

🩺 Step 7 — Check Ceph Cluster Health

ceph -s

Expected:

🗂️ Step 8 — Validate OSD Status on the Updated Node

ceph osd tree

Check that OSDs hosted on this node are:

You may also run:

ceph osd status

🧩 Step 9 — Validate Ceph Services on the Node

If using cephadm:

cephadm ls

Expected:

If using systemd-managed Ceph:

systemctl status ceph.target

🧮 Step 10 — Check Ceph Versions (Optional)

ceph versions

All nodes should be on the same Ceph version before proceeding to the next node.

🎛️ Step 11 — Return VMs to the Node

Once all checks pass:

📌 Notes