Why VMs freeze when a node goes down
The Proxmox Cluster File System (pmxcfs) requires a majority of nodes to be online to permit write operations. A 3-node cluster needs at least 2 online — losing one node means losing the majority, and pmxcfs will:
- Block all write operations
- Freeze or suspend running VMs
- Refuse to create or delete VMs
Symptoms
pvecm status
# Quorum information
# ------------------
# Date: Mon Jun 10 10:30:15 2026
# Quorum provider: corosync_votequorum
# Nodes: 1
# Node votes: 1
# Expected votes: 2
# Total votes: 1
# Quorum: 2 ← needs 2 but only has 1
# Flags:
#
# Membership information
# Status: not in quorum
Recovery options
Option 1: Bring the node back online (preferred)
If the other node was simply rebooted or had a temporary network issue — just wait for it to come back.
Option 2: Force local quorum (emergency)
Only use this when the other node is permanently dead:
# WARNING: run only on the ONLINE node
pvecm expected 1
# Verify
pvecm status
# Quorum: 1 (met!)
Option 3: Remove the dead node from the cluster
# From the surviving node
pvecm delnode <dead-node-name>
# Verify cluster view
pvecm nodes
Prevention
1. Use an odd number of nodes
3 or 5 nodes rather than 2 or 4. A 3-node cluster survives losing 1 node. A 2-node cluster loses quorum the moment one node goes down.
2. Add a QDevice for a 2-node cluster
If you only have 2 nodes, use a lightweight third machine (Raspberry Pi, VM, VPS) as a quorum device:
# On the QDevice host, install corosync-qnetd
apt install corosync-qnetd
systemctl enable --now corosync-qnetd
# From any Proxmox node
pvecm qdevice setup <qdevice-host-ip>
QDevice adds 1 vote to the cluster, enough for one of the two nodes to win the majority when the other is down.
3. Monitor quorum with alerts
# Quorum check script — add to cron every minute
#!/bin/bash
STATUS=$(pvecm status 2>/dev/null | grep "Quorum:" | awk '{print $2}')
EXPECTED=$(pvecm status 2>/dev/null | grep "Expected votes:" | awk '{print $3}')
TOTAL=$(pvecm status 2>/dev/null | grep "Total votes:" | awk '{print $3}')
if [ "$TOTAL" -lt "$EXPECTED" ]; then
echo "ALERT: Proxmox cluster quorum degraded! Total=$TOTAL Expected=$EXPECTED" \
| mail -s "[ALERT] Proxmox Quorum" [email protected]
fi
Or use Proxmox's built-in notifications (Datacenter → Notifications) with the cluster_quorum metric.