You need to do maintenance on a node. What should you do first?

The correct first step for planned maintenance on a compute node is to place it in the DRAIN state using the scontrol update command. This is the standard, non-disruptive procedure in Slurm. The DRAIN state prevents new jobs from being allocated to the node while allowing any currently running jobs to complete. Once all jobs have finished, the node's state transitions to IDLE+DRAIN, indicating it is safe to take offline for maintenance without interrupting user workloads.

B/C. Setting the node state to down is an abrupt action typically reserved for unexpected node failures. It can cause running jobs to be terminated and requeued, which is disruptive and not ideal for planned maintenance.

D. Disabling job scheduling on all compute nodes is excessive and unnecessary for maintenance on a single node. This action would halt the productivity of the entire cluster.

1. NVIDIA DGX SuperPOD with NVIDIA DGX A100 Deployment Guide (v1.1)

Section 5.1.1. Taking Nodes Offline for Maintenance: "To take a node offline for maintenance, the best practice is to drain the node. Draining a node prevents new jobs from being scheduled on the node but allows currently running jobs to complete. To drain a node, run the following command: scontrol update nodename= state=drain reason="maintenance""

2. SchedMD (Official Slurm Documentation), scontrol Man Page

Section: update command, NodeName parameter: "State=DRAIN - Indicate that a node is unavailable for use. No new jobs will be allocated to the node. Jobs already running on the node will be allowed to complete. The node state will be changed to idle when all of the jobs on it have completed." This contrasts with State=DOWN, which is described as for a node that is "unavailable for use" and often set automatically when a node fails.