When troubleshooting Slurm job scheduling issues, a common source of problems is jobs getting stuck in a pending state indefinitely. Which Slurm…

The scontrol command is the primary administrative tool for viewing and modifying the state of Slurm jobs, partitions, and nodes. To diagnose a job stuck in a pending state, the command scontrol show job provides the most comprehensive output. This includes a dedicated Reason field that explicitly states why the scheduler has not yet allocated resources to the job (e.g., (Resources), (Priority), (Dependency)). This level of detail is crucial for identifying the specific cause of the scheduling delay, making it the correct tool for in-depth troubleshooting as described in the scenario.

B. sacct: This command is used for querying the Slurm accounting database to report on past or running jobs, not for detailed, real-time diagnosis of why a pending job is not starting.

C. sinfo: This command provides a summary of the state of the cluster's nodes and partitions. It does not provide information about individual jobs or the specific reasons for their pending state.

1. SchedMD Official Slurm Documentation: The man page for scontrol describes its function. The show job option is detailed as the method to view comprehensive information about a job, including its state and the reason for that state.

Source: scontrol(1) man page, section show . Available at: https://slurm.schedmd.com/scontrol.html

2. Princeton University Research Computing Documentation: In the "Inspecting Jobs" guide, it explicitly states: "To find out why a job is pending, use the scontrol show job command and look for the JobState and Reason fields." This aligns with the use case in the question.

Source: Princeton University, Research Computing, "Inspecting Jobs". Available at: https://researchcomputing.princeton.edu/support/knowledge-base/slurm#InspectingJobs

3. Lawrence Livermore National Laboratory (LLNL) Slurm Documentation: LLNL's tutorials, which are highly authoritative as they are a primary developer of Slurm, differentiate the commands. They explain that while squeue shows a brief reason, scontrol show job is used for a "full, detailed listing of the job's characteristics," which is necessary for deep troubleshooting.

Source: LLNL Computing, "Slurm Tutorial: squeue". Available at: https://hpc.llnl.gov/documentation/tutorials/slurms-squeue-command