# Check system load ### 💡 Tip: to get a quick view of free resources per node (CPU and memory) (this is not a standard **slurm** function; it is an *ad-hoc* function implemented in /usr/local/bin) ``` $sfreeres ``` You’ll see something like: ``` CPU Mem (GiB) GPU Node (Free/Ttl) (Free/Ttl) (Free/Ttl) ------------------------------------------------ node1 16/16 153/153 0/0 node2 16/16 153/153 0/0 node3 16/16 153/153 0/0 node4 16/16 153/153 0/0 node5 16/16 153/153 0/0 node6 5/8 45/60 0/0 node7 5/8 46/60 0/0 node8 10/10 34/91 0/0 node9 2/10 39/91 0/0 node10 5/20 179/185 0/0 node11 5/20 161/185 0/0 node12 5/20 180/185 0/0 node13 5/20 160/185 0/0 node14 5/20 177/185 0/0 node15 5/20 177/185 0/0 node16 5/20 183/185 0/0 node17 32/32 497/497 0/0 ``` ### ⚙️ To see current active jobs ``` squeue ``` This shows job ID, partition, job name, user, state, time, number of nodes, and the node list. ``` JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 26152 ceab mosquito j.palmer PD 0:00 1 (Resources) 26252 ceab mosquito j.palmer PD 0:00 1 (Priority) 26301 ceab 986ff116 guille.s PD 0:00 1 (Priority) 26298_[1-5%5] ceab update_m j.palmer PD 0:00 1 (Priority) 26291 ceab Sens6 m.pardo R 26:21 1 node9 26289_5 ceab update_m j.palmer R 31:51 1 node9 26195 ceab bash roman.sa R 3:09:51 1 node9 25990 ceab julia_Ha m.pardo R 2-05:07:34 1 node9 25991 ceab julia_Ha m.pardo R 2-05:07:34 1 node9 25992 ceab julia_Ha m.pardo R 2-05:07:34 1 node9 25637 ceab austr7 jramoned R 6-04:15:32 1 node16 25633 ceab austr3 jramoned R 6-04:19:15 1 node12 25632 ceab austr2 jramoned R 6-04:20:29 1 node11 25631 ceab austr1 jramoned R 6-04:25:51 1 node10 25549 ceab julia_Ha m.pardo R 6-21:47:41 1 node9 22348 ceab arunc1.s xavier.r R 35-00:23:38 1 node6 22350 ceab arunc3.s xavier.r R 35-00:23:38 1 node7 ``` ### 🧩 Quick overview: all partitions and their states ``` sinfo ``` You’ll see something like: ``` PARTITION AVAIL TIMELIMIT NODES STATE NODELIST all up infinite 1 drain node8 all up infinite 10 mix node[6-7,9-16] all up infinite 6 idle node[1-5,17] ceab* up infinite 1 drain node8 ceab* up infinite 10 mix node[6-7,9-16] medium up infinite 5 idle node[1-5] mpi up infinite 7 mix node[10-16] amd up infinite 1 idle node17 ``` ### 🧩 Quick overview: all nodes and their states ``` sinfo -N -l ``` **Explanation:** * `-N`: show info by **node** instead of by partition. * `-l`: long format (shows state, CPUs, memory, etc.). You’ll see something like: ``` Thu Nov 13 12:21:48 2025 NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON node1 1 all idle 16 2:8:1 157000 0 1 (null) none node1 1 medium idle 16 2:8:1 157000 0 1 (null) none node2 1 all idle 16 2:8:1 157000 0 1 (null) none node2 1 medium idle 16 2:8:1 157000 0 1 (null) none node3 1 all idle 16 2:8:1 157000 0 1 (null) none node3 1 medium idle 16 2:8:1 157000 0 1 (null) none ... node8 1 ceab* drained 10 1:10:1 93000 0 1 (null) Kill task failed node8 1 all drained 10 1:10:1 93000 0 1 (null) Kill task failed node9 1 ceab* mixed- 10 1:10:1 93000 0 1 (null) none node9 1 all mixed- 10 1:10:1 93000 0 1 (null) none node10 1 ceab* mixed 20 2:10:1 189000 0 1 (null) none node10 1 mpi mixed 20 2:10:1 189000 0 1 (null) none node10 1 all mixed 20 2:10:1 189000 0 1 (null) none node11 1 ceab* mixed 20 2:10:1 189000 0 1 (null) none node11 1 all mixed 20 2:10:1 189000 0 1 (null) none node11 1 mpi mixed 20 2:10:1 189000 0 1 (null) none ... node16 1 ceab* mixed 20 2:10:1 189000 0 1 (null) none node16 1 all mixed 20 2:10:1 189000 0 1 (null) none node16 1 mpi mixed 20 2:10:1 189000 0 1 (null) none node17 1 amd idle 32 4:8:1 509000 0 1 (null) none node17 1 all idle 32 4:8:1 509000 0 1 (null) none ``` ### 📊 Occupancy per node (CPU and memory usage) ``` scontrol show node node17 ``` You’ll see something like: ``` NodeName=node17 Arch=x86_64 CoresPerSocket=8 CPUAlloc=0 CPUEfctv=32 CPUTot=32 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=node17 NodeHostName=node17 Version=24.11.5 OS=Linux 5.14.0-570.23.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 26 19:29:53 UTC 2025 RealMemory=509000 AllocMem=0 FreeMem=512962 Sockets=4 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=all,amd BootTime=2025-10-09T15:09:32 SlurmdStartTime=2025-10-09T15:09:59 LastBusyTime=2025-10-09T15:20:55 ResumeAfterTime=None CfgTRES=cpu=32,mem=509000M,billing=32 AllocTRES= CurrentWatts=0 AveWatts=0 ``` **Key fields:** * `CPUAlloc`, `CPUTot` — allocated vs. total CPUs * `AllocMem`, `RealMemory` — allocated vs. total memory * `Gres` — GPU allocation (if applicable) Or for all nodes: ``` scontrol show node ``` **scontrol show node** prints a lot of info, but you can extract only the fields you need using standard command-line tools like *grep* or *awk* ### 🧮 Compact summary: CPUs in use vs available ``` sinfo -o "%N %c %C" ``` **Columns explained:** * `%N` — Node name * `%c` — Total CPUs * `%C` — CPU state summary (`alloc/idle/other/total`) Example output: ``` NODELIST CPUS CPU_STATE node001 32 16/16/0/32 node002 32 0/32/0/32 ``` ### 🧠 GPU occupancy (if applicable) If your cluster has GPUs, you can check: ``` sinfo -o "%N %G" ``` or detailed usage with: ``` scontrol show node | grep -A10 Gres ```