Wiki cluster CEAB (antic cblabdoc)

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
# Check system load

### 💡 Tip: to get a quick view of free resources per node (CPU and memory)

(this is not a standard **slurm** function; it is an *ad-hoc* function implemented in /usr/local/bin)

```
$sfreeres
```

You’ll see something like:

```
                CPU        Mem (GiB)      GPU   
Node        (Free/Ttl)    (Free/Ttl)  (Free/Ttl)
------------------------------------------------
node1          16/16        153/153       0/0   
node2          16/16        153/153       0/0   
node3          16/16        153/153       0/0   
node4          16/16        153/153       0/0   
node5          16/16        153/153       0/0   
node6           5/8          45/60        0/0   
node7           5/8          46/60        0/0   
node8          10/10         34/91        0/0   
node9           2/10         39/91        0/0   
node10          5/20        179/185       0/0   
node11          5/20        161/185       0/0   
node12          5/20        180/185       0/0   
node13          5/20        160/185       0/0   
node14          5/20        177/185       0/0   
node15          5/20        177/185       0/0   
node16          5/20        183/185       0/0   
node17         32/32        497/497       0/0
```

### ⚙️ To see current active jobs

```
squeue
```

This shows job ID, partition, job name, user, state, time, number of nodes, and the node list.

```
        JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
        26152      ceab mosquito j.palmer PD       0:00      1 (Resources)
        26252      ceab mosquito j.palmer PD       0:00      1 (Priority)
        26301      ceab 986ff116 guille.s PD       0:00      1 (Priority)
26298_[1-5%5]      ceab update_m j.palmer PD       0:00      1 (Priority)
        26291      ceab    Sens6  m.pardo  R      26:21      1 node9
      26289_5      ceab update_m j.palmer  R      31:51      1 node9
        26195      ceab     bash roman.sa  R    3:09:51      1 node9
        25990      ceab julia_Ha  m.pardo  R 2-05:07:34      1 node9
        25991      ceab julia_Ha  m.pardo  R 2-05:07:34      1 node9
        25992      ceab julia_Ha  m.pardo  R 2-05:07:34      1 node9
        25637      ceab   austr7 jramoned  R 6-04:15:32      1 node16
        25633      ceab   austr3 jramoned  R 6-04:19:15      1 node12
        25632      ceab   austr2 jramoned  R 6-04:20:29      1 node11
        25631      ceab   austr1 jramoned  R 6-04:25:51      1 node10
        25549      ceab julia_Ha  m.pardo  R 6-21:47:41      1 node9
        22348      ceab arunc1.s xavier.r  R 35-00:23:38      1 node6
        22350      ceab arunc3.s xavier.r  R 35-00:23:38      1 node7
```


### 🧩 Quick overview: all partitions and their states

```
sinfo
```

You’ll see something like:

```
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
all          up   infinite      1  drain node8
all          up   infinite     10    mix node[6-7,9-16]
all          up   infinite      6   idle node[1-5,17]
ceab*        up   infinite      1  drain node8
ceab*        up   infinite     10    mix node[6-7,9-16]
medium       up   infinite      5   idle node[1-5]
mpi          up   infinite      7    mix node[10-16]
amd          up   infinite      1   idle node17
```

### 🧩 Quick overview: all nodes and their states

```
sinfo -N -l
```

**Explanation:**

* `-N`: show info by **node** instead of by partition.
* `-l`: long format (shows state, CPUs, memory, etc.).

You’ll see something like:

```
Thu Nov 13 12:21:48 2025
NODELIST   NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
node1          1       all        idle 16      2:8:1 157000        0      1   (null) none                
node1          1    medium        idle 16      2:8:1 157000        0      1   (null) none                
node2          1       all        idle 16      2:8:1 157000        0      1   (null) none                
node2          1    medium        idle 16      2:8:1 157000        0      1   (null) none                
node3          1       all        idle 16      2:8:1 157000        0      1   (null) none                
node3          1    medium        idle 16      2:8:1 157000        0      1   (null) none                
...
node8          1     ceab*     drained 10     1:10:1  93000        0      1   (null) Kill task failed    
node8          1       all     drained 10     1:10:1  93000        0      1   (null) Kill task failed    
node9          1     ceab*      mixed- 10     1:10:1  93000        0      1   (null) none                
node9          1       all      mixed- 10     1:10:1  93000        0      1   (null) none                
node10         1     ceab*       mixed 20     2:10:1 189000        0      1   (null) none                
node10         1       mpi       mixed 20     2:10:1 189000        0      1   (null) none                
node10         1       all       mixed 20     2:10:1 189000        0      1   (null) none                
node11         1     ceab*       mixed 20     2:10:1 189000        0      1   (null) none                
node11         1       all       mixed 20     2:10:1 189000        0      1   (null) none                
node11         1       mpi       mixed 20     2:10:1 189000        0      1   (null) none                
...
node16         1     ceab*       mixed 20     2:10:1 189000        0      1   (null) none                
node16         1       all       mixed 20     2:10:1 189000        0      1   (null) none                
node16         1       mpi       mixed 20     2:10:1 189000        0      1   (null) none                
node17         1       amd        idle 32      4:8:1 509000        0      1   (null) none                
node17         1       all        idle 32      4:8:1 509000        0      1   (null) none                
```

### 📊 Occupancy per node (CPU and memory usage)

```
scontrol show node node17
```

You’ll see something like:

```
NodeName=node17 Arch=x86_64 CoresPerSocket=8 
   CPUAlloc=0 CPUEfctv=32 CPUTot=32 CPULoad=0.00
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=node17 NodeHostName=node17 Version=24.11.5
   OS=Linux 5.14.0-570.23.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 26 19:29:53 UTC 2025 
   RealMemory=509000 AllocMem=0 FreeMem=512962 Sockets=4 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=all,amd 
   BootTime=2025-10-09T15:09:32 SlurmdStartTime=2025-10-09T15:09:59
   LastBusyTime=2025-10-09T15:20:55 ResumeAfterTime=None
   CfgTRES=cpu=32,mem=509000M,billing=32
   AllocTRES=
   CurrentWatts=0 AveWatts=0
   
```

**Key fields:**

* `CPUAlloc`, `CPUTot` — allocated vs. total CPUs
* `AllocMem`, `RealMemory` — allocated vs. total memory
* `Gres` — GPU allocation (if applicable)


Or for all nodes:

```
scontrol show node
```

**scontrol show node** prints a lot of info, but you can extract only the fields you need using standard command-line tools like *grep* or *awk*

### 🧮 Compact summary: CPUs in use vs available

```
sinfo -o "%N %c %C"
```

**Columns explained:**

* `%N` — Node name
* `%c` — Total CPUs
* `%C` — CPU state summary (`alloc/idle/other/total`)

Example output:

```
NODELIST     CPUS   CPU_STATE
node001      32     16/16/0/32
node002      32     0/32/0/32
```


### 🧠 GPU occupancy (if applicable)

If your cluster has GPUs, you can check:

```
sinfo -o "%N %G"
```

or detailed usage with:

```
scontrol show node | grep -A10 Gres
```