Slurm Use case

  1. 1st use case
  2. 2nd use case

1st use case :

The processing is "arbitrarily" cut into logical sections (Steps) executed successively for the following reasons :

- follow the step-by-step progress of the job during its execution,

and/or

- analyze the use of the resources of each Step (CPU, RAM, Disk ...) post-execution. Only one Task is executed at a time.

#SBATCH --job-name=SlurmJob-%j 
#SBATCH --output=SlurmJob-%j.out 

#SBATCH --ntasks=1 
#SBATCH --cpus-per-task=4 

echo "Job START - $(date)" >"SlurmJob-$SLURM_JOB_ID.log"

In the case of a single task, it is not necessary to specify -n / -N, the default value being the total number of Tasks, namely 1!

srun job-step-1.sh 'param1' 'param2' 
srun job-step-2.sh 'param' 
srun job-step-3.sh 

echo "Job END - $(date)" >>"SlurmJob-$SLURM_JOB_ID.log" 

2nd use case :

Perform the same or similar processing on a large number of files in parallel.

The total number of Tasks (--ntasks) is the number of files to process.

These files are here stored in a given directory.

SBATCH --job-name=SlurmJob-%j 
#SBATCH --output=SlurmJob-%j.out 

#SBATCH --ntasks=10 
#SBATCH --cpus-per-task=4 

# Parameter: 
# source file directory to process 
path='~/path/to/files'
 
# Script executed for each Task 
script='slurm-task.sh' 


# Prologue 
echo "Job START - $(date)" >"SlurmJob-$SLURM_JOB_ID.log" 

# process: 

while read f; do 
    # for each file, execution of one Step in a 
    # child process with an allocation of one Task
    # and here a single node. The number of CPUs per Task is
    # here pssed as parameter of the script, that allows a non static
    # number of threads for this processing.

    srun -n1 -N1 "$script" "$f" $SLURM_CPUS_PER_TASK 'param1' 'param2' & 

    # => Slurm will allocate 4 CPUs for the execution of: 
    #     slurm-task.sh  4 'param1' 'param2' 

done < <(ls "$path") 

# Wait the end of the execution of the childs processes ! 
wait 

# Epilogue 
echo "Job END - $(date)" >>"SlurmJob-$SLURM_JOB_ID.log"