The processing is "arbitrarily" cut into logical sections (Steps) executed successively for the following reasons :
- follow the step-by-step progress of the job during its execution,
- analyze the use of the resources of each Step (CPU, RAM, Disk ...) post-execution. Only one Task is executed at a time.
#SBATCH --job-name=SlurmJob-%j #SBATCH --output=SlurmJob-%j.out #SBATCH --ntasks=1 #SBATCH --cpus-per-task=4 echo "Job START - $(date)" >"SlurmJob-$SLURM_JOB_ID.log"
In the case of a single task, it is not necessary to specify -n / -N, the default value being the total number of Tasks, namely 1!
srun job-step-1.sh 'param1' 'param2' srun job-step-2.sh 'param' srun job-step-3.sh echo "Job END - $(date)" >>"SlurmJob-$SLURM_JOB_ID.log"
Perform the same or similar processing on a large number of files in parallel.
The total number of Tasks (--ntasks) is the number of files to process.
These files are here stored in a given directory.
SBATCH --job-name=SlurmJob-%j #SBATCH --output=SlurmJob-%j.out #SBATCH --ntasks=10 #SBATCH --cpus-per-task=4 # Parameter: # source file directory to process path='~/path/to/files' # Script executed for each Task script='slurm-task.sh' # Prologue echo "Job START - $(date)" >"SlurmJob-$SLURM_JOB_ID.log" # process: while read f; do # for each file, execution of one Step in a # child process with an allocation of one Task # and here a single node. The number of CPUs per Task is # here pssed as parameter of the script, that allows a non static # number of threads for this processing. srun -n1 -N1 "$script" "$f" $SLURM_CPUS_PER_TASK 'param1' 'param2' & # => Slurm will allocate 4 CPUs for the execution of: # slurm-task.sh 4 'param1' 'param2' done < <(ls "$path") # Wait the end of the execution of the childs processes ! wait # Epilogue echo "Job END - $(date)" >>"SlurmJob-$SLURM_JOB_ID.log"