Slurm basics :
A Slurm Batch is a file in which are defined the different elements necessary for the creation of a Job. It describes a request for resource allocation for the execution of one or more processes, and mainly contains the following information (not exhaustive) :
- The parameters of the Job and the requested resources (Job Name, tracking e-mail, CPUs, RAM ...) via #SBATCH options
- The processing to perform (shell script).
The batch is transmitted to Slurm via the "sbatch" command which, unless error or refusal, creates a Job and places it in the queue.
A Job is an allocation of resources (CPUs, RAM, time ...) reserved for the execution of a specific treatment.
- The allocation is defined in the Batch in number of Tasks (ntasks) multiplied by the number of CPUs per Task (cpus-per-task) and corresponds to the maximum resources usable in parallel.
- The script creates one or more Job Steps and manages the distribution of Tasks on the Compute nodes.
A Job Step represents a step or section of the processing performed by the Job. It executes one or more Tasks via the "srun" command. This division into Job Steps offers great flexibility in the organization of Job steps and the management and analysis of allocated resources :
- Steps can be executed sequentially or in parallel,
- a Step can initiate one or more Tasks executed in parallel,
- Steps are supported by sstat / sacct commands, allowing both tracking of the Step-by-Step progress of the Job during execution, and detailed resource usage statistics for each Step (during and after after execution).
A Task is a process to which the resources defined in the Batch are allocated by the option "--cpus-per-task". A Task can have these resources like any other process (creation of threads, sub-processes, possibly themselves multi-threaded).
This is the resource allocation unit of the Job. CPUs not used by a Task will be "lost", not usable by any other Task or Step. If the Task creates more processes / threads than allocated CPUs, these threads will share the allocation.
To better understand these principles, let's look at a batch example.