Running HPC Jobs with Bash and SLURM | 2026 HPC Training Series #4
Transcript
So everyone, welcome to Research Computing Workshop Series Part 4, scheduling your first HPC jobs with Bash. I'm teaching the class today along with my colleague Al from the HPC team. And objectives for today, it is not a Bash coding class.
We will specifically focus on Bash for Slurm, and Slurm is our HPC job controller. Hopefully, by the end of the class, you will be able to run, view, and analyze Slurm jobs, read and modify Slurm Bash scripts with confidence, request compute resources correctly, and debug the failed jobs. And if you would like to follow along but don't feel pressured, you can SSH or log in to our Insomnia cluster.
It's an HPC cluster at Columbia. I think all of you should have access, but again, don't worry if you don't have access. We can figure it out later.
But basically, I'm going to log in with you, and in order to get in, you would just issue an SSH command with your uni, and then insomniarcscolumbia.edu. So I already have some SSH keys set up, but you would be prompted for the password, and you would be prompted for Duo entry. I'm just going to give everybody a minute to catch up. Again, if you're having issues, please, I think it's best if you just post something in the chat, and one of my co-workers will help you.
If you would like to follow along with the exercise, you can also copy over some files that we already have in place. I have the information here on the screen. You know, make a new directory.
You can name it anything you want. But, you know, for instance, bash workshop, and then I'm going to get into the directory. I'm going to copy over some files from the storage directory that we have on the cluster.
Path is a little bit convoluted. So if you remember from previous Linux class, if you're not familiar with Linux, we're just copying everything from the directory and into the current place, and we're going to do a long listing, so we have a few files here now that we will use a little later. And just a quick recap.
What is bash? It is a shell for Linux. It is the default scripting language on essentially every Linux system, although on your Macs you might have zshell, and like any programming language, it has control flow built in, so it has loops, conditionals, variables, all of that. It is not a prerequisite for this class to know bash well.
You don't need it in order to submit Slurm jobs effectively. And again, Slurm, it is our HPC job scheduler that we use in Columbia. It is open source, it is full tolerant, highly scalable, it is used in a lot of top 500 companies.
Last time I checked, it was like 60 plus percent. It manages the computer resources for you, like CPUs, GPUs, memory, schedules and monitors job queues, and it acts as the cluster's work manager. I'm going to skip ahead a little bit, and basically how you would examine the job queue is the SQ command, so here we see everything that is currently running in the cluster when we issue the SQ command.
A little bit more to elaborate. Slurm allocates resources, it does the job management, it does the fair scheduling, and it's designed for large-scale workloads, and no Linux kernel modification is required. So, some of the commands that are very useful to learn and we will work with are srun, to do a run of the job interactively, as batch, to submit a batch script for later execution, SQ, as we just saw, to view status of the jobs in the queue, as info, shows the status of partitions and nodes, cancel, to cancel the jobs, and saft, examine jobs efficiently.
And I know I'm going fast, so if you are completely new to this, I know it might be a lot of information, but just bear with me, and, again, you can always just examine all of this on your own. I believe that you will have access to the cluster for another week or two. So, you know, like, you can play with it a little more.
And so, let's do a hands-on exercise and just, like, quickly submit an srun, a job with srun. What is the advantage of the interactive jobs versus as batch? So, interactive job lets you run application on a compute node if you were just on a regular login node, but with dedicated resources, and it lets you test things pretty quickly. Like, yeah, like we said, good for quick tests and debugging.
And if you would like to follow along, you can definitely run the srun command with me. So, srun and PTY means on the terminal, dash T says for how long you would like your script to run, and generally, on our clusters, you need to specify the partition. In our case, it's free, and I'm just going to do a bash shell.
And so, we do have quite a few free users on the cluster. Therefore, you might be waiting for a little bit to get your resources allocated, but in the end, the job is submitted. And many ways you can examine it, but, for instance, you can do sqme command, and you see your job running here.
And, of course, this is just for demo purposes. My job isn't doing much. It just opened the bash shell, but generally, you know, like, you would be doing actual tasks.
And, yeah, a little bit more information here. You know, sq has flags to examine the job itself, and what I just did with sq and seeing all of my jobs, and my job ran out of time because I only requested one minute. So, I got kicked off.
But, you know, perhaps we want more information about the job, and s account is a useful command that I mentioned before. S account with the dash J flag and the job number lets me examine more information about the job. Like, where it ran, if there were errors, exit codes, what partition, et cetera, et cetera.
Very, very useful. And for all of our users, we recommend using the SAF command, which basically examines the efficiency of jobs, and the problem that we see quite often is that users over request the resources, memory, CPU, unfortunately, GPUs also that are very expensive. So, it is very useful to see if your job was efficient at running, and going to examine the stats on my job with the SAF command.
And, again, because it was, you know, just like a demo job, the efficiency was pretty poor with the CPU, with the memory. And if your jobs are approaching zero percent, you maybe want to reexamine how many resources you are requesting on the cluster. And, of course, in real time, you can always examine it with the built in Linux PS command with some flags.
And, again, with like the SAF command. In order to stop a job, ask cancel is a good command. Or, you know, like you can just kill it directly, but with the Linux kill command, but more efficient to use a built in SLURM ask cancel command.
Ask control, as we mentioned, one of the very useful SLURM commands, and we can run it also on my previous command. It will give you stats about run times, like start, when it ended, because it's not always, I guess, obvious how long it took, and the amount of resources. So, show job.
Interesting. And I guess I forgot that it generally just, like, works on the running jobs. Not on the ones that finished.
So, let us look at the actual batch files that we have, and try to submit bat job instead of the interactive one. And I'm just quickly going to look at the contents of the first file. So, we have some directives here.
And as in any shell script, it starts with a shebang. Basically saying this is a batch file. But the lines underneath are not actually commons, even though they do start with a hash.
They are SLURM directives. And on our clusters, what you do need to specify is generally, or always, the account, which in our case is the free account. You do need to give your job a name, and you need to specify a test time.
And then you would have your script here. In our case, we'll just have host name. And in order to submit a job, we just run it on the terminal.
But I just feel like so. And the job is submitted. And as before, we can, you know, either do sq and grab for the actual job name or number, or we can do sq and dash dash me to see my jobs in particular.
So, why use the batch jobs over just interactive sessions? They do not require user input while running. And you can just write a script, you can submit it to the scheduler, and SLURM runs it for you on a compute node. No need to stay logged in or watch it.
It survives the terminal disconnects. You can do many jobs at once. You can do long jobs beyond the VPN connection.
And of course, it enables automation. So, this is the script that we ran. And what I mentioned before, the anatomy, a little bit more about the directives here.
And a little bit more that we just mentioned. And, you know, if you wanted to test the other script, we can examine the batch script too. It has a little more directives.
And does more. So, I think this is pretty self-explanatory, right? Like, how does the script work? You submit it, it is scheduled, SLURM decides when and where the job runs. And of course, on our cluster, you might need to wait a little bit, especially if you're on a free account, and then it gets executed.
A little more on that. You log in into the login node, you submit as batch command, scheduler, node, and so on. So, if you work on our clusters, you are likely to use the environment modules.
And just, again, I'm gonna cover this very quickly. But in order to load various software, you might need to load various modules. Like, if you're working with R, or if you're working with OpenMBI.
And in order to see what we have available, you can issue the module avail command. And you would see the list of everything that we have. So, you know, different versions of R. And GCC.
And a lot, a lot of other stuff. And of course, you can incorporate it into your workflow, into your as batch scripts. And just like, a very quick overview, how you would use it.
You would do module load to use a certain module. What is the benefit of this? Because instead of installing software in your home directory, you would actually use the modules, and that puts less, uses less resources on our cluster. So, it's a little bit more efficient when people share.
I'm actually gonna skip over this one. But yes, what I just wanted to reiterate, that over requesting resources actually hurts scheduling of your jobs. So, again, if you do work, please be very conscious of what you're requesting, and always review it with the SAF command.
The rule of thumb, always start small. Just submit short test job. And then, like we discussed, submit with SAF, with PS, see if the job was efficient, and then just adjust the directives and resubmit.
And so, when you're doing your batch files, you generally would, you know, like, let us actually see the second one. When you are running your jobs there, as batch command, you actually would request more resources than we just tested in the first job. And you would specify nodes, for instance, and tasks, and perhaps CPUs and memory.
And it is nuanced, and I believe that Al will go into it in more detail. But just quickly, quick overview. So, memory, real memory required per node.
Capital N, or dash dash nodes, requests minimum amount of nodes that you need for your job. Dash N requests enough resources for the tasks that you have. And it's a CPU per task.
So, dash C, CPUs per task, tells SLAM the job steps need a certain number of processors per task. And without it, it would be just one processor per task. So, for instance, if in your job you have four tasks, and you need three processors per task, it will be 12 CPUs that you will get allocated.
GPUs, and I think I'm going a little bit over time, so I'm gonna speed up, and I will, I guess, let Al go into this a little bit more. But just, like, briefly, please keep in mind that GPUs are expensive resources. You can request them even on the free account and get access to them.
And the dash G flag. But please be considerate how many GPUs you are requesting. And everybody is sharing the same resources on the cluster, obviously.
Yes. Sorry. A little bit more cautioning.
But GPUs are expensive, and you need to make sure that your code supports the GPUs. And before you play with it, please understand the directives and what they mean in Slurm. Link to the main pages about Slurm.
A little bit more further reading. And a little bit more of on the best practices. So, again, examine your jobs, look over the accounting, and, of course, review the logs.
Make sure that you create the logs in your jobs, like we're doing here in this example. And just briefly, some of the most common user mistakes that we have seen. We have a couple of login nodes on our each one of our clusters.
And sometimes users run the jobs on the login nodes instead of the compute nodes. Please don't do that. We have seen users requesting many CPUs for just a single thread code.
Also, please don't do that. Overrequesting the wall time. Or forgetting the submit directory and writing files, you know, somewhere and then forgetting them.
Then not capturing logs. Probably a bad idea because you want to see what your job has done. And a lot of small jobs just, like, do them in a batch.
And not testing with S run first is not ideal because, again, good testing mechanism. Some additional workshops, events, resources. There's CUIT and Columbia Libraries.
Research computing opportunities. There's, you can always reach out to the research data services. And SourcePack is a good resource.
And I think that's it for my part. I'm going to let Al switch over. And I'm going to stop screen sharing.
And thank you, everyone. Please, of course, let us know if you have any questions. And feel free to play with this for another week or so while you still have access.
Thank you, Helena. And just give me a moment to share my screen. I think you may need to stop sharing yours first.
I'm trying. But I'm just like... Let me try and just share and see if that... Yeah, I'm just like... Don't worry. Here we go.
My sharing will... Okay. I think I did it. Yes.
Okay. So, welcome, everyone, to running HPC jobs with Bash and Slurm part 2. And this will focus on array jobs, Slurm variables, and cluster submission queues. So, I'm Al Tucker.
And I'll be your instructor for this part. And I'm a senior research systems engineer with the Columbia HPC group. So, first, we'll talk about scaling with job arrays.
And job arrays are a more efficient way to submit large numbers of similar tasks. Instead of submitting each job one at a time, you can submit them all together as a single group. This reduces the overhead on the Slurm scheduler and helps the system handle large workloads more smoothly.
A job array won't make your calculations run faster. But it helps Slurm manage the tasks more efficiently and can improve overall scheduling responsiveness. So, here's an example of, say, a default way someone would think of running multiple jobs.
You would use a typical for loop in Bash asking for something to be run up to 20 times. And you would submit your job. And this counter here would be the index.
So, the first time through the loop, your job would be running time number 1 and then 2, 3, 4. And this is how you would keep track of a loop.
this is a typical thing with bash and here is just some example input of what your job my job sh would be so under this situation there would be 20 separate job ids for slurring the track individually and this creates a certain amount of overhead with a job array you can submit everything in one jump just specifying how many times you want this to iterate and your internal code of your job would basically be the same though the counter would be kept track of by slurm and a variable called slurm array task id so this is easier for slurm because this is just one job with 20 tasks inside of it and for the slurm scheduler that's a bit faster so again with the job array the variable slurm array task id is unique to each task in the array so there's one overall job id with an index of one two three and so forth so why is a job rate easier for slurm so here's an analogy to kind of help you understand a job array would be like keeping track of 20 children on a field trip to a museum versus the for loop which is like letting all 20 run loose wherever they want to go and you're trying to keep track of them individually you can see when it's exemplified like this that situation number one is easier for both you and for slurm so more about job arrays and here is an example this will be our example job id and again the index for each task in the job is slurm array task id so as we saw before in the previous part when you used sq and you saw jobs listed you would see with a job array your master job id with a subscript of one two three whatever as the job id is the index variable itself the parent job that started it all would be contained in another variable which would be the slurm array job id and not the task id some other array variables that may be useful when you're running job arrays is slurm array task count which is the total number of tasks in the array slurm array task max is the maximum index number of tasks in the array and slurm array task min is the minimum number of tasks in the array slurm provides a wealth of variables about your job and some may be useful whether you're using an array or not the slurm job id is a variable that's always available even if you're not using an array and that is the array the id of your job it's always present another variable provided by slurm that you can access in your scripts is slurm node list and this is the list of nodes that's allocated to your job at the start and that would be something that you would request now it's important to understand that this node list doesn't necessarily mean that these are all the nodes that were used when your job ran if you request more nodes than you need your job might be slower to start as slurm looks for enough free nodes to satisfy the request you made another variable that slurm provides that you can access in your scripts is the slurm submit dir and this is the directory from which the job was submitted meaning it's where the s-batch was run from now this may not be where all your necessary files are so we'll use an example here to try to go into that if you have all the files that you need in one particular directory it's best before you start anything to cd into that directory first then you would use your s-batch command to run your job now once your script is running slurm will have this directory in slurm submit dir this would let your your job know that everything is running from where your files are and if your script changed directories during its run all the changes would be relative to the path that you started from so now we're going to move on from job arrays and the next part we'll talk about the four queue submission types on the hpc clusters and some options and pointers for using each of them what we'll go over will be your group's normal partition first then what's called the short queue then the hpc test queue and then lastly we'll talk about the burst queue so the first queue type we're going to talk about is your normal account and this is the standard way you submit a job on the cluster so you submit it with your s-batch script under your group account the accounts that we've made for you are all in the free user group though some of you had pre-existing cluster accounts so you know what your your normal accounts are but this is a partial example of your submit script here and when you submit you would specify the account that you're using under the s-batch dash dash account line and other things would be wall time requested number of nodes and memory and we'll go more into these as we go on so now one thing you should always specify a wall time with your job otherwise it will quit with an error if you don't know how much time runtime your job needs always start small and work your way up you can ask for in general up to seven days but you don't always need in fact rarely need a full seven days in this example here i've specified only 42 minutes if you submit a job and you don't ask for enough wall time your job or slur rather will kill your job after that specified wall time and then you would just run it again requesting more time again start small and scale up also you should always at least ask for one node and always request an amount of memory to be allocated for your entire job and again the numbers depend on what you're doing and can be different for everyone so if you're not sure it's better to start small and scale up remember the more you ask for the longer your job could potentially wait before it runs while slurm frees up enough room relative to all the other jobs running but if you don't supply either of these your job could quit with unexpected errors now the short partition this allows you to run a job on any node in the cluster but only for a short time up to 12 hours at most one reason why you would use the short partition if your group doesn't own a gpu node and you need to use a gpu this would give you access to someone else's gpu temporarily again 12 hours is the maximum amount of wall time you can request on the short queue but start small and scale up here in this example i've only asked for seven hours and 10 minutes also in this example since i'm requesting a gpu part of my code is loading the cuda module which loads gpu libraries that are needed to use a gpu there are also two more newer queues in the hpc available to users the first is the hpc test queue and this is only available to groups that own their own nodes in the cluster meaning that if you're only part of the free group or the edu group you wouldn't be able to use the hpc test queue and it's specified with this syntax of dash dash qos explicitly asking for the test queue this queue was designed for those times when between jobs in your group needing to run and jobs in the short partition also running on your node you may experience a delay in your job being assigned a node so that's why we developed the hpc test queue by using the hpc test queue your job gets higher priority on your node over other jobs so that means your job could quickly gain access now the trade-off here is that the maximum wall time on the hpc test queue is only six hours we do this to strike the best balance between flexibility between a high priority and equal access for everyone again once when using this queue start smaller with your time request and work up with the hpc test queue potentially i can ask for six hours in this example i've only asked for two hours and 13 minutes so here's a caveat about queue options and take a moment to look at this example below on your screen here's a place where i like to stop and point out that sometimes not all possible combinations of options go well together so it's up to you the user to think through the ramifications of what you're requesting and the next slide will explain this using the hpc test queue as an example this gpu request may or may not work the reason is because the hpc test queue gives you higher priority on a node that your group owns so if your group does not own a gpu node you won't be able to request a gpu and your job would have an error now if your group does own a gpu node you'd get a gpu node automatically many nodes only have two gpus so it does make sense to request one or two as you may need an aside about gpus is also that just because you request a gpu node does not mean your job will use a gpu your program needs to be gpu aware and everyone's research varies so again this is a point where it's up to you as the user to understand what you're doing and make appropriate choices as you construct your jobs so just as a general rule of thumb if your program ran perfectly on a standard node meaning a node without a gpu unless you consciously made changes in your code just because you ran it on a gpu node would not make it use a gpu or run any faster now similar to the short queue the burst queue is another queue that lets you run jobs on any node in the cluster and here you have a higher limit you can have a wall time as long as 14 days in this example i'm requesting 10 days and two hours of wall time now there's a trade-off with the burst queue since your job could potentially occupy a node for up to two weeks the burst queue has a very low priority and it can be preempted by jobs from the node's owner this means that if you do use the burst queue your job could possibly be forced to quit and then be put back on the queue to restart later so in order for this to be efficient you're responsible for making sure your job will write temporary files and that's a technique we call checkpointing so that if your job is forced to stop it can later look at its own progress and start up from approximately where it left off instead of restarting from the very beginning reading these checkpoint files for appropriate use is also something that you as the user must account for in your code so fully explaining how to checkpoint is beyond the reach of this training session and it requires the user to understand when this may be useful not every job can be interrupted and started again later one thing we would like to point out though is that when you're writing temporary files on a job and this is whether for checkpointing or just in general place to write these files is in the slash local path on the node first secondly you could use your own home directory as a place and then lastly you would write temporary files to your group scrap space what you should not do is write files on the file system in the slash tmp or the slash bar slash tmp paths as these are paths on the nodes they're on smaller file systems and there is a danger of running the node out of space this can cause problems for your job and for the node in general so when you ask for a node and some memory for your code your job isn't alone on the node a standard node on the insomnia cluster has 512 gig of ram and 160 cores that's what something called hyper threading enabled slurm is going to handle sharing those resources with your job and with jobs from other users that are running on the cluster or on that particular node now there is an option called exclusivity and that's with the s fact dash dash exclusive flag that you would put into your code this is makes it possible for an entire node to be exclusively allocated to your job now again there's a caveat unless you're absolutely sure you need to use this the dash dash exclusive option is not always the best idea the reason is your job will have to wait until slurm can clear an entire node of all jobs before yours can begin so in a shared environment like the cluster where many people are running jobs in order to free up an entire node your job could be waiting days instead of minutes or hours before it actually starts when you use the dash dash exclusive flag another point i want to make about job options and reasoning through which ones make sense together and which may not it still does make sense with exclusivity to ask for memory exclusively to be allocated to your job the reason is dash dash exclusive means that you can use all the resources on the node it doesn't mean that you're automatically given all the resources on a node so if you ask for exclusive use of a node but you don't ask for any memory your job would only be given a small trivial default amount of memory which may be not enough this is similar in concept to how just running on a gpu node does not mean your code will actually use a gpu node you need to account for it in your code so when you ask for a node and some in some memory slurm will invisibly fill in a one to most questions that are applicable such as how many cpu's for the job use now for many jobs this is fine you can however be more precise in allocating resources for your job these may or may not make sense depending on specifically what type of work you are doing so with these options above dash dash end tasks dash dash cpus per task and dash dash mem per cpu you can specify on your node how many tasks your job should have as well as how many cpus your job each task rather should use and then how much memory each individual cpu should be given so in the next slide i'll attempt to use an analogy to show you what this means so using the same numbers from before if you're asking for a node you could think of a node as a department store one big place the tasks in this analogy would be the individual departments within the store cpus per task in this analogy would be each employee working within each department in the store and under this analogy memory per cpu would be how much each employee is paid to work within each department within the department store now if you don't make those suggestions or specifications slurm would default to just a one for everything and for many jobs that would be fine these are powerful processors not literal people but you can see what happens when you don't make decisions and slurm just fills in something for you it may not be enough so now about task and cpu requests memory is cumulative or this is more about how some options may not mix well so in this example notice that in both s-batch examples here you're requesting 24 gig of ram for the entire job but each one is going about it very differently in the top example you're specifying things more finely and you're allocating ram in individual portions for each cpu and every task so four tasks by two cpus per task by three gig of ram per cpu means overall you're getting 24 gig of ram for the entire job it's the same as in this example but you're just allocating it more finely here instead of one big batch for the entire job now what you don't want to do though is mistakenly mix conflicting options like i've done here since the mem per cpu request is cumulative by default you're requesting 24 a gig for the entire job already with just this but if you also use the dash dash mem directive to request 24 gig you've set up a conflict because you're asking for the same amount of ram twice just in different ways so in the best case this would cause your job to fail with an error again this is another example of just being aware of job options as you construct your jobs some job options may not work well together so here's another example note here that either of these examples would be a correct way to fix the conflicting memory problem that we had in the last slide in the first example 24 gig of ram is allocated for the entire job and the tasks and cpus would be free to fluidly use more or less out of a large pool of 24 gig of ram in the second example each individual cpu in each individual task would have three gig of ram to work with now of course in this example if some cpus only use one gig of ram you'd be wasting reserved resources or even worse if any individual cpu needed four gig or more your job could fail entirely so again i want to enforce that neither of these solutions is better inherently than the other and if this sounds confusing yes it can be in the world of linux there are often multiple ways to do the same thing and each one is valid deciding how much memory you need do you need to specify tasks how many tasks how to allocate ram all these are questions with no right answer there's only right for you depending on what programs you're using to do your research and sometimes this may just be as much art as it is science when you're starting out creating your slurm jobs a good way to get guidance on questions like these or from other colleagues in your lab or researchers in your field who are doing the same kind of work you're doing when you have a clear idea of what you want to do and then it still fails that is where the hpc support can be of best help so to wrap up these concepts here it's impossible to cover the full magnitude of slurm options and all the possible variables in any single session so we hope that you found this broad overview informative as you begin using our clusters a recording of this session will be posted later among other training videos we have at this url here also more specific help what can be found in the documentation of our columbia clusters insomnia ginsburg and terramoto and this can be found linked off of our main hpc page and this url and as always google and gemini are never more than a click away if you have questions we're wrapping up now you can feel free to ask but also you can always contact us with your questions at hpc-support at columbia.edu and that ends my part of the session and if there are any questions uh feel free to ask.
We will specifically focus on Bash for Slurm, and Slurm is our HPC job controller. Hopefully, by the end of the class, you will be able to run, view, and analyze Slurm jobs, read and modify Slurm Bash scripts with confidence, request compute resources correctly, and debug the failed jobs. And if you would like to follow along but don't feel pressured, you can SSH or log in to our Insomnia cluster.
It's an HPC cluster at Columbia. I think all of you should have access, but again, don't worry if you don't have access. We can figure it out later.
But basically, I'm going to log in with you, and in order to get in, you would just issue an SSH command with your uni, and then insomniarcscolumbia.edu. So I already have some SSH keys set up, but you would be prompted for the password, and you would be prompted for Duo entry. I'm just going to give everybody a minute to catch up. Again, if you're having issues, please, I think it's best if you just post something in the chat, and one of my co-workers will help you.
If you would like to follow along with the exercise, you can also copy over some files that we already have in place. I have the information here on the screen. You know, make a new directory.
You can name it anything you want. But, you know, for instance, bash workshop, and then I'm going to get into the directory. I'm going to copy over some files from the storage directory that we have on the cluster.
Path is a little bit convoluted. So if you remember from previous Linux class, if you're not familiar with Linux, we're just copying everything from the directory and into the current place, and we're going to do a long listing, so we have a few files here now that we will use a little later. And just a quick recap.
What is bash? It is a shell for Linux. It is the default scripting language on essentially every Linux system, although on your Macs you might have zshell, and like any programming language, it has control flow built in, so it has loops, conditionals, variables, all of that. It is not a prerequisite for this class to know bash well.
You don't need it in order to submit Slurm jobs effectively. And again, Slurm, it is our HPC job scheduler that we use in Columbia. It is open source, it is full tolerant, highly scalable, it is used in a lot of top 500 companies.
Last time I checked, it was like 60 plus percent. It manages the computer resources for you, like CPUs, GPUs, memory, schedules and monitors job queues, and it acts as the cluster's work manager. I'm going to skip ahead a little bit, and basically how you would examine the job queue is the SQ command, so here we see everything that is currently running in the cluster when we issue the SQ command.
A little bit more to elaborate. Slurm allocates resources, it does the job management, it does the fair scheduling, and it's designed for large-scale workloads, and no Linux kernel modification is required. So, some of the commands that are very useful to learn and we will work with are srun, to do a run of the job interactively, as batch, to submit a batch script for later execution, SQ, as we just saw, to view status of the jobs in the queue, as info, shows the status of partitions and nodes, cancel, to cancel the jobs, and saft, examine jobs efficiently.
And I know I'm going fast, so if you are completely new to this, I know it might be a lot of information, but just bear with me, and, again, you can always just examine all of this on your own. I believe that you will have access to the cluster for another week or two. So, you know, like, you can play with it a little more.
And so, let's do a hands-on exercise and just, like, quickly submit an srun, a job with srun. What is the advantage of the interactive jobs versus as batch? So, interactive job lets you run application on a compute node if you were just on a regular login node, but with dedicated resources, and it lets you test things pretty quickly. Like, yeah, like we said, good for quick tests and debugging.
And if you would like to follow along, you can definitely run the srun command with me. So, srun and PTY means on the terminal, dash T says for how long you would like your script to run, and generally, on our clusters, you need to specify the partition. In our case, it's free, and I'm just going to do a bash shell.
And so, we do have quite a few free users on the cluster. Therefore, you might be waiting for a little bit to get your resources allocated, but in the end, the job is submitted. And many ways you can examine it, but, for instance, you can do sqme command, and you see your job running here.
And, of course, this is just for demo purposes. My job isn't doing much. It just opened the bash shell, but generally, you know, like, you would be doing actual tasks.
And, yeah, a little bit more information here. You know, sq has flags to examine the job itself, and what I just did with sq and seeing all of my jobs, and my job ran out of time because I only requested one minute. So, I got kicked off.
But, you know, perhaps we want more information about the job, and s account is a useful command that I mentioned before. S account with the dash J flag and the job number lets me examine more information about the job. Like, where it ran, if there were errors, exit codes, what partition, et cetera, et cetera.
Very, very useful. And for all of our users, we recommend using the SAF command, which basically examines the efficiency of jobs, and the problem that we see quite often is that users over request the resources, memory, CPU, unfortunately, GPUs also that are very expensive. So, it is very useful to see if your job was efficient at running, and going to examine the stats on my job with the SAF command.
And, again, because it was, you know, just like a demo job, the efficiency was pretty poor with the CPU, with the memory. And if your jobs are approaching zero percent, you maybe want to reexamine how many resources you are requesting on the cluster. And, of course, in real time, you can always examine it with the built in Linux PS command with some flags.
And, again, with like the SAF command. In order to stop a job, ask cancel is a good command. Or, you know, like you can just kill it directly, but with the Linux kill command, but more efficient to use a built in SLURM ask cancel command.
Ask control, as we mentioned, one of the very useful SLURM commands, and we can run it also on my previous command. It will give you stats about run times, like start, when it ended, because it's not always, I guess, obvious how long it took, and the amount of resources. So, show job.
Interesting. And I guess I forgot that it generally just, like, works on the running jobs. Not on the ones that finished.
So, let us look at the actual batch files that we have, and try to submit bat job instead of the interactive one. And I'm just quickly going to look at the contents of the first file. So, we have some directives here.
And as in any shell script, it starts with a shebang. Basically saying this is a batch file. But the lines underneath are not actually commons, even though they do start with a hash.
They are SLURM directives. And on our clusters, what you do need to specify is generally, or always, the account, which in our case is the free account. You do need to give your job a name, and you need to specify a test time.
And then you would have your script here. In our case, we'll just have host name. And in order to submit a job, we just run it on the terminal.
But I just feel like so. And the job is submitted. And as before, we can, you know, either do sq and grab for the actual job name or number, or we can do sq and dash dash me to see my jobs in particular.
So, why use the batch jobs over just interactive sessions? They do not require user input while running. And you can just write a script, you can submit it to the scheduler, and SLURM runs it for you on a compute node. No need to stay logged in or watch it.
It survives the terminal disconnects. You can do many jobs at once. You can do long jobs beyond the VPN connection.
And of course, it enables automation. So, this is the script that we ran. And what I mentioned before, the anatomy, a little bit more about the directives here.
And a little bit more that we just mentioned. And, you know, if you wanted to test the other script, we can examine the batch script too. It has a little more directives.
And does more. So, I think this is pretty self-explanatory, right? Like, how does the script work? You submit it, it is scheduled, SLURM decides when and where the job runs. And of course, on our cluster, you might need to wait a little bit, especially if you're on a free account, and then it gets executed.
A little more on that. You log in into the login node, you submit as batch command, scheduler, node, and so on. So, if you work on our clusters, you are likely to use the environment modules.
And just, again, I'm gonna cover this very quickly. But in order to load various software, you might need to load various modules. Like, if you're working with R, or if you're working with OpenMBI.
And in order to see what we have available, you can issue the module avail command. And you would see the list of everything that we have. So, you know, different versions of R. And GCC.
And a lot, a lot of other stuff. And of course, you can incorporate it into your workflow, into your as batch scripts. And just like, a very quick overview, how you would use it.
You would do module load to use a certain module. What is the benefit of this? Because instead of installing software in your home directory, you would actually use the modules, and that puts less, uses less resources on our cluster. So, it's a little bit more efficient when people share.
I'm actually gonna skip over this one. But yes, what I just wanted to reiterate, that over requesting resources actually hurts scheduling of your jobs. So, again, if you do work, please be very conscious of what you're requesting, and always review it with the SAF command.
The rule of thumb, always start small. Just submit short test job. And then, like we discussed, submit with SAF, with PS, see if the job was efficient, and then just adjust the directives and resubmit.
And so, when you're doing your batch files, you generally would, you know, like, let us actually see the second one. When you are running your jobs there, as batch command, you actually would request more resources than we just tested in the first job. And you would specify nodes, for instance, and tasks, and perhaps CPUs and memory.
And it is nuanced, and I believe that Al will go into it in more detail. But just quickly, quick overview. So, memory, real memory required per node.
Capital N, or dash dash nodes, requests minimum amount of nodes that you need for your job. Dash N requests enough resources for the tasks that you have. And it's a CPU per task.
So, dash C, CPUs per task, tells SLAM the job steps need a certain number of processors per task. And without it, it would be just one processor per task. So, for instance, if in your job you have four tasks, and you need three processors per task, it will be 12 CPUs that you will get allocated.
GPUs, and I think I'm going a little bit over time, so I'm gonna speed up, and I will, I guess, let Al go into this a little bit more. But just, like, briefly, please keep in mind that GPUs are expensive resources. You can request them even on the free account and get access to them.
And the dash G flag. But please be considerate how many GPUs you are requesting. And everybody is sharing the same resources on the cluster, obviously.
Yes. Sorry. A little bit more cautioning.
But GPUs are expensive, and you need to make sure that your code supports the GPUs. And before you play with it, please understand the directives and what they mean in Slurm. Link to the main pages about Slurm.
A little bit more further reading. And a little bit more of on the best practices. So, again, examine your jobs, look over the accounting, and, of course, review the logs.
Make sure that you create the logs in your jobs, like we're doing here in this example. And just briefly, some of the most common user mistakes that we have seen. We have a couple of login nodes on our each one of our clusters.
And sometimes users run the jobs on the login nodes instead of the compute nodes. Please don't do that. We have seen users requesting many CPUs for just a single thread code.
Also, please don't do that. Overrequesting the wall time. Or forgetting the submit directory and writing files, you know, somewhere and then forgetting them.
Then not capturing logs. Probably a bad idea because you want to see what your job has done. And a lot of small jobs just, like, do them in a batch.
And not testing with S run first is not ideal because, again, good testing mechanism. Some additional workshops, events, resources. There's CUIT and Columbia Libraries.
Research computing opportunities. There's, you can always reach out to the research data services. And SourcePack is a good resource.
And I think that's it for my part. I'm going to let Al switch over. And I'm going to stop screen sharing.
And thank you, everyone. Please, of course, let us know if you have any questions. And feel free to play with this for another week or so while you still have access.
Thank you, Helena. And just give me a moment to share my screen. I think you may need to stop sharing yours first.
I'm trying. But I'm just like... Let me try and just share and see if that... Yeah, I'm just like... Don't worry. Here we go.
My sharing will... Okay. I think I did it. Yes.
Okay. So, welcome, everyone, to running HPC jobs with Bash and Slurm part 2. And this will focus on array jobs, Slurm variables, and cluster submission queues. So, I'm Al Tucker.
And I'll be your instructor for this part. And I'm a senior research systems engineer with the Columbia HPC group. So, first, we'll talk about scaling with job arrays.
And job arrays are a more efficient way to submit large numbers of similar tasks. Instead of submitting each job one at a time, you can submit them all together as a single group. This reduces the overhead on the Slurm scheduler and helps the system handle large workloads more smoothly.
A job array won't make your calculations run faster. But it helps Slurm manage the tasks more efficiently and can improve overall scheduling responsiveness. So, here's an example of, say, a default way someone would think of running multiple jobs.
You would use a typical for loop in Bash asking for something to be run up to 20 times. And you would submit your job. And this counter here would be the index.
So, the first time through the loop, your job would be running time number 1 and then 2, 3, 4. And this is how you would keep track of a loop.
this is a typical thing with bash and here is just some example input of what your job my job sh would be so under this situation there would be 20 separate job ids for slurring the track individually and this creates a certain amount of overhead with a job array you can submit everything in one jump just specifying how many times you want this to iterate and your internal code of your job would basically be the same though the counter would be kept track of by slurm and a variable called slurm array task id so this is easier for slurm because this is just one job with 20 tasks inside of it and for the slurm scheduler that's a bit faster so again with the job array the variable slurm array task id is unique to each task in the array so there's one overall job id with an index of one two three and so forth so why is a job rate easier for slurm so here's an analogy to kind of help you understand a job array would be like keeping track of 20 children on a field trip to a museum versus the for loop which is like letting all 20 run loose wherever they want to go and you're trying to keep track of them individually you can see when it's exemplified like this that situation number one is easier for both you and for slurm so more about job arrays and here is an example this will be our example job id and again the index for each task in the job is slurm array task id so as we saw before in the previous part when you used sq and you saw jobs listed you would see with a job array your master job id with a subscript of one two three whatever as the job id is the index variable itself the parent job that started it all would be contained in another variable which would be the slurm array job id and not the task id some other array variables that may be useful when you're running job arrays is slurm array task count which is the total number of tasks in the array slurm array task max is the maximum index number of tasks in the array and slurm array task min is the minimum number of tasks in the array slurm provides a wealth of variables about your job and some may be useful whether you're using an array or not the slurm job id is a variable that's always available even if you're not using an array and that is the array the id of your job it's always present another variable provided by slurm that you can access in your scripts is slurm node list and this is the list of nodes that's allocated to your job at the start and that would be something that you would request now it's important to understand that this node list doesn't necessarily mean that these are all the nodes that were used when your job ran if you request more nodes than you need your job might be slower to start as slurm looks for enough free nodes to satisfy the request you made another variable that slurm provides that you can access in your scripts is the slurm submit dir and this is the directory from which the job was submitted meaning it's where the s-batch was run from now this may not be where all your necessary files are so we'll use an example here to try to go into that if you have all the files that you need in one particular directory it's best before you start anything to cd into that directory first then you would use your s-batch command to run your job now once your script is running slurm will have this directory in slurm submit dir this would let your your job know that everything is running from where your files are and if your script changed directories during its run all the changes would be relative to the path that you started from so now we're going to move on from job arrays and the next part we'll talk about the four queue submission types on the hpc clusters and some options and pointers for using each of them what we'll go over will be your group's normal partition first then what's called the short queue then the hpc test queue and then lastly we'll talk about the burst queue so the first queue type we're going to talk about is your normal account and this is the standard way you submit a job on the cluster so you submit it with your s-batch script under your group account the accounts that we've made for you are all in the free user group though some of you had pre-existing cluster accounts so you know what your your normal accounts are but this is a partial example of your submit script here and when you submit you would specify the account that you're using under the s-batch dash dash account line and other things would be wall time requested number of nodes and memory and we'll go more into these as we go on so now one thing you should always specify a wall time with your job otherwise it will quit with an error if you don't know how much time runtime your job needs always start small and work your way up you can ask for in general up to seven days but you don't always need in fact rarely need a full seven days in this example here i've specified only 42 minutes if you submit a job and you don't ask for enough wall time your job or slur rather will kill your job after that specified wall time and then you would just run it again requesting more time again start small and scale up also you should always at least ask for one node and always request an amount of memory to be allocated for your entire job and again the numbers depend on what you're doing and can be different for everyone so if you're not sure it's better to start small and scale up remember the more you ask for the longer your job could potentially wait before it runs while slurm frees up enough room relative to all the other jobs running but if you don't supply either of these your job could quit with unexpected errors now the short partition this allows you to run a job on any node in the cluster but only for a short time up to 12 hours at most one reason why you would use the short partition if your group doesn't own a gpu node and you need to use a gpu this would give you access to someone else's gpu temporarily again 12 hours is the maximum amount of wall time you can request on the short queue but start small and scale up here in this example i've only asked for seven hours and 10 minutes also in this example since i'm requesting a gpu part of my code is loading the cuda module which loads gpu libraries that are needed to use a gpu there are also two more newer queues in the hpc available to users the first is the hpc test queue and this is only available to groups that own their own nodes in the cluster meaning that if you're only part of the free group or the edu group you wouldn't be able to use the hpc test queue and it's specified with this syntax of dash dash qos explicitly asking for the test queue this queue was designed for those times when between jobs in your group needing to run and jobs in the short partition also running on your node you may experience a delay in your job being assigned a node so that's why we developed the hpc test queue by using the hpc test queue your job gets higher priority on your node over other jobs so that means your job could quickly gain access now the trade-off here is that the maximum wall time on the hpc test queue is only six hours we do this to strike the best balance between flexibility between a high priority and equal access for everyone again once when using this queue start smaller with your time request and work up with the hpc test queue potentially i can ask for six hours in this example i've only asked for two hours and 13 minutes so here's a caveat about queue options and take a moment to look at this example below on your screen here's a place where i like to stop and point out that sometimes not all possible combinations of options go well together so it's up to you the user to think through the ramifications of what you're requesting and the next slide will explain this using the hpc test queue as an example this gpu request may or may not work the reason is because the hpc test queue gives you higher priority on a node that your group owns so if your group does not own a gpu node you won't be able to request a gpu and your job would have an error now if your group does own a gpu node you'd get a gpu node automatically many nodes only have two gpus so it does make sense to request one or two as you may need an aside about gpus is also that just because you request a gpu node does not mean your job will use a gpu your program needs to be gpu aware and everyone's research varies so again this is a point where it's up to you as the user to understand what you're doing and make appropriate choices as you construct your jobs so just as a general rule of thumb if your program ran perfectly on a standard node meaning a node without a gpu unless you consciously made changes in your code just because you ran it on a gpu node would not make it use a gpu or run any faster now similar to the short queue the burst queue is another queue that lets you run jobs on any node in the cluster and here you have a higher limit you can have a wall time as long as 14 days in this example i'm requesting 10 days and two hours of wall time now there's a trade-off with the burst queue since your job could potentially occupy a node for up to two weeks the burst queue has a very low priority and it can be preempted by jobs from the node's owner this means that if you do use the burst queue your job could possibly be forced to quit and then be put back on the queue to restart later so in order for this to be efficient you're responsible for making sure your job will write temporary files and that's a technique we call checkpointing so that if your job is forced to stop it can later look at its own progress and start up from approximately where it left off instead of restarting from the very beginning reading these checkpoint files for appropriate use is also something that you as the user must account for in your code so fully explaining how to checkpoint is beyond the reach of this training session and it requires the user to understand when this may be useful not every job can be interrupted and started again later one thing we would like to point out though is that when you're writing temporary files on a job and this is whether for checkpointing or just in general place to write these files is in the slash local path on the node first secondly you could use your own home directory as a place and then lastly you would write temporary files to your group scrap space what you should not do is write files on the file system in the slash tmp or the slash bar slash tmp paths as these are paths on the nodes they're on smaller file systems and there is a danger of running the node out of space this can cause problems for your job and for the node in general so when you ask for a node and some memory for your code your job isn't alone on the node a standard node on the insomnia cluster has 512 gig of ram and 160 cores that's what something called hyper threading enabled slurm is going to handle sharing those resources with your job and with jobs from other users that are running on the cluster or on that particular node now there is an option called exclusivity and that's with the s fact dash dash exclusive flag that you would put into your code this is makes it possible for an entire node to be exclusively allocated to your job now again there's a caveat unless you're absolutely sure you need to use this the dash dash exclusive option is not always the best idea the reason is your job will have to wait until slurm can clear an entire node of all jobs before yours can begin so in a shared environment like the cluster where many people are running jobs in order to free up an entire node your job could be waiting days instead of minutes or hours before it actually starts when you use the dash dash exclusive flag another point i want to make about job options and reasoning through which ones make sense together and which may not it still does make sense with exclusivity to ask for memory exclusively to be allocated to your job the reason is dash dash exclusive means that you can use all the resources on the node it doesn't mean that you're automatically given all the resources on a node so if you ask for exclusive use of a node but you don't ask for any memory your job would only be given a small trivial default amount of memory which may be not enough this is similar in concept to how just running on a gpu node does not mean your code will actually use a gpu node you need to account for it in your code so when you ask for a node and some in some memory slurm will invisibly fill in a one to most questions that are applicable such as how many cpu's for the job use now for many jobs this is fine you can however be more precise in allocating resources for your job these may or may not make sense depending on specifically what type of work you are doing so with these options above dash dash end tasks dash dash cpus per task and dash dash mem per cpu you can specify on your node how many tasks your job should have as well as how many cpus your job each task rather should use and then how much memory each individual cpu should be given so in the next slide i'll attempt to use an analogy to show you what this means so using the same numbers from before if you're asking for a node you could think of a node as a department store one big place the tasks in this analogy would be the individual departments within the store cpus per task in this analogy would be each employee working within each department in the store and under this analogy memory per cpu would be how much each employee is paid to work within each department within the department store now if you don't make those suggestions or specifications slurm would default to just a one for everything and for many jobs that would be fine these are powerful processors not literal people but you can see what happens when you don't make decisions and slurm just fills in something for you it may not be enough so now about task and cpu requests memory is cumulative or this is more about how some options may not mix well so in this example notice that in both s-batch examples here you're requesting 24 gig of ram for the entire job but each one is going about it very differently in the top example you're specifying things more finely and you're allocating ram in individual portions for each cpu and every task so four tasks by two cpus per task by three gig of ram per cpu means overall you're getting 24 gig of ram for the entire job it's the same as in this example but you're just allocating it more finely here instead of one big batch for the entire job now what you don't want to do though is mistakenly mix conflicting options like i've done here since the mem per cpu request is cumulative by default you're requesting 24 a gig for the entire job already with just this but if you also use the dash dash mem directive to request 24 gig you've set up a conflict because you're asking for the same amount of ram twice just in different ways so in the best case this would cause your job to fail with an error again this is another example of just being aware of job options as you construct your jobs some job options may not work well together so here's another example note here that either of these examples would be a correct way to fix the conflicting memory problem that we had in the last slide in the first example 24 gig of ram is allocated for the entire job and the tasks and cpus would be free to fluidly use more or less out of a large pool of 24 gig of ram in the second example each individual cpu in each individual task would have three gig of ram to work with now of course in this example if some cpus only use one gig of ram you'd be wasting reserved resources or even worse if any individual cpu needed four gig or more your job could fail entirely so again i want to enforce that neither of these solutions is better inherently than the other and if this sounds confusing yes it can be in the world of linux there are often multiple ways to do the same thing and each one is valid deciding how much memory you need do you need to specify tasks how many tasks how to allocate ram all these are questions with no right answer there's only right for you depending on what programs you're using to do your research and sometimes this may just be as much art as it is science when you're starting out creating your slurm jobs a good way to get guidance on questions like these or from other colleagues in your lab or researchers in your field who are doing the same kind of work you're doing when you have a clear idea of what you want to do and then it still fails that is where the hpc support can be of best help so to wrap up these concepts here it's impossible to cover the full magnitude of slurm options and all the possible variables in any single session so we hope that you found this broad overview informative as you begin using our clusters a recording of this session will be posted later among other training videos we have at this url here also more specific help what can be found in the documentation of our columbia clusters insomnia ginsburg and terramoto and this can be found linked off of our main hpc page and this url and as always google and gemini are never more than a click away if you have questions we're wrapping up now you can feel free to ask but also you can always contact us with your questions at hpc-support at columbia.edu and that ends my part of the session and if there are any questions uh feel free to ask.