Running Containerized Workloads with Singularity | 2026 HPC Training Series #5
Transcript
Welcome everyone to part five of our HPC workshop series. My name is Waqas and I'm a senior research systems engineer with HPC team at Columbia University Information Technology, CUIT. Today we are looking at singularity, also known as abtainer, and why containers are so useful in research computing.
The focus is practical, how we take a software environment that works in one place and run it consistently on an HPC cluster. We will start with concepts, compare singularity with Docker and then walk through the pulling the images and running the containers. By the end, you should recognize the key commands and understand where they fit into SLURM workflow.
So you do not need to be an expert in the containers to follow along, but familiarity with Linux commands and connecting to Insomnia will help. I will explain what each command is doing before we use it and if you are following along with the demonstration, please keep your terminal open so you can practice the commands as we go and if you are able to follow along, if you are unable to follow along for any reason, don't worry, a recording of this session will be available for you to review and practice at your own pace later. If you have any questions or if something is unclear at any point, please feel free to post your questions in the chat.
One of my colleagues will be monitoring the chat and will be able to assist you. All right. So here's our plan.
First, we will discuss why containers exist and the problems they solve, particularly software dependencies and reproducibility. Then we will look at common image registries and compare Docker with singularity. After that, we will move to Insomnia, obtain compute node allocation, load the module, and work from scratch storage.
And finally, we will explore pulling and running images, making modifications with a sandbox, writing definition files and integrating containers with modules and Slurm. The first half explains the why and the second half shows how. All right.
So what are containers? Think of a container as a packaged application environment. It can include the application, its runtime, libraries, and other dependencies. So you do not have to reconstruct that environment on every machine.
The container is portable because its software stack travels with it. Although the host hardware and kernel still matter. In singularity, a common finished image format is the single file SIF or .SIF. One important distinction, a container is not a virtual machine.
It shares the host system's kernel while isolating the application's user space environment. All right. So why containers? It comes into mind for the researchers that often researchers face the familiar problem.
It works on my laptop but not on the cluster. A different Python version, missing library, or conflicting dependency can stop a workflow. So containers help by packaging the software stack together.
They also support reproducibility. A fixed image makes it easier to return to the same environment later, though the inputs, hardware, and settings still matter. Excuse me.
On a shared APC cluster, singularity is especially useful because users can run containerized software without ordinary privileges. So the value is not just convenience, it's portability, control, a more predictable research environment. Excuse me.
So before building anything, we need to understand what are the container registries. Check whether a suitable image already exists. A container registry is a place where people and organizations publish prebuilt images.
Docker Hub has a general purpose software. Red Hat, QA, and GitHub Container Registry are also common sources. And media NGC is particularly relevant to GPU and machine learning workloads.
And Biocontainers focuses on scientific and bioinformatics tools. The typical HPC approach is to find a trusted starting image, customize it if necessary, and run it with singularity on the cluster. We should not treat every public image as trustworthy.
Review its origin, documentation, and contents before using it with research data. All right. So let's compare Apple to Apple.
Docker and singularity both use containers. But their typical environments differ. Docker is widely used for cloud services, application deployment, and image building workflows as compared to singularity or Attainer.
It is designed around the shared HPC systems where users don't have administrative access. It works naturally with command line, research programs, and can expose the host's GPU and MPI environment when configured properly. A key HPC benefit is that the user's identity carries into the container, helping file ownership and permissions behave as expected.
A common pattern is to prepare an image with Docker compatible tools, then run that image with singularity on the cluster. Docker itself also supports rootless modes. So the contrast here is about the workshop's typical deployment model.
All right. So using singularity on HPC, let's connect to the concepts to the practical work. So the most important rule is to separate a login activity from compute work.
So login nodes are for access and live preparation. Computationally intensive container tasks belong on compute nodes. So we request resources through Slurm using an interactive job for testing or a batch job for production.
Commands such as srun or slr provide that allocated environment. And once the allocation starts, we can load Attainer and run the container without affecting other users on shared login node. So for example, we begin on Insomnia login node.
Let me start with which Attainer. So in this example, it returns no Attainer message. So that is the behavior for the demonstration.
So do not start any hands-on container workload here, and it won't let you. So which basically only checks whether the command can be found. It does not allocate any compute resources or load any module.
And our next step is basically to ask this Slurm for an interactive compute node. All right. So let me go ahead and run the srun command here.
And then I'm going to explain each part of this. All right. So let me break down this what this command is doing.
Srun basically asking Slurm to start a job step. And then hyphen hyphen pty gives us an interactive terminal. Hyphen t is basically requesting one hour.
Hyphen a RCS specifies the account, and slash bin bash launches our shell. So Slurm may report that the job is waiting for resources. Once allocated, the shell prompt changes to a compute node hostname, such as in this case, I have INS093.
So that changed hostname tells us we are no longer working on the login node. So excuse me. So let's load the module.
And if we wanted to load that module on the login node, it would prevent you to do that, because there is kind of a preventive measure. So here on the compute node, I'm going to load that obtainer. And how can I check it? And if I run the same command, which obtainer.
So now you can see that this example points to the shared obtainer 1.4.5 installation. At this point, we have followed the essential sequence, login, request compute resources, load the module, and verify the executable. And now we are ready to work with images.
But before pulling a large image, move to the appropriate scratch space. The home directories have tighter quotas and may be less suitable for large container files. And I owe heavy work.
So your scratch path will be different. And here's mine. And then up until here, you will have the same.
But probably for you, it should be RCS. And also when you're asking for the compute node, you should use free account. All right.
And then next is my username. No, user's directory. And then my username.
All right. So if I do pwd, it gives me that I'm here on the scratch space. So we want our images, our temporary downloads, working files here in the high capacity workspace.
But also remember that the scratch is not a suitable place for backed up or long-term storage. So my next slide will give us the command to pull an image. So now we pull a singularity image, Docker image through singularity.
All right. So while the image is being pulled from the registry, let me walk you through what's happening behind the scenes. So the Docker call and slash slash prefix, it tells singularity to retrieve an OCI or Docker format image from a registry.
And then singularity converts it into its SIF format and writes the file such as Python 3.11.SIF in the current directory. So this is the key convenience. We can use the large ecosystem of published Docker images without running the Docker services on the cluster.
And by default, singularity cache is under this path, which is shown here. Cache location or cache allocation matters when home quarters are limited. The active container configuration may use container prefix equivalence.
It's going to take a little bit of time, maybe one or two minutes to pull the image. All right. So meanwhile, it is pulling the image.
Let me go over to the next slide. So there are three common ways to use an image. One is interactive shell inside the container, which is useful for exploration and troubleshooting.
Then singularity exec basically runs one specific command inside the container and returns control when it finishes. And then there is a third one, which is singularity run. It invokes the image's predefined run function or run action and commonly supplied by a run script in its definition file.
And we'll get to the definition files a little later. So for scheduled research workflows, exec is often especially clear because the exec command appears in this run script. We'll see how that works later.
It's taking a little longer than anticipated. There you go. So if I do LS, I have my SIF image.
So now that we have covered how to pull the container image, let's look at practical example of how researchers can benefit from Abtainer on an HPC cluster. So for example, imagine you're working on a research project that requires a different version of Python. For example, it requires 3.11, but the Python version available on the cluster is different.
Normally, you might need to install another version, configure the dependencies, or ask the HPC administrators for help. With Abtainer, we can use a Python environment that's already packaged inside a container without installing anything on the cluster or requiring any root privileges. So first, I'm going to check the Python version available on the host, which is our INS093 node, so Python.
It is 3.9.21. And then I'll run the same command inside our container using Abtainer exec and check out what is the container's Python version. So singularity. And then I'm going to write the exec Python, name of the image, and then command, which is Python3-version.
Notice that we are executing Python from inside the container rather than relying on the host environment. The versions may be different, as you can see. The host version was 3.9, and then the container's Python version is 3.11. All right.
So now let's take it one step further. For example, if I want to let me clear my screen. So if I want to know the operating system of my host, it should be so here you can see that it is running on Red Hat 9.3. And then if I do the same, but this time I'm going to use the shell of the container instead of using the exec command.
So I'm going to use singularity, shell, and then Python. So it will take me to inside the container, and I will check the version. Also, as compared to the OS release, look at the difference.
So inside the container, it is Debian operating system, and my host operating system was Red Hat Enterprise Linux. So these commands help demonstrate that we are using the container software environment while still running on HPC compute node. And finally, I'm going to type exit to exit out of this container.
The key takeaway is that researchers can run applications with their required versions without modifying the shared HPC operating system requesting the administrative privileges. So as a bonus, I want to show you a useful command that allows you to customize an existing container image. As we discussed earlier, singularity images in SIF format are immutable, meaning they are read only, and you cannot directly modify them after downloading.
So I'm going to use the sandbox functionality, singularity, build, sandbox, and then name of my file, directory, where I want to customize my image, and then name of the image for whom I'm taking advantage of. All right. So we can then use the hyphen hyphen writable.
So singularity shell. If I want to modify this image, so I want to be in writable mode. Okay.
And then I want to work temp directory of my image, and then my box. Okay. All right.
So this warning can be ignored. Sometimes the permissions messes up with this. Okay.
Let me clear this screen. Okay. For the demonstration purposes, I can do.
All right. And then I'm going to use this one. Okay.
There we are. So a little tweaks from my end, because I'm working on a compute node without having the root privileges. So let me create a directory, or maybe create a file, test.txt. So now that I have made my changes inside the writable sandbox, the final step is to package it back into a singularity image, since this is the executable kind of directory.
So I'm going to exit out from this, and then I'm going to build the image with my changes, having this test.txt file in it. So singularity build and Python. So I'm going to just give a name of 3.11, and then custom with my changes, and then name of my directory from where I'm trying to create this image from.
All right. So this basically creates a portable read-only image that includes all my customizations and can be deployed across compatible environments. Keep in mind that you may not be able to install additional packages or make certain modifications directly on HPC cluster, but you can build and customize your container on your own laptop, workstation, or virtual machine where you have the necessary privileges.
Once your environment is ready, you can use a lift and shift approach to transfer the completed SIF image to our HPC scratch space and run it using singularity. So this allows you to take full advantage of our HPC computing resources without having to install or configure your software dependencies directly on the cluster. And that's one of the biggest advantages of containerization.
Build your environment once and deploy it whenever you need it with minimal changes. So let me explain what I have done previously in four steps. Pull, sandbox, modify, rebuild.
So first, choose a trustworthy upstream image. Second, convert it to a writable sandbox when customization is required. Third, install or configure what your research application needs.
And then fourth, build the result as SIF. The advantage is that you can move the resulting image to compatible system and run that same packaged software environment there. So it also helps to document which base image and changes produce the final result so the process can be repeated.
All right. So another more reproducible way to build a customized container image is by using a definition file. In this example, we have a file called myapp.def, which specifies the container's base image as compared to here where I already had a pulled image from the registry and then I basically made modifications and recreated the image.
But in this example, we have the base image definition file, the software packages we need to install, and the environment configuration. We can then build our container using the same singularity build and then name of the image and then the path of this definition file. But keep in mind, building images from definition files typically requires root privileges.
So the recommended approach is to build and customize your container on your own virtual machine or laptop, then transfer the completed SIF image to cluster where you can run and take advantage of our HPC resources. Moving on to next. So now that we have seen how to run Python inside a container interactively, let's take it one step further and see how we can use containers to run actual workloads on our HPC cluster.
As you already know, on our cluster, we use Slurm to submit jobs as we did before and request compute resources. The good news is that running a container through Slurm is not much different from submitting a regular batch job. We are still using standard Slurm submission script, which you can see.
The only difference is that instead of running our application directly on the host, we are running it inside a container. So let me walk you through quickly through this example. At the top, we have our usual Slurm directives.
We are giving the job name, selecting the partition, requesting one hour of execution time. And you also notice that hyphen hyphen GRESS GPU one. This tells Slurm that our job needs one GPU, which makes sense because in this example, we are running a machine learning workload.
And next, we are loading the AppTainer module just like we did in our example before. And this makes the container runtime available within our job environment. And here is the thing getting interesting that instead of installing PyTorch, Python, and all their dependencies directly on the cluster, we are simply using singularity exec to run our Python training script inside a prebuilt container image.
And hyphen hyphen NV flag, basically, that's particularly important for GPU workloads because it allows the container to access the NVIDIA GPU and compatible driver libraries for the host. So, we are then passing our training script, train.py, along with a couple of parameters, such as number of training epochs and batch size, and that's pretty much it. We let Slurm handle the resource allocation while AppTainer handles the software environment.
So, imagine you're a researcher working with a specific version of PyTorch or TensorFlow, you don't have to worry about whether that exact version is installed on our cluster. As long as you have a compatible container image and the necessary resources, you can submit your workload through Slurm. And the best part, you don't need to have root privileges.
You don't need to modify anything on the host operating system. That's really where container becomes powerful on HPC, where we are combining Slurm's resource management with obtainer's portability software environment, making it much easier to run complex resource workloads. Another thing I wanted to mention was bind paths.
So far, we have seen how to run applications inside containers. But here's a question. What happens when your application needs to access files or datasets stored on the HPC cluster? Let's say you're running a Python application inside a container, but your research data is sitting in your Scratch directory on Insomnia.
How does the container access that data? That's where the bind paths come in. Think of a bind path as a bridge between the host system and the container. It allows us to make a directory from the host available inside the container without actually copying files into the container image.
So, let's take a look at first example. We are using Singularity exec with the hyphen, hyphen bind option and slash data colon slash MNT. The first path slash data represents a directory on host system, while slash MNT represents where that directory will appear inside the container.
So, when Python runs slash MNT run.py inside the container, it's actually accessing a file stored under slash data on the host, which is, in my case, INS093. So, the file stays on the host. We are simply making them accessible from inside the container.
And now, what if we want to make multiple directory available? That's where the second example demonstrates. We are binding both slash Scratch and slash projects into our PyTorch container. And because we are using the same paths on both sides, we don't need to specify a separate destination for each one.
So, this is particularly useful in HPC environment, where researchers often have large data sets stored in shared storage or Scratch space. Instead of packaging hundreds of gigabytes of research data into a container, we keep the container focused on the software environment and use bind paths to access our existing data. And here's another important point.
Any files your application writes to are writable. Bind host directory are saved directly to that directory. So, even after the container exits, your output files remain available on the HPC file system.
Also, keep in mind that bind paths don't give you additional permissions. If you don't have permission to access a directory on the host, you won't magically gain access to it from inside the container. And one last thing worth mentioning, AppTainer automatically binds certain directories by default, such as your home directory, just like it did if I go inside the container again.
So, if I do LS, all of the directories are already binded. I mean, so, it automatically binds certain directories, as I mentioned, such as my home directory and current working directory, depending on the system configuration. So, you won't always need to specify hyphen, hyphen bind manually.
So, the main takeaway is simple. The container provides you a software environment, while bind paths give that software access to your research data. And that's how we can keep our applications portable without having to move or duplicate large datasets every time you run a container.
All right. So, that brings us to the end of today's workshop. Before we wrap up, I want to quickly revisit three main takeaways from today's session.
First, portability. One of the biggest advantages of using containers is that we can package our applications, libraries, dependencies into a single image and move that image between different environments. As we demonstrated earlier, we were able to run Python 3.11 inside a container, even though our HPC compute node was running Python 3.9. We didn't have to install anything on the host or make changes to operating system.
And then, second is reproducibility. So, this is particularly important in research. Imagine you have spent months working on a project, and six months later, you need to re-run the same analysis.
By that time, some of the software packages or dependencies on the system might have changed. So, with containers, we can preserve the software environment used for our research, making it much easier to reproduce our work later. And then, finally, ease of deployment.
We have seen that we don't necessarily need to install every application directly on HPC cluster. We can pull an existing container image, run our application inside it, and even integrate containers with Slurm to execute our workloads on compute nodes. And for those who need additional customization, we can build or modify containers in an environment where we have the necessary permissions, then bring the finished image over to the cluster.
So, if there's one thing I would like you to remember from today's workshop is that containers allow us to focus more on our research and less on configuring and troubleshooting software environments. And that's really the value of Aptainer in an HPC environment. That concludes the material I had planned for today.
So, thank you all for joining, and I hope you found this session useful. And I'm happy to take any questions if you might have.
The focus is practical, how we take a software environment that works in one place and run it consistently on an HPC cluster. We will start with concepts, compare singularity with Docker and then walk through the pulling the images and running the containers. By the end, you should recognize the key commands and understand where they fit into SLURM workflow.
So you do not need to be an expert in the containers to follow along, but familiarity with Linux commands and connecting to Insomnia will help. I will explain what each command is doing before we use it and if you are following along with the demonstration, please keep your terminal open so you can practice the commands as we go and if you are able to follow along, if you are unable to follow along for any reason, don't worry, a recording of this session will be available for you to review and practice at your own pace later. If you have any questions or if something is unclear at any point, please feel free to post your questions in the chat.
One of my colleagues will be monitoring the chat and will be able to assist you. All right. So here's our plan.
First, we will discuss why containers exist and the problems they solve, particularly software dependencies and reproducibility. Then we will look at common image registries and compare Docker with singularity. After that, we will move to Insomnia, obtain compute node allocation, load the module, and work from scratch storage.
And finally, we will explore pulling and running images, making modifications with a sandbox, writing definition files and integrating containers with modules and Slurm. The first half explains the why and the second half shows how. All right.
So what are containers? Think of a container as a packaged application environment. It can include the application, its runtime, libraries, and other dependencies. So you do not have to reconstruct that environment on every machine.
The container is portable because its software stack travels with it. Although the host hardware and kernel still matter. In singularity, a common finished image format is the single file SIF or .SIF. One important distinction, a container is not a virtual machine.
It shares the host system's kernel while isolating the application's user space environment. All right. So why containers? It comes into mind for the researchers that often researchers face the familiar problem.
It works on my laptop but not on the cluster. A different Python version, missing library, or conflicting dependency can stop a workflow. So containers help by packaging the software stack together.
They also support reproducibility. A fixed image makes it easier to return to the same environment later, though the inputs, hardware, and settings still matter. Excuse me.
On a shared APC cluster, singularity is especially useful because users can run containerized software without ordinary privileges. So the value is not just convenience, it's portability, control, a more predictable research environment. Excuse me.
So before building anything, we need to understand what are the container registries. Check whether a suitable image already exists. A container registry is a place where people and organizations publish prebuilt images.
Docker Hub has a general purpose software. Red Hat, QA, and GitHub Container Registry are also common sources. And media NGC is particularly relevant to GPU and machine learning workloads.
And Biocontainers focuses on scientific and bioinformatics tools. The typical HPC approach is to find a trusted starting image, customize it if necessary, and run it with singularity on the cluster. We should not treat every public image as trustworthy.
Review its origin, documentation, and contents before using it with research data. All right. So let's compare Apple to Apple.
Docker and singularity both use containers. But their typical environments differ. Docker is widely used for cloud services, application deployment, and image building workflows as compared to singularity or Attainer.
It is designed around the shared HPC systems where users don't have administrative access. It works naturally with command line, research programs, and can expose the host's GPU and MPI environment when configured properly. A key HPC benefit is that the user's identity carries into the container, helping file ownership and permissions behave as expected.
A common pattern is to prepare an image with Docker compatible tools, then run that image with singularity on the cluster. Docker itself also supports rootless modes. So the contrast here is about the workshop's typical deployment model.
All right. So using singularity on HPC, let's connect to the concepts to the practical work. So the most important rule is to separate a login activity from compute work.
So login nodes are for access and live preparation. Computationally intensive container tasks belong on compute nodes. So we request resources through Slurm using an interactive job for testing or a batch job for production.
Commands such as srun or slr provide that allocated environment. And once the allocation starts, we can load Attainer and run the container without affecting other users on shared login node. So for example, we begin on Insomnia login node.
Let me start with which Attainer. So in this example, it returns no Attainer message. So that is the behavior for the demonstration.
So do not start any hands-on container workload here, and it won't let you. So which basically only checks whether the command can be found. It does not allocate any compute resources or load any module.
And our next step is basically to ask this Slurm for an interactive compute node. All right. So let me go ahead and run the srun command here.
And then I'm going to explain each part of this. All right. So let me break down this what this command is doing.
Srun basically asking Slurm to start a job step. And then hyphen hyphen pty gives us an interactive terminal. Hyphen t is basically requesting one hour.
Hyphen a RCS specifies the account, and slash bin bash launches our shell. So Slurm may report that the job is waiting for resources. Once allocated, the shell prompt changes to a compute node hostname, such as in this case, I have INS093.
So that changed hostname tells us we are no longer working on the login node. So excuse me. So let's load the module.
And if we wanted to load that module on the login node, it would prevent you to do that, because there is kind of a preventive measure. So here on the compute node, I'm going to load that obtainer. And how can I check it? And if I run the same command, which obtainer.
So now you can see that this example points to the shared obtainer 1.4.5 installation. At this point, we have followed the essential sequence, login, request compute resources, load the module, and verify the executable. And now we are ready to work with images.
But before pulling a large image, move to the appropriate scratch space. The home directories have tighter quotas and may be less suitable for large container files. And I owe heavy work.
So your scratch path will be different. And here's mine. And then up until here, you will have the same.
But probably for you, it should be RCS. And also when you're asking for the compute node, you should use free account. All right.
And then next is my username. No, user's directory. And then my username.
All right. So if I do pwd, it gives me that I'm here on the scratch space. So we want our images, our temporary downloads, working files here in the high capacity workspace.
But also remember that the scratch is not a suitable place for backed up or long-term storage. So my next slide will give us the command to pull an image. So now we pull a singularity image, Docker image through singularity.
All right. So while the image is being pulled from the registry, let me walk you through what's happening behind the scenes. So the Docker call and slash slash prefix, it tells singularity to retrieve an OCI or Docker format image from a registry.
And then singularity converts it into its SIF format and writes the file such as Python 3.11.SIF in the current directory. So this is the key convenience. We can use the large ecosystem of published Docker images without running the Docker services on the cluster.
And by default, singularity cache is under this path, which is shown here. Cache location or cache allocation matters when home quarters are limited. The active container configuration may use container prefix equivalence.
It's going to take a little bit of time, maybe one or two minutes to pull the image. All right. So meanwhile, it is pulling the image.
Let me go over to the next slide. So there are three common ways to use an image. One is interactive shell inside the container, which is useful for exploration and troubleshooting.
Then singularity exec basically runs one specific command inside the container and returns control when it finishes. And then there is a third one, which is singularity run. It invokes the image's predefined run function or run action and commonly supplied by a run script in its definition file.
And we'll get to the definition files a little later. So for scheduled research workflows, exec is often especially clear because the exec command appears in this run script. We'll see how that works later.
It's taking a little longer than anticipated. There you go. So if I do LS, I have my SIF image.
So now that we have covered how to pull the container image, let's look at practical example of how researchers can benefit from Abtainer on an HPC cluster. So for example, imagine you're working on a research project that requires a different version of Python. For example, it requires 3.11, but the Python version available on the cluster is different.
Normally, you might need to install another version, configure the dependencies, or ask the HPC administrators for help. With Abtainer, we can use a Python environment that's already packaged inside a container without installing anything on the cluster or requiring any root privileges. So first, I'm going to check the Python version available on the host, which is our INS093 node, so Python.
It is 3.9.21. And then I'll run the same command inside our container using Abtainer exec and check out what is the container's Python version. So singularity. And then I'm going to write the exec Python, name of the image, and then command, which is Python3-version.
Notice that we are executing Python from inside the container rather than relying on the host environment. The versions may be different, as you can see. The host version was 3.9, and then the container's Python version is 3.11. All right.
So now let's take it one step further. For example, if I want to let me clear my screen. So if I want to know the operating system of my host, it should be so here you can see that it is running on Red Hat 9.3. And then if I do the same, but this time I'm going to use the shell of the container instead of using the exec command.
So I'm going to use singularity, shell, and then Python. So it will take me to inside the container, and I will check the version. Also, as compared to the OS release, look at the difference.
So inside the container, it is Debian operating system, and my host operating system was Red Hat Enterprise Linux. So these commands help demonstrate that we are using the container software environment while still running on HPC compute node. And finally, I'm going to type exit to exit out of this container.
The key takeaway is that researchers can run applications with their required versions without modifying the shared HPC operating system requesting the administrative privileges. So as a bonus, I want to show you a useful command that allows you to customize an existing container image. As we discussed earlier, singularity images in SIF format are immutable, meaning they are read only, and you cannot directly modify them after downloading.
So I'm going to use the sandbox functionality, singularity, build, sandbox, and then name of my file, directory, where I want to customize my image, and then name of the image for whom I'm taking advantage of. All right. So we can then use the hyphen hyphen writable.
So singularity shell. If I want to modify this image, so I want to be in writable mode. Okay.
And then I want to work temp directory of my image, and then my box. Okay. All right.
So this warning can be ignored. Sometimes the permissions messes up with this. Okay.
Let me clear this screen. Okay. For the demonstration purposes, I can do.
All right. And then I'm going to use this one. Okay.
There we are. So a little tweaks from my end, because I'm working on a compute node without having the root privileges. So let me create a directory, or maybe create a file, test.txt. So now that I have made my changes inside the writable sandbox, the final step is to package it back into a singularity image, since this is the executable kind of directory.
So I'm going to exit out from this, and then I'm going to build the image with my changes, having this test.txt file in it. So singularity build and Python. So I'm going to just give a name of 3.11, and then custom with my changes, and then name of my directory from where I'm trying to create this image from.
All right. So this basically creates a portable read-only image that includes all my customizations and can be deployed across compatible environments. Keep in mind that you may not be able to install additional packages or make certain modifications directly on HPC cluster, but you can build and customize your container on your own laptop, workstation, or virtual machine where you have the necessary privileges.
Once your environment is ready, you can use a lift and shift approach to transfer the completed SIF image to our HPC scratch space and run it using singularity. So this allows you to take full advantage of our HPC computing resources without having to install or configure your software dependencies directly on the cluster. And that's one of the biggest advantages of containerization.
Build your environment once and deploy it whenever you need it with minimal changes. So let me explain what I have done previously in four steps. Pull, sandbox, modify, rebuild.
So first, choose a trustworthy upstream image. Second, convert it to a writable sandbox when customization is required. Third, install or configure what your research application needs.
And then fourth, build the result as SIF. The advantage is that you can move the resulting image to compatible system and run that same packaged software environment there. So it also helps to document which base image and changes produce the final result so the process can be repeated.
All right. So another more reproducible way to build a customized container image is by using a definition file. In this example, we have a file called myapp.def, which specifies the container's base image as compared to here where I already had a pulled image from the registry and then I basically made modifications and recreated the image.
But in this example, we have the base image definition file, the software packages we need to install, and the environment configuration. We can then build our container using the same singularity build and then name of the image and then the path of this definition file. But keep in mind, building images from definition files typically requires root privileges.
So the recommended approach is to build and customize your container on your own virtual machine or laptop, then transfer the completed SIF image to cluster where you can run and take advantage of our HPC resources. Moving on to next. So now that we have seen how to run Python inside a container interactively, let's take it one step further and see how we can use containers to run actual workloads on our HPC cluster.
As you already know, on our cluster, we use Slurm to submit jobs as we did before and request compute resources. The good news is that running a container through Slurm is not much different from submitting a regular batch job. We are still using standard Slurm submission script, which you can see.
The only difference is that instead of running our application directly on the host, we are running it inside a container. So let me walk you through quickly through this example. At the top, we have our usual Slurm directives.
We are giving the job name, selecting the partition, requesting one hour of execution time. And you also notice that hyphen hyphen GRESS GPU one. This tells Slurm that our job needs one GPU, which makes sense because in this example, we are running a machine learning workload.
And next, we are loading the AppTainer module just like we did in our example before. And this makes the container runtime available within our job environment. And here is the thing getting interesting that instead of installing PyTorch, Python, and all their dependencies directly on the cluster, we are simply using singularity exec to run our Python training script inside a prebuilt container image.
And hyphen hyphen NV flag, basically, that's particularly important for GPU workloads because it allows the container to access the NVIDIA GPU and compatible driver libraries for the host. So, we are then passing our training script, train.py, along with a couple of parameters, such as number of training epochs and batch size, and that's pretty much it. We let Slurm handle the resource allocation while AppTainer handles the software environment.
So, imagine you're a researcher working with a specific version of PyTorch or TensorFlow, you don't have to worry about whether that exact version is installed on our cluster. As long as you have a compatible container image and the necessary resources, you can submit your workload through Slurm. And the best part, you don't need to have root privileges.
You don't need to modify anything on the host operating system. That's really where container becomes powerful on HPC, where we are combining Slurm's resource management with obtainer's portability software environment, making it much easier to run complex resource workloads. Another thing I wanted to mention was bind paths.
So far, we have seen how to run applications inside containers. But here's a question. What happens when your application needs to access files or datasets stored on the HPC cluster? Let's say you're running a Python application inside a container, but your research data is sitting in your Scratch directory on Insomnia.
How does the container access that data? That's where the bind paths come in. Think of a bind path as a bridge between the host system and the container. It allows us to make a directory from the host available inside the container without actually copying files into the container image.
So, let's take a look at first example. We are using Singularity exec with the hyphen, hyphen bind option and slash data colon slash MNT. The first path slash data represents a directory on host system, while slash MNT represents where that directory will appear inside the container.
So, when Python runs slash MNT run.py inside the container, it's actually accessing a file stored under slash data on the host, which is, in my case, INS093. So, the file stays on the host. We are simply making them accessible from inside the container.
And now, what if we want to make multiple directory available? That's where the second example demonstrates. We are binding both slash Scratch and slash projects into our PyTorch container. And because we are using the same paths on both sides, we don't need to specify a separate destination for each one.
So, this is particularly useful in HPC environment, where researchers often have large data sets stored in shared storage or Scratch space. Instead of packaging hundreds of gigabytes of research data into a container, we keep the container focused on the software environment and use bind paths to access our existing data. And here's another important point.
Any files your application writes to are writable. Bind host directory are saved directly to that directory. So, even after the container exits, your output files remain available on the HPC file system.
Also, keep in mind that bind paths don't give you additional permissions. If you don't have permission to access a directory on the host, you won't magically gain access to it from inside the container. And one last thing worth mentioning, AppTainer automatically binds certain directories by default, such as your home directory, just like it did if I go inside the container again.
So, if I do LS, all of the directories are already binded. I mean, so, it automatically binds certain directories, as I mentioned, such as my home directory and current working directory, depending on the system configuration. So, you won't always need to specify hyphen, hyphen bind manually.
So, the main takeaway is simple. The container provides you a software environment, while bind paths give that software access to your research data. And that's how we can keep our applications portable without having to move or duplicate large datasets every time you run a container.
All right. So, that brings us to the end of today's workshop. Before we wrap up, I want to quickly revisit three main takeaways from today's session.
First, portability. One of the biggest advantages of using containers is that we can package our applications, libraries, dependencies into a single image and move that image between different environments. As we demonstrated earlier, we were able to run Python 3.11 inside a container, even though our HPC compute node was running Python 3.9. We didn't have to install anything on the host or make changes to operating system.
And then, second is reproducibility. So, this is particularly important in research. Imagine you have spent months working on a project, and six months later, you need to re-run the same analysis.
By that time, some of the software packages or dependencies on the system might have changed. So, with containers, we can preserve the software environment used for our research, making it much easier to reproduce our work later. And then, finally, ease of deployment.
We have seen that we don't necessarily need to install every application directly on HPC cluster. We can pull an existing container image, run our application inside it, and even integrate containers with Slurm to execute our workloads on compute nodes. And for those who need additional customization, we can build or modify containers in an environment where we have the necessary permissions, then bring the finished image over to the cluster.
So, if there's one thing I would like you to remember from today's workshop is that containers allow us to focus more on our research and less on configuring and troubleshooting software environments. And that's really the value of Aptainer in an HPC environment. That concludes the material I had planned for today.
So, thank you all for joining, and I hope you found this session useful. And I'm happy to take any questions if you might have.