Scaling Your Research with HPC | 2026 HPC Training Series #1
Transcript
Okay, so it's about time we will begin. So, welcome to Research Computing Services intro to HPC training services, session one, an introduction to scaling your research with HPC. I am Al Tucker, and I will be your instructor for this session.
I am part of Research Computing Services High Performance Computing Group. And you can ask questions of the HPC group by emailing hpc-support at columbia.edu. Also, as the session goes along, feel free to ask questions in the chat. There are two other members of the HPC group here who can answer your questions in the chat.
And actually, I'll give them each a chance to introduce themselves right now before we move on.
Hi, I'm Helena. I will be answering your questions should you have any.
Hi, my name is Waqas. I'm part of the HPC team as well, and I'll be answering your questions.
Okay, and also, after the session is over, there will be time for Q&A as well.
So, you can also wait till then to ask questions live if you wish, instead of just typing. So, anyway, let us move on. So, a brief overview of what this session is.
This is a beginner session, and it's meant to provide an introduction to what an HPC is, how one is used, both here at Columbia and in the world beyond. There is no prior knowledge assumed beyond having just used a computer at any point. No account on a cluster is required.
There will be no hands-on exercises. Also, this session will be recorded and placed online for your later review at a link, which will be emailed to you after the class. So, first, what is an HPC? How does it differ from consumer deathtops that you're used to? So, nearly all high-performance computing clusters run on the Linux operating system, and this includes ours.
Which is built on a version called Red Hat Linux. There are other versions of Linux more suitable to use at home, such as Rocky Linux, Ubuntu, Linux Mint, and more. All these versions of Linux are what's called open source, meaning that they're free to download and install on your computer.
The standard way to use a Linux system is through the command line interface, or the CLI. So, the CLI is similar to what you see when you're using the terminal application on macOS, or Windows Subsystem for Linux 2, also called WSL2, which is present in Windows 10 or higher. Since some skill and comfort with the command line is required to work on the cluster, all of the following courses in this series teach commands and techniques to bring you up to speed on the CLI.
Exploring the command line on your own computer can be a stepping stone to more easily mastering the CLI on the HPC. So, a good way to prepare yourself for the CLI is by doing your normal, everyday tasks on your laptop using the Mac terminal, or WSL2, instead of using the typical mouse and menu graphical interface. Ordinary things, such as creating a text edit or notepad file or making a new folder, can be done on your own laptop using the terminal or WSL2, instead of using the graphical interface.
And these are two examples of doing the same thing under the Mac or WSL2, making a folder and then creating a text file within that folder. You can Google to find many tutorials to learn your own command line, and these are just some examples here using the Mac terminal, WSL2. Also, there's something called PowerShell, which is present on Windows.
It's very specific to Windows, but it is similar in a certain sense, and that's something you can use on Windows as well. These are three different links for tutorials, but there are many others out there. You can just Google them and follow whichever one you please, if you want.
So, to further define exactly what is an HPC, a high-performance computing cluster is composed of rooms full of hundreds or thousands of computers, each one many times faster than your desktop or laptop. And they're all networked together to solve large, complex problems in unison, much faster than any single machine. Another term you may hear, high-throughput computing, is similar, but focuses on repeatedly doing large numbers of smaller, independent tasks.
The focus is to solve a large volume of simpler things faster. Now, this isn't necessarily an either-or situation. A computing cluster can do both kinds of tasks, and Columbia's HPC system can do both.
It all depends on what type of problem you choose to solve. Another term you may hear is supercomputer, and it's actually somewhat misleading. A supercomputer is not one big computer.
Not anymore. We're way past the 50s. But the term refers to the top HPC clusters in the world.
They're often custom-built at great expense to solve very specific problems. At the heart, though, a supercomputer is merely a top-tier, elite HPC. Another way an HPC differs from your typical laptop or desktop is that an HPC is a shared environment that many people use at one time.
Now, one thing to handle this sharing is something called a workload manager, which is employed on the cluster. There are different types, but the workload manager we use in our clusters is called SLURM. When you have a task or a job you want to run on the cluster, you submit the job to SLURM, and SLURM keeps track of how many other people are using the cluster at the same time and how many resources each person is using.
So what this means in the shared environment is that when you submit a job, your job might not start immediately. It could even start hours later. It depends on how much power you want to use for your job relative to how much power other people are using.
When your job does start, though, it will run hundreds of times faster than it ever could on your laptop or desktop. So each one of the many networked computers is called a node, and the ones that actually do the jobs are compute nodes. So these nodes are commonly reached through one or more front-end login nodes, which control access to the cluster.
So you can think of the login nodes as the lobby controlling access to the theater of compute nodes, where the show actually goes on. The compute nodes in each cluster work in parallel with each other, boosting processing speed to deliver high-performance computing. So here is a simplified diagram of our clusters.
So here's you, and you can be in your office on campus or at home, and you reach the login node remotely through something we call SSH. Your initial landing on the cluster is the login node, the lobby of the theater. And then you have access to all of the compute nodes that are all networked together behind the login node, and this is where your jobs will actually run.
And all of these, through a high-speed network, are connected to a centralized storage. This storage, since it's connected over the network, lets you have the exact same home directory on the login node that you do on any compute node. So you don't need to move files around from one node to the other.
Any node you use, your files are just there. So more about storage. The typical storage on an HPC is also far different from the types you commonly know.
I mean, you're all familiar probably with hard disks, SATA disks, external disks, NVMe SSDs. These are all things you find in laptops and desktops, and HPC is much more complex when it comes to storage. Storage on a typical HPC is composed of distinct storage array nodes, and here we see four of them.
Each one is filled with multiple high-capacity disks. All of them work in tandem, connected over that specialized internal network that we pointed out in the other diagram. So all of these provide one vast amount of space, and that space can be partitioned or sectioned off by the cluster operating system as needed.
So the format of the disks is also very different. On your personal computer, your disk has a file system. And here are some of the ones that you may be used to now.
On Windows, FAT32 is common for the format of a disk. Mac OS uses APFS. Linux uses something called EXT4.
And these are all ways of organizing data on the disk so the operating system can make sense of the data and save and retrieve files. In HPC, this is much more complex. HPCs generally use what's called a parallel file system.
And this is a simplified diagram of a parallel file system. This is a type of system that allows individual parts of any single file to be written across multiple disks. All of the disks work in parallel, allowing multiple compute nodes to more easily read and write data at higher speeds.
So what are the benefits of a parallel file system? Well, there are many different types. What we use is something called, or I should say what we principally use, is something called GPFS, which stands for General Parallel File System. So some of the benefits of a GPFS file system is you have high throughput, so multiple processes can read and write data simultaneously.
It scales well, so thousands of nodes can read and write petabytes of data, and performance scales up both with compute nodes and with storage. You get reduced I.O., input-output bottlenecks, because this distributes the input-output load across multiple disks and servers, as we saw in the previous diagrams of many disks working together. You also have concurrent access to many different applications or many data files by multiple users.
So this ensures data consistency and integrity during parallel file system operations. GPFS or any parallel file system is optimized for HPC and AI workloads, like simulations, training large models, or big data analytics, and it works well with job schedulers and MPI-based applications. Parallel file systems are also very fault-tolerant and redundant, so systems keep running smoothly, even if one disk or even multiple disks fail.
You don't lose data, and to the user, it's entirely transparent. Also, parallel file systems are very efficient at large file handling. They excel at reading files with simulation outputs, large data sets, and splitting and distributing huge files across disks.
So next, we're going to talk about the difference between a parallel file system and something you may all be more familiar with, an NFS mount. So NFS, or a network file system, is something that you may already be familiar with because it's commonly used within labs or even at home settings, and it's a way to share files across multiple computers. Parallel file systems are things that are more common on HPCs, and again, as I go over this model, it's important to realize this is all in very general.
We do have some usage of NFS on our HPC clusters, and it's for a very specific reason, and as our model grows over time, we want you to phase that out. However, in general, a parallel file system is something that's designed for high performance and large scale workloads, whereas NFS is just a general purpose type of file sharing system. Also, the input-output parallel file systems excel at letting data being read and written to multiple clients, whereas NFS tends to be serialized.
It's for reading and writing data to a single server in general. You have much better performance on a parallel file system, whereas the single server model of NFS can limit your bandwidth and make things slower, especially if you're doing large numbers of reads and writes. Parallel file systems scale up to thousands of nodes, whereas NFS in general is better for smaller medium networks.
Some types of parallel file systems are GPFS, as we mentioned before. Also, something we use on one of our clusters is something called Lustre, which is just another type of parallel file system. Other things out there in the world you may hear of are BGFS and DAOS.
DAOS are different types of parallel file systems. NFS, generally you're going to find out in the world. NFS version 4, which is the most current, you may find NFS version 3. 2 and 1 are just too old.
You wouldn't be seeing those running out in the wild, but these are some examples of different file systems. Parallel file systems, more redundant, may offer failover options. NFS typically has a single point of failure.
So if a single place from which the files are being served goes awry, you may lose access to the files. Parallel file systems also split your files across multiple disks. NFS, in general, data resides on one server.
Parallel file systems are also much better for concurrent access, meaning reading multiple files more at one time. NFS can struggle under high concurrency. Parallel file systems, in general, are the things you will see in HPCs.
NFS file systems are things that most often you will see in smaller environments. And again, this is a very general summary. As I said, we do have some usage of NFS in our HPC for a specific reason.
And over time, we do aim to phase that out. So now let's talk about how you access an HPC. You gain access to an HPC through something called secure shell or SSH.
And it was that dotted line that was between you and the login node in the previous diagram. So below is an example of using SSH in the macOS terminal. So first, I'm on my local computer.
And then I use SSH to reach Insomnia, which is actually our latest cluster. After accessing Insomnia, you can see things don't change much. They look very much the same.
But one way to tell you're on an entirely different computer after SSH is that your prompt line, that's the line with the dollar sign, changes. So initially, you see I'm on my local computer. After SSH-ing, this is the way to tell that you're on a remote computer.
Here it's saying I'm under my user ID. We call them unis here, but on other systems, it's just called a user ID. And specifically, I've logged into the second login node of the Insomnia cluster here.
You can also tell you're on an entirely different computer because the files that you see when you use the listing command are going to be entirely different. So the cloud is an example of an HPC. Cloud computing at its heart is an HPC, but it's used for a different purpose.
So behind things like Facebook, Instagram, X, or Blue Sky are similar massive groups of individual node computers all working together. When you submit a job on the HPC consisting of some type of scientific calculation, a job on Instagram would be your message to someone or a post on their wall. Storage on an HPC, which you would use to hold research data, would function in a similar way.
Only in the cloud, storage is holding pictures of your cat or a party or whatever you decide to put up there. But the concept is very similar. Also, at its heart, online gaming is an HPC.
The interface may look different, but you log into an HPC using a username and a password, the same way you would log into an online game with a username and password. Instead of research data, the data stored is the statistics of your character in the game and their place in the online world. Jobs in an online game are your directions to move, kick, shoot, or talk to another character, instead of a job on an HPC, which would be like a direction to a program like MATLAB to run a scientific calculation.
Now we're going to talk about some research uses of HPCs here at Columbia. In general, HPC tends to serve one of two purposes. An HPC can process calculations that you could do on your laptop.
Only the HPC does them so much faster, you can get vastly more work done in a short time period. Another thing with an HPC, because of the large processing power and vast amounts of storage, HPCs also enable complex processing of large data sets. So you can do things on an HPC that would not be possible any other way.
So some use cases here at Columbia. Alex Urban's group in the Department of Chemical Engineering looks at the various aspects of batteries and energy storage, and what new materials can be used for this purpose, or how existing materials can be made more efficient. In one collaboration with a colleague, this group examines hydrolyzing seawater to get oxygen.
Seawater being incredibly impure makes this a very difficult problem to solve. Various additives can be used for this job, but the process is so complex that running physical experiments to exhaustively test every combination of useful additive just is not practical. So they use GPUs on the HPC to train and refine models to stimulate these experiments thousands of times to narrow down which combinations have the most promise to work in the real world.
Also, I want to make one aside here. The term GPU, it stands for Graphics Processing Unit, because originally GPUs were designed for things like video games to move characters around on the screen much faster. It was only later that people realized that moving data around at high speeds was something that could be applicable to research data.
Now, your problems have to be put in a different format, a specific format, to be useful for a GPU. But once you can, then a GPU can solve problems much faster, and these days GPUs are primarily used in situations like research. Also things like Wall Street, where they need to move a lot of data and process a lot of data very fast.
But anyway, getting back to Columbia. This kind of work that Alex Urban's group does is something that even on a powerful laptop, it would just take so, so long that advances in their field would be very few and over much longer periods of time. Other uses here at Columbia.
Professor Molly Przorski's group in the Department of Biological Sciences focuses on the genetics of evolution. Different lab members focus on different aspects. Will Milligan studies the evolution of mutation and primates.
His work involves analyzing very large data files and thousands of small cluster jobs at a time. Hannah Munby looks for commonalities among large amounts of raw genetic sequencing data. This is a memory intensive process called variant culling.
The high memory and storage requirements far exceed what a consumer laptop or desktop could fulfill. Misha Langley looks at gene expressions in water buffalo for the purpose of examining dubious claims made in previous studies. The extremely large amounts of data require the storage and speed of an HPC to work with.
Other usage cases. Professor Ying Wei and her Five Sigma group in the Department of Biostatistics do single cell genomics work on the mental health aspects of various diseases under grants through the Columbia Data Science Institute. Under a specific grant related to Alzheimer's disease, they try to find links among various types of data gathered under different circumstances.
Clinical data, medical data, observational data, self-reported data, drug interaction data, and more. Since, especially in the field of Alzheimer's, this data is all gathered under very different circumstances with different end goals in mind. Even though it all falls under Alzheimer's research, trying to find commonalities to create useful models about the progression of the disease is a massive task that simply is not possible without an HPC.
Lab member Zhilu takes advantage of the short partition, which is something on the cluster that allows a user limited access to every single node in the cluster to debug code for later use during long runs on the specific nodes that their group owns. Luca Cammaso is a PI in the multi-messenger science group in the Department of Astrophysics. His focus is studying plasma in the vicinity of black holes to determine the types of photons, neutrinos, cosmic rays, or other possible particles that may exist in the vicinity of a black hole.
Now, this is the kind of work that can't be physically manipulated in a lab. So what the cluster does is it allows him to run processor and memory intensive simulations on huge data sets that only an HPC could handle. His resource requirements are in fact so large that even with access to the HPC here at Columbia, larger jobs need to be run on even more powerful clusters at places like NASA or Lawrence Livermore Laboratories.
Now, besides Columbia's HPC clusters, there's some other resources that as a Columbia member, you can access. And one thing is called Empire AI. Now, it was first announced in April 2024 and what Empire AI is is a statewide initiative backed by the governor.
Its purpose is to establish next generation GPU-only computing clusters specifically for AI research done by educational institutions in the state of New York. Now only a select number of institutions are part of Empire AI and Columbia University was one of the first of these and you can find out more about Empire AI by visiting the CUIT website at this URL. There are additional workshops, events, and trainings that are also held here at Columbia.
One such entity is the Foundations of Research Computing which holds events here at Columbia. You can see them at this URL here and it holds different workshops or boot camps. Also there's a Python user group that meets in Butler Library.
Also part of the libraries is Research Data Services and this is a URL here. We can see the services that they provide. They have data clubs on Python and R and also a map club about GIS data and this brings us to the end of our introductory session.
If you want to find out more, have any other questions, want an account on one of our clusters, you can email the HPC group at hpc-support at columbia.edu. So now that we've come to the end of the session, I haven't been able to keep track of the chat as I've been lecturing so if there are any questions that people have to answer, you can put them in the chat or unmute yourself and just ask your question live and if I have an answer, I'll be happy to share it with you.
I am part of Research Computing Services High Performance Computing Group. And you can ask questions of the HPC group by emailing hpc-support at columbia.edu. Also, as the session goes along, feel free to ask questions in the chat. There are two other members of the HPC group here who can answer your questions in the chat.
And actually, I'll give them each a chance to introduce themselves right now before we move on.
Hi, I'm Helena. I will be answering your questions should you have any.
Hi, my name is Waqas. I'm part of the HPC team as well, and I'll be answering your questions.
Okay, and also, after the session is over, there will be time for Q&A as well.
So, you can also wait till then to ask questions live if you wish, instead of just typing. So, anyway, let us move on. So, a brief overview of what this session is.
This is a beginner session, and it's meant to provide an introduction to what an HPC is, how one is used, both here at Columbia and in the world beyond. There is no prior knowledge assumed beyond having just used a computer at any point. No account on a cluster is required.
There will be no hands-on exercises. Also, this session will be recorded and placed online for your later review at a link, which will be emailed to you after the class. So, first, what is an HPC? How does it differ from consumer deathtops that you're used to? So, nearly all high-performance computing clusters run on the Linux operating system, and this includes ours.
Which is built on a version called Red Hat Linux. There are other versions of Linux more suitable to use at home, such as Rocky Linux, Ubuntu, Linux Mint, and more. All these versions of Linux are what's called open source, meaning that they're free to download and install on your computer.
The standard way to use a Linux system is through the command line interface, or the CLI. So, the CLI is similar to what you see when you're using the terminal application on macOS, or Windows Subsystem for Linux 2, also called WSL2, which is present in Windows 10 or higher. Since some skill and comfort with the command line is required to work on the cluster, all of the following courses in this series teach commands and techniques to bring you up to speed on the CLI.
Exploring the command line on your own computer can be a stepping stone to more easily mastering the CLI on the HPC. So, a good way to prepare yourself for the CLI is by doing your normal, everyday tasks on your laptop using the Mac terminal, or WSL2, instead of using the typical mouse and menu graphical interface. Ordinary things, such as creating a text edit or notepad file or making a new folder, can be done on your own laptop using the terminal or WSL2, instead of using the graphical interface.
And these are two examples of doing the same thing under the Mac or WSL2, making a folder and then creating a text file within that folder. You can Google to find many tutorials to learn your own command line, and these are just some examples here using the Mac terminal, WSL2. Also, there's something called PowerShell, which is present on Windows.
It's very specific to Windows, but it is similar in a certain sense, and that's something you can use on Windows as well. These are three different links for tutorials, but there are many others out there. You can just Google them and follow whichever one you please, if you want.
So, to further define exactly what is an HPC, a high-performance computing cluster is composed of rooms full of hundreds or thousands of computers, each one many times faster than your desktop or laptop. And they're all networked together to solve large, complex problems in unison, much faster than any single machine. Another term you may hear, high-throughput computing, is similar, but focuses on repeatedly doing large numbers of smaller, independent tasks.
The focus is to solve a large volume of simpler things faster. Now, this isn't necessarily an either-or situation. A computing cluster can do both kinds of tasks, and Columbia's HPC system can do both.
It all depends on what type of problem you choose to solve. Another term you may hear is supercomputer, and it's actually somewhat misleading. A supercomputer is not one big computer.
Not anymore. We're way past the 50s. But the term refers to the top HPC clusters in the world.
They're often custom-built at great expense to solve very specific problems. At the heart, though, a supercomputer is merely a top-tier, elite HPC. Another way an HPC differs from your typical laptop or desktop is that an HPC is a shared environment that many people use at one time.
Now, one thing to handle this sharing is something called a workload manager, which is employed on the cluster. There are different types, but the workload manager we use in our clusters is called SLURM. When you have a task or a job you want to run on the cluster, you submit the job to SLURM, and SLURM keeps track of how many other people are using the cluster at the same time and how many resources each person is using.
So what this means in the shared environment is that when you submit a job, your job might not start immediately. It could even start hours later. It depends on how much power you want to use for your job relative to how much power other people are using.
When your job does start, though, it will run hundreds of times faster than it ever could on your laptop or desktop. So each one of the many networked computers is called a node, and the ones that actually do the jobs are compute nodes. So these nodes are commonly reached through one or more front-end login nodes, which control access to the cluster.
So you can think of the login nodes as the lobby controlling access to the theater of compute nodes, where the show actually goes on. The compute nodes in each cluster work in parallel with each other, boosting processing speed to deliver high-performance computing. So here is a simplified diagram of our clusters.
So here's you, and you can be in your office on campus or at home, and you reach the login node remotely through something we call SSH. Your initial landing on the cluster is the login node, the lobby of the theater. And then you have access to all of the compute nodes that are all networked together behind the login node, and this is where your jobs will actually run.
And all of these, through a high-speed network, are connected to a centralized storage. This storage, since it's connected over the network, lets you have the exact same home directory on the login node that you do on any compute node. So you don't need to move files around from one node to the other.
Any node you use, your files are just there. So more about storage. The typical storage on an HPC is also far different from the types you commonly know.
I mean, you're all familiar probably with hard disks, SATA disks, external disks, NVMe SSDs. These are all things you find in laptops and desktops, and HPC is much more complex when it comes to storage. Storage on a typical HPC is composed of distinct storage array nodes, and here we see four of them.
Each one is filled with multiple high-capacity disks. All of them work in tandem, connected over that specialized internal network that we pointed out in the other diagram. So all of these provide one vast amount of space, and that space can be partitioned or sectioned off by the cluster operating system as needed.
So the format of the disks is also very different. On your personal computer, your disk has a file system. And here are some of the ones that you may be used to now.
On Windows, FAT32 is common for the format of a disk. Mac OS uses APFS. Linux uses something called EXT4.
And these are all ways of organizing data on the disk so the operating system can make sense of the data and save and retrieve files. In HPC, this is much more complex. HPCs generally use what's called a parallel file system.
And this is a simplified diagram of a parallel file system. This is a type of system that allows individual parts of any single file to be written across multiple disks. All of the disks work in parallel, allowing multiple compute nodes to more easily read and write data at higher speeds.
So what are the benefits of a parallel file system? Well, there are many different types. What we use is something called, or I should say what we principally use, is something called GPFS, which stands for General Parallel File System. So some of the benefits of a GPFS file system is you have high throughput, so multiple processes can read and write data simultaneously.
It scales well, so thousands of nodes can read and write petabytes of data, and performance scales up both with compute nodes and with storage. You get reduced I.O., input-output bottlenecks, because this distributes the input-output load across multiple disks and servers, as we saw in the previous diagrams of many disks working together. You also have concurrent access to many different applications or many data files by multiple users.
So this ensures data consistency and integrity during parallel file system operations. GPFS or any parallel file system is optimized for HPC and AI workloads, like simulations, training large models, or big data analytics, and it works well with job schedulers and MPI-based applications. Parallel file systems are also very fault-tolerant and redundant, so systems keep running smoothly, even if one disk or even multiple disks fail.
You don't lose data, and to the user, it's entirely transparent. Also, parallel file systems are very efficient at large file handling. They excel at reading files with simulation outputs, large data sets, and splitting and distributing huge files across disks.
So next, we're going to talk about the difference between a parallel file system and something you may all be more familiar with, an NFS mount. So NFS, or a network file system, is something that you may already be familiar with because it's commonly used within labs or even at home settings, and it's a way to share files across multiple computers. Parallel file systems are things that are more common on HPCs, and again, as I go over this model, it's important to realize this is all in very general.
We do have some usage of NFS on our HPC clusters, and it's for a very specific reason, and as our model grows over time, we want you to phase that out. However, in general, a parallel file system is something that's designed for high performance and large scale workloads, whereas NFS is just a general purpose type of file sharing system. Also, the input-output parallel file systems excel at letting data being read and written to multiple clients, whereas NFS tends to be serialized.
It's for reading and writing data to a single server in general. You have much better performance on a parallel file system, whereas the single server model of NFS can limit your bandwidth and make things slower, especially if you're doing large numbers of reads and writes. Parallel file systems scale up to thousands of nodes, whereas NFS in general is better for smaller medium networks.
Some types of parallel file systems are GPFS, as we mentioned before. Also, something we use on one of our clusters is something called Lustre, which is just another type of parallel file system. Other things out there in the world you may hear of are BGFS and DAOS.
DAOS are different types of parallel file systems. NFS, generally you're going to find out in the world. NFS version 4, which is the most current, you may find NFS version 3. 2 and 1 are just too old.
You wouldn't be seeing those running out in the wild, but these are some examples of different file systems. Parallel file systems, more redundant, may offer failover options. NFS typically has a single point of failure.
So if a single place from which the files are being served goes awry, you may lose access to the files. Parallel file systems also split your files across multiple disks. NFS, in general, data resides on one server.
Parallel file systems are also much better for concurrent access, meaning reading multiple files more at one time. NFS can struggle under high concurrency. Parallel file systems, in general, are the things you will see in HPCs.
NFS file systems are things that most often you will see in smaller environments. And again, this is a very general summary. As I said, we do have some usage of NFS in our HPC for a specific reason.
And over time, we do aim to phase that out. So now let's talk about how you access an HPC. You gain access to an HPC through something called secure shell or SSH.
And it was that dotted line that was between you and the login node in the previous diagram. So below is an example of using SSH in the macOS terminal. So first, I'm on my local computer.
And then I use SSH to reach Insomnia, which is actually our latest cluster. After accessing Insomnia, you can see things don't change much. They look very much the same.
But one way to tell you're on an entirely different computer after SSH is that your prompt line, that's the line with the dollar sign, changes. So initially, you see I'm on my local computer. After SSH-ing, this is the way to tell that you're on a remote computer.
Here it's saying I'm under my user ID. We call them unis here, but on other systems, it's just called a user ID. And specifically, I've logged into the second login node of the Insomnia cluster here.
You can also tell you're on an entirely different computer because the files that you see when you use the listing command are going to be entirely different. So the cloud is an example of an HPC. Cloud computing at its heart is an HPC, but it's used for a different purpose.
So behind things like Facebook, Instagram, X, or Blue Sky are similar massive groups of individual node computers all working together. When you submit a job on the HPC consisting of some type of scientific calculation, a job on Instagram would be your message to someone or a post on their wall. Storage on an HPC, which you would use to hold research data, would function in a similar way.
Only in the cloud, storage is holding pictures of your cat or a party or whatever you decide to put up there. But the concept is very similar. Also, at its heart, online gaming is an HPC.
The interface may look different, but you log into an HPC using a username and a password, the same way you would log into an online game with a username and password. Instead of research data, the data stored is the statistics of your character in the game and their place in the online world. Jobs in an online game are your directions to move, kick, shoot, or talk to another character, instead of a job on an HPC, which would be like a direction to a program like MATLAB to run a scientific calculation.
Now we're going to talk about some research uses of HPCs here at Columbia. In general, HPC tends to serve one of two purposes. An HPC can process calculations that you could do on your laptop.
Only the HPC does them so much faster, you can get vastly more work done in a short time period. Another thing with an HPC, because of the large processing power and vast amounts of storage, HPCs also enable complex processing of large data sets. So you can do things on an HPC that would not be possible any other way.
So some use cases here at Columbia. Alex Urban's group in the Department of Chemical Engineering looks at the various aspects of batteries and energy storage, and what new materials can be used for this purpose, or how existing materials can be made more efficient. In one collaboration with a colleague, this group examines hydrolyzing seawater to get oxygen.
Seawater being incredibly impure makes this a very difficult problem to solve. Various additives can be used for this job, but the process is so complex that running physical experiments to exhaustively test every combination of useful additive just is not practical. So they use GPUs on the HPC to train and refine models to stimulate these experiments thousands of times to narrow down which combinations have the most promise to work in the real world.
Also, I want to make one aside here. The term GPU, it stands for Graphics Processing Unit, because originally GPUs were designed for things like video games to move characters around on the screen much faster. It was only later that people realized that moving data around at high speeds was something that could be applicable to research data.
Now, your problems have to be put in a different format, a specific format, to be useful for a GPU. But once you can, then a GPU can solve problems much faster, and these days GPUs are primarily used in situations like research. Also things like Wall Street, where they need to move a lot of data and process a lot of data very fast.
But anyway, getting back to Columbia. This kind of work that Alex Urban's group does is something that even on a powerful laptop, it would just take so, so long that advances in their field would be very few and over much longer periods of time. Other uses here at Columbia.
Professor Molly Przorski's group in the Department of Biological Sciences focuses on the genetics of evolution. Different lab members focus on different aspects. Will Milligan studies the evolution of mutation and primates.
His work involves analyzing very large data files and thousands of small cluster jobs at a time. Hannah Munby looks for commonalities among large amounts of raw genetic sequencing data. This is a memory intensive process called variant culling.
The high memory and storage requirements far exceed what a consumer laptop or desktop could fulfill. Misha Langley looks at gene expressions in water buffalo for the purpose of examining dubious claims made in previous studies. The extremely large amounts of data require the storage and speed of an HPC to work with.
Other usage cases. Professor Ying Wei and her Five Sigma group in the Department of Biostatistics do single cell genomics work on the mental health aspects of various diseases under grants through the Columbia Data Science Institute. Under a specific grant related to Alzheimer's disease, they try to find links among various types of data gathered under different circumstances.
Clinical data, medical data, observational data, self-reported data, drug interaction data, and more. Since, especially in the field of Alzheimer's, this data is all gathered under very different circumstances with different end goals in mind. Even though it all falls under Alzheimer's research, trying to find commonalities to create useful models about the progression of the disease is a massive task that simply is not possible without an HPC.
Lab member Zhilu takes advantage of the short partition, which is something on the cluster that allows a user limited access to every single node in the cluster to debug code for later use during long runs on the specific nodes that their group owns. Luca Cammaso is a PI in the multi-messenger science group in the Department of Astrophysics. His focus is studying plasma in the vicinity of black holes to determine the types of photons, neutrinos, cosmic rays, or other possible particles that may exist in the vicinity of a black hole.
Now, this is the kind of work that can't be physically manipulated in a lab. So what the cluster does is it allows him to run processor and memory intensive simulations on huge data sets that only an HPC could handle. His resource requirements are in fact so large that even with access to the HPC here at Columbia, larger jobs need to be run on even more powerful clusters at places like NASA or Lawrence Livermore Laboratories.
Now, besides Columbia's HPC clusters, there's some other resources that as a Columbia member, you can access. And one thing is called Empire AI. Now, it was first announced in April 2024 and what Empire AI is is a statewide initiative backed by the governor.
Its purpose is to establish next generation GPU-only computing clusters specifically for AI research done by educational institutions in the state of New York. Now only a select number of institutions are part of Empire AI and Columbia University was one of the first of these and you can find out more about Empire AI by visiting the CUIT website at this URL. There are additional workshops, events, and trainings that are also held here at Columbia.
One such entity is the Foundations of Research Computing which holds events here at Columbia. You can see them at this URL here and it holds different workshops or boot camps. Also there's a Python user group that meets in Butler Library.
Also part of the libraries is Research Data Services and this is a URL here. We can see the services that they provide. They have data clubs on Python and R and also a map club about GIS data and this brings us to the end of our introductory session.
If you want to find out more, have any other questions, want an account on one of our clusters, you can email the HPC group at hpc-support at columbia.edu. So now that we've come to the end of the session, I haven't been able to keep track of the chat as I've been lecturing so if there are any questions that people have to answer, you can put them in the chat or unmute yourself and just ask your question live and if I have an answer, I'll be happy to share it with you.