Navigating HPC with Linux | 2026 HPC Training Series #3
Transcript
Max: Good afternoon, everyone. Welcome to the third installment of our intro series. I wanted to give a quick introduction before I hand the mic over to Waqas, who's going to be your host this afternoon. So my name is Max Shortte, I am the manager of the HPC team at CUIT. My team and I are responsible for serving and delivering high-performance computing to the research community here at Columbia University. We look after everything from the physical notes, such as the login, execute, transfer, and head nodes. The network layers, which consist of InfiniBAN and Ethernet, as well as physical storage. We also service the software stack, from Globus for large-scale data transfers, anything module-related, such as R, CUDA, Conda, MATLAB. As well as Slurm, which is a scheduler, and our provision systems, which are Confluence and Bright.
So, if you look at this first slide here… Waqas, go back one more slide for me. We currently have 3 clusters in operations. That's Terremoto, which is our oldest cluster, Ginsburg, which is our largest cluster, and Insomnia, which is our newest cluster. Now, if you were to view these three clusters of one resource pool, we'd be pushing close to 25,000 CPU cores. 250 GPUs, 160TB of error correcting RAM, as well as 500 nodes.
Next slide, Waqas. So…This slide actually draws down to Insomnia, which is our newest cluster, and the reason why I wanted to focus on this current cluster is because, number one, this is the cluster you guys are going to be navigating on this afternoon. Number two, Insomnia is our newest cluster, and it's the only cluster that we'd be investing in going forward and expanding. So currently, Insomnia has 92 nodes, 14,500 CPU cores, 60TB of RAM, and 92 GPUs. Next slide, Waqas. So, this is a breakdown of insomnia by department representation. Currently, we have 26 accounts, 26 partitions supporting 700 accounts. Currently, system biology has the largest footprint with 21%, but we have a solid representation from social science, statistics, and engineering. So what does that mean? That means insomnia has been widely adopted by a wide range of disciplines across Columbia University, which is… which is good for us. So, guys, next slide?
So, before we get started, and I pass the mic over to Waqas, I just wanted to give a couple of rules, house rules. Number one, these sessions will be recorded, so you have the luxury now to actually follow along and execute later on this afternoon, once we hand out the video and the slides. Number two, if you have any questions, please paste them in the Zoom chat window. One of my team will actually address those questions for you. And number three, Waqas will be reserving 15 minutes at the end of the session for you guys to voice any questions at all. So, Waqas I pass this over to you.
Waqas: Thank you, Max. Good introduction. My name is Waqas Hanif, and I'm part of the HPC group here at Columbia. Actually, I'm very excited to do this for you folks. So, first off, what is Linux? Linux is an operating system that manages hardware and software, allowing users to interact with their computer. It's known for being open source, secure, and highly customizable. On Windows or Mac OS, Linux is free and comes in many different versions, called distributions.
So, if you… you're seeing these icons, these are basically multiple distribution icons. First is Ubuntu, Red Hat, Fedora, and, I think Rocky Linux. So, it powers everything from servers and supercomputers to smartphones, embedded devices, you name it. In short, Linux is everywhere, even if you don't realize you're using it. So why Linux? It's open source, free access to the source code, allows customization and transparency, it's flexibility. It runs on everything from desktops to servers and embedded systems, security, strong permission management, and fast community-driven updates. Stability. Rarely crashes with long uptime, making it ideal for critical systems. Performance-wise, efficient, multitasking, and great for handling heavy workloads. Obviously the cost. Many distributions are free, so it reduces the software expenses. And also, community support. It is large, active community, providing help and resources.
So why I chose Linux, when I started with Linux, my main goal was to host my own website. As a student, I didn't have the budget for expensive Windows Server licenses. So, Windows required costly software like IIS for web hosting, approximately $2,000, and Microsoft SQL Server for databases, around $4,000. So, Linux provided a free open-source alternative with Apache for web hosting and MySQL for database, both widely used and community-supported. Plus, Linux is known for its strong security and reliability, making it a great long-term choice. So by choosing Linux, I had full control over my system without spending a dime on licensing. So I hope that everyone has the access to the cluster. You'll have to log into your nodes, or you'll have to SSH to the nodes, if I say correctly. So that's how you do it. Your uni at insomnia… sorry, sshuni at insomnia.rcs.columbia.edu. That's how you're gonna SSH to the system, which is basically gonna give you the access to the cluster. So, I suppose everyone has the access. And this workshop is hands-on, so I'm gonna start with the hands-on, commands. So we start with the… with the system information. As I suppose, we have the access to the server. So, first of all, host… Name. It basically displays the name of the system. It is useful for identifying the machine, especially in networks. So next we're gonna see uptime. It shows how long the system has been Running. It's great for, checking the system's stability. And, of course, date, it prints basically the current system date and time. It helps in scheduling tasks or verifying system time settings. So, moving on to the next command. Pwd. So… It basically is a print working directory. So, it will… It's a simple but essential Linux command. It displays the full path of the directory you are in currently. This is useful when navigating the file system, especially in complex directory structures. And, no, I'm gonna go with another one.
Yeah, so what is directory structure? Basically, it is known as a file system hierarchy, standard FHS. All files and directories appear under the root directory, which starts with slash, even if they're, like, stored in different physical or virtual devices. Okay, and then I'm gonna move on to the next slide. Okay, so wherever you are, no matter what, on the system, the CD command is used to change the directory, and it will take back to The parent directly or home directory. It's just gonna… Okay, so if I do a CD tilde, maybe I'm gonna… go to Downloads, if I want to go to the downloads, is that… Change the directory, which is CD, And downwards, and it's gonna… And if I want to print that directly where I am right now. It's gonna give me the full path of the directory. So CD basically moves you… From here to the other directories. Chain directory is the short for… CD is the short for the chain directory. It is a quick way to navigate to the parent directory of all the directories on the system. So if I hit CD again, and PWD… It basically took me to my home directory, which is my parent directory. All right, and then if we want to go to the CD slash, it will take me to the root directory, which is the The top of the directories, top of the line. Probably you won't be able to, It's… it depends on, the access of the user, and so…
We, as the system admins, we have the full privileges, so that's why we can go to the root directory. Okay, moving on to the next. How can you create a directory? So, the mkdir command is used to create the directories. So I'm gonna create a directory here. I'm gonna create a test directly. Oops, not… Inappropriate directory, which is my home directory. I'm jumping a, awesome. Just go to dis… I maybe ran out of the inot numbers. Okay, so I have successfully created the test directly, and let's test out. Let me look here… Alright, so the test directory is created using the MKDIR, which is make directory, and the test directory. And then we're gonna go to the next command, which is basically Touch. Touch is basically the command used to create an empty file. Or update the timestamp of an existing file. So I'm gonna create a… file a name. my file. dot TXT. I'm fighting. Want to check out. that file. here is my file.txt, so it basically created, file. Later, I wanted to show you how to list the files, but since I'm using it, you can see that ls command is basically used to list the files or directories in a present working directory. So ls is, it's… Stands for, listing, listing a directory or a file. Alright, so, moving on to the next one… Oh, I didn't miss anything. Okay, so if I want to remove file or directory, I just… Right, RM. Which is… which stands for remove. And then… I'm gonna remove this directory. Since I'm removing the directory. I'm gonna use a flag, which is hyphen R, so that it recursively removes the directory and its own, kind of, files in it, or whatever, in it. the content in it. So, for files, I'm gonna use only RM. So let's remove this myfile.txt RM. And my file.txt. And it basically will ask, do you want to remove? And I'll probably hit Y, it's gonna move it. And if I want to check if it was removed or not, myfile.txt is no more. And if I use RM… Test. Basically, it will… it will prompt me that you cannot remove the directory without this flag, which is hyphen R. So, it says RM cannot be removed. Test is a directory, so I'm gonna remove it using the hyphen R flag. And then… If I hit enter, it will ask, do you want to remove this? IPS, or by remove it. To check it, just LS. Alright, so that's how we remove the files using the RL. As well as the directories using the rm-r. flag. So, this is the flag we use to remove the batteries. Alright, so we have gone through PWD, CD. So, basically, if I am in my… any of the directories…
Sam Cho: Waqas increased cost to.
Waqas: Okay, thank you. Thank you, Sam. Okay, so if I am in my home directory, and if I go to CD Downloads, which is 10 directory to downloads, and… Let's see, what do we have here? And if I go… another directory, which is MPM. Oh, it's not a direct user file. Let's create a directory. Test 1, test 2, and I'm gonna go into Test2 directory. So, CD dot dot, it's basically gonna take you back to one directory above, whichever directory you are. So, see, for example, if I'm in, test directory. So, in theory, cd. is gonna take me back to downloads. See, dot dot… Resident Directory… So it took me one directory back. And if I am… Gonna go back to Test 2, if I just hit CD, it should take me back to my home directory, which is the parent directory, which is WH2612, And if I want to check PWD, There I am. So there's a difference in CD dot dot, and then CD. And also, you can achieve the same results, if you are into CD downloads, test… And, PWD to check out where I am. If I want to go back to home, I just… By tilde. Which is gonna take me back to my home directory, which is WH2612. Oops, PWT… I am, so… CD or CD tilde, to take you back to your home directory. CD dot dot is just to take you back one directory above. That's the difference. Alright, we talked about… Changing the directory… CD, PWD, CD tilde… Alright, so I have just… Showed you how to change the directory to test. I'm gonna skip this slide. All right, so, clear here, path completion. What is it? So, if… if I want to, basically. If I don't want to write down the whole path of the directory, I just use the tab button. How can I do that? Because I need to know… Okay. So, see, for example, if we have this downloads, and I just don't want to type the whole download, so what I'm gonna… what I'm gonna do is that, CD, and then… uppercase T, and then hit tab. So I have two. with the D. One is desktop, one is downloads. So… I'm gonna… write down one more word, which is the O, and then hit tab. So it's gonna complete the whole, whole word. Whole path. So… Basically, tab completion is to auto-complete the file or directory name starting with D. And, let's say if I want to go… into another directory, which is desktop, so I'm gonna hit DE, and then space, sorry, Tab. is gonna complete the whole path. So, CD into desktop, So, it is very useful. When you're using… when you're navigating into the, different directories or into the paths, it's very helpful. Alright, so… I hope I didn't say anything.
Okay, so LS, I told you that this is to list the directory or the files. Ls-l. So, every command has multiple, kind of, flags. So, for listing. it has another flag, which is hyphen L. Hyphen L is for a long listing, so I'm gonna show you the difference. If I do LS, as I did before. It is listing the content of the directory. And if I hit LS hyphen L, It gives me more information about that content. So, for example, it's giving me the permissions, who's the owner, who's the group, what date it was created, and multiple different things. So, ls… this command simply shows you what's inside your directory, ls-l", if you need more details, this command gives you a deeper look at your files. Okay… did that. Alright, so yeah, as I was talking about it. So, if you check out, this… this is basically the information about the user that owns the file. And then, if the next is the group that owns this file. So sometimes you have to give the permission to a whole group who can access the file, manipulate the file. So, you have to kind of assign a group to that file, or a directory. So this is… this is what that information is, that who is the… who is the owner… who is the owner and the group of that file, that file belongs to. And of course, the size of the file, this is basically the size of the file. And the directories. And next is basically the last time the file was updated or created. So, this section shows that this file was created on April 3. All right, and lastly, yeah, of course, the name of the file. All right, so another, basically, flag is that ls-A, it lists all files, including the hidden ones. So, if I… here, you don't see any hidden file. And how can I say that? Because there is no dot Kind of in front of… At the beginning of the file name. So if I want to do LS-A, You are seeing the difference? And if I… To the long listing of the hidden files, or all files. So, some of the files now you will be seeing are, basically, it starts with the… with the dot dot, so… This option reveals the files and directories that are normally hidden. For example, some of the system files, some of the environmental files, If I want to create any… Well, let's not create the hidden file. That's… But anyway, LS hyphen A, A for all, and L for longlist. And yes, of course, we have combined two options, two flags, all together, so that, I mean, we can achieve the same goal by using ls-a-L, But I just combined both of the flags with only one hyphen, so… Both of them are right. Alright, so moving on to the next… Alright, so how can we copy a file? I'm gonna create a file here. Let's… Clear out the clutter, by using the clear command. and the screen will be clear out, yes, this is another command. And I'm gonna create a file. Name… Hmm, that's fine. And if I want to see if the file was created. And the test file is created.
Okay. One more thing before we move forward is that if we do ls-l. One thing I want to share is that How to… how to find out that which one is the directory and which one is the file? Is just looking at these… a small, kind of, either it's a D or a hyphen. A hyphen represents it's a file, a D represents it's a directory. And then sometimes there is a L, which is… which basically shows that this is a link file, which is a deeper concept, but just wanted to let you know that. How can we know… how can I know that what is… what is the file, what is the directory? So, my test file was created. Here. And then if I want to move this test file, or copy this test file into a test directory, which is here. Okay, so I'm gonna go into the test directory first to see if… Test file is not here, so this, basically, directory is an empty directory. And if I go back one more directory, ls… And if I want to copy this file, which is test file, so I'm gonna use CPMANT, And then test… bottle. So, path of the file is… this one. basically, I am in the current directory, that's why I'm just giving the name of the file, that's… That's another thing I will show you later, that how can we give the paths Like, absolute path or the relative path. But, so, CP… path of the file, and then the destination where I want to put that file to. So I just want to… Paste this file in test directory. I should write. Put it in the downloads. So I was in a different directory. Hold on. So I'm gonna create a file again here. doc.txt… And cptcast.txt to downloads. Hit enter. And then I'm gonna go to Downloads to test out. If the file was… Copied over. So test.txt was copied over. So I created the test.txt in my home directory. And then I copied over that test.txt into downloads folder or directory. And then I listed… If… and to make sure that the file was copied over, which is test.txt. Alright, so… That's the… Okay, and if I want… if I want to remove the file, of course, RM has done it. BXT. It's gonna prompt me. If I want to remove it or not, yes? And to make sure if it was removed, LS. So I'm gonna go back to my home directory just to… Clear off the confusions. Here I am. Present working directory is my home directory. Alright, so that's how you remove the file. And then, if I want to… so basically, before, I copied over test.txt. to downloads directly. But what if I don't… want to keep it here in my home directory. If I want to move the file, so I'm gonna use the command named MV, which is move. And then test.txt. to downloads. Okay. Hit enter. And I'm gonna go back to downloads. Alas. test.txt, which I removed before, is here, and if I go back to my home directory. It is not here anymore. So that's how you move the files. From one path to another path. From one location to another location. Yes, we can move, So, yeah, from MVCommand, we can… move the files or directories. Also, we can rename them. So, see, for example, if I go back to downloads. And if I want to rename test.txt to something else. So I've just used mv command and test dot txt, to my file. Three. dot TXG. And if I want to test it out. So, this test.txt got converted into my file 3.txt. All right, so we can achieve two goals with MVCommand. One is to move the files or directories from one location to another, and then other kind of a goal is we can rename the files and directories using the MV command. Alright?
So… Now we're gonna move… towards reading a file, but using the cat is another command, so I'm gonna go back one more step. And we… I have a file here named Columbia.txt, and if I want to read the file. I'll just type cat and columbia.txt, and I'm extensively using tab command. Tab-tab, sorry. And basically, it will print out whatever inside the file is on the screen. That's how we read the files, not… All the time, but sometimes, if it's a small file. Alright, so, moving on to the next command. And, so, yes, ID and groups. So, for example, if I want to type just ID. and my uni, if I don't type it, it will gonna show my username. from wherever I'm logging. wrong. So, if I do ID… For you, it will show maybe some intro series groups. For me, it's showing a different, view. So, basically, it shows you, it basically displays the details about the user, including their user ID, their group ID, And, any additional groups they belong to. So for… for you folks, you probably are Belonging to one group, or maybe two or three, it depends. Another thing is, groups. Okay, a simpler way to see which groups a user belong to. All right Okay, I'm gonna clear out the clutter. And then I'm gonna move on to… for example, if I want to see what is my username? I'll just type in, who am I? And it will print out that… The… what is my username? Okay? And there is a… there is a different who am I, which is who space M space I. It basically shows the user who started the session, which can be different if you have switched users. For example, I switched from root user to my username, or basically I… asked this resource, which is the INS040, it's a node, I asked, but… Who am I is the go-to command. Okay, moving on to the next. There is another command, who? Who basically… why is it not showing anything? Okay, so let's say… Let me get my privileges… There's not showing me anything. Okay, who basically prints the, who lists all the current, logged-in users, along with their session details? So, if I do one more… It's basically, it's gonna show you that, All the users currently logged in, various session details, such as terminal, login time, remote host. So I was logged into, a different kind of, node. It doesn't have any… doesn't have any… other users, so that's why it was not showing anything, but… Who basically shows you all of the details, who is currently logged into the system, and when, and which IP phone.
Okay, there's another command… ATSW. It's basically… It gives more detail about the… basically, it shows who is logged in and what they're doing, including their login time, idle time. The command they are currently running. So it is kind of a… gives you more in-depth, view of who is logged in and what are they doing. So there are different columns, like user, terminal, or device the user is logged in from, for example, TTY, PDS. And then their login time, when they logged in, idle time, the amount of time the user has been idle for. JCPU, the time used for all processes associated with the user terminal, and then the PCPU is the time used by the current process running under that terminal. And then the process ID is PID, we call it PID. So the ID of the user is currently Basically, the current session up terminal. Alright, so… Another useful command is basically MAN. MAN is short for manual. It is used to access the manual pages for a command, providing detailed information on its usage, options, and syntax. For example, I'm gonna… If I want to learn more about the LS command, I just do man ls. It basically gives you what it does, description, flags, what flags I can use with the man, sorry, with the LS command. Different options. So basically, it is the… it is your go-to resource for learning about commands and their options directly from the terminal. You don't have to Google anything, you just type man, and then name, and then command, and then it will give you what you can do with that command. Alright, so file manipulation, basically, for example, I showed you earlier… this… Columbia.txt… So, for example, if I want to… particularly Fetch University word. From, from this whole file. what I will do is… Breath. And then… University… And name of the file. So, I'm using prep command. I want to use… I want to fetch this university from the whole, whole file. And it will show me, highlighted, that the university was here, here, and at this place. So, it is… it's a powerful search tool used to filter the text based on the patterns. Useful for scanning log files, configuration files, or large data sets. Okay, and then moving on to the pipes. So… Pipes basically allow chaining the commands. passing the output of one command as input to the other, and I'm going to show you how. This is basically useful for narrowing down large outputs, making it easier to find specific information So, let's see, how can we do CAT? columbia.txt, and then If I want to grab location, So basically. I am reading this file, and then… whatever output of that file was, I'm just piping it and using another command. Making the output of the first command as an input of the other command. And then that command is grep, and then it's gonna grep the location word. Okay? I don't have one, let's try University. Okay. So, before, I just grabbed the university word from Columbia.txt, but this time, I'm basically catting the Columbia.txt file, making the output of that file as an input of the grap command, and then fetching the university word. Okay, so it has basically… It's, it, it is, it is filtering the, the output. off the CAC mode. And filtering… the word university. It's showing the lines which only have the university. Alright? Okay, next command is tail and head. So, for example, if I… clear this out. Both of these head and tail are useful for previewing the large files without opening them. It's great for checking the beginning or end of the logs or configuration files. So, I'm gonna use head. Columbia.txt. So… By default, it shows the first 10 lines of the columbia.txt. Which includes the spaces between the lines. So… Head is basically printing out the first 10 lines, including these spaces, of that columbia.txt. And if I want to reduce that number, so what I'm gonna do is… and… oh, another thing I wanted to share was up arrow. So, for example, I have typed this command, and if I want to use that again, I just hit up arrow, and it's gonna basically, I mean, it's gonna bring the command again. op… I did not want it to do that. I'm gonna square it out. And then add. iPhone 5, I want to print the 5, first five lines at columbia.txt. So now it printed, basically, first 5 lines. 1, 2, 3, 4, 5. If I want to reduce that number… And tattoo. So that's how… it's basically printing out the first two lines. Of the… of the file. And same goes for the tail. it's gonna… print the last lines. By default, 10, and if I want to use the… only two lines of columbia.txt, and for tail, I'm going to use hyphen n for numbers. Okay, so last two lines are basically spaces. It's available can… Yeah, that's… So it basically printed the last pipelines. A tail and head is also useful for, kind of, reviewing the configuration files and the logs. End of the longs are the start of the longs. Output redirection. So, it is also a very, useful feature. The greater sign operator basically redirects output to a file. If the file exists, it replaces its content. It is useful for logging the timestamps or saving the command output for the later use. So for example, as I showed you before. Date command basically prints out the date and time of the system, so if I want to redirect the output of this date command, I'm just gonna type in date. And then this operator, and then if I want to give the name of the file, system date. And then, if I hit enter, LS… It basically created that file, and let's read this file, system date. And it basically have the content. Which is… The date of the system, date and time. All right? So, it is also very useful. Okay, output redirection… And a siphon L… Okay, yes. Just… this is kind of a… Repetitive.
Okay, so there is the difference between single… There is a difference between single operator and the double. So, basically, single is gonna create that file if that file does not exist. But it will replace the content of the file. Say, for example, if I want to print out ls-l, and… Redirect that to system date. It will basically remove this content, and it's gonna… Basically, add, or maybe, output of the LS-L to that System date command, system gate file. And then if I want to cat that… So, this file now… has the content of the LS hyphen L command. So… if I don't want to lose the… the current content, and… If I want to add more, I'll use the double redirection sign. So… If I want to use date and double redirection sign, So… This double redirection sign is gonna append the content. Which is, basically, it's gonna keep the content, off. the current… File, and then… add more to it. So, if I want to do CAD system date. So now, we have this whole content, plus the date and time of the system. So, double redirection signs append. Single redirection signs, basically, sign is gonna create the new content, or, remove the old content. That's, that's… what it is. Alright, so moving on to the next command, which is Echo. Echo is gonna… it is great for scripting, debugging, and weekly display values. So, see, for example, echo, hide. It's gonna print high. Echo, Columbia. Snapdragon Columbia. It's simple, yet essential for automating the tasks in Linux. So it is basically used for, displaying the messages, printing the variables, like, see, for example, if I do echo, Boom. So, it will… it is… it basically printed the variable home. It's the environmental variable, which basically is that path of my home directory. Alright. Moving on to the next one, which is multiple commands. So, see, for example, I want to combine two commands, or three commands, or four commands, so how can I achieve that? By this semicolon, which is, date, and then semicolon, and then if I want to do echo… Columbia. So it's gonna… print out Echo Columbia, ECHO is one command. And then date is another command. So… If I hit enter. It ran both commands in one go. See, for example, date. Colon, LS. column. What can we do, hotels? Let's see, for example… Mang… No, not BAM. For example. what else I can do? I'm thinking about easiest, command. Colombia… dot txt… And… it's gonna do all of it. So, it's gonna print the dates output, ls, and then it's gonna print the output of cat columbia.txt. So, the first is… The second is LS. And then the third output is for… cat columbia.txt. So we are… we have combined multiple commands in one go. Moving on to the system health. That's another thing, day to day. Basically, the tools… Which are essential for troubleshooting the servers. We rely on these tools daily, especially when servers are running slow or encountering the issues. For instance, if a server is sluggish, we start with these tools to pinpoint the problem, whether it's a resource usage, disk space, or network issues. So, knowing how to use them is crucial for effective troubleshooting. So, I'm going to start with 3 hyphen H. Let me clear this flutter. So, free. hyphen so free is the command, and then hyphen H stands for the human readable. And then it's gonna basically print the, detailed report on this, on your system's memory usage. It's M4M memory. So, as I mentioned, that hyphen-H option shows memory in human-readable format, which is, for example, MBs or GBs, while hyphen M shows it's it in megabytes. Okay, and then moving on to another command, which is top. It's also… Very powerful tool for system monitoring. Allowing you to spot performance issues quickly and manage processes. It's an essential tool for any Linux admin to keep track of the system health. It shows the real-time CPU usage, helping you track which processes are using the most resources. It lists all running processes and updates every 3 seconds by default. There are many options to customize the display while, like, sorting by memory or CPU usage. Queue for quitting. Okay, and then the edge top, it is more interactive. I'm gonna do it, actually… John Yep, as I was saying, that it provides a visually appealing color-coded interface with an intuitive layout, unlike top, which requires manual PID input, which is process ID input for actions. H stop basically allows you to scroll through processes and interact with them using arrow keys, making it easier to manage tasks. I'm not gonna go deeper into it because of the time constraints. But HDOP is also very useful. And other than that, VM stat. So, it basically… Let's… let's do it. Can be Amstacked. And it basically… It's a lightweight tool that provides a real-time snapshot of the system performance. It reports on CPU usage. memory swap, disk I.O, and system processes. Okay… P.S. Pss basically provides a full system-wide view of running processes, including persistent processes and those owned by the other users. Oh, P.S. And then there are multiple flags with PS, TSIF and E. It basically provides all, like, full system-wide view of the running processes, including the system process, and those owned by the other users. It is basically a useful command for admins to monitor the system. Activity, and then troubleshoot resource issues. PS hyphen AUX. Okay, PS hyphen AUX lists every process, with a lot more information. As you can see, it has multiple tabs, if I go on top of the… Too much. So yeah, it's, who's the user, what is the process ID, how much percent of the CPU memory is using virtual memory, and start time. And then what command they are using. A lot of, information. And one critical thing is inode numbers. So, in Linux, every file or directory is represented by an inode, which holds the metadata, like permissions, ownership, and size, but not the file name of the content. So, each file, each new file consumes one inode, so even if you have a disk space left, you might not be able to create more files if the inode limit is reached. So to check the iNode usage, we just use DF-I, It's gonna show me a lot of different things. Maybe human-readable, the Python, image. So, human readable, basically, it's gonna print you, the memory and usage, how many inodes I have used, how many inodes on the system, and how much are free, percentage of it, and for the mount systems, our mount points.
So, it basically is useful in environments with lots of small files, like logs or temporary data. Okay… Exit command is basically gonna lock you out of the system, or you can Press Ctrl and D. I hope… I was able to make it in one hour. Sorry if I was going too fast, because of the… we squeezed the 3-hour session, removing a lot of slides, to 1 hour. And then I try my best to show whatever I have. So, any questions?
Jessica: There is a question, is there… are there functions that can be operated on WinSCP directly?
Waqas: Winscp is, we can copy the files. We have used it in the past, but we highly encourage to use Globus for the large files. And then the, if you want to use… if you want to copy over the small files, you can use the SCP. Scp is the secure copy protocol. So… when SCP is not encouraged in the system. Correct me if I'm wrong, Max.
Jessica: Okay, I think that probably answers the user's question. If you… if you have any follow-ups, you're welcome to raise your… raise the… the follow-ups, directly here. Not you, Waqas, the user, sorry.
Waqas: Yep, here are some books that really helped me, both in learning the Linux and passing the Red Hat Certified Engineer, as well as Admin exam. So these books cover basically everything from basics to advanced topics, making them great for anyone, looking to dive and dive deep into the LX.
Halayn: And then what's the next class, Waqas?
Waqas: Next class, I am not sure about that. Jess, can you please…
Halayn: So with the scripting, it's scripting next, right? So you take these…and learn how to use them in scripts.
Jessica: Yeah, HPC jobs… scheduling HPC jobs with Bash is our next class, on Tuesday. I am going to… if anybody does have feedback on this course today, please let us know, and put the feedback form in our chat here, and I'll email it afterwards, but this class is a new… new way of presenting this information, and we want to hear from you guys if you have, suggestions, or if you liked it, appreciate it. I know there are some folks here that have been in the last semester, so it's great for that. Good to hear. But otherwise, Waqas is here for your questions, otherwise we will… we can close it out. Sure. Thank you, everyone, for joining in. Thanks, everybody. Have a good day.
So, if you look at this first slide here… Waqas, go back one more slide for me. We currently have 3 clusters in operations. That's Terremoto, which is our oldest cluster, Ginsburg, which is our largest cluster, and Insomnia, which is our newest cluster. Now, if you were to view these three clusters of one resource pool, we'd be pushing close to 25,000 CPU cores. 250 GPUs, 160TB of error correcting RAM, as well as 500 nodes.
Next slide, Waqas. So…This slide actually draws down to Insomnia, which is our newest cluster, and the reason why I wanted to focus on this current cluster is because, number one, this is the cluster you guys are going to be navigating on this afternoon. Number two, Insomnia is our newest cluster, and it's the only cluster that we'd be investing in going forward and expanding. So currently, Insomnia has 92 nodes, 14,500 CPU cores, 60TB of RAM, and 92 GPUs. Next slide, Waqas. So, this is a breakdown of insomnia by department representation. Currently, we have 26 accounts, 26 partitions supporting 700 accounts. Currently, system biology has the largest footprint with 21%, but we have a solid representation from social science, statistics, and engineering. So what does that mean? That means insomnia has been widely adopted by a wide range of disciplines across Columbia University, which is… which is good for us. So, guys, next slide?
So, before we get started, and I pass the mic over to Waqas, I just wanted to give a couple of rules, house rules. Number one, these sessions will be recorded, so you have the luxury now to actually follow along and execute later on this afternoon, once we hand out the video and the slides. Number two, if you have any questions, please paste them in the Zoom chat window. One of my team will actually address those questions for you. And number three, Waqas will be reserving 15 minutes at the end of the session for you guys to voice any questions at all. So, Waqas I pass this over to you.
Waqas: Thank you, Max. Good introduction. My name is Waqas Hanif, and I'm part of the HPC group here at Columbia. Actually, I'm very excited to do this for you folks. So, first off, what is Linux? Linux is an operating system that manages hardware and software, allowing users to interact with their computer. It's known for being open source, secure, and highly customizable. On Windows or Mac OS, Linux is free and comes in many different versions, called distributions.
So, if you… you're seeing these icons, these are basically multiple distribution icons. First is Ubuntu, Red Hat, Fedora, and, I think Rocky Linux. So, it powers everything from servers and supercomputers to smartphones, embedded devices, you name it. In short, Linux is everywhere, even if you don't realize you're using it. So why Linux? It's open source, free access to the source code, allows customization and transparency, it's flexibility. It runs on everything from desktops to servers and embedded systems, security, strong permission management, and fast community-driven updates. Stability. Rarely crashes with long uptime, making it ideal for critical systems. Performance-wise, efficient, multitasking, and great for handling heavy workloads. Obviously the cost. Many distributions are free, so it reduces the software expenses. And also, community support. It is large, active community, providing help and resources.
So why I chose Linux, when I started with Linux, my main goal was to host my own website. As a student, I didn't have the budget for expensive Windows Server licenses. So, Windows required costly software like IIS for web hosting, approximately $2,000, and Microsoft SQL Server for databases, around $4,000. So, Linux provided a free open-source alternative with Apache for web hosting and MySQL for database, both widely used and community-supported. Plus, Linux is known for its strong security and reliability, making it a great long-term choice. So by choosing Linux, I had full control over my system without spending a dime on licensing. So I hope that everyone has the access to the cluster. You'll have to log into your nodes, or you'll have to SSH to the nodes, if I say correctly. So that's how you do it. Your uni at insomnia… sorry, sshuni at insomnia.rcs.columbia.edu. That's how you're gonna SSH to the system, which is basically gonna give you the access to the cluster. So, I suppose everyone has the access. And this workshop is hands-on, so I'm gonna start with the hands-on, commands. So we start with the… with the system information. As I suppose, we have the access to the server. So, first of all, host… Name. It basically displays the name of the system. It is useful for identifying the machine, especially in networks. So next we're gonna see uptime. It shows how long the system has been Running. It's great for, checking the system's stability. And, of course, date, it prints basically the current system date and time. It helps in scheduling tasks or verifying system time settings. So, moving on to the next command. Pwd. So… It basically is a print working directory. So, it will… It's a simple but essential Linux command. It displays the full path of the directory you are in currently. This is useful when navigating the file system, especially in complex directory structures. And, no, I'm gonna go with another one.
Yeah, so what is directory structure? Basically, it is known as a file system hierarchy, standard FHS. All files and directories appear under the root directory, which starts with slash, even if they're, like, stored in different physical or virtual devices. Okay, and then I'm gonna move on to the next slide. Okay, so wherever you are, no matter what, on the system, the CD command is used to change the directory, and it will take back to The parent directly or home directory. It's just gonna… Okay, so if I do a CD tilde, maybe I'm gonna… go to Downloads, if I want to go to the downloads, is that… Change the directory, which is CD, And downwards, and it's gonna… And if I want to print that directly where I am right now. It's gonna give me the full path of the directory. So CD basically moves you… From here to the other directories. Chain directory is the short for… CD is the short for the chain directory. It is a quick way to navigate to the parent directory of all the directories on the system. So if I hit CD again, and PWD… It basically took me to my home directory, which is my parent directory. All right, and then if we want to go to the CD slash, it will take me to the root directory, which is the The top of the directories, top of the line. Probably you won't be able to, It's… it depends on, the access of the user, and so…
We, as the system admins, we have the full privileges, so that's why we can go to the root directory. Okay, moving on to the next. How can you create a directory? So, the mkdir command is used to create the directories. So I'm gonna create a directory here. I'm gonna create a test directly. Oops, not… Inappropriate directory, which is my home directory. I'm jumping a, awesome. Just go to dis… I maybe ran out of the inot numbers. Okay, so I have successfully created the test directly, and let's test out. Let me look here… Alright, so the test directory is created using the MKDIR, which is make directory, and the test directory. And then we're gonna go to the next command, which is basically Touch. Touch is basically the command used to create an empty file. Or update the timestamp of an existing file. So I'm gonna create a… file a name. my file. dot TXT. I'm fighting. Want to check out. that file. here is my file.txt, so it basically created, file. Later, I wanted to show you how to list the files, but since I'm using it, you can see that ls command is basically used to list the files or directories in a present working directory. So ls is, it's… Stands for, listing, listing a directory or a file. Alright, so, moving on to the next one… Oh, I didn't miss anything. Okay, so if I want to remove file or directory, I just… Right, RM. Which is… which stands for remove. And then… I'm gonna remove this directory. Since I'm removing the directory. I'm gonna use a flag, which is hyphen R, so that it recursively removes the directory and its own, kind of, files in it, or whatever, in it. the content in it. So, for files, I'm gonna use only RM. So let's remove this myfile.txt RM. And my file.txt. And it basically will ask, do you want to remove? And I'll probably hit Y, it's gonna move it. And if I want to check if it was removed or not, myfile.txt is no more. And if I use RM… Test. Basically, it will… it will prompt me that you cannot remove the directory without this flag, which is hyphen R. So, it says RM cannot be removed. Test is a directory, so I'm gonna remove it using the hyphen R flag. And then… If I hit enter, it will ask, do you want to remove this? IPS, or by remove it. To check it, just LS. Alright, so that's how we remove the files using the RL. As well as the directories using the rm-r. flag. So, this is the flag we use to remove the batteries. Alright, so we have gone through PWD, CD. So, basically, if I am in my… any of the directories…
Sam Cho: Waqas increased cost to.
Waqas: Okay, thank you. Thank you, Sam. Okay, so if I am in my home directory, and if I go to CD Downloads, which is 10 directory to downloads, and… Let's see, what do we have here? And if I go… another directory, which is MPM. Oh, it's not a direct user file. Let's create a directory. Test 1, test 2, and I'm gonna go into Test2 directory. So, CD dot dot, it's basically gonna take you back to one directory above, whichever directory you are. So, see, for example, if I'm in, test directory. So, in theory, cd. is gonna take me back to downloads. See, dot dot… Resident Directory… So it took me one directory back. And if I am… Gonna go back to Test 2, if I just hit CD, it should take me back to my home directory, which is the parent directory, which is WH2612, And if I want to check PWD, There I am. So there's a difference in CD dot dot, and then CD. And also, you can achieve the same results, if you are into CD downloads, test… And, PWD to check out where I am. If I want to go back to home, I just… By tilde. Which is gonna take me back to my home directory, which is WH2612. Oops, PWT… I am, so… CD or CD tilde, to take you back to your home directory. CD dot dot is just to take you back one directory above. That's the difference. Alright, we talked about… Changing the directory… CD, PWD, CD tilde… Alright, so I have just… Showed you how to change the directory to test. I'm gonna skip this slide. All right, so, clear here, path completion. What is it? So, if… if I want to, basically. If I don't want to write down the whole path of the directory, I just use the tab button. How can I do that? Because I need to know… Okay. So, see, for example, if we have this downloads, and I just don't want to type the whole download, so what I'm gonna… what I'm gonna do is that, CD, and then… uppercase T, and then hit tab. So I have two. with the D. One is desktop, one is downloads. So… I'm gonna… write down one more word, which is the O, and then hit tab. So it's gonna complete the whole, whole word. Whole path. So… Basically, tab completion is to auto-complete the file or directory name starting with D. And, let's say if I want to go… into another directory, which is desktop, so I'm gonna hit DE, and then space, sorry, Tab. is gonna complete the whole path. So, CD into desktop, So, it is very useful. When you're using… when you're navigating into the, different directories or into the paths, it's very helpful. Alright, so… I hope I didn't say anything.
Okay, so LS, I told you that this is to list the directory or the files. Ls-l. So, every command has multiple, kind of, flags. So, for listing. it has another flag, which is hyphen L. Hyphen L is for a long listing, so I'm gonna show you the difference. If I do LS, as I did before. It is listing the content of the directory. And if I hit LS hyphen L, It gives me more information about that content. So, for example, it's giving me the permissions, who's the owner, who's the group, what date it was created, and multiple different things. So, ls… this command simply shows you what's inside your directory, ls-l", if you need more details, this command gives you a deeper look at your files. Okay… did that. Alright, so yeah, as I was talking about it. So, if you check out, this… this is basically the information about the user that owns the file. And then, if the next is the group that owns this file. So sometimes you have to give the permission to a whole group who can access the file, manipulate the file. So, you have to kind of assign a group to that file, or a directory. So this is… this is what that information is, that who is the… who is the owner… who is the owner and the group of that file, that file belongs to. And of course, the size of the file, this is basically the size of the file. And the directories. And next is basically the last time the file was updated or created. So, this section shows that this file was created on April 3. All right, and lastly, yeah, of course, the name of the file. All right, so another, basically, flag is that ls-A, it lists all files, including the hidden ones. So, if I… here, you don't see any hidden file. And how can I say that? Because there is no dot Kind of in front of… At the beginning of the file name. So if I want to do LS-A, You are seeing the difference? And if I… To the long listing of the hidden files, or all files. So, some of the files now you will be seeing are, basically, it starts with the… with the dot dot, so… This option reveals the files and directories that are normally hidden. For example, some of the system files, some of the environmental files, If I want to create any… Well, let's not create the hidden file. That's… But anyway, LS hyphen A, A for all, and L for longlist. And yes, of course, we have combined two options, two flags, all together, so that, I mean, we can achieve the same goal by using ls-a-L, But I just combined both of the flags with only one hyphen, so… Both of them are right. Alright, so moving on to the next… Alright, so how can we copy a file? I'm gonna create a file here. Let's… Clear out the clutter, by using the clear command. and the screen will be clear out, yes, this is another command. And I'm gonna create a file. Name… Hmm, that's fine. And if I want to see if the file was created. And the test file is created.
Okay. One more thing before we move forward is that if we do ls-l. One thing I want to share is that How to… how to find out that which one is the directory and which one is the file? Is just looking at these… a small, kind of, either it's a D or a hyphen. A hyphen represents it's a file, a D represents it's a directory. And then sometimes there is a L, which is… which basically shows that this is a link file, which is a deeper concept, but just wanted to let you know that. How can we know… how can I know that what is… what is the file, what is the directory? So, my test file was created. Here. And then if I want to move this test file, or copy this test file into a test directory, which is here. Okay, so I'm gonna go into the test directory first to see if… Test file is not here, so this, basically, directory is an empty directory. And if I go back one more directory, ls… And if I want to copy this file, which is test file, so I'm gonna use CPMANT, And then test… bottle. So, path of the file is… this one. basically, I am in the current directory, that's why I'm just giving the name of the file, that's… That's another thing I will show you later, that how can we give the paths Like, absolute path or the relative path. But, so, CP… path of the file, and then the destination where I want to put that file to. So I just want to… Paste this file in test directory. I should write. Put it in the downloads. So I was in a different directory. Hold on. So I'm gonna create a file again here. doc.txt… And cptcast.txt to downloads. Hit enter. And then I'm gonna go to Downloads to test out. If the file was… Copied over. So test.txt was copied over. So I created the test.txt in my home directory. And then I copied over that test.txt into downloads folder or directory. And then I listed… If… and to make sure that the file was copied over, which is test.txt. Alright, so… That's the… Okay, and if I want… if I want to remove the file, of course, RM has done it. BXT. It's gonna prompt me. If I want to remove it or not, yes? And to make sure if it was removed, LS. So I'm gonna go back to my home directory just to… Clear off the confusions. Here I am. Present working directory is my home directory. Alright, so that's how you remove the file. And then, if I want to… so basically, before, I copied over test.txt. to downloads directly. But what if I don't… want to keep it here in my home directory. If I want to move the file, so I'm gonna use the command named MV, which is move. And then test.txt. to downloads. Okay. Hit enter. And I'm gonna go back to downloads. Alas. test.txt, which I removed before, is here, and if I go back to my home directory. It is not here anymore. So that's how you move the files. From one path to another path. From one location to another location. Yes, we can move, So, yeah, from MVCommand, we can… move the files or directories. Also, we can rename them. So, see, for example, if I go back to downloads. And if I want to rename test.txt to something else. So I've just used mv command and test dot txt, to my file. Three. dot TXG. And if I want to test it out. So, this test.txt got converted into my file 3.txt. All right, so we can achieve two goals with MVCommand. One is to move the files or directories from one location to another, and then other kind of a goal is we can rename the files and directories using the MV command. Alright?
So… Now we're gonna move… towards reading a file, but using the cat is another command, so I'm gonna go back one more step. And we… I have a file here named Columbia.txt, and if I want to read the file. I'll just type cat and columbia.txt, and I'm extensively using tab command. Tab-tab, sorry. And basically, it will print out whatever inside the file is on the screen. That's how we read the files, not… All the time, but sometimes, if it's a small file. Alright, so, moving on to the next command. And, so, yes, ID and groups. So, for example, if I want to type just ID. and my uni, if I don't type it, it will gonna show my username. from wherever I'm logging. wrong. So, if I do ID… For you, it will show maybe some intro series groups. For me, it's showing a different, view. So, basically, it shows you, it basically displays the details about the user, including their user ID, their group ID, And, any additional groups they belong to. So for… for you folks, you probably are Belonging to one group, or maybe two or three, it depends. Another thing is, groups. Okay, a simpler way to see which groups a user belong to. All right Okay, I'm gonna clear out the clutter. And then I'm gonna move on to… for example, if I want to see what is my username? I'll just type in, who am I? And it will print out that… The… what is my username? Okay? And there is a… there is a different who am I, which is who space M space I. It basically shows the user who started the session, which can be different if you have switched users. For example, I switched from root user to my username, or basically I… asked this resource, which is the INS040, it's a node, I asked, but… Who am I is the go-to command. Okay, moving on to the next. There is another command, who? Who basically… why is it not showing anything? Okay, so let's say… Let me get my privileges… There's not showing me anything. Okay, who basically prints the, who lists all the current, logged-in users, along with their session details? So, if I do one more… It's basically, it's gonna show you that, All the users currently logged in, various session details, such as terminal, login time, remote host. So I was logged into, a different kind of, node. It doesn't have any… doesn't have any… other users, so that's why it was not showing anything, but… Who basically shows you all of the details, who is currently logged into the system, and when, and which IP phone.
Okay, there's another command… ATSW. It's basically… It gives more detail about the… basically, it shows who is logged in and what they're doing, including their login time, idle time. The command they are currently running. So it is kind of a… gives you more in-depth, view of who is logged in and what are they doing. So there are different columns, like user, terminal, or device the user is logged in from, for example, TTY, PDS. And then their login time, when they logged in, idle time, the amount of time the user has been idle for. JCPU, the time used for all processes associated with the user terminal, and then the PCPU is the time used by the current process running under that terminal. And then the process ID is PID, we call it PID. So the ID of the user is currently Basically, the current session up terminal. Alright, so… Another useful command is basically MAN. MAN is short for manual. It is used to access the manual pages for a command, providing detailed information on its usage, options, and syntax. For example, I'm gonna… If I want to learn more about the LS command, I just do man ls. It basically gives you what it does, description, flags, what flags I can use with the man, sorry, with the LS command. Different options. So basically, it is the… it is your go-to resource for learning about commands and their options directly from the terminal. You don't have to Google anything, you just type man, and then name, and then command, and then it will give you what you can do with that command. Alright, so file manipulation, basically, for example, I showed you earlier… this… Columbia.txt… So, for example, if I want to… particularly Fetch University word. From, from this whole file. what I will do is… Breath. And then… University… And name of the file. So, I'm using prep command. I want to use… I want to fetch this university from the whole, whole file. And it will show me, highlighted, that the university was here, here, and at this place. So, it is… it's a powerful search tool used to filter the text based on the patterns. Useful for scanning log files, configuration files, or large data sets. Okay, and then moving on to the pipes. So… Pipes basically allow chaining the commands. passing the output of one command as input to the other, and I'm going to show you how. This is basically useful for narrowing down large outputs, making it easier to find specific information So, let's see, how can we do CAT? columbia.txt, and then If I want to grab location, So basically. I am reading this file, and then… whatever output of that file was, I'm just piping it and using another command. Making the output of the first command as an input of the other command. And then that command is grep, and then it's gonna grep the location word. Okay? I don't have one, let's try University. Okay. So, before, I just grabbed the university word from Columbia.txt, but this time, I'm basically catting the Columbia.txt file, making the output of that file as an input of the grap command, and then fetching the university word. Okay, so it has basically… It's, it, it is, it is filtering the, the output. off the CAC mode. And filtering… the word university. It's showing the lines which only have the university. Alright? Okay, next command is tail and head. So, for example, if I… clear this out. Both of these head and tail are useful for previewing the large files without opening them. It's great for checking the beginning or end of the logs or configuration files. So, I'm gonna use head. Columbia.txt. So… By default, it shows the first 10 lines of the columbia.txt. Which includes the spaces between the lines. So… Head is basically printing out the first 10 lines, including these spaces, of that columbia.txt. And if I want to reduce that number, so what I'm gonna do is… and… oh, another thing I wanted to share was up arrow. So, for example, I have typed this command, and if I want to use that again, I just hit up arrow, and it's gonna basically, I mean, it's gonna bring the command again. op… I did not want it to do that. I'm gonna square it out. And then add. iPhone 5, I want to print the 5, first five lines at columbia.txt. So now it printed, basically, first 5 lines. 1, 2, 3, 4, 5. If I want to reduce that number… And tattoo. So that's how… it's basically printing out the first two lines. Of the… of the file. And same goes for the tail. it's gonna… print the last lines. By default, 10, and if I want to use the… only two lines of columbia.txt, and for tail, I'm going to use hyphen n for numbers. Okay, so last two lines are basically spaces. It's available can… Yeah, that's… So it basically printed the last pipelines. A tail and head is also useful for, kind of, reviewing the configuration files and the logs. End of the longs are the start of the longs. Output redirection. So, it is also a very, useful feature. The greater sign operator basically redirects output to a file. If the file exists, it replaces its content. It is useful for logging the timestamps or saving the command output for the later use. So for example, as I showed you before. Date command basically prints out the date and time of the system, so if I want to redirect the output of this date command, I'm just gonna type in date. And then this operator, and then if I want to give the name of the file, system date. And then, if I hit enter, LS… It basically created that file, and let's read this file, system date. And it basically have the content. Which is… The date of the system, date and time. All right? So, it is also very useful. Okay, output redirection… And a siphon L… Okay, yes. Just… this is kind of a… Repetitive.
Okay, so there is the difference between single… There is a difference between single operator and the double. So, basically, single is gonna create that file if that file does not exist. But it will replace the content of the file. Say, for example, if I want to print out ls-l, and… Redirect that to system date. It will basically remove this content, and it's gonna… Basically, add, or maybe, output of the LS-L to that System date command, system gate file. And then if I want to cat that… So, this file now… has the content of the LS hyphen L command. So… if I don't want to lose the… the current content, and… If I want to add more, I'll use the double redirection sign. So… If I want to use date and double redirection sign, So… This double redirection sign is gonna append the content. Which is, basically, it's gonna keep the content, off. the current… File, and then… add more to it. So, if I want to do CAD system date. So now, we have this whole content, plus the date and time of the system. So, double redirection signs append. Single redirection signs, basically, sign is gonna create the new content, or, remove the old content. That's, that's… what it is. Alright, so moving on to the next command, which is Echo. Echo is gonna… it is great for scripting, debugging, and weekly display values. So, see, for example, echo, hide. It's gonna print high. Echo, Columbia. Snapdragon Columbia. It's simple, yet essential for automating the tasks in Linux. So it is basically used for, displaying the messages, printing the variables, like, see, for example, if I do echo, Boom. So, it will… it is… it basically printed the variable home. It's the environmental variable, which basically is that path of my home directory. Alright. Moving on to the next one, which is multiple commands. So, see, for example, I want to combine two commands, or three commands, or four commands, so how can I achieve that? By this semicolon, which is, date, and then semicolon, and then if I want to do echo… Columbia. So it's gonna… print out Echo Columbia, ECHO is one command. And then date is another command. So… If I hit enter. It ran both commands in one go. See, for example, date. Colon, LS. column. What can we do, hotels? Let's see, for example… Mang… No, not BAM. For example. what else I can do? I'm thinking about easiest, command. Colombia… dot txt… And… it's gonna do all of it. So, it's gonna print the dates output, ls, and then it's gonna print the output of cat columbia.txt. So, the first is… The second is LS. And then the third output is for… cat columbia.txt. So we are… we have combined multiple commands in one go. Moving on to the system health. That's another thing, day to day. Basically, the tools… Which are essential for troubleshooting the servers. We rely on these tools daily, especially when servers are running slow or encountering the issues. For instance, if a server is sluggish, we start with these tools to pinpoint the problem, whether it's a resource usage, disk space, or network issues. So, knowing how to use them is crucial for effective troubleshooting. So, I'm going to start with 3 hyphen H. Let me clear this flutter. So, free. hyphen so free is the command, and then hyphen H stands for the human readable. And then it's gonna basically print the, detailed report on this, on your system's memory usage. It's M4M memory. So, as I mentioned, that hyphen-H option shows memory in human-readable format, which is, for example, MBs or GBs, while hyphen M shows it's it in megabytes. Okay, and then moving on to another command, which is top. It's also… Very powerful tool for system monitoring. Allowing you to spot performance issues quickly and manage processes. It's an essential tool for any Linux admin to keep track of the system health. It shows the real-time CPU usage, helping you track which processes are using the most resources. It lists all running processes and updates every 3 seconds by default. There are many options to customize the display while, like, sorting by memory or CPU usage. Queue for quitting. Okay, and then the edge top, it is more interactive. I'm gonna do it, actually… John Yep, as I was saying, that it provides a visually appealing color-coded interface with an intuitive layout, unlike top, which requires manual PID input, which is process ID input for actions. H stop basically allows you to scroll through processes and interact with them using arrow keys, making it easier to manage tasks. I'm not gonna go deeper into it because of the time constraints. But HDOP is also very useful. And other than that, VM stat. So, it basically… Let's… let's do it. Can be Amstacked. And it basically… It's a lightweight tool that provides a real-time snapshot of the system performance. It reports on CPU usage. memory swap, disk I.O, and system processes. Okay… P.S. Pss basically provides a full system-wide view of running processes, including persistent processes and those owned by the other users. Oh, P.S. And then there are multiple flags with PS, TSIF and E. It basically provides all, like, full system-wide view of the running processes, including the system process, and those owned by the other users. It is basically a useful command for admins to monitor the system. Activity, and then troubleshoot resource issues. PS hyphen AUX. Okay, PS hyphen AUX lists every process, with a lot more information. As you can see, it has multiple tabs, if I go on top of the… Too much. So yeah, it's, who's the user, what is the process ID, how much percent of the CPU memory is using virtual memory, and start time. And then what command they are using. A lot of, information. And one critical thing is inode numbers. So, in Linux, every file or directory is represented by an inode, which holds the metadata, like permissions, ownership, and size, but not the file name of the content. So, each file, each new file consumes one inode, so even if you have a disk space left, you might not be able to create more files if the inode limit is reached. So to check the iNode usage, we just use DF-I, It's gonna show me a lot of different things. Maybe human-readable, the Python, image. So, human readable, basically, it's gonna print you, the memory and usage, how many inodes I have used, how many inodes on the system, and how much are free, percentage of it, and for the mount systems, our mount points.
So, it basically is useful in environments with lots of small files, like logs or temporary data. Okay… Exit command is basically gonna lock you out of the system, or you can Press Ctrl and D. I hope… I was able to make it in one hour. Sorry if I was going too fast, because of the… we squeezed the 3-hour session, removing a lot of slides, to 1 hour. And then I try my best to show whatever I have. So, any questions?
Jessica: There is a question, is there… are there functions that can be operated on WinSCP directly?
Waqas: Winscp is, we can copy the files. We have used it in the past, but we highly encourage to use Globus for the large files. And then the, if you want to use… if you want to copy over the small files, you can use the SCP. Scp is the secure copy protocol. So… when SCP is not encouraged in the system. Correct me if I'm wrong, Max.
Jessica: Okay, I think that probably answers the user's question. If you… if you have any follow-ups, you're welcome to raise your… raise the… the follow-ups, directly here. Not you, Waqas, the user, sorry.
Waqas: Yep, here are some books that really helped me, both in learning the Linux and passing the Red Hat Certified Engineer, as well as Admin exam. So these books cover basically everything from basics to advanced topics, making them great for anyone, looking to dive and dive deep into the LX.
Halayn: And then what's the next class, Waqas?
Waqas: Next class, I am not sure about that. Jess, can you please…
Halayn: So with the scripting, it's scripting next, right? So you take these…and learn how to use them in scripts.
Jessica: Yeah, HPC jobs… scheduling HPC jobs with Bash is our next class, on Tuesday. I am going to… if anybody does have feedback on this course today, please let us know, and put the feedback form in our chat here, and I'll email it afterwards, but this class is a new… new way of presenting this information, and we want to hear from you guys if you have, suggestions, or if you liked it, appreciate it. I know there are some folks here that have been in the last semester, so it's great for that. Good to hear. But otherwise, Waqas is here for your questions, otherwise we will… we can close it out. Sure. Thank you, everyone, for joining in. Thanks, everybody. Have a good day.