Introduction to Linux Workshop 2025 | HPC Training Series #1
Transcript
All right, so as Max mentioned that this week we are starting with an introduction to Linux, covering the basics, what Linux is, why it's popular, and some key concepts.
And then next week we will move on to scripting in Linux and where we will learn how to automate the tasks using shell scripts. It's a crucial skill for efficiency and system management. So don't forget to register for next week's session if you haven't already.
So moving on to the Linux, what is Linux? Linux is basically an operating system that manages hardware and software allowing users to interact with their computer. It's known for being open source, secure, and highly customizable. Unlike Windows or macOS, Linux is free and comes in many different versions called distros.
It powers everything from servers and supercomputers to smartphones and embedded devices. In short, Linux is everywhere even if you don't realize you are using it. Why Linux? So it's free access to the source code allows customization and transparency.
It runs on everything from desktops to servers and embedded systems. Strong permissions, I mean strong permission management and fast community-driven updates. It rarely crashes with long uptime making it ideal for critical systems.
Efficient multitasking and great for handling heavy workloads. So many distributions are free and reducing the software expenses and large active community providing help and resources. So that's why Linux.
So why I chose Linux? When I started with Linux my main goal was to host my own website. As a student I didn't have the budget for expensive Windows server licenses so Windows required costly software like Microsoft IIS for web hosting which is like around $2,000 and then Microsoft SQL server for database is around $4,000. So the license itself.
But Linux provided a free and open source alternative with Apache for web hosting and MySQL for database. Both widely used and community supported. Plus Linux is known for its strong security and reliability making it a great long-term choice.
So by choosing Linux I had full control over my system without spending a dime on licensing. All right before jumping into navigating Linux you will probably needing access to the system. So there is a system you might have received and you should have received the email about the system and how to log into.
But let me do it in front of you. So basically here is the login you'll be using to get into Insomnia. The system name is insomnia.rcs.columbia.edu and you would use your uni prefacing the domain name.
So I'm going to do it in front of you and before that I mean for Windows 10 and 11 you can actually use the command prompt that's built in or you can download PuTTY which would be needed for older versions of Windows like 7 and 8. All right so you can download the PuTTY using the link. You can just type the link and if you have already downloaded it it's fine. For Mac users you would run the program called Terminal.
You can just search it in your search bar and then just type Terminal and it will open up the window. Okay I'll give you two minutes and I'm gonna do it in front of you so we can mimic the session. So ssh and then your uni which is in my case wh2612 at and then insomnia.rcs.columbia.edu. So why will we will be using the ssh? Why we are using secure shell? I wanted to give a little bit background on the security that's happening behind the scenes.
You know before this secure shell environment that we are going to use now everything was run by telnet and everything you typed into it could be sniffed on including usernames and passwords. So over the years programs were developed and these are all open source. So now we use this secure shell program where everything you're seeing on this slide is just I mean it's encrypted.
See if I hit enter so don't get afraid of it. Okay I want to talk just a second on what is that some of you are seeing on the screen. There is a name for this.
It's basically called trust on first use. The message you get the first time you log into any server any linux server. So you would definitely want to type yes and press enter and it will prompt you to write your or type your password that should be your uni password.
All right so if you're entering your password and don't see anything appearing on the screen don't worry that's normal. Linux doesn't show password characters for security reasons. This is to prevent anyone from guessing your password length just by looking at it.
So just type your password carefully and hit enter. It's still being entered even if you don't see it. Okay so I'm going to hit enter.
So at this point most of you should be logged into the server. I'm going to take a pause for like two to five seconds and we'll continue from there. Anyone with any technical difficulties just drop a question or any errors in the chat window and one of my team members will address it.
Any questions? Okay Syed or Wakas let's proceed. Okay all right so now that we have accessed the server let's start with some basic commands to gather some information. All right so first we can check the system's name using post name command.
So is my text big enough Max can you please make it slightly larger that's perfect thank you and rather using command prompt I have my own software to use which is Mobaxter. This is also another facility or it's kind of software where you can use the SSH. You guys can continue with your command prompt or a terminal.
Okay so let me go ahead and use my Mobaxter which is also another software. So host name basically displays the name of the system. It is useful for identifying the machine especially in networks.
Okay all right next we want to see how long the system has been running so we will use uptime for that. It shows you how long the system has been running. It is great for checking system stability so it is showing basically the time and days and how many users.
All right so basically it checks the stability of the systems. Okay finally I want to introduce you to date this basically prints the date current date of the system and time. So it helps in scheduling tasks or verifying system time setting.
All right so next command I wanted to show you was PWD. It basically is a simple but essential Linux command. It displays the full path of the current directory you are in and this is useful when navigating the file system especially in complex directory structures and so we will be using this PWD in our presentation and this intro series a lot.
So you might have a question in mind that what is a directory structure? So directory structure is basically known as file system hierarchy standards. All files and directories appear under the root directory which is slash. So if you see here you see a slash so this is the parent directory.
So it starts with parent directory so there is another directory and there is another directory and there is another directory right after it. So my present working directory will be WH2612. All right so next command basically is CD.
The CD command is used to change directories in Linux. Okay it's a quick way to navigate to the parent of all directories on the system. So see for example if I want to change to my parent directory instead of here I will just use CD space slash.
So it will take me to my parent directory which is slash. You see the difference before it was showing a tilde which is a symbol for home directory. Now it is showing slash which is the parent directory and I can verify it using pwd.
Okay and now if I want to go back to my home directory I can simply type CD. So CD will take me back to my home directory. Most of you guys probably won't be able to go to your parent directory but as an admin I can go there.
So I'm going to change it to my home directory and I can verify it using pwd. Next I wanted to introduce another command mkdir. So it is short for make directory.
This command is used to create the directories and this creates a new directory. Let me create a new directory mkdir space tmp. So I'm going to create a directory named tmp.
Okay so how can I verify that I have created this directory named tmp. So there is another command for that it is ls short for list. So it will list the content of my current working directory.
So if you guys can see it is here. All right so creating files. So now we learned how to create the directory.
We can create the files using touch command. Since this myfile.txt is already created I'm going to create another file. It'll be myfile2.txt. How can I verify ls.
So if you can see myfile2.txt is here but before when I listed it it was not there. Okay or maybe you can create another file if it's confusing. Example ls and I can see that example is here.
Example file is here. Okay how to remove the directory. So let me remove the directory.
Let me remove tmp directory. So rmdir and name of the directory. It will remove the tmp directory.
How can I verify? I can verify using ls. So there is no tmp directory here anymore. All right so I'm going to create the directory.
Okay so before creating another directory or moving forward I'm going to give you another command to clear the screen so that you don't see any clutter here. Okay so if I type clear it will clear out the screen. Okay so I'm going to create one more time tmp.
I'm going to change to my newly created directory cd into tmp and then I can verify using pwd. You see the difference before it was showing me when I was in my home directory. It was showing me till here but now it is showing me till slash tmp and if I want to go back one directory above I can use this dot dot so cd space dot dot.
So it's basically it's going to take me one directory above. So pwd now it's showing me that I am here at my home directory. Before I entered pwd I was in tmp directory.
Okay moving on to next one. See for example if I was in slash and I wanted to go back to my home directory as I mentioned I just type cd. It will take me back to my home directory or I can use tilde a tilde.
It is a special character that acts as a as a shortcut to your home directory. Okay so cd without any argument is going to take you to the home directory as well as cd tilde. It's going to also take you to the home directory.
How can I verify pwd? All right so there are two types of paths. One is absolute path. It begins with slash specifying the full path from the root directory as I mentioned before.
Starts from slash. Okay and then there is another path which is it doesn't start with slash but it is relevant to the current working directory. Okay so I'm going to give you an example of that.
Either I can do cd into tmp and then it will change the directory to tmp and I'm not using the absolute path. I'm just using relative path because I know that this tmp directory is already in my present working directory so we can verify that. So we are in our home directory and tmp is already there so I can just change my directory to tmp one and I'm going to go back to one directory above and now I can achieve the same goal by using the absolute path which which starts with slash.
So cd slash insomnia. I'm just looking at it here then home then wh2612 in your case your uni and then name of the directory. So now I am giving the full path absolute path.
Okay hit enter verify pwd. So I have achieved the same goal by entering the absolute path. Before I was entering only the relative path which was cd into tmp because I knew that this directory is already in my present working directory.
Tab completion. It is a lifesaver. Helps quickly complete the commands file names or directory names by pressing the tab key.
So see for example if I was typing this whole path. Okay let me go back one directory above. Let me clear my screen and if I want to go back to tmp directory using the absolute path.
So I'm gonna use tab completion this time. So cd into in. I just entered two words uh two characters and then hit tab.
It automatically completed the path and then if I enter h and then tab it's going to automatically complete the home. But if I enter wh there might be another user with wh. So I if I hit and if I hit tab it's going to say okay I can complete up until two but you need to see what else this home directory have.
So I can double tap it and it will show me the entries. So there is the username wh2526, wh2581, wh2612. So I need to specify the second character and then tap.
So it will complete the remaining part. Okay and then tmp and if you can see I just entered tm and then tab just completed the whole path. So tab completion is a kind of work both for commands and files and directories and it reduced the errors basically.
Next I'm gonna show you ls. We already know what is ls. I have already presented that to you but this time I'm gonna use another flag for in front of ls command.
So ls hyphen l. So l basically it's gonna if you if you need more details this command gives you deeper look at your kind of a content of the present working directory. Okay so you will see the difference when I enter ls and then if I'm entering ls hyphen l so it's basically long listing the directory the present working directory. Okay don't be scared of this this whole stuff I'm going to explain that to you.
So first is basically file type and permissions. So how can I know that this is a directory or this is a file? Basically it tells me in the start if it's a d here it's a directory. If it's a dash it's a file and sometimes you will see l in the beginning it will show that it will it is a link.
All right so next the permissions for for the file are directory. Okay and then the user who owns this file. So the second column is basically it shows that the file's owner and then the third column it indicates the group associated with the file.
Okay so in my case it is user group and then this next column it shows the size of the file in bytes. Okay and then the next part from here it shows basically the last modification date and time of that file or directory and then obviously the name of the file or directory. So ls-l gives us more kind of a detailed version of the contents of the of the directory which we are in currently we are in home directory.
So in our case there are a lot of files and directories you might be seeing maybe nothing in your in your scenario. Okay or maybe you can create a file and directory using mkdir a touch and then you can long list those. Okay next thing I want to show you is ls-a so if I enter ls-a you will see a difference some of the different files will show up I'm going to show it to you so ls has this content and let's see ls-a what does it show okay so ls-a is showing some of the files which are starting with dot I'm going to explain those to you so if we did ls those files were not there but if we do ls-a some of the dot files are there so this option reveals the files and directories that are normally hidden so whenever you see dot in front of or at the start of the file or directory so it means that's a hidden file or directory and let me combine two flags ls-al so I'm gonna list the hidden directories as well as long listing those here you are so you're seeing the difference you're seeing the dot bash profile dot bash rc these are hidden files these are system files basically so this option basically it reveals files and directories that are normally hidden and hidden from ls command okay so moving on to the next slide so these two dots these these files generally store your preferences they're typically such as your shell or other applications you can use them automatically to automatically set up your environment when you log in so you don't have to manually configure settings each time okay so dot files are a convenient way to save and manage your personalized kind of settings without affecting the visibility of your other files any questions so far all right let's move on now let's see some basic file operations um I wanted to introduce you to cp command how can we copy a file into a directory or maybe a directory to a directory okay so let me clear my screen and I'm gonna list the content of my present working directory and let me cd into tmp and let me list if it has something so my tmp directory doesn't have anything I'm gonna go back to my home directory and let's copy a file ls and if we do have any test file or maybe we can copy this my file too to my tmp directory okay so cp so that's how the syntax will work cp space and name of the file our directory and then the destination path you can use absolute path you can use relative path it's up to you but in order for me to save the time I'm going to use the relative path okay so I'm gonna copying this my file to into tmp directory okay and then I'm going to go into tmp directory to verify if my file was copied there I'm going to verify by using ls. Okay, see the difference. Before it was showing nothing when I entered the ls and now when I copied the file it's showing me the content of the my of present working directory which is tmp in this case.
So that's how you copy the files into the directories or directories into directory other directories or destinations. All right, so scp is a command line tool for securely transferring files between computers over ssh. It's useful when you need to quickly copy files between local and remote systems or even between two remote servers which I'm going to show it to you.
So it's basically copying from your local machine to your server anything from your local machine to the remote server or maybe from remote server to a remote server or maybe from a remote server to your local machine. Okay, so the syntax how is it going to work let me go to my other remote server and if I list I have created a kind of a test directory and I'm gonna go into that directory list the contents. So let's see if we can copy the example file which we created here.
Okay, so I'm gonna so that's how the syntax is gonna work scp and then the name of the file so my tab is not working because I'm in tmp directory. Okay, go back one directory above and then scp example file and then my uni at name of the server or ip you can use ip as well dot rcs and then colon and then path to the destination. So in my case if I want to know the path for my destination directory we are seeing the absolute path and that's what we need for scp.
So I'm gonna paste that path and let's see if it's going to work or not. So remember there is no example file here when we list the content of present working directory. Okay, so let's see of course it's going to ask me password for the remote server so I'm going to enter my password and if you're seeing this 100 percent and then the kbs of the file it means we have successfully sent out the file.
So let's verify it here see the difference the example file was not here but now it is here. Okay, it is very helpful using this scp when you're trying to fetch some files to your local machine or when you're trying to push your files from your local machine to the remote server. So this is the syntax scp name of the file which is in your current directory and then the username at name of the server colon and then the path absolute path to the directory to the destination directory.
That's how you use scp. Okay, all right so let's move on to the next command. So let me clear my screen here and I want to show you how to remove the files.
Now we know that how to create the files are directories. Let me show you how to remove those. So ls to list the files.
So see for example if I want to delete this example file. So how can I do that just simple rm space name of the file. So it will ask you that do you want do you really want to remove this empty file.
So I'll say yes and we can verify using ls. Okay, so example is not here anymore but remember in order for us to remove directories we need to specify a flag which is hyphen r. Hyphen r stands for recursively. So this directory might have some content in it.
So let me let me show you if I don't use this hyphen r what error is it's going to give me cannot remove tmp is a directory. So it's going to give you error because it's a directory. So we need to use hyphen r flag.
Okay, so it's going to ask you question that do you want me to delete and descend into the directory and delete the contents. I'll say yes and then it's going to ask me for each file or each content in the directory one by one. Okay, so it deleted the directory and its contents.
Let's verify there is no tmp directory here. So if I don't want to get prompted for the questions it's asking I can go ahead and let me create one more time tmp directory and some files in there. Okay, so this time I'm going to use another flag which is f. F is forcefully to avoid these questions to be asked.
So I have created the directory one more time just to show you how can we avoid these questions. So rm hyphen rf and then specify the name of the directory. So this time it won't ask me any question but it will delete the whole directory and its contents.
So far so good. Any questions? Okay, moving on to my next command which is mv. Okay, let me see.
Let me clear my screen one more time. Ls. So let's see if I can move my file.
This is my file two into the directory. So I'm going to create one more time the tmp directory and then let's go into tmp directory and let's see if it has any content. So ls is not going to give any output means this directory doesn't have any content.
Okay, I'm going to go one back one step back to my home directory where my files are and then move this my file two into the tmp directory. Okay, so the syntax is mv and name of the file and specify the path the destination. So mv source and then destination.
In my case this is my file two and then tmp. Okay, let's see if it has moved. So before I showed you how to copy a file but it didn't move the file.
This time it just moved the file so I can verify it here. Okay, let me see. So there is a confusion between my file 2.txt and my file 2. So my file 2 has has moved from my present working directory which is my home and let's see if it is in tmp ls.
So my file 2 is here. Let me go back and create an example file and avoid the confusion. There's only one example file here and then I'm going to move this example file to my tmp directory.
Okay, ls to verify if this file has been moved to the tmp directory and let's go back to tmp directory and list. So here is my example file. So that's how you move the files.
There is another purpose of mv command which is you can rename the files. So let me rename this example file to something else and without changing the content of the file. So let me see if I can give it a name tree.
Okay and then I'm gonna list the contents of present working directory. So example file has been renamed to tree. So that's another purpose of mv command.
All right, so moving on to next slide and let me clear my screen. Okay, so reading the files in Linux is very simple. You just type cat which is short for concatenate and then if you want to give a name of a file it will print out whatever the file content is.
So I just specified cat and then name of the file and it has printed all the content of the file. A small exercise for you guys is that you can create a file using touch columbia.txt in your home directory and just copy and paste a content from anything from the internet so we can use later on because I'm going to show you how to edit the files, how to write into the file, how to navigate into the file. All right, but that's how you can read the files.
Cat specified the name of the file and it will print out everything in that file. It is very useful when you're troubleshooting when you're trying to read different files or maybe shell scripts. Okay, moving on to the next slide.
So let me clear my screen. Okay, id is basically it's useful for checking a user's uid, gid which is a group id and a group membership. So I'm going to use id command and then my uni and it's going to show me what is my user id, what is my group id and how many groups do I belong to as a user.
Okay, and these groups can also be printed using the same syntax but using the command groups. It's a simpler way to see which groups a user belongs to. Okay, so in your case you might be seeing only one group or maybe two groups.
In my case, these are a bunch of groups. Okay, next slide. This command, who am I? So basically useful to quickly confirm which user account you're operating under.
Okay, so let me clear my screen. So there are two who am I's, one without any spaces, one with spaces. Okay, so let's try without any space.
Who am I? So it's going to print out my user name. Okay, now we're going to try with spaces. Who am I? So it shows basically the user who started the session which can be a different if you have switched to maybe a super user or maybe to another user but this one will show you that which user started this session.
Okay, so this is a difference between who am I and who am I with the spaces. So it will show you that which user started the session but this one shows you just your user ID. Okay, moving on to the next one.
Who? So let's check who is logged into the system. There are a lot of users. So who basically lists all the currently logged in users along with their session details such as terminal, login time, and remote host.
Okay, it is useful for monitoring active users on a multiple and on a multi-user system. See for example this one. Okay, so it shows you the user name, login name of the user.
This shows you the terminal or device the user is logged in from and this one shows you basically the date and time when the user logged in. And the last thing the IP what is this? This is basically it refers to the remote address from which the user is logged in from and this is very powerful command when you are trying to troubleshoot. Okay, let me clear my screen.
Okay, let's see more details on the on the user info. So I'm using the short version of who which is w. It gives me a little bit more details. So a w shows who is logged in and what they are doing including their login time, idle time, and the command they are currently running.
Okay, so this w provides the more comprehensive view of the user activity on the system helping you to monitor user sessions and activity in real. So as I mentioned this is just the user name. This is the terminal they are logging from.
This is the basically login time and this is the idle time the amount of time the user has been idle for and then this jcpu is basically the time used by all processes associated with the user's terminal and then this sorry this jcpu and then this pcpu is basically the time used by the current process running under that terminal and there is a what. This column shows the current command or activity being executed by the user in the terminal session. Okay, so it is a bit more detail and also it gives us the the uptime of the of the server as well and how many users are logged in.
Okay, all right let's dive into the manual pages. It's very helpful and then it's gonna it is a lifesaver. Okay, let me clear my screen.
So man is used to access basically the manual pages for a command providing detailed information on its usage options and syntax of say for example if I do man ls since we are using ls for so many times now. So man is basically short for manual so it's it's your go-to resource for learning about commands and their options directly from the terminal so you don't have to google if you want to learn about the the flags of the command. Okay, so say for example we were using hyphen a flag so it is also in the man page or manual page of the ls command and if I want to search any flag for for that command it's basically very easy.
I can just enter slash and if I want to see if hyphen l is there because we were using hyphen l before as well. All right so where is hyphen l there are a lot of hyphen l it's hyphen l hyphen l here oh it has highlighted it. So use a long listing format so it is it also explains what it does what this flag is doing if you are entering it in front of your command.
So it is very useful it's uh it's going to show you the syntax as well for the commands so what syntax you have to use okay so to quit out of it just press q. All right so let's explore some some essential file manipulation tools okay grep it's basically a powerful search tool used for you know to to filter text based on patterns okay so it is used for scanning log files configuration files or large data sets in my case let me clear my screen I do have columbia.txt and I'm going to read this columbia.txt so it has a lot a lot of information so let's see if I can filter any information using grep grep and I want to filter university word and columbia.txt if you haven't created that file columbia.txt just create it so it's going to help you to navigate through just puts any stuff copy from google and just put it there so that you can practice okay so let's see if we can grep the university word from the columbia.txt so now it is it is not showing me as compared to the cat command it is not showing me the whole content of the file but just the university word the lines which has the university word okay pipes let's see how pipe works so pipe basically it allows chaining the commands passing the output of one command as input to the another this is this is very useful for narrowing down large outputs making it easier to find specific information okay it is also more efficient way to to do this let me do this grep using using pipe so i'm going to read the file using cat columbia.txt and then i have the output of that cat command it's going to store it in the pipe and then i'm going to grab the word let's say manhattan or maybe location okay all right so it has just printed the line which had location in it so normally when you cat the cat file it will print out everything on the screen but in this case it did not it just stored the output into the pipe and then from that i manipulated that using grep okay so it is very powerful the pipes are very powerful okay tail and head so let me clear my screen okay let me do head columbia.txt so what it does is basically both of these are very useful for previewing the large files without opening them okay it's a head basically shows the first 10 lines of the of the file okay including the spaces between them so let's see if i can add columbia.txt so it just printed out the first 10 lines including the spaces okay and if i want to specify only first two lines or first three lines or maybe first five lines i can just uh enter hyphen n flag so let's do that head hyphen n and then let's see if i can print first two lines of columbia.txt so see the difference it has just printed first line and then the space after that as compared to the head okay let's see what does tail prints out so instead of the top of the the file it is it is just printing the bottom like the lower part lower 10 lines of the of the file and if i want to specify just the last line so i can just tail hyphen n n and 1 and columbia.txt so my expected output is to just to print this last line there we are okay so it is kind of a very useful uh tool to you know basically if you're searching for something in the logs maybe in the error messages it is very useful to tailor it to the last few lines or maybe first few lines so this is tail and head okay let me clear my screen all right so how can we edit the files when it comes to the text editors there is no single obvious choice but vi or vim stands out for linux admins why vi it is it is very powerful it's simple but can feel tricky at first okay and it is universally available meaning that it is pre-installed on all linux distros unix systems and even macs so it is it is very reliable when when a server crashes vi is often the only editor available making it essential for quick fixes so if you're debugging any any crashes and then you're not into the operating system itself vi editor will also will be there okay so and this is this is why many linux admins including myself rely on it so it's always there when you need it so let me manipulate the file or edit the file using vi editor so what is the what is the syntax for that vi and the name of the file columbia.txt okay so there are two main kind of modes in vi editor one is insert mode or edit mode and the second one is command mode which is which is which i'm going to show it to you so for me to go into the insert mode i just enter i and it will show you here that you are in insert mode bottom of left bottom of the line okay and and if you want to go back to the command mode you just press escape and this insert mode will go away okay in insert mode you can just type anything you want okay and then it's just like a normal editor but in escape mode you cannot type anything unless you're you type I, which takes you to the insert mode. All right, so let's explore some of the options in escape mode. If you are at the bottom of the file and if you want to go up top, and while you're in escape mode, just hit GG, or G twice, so it will take you to the top of the file.
And if you want to go to the bottom of the file, shift G will take you to the bottom of the file. So it's GG to the top of the file and shift G to the bottom of the file. All right, so the cursor will move from top of the page to the end of the page, or the file.
And if you want to, if you're at the beginning of the line and you want to go here to the end of the line, you just press shift 4 while in escape mode. This is a disclaimer, while in escape mode, you just press shift 4 and the cursor will be at the end of the line. And then if you want to go back to the start of the line, you can just press shift 6. Okay, shift 4 to the end of the line, shift 6 to the, okay, shift 6 to the start of the line, and then shift 4 to the end of the line.
All right, so if I want to delete this whole line, how can I do that? One way is to go to insert mode and then press backspace all the way to the end or to the start of the line. Another method is to just go into the escape mode and hit DD and it will delete the whole line. Okay, and if you want to undo your changes while in escape mode, you just press U which will undo the changes.
All right, so go to the start of the line by pressing shift 6 and then DD. It will delete the whole line. And then to undo it while in escape mode, press U. Okay, so there are a few more things for the vi editor I wanted to show you is while in escape mode, just press colon and then set spell.
It will basically turn on the inbuilt spell checker. So if I want to do that, hit enter and it will show you where the spelling mistakes are. And if you want to turn it off while in escape mode, just hit colon and then set no spell.
So it will just take it away. Another thing is if you want to see the number of the lines, line numbering, how can we do that? Set number. It will set the numbers of the line including the spaces and in order for us to remove that no number.
Okay, so now we know how to edit a file. Let's save this file. So in escape mode, hit colon and then W to write and then Q to quit.
And it will save the file and quit out of it. Let me go back to this file and make some changes. Let me delete this file, delete this line and then save it, quit out of it and hit enter.
Okay, and then if I go back to this columbia.txt, that line shouldn't be there. See, for example, if I want to delete this file and I am just using Q, not saving anything, not using W, it will ask me that no right since last change, add exclamation to override. Okay, so it's basically asking me to forcefully quit out of it.
Errors are your best friend in Linux, by the way. So I have deleted that file, deleted it. Say again, Max? I'll say you think so.
Yeah, you learn a lot from mistakes or errors. Okay, so let's see if that file, that line is there or not. That line is still there because I didn't save it.
Okay, so this line came back because I didn't save it. So this time I'm going to save it and then go out. And then once I'll come back, the line is not there.
Okay, so far so good. Okay, let's take a look at a few simple commands. Let's get out of this file, clear my screen.
And then as I mentioned, date is just, it's going to print out the date and time. So if I want to save the output of this command, I can just simply use this redirection sign. Okay, this is just, it redirects the output of a file or a command.
So let's do that. Date, redirection, and then let's say if I, the date of system. So this is going to create a file named the date of system and then it's going to redirect the output of this command into that file.
Okay, so let me read that file and tap. So it basically created the file with this content. And if I want to add more content to it, let's see if I want to redirect uptimes output.
This time I'm not going to use the single redirection sign. Why? Because it's going to replace this old kind of a stuff and add the new stuff, which is output of uptime command. But if I want to append it, I'll use double redirection sign.
Okay, the date of the system. So in theory, it should show the this date and then uptime of the system as well. Okay, let's cat it.
All right, so it has appended a few more stuff, which is the uptime command. Okay, so single redirection sign is going to create the file and then create the output of one command to the file it creates. Or maybe you can, if you want to nullify the content of the file, you can just create another.
Let's see if I can do echo high. So it basically did not append high, but it just removed the old stuff and then added the high just because I have used the single redirection. Okay, so you might be wondering what is echo.
Echo is nothing just to print something. Echo hello. It will just print out whatever is written in front of it.
And if you want to write more than one word, you just need to enclose it into double quotation. So basically it is more useful than it seems. It's commonly used to display the messages, print the variables, and even to write into the files just like I did.
Okay, we can combine multiple commands as well. So let's say if I want to combine that command with the uptime, and that is the only thing you need to have, semicolon. And I want to basically use another command, which is, what can we do? Who am I? Okay, so I have combined three commands using semicolon.
Okay, so first command output is this one, which is this command. And then second output is this one. And then the third output is this one.
So I have combined three commands in a single go. Any questions so far? Let's continue. All right.
So I'm going to show some of the tools that are essentials for troubleshooting servers, but they're just as useful for your home machine. Day-to-day, we rely on these tools, especially when servers are running slow or encountering any issues. For instance, if a server is sluggish, we start with these tools to pinpoint the problem, whether it's the resource usage, disk space, or network issues.
So knowing how to use them is crucial for effective troubleshooting. So I'm going to move forward to a free command. Let me clear my screen first.
And I'm going to use two flags, H and M. So H is basically human readable. Okay, so it's going to show me how much is the total memory, how much is used. Okay, so as well as the free memory.
Okay, next command I want to show you is TOP. TOP basically is a very powerful tool for system monitoring. It allows you to spot performance issues quickly and manages, you can manage the processes.
This command is perfect for real-time system monitoring. So it shows the real-time CPU usage, helping you to track with which processes are using the most resources. It also lists all the running processes and then updates every three seconds by default.
Okay, so there are many options to customize the display as well. So here, this is the process ID, which user. Okay, there are a lot of stuff here, which is not the scope of today's class.
But as the overview, you can monitor the CPU usage and the memory usage percentage. And there's a time for that process. Okay, it also shows you the load average, how many users, how much time the server has been up from, and then how many processes are there.
There are zombie processes. That's also beyond the scope of this workshop. But this is a very powerful command.
So I'm going to quit out of it. I just wanted to give you an overview of this. So how can we quit? Just press Q. Okay, I'm going to clear my screen.
HTOP, it's also, it provides the visually appealing color-coded interface with an intuitive layout. Unlike TOP, which requires manual PID input for actions, HTOP basically allows you to scroll through the process and interact with them using the arrow keys. Okay, and then if you want to kill any process, which is taking a lot of resources, you just press F9.
Don't do that right now. But this is kind of, it basically gives you a visually appealing view of the server. Okay, another good command is vmstat.
Basically, it's a lightweight tool that provides a real-time snapshot of the system performance, and it reports on CPU usage, memory usage, IO, and system processes, so helping to diagnose performance bottlenecks. So unlike more detailed tools like HTOP, vmstat gives an overview of the system health with the minimal resources. Okay, all right, so PS.
PS is basically, it's very useful for checking the processes owned by your user, giving you a quick overview of what's running in your current session. So there are a few flags we use with the PS, which are PS-E to list the processes, every process on the system. That's also, and my favorite is PS-AUX.
Okay, so it's going to list all the processes, and screen will go all the way to down, because, and if I want to, and let me show you the power of pipe, and then grab. So if I want to specify which user, sorry, in my case, WH2612, so I'm just saving the output of PS-AUX command into a pipe, and then grab my username, okay, to see what processes am I running. Okay, so it just filtered out my username, and what processes, and what are the process IDs, and what time, and how much CPU usage they're using, and all that.
Okay, all right, last but not least, how to exit out of the, or close the terminal. There are three commands I'm going to share with you. One is exit.
Okay, so I have just exited out of the session, the secure shell session, and I'm just returning back. So let's take some time, and then you can use logout, and then if you want to use the short key, which is control D. So I'm back here, so I'm going to use control D to log it out. Okay, so that's an overview, or a high-level introduction to the Linux.
Any questions? Oh, by the way, outside of these introductory sessions, Columbia offers additional training material and workshops, such as software carpentry classes, where you can receive training in Unix, and programming courses like Python. So links to these resources are listed on this presentation, which will be shared with you, and one final touch I wanted to give was some books that really helped me, both in learning Linux, and passing the Red Hat Certified System Administrator exam. These books cover almost everything from basics to advanced topics, making them great for anyone looking to advance their kind of expertise in Linux.
If you're serious about mastering Linux, these are highly recommended reads, and these are our contact information. If you want to reach out, you can reach out to us at hpc-port.columbia.edu, and that's all from my side. Back over to you.
Yeah, excellent job, Wakas. A stellar job. Everyone, thanks again to Wakas, and please join us next week.
We have the Intro to Bash session, Intro to Python a week after that, and before getting to Intro to HPC, these two workshops will give you a good foundation, and it builds up to the Intro to HPC workshop, so it integrates into one, and I'll give you a good footprint and a good foundation to how to use our clusters. Simple things like navigating, loading modules, and also implementing Python modules to actually speed up your jobs as well. But I hope you guys join us for next week.
Like Wakas stated, you have our information. If you want to follow along the slides afterwards, if you get into any errors, email us directly. If you do have any issues in your accounts, you can't log in, permission issues, whatever, open a ticket to HPC support, and one of us will actually answer for you.
But Wakas, thanks again. Great job. Thank you so much, everyone, for chiming in, and thank you, Max.
Thank you. Thank you, everyone.
And then next week we will move on to scripting in Linux and where we will learn how to automate the tasks using shell scripts. It's a crucial skill for efficiency and system management. So don't forget to register for next week's session if you haven't already.
So moving on to the Linux, what is Linux? Linux is basically an operating system that manages hardware and software allowing users to interact with their computer. It's known for being open source, secure, and highly customizable. Unlike Windows or macOS, Linux is free and comes in many different versions called distros.
It powers everything from servers and supercomputers to smartphones and embedded devices. In short, Linux is everywhere even if you don't realize you are using it. Why Linux? So it's free access to the source code allows customization and transparency.
It runs on everything from desktops to servers and embedded systems. Strong permissions, I mean strong permission management and fast community-driven updates. It rarely crashes with long uptime making it ideal for critical systems.
Efficient multitasking and great for handling heavy workloads. So many distributions are free and reducing the software expenses and large active community providing help and resources. So that's why Linux.
So why I chose Linux? When I started with Linux my main goal was to host my own website. As a student I didn't have the budget for expensive Windows server licenses so Windows required costly software like Microsoft IIS for web hosting which is like around $2,000 and then Microsoft SQL server for database is around $4,000. So the license itself.
But Linux provided a free and open source alternative with Apache for web hosting and MySQL for database. Both widely used and community supported. Plus Linux is known for its strong security and reliability making it a great long-term choice.
So by choosing Linux I had full control over my system without spending a dime on licensing. All right before jumping into navigating Linux you will probably needing access to the system. So there is a system you might have received and you should have received the email about the system and how to log into.
But let me do it in front of you. So basically here is the login you'll be using to get into Insomnia. The system name is insomnia.rcs.columbia.edu and you would use your uni prefacing the domain name.
So I'm going to do it in front of you and before that I mean for Windows 10 and 11 you can actually use the command prompt that's built in or you can download PuTTY which would be needed for older versions of Windows like 7 and 8. All right so you can download the PuTTY using the link. You can just type the link and if you have already downloaded it it's fine. For Mac users you would run the program called Terminal.
You can just search it in your search bar and then just type Terminal and it will open up the window. Okay I'll give you two minutes and I'm gonna do it in front of you so we can mimic the session. So ssh and then your uni which is in my case wh2612 at and then insomnia.rcs.columbia.edu. So why will we will be using the ssh? Why we are using secure shell? I wanted to give a little bit background on the security that's happening behind the scenes.
You know before this secure shell environment that we are going to use now everything was run by telnet and everything you typed into it could be sniffed on including usernames and passwords. So over the years programs were developed and these are all open source. So now we use this secure shell program where everything you're seeing on this slide is just I mean it's encrypted.
See if I hit enter so don't get afraid of it. Okay I want to talk just a second on what is that some of you are seeing on the screen. There is a name for this.
It's basically called trust on first use. The message you get the first time you log into any server any linux server. So you would definitely want to type yes and press enter and it will prompt you to write your or type your password that should be your uni password.
All right so if you're entering your password and don't see anything appearing on the screen don't worry that's normal. Linux doesn't show password characters for security reasons. This is to prevent anyone from guessing your password length just by looking at it.
So just type your password carefully and hit enter. It's still being entered even if you don't see it. Okay so I'm going to hit enter.
So at this point most of you should be logged into the server. I'm going to take a pause for like two to five seconds and we'll continue from there. Anyone with any technical difficulties just drop a question or any errors in the chat window and one of my team members will address it.
Any questions? Okay Syed or Wakas let's proceed. Okay all right so now that we have accessed the server let's start with some basic commands to gather some information. All right so first we can check the system's name using post name command.
So is my text big enough Max can you please make it slightly larger that's perfect thank you and rather using command prompt I have my own software to use which is Mobaxter. This is also another facility or it's kind of software where you can use the SSH. You guys can continue with your command prompt or a terminal.
Okay so let me go ahead and use my Mobaxter which is also another software. So host name basically displays the name of the system. It is useful for identifying the machine especially in networks.
Okay all right next we want to see how long the system has been running so we will use uptime for that. It shows you how long the system has been running. It is great for checking system stability so it is showing basically the time and days and how many users.
All right so basically it checks the stability of the systems. Okay finally I want to introduce you to date this basically prints the date current date of the system and time. So it helps in scheduling tasks or verifying system time setting.
All right so next command I wanted to show you was PWD. It basically is a simple but essential Linux command. It displays the full path of the current directory you are in and this is useful when navigating the file system especially in complex directory structures and so we will be using this PWD in our presentation and this intro series a lot.
So you might have a question in mind that what is a directory structure? So directory structure is basically known as file system hierarchy standards. All files and directories appear under the root directory which is slash. So if you see here you see a slash so this is the parent directory.
So it starts with parent directory so there is another directory and there is another directory and there is another directory right after it. So my present working directory will be WH2612. All right so next command basically is CD.
The CD command is used to change directories in Linux. Okay it's a quick way to navigate to the parent of all directories on the system. So see for example if I want to change to my parent directory instead of here I will just use CD space slash.
So it will take me to my parent directory which is slash. You see the difference before it was showing a tilde which is a symbol for home directory. Now it is showing slash which is the parent directory and I can verify it using pwd.
Okay and now if I want to go back to my home directory I can simply type CD. So CD will take me back to my home directory. Most of you guys probably won't be able to go to your parent directory but as an admin I can go there.
So I'm going to change it to my home directory and I can verify it using pwd. Next I wanted to introduce another command mkdir. So it is short for make directory.
This command is used to create the directories and this creates a new directory. Let me create a new directory mkdir space tmp. So I'm going to create a directory named tmp.
Okay so how can I verify that I have created this directory named tmp. So there is another command for that it is ls short for list. So it will list the content of my current working directory.
So if you guys can see it is here. All right so creating files. So now we learned how to create the directory.
We can create the files using touch command. Since this myfile.txt is already created I'm going to create another file. It'll be myfile2.txt. How can I verify ls.
So if you can see myfile2.txt is here but before when I listed it it was not there. Okay or maybe you can create another file if it's confusing. Example ls and I can see that example is here.
Example file is here. Okay how to remove the directory. So let me remove the directory.
Let me remove tmp directory. So rmdir and name of the directory. It will remove the tmp directory.
How can I verify? I can verify using ls. So there is no tmp directory here anymore. All right so I'm going to create the directory.
Okay so before creating another directory or moving forward I'm going to give you another command to clear the screen so that you don't see any clutter here. Okay so if I type clear it will clear out the screen. Okay so I'm going to create one more time tmp.
I'm going to change to my newly created directory cd into tmp and then I can verify using pwd. You see the difference before it was showing me when I was in my home directory. It was showing me till here but now it is showing me till slash tmp and if I want to go back one directory above I can use this dot dot so cd space dot dot.
So it's basically it's going to take me one directory above. So pwd now it's showing me that I am here at my home directory. Before I entered pwd I was in tmp directory.
Okay moving on to next one. See for example if I was in slash and I wanted to go back to my home directory as I mentioned I just type cd. It will take me back to my home directory or I can use tilde a tilde.
It is a special character that acts as a as a shortcut to your home directory. Okay so cd without any argument is going to take you to the home directory as well as cd tilde. It's going to also take you to the home directory.
How can I verify pwd? All right so there are two types of paths. One is absolute path. It begins with slash specifying the full path from the root directory as I mentioned before.
Starts from slash. Okay and then there is another path which is it doesn't start with slash but it is relevant to the current working directory. Okay so I'm going to give you an example of that.
Either I can do cd into tmp and then it will change the directory to tmp and I'm not using the absolute path. I'm just using relative path because I know that this tmp directory is already in my present working directory so we can verify that. So we are in our home directory and tmp is already there so I can just change my directory to tmp one and I'm going to go back to one directory above and now I can achieve the same goal by using the absolute path which which starts with slash.
So cd slash insomnia. I'm just looking at it here then home then wh2612 in your case your uni and then name of the directory. So now I am giving the full path absolute path.
Okay hit enter verify pwd. So I have achieved the same goal by entering the absolute path. Before I was entering only the relative path which was cd into tmp because I knew that this directory is already in my present working directory.
Tab completion. It is a lifesaver. Helps quickly complete the commands file names or directory names by pressing the tab key.
So see for example if I was typing this whole path. Okay let me go back one directory above. Let me clear my screen and if I want to go back to tmp directory using the absolute path.
So I'm gonna use tab completion this time. So cd into in. I just entered two words uh two characters and then hit tab.
It automatically completed the path and then if I enter h and then tab it's going to automatically complete the home. But if I enter wh there might be another user with wh. So I if I hit and if I hit tab it's going to say okay I can complete up until two but you need to see what else this home directory have.
So I can double tap it and it will show me the entries. So there is the username wh2526, wh2581, wh2612. So I need to specify the second character and then tap.
So it will complete the remaining part. Okay and then tmp and if you can see I just entered tm and then tab just completed the whole path. So tab completion is a kind of work both for commands and files and directories and it reduced the errors basically.
Next I'm gonna show you ls. We already know what is ls. I have already presented that to you but this time I'm gonna use another flag for in front of ls command.
So ls hyphen l. So l basically it's gonna if you if you need more details this command gives you deeper look at your kind of a content of the present working directory. Okay so you will see the difference when I enter ls and then if I'm entering ls hyphen l so it's basically long listing the directory the present working directory. Okay don't be scared of this this whole stuff I'm going to explain that to you.
So first is basically file type and permissions. So how can I know that this is a directory or this is a file? Basically it tells me in the start if it's a d here it's a directory. If it's a dash it's a file and sometimes you will see l in the beginning it will show that it will it is a link.
All right so next the permissions for for the file are directory. Okay and then the user who owns this file. So the second column is basically it shows that the file's owner and then the third column it indicates the group associated with the file.
Okay so in my case it is user group and then this next column it shows the size of the file in bytes. Okay and then the next part from here it shows basically the last modification date and time of that file or directory and then obviously the name of the file or directory. So ls-l gives us more kind of a detailed version of the contents of the of the directory which we are in currently we are in home directory.
So in our case there are a lot of files and directories you might be seeing maybe nothing in your in your scenario. Okay or maybe you can create a file and directory using mkdir a touch and then you can long list those. Okay next thing I want to show you is ls-a so if I enter ls-a you will see a difference some of the different files will show up I'm going to show it to you so ls has this content and let's see ls-a what does it show okay so ls-a is showing some of the files which are starting with dot I'm going to explain those to you so if we did ls those files were not there but if we do ls-a some of the dot files are there so this option reveals the files and directories that are normally hidden so whenever you see dot in front of or at the start of the file or directory so it means that's a hidden file or directory and let me combine two flags ls-al so I'm gonna list the hidden directories as well as long listing those here you are so you're seeing the difference you're seeing the dot bash profile dot bash rc these are hidden files these are system files basically so this option basically it reveals files and directories that are normally hidden and hidden from ls command okay so moving on to the next slide so these two dots these these files generally store your preferences they're typically such as your shell or other applications you can use them automatically to automatically set up your environment when you log in so you don't have to manually configure settings each time okay so dot files are a convenient way to save and manage your personalized kind of settings without affecting the visibility of your other files any questions so far all right let's move on now let's see some basic file operations um I wanted to introduce you to cp command how can we copy a file into a directory or maybe a directory to a directory okay so let me clear my screen and I'm gonna list the content of my present working directory and let me cd into tmp and let me list if it has something so my tmp directory doesn't have anything I'm gonna go back to my home directory and let's copy a file ls and if we do have any test file or maybe we can copy this my file too to my tmp directory okay so cp so that's how the syntax will work cp space and name of the file our directory and then the destination path you can use absolute path you can use relative path it's up to you but in order for me to save the time I'm going to use the relative path okay so I'm gonna copying this my file to into tmp directory okay and then I'm going to go into tmp directory to verify if my file was copied there I'm going to verify by using ls. Okay, see the difference. Before it was showing nothing when I entered the ls and now when I copied the file it's showing me the content of the my of present working directory which is tmp in this case.
So that's how you copy the files into the directories or directories into directory other directories or destinations. All right, so scp is a command line tool for securely transferring files between computers over ssh. It's useful when you need to quickly copy files between local and remote systems or even between two remote servers which I'm going to show it to you.
So it's basically copying from your local machine to your server anything from your local machine to the remote server or maybe from remote server to a remote server or maybe from a remote server to your local machine. Okay, so the syntax how is it going to work let me go to my other remote server and if I list I have created a kind of a test directory and I'm gonna go into that directory list the contents. So let's see if we can copy the example file which we created here.
Okay, so I'm gonna so that's how the syntax is gonna work scp and then the name of the file so my tab is not working because I'm in tmp directory. Okay, go back one directory above and then scp example file and then my uni at name of the server or ip you can use ip as well dot rcs and then colon and then path to the destination. So in my case if I want to know the path for my destination directory we are seeing the absolute path and that's what we need for scp.
So I'm gonna paste that path and let's see if it's going to work or not. So remember there is no example file here when we list the content of present working directory. Okay, so let's see of course it's going to ask me password for the remote server so I'm going to enter my password and if you're seeing this 100 percent and then the kbs of the file it means we have successfully sent out the file.
So let's verify it here see the difference the example file was not here but now it is here. Okay, it is very helpful using this scp when you're trying to fetch some files to your local machine or when you're trying to push your files from your local machine to the remote server. So this is the syntax scp name of the file which is in your current directory and then the username at name of the server colon and then the path absolute path to the directory to the destination directory.
That's how you use scp. Okay, all right so let's move on to the next command. So let me clear my screen here and I want to show you how to remove the files.
Now we know that how to create the files are directories. Let me show you how to remove those. So ls to list the files.
So see for example if I want to delete this example file. So how can I do that just simple rm space name of the file. So it will ask you that do you want do you really want to remove this empty file.
So I'll say yes and we can verify using ls. Okay, so example is not here anymore but remember in order for us to remove directories we need to specify a flag which is hyphen r. Hyphen r stands for recursively. So this directory might have some content in it.
So let me let me show you if I don't use this hyphen r what error is it's going to give me cannot remove tmp is a directory. So it's going to give you error because it's a directory. So we need to use hyphen r flag.
Okay, so it's going to ask you question that do you want me to delete and descend into the directory and delete the contents. I'll say yes and then it's going to ask me for each file or each content in the directory one by one. Okay, so it deleted the directory and its contents.
Let's verify there is no tmp directory here. So if I don't want to get prompted for the questions it's asking I can go ahead and let me create one more time tmp directory and some files in there. Okay, so this time I'm going to use another flag which is f. F is forcefully to avoid these questions to be asked.
So I have created the directory one more time just to show you how can we avoid these questions. So rm hyphen rf and then specify the name of the directory. So this time it won't ask me any question but it will delete the whole directory and its contents.
So far so good. Any questions? Okay, moving on to my next command which is mv. Okay, let me see.
Let me clear my screen one more time. Ls. So let's see if I can move my file.
This is my file two into the directory. So I'm going to create one more time the tmp directory and then let's go into tmp directory and let's see if it has any content. So ls is not going to give any output means this directory doesn't have any content.
Okay, I'm going to go one back one step back to my home directory where my files are and then move this my file two into the tmp directory. Okay, so the syntax is mv and name of the file and specify the path the destination. So mv source and then destination.
In my case this is my file two and then tmp. Okay, let's see if it has moved. So before I showed you how to copy a file but it didn't move the file.
This time it just moved the file so I can verify it here. Okay, let me see. So there is a confusion between my file 2.txt and my file 2. So my file 2 has has moved from my present working directory which is my home and let's see if it is in tmp ls.
So my file 2 is here. Let me go back and create an example file and avoid the confusion. There's only one example file here and then I'm going to move this example file to my tmp directory.
Okay, ls to verify if this file has been moved to the tmp directory and let's go back to tmp directory and list. So here is my example file. So that's how you move the files.
There is another purpose of mv command which is you can rename the files. So let me rename this example file to something else and without changing the content of the file. So let me see if I can give it a name tree.
Okay and then I'm gonna list the contents of present working directory. So example file has been renamed to tree. So that's another purpose of mv command.
All right, so moving on to next slide and let me clear my screen. Okay, so reading the files in Linux is very simple. You just type cat which is short for concatenate and then if you want to give a name of a file it will print out whatever the file content is.
So I just specified cat and then name of the file and it has printed all the content of the file. A small exercise for you guys is that you can create a file using touch columbia.txt in your home directory and just copy and paste a content from anything from the internet so we can use later on because I'm going to show you how to edit the files, how to write into the file, how to navigate into the file. All right, but that's how you can read the files.
Cat specified the name of the file and it will print out everything in that file. It is very useful when you're troubleshooting when you're trying to read different files or maybe shell scripts. Okay, moving on to the next slide.
So let me clear my screen. Okay, id is basically it's useful for checking a user's uid, gid which is a group id and a group membership. So I'm going to use id command and then my uni and it's going to show me what is my user id, what is my group id and how many groups do I belong to as a user.
Okay, and these groups can also be printed using the same syntax but using the command groups. It's a simpler way to see which groups a user belongs to. Okay, so in your case you might be seeing only one group or maybe two groups.
In my case, these are a bunch of groups. Okay, next slide. This command, who am I? So basically useful to quickly confirm which user account you're operating under.
Okay, so let me clear my screen. So there are two who am I's, one without any spaces, one with spaces. Okay, so let's try without any space.
Who am I? So it's going to print out my user name. Okay, now we're going to try with spaces. Who am I? So it shows basically the user who started the session which can be a different if you have switched to maybe a super user or maybe to another user but this one will show you that which user started this session.
Okay, so this is a difference between who am I and who am I with the spaces. So it will show you that which user started the session but this one shows you just your user ID. Okay, moving on to the next one.
Who? So let's check who is logged into the system. There are a lot of users. So who basically lists all the currently logged in users along with their session details such as terminal, login time, and remote host.
Okay, it is useful for monitoring active users on a multiple and on a multi-user system. See for example this one. Okay, so it shows you the user name, login name of the user.
This shows you the terminal or device the user is logged in from and this one shows you basically the date and time when the user logged in. And the last thing the IP what is this? This is basically it refers to the remote address from which the user is logged in from and this is very powerful command when you are trying to troubleshoot. Okay, let me clear my screen.
Okay, let's see more details on the on the user info. So I'm using the short version of who which is w. It gives me a little bit more details. So a w shows who is logged in and what they are doing including their login time, idle time, and the command they are currently running.
Okay, so this w provides the more comprehensive view of the user activity on the system helping you to monitor user sessions and activity in real. So as I mentioned this is just the user name. This is the terminal they are logging from.
This is the basically login time and this is the idle time the amount of time the user has been idle for and then this jcpu is basically the time used by all processes associated with the user's terminal and then this sorry this jcpu and then this pcpu is basically the time used by the current process running under that terminal and there is a what. This column shows the current command or activity being executed by the user in the terminal session. Okay, so it is a bit more detail and also it gives us the the uptime of the of the server as well and how many users are logged in.
Okay, all right let's dive into the manual pages. It's very helpful and then it's gonna it is a lifesaver. Okay, let me clear my screen.
So man is used to access basically the manual pages for a command providing detailed information on its usage options and syntax of say for example if I do man ls since we are using ls for so many times now. So man is basically short for manual so it's it's your go-to resource for learning about commands and their options directly from the terminal so you don't have to google if you want to learn about the the flags of the command. Okay, so say for example we were using hyphen a flag so it is also in the man page or manual page of the ls command and if I want to search any flag for for that command it's basically very easy.
I can just enter slash and if I want to see if hyphen l is there because we were using hyphen l before as well. All right so where is hyphen l there are a lot of hyphen l it's hyphen l hyphen l here oh it has highlighted it. So use a long listing format so it is it also explains what it does what this flag is doing if you are entering it in front of your command.
So it is very useful it's uh it's going to show you the syntax as well for the commands so what syntax you have to use okay so to quit out of it just press q. All right so let's explore some some essential file manipulation tools okay grep it's basically a powerful search tool used for you know to to filter text based on patterns okay so it is used for scanning log files configuration files or large data sets in my case let me clear my screen I do have columbia.txt and I'm going to read this columbia.txt so it has a lot a lot of information so let's see if I can filter any information using grep grep and I want to filter university word and columbia.txt if you haven't created that file columbia.txt just create it so it's going to help you to navigate through just puts any stuff copy from google and just put it there so that you can practice okay so let's see if we can grep the university word from the columbia.txt so now it is it is not showing me as compared to the cat command it is not showing me the whole content of the file but just the university word the lines which has the university word okay pipes let's see how pipe works so pipe basically it allows chaining the commands passing the output of one command as input to the another this is this is very useful for narrowing down large outputs making it easier to find specific information okay it is also more efficient way to to do this let me do this grep using using pipe so i'm going to read the file using cat columbia.txt and then i have the output of that cat command it's going to store it in the pipe and then i'm going to grab the word let's say manhattan or maybe location okay all right so it has just printed the line which had location in it so normally when you cat the cat file it will print out everything on the screen but in this case it did not it just stored the output into the pipe and then from that i manipulated that using grep okay so it is very powerful the pipes are very powerful okay tail and head so let me clear my screen okay let me do head columbia.txt so what it does is basically both of these are very useful for previewing the large files without opening them okay it's a head basically shows the first 10 lines of the of the file okay including the spaces between them so let's see if i can add columbia.txt so it just printed out the first 10 lines including the spaces okay and if i want to specify only first two lines or first three lines or maybe first five lines i can just uh enter hyphen n flag so let's do that head hyphen n and then let's see if i can print first two lines of columbia.txt so see the difference it has just printed first line and then the space after that as compared to the head okay let's see what does tail prints out so instead of the top of the the file it is it is just printing the bottom like the lower part lower 10 lines of the of the file and if i want to specify just the last line so i can just tail hyphen n n and 1 and columbia.txt so my expected output is to just to print this last line there we are okay so it is kind of a very useful uh tool to you know basically if you're searching for something in the logs maybe in the error messages it is very useful to tailor it to the last few lines or maybe first few lines so this is tail and head okay let me clear my screen all right so how can we edit the files when it comes to the text editors there is no single obvious choice but vi or vim stands out for linux admins why vi it is it is very powerful it's simple but can feel tricky at first okay and it is universally available meaning that it is pre-installed on all linux distros unix systems and even macs so it is it is very reliable when when a server crashes vi is often the only editor available making it essential for quick fixes so if you're debugging any any crashes and then you're not into the operating system itself vi editor will also will be there okay so and this is this is why many linux admins including myself rely on it so it's always there when you need it so let me manipulate the file or edit the file using vi editor so what is the what is the syntax for that vi and the name of the file columbia.txt okay so there are two main kind of modes in vi editor one is insert mode or edit mode and the second one is command mode which is which is which i'm going to show it to you so for me to go into the insert mode i just enter i and it will show you here that you are in insert mode bottom of left bottom of the line okay and and if you want to go back to the command mode you just press escape and this insert mode will go away okay in insert mode you can just type anything you want okay and then it's just like a normal editor but in escape mode you cannot type anything unless you're you type I, which takes you to the insert mode. All right, so let's explore some of the options in escape mode. If you are at the bottom of the file and if you want to go up top, and while you're in escape mode, just hit GG, or G twice, so it will take you to the top of the file.
And if you want to go to the bottom of the file, shift G will take you to the bottom of the file. So it's GG to the top of the file and shift G to the bottom of the file. All right, so the cursor will move from top of the page to the end of the page, or the file.
And if you want to, if you're at the beginning of the line and you want to go here to the end of the line, you just press shift 4 while in escape mode. This is a disclaimer, while in escape mode, you just press shift 4 and the cursor will be at the end of the line. And then if you want to go back to the start of the line, you can just press shift 6. Okay, shift 4 to the end of the line, shift 6 to the, okay, shift 6 to the start of the line, and then shift 4 to the end of the line.
All right, so if I want to delete this whole line, how can I do that? One way is to go to insert mode and then press backspace all the way to the end or to the start of the line. Another method is to just go into the escape mode and hit DD and it will delete the whole line. Okay, and if you want to undo your changes while in escape mode, you just press U which will undo the changes.
All right, so go to the start of the line by pressing shift 6 and then DD. It will delete the whole line. And then to undo it while in escape mode, press U. Okay, so there are a few more things for the vi editor I wanted to show you is while in escape mode, just press colon and then set spell.
It will basically turn on the inbuilt spell checker. So if I want to do that, hit enter and it will show you where the spelling mistakes are. And if you want to turn it off while in escape mode, just hit colon and then set no spell.
So it will just take it away. Another thing is if you want to see the number of the lines, line numbering, how can we do that? Set number. It will set the numbers of the line including the spaces and in order for us to remove that no number.
Okay, so now we know how to edit a file. Let's save this file. So in escape mode, hit colon and then W to write and then Q to quit.
And it will save the file and quit out of it. Let me go back to this file and make some changes. Let me delete this file, delete this line and then save it, quit out of it and hit enter.
Okay, and then if I go back to this columbia.txt, that line shouldn't be there. See, for example, if I want to delete this file and I am just using Q, not saving anything, not using W, it will ask me that no right since last change, add exclamation to override. Okay, so it's basically asking me to forcefully quit out of it.
Errors are your best friend in Linux, by the way. So I have deleted that file, deleted it. Say again, Max? I'll say you think so.
Yeah, you learn a lot from mistakes or errors. Okay, so let's see if that file, that line is there or not. That line is still there because I didn't save it.
Okay, so this line came back because I didn't save it. So this time I'm going to save it and then go out. And then once I'll come back, the line is not there.
Okay, so far so good. Okay, let's take a look at a few simple commands. Let's get out of this file, clear my screen.
And then as I mentioned, date is just, it's going to print out the date and time. So if I want to save the output of this command, I can just simply use this redirection sign. Okay, this is just, it redirects the output of a file or a command.
So let's do that. Date, redirection, and then let's say if I, the date of system. So this is going to create a file named the date of system and then it's going to redirect the output of this command into that file.
Okay, so let me read that file and tap. So it basically created the file with this content. And if I want to add more content to it, let's see if I want to redirect uptimes output.
This time I'm not going to use the single redirection sign. Why? Because it's going to replace this old kind of a stuff and add the new stuff, which is output of uptime command. But if I want to append it, I'll use double redirection sign.
Okay, the date of the system. So in theory, it should show the this date and then uptime of the system as well. Okay, let's cat it.
All right, so it has appended a few more stuff, which is the uptime command. Okay, so single redirection sign is going to create the file and then create the output of one command to the file it creates. Or maybe you can, if you want to nullify the content of the file, you can just create another.
Let's see if I can do echo high. So it basically did not append high, but it just removed the old stuff and then added the high just because I have used the single redirection. Okay, so you might be wondering what is echo.
Echo is nothing just to print something. Echo hello. It will just print out whatever is written in front of it.
And if you want to write more than one word, you just need to enclose it into double quotation. So basically it is more useful than it seems. It's commonly used to display the messages, print the variables, and even to write into the files just like I did.
Okay, we can combine multiple commands as well. So let's say if I want to combine that command with the uptime, and that is the only thing you need to have, semicolon. And I want to basically use another command, which is, what can we do? Who am I? Okay, so I have combined three commands using semicolon.
Okay, so first command output is this one, which is this command. And then second output is this one. And then the third output is this one.
So I have combined three commands in a single go. Any questions so far? Let's continue. All right.
So I'm going to show some of the tools that are essentials for troubleshooting servers, but they're just as useful for your home machine. Day-to-day, we rely on these tools, especially when servers are running slow or encountering any issues. For instance, if a server is sluggish, we start with these tools to pinpoint the problem, whether it's the resource usage, disk space, or network issues.
So knowing how to use them is crucial for effective troubleshooting. So I'm going to move forward to a free command. Let me clear my screen first.
And I'm going to use two flags, H and M. So H is basically human readable. Okay, so it's going to show me how much is the total memory, how much is used. Okay, so as well as the free memory.
Okay, next command I want to show you is TOP. TOP basically is a very powerful tool for system monitoring. It allows you to spot performance issues quickly and manages, you can manage the processes.
This command is perfect for real-time system monitoring. So it shows the real-time CPU usage, helping you to track with which processes are using the most resources. It also lists all the running processes and then updates every three seconds by default.
Okay, so there are many options to customize the display as well. So here, this is the process ID, which user. Okay, there are a lot of stuff here, which is not the scope of today's class.
But as the overview, you can monitor the CPU usage and the memory usage percentage. And there's a time for that process. Okay, it also shows you the load average, how many users, how much time the server has been up from, and then how many processes are there.
There are zombie processes. That's also beyond the scope of this workshop. But this is a very powerful command.
So I'm going to quit out of it. I just wanted to give you an overview of this. So how can we quit? Just press Q. Okay, I'm going to clear my screen.
HTOP, it's also, it provides the visually appealing color-coded interface with an intuitive layout. Unlike TOP, which requires manual PID input for actions, HTOP basically allows you to scroll through the process and interact with them using the arrow keys. Okay, and then if you want to kill any process, which is taking a lot of resources, you just press F9.
Don't do that right now. But this is kind of, it basically gives you a visually appealing view of the server. Okay, another good command is vmstat.
Basically, it's a lightweight tool that provides a real-time snapshot of the system performance, and it reports on CPU usage, memory usage, IO, and system processes, so helping to diagnose performance bottlenecks. So unlike more detailed tools like HTOP, vmstat gives an overview of the system health with the minimal resources. Okay, all right, so PS.
PS is basically, it's very useful for checking the processes owned by your user, giving you a quick overview of what's running in your current session. So there are a few flags we use with the PS, which are PS-E to list the processes, every process on the system. That's also, and my favorite is PS-AUX.
Okay, so it's going to list all the processes, and screen will go all the way to down, because, and if I want to, and let me show you the power of pipe, and then grab. So if I want to specify which user, sorry, in my case, WH2612, so I'm just saving the output of PS-AUX command into a pipe, and then grab my username, okay, to see what processes am I running. Okay, so it just filtered out my username, and what processes, and what are the process IDs, and what time, and how much CPU usage they're using, and all that.
Okay, all right, last but not least, how to exit out of the, or close the terminal. There are three commands I'm going to share with you. One is exit.
Okay, so I have just exited out of the session, the secure shell session, and I'm just returning back. So let's take some time, and then you can use logout, and then if you want to use the short key, which is control D. So I'm back here, so I'm going to use control D to log it out. Okay, so that's an overview, or a high-level introduction to the Linux.
Any questions? Oh, by the way, outside of these introductory sessions, Columbia offers additional training material and workshops, such as software carpentry classes, where you can receive training in Unix, and programming courses like Python. So links to these resources are listed on this presentation, which will be shared with you, and one final touch I wanted to give was some books that really helped me, both in learning Linux, and passing the Red Hat Certified System Administrator exam. These books cover almost everything from basics to advanced topics, making them great for anyone looking to advance their kind of expertise in Linux.
If you're serious about mastering Linux, these are highly recommended reads, and these are our contact information. If you want to reach out, you can reach out to us at hpc-port.columbia.edu, and that's all from my side. Back over to you.
Yeah, excellent job, Wakas. A stellar job. Everyone, thanks again to Wakas, and please join us next week.
We have the Intro to Bash session, Intro to Python a week after that, and before getting to Intro to HPC, these two workshops will give you a good foundation, and it builds up to the Intro to HPC workshop, so it integrates into one, and I'll give you a good footprint and a good foundation to how to use our clusters. Simple things like navigating, loading modules, and also implementing Python modules to actually speed up your jobs as well. But I hope you guys join us for next week.
Like Wakas stated, you have our information. If you want to follow along the slides afterwards, if you get into any errors, email us directly. If you do have any issues in your accounts, you can't log in, permission issues, whatever, open a ticket to HPC support, and one of us will actually answer for you.
But Wakas, thanks again. Great job. Thank you so much, everyone, for chiming in, and thank you, Max.
Thank you. Thank you, everyone.