Command Line Wizardry 2.
Today, I thought I might just write about some general commands that help navigate files safely.
Now, the commands in the arsenal of what we’ll be using are:cat, more, grep, awk, sed, and vim
Now, in general if you’re working with files you fall into two categories: Looking at them, and editing them. It is really important to know how to do both of those tasks separately so you eliminate a lot of accidental mistakes.
So lets talk about looking at files. The first command is cat. Cat is a concatenate tool. It is most commonly used for dumping output to the console, so if you did:
cat foo.bar
It would spill its guts on the screen. Sometimes that is exactly what you want, especially if you are going to pipe the output somewhere else. Usually you want to read the contents, and the command more is used for that.
more foo.bar
This will also spill the guts of the file out, but it will only do it one screen at a time, it is like a file viewer. Press spacebar to hop a page down, q to quit, and you can even search in a file. To search type a / followed by the search string and press enter. It will find the first match. Press n to find the next match. Say you want to go to a specific line number, type the line number in and press G. That is a capital G.
The next tool that is extraordinarily useful is grep. Grep finds things in files. Grep is way too powerful to go into a huge amount of detail, but the basic format is.
grep -options search_string files
So, without options it will output the filename and the matched line. Sometimes you want line numbers (So you can check it out in more). So you can use the n option. Sometimes you want to ignore case because people who made the file are inconsistent and that is the i option. Now grep’s real power is that you can use regular expressions to search. I’m not going into regular expressions here, but if you need to find strings of data that are somewhat variable, then they are the way to do it.
Up until this point, the commands I’ve explained will not alter a file’s contents, but allow you some pretty powerful methods to find what you want in them. The next group of tools will allow you to manipulate them, and require a certain amount of care.
First up is awk. Awk is actually an interpreted programming language. It is really powerful and equally cryptic. Now, it is really useful for finding data in formatted text. Lets say you only wanted to look at file sizes.
ls -l | awk '{print $5}'
The ls -l will list the contents of the directory with the -l flag listing all the details such as access rights, ownership, size, date, and name. Then it is piped into (Fed into), the awk command. Now the part in single quotes is the program. The program consists of a print command and the variable $5. See, awk splits strings up by default on whitespace and creates an array of $0-$X where $X is the last item split. $0 is the entire line. So for this example, print $1 will print the rights, $2 will print the number of files contained, $3 the owner, $4 the group, $5 the size, $6 month, $7 day, $8 time, and $9 the actual file name.
Now awk is way more powerful than that. You can use conditionals to say, “Hey, if there are 5 fields on the line, do this,” which is really great if you have a file with lots of different sorts of information in it.
Next up in the commands is sed. Sed is like awk in that it is extremely powerful and somewhat hard to use. Sed is based on commands and addresses. Addresses are almost always defined by regular expressions, but other options exist. By default sed doesn’t alter a file, but it can if you aren’t careful. The option to be especially careful of is the -i option which tells sed to do all of its work, “In place.” So, the basic sed command that is most useful is:
sed -e 's/foo/bar/g' < file
This brings the content of file into sed, which has the command of s which is search for foo and replace with bar. The g at the end means globally, which will do that in the entire file. So what you have is a command line find and replace. Now, like I said, this won’t alter the file, but instead it will spit out all of the work it did to your screen. Its a great way to test your command out first. If you wanted to have it edit the file in place you can use this.
sed -i bak -e 's/foo/bar/g' file
Now, the new edition is the -i option. That tells sed to operate on the file instead of just telling you what it would do. The -i option asks for an extension for a backup file to be created. Thats because people make more mistakes with this one option than not. So it will create a backup of your file named file.bak. If you are 100% sure about your command you can do this.
sed -i '' -e 's/foo/bar/g' file
There is no turning back from that one, it will find all occurrences of foo and change them to bar.
Lastly is vim. Vim is an editor. Vim is an editor on drugs. The first time you use it, you probably won’t be able to figure out how to quit, move the cursor, or even type. Sounds great doesn’t it? Once you get over it’s interface, it can be extremely efficient to edit files in. There are tons of vim tutorials out there and I recommend you use them. Of course, most systems with vim have the vitutor command which will teach you as well.
So to edit a file you would do this:
vim file
That will open up vim with the contents of the file loaded. I will do my best to give a quick and dirty overview of how vim operates, so at least you can move around and quit.
So, first up is moving around. That is done by the hjkl keys. That takes a little while to get used to, but becomes natural quickly enough. If you want it to be second nature, play nethack a little bit.
Okay, so you can move around. Now, how do you quit? Well a fair bit of commands have to be entered a special way. The best way to describe this is that when you’re moving around you’re kind of in a editing mode where keys do various things. Now if you press : you will be able to enter commands. You’ll see it down in the lower left corner. Press escape to abort whichever mode you are in. That is really important to remember. If you aren’t sure what you are doing, escape will probably bring you back to normal.
So you press : and now there are a couple of different ways to quit. To write the file and quit you do :wq To exit without saving you do :qa!
Now, you can move around and quit. Lets talk some editing. First up is the i key. If you press i in vim you will enter insert mode and be able to type wherever your cursor is. To stop typing press escape. Be wary of using actual arrow keys and backspace. They may not be set up. If thats the case you’ll have to do some googling to map those keys correctly.
Now, more often than not you’ll just want to change a word or letter or line. There are super easy ways to do that.
To change something start with the c key. Now if you want to change a word press w. So actually you will move the cursor over a word and press cw. Then you should see a $ pop up at the end of that word. Simply type your changes, and press escape. If you want to change a letter, move over top of it and press r. Then type the key you want to replace it with.
If you want to type after your cursor, press a. If you want to type on the next line, press o. Capital O does something very similar, play around with them both.
To delete things, you will use the d command. Now if it is just one letter, just press x over the character you want gone. Otherwise we’ll use d. If you want the word deleted, move the cursor over it and press dw. If you want the entire line, press dd. If you want everything from your cursor to the end of the line press d$.
I should mention that you have one level of undo with the u key.
Vim is really powerful and worth learning if you find yourself in a shell. There is a huge division among followers of vim vs emacs. I use vim because that was simply what I first saw, but getting comfortable with one is really important.
Thats all I have for now, but I’m pretty sure thats enough to put 80% of you to sleep.
One more command that will help out hugely when you are first learning about how to use files is a good way to back one up. Here is the command I use for that:
cp foo foo.`date +%m%d%Y`
Now, this is the copy command. It copies the file named foo to foo.02122010. The `date +%m%d%y` is actually a nested command, anything in “ is run as a command. So this runs the date command and the +%m%d%Y formats the string to monthdayyear. Very helpful for backing things up before you go monkey around.
February 12th, 2010 at 2:46 pm by rl