BASH shell variables, commands and scripting

Certify and Increase Opportunity.
Be
Govt. Certified Linux Administrator

Back to Tutorial

First, a convention. I’ll list things for you to type in this format:

$ date

The computer’s reply like this:

Tue Dec 23 10:52:51 PST 2003

The “$” symbol in the user entry above, is a generic shell prompt, and yours will almost certainly look different (but it will include a similar symbol). I’ll be using one of two prompts (this is a common convention, worth remembering): I’ll use “$” to refer to a normal user session, and “#” to refer to a root session.

Avoid using root sessions and permissions unless it is required. Misused root authority can cause very serious harm to your system. Since this is a tutorial in which you will want to experiment with different commands, limit the chance for harm by doing so as an ordinary user. To put this another way, enter this example:

# whoami

root

If your session produced the result shown above, please — log out and become an ordinary user. In shell programming, spaces matter. If you see spaces between words and characters in these examples, be sure to include the spaces. In shell programming, case matters also. If you don’t get the results shown on this page, look at the case of your entries.

As you may be aware, a Linux filesystem is in the form of a large tree with many branches called “subdirectories”. When you issue a shell command, it is often necessary to know where you are in the “tree”. Type this example:

$ pwd

/path/path/path

When you try this example (“pwd” means “print working directory”), your current working directory will be printed. You can decide where you are in the tree. Type this example:

$ cd ~

$ pwd

/home/username

The symbol “~” is a special shortcut character that can be used to refer to your home directory. You could have typed this —

$ cd /home/username

— and accomplished the same result, but if you think about it, the “~” character is more portable. Later, when you are writing shell scripts, you might want a command that moves to any user’s home directory.

Listing Files – Directories contain files, and you can list them in a simple, compact format:

$ ls

filename filename filename …

• Or you can list them in more detail:

$ ls -la

(detailed list, one file per line)

• And, very important, to find out what a command’s options are, use the “man” (manual) command:

$ man ls

(manual page for “ls”)

The “man” command allows you to learn a command’s options. You still have to remember the command’s name.

• To find files by name:

$ find . -name ‘*.jpg’

(list of files with .jpg suffix in current and all child directories)

• To create a text diagram of the directory tree:

$ tree -d .

The “tree” command is less useful now that directory trees have become so complicated, and now that most distributions support X Windows and sophisticated filesystem browsers.

Examining Files – There are a number of things you can do to find out more about the files in the list. Here are just a few:

• The “file” command tries to identify files by examining their contents:

$ file tux_small.png

tux_small.png: PNG image data, 128 x 151, 8-bit/color RGB, non-interlaced

The next example uses the obscurely named “cat” command. It prints the contents of a file. Unfortunately if the file’s contents are not readable, they get printed anyway.

$ cat zipcodes.txt

(prints the entire contents of a file named “zipcodes.txt”)

• If a file is too long to be viewed on one page, you can say:

$ more zipcodes.txt

(prints file one screenful at a time)

• You can also use “grep” to print only those parts of a file you are interested in:

$ grep 10001 zipcodes.txt

(prints only those lines that have the character string “10001” in them)

The “grep” command is very useful, unfortunately it has a difficult-to-remember name. Be sure to:

Pipelines and Redirection – You can use a pipeline (symbolized by “|”) to make the output of one command serve as the input to another command. This idea can be used to create a combination of commands to accomplish something no single command can do. Enter this command:

$ echo “cherry apple peach”

cherry apple peach

Okay, let’s say we want to sort these words alphabetically. There is a command “sort”, but it sorts entire lines, not words, so we need to break this single line into individual lines, one line per word.

• Step one: pipe the output of “echo” into a translation (tr) command that will replace spaces with linefeeds (represented by “\n”):

$ echo “cherry apple peach” | tr ” ” “\n”

cherry

apple

peach

Success: each word appears on a separate line. Now we are ready to sort.

• Step two: add the sort command:

$ echo “cherry apple peach” | tr ” ” “\n” | sort

apple

cherry

peach

Let’s try reversing the order of the sort:

$ echo “cherry apple peach” | tr ” ” “\n” | sort -r

peach

cherry

apple

A pipeline (“|”) takes the output of one command and makes it the input to another command. Normally the output from commands is printed on the screen. But using the symbol “>”, you can redirect the output to a file:

$ date > RightNow.txt

$ cat RightNow.txt

Tue Dec 23 14:43:33 PST 2003

The above example used “>” to replace the content of any existing file having the name “RightNow.txt”. To append new data to an existing file, use “>>” instead:

$ date >> RightNow.txt

$ cat RightNow.txt

Tue Dec 23 14:43:33 PST 2003

Tue Dec 23 14:46:10 PST 2003

Use “>” to overwrite any existing file, use “>>” to append to any existing file. In both cases, if no file exists, one is created. Many commands have inputs as well as outputs. The input defaults to the keyboard, the output defaults to the screen.

• To redirect the output to a file, use “>” or “>>” as shown above.
• To make the output of a command serve as the input of another command, use “|”.
• To make the contents of a file serve as the input to a command, use “<“:

$ wc < RightNow.txt

2 12 58

As is so often the case in shell programming, there is at least one other way to produce the above result –

$ cat RightNow.txt | wc

2 12 58

Shell Script Basics – A shell script is a plain-text file that contains shell commands. It can be executed by typing its name into a shell, or by placing its name in another shell script. To be executable, a shell script file must meet some conditions:

• The file must have a special first line that names an appropriate command processor. For this tutorial, the following will work in most cases – #!/bin/bash. If this example doesn’t work, you will need to find out where your Bash shell executable is located and substitute that location in the above example.
• The file must be made executable by changing its permission bits. An example – $ chmod +x (shell script filename)

A shell script file may optionally have an identifying suffix, like “.sh”. This only helps the user remember which files are which. The command processor responsible for executing the file uses the executable bit, plus the file’s first line, to decide how to handle a shell script file. One normally executes a shell script as – $ ./scriptname.sh

This special entry is a way to tell the command processor that the desired script is located in the current directory. Always remember: if you cannot get your shell script to run, remember this trick to provide its location as well as its name.

Sample Shell Script – This will get you past the details of writing and launching a simple script. Choose a text editor you want to use. It can be a command-line editor like emacs, pico or vi, or an X Windows editor if you have this option. A shell script is little more than a list of commands that are run in sequence. Conventionally, a shellscript should start with a line such as the following:

#!/bin/bash

This indicates that the script should be run in the bash shell regardless of which interactive shell the user has chosen. This is very important, since the syntax of different shells can vary greatly. Here’s a very simple example of a shell script. It just runs a few simple commands

#!/bin/bash

echo “hello, $USER. I wish to list some files of yours”

echo “listing files in the current directory, $PWD”

ls # list files

Firstly, notice the comment on line 4. In a bash script, anything following a pound sign # (besides the shell name on the first line) is treated as a comment. ie the shell ignores it. It is there for the benifit of people reading the script.

$USER and $PWD are variables. These are standard variables defined by the bash shell itself, they needn’t be defined in the script. Note that the variables are expanded when the variable name is inside double quotes. Expanded is a very appropriate word: the shell basically sees the string $USER and replaces it with the variable’s value then executes the command.

Variables – Any programming language needs variables. You define a variable as follows:

X=”hello”

and refer to it as follows:

$X

More specifically, $X is used to denote the value of the variable X. Some things to take note of regarding semantics:

• bash gets unhappy if you leave a space on either side of the = sign. For example, the following gives an error message – X = hello
• while I have quotes in my example, they are not always necessary. where you need quotes is when your variable names include spaces, as X=hello world # error and X=”hello world” # OK

This is because the shell essentially sees the command line as a pile of commands and command arguments seperated by spaces. foo=baris considered a command. The problem with foo = bar is the shell sees the word foo seperated by spaces and interprets it as a command. Likewise, the problem with the command X=hello world is that the shell interprets X=hello as a command, and the word “world” does not make any sense (since the assignment command doesn’t take arguments).

Single and double quotes – Basically, variable names are exapnded within double quotes, but not single quotes. If you do not need to refer to variables, single quotes are good to use as the results are more predictable. An example

#!/bin/bash

echo -n ‘$USER=’ # -n option stops echo from breaking the line

echo “$USER”

echo “\$USER=$USER” # this does the same thing as the first two lines

The output looks like this (assuming your username is elflord)

$USER=elflord

$USER=elflord

so the double quotes still have a work around. Double quotes are more flexible, but less predictable. Given the choice between single quotes and double quotes, use single quotes.

Sometimes, it is a good idea to protect variable names in double quotes. This is usually the most important if your variables value either (a) contains spaces or (b) is the empty string. An example

#!/bin/bash

X=””

if [ -n $X ]; then       # -n tests to see if the argument is non empty

echo “the variable X is not the empty string”

fi

This script will give the following output:

the variable X is not the empty string

Because the shell expands $X to the empty string. The expression [ -n ] returns true (since it is not provided with an argument). A better script would have been:

#!/bin/bash

X=””

if [ -n “$X” ]; then     # -n tests to see if the argument is non empty

echo “the variable X is not the empty string”

fi

In this example, the expression expands to [ -n “” ] which returns false, since the string enclosed in inverted commas is clearly empty. Just to convince you that the shell really does “expand” variables in the sense I mentioned before, here is an example:

#!/bin/bash

LS=”ls”

LS_FLAGS=”-al”

$LS $LS_FLAGS $HOME

This looks a little enigmatic. What happens with the last line is that it actually executes the command

ls -al /home/elflord

(assuming that /home/elflord is your home directory). That is, the shell simply replaces the variables with their values, and then executes the command. Here’s a potential problem situation. Suppose you want to echo the value of the variable X, followed immediately by the letters “abc”. Question: how do you do this ? Let’s have a try :

#!/bin/bash

X=ABC

echo “$Xabc”

This gives no output. What went wrong ? The answer is that the shell thought that we were asking for the variable Xabc, which is uninitialised. The way to deal with this is to put braces around X to seperate it from the other characters. The following gives the desired result:

#!/bin/bash

X=ABC

echo “${X}abc”

Conditionals, if/then/elif – Sometimes, it’s necessary to check for certain conditions. Does a string have 0 length ? does the file “foo” exist, and is it a symbolic link , or a real file ? Firstly, we use the if command to run a test. The syntax is as follows:

if condition

then

statement1

statement2

……….

fi

Sometimes, you may wish to specify an alternate action when the condition fails. Here’s how it’s done.

if condition

then

statement1

statement2

……….

else

statement3

fi

alternatively, it is possible to test for another condition if the first “if” fails. Note that any number of elifs can be added.

if condition1

then

statement1

statement2

……….

elif condition2

then

statement3

statement4

……..

elif condition3

then

statement5

statement6

……..

fi

The statements inside the block between if/elif and the next elif or fi are executed if the corresponding condition is true. Actually, any command can go in place of the conditions, and the block will be executed if and only if the command returns an exit status of 0 (in other words, if the command exits “succesfully” ). However, in the course of this document, we will be only interested in using “test” or “[ ]” to evaluate conditions. The command used in conditionals nearly all the time is the test command. Test returns true or false (more accurately, exits with 0 or non zero status) depending respectively on whether the test is passed or failed. It works like this:

test operand1 operator operand2

for some tests, there need be only one operand (operand2) The test command is typically abbreviated in this form:

[ operand1 operator operand2 ]

To bring this discussion back down to earth, we give a few examples:

#!/bin/bash

X=3

Y=4

empty_string=””

if [ $X -lt $Y ]          # is $X less than $Y ?

then

echo “\$X=${X}, which is smaller than \$Y=${Y}”

fi

if [ -n “$empty_string” ]; then

echo “empty string is non_empty”

fi

if [ -e “${HOME}/.fvwmrc” ]; then                           # test to see if ~/.fvwmrc exists

echo “you have a .fvwmrc file”

if [ -L “${HOME}/.fvwmrc” ]; then                  # is it a symlink ?

echo “it’s a symbolic link

elif [ -f “${HOME}/.fvwmrc” ]; then     # is it a regular file ?

echo “it’s a regular file”

fi

else

echo “you have no .fvwmrc file”

fi

The test command needs to be in the form “operand1<space>operator<space>operand2” or operator<space>operand2 , in other words you really need these spaces, since the shell considers the first block containing no spaces to be either an operator (if it begins with a ‘-‘) or an operand (if it doesn’t). So for example; this

if [ 1=2 ]; then

echo “hello”

fi

gives exactly the “wrong” output (ie it echos “hello”, since it sees an operand but no operator.)

Another potential trap comes from not protecting variables in quotes. We have already given an example as to why you must wrap anything you wish to use for a -n test with quotes. However, there are a lot of good reasons for using quotes all the time, or almost all of the time. Failing to do this when you have variables expanded inside tests can result in very wierd bugs. Here’s an example: For example,

#!/bin/bash

X=”-n”

Y=””

if [ $X = $Y ] ; then

echo “X=Y”

fi

This will give misleading output since the shell expands our expression to

[ -n = ]

and the string “=” has non zero length.

Test operators – Here’s a quick list of test operators.

operator	produces true if…	number of operands
-n	operand non zero length	1
-z	operand has zero length	1
-d	there exists a directory whose name is operand	1
-f	there exists a file whose name is operand	1
-eq	the operands are integers and they are equal	2
-neq	the opposite of -eq	2
=	the operands are equal (as strings)	2
!=	opposite of =	2
-lt	operand1 is strictly less than operand2 (both operands should be integers)	2
-gt	operand1 is strictly greater than operand2 (both operands should be integers)	2
-ge	operand1 is greater than or equal to operand2 (both operands should be integers)	2
-le	operand1 is less than or equal to operand2 (both operands should be integers)	2

Loops – Loops are constructions that enable one to reiterate a procedure or perform the same procedure on several different items. There are the following kinds of loops available in bash – for loops and while loops

For loops – The syntax for the for loops is best demonstrated by example.

#!/bin/bash

for X in red green blue

do

echo $X

done

The for loop iterates the loop over the space seperated items. Note that if some of the items have embedded spaces, you need to protect them with quotes. Here’s an example:

#!/bin/bash

colour1=”red”

colour2=”light blue”

colour3=”dark green”

for X in “$colour1″ $colour2″ $colour3”

do

echo $X

done

Can you guess what would happen if we left out the quotes in the for statement ? This indicates that variable names should be protected with quotes unless you are pretty sure that they do not contain any spaces.

Globbing – The shell expands a string containing a * to all filenames that “match”. A filename matches if and only if it is identical to the match string after replacing the stars * with arbitrary strings. For example, the character “*” by itself expands to a space seperated list of all files in the working directory (excluding those that start with a dot “.” ) So

echo *

lists all the files and directories in the current directory.

echo *.jpg

lists all the jpeg files.

echo ${HOME}/public_html/*.jpg

lists all jpeg files in your public_html directory.

As it happens, this turns out to be very useful for performing operations on the files in a directory, especially used in conjunction with a for loop. For example:

#!/bin/bash

for X in *.html

do

grep -L ‘<UL>’ “$X”

done

Wildcards are single characters that stand for or substitute for one or more values. Bash uses the familiar * and ? as wildcards. * stands for one or more characters and ? represents any single character. Set operators allow you to define a range or set of characters to use as wildcards. The notation for defining a set is [set ], where [ and ] delimit the range and set lists the alphanumeric characters making up the range. The range can be inclusive, disjoint, discrete, or a combination of all three. An inclusive set includes all the characters in the set, and is defined using a hyphen, for example, [b–f] . A disjoint set is at least two inclusive ranges separated by a comma (,), such as [1–4,7–0]. A discrete set refers to a simple list of characters, such as [13579]

While Loops – While loops iterate “while” a given condition is true. An example of this:

#!/bin/bash

X=0

while [ $X -le 20 ]

do

echo $X

X=$((X+1))

done

This raises a natural question: why doesn’t bash allow the C like for loops

for (X=1,X<10; X++)

As it happens, this is discouraged for a reason: bash is an interpreted language, and a rather slow one for that matter. For this reason, heavy iteration is discouraged.

Command Substitution – Command Substitution is a very handy feature of the bash shell. It enables you to take the output of a command and treat it as though it was written on the command line. For example, if you want to set the variable X to the output of a command, the way you do this is via command substitution. There are two means of command substitution: brace expansion and backtick expansion.

Brace expansion workls as follows: $(commands) expands to the output of commands This permits nesting, so commands can include brace expansions. Backtick expansion expands `commands` to the output of commands. An example is as

#!/bin/bash

files=”$(ls)”

web_files=`ls public_html`

echo “$files”     # we need the quotes to preserve embedded newlines in $files

echo “$web_files” # we need the quotes to preserve newlines

X=`expr 3 \* 2 + 4` # expr evaluate arithmatic expressions. man expr for details.

echo “$X”

The advantage of the $() substitution method is almost self evident: it is very easy to nest. It is supported by most of the bourne shell varients (the POSIX shell or better is OK). However, the backtick substitution is slightly more readable, and is supported by even the most basic shells (any #!/bin/sh version is just fine). If strings are not quote-protected in the above echo statement, new lines are replaced by spaces in the output.

Input and Output – The read built-in command is the counterpart of the echo and printf commands. The syntax of the read command is as follows:

read [options] NAME1 NAME2 … NAMEN

One line is read from the standard input, or from the file descriptor supplied as an argument to the -u option. The first word of the line is assigned to the first name, NAME1, the second word to the second name, and so on, with leftover words and their intervening separators assigned to the last name, NAMEN. If there are fewer words read from the input stream than there are names, the remaining names are assigned empty values. The characters in the value of the IFS variable are used to split the input line into words or tokens.

In bash, the return value of a program is stored in a special variable called $?. This illustrates how to capture the return value of a program, I assume that the directory dada does not exist.

#!/bin/bash

cd /dada &> /dev/null

echo rv: $?

cd $(pwd) &> /dev/null

echo rv: $?

Bash sports a surprisingly rich collection of input and output features. Many of them, such as manipulating file descriptors or reading and writing to device files. The basic I/O redirectors, > and <, redirect output from stdout and input from stdin, respectively. Most often, > is used to save the output from a command to a file. For example, the command cat /etc/passwd > outcaptures the output of the cat command and sends it to a file named out in the current directory. To capture stderr output to a file, put a 2 immediately in front of >. For example, if you invoke tar with no options or arguments, it displays an error message to stderr (usually the display).