SHELL SCRIPTS
Usually when people say “shell scripting” they have on mind bash, ksh, sh, ash, or similar Linux/UNIX scripting language. Scripting is another way to communicate with the computer. Using graphic windows interface (not matter windows or Linux) users can move the mouse and click on the various objects like buttons, lists, checkboxes, and so on. But it is a very inconvenient way witch requires user participation and accuracy each time he would like to ask the computer / server to do the same tasks (let's say to convert photos or download new movies, mp3, etc). To make all these things easily accessible and automated we could use shell scripts.
Some programming languages like Pascal, FoxPro, C, java needs to be compiled before they could be executed. They need an appropriate compiler to make our code do some job.
Other programming languages like PHP, JavaScript, VisualBasic do not need a compiler. So they need interpreters and we could run our program without compiling the code.
The shell scripts are also like interpreters, but it is usually used to call externally compiled programs. Then captures the outputs, exit codes, and act accordingly.
One of the most popular shell scripting languages in the Linux world is the bash. And I think (this is my own opinion) this is because bash shell allows users easily navigate through the history commands (previously executed) by default, in opposite ksh which requires some tuning in .profile or remember some “magic” key combination to walk through history and amend commands.
OK, I think this is enough for the introduction and I leaving for your judge which environment is most comfortable for you. Since now I will speak only about bash and scripting. In the following examples, I will use CentOS 6.6 and bash-4.1.2. Just make sure you have the same or greater version.
Shell Script Streams
The shell scripting is something similar to the conversation of several persons. Just imagine that all commands like the persons who able to do something if you properly ask them. Let's say you would like to write the document. First of all, you need the paper, then you need to say the content to someone to write it, and finally, you would like to store it somewhere. Or you would like to build a house, so you will ask appropriate persons to clean up space. After they say “it's done” then other engineers could build for you the walls. And finally, when engineers also tell “It's done” you can ask the painters to color your house. And what would happen if you ask the painters coloring your walls before they are built? I think they will start to complain. Almost all commands like the persons could speak and if they did its job without any issues they speak to “standard output”. If they can’t do what you are asking — they speaking to the “standard error”. So finally all commands listening for you through “standard input”.
A quick example- when you opening Linux terminal and writing some text — you speaking to bash through “standard input”. So ask the bash shell who am i
Now let's ask something that bash will not understand us:
The first word before “:” usually is the command which complaining to you. Actually each of these streams has its own index number:
If you really would like to know to witch output command said something — you need to redirect (to use “greater than “> “ symbol after command and stream index) that speech to file:
In this example, we tried to redirect 1 (stdout) stream to file named output.txt. Let's look does to the content of that file. We use the command cat for that:
Seams that are empty. Ok now let's try to redirect 2 (stderr) stream:
Ok, we see that complaints are gone. Let's check the file:
Exactly! We see that all complaints were recorded to the errors.txt file.
Sometimes commands produce stdout and stderr simultaneously. To redirect them to separate files we can use the following syntax:
To shorten this syntax a bit we can skip the “1” as by default the stdout stream will be redirected:
OK, let’s try to do something “bad”. let's remove the file1 and folder1 with the rm command:
Now check our output files:
As we see the streams were separated into different files. Sometimes it is not handy as usual, we want to see the sequence when the errors appeared — before or after some actions. For that we can redirect both streams to the same file:
We can redirect one stream to another:
Let me explain. All stdout of the command will be redirected to the out_err.txt. The errout will be redirected to the 1-st stream which (as I already explained above) will be redirected to the same file. Let see the example:
Looking at the combined output we can state that first of all rm command tried to remove the folder2 and it was not a success as Linux require the -r key for rm command to allow remove folders. At the second the file2 was removed. By providing the -v (verbose) key for the rm command we asking rm command to inform as about each removed file or folder.
This is almost all you need to know about redirection. I say almost because there is one more very important redirection which called “piping”. By using | (pipe) symbol we usually redirecting the stdout stream.
Let's say we have the text file:
and we need to find the lines in it with the words “Hello”. Linux has the grep command for that:
This is OK when we have a file and would like to search in it. But what if we need to find something in the output of another command? Yes, of course, we can redirect the output to the file and then look in it:
As we see this way is not very handy as soon we will mess the space with temporary files. For that, we can use the pipes. They allow us to redirect one command stdout to another command stdin streams:
As we see, we get the same result without any temporary files. We have redirected frisk stdout to the grep stdin.
There are several other redirections but we will speak about them later.
Displaying custom messages in the shell
As we already know usually communication with and within the shell is going as dialog. So let's create some real script which also will speak with us. It will allow you to learn some simple commands and better understand the scripting concept.
Imagine we are working in some company as a help desk manager and we would like to create some shell script to register the call information: phone number, User name, and a brief description of the issue. We going to store it in the plain text file data.txt for future statistics. The script itself should work in a dialog way to make life easy for help desk workers. So first of all we need to display the questions. For displaying any messages there are echo and printf commands. Both of them displaying messages, but printf is more powerful as we can nicely form output to align it to the right, left, or leave dedicated space for a message. Let's start with a simple one. For file creation please use your favorite text editor (kate, nano, vi, …) and create the file named note.sh with the command inside:
or
echo -n “Phone number?”
Example
How to run/execute a script
After you have saved the file we can run it with bash command by providing our file as an argument:
Actually to use this way for script execution is not handy. It would be more comfortable to just execute the script without any bash command as a prefix. To make it executable we can use chmod command:
As we see, before chmod command execution, the script has only read (r) and write
permissions. After chmod +x it got execute (x) permissions. (More details about permissions I am going to describe in the next article.) Now we can simply run it:
It also works.
Everything would be OK if all Linux users would have the same default shell. If we simply execute this script default user shell will be used to parse script content and run the commands. Different shells have a bit different syntax, internal commands, etc. So to guarantee the bash will be used for our script we should add #!/bin/bash as the first line. In this way default user shell will call /bin/bash and only then will execute following shell commands in the script:
Only now we will be 100% sure that bash will be used to parse our script content. Let's move on.
Reading the inputs
After we have displayed the message script should wait for an answer from the user. There is the command read:
After execution script will wait for the user input until he presses the [ENTER] key:
Everything you have input will be stored to the variable To display the value of the variable we can use the same echo command:
Ok, now we are ready to add the rest questions:
Using stream redirection
Perfect! There is left to redirect everything to the file data.txt. As a field separator, we going to use / (slash) symbol.
Note: You can choose any which you think is the best, bat be sure that content will not have these symbols inside. It will cause extra fields in the line.
Bingo. Let’s run once again:
Our file is growing. Let's add the date in the front of each line. This will be useful later when playing with data while calculating statistics. For that we can use command date and give it some format as I do not like default one:
There are several ways to read the command output to the variable. In this simple situation, we will use ` (backquotes). Instead of back quotes, we can also use $( ). Say for example now = $(date)
Hmmm… Our script looks a bit ugly. Let's prettify it a bit. If you would read manual about read command you would find that read command also could display some messages. For this we should use -p key and message:
I agree it looks much better!
And the cursor is right after the message (not in a new line) what makes a bit sense.
It’s time to improve our script. If the user works all day with the calls it is not very handy to run it each time. Let’s add all these actions in the never-ending loop:
Using pipe redirection
Let's add more functionality to our “Frankenstein” I would like the script will display some statistics after each call. Let's say we want to see how many times each number called us. For that we should cat the data.txt file:
Now all this output we can redirect to the cut command to cut each line into the chunks (our delimiter “/”) and print the second field:
Now, this output we can redirect to another command to sort:
and leave only unique lines. To count unique entries just add -c key for uniq command:
Just add this to end of our loop:
The current scenario is going through well-known steps like:
- Display message
- Get user input
- Store values to the file
- Do something with stored data
But what if user has several responsibilities and he needs sometimes to input data, sometimes to do statistic calculations, or might be to find something in stored data? For that we need to implement switches / cases. In next article i will show you how to use them and how to nicely form the output. It is useful while “drawing” the tables in the shell.
*******************************************************************
The script helps to remember all main points during creating user and quickly create a new users. If admin needs to create 100 users at once this script could be helpful. Script use command useradd with keys and groupadd.
Shell Script
#!/bin/bash
while [ x$username = “x” ]; do
read -p “Please enter the username you wish to create : “ username
if id -u $username >/dev/null 2>&1; then
echo “User already exists”
username=””
fi
done
while [ x$group = “x” ]; do
read -p “Please enter the primary group. If group not exist, it will be created : “ group
if id -g $group >/dev/null 2>&1; then
echo “Group exist”
else
groupadd $group
fi
done
read -p “Please enter bash [/bin/bash] : “ bash
if [ x”$bash” = “x” ]; then
bash=”/bin/bash”
fi
read -p “Please enter homedir [/home/$username] : “ homedir
if [ x”$homedir” = “x” ]; then
homedir=”/home/$username”
fi
read -p “Please confirm [y/n]” confirm
if [ “$confirm” = “y” ]; then
useradd -g $group -s $bash -d $homedir -m $username
fi
Sample Result
sudo ./linux_user.sh
Please enter the username you wish to create : test
Please enter the primary group. If group not exist, it will be created : test
Please enter bash [/bin/bash] :
Please enter homedir [/home/test] :
Please confirm [y/n]y
22:12:58 [test@Desktop] :~ id test
uid=1003(test) gid=1003(test) groups=1003(test)
Learn Above Script Line by Line
#Write to console ask to enter group and save input to group variable
read -p “Please enter the primary group. If group not exist, it will be created : “ group
#check if group already exist
if id -g $group >/dev/null 2>&1; then
#just warn that group already exist
echo “Group exist”
else
#if group not exist — create one more
groupadd $group
fi
#end of while loop
done
#ask to enter preferred bash
read -p “Please enter bash [/bin/bash] : “ bash
#check if no input
if [ x”$bash” = “x” ]; then
#if no input, use default bash
bash=”/bin/bash”
fi
#ask to enter preferred homedir
read -p “Please enter homedir [/home/$username] : “ homedir
#check if no input
if [ x”$homedir” = “x” ]; then
#if no input , use default homedir
homedir=”/home/$username”
fi
#ask to confirm all inputs
read -p “Please confirm [y/n]” confirm
#if input y
if [ “$confirm” = “y” ]; then
#command to add user with all entered info
useradd -g $group -s $bash -d $homedir -m $username
fi
*******************************************************************
Shell Script that checks if service is running and if its not then will try to start it. It's good for a system admin to make sure the crucial services are running all the time.
Shell Script
#!/bin/bash
if [ “$#” = 0 ]
then
echo “Usage $0 “exit 1
fi
service=$1
is_running=`ps aux | grep -v grep| grep -v “$0” | grep $service| wc -l | awk ‘{print $1}’`
if [ $is_running != “0” ] ;
then
echo “Service $service is running”
else
echo
initd=`ls /etc/init.d/ | grep $service | wc -l | awk ‘{ print $1 }’`
if [ $initd = “1” ];
then
startup=`ls /etc/init.d/ | grep $service`echo -n “Found startap script /etc/init.d/${startup}. Start it? Y/n ? “
read answer
if [ $answer = “y” -o $answer = “Y” ];
then
echo “Starting service…”/etc/init.d/${startup} start
fi
fi
fi
Results
bobbin@linoxide:/$ ./service.sh apparmor
Service apparmor is not running
Found startap script /etc/init.d/apparmor. Start it? Y/n ? Y
Starting service…
* Starting AppArmor profiles [OK]
Learn Above Script Line By Line
#check if service name passed to script as argument, if there no arguments (0) do next
if [ “$#” = 0 ]
then
“Usage $0 “ #write to terminal usage
echo
#since no arguments — we need to exit script and user re-run it
exit 1
fi
#get service name from first argument
service=$1
#this check ,if service running using ps command, after we remove our process from output, since script will also match, with wc we count number of matching lines.
is_running=`ps aux | grep -v grep| grep -v “$0” | grep $service| wc -l | awk ‘{print $1}’`
#is number of lines are not 0 do next
if [ $is_running != “0” ] ;
then
#just put this line to terminal
echo “Service $service is running”
#if number of precesses is 0
else
#Service $service is not running” #just put this string to terminal
echo
#checking for files in /etc/init.d
(directory with start-up scripts) with name similar to service
initd=`ls /etc/init.d/ | grep $service | wc -l | awk ‘{ print $1 }’`
#if there is script with similar name
if [ $initd = “1” ];
then
#this line get name of startup script (ls — lists files in directory
startup=`ls /etc/init.d/ | grep $service`
echo -n “Found startap script /etc/init.d/${startup}. Start it? Y/n ? “
#waiting for user answer
read answer
#if answer Y or y
if [ $answer = “y” -o $answer = “Y” ];
then
echo “Starting service…”
#running startup script
/etc/init.d/${startup} start
fi
#exit of if loop
fi
*******************************************************************
The script that check used space on all mounted devices and warn if the used space is more than the threshold. In this scenario we were using 25% of disk, that’s why the threshold is so small.
Linux Shell Script
#!/bin/bash
threshold=”20″
i=2
result=`df -kh |grep -v “Filesystem” | awk ‘{ print $5 }’ | sed ‘s/%//g’`
for percent in $result; do
if ((percent > threshold))
thenpartition=`df -kh | head -$i | tail -1| awk ‘{print $1}’`
echo “$partition at $(hostname -f) is ${percent}% full”
fi
let i=$i+1
done
Test Script Result
bobbin@linoxide:/$ ./df_script.sh
/dev/sda6 at linoxide.lviv.example.com is 25% full
Learn Above Shell Script
#This set threshold value
threshold=”20″
#Counter, will be used later, set to 2, since first line in df output is description.
i=2
#Getting list of percentage of all disks, df -kh show all disk usage, grep -v — without description line, awk ‘{ print $5 }’ — we need only 5th value from line and sed ‘s/%//g’ — to remove % from result.
result=`df -kh |grep -v “Filesystem” | awk ‘{ print $5 }’ | sed ‘s/%//g’`
#for every value in result we start loop.
for percent in $result; do
#compare, if current value bigger than threshold, if yes next lines.
if ((percent > threshold))
then
#taking name of partition, here we use counter. Df list of all partitions, head — take only $i lines from top, tail -1 take only last line, awk ‘{print $1}’ — take only first value in line.
partition=`df -kh | head -$i | tail -1| awk ‘{print $1}’`
#print to console — what partition and how much used in %.
echo “$partition at $(hostname -f) is ${percent}% full”
#end of if loop
fi
#counter increased by 1.
let i=$i+1
#end of for loop.
done
*******************************************************************
This script / daemon that send email of every login / logout to server. This helps sysadmin aware of login and logout to the server for monitoring purposes. Each line has its comment to understand how it works.
Shell Script
#!/bin/bash
subject=”New login on $(hostname -f)”
email=”test@linoxide.com”while true; do
LASTLOGIN=`last | head -1`
if [ x”$PREVLOGIN” = “x” ]; then
PREVLOGIN=$LASTLOGIN
else
if [ “$LASTLOGIN” != “$PREVLOGIN” ];
then
#echo “$LASTLOGIN” | mail -s “$subject” “$email”echo “$LASTLOGIN $subject”
Script Output
Since it’s daemon, and while never ends, exited after pressing CTRL+C
Learning Shell Script
#Variable to set subject of email
subject=”New login on $(hostname -f)”
#email address
email=”test@linoxide.com”
#To run as daemon, this will be false everytime.
while true; do
#We take last line from output of last command. Last command show last logins/logouts, newest on the top. Head -1 to take first line.
LASTLOGIN=`last | head -1`
#compare if previous value if empty, to avoid sending emails on restart.
if [ x”$PREVLOGIN” = “x” ]; then
#if previous value empty — write to previous value current last line with login.
PREVLOGIN=$LASTLOGIN
#if previous value is not empty — do next:
else
#compare if previous value and last value the same, if not do next:
if [ “$LASTLOGIN” != “$PREVLOGIN” ];
#uncoment this line if you want to receive emails.First we echo lastlogin line, after redirect it to mail command, mail command to send emails, -s to specify subject, last agrument email to where to sent.
then
#echo “$LASTLOGIN” | mail -s “$subject” “$email”
#this line just to write to terminal result.
echo “$LASTLOGIN $subject”
#if email sent we write to previous value current value, so will sent email only on new entry.
PREVLOGIN=$LASTLOGIN
#finish of if loop, if previous entry the same as last one no need to do anything.
fi
#finish of first if loop.
fi
#sleep 1 second, to do nothing, to not use CP.
sleep 1
#finish of while loop.
done
*******************************************************************
This shell script enables you to perform a ping to the remote host and check whether the mentioned port is opened on that server. This helps system admin for ping test and also make sure if there any issues with specific ports.
Shell Script
#!/bin/bash
if ping -q -c 4 $host >/dev/null
thenping_result=”OK”
else
ping_result=”NOT OK”fi
nc_result=`nc -z $host $port; echo $?`
message=”Ping to host — ${ping_result}, port $port ${port_result}.”
if [ “$ping_result” != “OK” -o “$nc_result” != “0” ];
then
echo “$message”echo “$message” | mail -s “$subject” $email
fi
Script Output
Ping to localhost and check is 22 port opened (ssh server)
bobbin@linoxide:/$ ./script 127.0.0.1 22
Ping to host — OK, port 22 not opened.
desktop:~/$
Learning Shell Script
#Check if service name passed to script as argument, if there no arguments (0) do next
if [ “$#” = “0” ];
then
#Write to terminal usage
echo “Usage: $0 “
#Since no arguments — we need to exit script and user re-run #Writing parameters to variables
host=$1
port=$2
email=”test@linoxide.com”
subject=”Script result”
#Check if ping ok -q to quite mod, -c 4 for 4 checks
if ping -q -c 4 $host >/dev/null
then
#Next lines writes result variable
ping_result=”OK”
else
ping_result=”NOT OK”
fi
#Next command check if port opened via nc command, and getting exit status of nc command
nc_result=`nc -z $host $port; echo $?`
#Check of exit status of nc command, and write results to variables
if [ $nc_result != 0 ];
then
port_result=”not opened”
else
port_result=”opened”
fi
#Writing message that script will email and write to output
message=”Ping to host — ${ping_result}, port $port ${port_result}.”
#Next check if ping or port check is failed (ping if not OK and exit status of nc if not 0)
if [ “$ping_result” != “OK” -o “$nc_result” != “0” ];
then
echo “$message”#this line write warning message to terminal
echo “$message” | mail -s “$subject” $email #this line send email
fi
*******************************************************************
This article we bring shell scripts to backup your files / directories from you local linux machine to a remote linux server using rsync command. This would be an interactive way to perform backup , where you need to provide remote backup server hostname / ip address and folder location. We keep a separate file where you need to provide files / directories that need backup. We have added two scripts where first script ask password after each file had been copied (if you have enabled ssh authentication keys , then password will be not be asked) and in second script password will be prompted only once.
We are going to backup bckup.txt, dataconfig.txt, docs and oracledb.
[root@Fedora21 tmp]# ls -l
total 12
-rw-r-r-. 1 root root 0 May 15 10:43 bckrsync.sh
-rw-r-r-. 1 root root 0 May 15 10:44 bckup.txt
-rw-r-r-. 1 root root 0 May 15 10:46 dataconfig.txt
drwxr-xr-x. 2 root root 4096 May 15 10:45 docs
drwxr-xr-x. 2 root root 4096 May 15 10:44 oracledb
This file contains backup files / dir details
[root@Fedora21 tmp]# cat /tmp/bckup.txt
/tmp/oracledb
/tmp/dataconfig.txt
/tmp/docs
[root@Fedora21 tmp]#
Script 1 :
#!/bin/bash
#We will save path to backup file in variable
backupf=’/tmp/bckup.txt’#Next line just prints message
echo “Shell Script Backup Your Files / Directories Using rsync”#next line check if entered value is not null, and if null it will reask user to enter Destination Server
while [ x$desthost = “x” ]; do#next line prints what userd should enter, and stores entered value to variable with name desthost
read -p “Destination backup Server : “ desthost#next line finishes while loop
done#next line check if entered value is not null, and if null it will reask user to enter Destination Path
while [ x$destpath = “x” ]; do#next line prints what userd should enter, and stores entered value to variable with name destpath
read -p “Destination Folder : “ destpath#next line finishes while loop
done#Next line will start reading backup file line by line
for line in `cat $backupf`#and on each line will execute next
do#print message that file/dir will be copied
echo “Copying $line … “
#copy via rsync file/dir to destinationrsync -ar “$line” “$desthost”:”$destpath”
#this line just print done
echo “DONE”#end of reading backup file
done
Running the script with output
[root@Fedora21 tmp]# ./bckrsync.sh
Shell Script Backup Your Files / Directories Using rsync
Destination backup Server : 104.*.*.41
Destination Folder : /tmp
Copying /tmp/oracledb …
The authenticity of host ‘104.*.*.41 (104.*.*.41)’ can’t be established.
ECDSA key fingerprint is 96:11:61:17:7f:fa:……
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘104.*.*.41’ (ECDSA) to the list of known hosts.
root@104.*.*.41’s password:
DONE
Copying /tmp/dataconfig.txt …
root@104.*.*.41’s password:
DONE
Copying /tmp/docs …
root@104.*.*.41’s password:
DONE
[root@Fedora21 tmp]#
Script 2 :
#!/bin/bash
#We will save path to backup file in variable
backupf=’/tmp/bckup.txt’#Next line just prints message
echo “Shell Script Backup Your Files / Directories Using rsync”#next line check if entered value is not null, and if null it will reask user to enter Destination Server
while [ x$desthost = “x” ]; do#next line prints what userd should enter, and stores entered value to variable with name desthost
read -p “Destination backup Server : “ desthost#next line finishes while loop
done#next line check if entered value is not null, and if null it will reask user to enter Destination Path
while [ x$destpath = “x” ]; do#next line prints what userd should enter, and stores entered value to variable with name destpath
read -p “Destination Folder : “ destpath#next line finishes while loop
done#Next line will start reading backup file line by line
for line in `cat $backupf`#and on each line will execute next
dospawn rsync -ar ${line} ${desthost}:${destpath}
#as result of previous command we expect “password” promtp
expect “*?assword:*”
#next command enters password from script
send “${password}\r”
#next command tells that we expect end of file (everything finished on remote server)
expect eof
#end of expect pard
EOD
#this line just print done
echo “DONE”#end of reading backup file
done
Screenshot running the second script with output
I hope these scripts help you to take backup !!
shell script that will copy files/directory to the source to destination. This script will allow copy multiple files and directories. The script will only work assuming you have configured ssh key-based authentication between two machines.
The script will request to input the destination hostname or IP address and path of the destination folder. On the new line, you need to provide filename or directory that needs to be copied. You will be only allowed to copy one file or directory per line. When hitting enter, you will be allowed to provide the next file/directory. Script will exit once it receives a blank field.
#!/bin/bash
#next line prints hearer of script
echo “Interactive Script to Copy File (files) / Directory using scp”
#next line check if entered value is not null, and if null it will reask user to enter Destination Server
while [ x$desthost = “x” ]; do
#next line prints what userd should enter, and stores entered value to variable with name desthost
read -p “Destination Server Name : “ desthost
#next line finishes while loop
done
#next line check if entered value is not null, and if null it will reask user to enter Destination Path
while [ x$destpath = “x” ]; do
#next line prints what userd should enter, and stores entered value to variable with name destpath
read -p “Destination Path : “ destpath
#next line finishes while loop
done
#next line put null value to variable filename
filename=’null’
#next line check if entered value is null, and If not null it will reask user to enter file(s) to copy
while ! [ x”$filename” = “x” ]; do
#next line prints what userd should enter, and stores entered value to variable with name filename
read -p “Path to source directory / file : “ filename
#next line checks if entered value is not null, and if not null it will copy file(s)
if ! [ x”$filename” = “x” ];
then
#next line prints header
echo -n “Copying $filename … “
#next like copy pre-entered file(s) or dir to destination path on destination server
scp -r “$filename” “$desthost”:”$destpath”
#end of if
fi
#next line finishes while loop
done
Running the scp script
[root@TestNode1 ~]# sh scpcopyscript.sh
Interactive Script to Copy File (files) / Directory using scp
Destination Server Name : 192.168.0.2
Destination Path : /tmp
Path to source directory / file : /root/backup.txt
backup.txt 100% 0 0.0KB/s 00:00
Path to source directory / file : /root/docsdir
file2.txt 100% 0 0.0KB/s 00:00
file3.txt 100% 0 0.0KB/s 00:00
file1.txt 100% 0 0.0KB/s 00:00
Path to source directory / file :
[root@TestNode1 ~]#
*******************************************************************
we will cover a few ways you can use to send emails with attachments from the command line interface on Linux. It can have quite a few uses, for example to send an archive with an application from an application server to email or you can use the commands in scripts to automate some process. For our examples,we will use the file foo.tar.gz as our attachment.
There are various ways to send emails from command line using different mail clients but here I am sharing few mail client utility used by most users like mailx, mutt and swaks.
All the tools we will present to you are very popular and present in the repositories of most Linux distributions, you can install them using the following commands:
For Debian / Ubuntu systems
apt-get install mutt
apt-get install swaks
apt-get install mailx
apt-get install sharutils
For Red Hat based systems like CentOS or Fedora
yum install mutt
yum install swaks
yum install mailx
yum install sharutils
1) Using mail / mailx
The mailx utility found as the default mailer application in most Linux distributions now includes the support to attach file. If it is not available you can easily install using the following commands please take note that this may not be supported in older versions, to check this you can use the command:
$ man mail
And the first line should look like this:
mailx [-BDdEFintv~] [-s subject] [-a attachment ] [-c cc-addr] [-b bcc-addr] [-r from-addr] [-h hops] [-A account] [-S variable[=value]] to-addr . . .
As you can see it supports the -a attribute to add a file to the email and -s attribute to subject to the email. Use a few of the below examples to send mails.
a) Simple Mail
Run the mail command, and then mailx would wait for you to enter the message of the email. You can hit enter for new lines. When done typing the message, press Ctrl+D, and mailx would display EOT.
After than mailx automatically delivers the email to the destination.
$ mail user@example.com
b) To send an email with the subject
$ echo “Email text” | mail -s “Test Subject” user@example.com
-s is used for defining subject for the email.
c) To send a message from a file
$ mail -s “message send from file” user@example.com < /path/to/file
d) To send message piped using the echo command
$ echo “This is message body” | mail -s “This is Subject” user@example.com
e) To send an email with an attachment
$ echo “Body with attachment “| mail -a foo.tar.gz -s “attached file” user@example.com
-a is used for attachments
2) mutt
Mutt is a text-based email client for Unix-like systems. It was developed over 20 years ago and it’s an important part of Linux history, one of the first clients to support scoring and threading capabilities. Use a few of the below examples to send emails.
a) Send an email with subject & body massage from a file
$ mutt -s “Testing from mutt” user@example.com < /tmp/message.txt
b) To send body message piped using the echo command
$ echo “This is the body” | mutt -s “Testing mutt” user@example.com
c) To send an email with an attachment
$ echo “This is the body” | mutt -s “Testing mutt” user@example.com -a /tmp/foo.tar.gz
d) To send an email with multiple attachments
$ echo “This is the body” | mutt -s “Testing” user@example.com -a foo.tar.gz -a bar.tar.gz
3) swaks
Swaks stands for Swiss Army Knife for SMTP and it is a featureful, flexible, scriptable, transaction-oriented SMTP test tool written and maintained by John Jetmore. You can use the following syntax to send an email with attachment:
$ swaks -t “foo@bar.com” -header “Subject: Subject” -body “Email Text” -attach foo.tar.gz
The important thing about Swaks is that it will also debug the full mail transaction for you, so it is a very useful tool if you also wish to debug the mail sending process:
As you can see it gives you full details about the sending process including what capabilities the receiving mail server supports, each step of the transaction between the 2 servers.
4) uuencode
Email transport systems were originally designed to transmit characters with a seven-bit encoding — like ASCII. This meant they could send messages with plain text but not “binary” text, such as program files or image files that used all of an eight-bit byte. The program is used to solve this limitation is “uuencode”( “UNIX to UNIX encoding”) which encode the mail from binary format to text format that is safe to transmit & program is used to decode the data is called “uudecode”
We can easily send binary text such as program files or image files using uuencode with mailx or mutt email client is shown by the following example:
$ uuencode example.jpeg example.jpeg | mail user@example.com
Shell Script: Explain how to send email
#!/bin/bash
# Function to check if entered file names are really files
function check_files()
{
output_files=””
for file in $1
do
if [ -s $file ] then
output_files=”${output_files}${file} “
fi
done
echo $output_files
}echo
echo
# Getting the Subject from user
echo -n -e “Enter e-mail subject:\n[Enter] “
read SUBJECTecho
if [ “$SUBJECT” == “” ] then
echo “Proceeding without the subject…”
fiecho
# Making sure file names are poiting to real files
attachments=$(check_files “$att”)
echo “Attachments: $attachments”echo
# Composing body of the message
echo “Enter message. To mark the end of message type ;; in new line.”
read lineSENDMAILCMD=”mutt -e \”set from=$FROM\” -s \”$SUBJECT\” \
$ATTACHMENTS — \”$TO\” <<< \”$BODY\””
echo $SENDMAILCMDmutt -e “set from=$FROM” -s “$SUBJECT” $ATTACHMENTS — $TO <<< $BODY
Script Output
Enter the e-mail address you wish to send mail from:
[Enter] test@gmail.comEnter the e-mail address you wish to send mail to:
[Enter] test@gmail.comEnter e-mail subject:
[Enter] Message subjectProvide the list of attachments. Separate names by space.
If there are spaces in file name, quote file name with “.
send_mail.shAttachments: send_mail.sh
Conclusion
There are many ways to send emails from command line/shell script but here we have shared 4 tools available for unix / linux based distros. I hope you enjoyed reading our article and please provide your valuable comments and also let us know if you know about any new tools.
*******************************************************************
we are introducing a shell script to perform Linux system health check. This script collects system information and status like hostname, kernel version, uptime, cpu / memory / disk usage. The script uses hostname, uptime, who, mpstat, lscpu, ps, top, df, free, bc commands to get system information and cut, grep, awk, and sed for text processing. The output of the script is a text file that will be generated in the current directory. A variable is set to provide an email address to which script can send a report file. Apart from system status, the script will check a predefined threshold for CPU load and filesystem size.
Remember: Make sure you have all the above commands working, to output all results correctly.
Understanding linuxhealthcheck.sh Script
#Print header, hostname (hostname command used), Kernel version (uname -r) , Uptime (from uptime command) and Last reboot time (from who command)
echo -e “
#!/bin/bashfunction sysstat {
echo -e “
####################################################################
Health Check Report (CPU,Process,Disk Usage, Memory)
####################################################################
#hostname command returns hostname
Hostname : `hostname`#uname command with key -r returns Kernel version
Kernel Version : `uname -r`#uptime command used to get uptime, and with sed command we cat process output to get only uptime.
Uptime : `uptime | sed ‘s/.*up \([^,]*\), .*/\1/’`#who command is used to get last reboot time, awk for processing output
Last Reboot Time : `who -b | awk ‘{print $3,$4}’`*********************************************************************
CPU Load → Threshold < 1 Normal > 1 Caution , > 2 Unhealthy
*********************************************************************
“
#here we check if mpstat command is in our system
MPSTAT=`which mpstat`#here we get exit code from previous command
MPSTAT=$?#if exit status in not 0, this means that mpstat command is not found (or not exist in our system)
if [ $MPSTAT != 0 ]then
echo “Please install mpstat!”
echo “On Debian based systems:”
echo “sudo apt-get install sysstat”
echo “On RHEL based systems:”
echo “yum install sysstat”
else
echo -e “”
#here we check in same way if lscpu installed
LSCPU=`which lscpu`LSCPU=$?
if [ $LSCPU != 0 ]
then
RESULT=$RESULT” lscpu required to procedure accurate results”
else
#if we have lscpu installed, we can get number of CPU’s on our system and get statistic for each using mpstat command.
cpus=`lscpu | grep -e “^CPU(s):” | cut -f2 -d: | awk ‘{print $1}’`i=0
#here we make loop to get and print CPU usage statistic for each CPU.
while [ $i -lt $cpus ]do
#here we get statistic for CPU and print it. Awk command help to do this, since output doesn’t allow this to do with grep. AWK check if third value is equal to variable $i (it changes from 0 to number of CPU), and print %usr value for this CPU
echo “CPU$i : `mpstat -P ALL | awk -v var=$i ‘{ if ($3 == var ) print $4 }’ `”#here we increment $i variable for loop
let i=$i+1done
fi
echo -e “
#here with uptime command we get load average for system, and cut command helps to process result.
Load Average : `uptime | awk -F’load average:’ ‘{ print $2 }’ | cut -f1 -d,`#same as before, but with awk command we check if system is Normal (if value less than 1, Caution (if between 1 and 2) and Unhealthy.
Heath Status : `uptime | awk -F’load average:’ ‘{ print $2 }’ | cut -f1 -d, | awk ‘{if ($1 > 2) print“Unhealthy”; else if ($1 > 1) print “Caution”; else print “Normal”}’`
“
fi
echo -e “
******************************************************************
Process
******************************************************************
Top memory using processs/application
PID %MEM RSS COMMAND
#with ps command we get list of processes, awk show only needed columns. After with sort command we sort it by third column and we need only top 10, that why we used head command
`ps aux | awk ‘{print $2, $4, $6, $11}’ | sort -k3rn | head -n 10`Top CPU using process/application
#with top command we get top CPU using processes, and with combination of head and tail we get top 10.
`top b -n1 | head -17 | tail -11`**********************************************************************
Disk Usage → Threshold < 90 Normal > 90% Caution > 95 Unhealthy
**********************************************************************
“
#we get disk usage with df command. -P key used to have postfix like output (there was problems with network shares, etc and -P resolve this problems). We print output to temp file to work with info more than one.
df -Pkh | grep -v ‘Filesystem’ > /tmp/df.status#We create loop to process line by line from df.status
while read DISKdo
#here we get line from df.status and print result formatted with awk command
LINE=`echo $DISK | awk ‘{print $1,”\t”,$6,”\t”,$5,” used”,”\t”,$4,” free space”}’`echo -e $LINE
echo
done < /tmp/df.status
echo -e “
Heath Status”
echo
#here almost same loop, but we check disk usage, and print Normal if value less 90, Caution if between 90 and 95, and Unhealthy if greater than 95)
while read DISKdo
USAGE=`echo $DISK | awk ‘{print $5}’ | cut -f1 -d%`
if [ $USAGE -ge 95 ]
then
STATUS=’Unhealthy’
elif [ $USAGE -ge 90 ]
then
STATUS=’Caution’
else
STATUS=’Normal’
fi
LINE=`echo $DISK | awk ‘{print $1,”\t”,$6}’`
#here we print result with status
echo -ne $LINE “\t\t” $STATUSecho
done < /tmp/df.status
#here we remove df.status file
rm /tmp/df.status#here we get Total Memory, Used Memory, Free Memory, Used Swap and Free Swap values and save them to variables.
TOTALMEM=`free -m | head -2 | tail -1| awk ‘{print $2}’`
#All variables like this is used to store values as float (we are using bc to make all mathematics operations, since without bc all values will be integer). Also we use if to add zero before value, if value less than 1024, and result of dividing will be less than 1.
TOTALBC=`echo “scale=2;if($TOTALMEM 0) print 0;$TOTALMEM/1024″| bc -l`
USEDMEM=`free -m | head -2 | tail -1| awk ‘{print $3}’`
USEDBC=`echo “scale=2;if($USEDMEM 0) print 0;$USEDMEM/1024″|bc -l`
FREEMEM=`free -m | head -2 | tail -1| awk ‘{print $4}’`
FREEBC=`echo “scale=2;if($FREEMEM 0) print 0;$FREEMEM/1024″|bc -l`echo -e “
********************************************************************
Memory
********************************************************************
Physical Memory
Total\tUsed\tFree\t%Free
# as we get values in GB, also we get % of usage dividing Free by Total
${TOTALBC}GB\t${USEDBC}GB \t${FREEBC}GB\t$(($FREEMEM * 100 / $TOTALMEM ))%Swap Memory
Total\tUsed\tFree\t%Free
#Same as above — values in GB, and in same way we get % of usage
${TOTALSBC}GB\t${USEDSBC}GB\t${FREESBC}GB\t$(($FREESWAP * 100 / $TOTALSWAP ))%
“}
#here we make filename value, using hostname, and date.
FILENAME=”health-`hostname`-`date +%y%m%d`-`date +%H%M`.txt”#here we run function and save result to generated filename
sysstat > $FILENAME#here we print output to user.
echo -e “Reported file $FILENAME generated in current directory.” $RESULT#here we check if user provide his email address to send email
if [ “$EMAIL” != “ ]then
then
echo “The program ‘mail’ is currently not installed.”
#if mailx installed, we send email with report to user
elsecat $FILENAME | mail -s “$FILENAME” $EMAIL
fi
fi
Don’t copy the script from above as it may not work, download linuxsystemhealth.sh script from here.
System health report
[root@linoxide script]# cat health-linoxide.com-140831–0909.txt | more#####################################################################
Health Check Report (CPU,Process,Disk Usage, Memory)
#####################################################################*********************************************************************
CPU Load → Threshold < 1 Normal > 1 Caution , > 2 Unhealthy
*********************************************************************CPU0 : 0.06
Load Average : 0.00
Heath Status : Normal
=> Top memory using processs/application
PID %MEM RSS COMMAND
1361 12.6 127896 /usr/lib/systemd/systemd-journald
1642 7.1 72252 /usr/sbin/rsyslogd
1644 1.9 20148 /usr/bin/python
2340 1.8 19092 /sbin/dhclient
1634 1.4 14748 /usr/sbin/NetworkManager
31410 0.8 8724 sshd:
31441 0.7 7888 sshd:
31432 0.7 7784 sshd:
2558 0.5 5988 /usr/sbin/sshd
1 0.5 5412 /sbin/init=> Top CPU using process/application
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 98572 5412 2796 S 0.0 0.5 0:11.82 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.51 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:03.33 kworker/u2:0
7 root 20 0 0 0 0 S 0.0 0.0 0:04.17 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs*********************************************************************
Disk Usage → Threshold < 90 Normal > 90% Caution > 95 Unhealthy
*********************************************************************/dev/root / 11% used 13G free space
devtmpfs /dev 0% used 495M free space
tmpfs /dev/shm 0% used 496M free space
tmpfs /run 1% used 495M free space
tmpfs /sys/fs/cgroup 0% used 496M free space
tmpfs /tmp 0% used 496M free space
Heath Status
=> Physical Memory
Total Used Free %Free
0.96GB 0.46GB 0.50GB 52%
=> Swap Memory
Total Used Free %Free
0.24GB 0GB 0.24GB 100%
Regular backup of mysql databases is also as important as backing up your code. So I wrote this script to backup all of my databases on the local disk. I then added gzip to compress the sql file to save disk space. You can always copy them over the network so as to retain multiple copies of your database.
Please replace the “mysqlpassword” mentioned in the script with your mysql root password. This script will create a file as dblist in /usr/local that will list all the databases in MySQL Server.
The DB's are backed up in /Backup/mysqlbackup directory with the directory name as the date of the day. The path of the dblist and backup destination can be modified by users.
#!/bin/bash
### Create Directory with Date where Database backup will be stored. ####
### List all the databases in /usr/local/dblist file. ####
mysql -u root -p’mysqlpassword’ -e ‘show databases’ >/usr/local/dblist
list=$(cat /usr/local/dblist)
echo $foldername### Create Backup Directory in /Backup/mysqlbackup … ####
mkdir -p /Backup/mysqlbackup/$foldernamefor i in $list
do
You can put this shell script in crontab and run every day. In this way, you will have daily backups of all your databases.
Sample Output
./mysql.sh
17Sep2013_backups
database1
database1.sql.gz file saved…
hello_db
hello_db.sql.gz file saved…
site2
site2.sql.gz file saved…
test
test.sql.gz file saved…
This shell script will create a dump of all databases from the MySQL server. Allow the system administrators to have backups of all databases. The output would be in a tar.gz zipped format. The script is using mysqldump and tar command for this purpose.
Shell Script
#!/bin/bash
#Check if user input hostname, if not — exit from script
if [ “$#” = “0” ]; thenecho “Usage: `basename $0` “
exit 2
fi
#We use some variables:
#we save hostname that user input to MYSQL variable
MYSQL=$1#Folder to keep backups
BACKUP_DIR=”/data”#we getting current timestamp, to use in filenames
NOW=`date +%Y%m%d-%H%M%S`#name of file will be HOSTNAME-CURRENT_TIMESTAMP
NAME=${MYSQL}-${NOW}#we use mysql access without password
PASSWORD=””#This will create folder, where we will keep all dumps
mkdir ${BACKUP_DIR}/${NAME}#Telling user that we start to process hostnname
echo “Processing ${MYSQL}…”#This will get list all of databases, we use mysql command to login to server, and with awk get only
#database name without description and database with name information_schema
DBS=`echo “show databases;” | mysql -host=${MYSQL} -user=root -password=${PASSWD} | \awk ‘{if ($1 != “Database” && $1 != “information_schema”) {print $1}}’`
#Now we will process every database, we will create dump for each
for DB in ${DBS}; do#telling user that we starting to dump each database
echo -n “ Dumping ${DB}… “#if database is not mysql we need to lock tables, to avoid problems when user during same time save
#something to db
if [ ${DB} != “mysql” ]; thenLOCKS=”-lock-tables”
else
LOCKS=””
fi
#command that will create dump
mysqldump -host=${MYSQL} -user=root -password=${PASSWD} -force \-allow-keywords -complete-insert -add-drop-table -add-locks -flush-logs \
${LOCKS} -result-file=${BACKUP_DIR}/${NAME}/${DB}-`date +%Y%m%d-%H%M%S`.sql ${DB} < /dev/null #telling user that process done for this database echo “Done” done echo #Now we will compress all dumps (stored in directory) to single file echo -n “ Compressing files… “ #with tar command we compress directory tar -czf ${BACKUP_DIR}/mysql-${NAME}.tar.gz ${BACKUP_DIR}/${NAME} >/dev/null 2>&1
echo Done
#removing uneeded directory
rm -rf ${BACKUP_DIR}/${NAME}echo “Dump of host ${MYSQL} finished”
Output
#./mysql_backup.sh localhost
Processing localhost…
Dumping mysql… Done
Dumping test… Done
Compressing files… Done
Dump of host localhost finished
#ls -la /data/
# to get list of files inside directory
total 152drwxrwxrwx 2 root root 4096 Mar 4 14:44 .
drwxr-xr-x 23 root root 4096 Mar 4 14:25 ..
-rw-r-r- 1 yevhen yevhen 139361 Mar 4 14:44 mysql-localhost-20130304–144450.tar.gz
Unpack the tar.gz file
server# tar -zxvf mysql-localhost-20130304–144450.tar.gz
data/localhost-20130304–144450/
data/localhost-20130304–144450/test-20130304–144450.sql
data/localhost-20130304–144450/mysql-20130304–144450.sql
Shell script helps the user to create a user in the Mysql database, when the user doesn’t know about MySQL, but need quickly add a new user. The script should run only from users that have passwordless access to MySQL.
Shell Script
#!/bin/bash
#Ask user to enter database name and save input to dbname variable
read -p “Please Enter Database Name:” dbname#checking if database exist
mysql -Bse “USE $dbname” 2> /dev/null#if database exist:
if [ $? -eq 0 ]; then#ask user about username
read -p “Please enter the username you wish to create : “ username#ask user about allowed hostname
read -p “Please Enter Host To Allow Access Eg: %,ip or hostname : “ host#ask user about password
read -p “Please Enter the Password for New User ($username) : “ password#mysql query that will create new user, grant privileges on database with entered password
query=”GRANT ALL PRIVILEGES ON $dbname.* TO $username@’$host’ IDENTIFIED BY ‘$password’”;#ask user to confirm all entered data
read -p “Executing Query : $query , Please Confirm (y/n) : “ confirm#if user confims then
if [ “$confirm” == ‘y’ ]; then#run query
mysql -e “$query”#update privileges, without this new user will be created but not active
mysql -e “flush privileges”else
#if user didn’t confirm entered data
read -p “Aborted, Press any key to continue..”#just exit
fielse
#If database not exit — warn user and exit
echo “The Database: $dbname does not exist, please specify a database that exists”;fi
RESULT
Server1:/# ./mysql_create_user.sh
Please Enter Database Name:test
Please enter the username you wish to create : test
Please Enter Host To Allow Access Eg: %,ip or hostname : localhost
Please Enter the Password for New User (test) : 12321q
Executing Query : GRANT ALL PRIVILEGES ON test.* TO test@’localhost’ IDENTIFIED BY ‘12321q’ , Please Confirm (y/n) : y
Testing
Server1:/# mysql -u test -p12321q
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 54
Server version: 5.1.66–0+squeeze1 (Debian)
Copyright © 2000, 2012, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mysql> Bye