Shell Scripting Deep Dive
Shell Scripting Deep Dive
Watch on
What is Shell Scripting?
Shell Script
A shell script is a computer program written in the
shell programming language that consists of a
series of commands to be executed by the shell
@Sandip Das
Why Use Shell Scripting?
Reasons to use shell scripting:
Automation of repetitive tasks:
Shell scripts can be used to automate repetitive or complex tasks, making it easier to manage and maintain systems.
System administration
Shell scripts can be used to perform various system administration tasks, such as setting up a new user, managing disk space, or monitoring system
performance.
Improved efficiency
Shell scripts can be written to run faster than manual methods, providing improved efficiency and reducing the amount of time spent on repetitive
tasks.
Portability
Shell scripts can be run on various operating systems, including Linux, Unix, and macOS, making it a portable scripting language.
Integration with other tools:
Shell scripts can be integrated with other tools and applications, providing the ability to create complex workflows and automations.
Cost-effective
Shell scripting is a cost-effective method of automation, as it does not require specialized software or hardware to run.
Improved accuracy
By automating tasks, shell scripts can help reduce the chance of human error, leading to improved accuracy and reliability.
@Sandip Das
Years of Evolution & Types of Shell Scripting
Unix operating
1971: The Unix operating system was introduced, which included the first shell,
called the Thompson shell.
system introduced
The Bourne shell
1979: The Bourne shell, sh, was introduced as the standard shell for Unix, and 1971
became widely adopted as the default shell for Unix-based operating systems.
1979
1987: The C shell, csh, was introduced as an alternative to the Bourne shell, and
was designed to have a more C-like syntax. The C shell
1987
1989: The Korn shell, ksh, was introduced, which combined features of the
Bourne and C shells and added new features such as arrays and job control.
The Korn shell
1991: The bash shell, short for Bourne-Again shell, was introduced as a free
1989
software replacement for the original Bourne shell. It became the default shell for
many Linux distributions and is still widely used today. The bash shell
1995: The Z shell, zsh, was introduced, which added new features such as
advanced autocompletion, spelling correction, and customizable prompts. 1991
In the years since, shell scripting has continued to evolve and improve, with the
The Z shell
introduction of new shells and the addition of new features and capabilities.
Today, shell scripting is an essential tool for system administrators, developers,
and data scientists, and is used in a wide range of applications and industries.
1995
@Sandip Das
Getting
Started
Shell Scripting Basics
@Sandip Das
A shell script is a plain text file that contains a series of shell commands, functions, and
logic that can be executed in a shell environment. Here's the basic syntax and structure
of a shell script:
The first line of the script, #!/bin/bash, specifies the shell interpreter to be used for
executing the script. The rest of the script consists of comments, preceded by the #
symbol, and commands to be executed by the shell.
Each command in the script is executed in sequence, one after the other, until the end of
the script is reached. A command can be a built-in shell command, a command from a
Unix-based system, or a custom function or script.
In addition to commands, shell scripts can also include logic, such as conditional
statements and loops, to control the flow of execution and make decisions based on the
results of commands.
This script takes a single argument, checks whether it is greater than 0, and outputs a
message accordingly.
Shell scripts can also be used to automate tasks, such as setting up a development
environment, deploying an application, or processing data. To make a shell script
executable, it must have the executable permission set using the chmod command. Once
the script is executable, it can be run by calling the script file in the terminal or by using
the ./ operator.
Let's learn a bit about bash Scripting
Bash (AKA Bourne Again Shell) is a type of interpreter that processes shell commands. A shell interpreter takes commands
in plain text format and calls Operating System services to do something. For example, the ls command lists the files and
folders in a directory.
A shell scripting is writing a program for the shell to execute and a shell script is a file or program that the shell will execute.
Bash (bash) is one of many available (yet the most commonly used) Unix shells. and is a replacement/improvement of the
original Bourne shell (sh).
Shell scripting is scripting in any shell, whereas Bash scripting is scripting specifically for Bash. In practice, however, "shell
script" and "bash script" are often used interchangeably, unless the shell in question is not Bash.
@Sandip Das
Basic Liux
Commands
Basic Linux Commands Part 1
We need to know basic Linux/Unix Commands before jumping into shell scripting
cp -p source destination - Will copy the file from source to
pwd - Get the full path of the current working ls -l - List the files and directories in the current directory in long destination. -p stands for preservation. It preserves the
directory. (table) format (It is recommended to use -l with ls for better original attributes of file while copying like file owner,
timestamp, group, permissions etc.
cd - Navigate to the last directory you were readability).
working in. cp -R source_directory destination_directory - Will copy
ls -ld dir-name - List information about the directory dir-name source directory to specified destination recursively.
cd ~ or just cd Navigate to the current user's
home directory. instead of its contents. mv file1 file2 - In Linux there is no rename command as
cd .. Go to the parent directory of current such. Hence mv moves/renames the file1 to file2
directory ls -a - List all the files including the hidden ones (File names
rm -i filename - Asks you before every file removal for
(mind the space between cd and ..) starting with a . are hidden files in Linux). confirmation. IF YOU ARE A NEW USER TO LINUX
echo: Print text to the terminal. COMMAND LINE, YOU SHOULD ALWAYS USE rm -i. You
can specify multiple files.
cat: Concatenate files and print them to the ls -F - Appends a symbol at the end of a file name to indicate its
terminal. type (* means executable, / means directory, @ means symbolic rm -R dir-name - Will remove the directory dir-name
grep: Search for a pattern in a file or output. link, = means socket, | means named pipe, > means door) recursively.
sort: Sort the lines of a file or output. rm -rf dir-name - Will remove the directory dir recursively,
wc: Count the number of lines, words, and ls -lt - List the files sorted by last modified time with most ignoring non-existent files and will never prompt for
characters in a file or output. anything. BE CAREFUL USING THIS COMMAND! You can
recently modified files showing at the top (remember -l option specify multiple directorie
head: Print the first few lines of a file or provides the long format which has better readability)
output. rmdir dir-name - Will remove the directory dir-name, if it's
tail: Print the last few lines of a file or output. empty. This command can only remove empty directories.
ls -lh - List File Sizes in Human Readable Format
find: Search for files and directories based on mkdir dir-name - Create a directory dir-name.
various criteria.
ls -lR - Shows all subdirectories recursively.
mkdir -p dir-name/dir-name - Create a directory hierarchy.
Create parent directories as needed, if they don't exist.
tree - Will generate a tree representation of the file system You can specify multiple directories.
starting from the current directory. touch filename - Create a file filename, if it doesn't exist,
@Sandip Das otherwise change the timestamp of the file to current time
Basic Linux Commands Part 2 @Sandip Das
chmod go=+r myfile - Add read permission for the owner and the group.
chmod a +rwx myfile - Allow all users to read, write or execute myfile.
In more simple way:
chmod go -r myfile - Remove read permission from the group and others. 0 = 0 = nothing
1 = 1 = execute
chown owner1 filename - Change ownership of a file to user owner1. 2=2= write
3 = 2 + 1 = w+x
chgrp grp_owner filename - Change primary group ownership of file filename to 4 = 4 = read
group grp_owner.
5 = 4+1 = r+x
chgrp -R grp_owner dir-name - Change primary group ownership of directory dir-
6 = 4 + 2 = r+w
name to group grp_owner 7 = 4 + 2 + 1 = r+w+x
recursively. To change group ownership of a directory and everything within that
directory, use this command.
@Sandip Das
echo
Here's a simple example of using the echo command:
#!/bin/bash
The echo command is a common and simple way to provide output in shell scripts. It can be
used to display messages, status updates, and the values of variables, among other things.
The output can be redirected to a file or piped to another command for further processing.
@Sandip Das
read
Here's a simple example of using the read command:
#!/bin/bash
The read command is useful for allowing the user to provide input to the script, which can
be used to customize the behavior of the script or to gather information from the user. The
input can be further processed and used in various ways, such as making decisions based
on the input, or storing the input in variables for later use.
let's learn
common
operations
in Shell
Scripting
Variables
Variables in shell scripting are used to store values and can be referenced later in the
script. In shell scripting, variables are defined by assigning a value to a name. Here's an
example
In this example, a variable name is defined and assigned the value "John Doe". The value
of the variable is referenced by prefixing the variable name with the $ symbol.
There are two types of variables in shell scripting:
1. Environment variables: These are predefined variables that are set by the operating
system or by the shell environment, and are used to store information such as the
current user, the home directory, and the current working directory.
#!/bin/bash
2. User-defined variables: These are variables that are created by the user to store NAME="John"
specific values, such as the name of a file, a user name, or a password. echo $NAME
echo "$NAME"
Variables in shell scripting are case-sensitive and can contain letters, numbers, and
echo "${NAME}!"
underscores. They should not contain spaces, and the name should start with a letter or
an underscore.
When defining a variable, it is recommended to use uppercase letters for environment
variables and lowercase letters for user-defined variables to help distinguish between
them.
It is also recommended to use quotes around the value when assigning a value to a
variable, especially when the value contains spaces or special characters, to ensure that
the value is interpreted as a single string.
@Sandip Das
Basics Substrings Manipulation
String
${FOO:0:3} - Substring (position, length) STR="HELLO WORLD!"
name="John" ${FOO:(-3):3} - Substring from the right echo ${STR,}
echo ${name} #=> "hELLO WORLD!" (lowercase 1st
echo ${name/J/j} letter)
#=> "john" (substitution) echo ${STR,,}
echo ${name:0:2} Concatenating strings #=> "hello world!" (all lowercase)
#=> "Jo" (slicing)
echo ${name::2} string1="Hello" STR="hello world!"
#=> "Jo" (slicing) string2="World!" echo ${STR^}
In shell scripting, strings are echo ${name::-1} concatenated_string=$string1+$string2 #=> "Hello world!" (uppercase 1st letter)
#=> "Joh" (slicing) echo $concatenated_string echo ${STR^^}
sequences of characters that are echo ${name:(-1)} #=> "HELLO WORLD!" (all uppercase)
#=> "n" (slicing from right)
used to represent text. Strings can be echo ${name:(-2):1} Replace strings Length
#=> "h" (slicing from right)
assigned to variables, passed as echo ${food:-Cake} ${string/pattern/replacement} ${#FOO} - Length of $FOO
#=> $food or "Cake" e.g. e.g.
arguments to commands, and used in length=2 name="John Doe" string="Hello, World!"
echo ${name:0:length} #=> "Jo" echo "Replaced: ${name/Doe/Smith}" length=${#string}
various ways throughout a script. echo $length
Match and extract
Regular expressions can be used to match and extract substrings from a string using the grep or awk command. For example:
Split strings
Strings can be split into an array of substrings using the IFS (Internal Field Separator) variable and the read command. For
example:
string="John Doe, 30, Male"
string="John Doe, 30, Male"
IFS=", "
IFS=","
read name age gender <<< "$string"
read -a array <<< "$string"
echo "Name: $name"
echo "Name: ${array[0]}"
echo "Age: $age"
echo "Age: ${array[1]}"
echo "Gender: $gender"
echo "Gender: ${array[2]}"
Join strings
An array of strings can be joined into a single string using the IFS variable and the printf command. For example
Backquotes (``): This form of command substitution uses backquotes (`) to enclose the
command that you want to substitute. The output of the command will be the result of the
#!/bin/bash
result=`ls -l`
echo "Result: $result"
Command substitution is a feature This will run the ls -l command and capture its output, which will then be stored in the result
in shell scripting that allows the variable.
output of a command to be Dollar Sign and Parentheses ($()): This form of command substitution is similar to
substituted in place of the backquotes, but
uses a dollar sign and parentheses instead. The syntax is $(command). For example:
command itself. In other words, it
allows you to capture the output of #!/bin/bash
result=$(ls -l)
a command and use it in your script echo "Result: $result"
as a string value.
This will run the ls -l command and capture its output, which will then be stored in the
result variable.
Both forms of command substitution are widely used in shell scripts to capture the output of
commands and use it for further processing or as input to other commands.
@Sandip Das
Pipe |
A pipe in shell scripting is a feature
that allows you to connect two or
more commands together and pass
the output of one command as the
input to another command. It is
represented by the | symbol.
In this example, the ls command is used to list all the files in the current directory. The output
of the ls command is then piped to the grep command, which filters the list of files and only
displays files that have a ".txt" extension.
The result of this script will be a list of all the ".txt" files in the current directory, displayed on
the terminal.
This is a simple example, but it demonstrates how pipes can be used to connect commands
together and pass the output of one command as the input to another command, making it
easy to process and filter data in multiple stages.
@Sandip Das
Variable Conditions Basic Example More Example
[[ -z STRING ]] - Empty string
Conditional
[[ -n STRING ]] - Not empty string If statement: # String
[[ STRING == STRING ]] - Equal if [ -e file.txt ]; then if [[ -z "$string" ]]; then
[[ STRING != STRING ]] - Not Equal echo "File exists" echo "String is empty"
[[ NUM -eq NUM ]] - Equal
statements
fi elif [[ -n "$string" ]]; then
[[ NUM -ne NUM ]] - Not equal echo "String is not empty"
[[ NUM -lt NUM ]] - Less than else
[[ NUM -le NUM ]] - Less than or equal if-else statement: echo "This never happens"
[[ NUM -gt NUM ]] - Greater than if [ -e file.txt ]; then fi
[[ NUM -ge NUM ]] - Greater than or echo "File exists"
equal else # Combinations
Conditional statements are a [[ STRING =~ STRING ]] - Regexp echo "File does not exist" if [[ X && Y ]]; then
fundamental part of shell scripting (( NUM < NUM )) - Numeric conditions fi ...
[[ -o noclobber ]] - If OPTIONNAME is fi
and allow you to control the flow of enabled if-elif-else statement:
your scripts based on the outcome [[ ! EXPR ]] - Not if [ -f file.txt ]; then # Equal
[[ X && Y ]] - And echo "File is a regular file" if [[ "$A" == "$B" ]]
of certain conditions. The most [[ X || Y ]] - Or elif [ -d file.txt ]; then
commonly used conditional echo "File is a directory" # Regex
File Conditions else if [[ "A" =~ . ]]
statements in shell scripting are if, [[ -e FILE ]] - Exists echo "File does not exist"
if-else, and if-elif-else. [[ -r FILE ]] - Readable fi if (( $a < $b )); then
[[ -h FILE ]] - Symlink echo "$a is smaller than $b"
[[ -d FILE ]] - Directory fi
[[ -w FILE ]] - Writable
[[ -s FILE ]] - Size is > 0 bytes if [[ -e "file.txt" ]]; then
[[ -f FILE ]] - File echo "file exists"
[[ -x FILE ]] - Executable fi
[[ FILE1 -nt FILE2 ]] - 1 is more recent
than 2
[[ FILE1 -ot FILE2 ]] - 2 is more recent
than 1
[[ FILE1 -ef FILE2 ]] - Same files
@Sandip Das
Loop
For Loops
Basic For Loop C-like for loop Ranges
#!/bin/bash #!/bin/bash #!/bin/bash
for ((i = 0 ; i < 100 ; i++)); do for i in {1..5}; do
Loops are a key aspect of shell # Loop through a list of numbers echo $i echo "Welcome $i"
for i in 1 2 3 4 5; do done done
scripting and allow you to repeat a echo $i
set of commands multiple times done With step size
based on certain conditions. The for i in {5..50..5}; do
echo "Welcome $i"
two most commonly used loops in done
shell scripting are for loops and While Loops
while loops.
Basic While Loop Read Lines
#!/bin/bash #!/bin/bash
Both for loops and while loops are important tools for automating repetitive tasks and processing data in shell scripting.
They can be combined with other scripting constructs such as conditional statements and variables to create powerful
scripts.
@Sandip Das
Functions
Defining & Calling functions basic Example:
#!/bin/bash
Defining & Calling functions
myfunc() { # Define a function that takes two arguments
echo "hello $1" function add {
} result=$(($1 + $2))
echo $result
Functions are a way to organize # Same as above (alternate syntax) }
and reuse code in shell scripting. A
function myfunc() { # Call the function and store the result
function is a self-contained block of echo "hello $1" result=$(add 10 20)
code that can be called from other }
# Display the result
parts of your script. Functions can myfunc "John" echo "The result is $result"
take arguments and return values,
which allows you to pass data to Returning values
myfunc() {
and from the function. local myresult='some value' Arguments
echo $myresult $# -> Number of arguments
} $* -> All positional arguments (as a single word)
$@ -> All positional arguments (as separate
result="$(myfunc)" strings)
$1 -> First argument
$_ -> Last argument of the previous command
Raising Condition based errors
myfunc() { Note: $@ and $* must be quoted in order to
return 1 perform as described. Otherwise, they do exactly
} the same thing (arguments as separate strings).
if myfunc; then See Special parameters.
echo "success"
else
echo "failure"
@Sandip Das
fi
Array
Defining arrays
Fruits=('Apple' 'Banana' 'Orange')
Fruits[0]="Apple"
Fruits[1]="Banana"
Fruits[2]="Orange"
Operations
Fruits=("${Fruits[@]}" "Watermelon") # Push
Fruits+=('Watermelon') # Also Push
Fruits=( ${Fruits[@]/Ap*/} ) # Remove by regex match
unset Fruits[2] # Remove one item
Fruits=("${Fruits[@]}") # Duplicate
Fruits=("${Fruits[@]}" "${Veggies[@]}") # Concatenate
lines=(`cat "logfile"`) # Read from file
Iteration
for i in "${arrayName[@]}"; do
echo $i
done
@Sandip Das
Dictionaries Defining arrays Working with dictionaries
#!/bin/bash #!/bin/bash
@Sandip Das
Debugging
Use set -x
The set -x option is a shell option that can be used to enable
debugging in shell scripts. When this option is set, each command
that is executed will be displayed on the screen, along with its
arguments and its exit status. This can help to identify which
commands are causing errors and what their outputs are.
e.g.
#!/bin/bash
Debugging shell scripts can # Enable debugging
set -x
be challenging, especially # A simple script that adds two numbers
num1=10
when the scripts are num2=20
complex or have multiple result=$((num1 + num2))
echo "The result is $result"
components. However, there Use echo statements: The echo command can be used to display Use trap statements: The trap command can be used to catch
are several techniques that the values of variables and other information during the execution signals or events during the execution of a shell script. For
of a shell script. This can help to track the flow of the script and example, a trap statement can be used to catch a SIGINT signal,
can be used to help find and identify where errors are occurring which is generated when the user presses Ctrl + C. The trap
e.g. statement can then be used to display information about the state
fix errors in shell scripts of the script, or to perform cleanup actions before the script
#!/bin/bash terminates.
e.g.
# A simple script that calculates the factorial of a number
#!/bin/bash
read -p "Enter a number: " num
# A script that demonstrates the use of the trap statement
fact=1
function cleanup() {
for((i=1;i<=num;i++)) echo "Cleaning up before exiting"
do }
fact=$((fact * i))
echo "Iteration $i: Factorial is $fact" trap cleanup INT
done
echo "Running the script"
echo "The factorial of $num is $fact"
# Simulate an error
false
Use Debugger, check logs: Modern shell scripting come with
debuggers, and also model IDE also comes with smart debugging echo "Script finished"
tools, use them e.g. Shell Syntax plugin for VS Code
@Sandip Das
Integrating Shell Scripts with Other Tools
Cron Jobs: You can use cron to schedule the execution of shell scripts at specific times or intervals, making it easy to automate tasks such
as daily backups or weekly system maintenance.
Database Management Systems: Shell scripts can be used to interact with databases such as MySQL, PostgreSQL, and Oracle,
automating tasks such as data import, export, and backup.
Web Applications: Shell scripts can be integrated with web applications, such as WordPress, to automate tasks such as backups,
updates, and migrations.
Monitoring and Logging Tools: Shell scripts can be integrated with monitoring and logging tools, such as Nagios and Logrotate, to
automate log management and improve system monitoring.
Version Control Systems: Shell scripts can be integrated with version control systems, such as Git and SVN, to automate tasks such as
branching, merging, and deploying code changes.
@Sandip Das
Back-up Script Here's a simple example of a shell script that backs up a directory to a specified location:
#!/bin/bash
Backup and recovery is a common
use case for shell scripting. Shell # Define the source and destination directories
src_dir='/path/to/src'
scripts can be used to automate the dst_dir='/path/to/dst'
process of backing up important data,
such as files, databases, and system # Copy the source directory to the destination
configurations, and storing them in a rsync -av $src_dir $dst_dir
safe location. The scripts can also be # Confirm that the backup was successful
used to schedule regular backups, so if [ $? -eq 0 ]; then
that the data can be recovered in the echo "Backup successful."
event of a failure. else
echo "Backup failed."
fi
In this example, the rsync command is used to copy the contents of the src_dir to the dst_dir. The script then
checks the exit status of the rsync command to confirm that the backup was successful. If the exit status is 0,
the backup was successful, and the script outputs a message to that effect. If the exit status is not 0, the
backup failed, and the script outputs an error message.
This is just a simple example, but shell scripts can be used to implement more sophisticated backup and
recovery strategies, such as incremental backups, compression, encryption, and remote backups to cloud
storage.
@Sandip Das
System Here's a simple example of a shell script that checks disk space and sends an email notification if the disk usage
exceeds a specified threshold:
Maintenance #!/bin/bash
In this example, the script uses the df command to check the disk usage and the awk command to extract the
usage percentage. The script then checks if the usage percentage is greater than or equal to the threshold and
sends an email notification if that's the case.
This is just a simple example, but shell scripts can be used to automate a wide range of system maintenance
tasks, such as checking logs for errors, updating software, rotating backups, and much more. The advantage of
using shell scripts is that they can be automated and scheduled, so the tasks can be performed without any
manual intervention.
@Sandip Das
Challenge Details:
2) Ask the user to input two numbers using the "read" command
Build a Simple 3) Then ask the user to choose between operations e.g.
echo "Enter Choice:"
echo "1. Addition"
Calculator
echo "2. Subtraction"
echo "3. Multiplication"
echo "4. Division"
read ch
4) Based on the number input, do match condition and call the function
Note: Create a GitHub repo and push the code there , then tag me in a Linkedin post and/or Twitter post or in
my youtube channel's this session video, I will take a look as many as possible 👍
@Sandip Das
Thanks for
attending the
session :)
Follow on:
Sandip Das
@LearnTechWithSandip