Definition of Algorithm

The word Algorithm means” A set of finite rules or instructions to be followed in calculations or other
problem-solving operations”

A procedure for solving a mathematical problem in a finite number of steps that frequently involves
recursive operations.

Use of the Algorithms:

Algorithms play a crucial role in various fields and have many applications. Some of the key areas
where algorithms are used include:

Computer Science: Algorithms form the basis of computer programming and are used to solve
problems ranging from simple sorting and searching to complex tasks such as artificial intelligence
and machine learning.

Mathematics: Algorithms are used to solve mathematical problems, such as finding the optimal
solution to a system of linear equations or finding the shortest path in a graph.

Operations Research: Algorithms are used to optimize and make decisions in fields such as
transportation, logistics, and resource allocation.

Artificial Intelligence: Algorithms are the foundation of artificial intelligence and machine learning,
and are used to develop intelligent systems that can perform tasks such as image recognition, natural
language processing, and decision-making.

Data Science: Algorithms are used to analyze, process, and extract insights from large amounts of
data in fields such as marketing, finance, and healthcare.

What is the need for algorithms?

Algorithms are necessary for solving complex problems efficiently and effectively.

They help to automate processes and make them more reliable, faster, and easier to perform.

Algorithms also enable computers to perform tasks that would be difficult or impossible for humans
to do manually.

They are used in various fields such as mathematics, computer science, engineering, finance, and
many others to optimize processes, analyze data, make predictions, and provide solutions to problems.

What are the Characteristics of an Algorithm?

As one would not follow any written instructions to cook the recipe, but only the standard one.
Similarly, not all written instructions for programming are an algorithm. For some instructions to be
an algorithm, it must have the following characteristics:

Clear and Unambiguous: The algorithm should be unambiguous. Each of its steps should be clear in
all aspects and must lead to only one meaning.
Well-Defined Inputs: If an algorithm says to take inputs, it should be well-defined inputs. It may or
may not take input.

Well-Defined Outputs: The algorithm must clearly define what output will be yielded and it should
be well-defined as well. It should produce at least 1 output.

Finite-ness: The algorithm must be finite, i.e. it should terminate after a finite time.

Feasible: The algorithm must be simple, generic, and practical, such that it can be executed with the
available resources. It must not contain some future technology or anything.

Language Independent: The Algorithm designed must be language-independent, i.e. it must be just
plain instructions that can be implemented in any language, and yet the output will be the same, as
expected.

Input: An algorithm has zero or more inputs. Each that contains a fundamental operator must accept
zero or more inputs.

Output: An algorithm produces at least one output. Every instruction that contains a fundamental
operator must accept zero or more inputs.

Definiteness: All instructions in an algorithm must be unambiguous, precise, and easy to interpret. By
referring to any of the instructions in an algorithm one can clearly understand what is to be done.
Every fundamental operator in instruction must be defined without any ambiguity.

Finiteness: An algorithm must terminate after a finite number of steps in all test cases. Every
instruction which contains a fundamental operator must be terminated within a finite amount of time.
Infinite loops or recursive functions without base conditions do not possess finiteness.

Effectiveness: An algorithm must be developed by using very basic, simple, and feasible operations
so that one can trace it out by using just paper and pencil.

Properties of Algorithm:

It should terminate after a finite time.

It should produce at least one output.

It should take zero or more input.

It should be deterministic means giving the same output for the same input case.

Every step in the algorithm must be effective i.e. every step should do some work.

Analysis of Algorithm

In the analysis of the algorithm, it generally focused on CPU (time) usage, Memory usage, Disk
usage, and Network usage. All are important, but the most concern is about the CPU time. Be careful
to differentiate between:

Performance: How much time/memory/disk/etc. is used when a program is run. This depends on the
machine, compiler, etc. as well as the code we write.
Complexity: How do the resource requirements of a program or algorithm scale, i.e. what happens as
the size of the problem being solved by the code gets larger.

Note: Complexity affects performance but not vice-versa.

Algorithm Analysis:

Algorithm analysis is an important part of computational complexity theory, which provides

theoretical estimation for the required resources of an algorithm to solve a specific computational
problem. Analysis of algorithms is the determination of the amount of time and space resources
required to execute it.

Why Analysis of Algorithms is important?

To predict the behavior of an algorithm without implementing it on a specific computer.

It is much more convenient to have simple measures for the efficiency of an algorithm than to
implement the algorithm and test the efficiency every time a certain parameter in the underlying
computer system changes.

It is impossible to predict the exact behavior of an algorithm. There are too many influencing factors.

The analysis is thus only an approximation; it is not perfect.

More importantly, by analyzing different algorithms, we can compare them to determine the best one
for our purpose.

Types of Algorithm Analysis:

 Best case
 Worst case
 Average case

Best case: Define the input for which algorithm takes less time or minimum time. In the best case
calculate the lower bound of an algorithm. Example: In the linear search when search data is present
at the first location of large data then the best case occurs.

Worst Case: Define the input for which algorithm takes a long time or maximum time. In the worst
calculate the upper bound of an algorithm. Example: In the linear search when search data is not
present at all then the worst case occurs.

Average case: In the average case take all random inputs and calculate the computation time for all
inputs.

And then we divide it by the total number of inputs.

Average case = all random case time / total no of case

Time Complexity and Space Complexity

Generally, there is always more than one way to solve a problem in computer science with different
algorithms. Therefore, it is highly required to use a method to compare the solutions in order to judge
which one is more optimal. The method must be:

Independent of the machine and its configuration, on which the algorithm is running on.

Shows a direct correlation with the number of inputs.

Can distinguish two algorithms clearly without ambiguity.

There are two such methods used, time complexity and space complexity which are discussed below:

Time Complexity: The time complexity of an algorithm quantifies the amount of time taken by an
algorithm to run as a function of the length of the input. Note that the time to run is a function of the
length of the input and not the actual execution time of the machine on which the algorithm is running
on.

Definition– The valid algorithm takes a finite amount of time for execution. The time required by the
algorithm to solve given problem is called time complexity of the algorithm. Time complexity is very
useful measure in algorithm analysis.

It is the time needed for the completion of an algorithm. To estimate the time complexity, we need to
consider the cost of each fundamental instruction and the number of times the instruction is executed.

Example 1: Addition of two scalar variables

Algorithm ADD SCALAR(A, B)

//Description: Perform arithmetic addition of two numbers

//Input: Two scalar variables A and B

//Output: variable C, which holds the addition of A and B

C <- A + B

return C

The addition of two scalar numbers requires one addition operation. the time complexity of this
algorithm is constant, so T(n) = O(1) .

In order to calculate time complexity on an algorithm, it is assumed that a constant time c is taken to
execute one operation, and then the total operations for an input length on N are calculated. Consider
an example to understand the process of calculation: Suppose a problem is to find whether a pair (X,
Y) exists in an array, A of N elements whose sum is Z. The simplest idea is to consider every pair and
check if it satisfies the given condition or not.

The pseudo-code is as follows:

int a[n];

for(int i = 0;i < n;i++)

cin >> a[i]

for(int i = 0;i < n;i++)

for(int j = 0;j < n;j++)

if(i!=j && a[i]+a[j] == z)

return true

return false

Assuming that each of the operations in the computer takes approximately constant time, let it be c.
The number of lines of code executed actually depends on the value of Z. During analyses of the
algorithm, mostly the worst-case scenario is considered, i.e., when there is no pair of elements with
sum equals Z. In the worst case,

N*c operations are required for input.

The outer loop i loop runs N times.

For each i, the inner loop j loop runs N times.

So total execution time is N*c + N*N*c + c. Now ignore the lower order terms since the lower order
terms are relatively insignificant for large input, therefore only the highest order term is taken
(without constant) which is N*N in this case. Different notations are used to describe the limiting
behavior of a function, but since the worst case is taken so big-O notation will be used to represent the
time complexity.

Hence, the time complexity is O(N2) for the above algorithm. Note that the time complexity is solely
based on the number of elements in array A i.e the input length, so if the length of the array will
increase the time of execution will also increase.

Order of growth is how the time of execution depends on the length of the input. In the above
example, it is clearly evident that the time of execution quadratically depends on the length of the
array. Order of growth will help to compute the running time with ease.

Let’s calculate the time complexity of the below algorithm:

count = 0

for (int i = N; i > 0; i /= 2)

for (int j = 0; j < i; j++)

count++;
This is a tricky case. In the first look, it seems like the complexity is O(N * log N). N for the j′s loop
and log(N) for i′s loop. But it’s wrong. Let’s see why.

Think about how many times count++ will run.

When i = N, it will run N times.

When i = N / 2, it will run N / 2 times.

When i = N / 4, it will run N / 4 times.

And so on.

The total number of times count++ will run is N + N/2 + N/4+…+1= 2 * N. So the time complexity
will be O(N).

Space Complexity:

Definition –

Problem-solving using computer requires memory to hold temporary data or final result while the
program is in execution. The amount of memory required by the algorithm to solve given problem is
called space complexity of the algorithm.

The space complexity of an algorithm quantifies the amount of space taken by an algorithm to run as
a function of the length of the input. Consider an example: Suppose a problem to find the frequency of
array elements.

It is the amount of memory needed for the completion of an algorithm.

To estimate the memory requirement we need to focus on two parts:

(1) A fixed part: It is independent of the input size. It includes memory for instructions (code),
constants, variables, etc.

(2) A variable part: It is dependent on the input size. It includes memory for recursion
stack,referenced variables, etc.

Example : Addition of two scalar variables

Algorithm ADD SCALAR(A, B)

//Description: Perform arithmetic addition of two numbers

//Input: Two scalar variables A and B

//Output: variable C, which holds the addition of A and B

C <— A+B

return C

The addition of two scalar numbers requires one extra memory location to hold the result. Thus the
space complexity of this algorithm is constant, hence S(n) = O(1).

The pseudo-code is as follows:

int freq[n];

int a[n];

for(int i = 0; i<n; i++)

cin>>a[i];

freq[a[i]]++;

Here two arrays of length N, and variable i are used in the algorithm so, the total space used is N * c +
N * c + 1 * c = 2N * c + c, where c is a unit space taken. For many inputs, constant c is insignificant,
and it can be said that the space complexity is O(N).

There is also auxiliary space, which is different from space complexity. The main difference is where
space complexity quantifies the total space used by the algorithm, auxiliary space quantifies the extra
space that is used in the algorithm apart from the given input. In the above example, the auxiliary
space is the space used by the freq[] array because that is not part of the given input. So total auxiliary
space is N * c + c which is O(N) only.

Types of Asymptotic Notations

The main idea of asymptotic analysis is to have a measure of the efficiency of algorithms that don’t
depend on machine-specific constants and don’t require algorithms to be implemented and time taken
by programs to be compared. Asymptotic notations are mathematical tools to represent the time
complexity of algorithms for asymptotic analysis.

Asymptotic Notations:

.Asymptotic Notations are mathematical tools that allow you to analyze an algorithm’s running time
by identifying its behavior as its input size grows.

.This is also referred to as an algorithm’s growth rate.

You can’t compare two algorithm’s head to head.

.You compare space and time complexity using asymptotic analysis.

.It compares two algorithms based on changes in their performance as the input size is increased or
decreased.

There are mainly three asymptotic notations:

Big-O Notation (O-notation)

Omega Notation (Ω-notation)

Theta Notation (Θ-notation)

1. Theta Notation (Θ-Notation):

Theta notation encloses the function from above and below. Since it represents the upper and the
lower bound of the running time of an algorithm, it is used for analyzing the average-case complexity
of an algorithm.

.Theta (Average Case) You add the running times for each possible input combination and take the
average in the average case.
Let g and f be the function from the set of natural numbers to itself. The function f is said to be Θ(g),
if there are constants c1, c2 > 0 and a natural number n0 such that c1* g(n) ≤ f(n) ≤ c2 * g(n) for all n
≥ n0

Mathematical Representation of Theta notation:

Θ (g(n)) = {f(n): there exist positive constants c1, c2 and n0 such that 0 ≤ c1 * g(n) ≤ f(n) ≤ c2 * g(n)
for all n ≥ n0}

Note: Θ(g) is a set

The above expression can be described as if f(n) is theta of g(n), then the value f(n) is always between
c1 * g(n) and c2 * g(n) for large values of n (n ≥ n0). The definition of theta also requires that f(n)
must be non-negative for values of n greater than n0.

The execution time serves as both a lower and upper bound on the algorithm’s time complexity.

It exist as both, most, and least boundaries for a given input value.

A simple way to get the Theta notation of an expression is to drop low-order terms and ignore leading
constants. For example, Consider the expression 3n3 + 6n2 + 6000 = Θ(n3), the dropping lower order
terms is always fine because there will always be a number(n) after which Θ(n3) has higher values
than Θ(n2) irrespective of the constants involved. For a given function g(n), we denote Θ(g(n)) is
following set of functions.
Examples :

{ 100 , log (2000) , 10^4 } belongs to Θ(1)

{ (n/4) , (2n+3) , (n/100 + log(n)) } belongs to Θ(n)

{ (n^2+n) , (2n^2) , (n^2+log(n))} belongs to Θ( n2)

Note: Θ provides exact bounds.

2. Big-O Notation (O-notation):

Big-O notation represents the upper bound of the running time of an algorithm. Therefore, it gives the
worst-case complexity of an algorithm.

.It is the most widely used notation for Asymptotic analysis.

.It specifies the upper bound of a function.

.The maximum time required by an algorithm or the worst-case time complexity.

.It returns the highest possible output value(big-O) for a given input.

.Big-Oh(Worst Case) It is defined as the condition that allows an algorithm to complete statement
execution in the longest amount of time possible.

If f(n) describes the running time of an algorithm, f(n) is O(g(n)) if there exist a positive constant C
and n0 such that, 0 ≤ f(n) ≤ cg(n) for all n ≥ n0

It returns the highest possible output value (big-O)for a given input.

The execution time serves as an upper bound on the algorithm’s time complexity.
Mathematical Representation of Big-O Notation:

O(g(n)) = { f(n): there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0 }

For example, Consider the case of Insertion Sort. It takes linear time in the best case and quadratic
time in the worst case. We can safely say that the time complexity of the Insertion sort is O(n2).

Note: O(n2) also covers linear time.

If we use Θ notation to represent the time complexity of Insertion sort, we have to use two statements
for best and worst cases:

The worst-case time complexity of Insertion Sort is Θ(n2).

The best case time complexity of Insertion Sort is Θ(n).

The Big-O notation is useful when we only have an upper bound on the time complexity of an
algorithm. Many times we easily find an upper bound by simply looking at the algorithm.

Examples :

{ 100 , log (2000) , 10^4 } belongs to O(1)

U { (n/4) , (2n+3) , (n/100 + log(n)) } belongs to O(n)

U { (n^2+n) , (2n^2) , (n^2+log(n))} belongs to O( n^2)

Note: Here, U represents union, we can write it in these manner because O provides exact or upper
bounds .

3. Omega Notation (Ω-Notation):

Omega notation represents the lower bound of the running time of an algorithm. Thus, it provides the
best case complexity of an algorithm.

The execution time serves as a lower bound on the algorithm’s time complexity.

It is defined as the condition that allows an algorithm to complete statement execution in the shortest
amount of time.

Let g and f be the function from the set of natural numbers to itself. The function f is said to be Ω(g),
if there is a constant c > 0 and a natural number n0 such that c*g(n) ≤ f(n) for all n ≥ n0

Mathematical Representation of Omega notation :

Ω(g(n)) = { f(n): there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n0 }

Let us consider the same Insertion sort example here. The time complexity of Insertion Sort can be
written as Ω(n), but it is not very useful information about insertion sort, as we are generally
interested in worst-case and sometimes in the average case.
Examples :

{ (n^2+n) , (2n^2) , (n^2+log(n))} belongs to Ω( n^2)

U { (n/4) , (2n+3) , (n/100 + log(n)) } belongs to Ω(n)

U { 100 , log (2000) , 10^4 } belongs to Ω(1)

Note: Here, U represents union, we can write it in these manner because Ω provides exact or lower
bounds.

Properties of Asymptotic Notations:

1. General Properties:

If f(n) is O(g(n)) then a*f(n) is also O(g(n)), where a is a constant.

Example:

f(n) = 2n²+5 is O(n²)

then, 7*f(n) = 7(2n²+5) = 14n²+35 is also O(n²).

Similarly, this property satisfies both Θ and Ω notation.

We can say,

If f(n) is Θ(g(n)) then a*f(n) is also Θ(g(n)), where a is a constant.

If f(n) is Ω (g(n)) then a*f(n) is also Ω (g(n)), where a is a constant.

2. Transitive Properties:

If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) = O(h(n)).

Example:

If f(n) = n, g(n) = n² and h(n)=n³

n is O(n²) and n² is O(n³) then, n is O(n³)

Similarly, this property satisfies both Θ and Ω notation.

We can say,

If f(n) is Θ(g(n)) and g(n) is Θ(h(n)) then f(n) = Θ(h(n)) .

If f(n) is Ω (g(n)) and g(n) is Ω (h(n)) then f(n) = Ω (h(n))

3. Reflexive Properties:

Reflexive properties are always easy to understand after transitive.

If f(n) is given then f(n) is O(f(n)). Since MAXIMUM VALUE OF f(n) will be f(n) ITSELF!

Hence x = f(n) and y = O(f(n) tie themselves in reflexive relation always.

Example:

f(n) = n² ; O(n²) i.e O(f(n))

Similarly, this property satisfies both Θ and Ω notation.

We can say that,

If f(n) is given then f(n) is Θ(f(n)).

If f(n) is given then f(n) is Ω (f(n)).

4. Symmetric Properties:
If f(n) is Θ(g(n)) then g(n) is Θ(f(n)).

Example:

If(n) = n² and g(n) = n²

then, f(n) = Θ(n²) and g(n) = Θ(n²)

This property only satisfies for Θ notation.

5. Transpose Symmetric Properties:

If f(n) is O(g(n)) then g(n) is Ω (f(n)).

Example:

If(n) = n , g(n) = n²

then n is O(n²) and n² is Ω (n)

This property only satisfies O and Ω notations.

6. Some More Properties:

1. If f(n) = O(g(n)) and f(n) = Ω(g(n)) then f(n) = Θ(g(n))

2. If f(n) = O(g(n)) and d(n)=O(e(n)) then f(n) + d(n) = O( max( g(n), e(n) ))

Example:

f(n) = n i.e O(n)

d(n) = n² i.e O(n²)

then f(n) + d(n) = n + n² i.e O(n²)

3. If f(n)=O(g(n)) and d(n)=O(e(n)) then f(n) * d(n) = O( g(n) * e(n))

Example:

f(n) = n i.e O(n)

d(n) = n² i.e O(n²)

then f(n) * d(n) = n * n² = n³ i.e O(n³)

_______________________________________________________________________________

Note: If f(n) = O(g(n)) then g(n) = Ω(f(n))