Numerical methods to find solutions of equations

Contents
Introduction .................................................................................................................................................. 1
Norms: Vectors and matrices.................................................................................................................... 1
Code in Python ...................................................................................................................................... 2
Symbolic calculus ...................................................................................................................................... 3
Numerical methods for non-linear equations in R...................................................................................... 4
Bisection method ...................................................................................................................................... 4
Methodology......................................................................................................................................... 4
Python code .......................................................................................................................................... 4
Contraction method.................................................................................................................................. 5
Methodology......................................................................................................................................... 5
Python code .......................................................................................................................................... 6
Other Methods: Newton, secant, chord ................................................................................................... 6
Methodology......................................................................................................................................... 6
Python code .......................................................................................................................................... 7
Numerical methods for non-linear equations in 𝑹𝒏 .................................................................................. 8
Contraction principle ................................................................................................................................ 8

Introduction
Norms: Vectors and matrices
Definition:
1
For 𝑥 = (𝑥1 , 𝑥2 , … , 𝑥𝑑 ) ∈ 𝑅 𝑑 , 𝑝 ∈ 𝑅+ , 𝑝 < ∞, ||𝑥||𝑝 = (|𝑥1 |𝑝 + |𝑥2 |𝑝 + ⋯ + |𝑥𝑑 |𝑝 )𝑝 .

If 𝑝 = ∞, ||𝑥||𝑝 = max(|𝑥1 |, … , |𝑥𝑑 |).

Definition 2:

If 𝐴 ∈ 𝑀𝑚,𝑛 (𝐶), ||𝐴||𝛼𝛽 = sup ||𝐴𝑥||𝛽 . If 𝛼 = 𝛽, ||𝐴||𝛼𝛼 = ||𝐴||𝛼 .

||𝑥||𝛼 ≤1

Example: ||𝐴||1 = max ∑𝑚 𝑛

𝑖=1|𝑎𝑖𝑗 |; ||𝐴||∞ = max ∑𝑗=1 |𝑎𝑖𝑗 |.
1≤𝑗≤𝑛 1≤𝑖≤𝑚

2
Frobenius norm: ||𝐴||𝐹 = √∑𝑚 𝑛
𝑖=1 ∑𝑗=1|𝑎𝑖𝑗 |

Properties:
||𝑨||𝟐 = √𝝆(𝑨𝒕 𝑨) where 𝝆(𝑿) = spectral radius of 𝑿 ∈ 𝑴𝒎 (𝑪): = 𝒎𝒂𝒙{|𝝀𝟏 |, … , |𝝀𝒎 |}

Code in Python
import numpy as np
def test_norms():
def test1():
"I will test norm1 of vector: the sum of absolute values of elems"
v = np.array([1,2,3,4])
v2 = np.array([1,-2,3,-4])
print("You are inside test1")
print("Norm-1 of v is:",np.linalg.norm(v,1))
print("Norm-1 of v2 is:",np.linalg.norm(v2,1))
test1()
def test2():
v = np.array([1,2,3,4])
print("Inside test 2")
print("Usual norm of v is",np.linalg.norm(v))
print("2-norm of V is",np.linalg.norm(v,2))
print("Infinity norm of V is",np.linalg.norm(v,np.inf))
print("As an alternative, the same norm of V is",np.max(np.abs(v)))
"by default the norm computed is the norm-2"
def test3():
v = np.array([[1,2,3,4],[-1,2,3,3]])
print("Inside test 3:")
print("Norm 2 of the matrix v is:",np.linalg.norm(v))
def test4():
v = np.array([[1,2,3,4],[-1,2,3,3]])
print("The inf norm of matrix v is:",np.linalg.norm(v,np.inf))
res1,res2 = np.sum(np.abs(v[0])),np.sum(np.abs(v[1]))
print("Alternatively, norm inf of matrix v is",np.max([res1,res2]))
test2()
test3()
test4()

Exercise1 :

a. Implement your own functions for norms (p and ∞).(At least for vectors).

b. Verify in Python the property ||𝐴||2 = √𝜌(𝐴𝑡 𝐴) .

Solution :

a.

import numpy as np
def norms(num):
def norm1(v):
"v is a vector"
return np.sum(np.abs(v))
 def norm2(v):
"v is also vector"
return np.sqrt(np.sum(v**2))
def test():
v = np.array([3,4,5])
print(norm2(v))
def norm(v,p):
return (np.sum(v**p))**(1/p)
def test2():
v = np.array([3,4,5])
print(norm(v,2))
dic = {1:norm1,2:norm2,3:norm}
return dic[num]
You can add test1() and test2() calls inside norms library to witness the functionality of the defined
functions.

b.

def exercise():
def spectral_radius(A):
eig = np.linalg.eig(A)[0]
return np.max(np.abs(eig))
def quantity(A):
"the RHS of the equality"
X = np.dot(np.transpose(A),A)
return np.sqrt(spectral_radius(X))
def test():
v = np.array([[1,2,3,4],[-1,2,3,3]])
print("Norm 2 of the matrix v is:",np.linalg.norm(v,2))
print("The RHS quantity is:",quantity(v))
test()

Symbolic calculus
Sympy is a library defined to handle symbolic calculus.

This library has various features for limits, differentiation, integration, etc.

import sympy as sym

import numpy as np
def test_symbols():
def test1():
x = sym.Symbol('x')
print(sym.diff(sym.sin(x),x))
test1()
def test2():
h = lambda x: sym.sin(x)
print('The sine function in pi/2',h(np.pi/2))
h2 = lambda x: sym.diff(h(x))
print('The derivative of sine in pi/2=',h2(np.pi/2))
test2()
 def test3():
x = sym.Symbol('x')
y = x**2+1
yprime = y.diff(x)
print(yprime)
test3()
def test4():
x = sym.Symbol('x')
y = x**2+1
yprime = y.diff(x)
f = sym.lambdify(x,yprime,'numpy')
print(f(np.array([1,2,3,4,5])))
test4()

Numerical methods for non-linear equations in R

Bisection method
Methodology
Let 𝑎 < 𝑏 ∈ 𝑅 and 𝑓: [𝑎, 𝑏] → 𝑅 a continuous function with 𝑓(𝑎) ⋅ 𝑓(𝑏) < 0. Then ∃𝑧 ∈ [𝑎, 𝑏] such that
𝑓(𝑧) = 0.
We define (𝑎𝑛 )𝑛≥0 , (𝑏𝑛 )𝑛≥0 , (𝑐𝑛 )𝑛≥0 such that :
𝑎+𝑏
• 𝑎0 = 𝑎, 𝑏0 = 𝑏, 𝑐0 = ;
2
• For 𝑛 ≥ 1
𝑎𝑛 = 𝑎𝑛−1
o If 𝑓(𝑐𝑛−1 ) = 0 then { 𝑏𝑛 = 𝑏𝑛−1
𝑐𝑛 = 𝑐𝑛−1
𝑎𝑛 = 𝑎𝑛−1
o If 𝑓(𝑎𝑛−1 )𝑓(𝑐𝑛−1 ) < 0 then { 𝑏𝑛 = 𝑐𝑛−1
𝑐𝑛 = (𝑎𝑛 + 𝑏𝑛 )/2
𝑎𝑛 = 𝑐𝑛−1
o If 𝑓(𝑎𝑛−1 )𝑓(𝑐𝑛−1 ) > 0 then { 𝑏𝑛 = 𝑏𝑛−1 .
𝑐𝑛 = (𝑎𝑛 + 𝑏𝑛 )/2

Suppose that f has a unique root in [𝑎, 𝑏].

Then the sequence (𝑐𝑛 )𝑛≥0 built above converges to the unique solution 𝑧 ∈ [𝑎, 𝑏] of equation 𝑓(𝑥) =
𝑏−𝑎
0 and |𝑐𝑛 − 𝑧| ≤ 2𝑛
.

Python code
import numpy as np
def bisection_method():
def condition(f,a,b):
"We test if f(a)f(b)<0"
if f(a)*f(b)>=0:
return 0
else:
return 1
 def bisect(f,a,b,tol):
if condition(f,a,b)==0:
raise Exception("Choose another pair of endpoints")
else:
x,y,z = a,b,(a+b)/2
err,n = 1,1
while err>=tol and n<30:
if f(z) == 0:
return z,err,n
elif f(x)*f(z)<0:
x=x
y=z
z = (x+y)/2
err = np.abs(z-x)
n = n+1
elif f(x)*f(z)>0:
x=z
z = (x+y)/2
err = np.abs(z-x)
n = n+1
return z,err,n
def test1():
f = lambda x: np.exp(x)-x-2
a,b = 1,2
res = bisect(f,a,b,0.00001)[0]
print(f(res))
print(bisect(f,a,b,0.00001))
test1()

Contraction method
Methodology
Let 𝐼 ⊂ 𝑅 an interval and 𝑓: 𝐼 → 𝑅 a function. The function 𝑓 is called contraction if and only if :

a) ∃𝑞 ∈ (0,1) such that |𝑓(𝑥) − 𝑓(𝑦)| ≤ 𝑞|𝑥 − 𝑦|, ∀𝑥, 𝑦 ∈ 𝐼 ;

𝑏) 𝑓(𝐼) ⊂ 𝐼.

If 𝑓: [𝑎, 𝑏] → 𝑅 is differentiable and :

a) If ∃𝑞 ∈ (0,1) such that |𝑓 ′ (𝑥)| ≤ 𝑞 for which 𝑥 ∈ [𝑎, 𝑏] ⇒ |𝑓(𝑥) − 𝑓(𝑦)| ≤ 𝑞|𝑥 − 𝑦|, ∀𝑥, 𝑦 ∈ [𝑎, 𝑏].
𝑎+𝑏 𝑎+𝑏 𝑏−𝑎
b) If |𝑓(𝑥) − 𝑓(𝑦)| ≤ 𝑞|𝑥 − 𝑦|and |𝑓 ( 2
)− 2 | ≤ (1 − 𝑞) ⋅ 2
then 𝑓([𝑎, 𝑏]) ⊂ [𝑎, 𝑏].

Main statement :
Let 𝒇: 𝑰 → 𝑹 be a contraction and 𝑥0 ∈ 𝐼. We define the sequence (𝑥𝑛 )𝑛 through the relation 𝑥𝑛+1 =
𝑓(𝑥𝑛 ), ∀𝑛 ∈ 𝑁. Then the equation 𝑓(𝑥) = 𝑥 has a unique solution 𝑧 ∈ [𝑎, 𝑏] and the sequence 𝑥𝑛 → 𝑧,
with the following error evaluation formula :
𝑞 𝑞𝑛
|𝑥𝑛 − 𝑧| ≤ |𝑥𝑛 − 𝑥𝑛−1 | ≤ |𝑥 − 𝑥0 |, ∀𝑛 ∈ 𝑁.
1−𝑞 1−𝑞 1

Python code
import numpy as np
def contraction(f,x0,tol):
"this is for functions f:R->R"
err,n = 1,0
x = x0
while err>=tol and n<30:
y = f(x)
err = np.abs(x-y)
n = n+1
x=y
return x,err,n

def test():
f = lambda x:x**2+x-7/16
x0 = -5/8
tol = 0.00001
print(contraction(f,x0,tol))
res = contraction(f,x0,tol)[0]
print(f(res))
#res will be a fixed point for f, f(res)=res

Other Methods: Newton, secant, chord

Methodology
1. Chord method

Conditions: Let [𝑎, 𝑏] ⊂ 𝑅, 𝑓: [𝑎, 𝑏] → 𝑅. Suppose the next conditions are met:

a. 𝑓 ′′ (𝑥) ≠ 0, ∀𝑥 ∈ [𝑎, 𝑏];

b. 𝑓(𝑎)𝑓 ′′ (𝑎) > 0;
c. 𝑓(𝑎) ⋅ 𝑓(𝑏) < 0.
𝑥0 𝑓(𝑥𝑛 )−𝑥𝑛 𝑓(𝑥0 )
Then the sequence 𝑥0 = 𝑎, 𝑥1 = 𝑏, 𝑥𝑛+1 = converges to the unique solution of
𝑓(𝑥𝑛 )−𝑓(𝑥0 )
𝑓(𝑥) = 0.
2. Secant method

Conditions: Let [𝑎, 𝑏] ⊂ 𝑅, 𝑓: [𝑎, 𝑏] → 𝑅. Suppose the following statements hold:

a. 𝑓 ′ (𝑥) ≠ 0, ∀𝑥 ∈ [𝑎, 𝑏].

b. (𝑥𝑛 ) ⊂ [𝑎, 𝑏].
c. 𝑓(𝑎) ⋅ 𝑓(𝑏) < 0.
 𝑥𝑛−1 𝑓(𝑥𝑛 )−𝑥𝑛 𝑓(𝑥𝑛−1 )
Then the sequence 𝑥0 , 𝑥1 ∈ [𝑎, 𝑏], 𝑥𝑛+1 = 𝑓(𝑥𝑛 )−𝑓(𝑥𝑛−1 )
converges to the unique solution
of 𝑓(𝑥) = 0.
3. Newton method
Conditions: Let the following conditions considered as satisfied:
a. 𝑓 ′ , 𝑓 ′′ ≠ 0 𝑜𝑛 [𝑎, 𝑏]
b. 𝑓(𝑎)𝑓(𝑏) < 0
c. Let 𝑥0 ∈ [𝑎, 𝑏] such that 𝑓(𝑥0 )𝑓 ′′ (𝑥0 ) > 0.
𝑓(𝑥 )
Then the sequence 𝑥𝑛+1 = 𝑥𝑛 − 𝑓′ (𝑥𝑛 ) , ∀𝑛 ∈ 𝑁 converges to the unique solution 𝑧 of
𝑛
𝑓(𝑥) = 0.
The error is defined by the following inequality:
|𝑥𝑛 − 𝑧| ≤ |𝑓(𝑥𝑛 )|/( min |𝑓 ′ (𝑥)|) .
𝑥∈[𝑎,𝑏]

Remarks:

1. Secant method requires less information about the function, 𝑓 must be only one time differentiable
(compared to two times in Newton or chord), but the number of operations involved is bigger,
because at step 𝑛 + 1 it depends on the solutions found at steps 𝑛& 𝑛 − 1.
2. Newton method requires the biggest number of conditions to be fulfilled but is the fastest method.

Python code
import numpy as np
def num_method(num):
def chord(f,a,b,no_iter):
if f(a)*f(b)<0:
x,y=a,b
for i in range(no_iter):
x,y = y,(a*f(x)-x*f(a))/(f(x)-f(a))
return y
else:
return "Choose another pair of endpoints"

def secant(f,x0,x1,no_iter):
if f(x0)*f(x1)<0:
x,y = x0,x1
for i in range(no_iter):
x,y = y,(x*f(y)-y*f(x))/(f(y)-f(x))
return y
else:
return "Choose another pair of values for x0 and x1"
def Newton(f,fprime,x0,no_iter):
x = x0
for i in range(no_iter):
x = x-f(x)/fprime(x)
return x
dic = {1:chord,2:secant,3:Newton}
return dic[num]

def tests():
def test_chord():
 g=num_method(1)
h = lambda x:np.exp(x)-x-2
print(g(h,0,2,20))
def test_secant():
h = lambda x:np.exp(x)-x-2
g = num_method(2)
print(g(h,0,4,50))
def test_Newton():
h1 = lambda x:np.exp(x)-x-2
h1_prime = lambda x:np.exp(x)-1
g = num_method(3)
#print(g(h1,h1_prime,0,10))
print(g(h1,h1_prime,1,10))
def test_Newton2():
import scipy.optimize as scp
h1 = lambda x:np.exp(x)-x-2
h1_prime = lambda x:np.exp(x)-1
print(scp.newton(h1,1,h1_prime)) #uses Newton method
print(scp.newton(h1,1))
test_Newton()
test_Newton2()

Numerical methods for non-linear equations in 𝑹𝒏

Contraction principle
Let 𝑚 ∈ 𝑁 ∗ , 𝑀 ⊂ 𝑅 𝑚 and || ⋅ || a norm on 𝑅 𝑚 . A function 𝑓: 𝑀 → 𝑅 𝑚 is called 𝑞 −contraction of the set
M, with respect to the norm || ⋅ || if:

a) ∃𝑞 ∈ (0,1) such that ||𝑓(𝑥) − 𝑓(𝑦)|| ≤ 𝑞||𝑥 − 𝑦||, ∀𝑥, 𝑦 ∈ 𝑀 and

𝑏) 𝑓(𝑀) ⊂ 𝑀.

Statement:

Let be || ⋅ || a norm on 𝑅 𝑚 ,M a closed set in 𝑅 𝑚 ,𝑓: 𝑀 → 𝑅 𝑚 a q-contraction of M. Then the equation

𝑓(𝑥) = 𝑥 has a unique solution 𝑧 ∈ 𝑀 and for every 𝑥0 ∈ 𝑀, the sequence defined through the relation
𝑥 (𝑛+1) = 𝑓(𝑥 (𝑛) ) → 𝑧. We have the evaluation:
𝑞 𝑞𝑛
||𝑥 (𝑛) − 𝑧|| ≤ 1−𝑞 ||𝑥 (𝑛) − 𝑥 (𝑛−1) || ≤ 1−𝑞 ||𝑥 (1) − 𝑥 (0) ||.

Exercise:
80+𝑥 3 −𝑥 2 −3𝑦𝑧
𝑥= 100
60−𝑦 3 −𝑧 2 +4𝑦𝑧
Solve the following system: 𝑦 = on the set 𝑉 = [0,2]3 , starting from 𝑥 (0) = (1,1,1).
86
40+𝑧 3 +𝑦 2 −5𝑥𝑦
{𝑧 = 72
Python code:

def function():
F = lambda x: np.array([(80+x[0]**3-x[0]**2-3*x[1]*x[2])/100,\
(60-x[1]**3-x[2]**2+4*x[0]*x[2])/86,\
(40+x[2]**3+x[1]**2-5*x[0]*x[1])/72])
return F

def fixed_point_function(x0,tol):
'I will compute the fixed point on [0,2]x[0,2]x[0,2] starting from the '
'point x0'
x,err,n = x0,1,1
F = function()
while err>=tol and n<30:
z = F(x)
err = np.linalg.norm(z-x,np.inf)
n = n+1
x=z
return x,err,n

def test_fixed_point_function():
x0 = np.array([1,1,1])
tol = np.exp(-5)
print(fixed_point_function(x0,tol))

Exercise 2:

For the previous system show at each iteration up to the 10th the solution 𝑥 (𝑛) .

Solution:
def fixed_point_function2(x0):
x,n=x0,1
F=function()
while n<=10:
z=F(x)
err = np.linalg.norm(z-x,np.inf)
n = n+1
x = z
print(x,err)

def test_fixed_point_function2():
x0=np.array([1,1,1])
fixed_point_function2(x0)