EC 636 - Fall 2019

Introduction to Probability Theory

Dr. Nabil Drawil

University of Tripoli, Libya
Department of Computer Engineering

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 1 / 27

Outline

1 Sample Space and Event

2 Probability of An Event

3 Independent Events
Independent Events

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 2 / 27

 Sample Space and Event

Experiment Sample Space and Event

Experiment (E): Any procedure that (1) can be repeated

theoretically an infinite number of times and (2) has a well-defined
set of possible outcomes.
Random experiment: conditions of the experiment only partially
determined the outcome
repeatable
variable outcomes
statistical regularity

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 3 / 27

 Sample Space and Event

Sample Space (S)

The set of all possible outcomes of an E

Some sample spaces
coin toss S = {H, T }, {0, 1}
roll of two dice S = {(1, 1), (1, 2), . . . , (6, 6)}
component lifetime S = {l ∈ (0, ∞)}
daily high temperature S = {(ti , i = 1, 2, . . . )}
noise waveform S = {(n(t), t ∈ <), each n(t) ∈ <}

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 4 / 27

 Sample Space and Event

Sample Space (S), cont.

Consider roll of two dice

die #1 s S

S
6

4
an event
A={dice sum to 7}
3

die #2
1 2 3 4 5 6

This is a product space

S1 = {1, 2, 3, 4, 5, 6}
S2 = {1, 2, 3, 4, 5, 6}
S = S1 × S2 = {(s1 , s2 ) : s1 ∈ S1 , s2 ∈ S2 }
where (s1 , s2 ) are ordered pairs

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 5 / 27

 Sample Space and Event

Types of Sample Space

Finite e.g., S = {0, 1} the binary alphabet

Countable e.g., S = {0, 1, 2, . . . } the non-negative integers
Uncountable (mostly continuous) e.g., S = [0, 1] the closed unit
interval
where finite and countable sample spaces are called discrete sample
space.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 6 / 27

 Sample Space and Event

Event: a subset of S

Any designated collection of possible outcomes of an experiment

including individual outcomes, the entire sample space, and the null
set , constitutes an event.

An event is said to occur if the outcome is one of the constituent

members of that event.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 7 / 27

 Sample Space and Event

Elementary Event

Typical form of an elementary event.

Single value s ∈ S
Vector s = (s1 , s2 , . . . , sk ), each si ∈ Si . i.e.,
S = S1 × S2 × · · · × Sn
Sequence s = (si , i = 1, 2, . . . ), each si ∈ Si
Waveform s = (s(t), t ∈ <)

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 8 / 27

 Probability of An Event

Probability of An Event
n(A)
Relative frequency of an event A = n
n(A)
0≤ n ≤1
If A and B are disjoint events,
then n(A∪B)
n = n(A)
n + n
n(B)

Example
Experiment: toss a fair coin. S = {H, T }, A = {H}
n(A) 1
n −→ 2 as n −→ ∞

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 9 / 27

 Probability of An Event

Probability of An Event (cont.)

n(A)
Probability of an event A : P(A) = lim n
n−→∞
It indicates the likelihood that event A will occur.

Review: set constructions P1-4

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 10 / 27

 Probability of An Event

Axioms of Probability

1 0 ≤ P(A) ≤ 1 for all A ⊆ S and P(S) = 1: non negative and

normalized.
2 If A ∩ B = φ, A ⊂ S and B ⊂ S, then P(A ∪ B) = P(A) + P(B).
3 If A1 , A2 , . . . are finite or infinite disjoint events,
n n
!
[ X
P Ai = P(Ai ) finitely additive
i=1 i=1

∞ ∞
!
[ X
P Ai = P(Ai ) countably additive
i=1 i=1

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 11 / 27

 Probability of An Event

Review: Set Constructions

n
S
Union Ai points in any set of collection
i=1
n
T
intersection Ai points in every set of collection
i=1
disjoint (or mutually exclusive ) collection has Ai ∩ Aj = φ for i 6= j.
n
S
exhaustive collaction has Ai = S.
i=1
Partition is a disjoint exhaustive collection.

A1 A2 A3
A4

An
S
A5

A6

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 12 / 27

 Probability of An Event

A Special Case

Finite sample space:

S = {s1 , s2 , . . . , sn }

0 ≤ P(si ) ≤ 1
Pn
P(si ) = 1
i=1
S P
If A = si ; then P(A) = P(si )
i∈I i∈I

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 13 / 27

 Probability of An Event

Basic Properties of probability

1 P(Ā) = 1 − P(A) for A ⊆ S.

2 P(A) ≤ P(B) for A ⊆ B. Bi
3 If B1 , B2 , . . . , Bn form a partition of S. then for any
A ⊆ S.
n

S
X
P(A) = P(A ∩ Bi )
i=1

4 P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

for A, B ⊆ S.
5 Given A1 , A2 , . . . , An ⊆ S
A
 
n
[ n
X X
P Ai  = P(Ai ) − P(Ai ∩ Ai )
1 2
i=1 i=1 1≤i1 <i2 ≤n
X
+ P(Ai ∩ Ai ∩ Ai ) + . . .
1 2 3
1≤i1 <i2 <i3 ≤n

n−1
X
+ (−1) P(Ai ∩ Ai ∩ · · · ∩ Ain )
1 2
1≤i1 <i2 <···<in ≤n

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 14 / 27

 Probability of An Event

Conditional Probability

A given B or A|B is an event in a new space defined by

B → P(A|B) should be normalized by P(B).
Outcomes in A|B are outcomes in A ∩ B
P(A ∩ B)
P(A|B) = , if P(B) > 0
P(B)
can be viewed as a cutting down of the original sample space
S→B
A → A/B

P(A ∩ B) = P(A|B)P(B) = P(B|A)P(A)

P(A ∩ B ∩ C) = P(A|B ∩ C)P(B ∩ C) = P(A|B ∩ C)P(B|C)P(C)

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 15 / 27

 Probability of An Event

Example

Family has two children

A = {both are boys}
B = {at least one is a boy}

Child 2

girl
B
A
boy
Child 1

boy girl

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 16 / 27

 Probability of An Event

Example

Family has two children

A = {both are boys}
B = {at least one is a boy}

Child 2

girl
B
A
boy
Child 1

boy girl

1 3 1 1
P(A) = 4 P(B) = 4 P(A ∩ B) = 4 P(A|B) = P(A ∩ B)/P(B) = 3

A and B are dependent events

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 16 / 27

 Probability of An Event

Excercises

See problems on conditional

probability (e.g., 12 to 14).

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 17 / 27

 Probability of An Event

Bayes’ Formula

If B1 , B2 , . . . , form a partition of S.
i.e., Bi ∩ Bj = φ for i 6= j - disjoint
∞
S
Bi = S - exhaustive
i=1
∞  ∞
S S
then A = A ∩ S = A ∩ Bi = (A ∩ Bi )
i=1 i=1

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 18 / 27

 Probability of An Event

Bayes’ Formula (cont.)

Total probability formula

∞
[ ∞
X ∞
X
P(A) = P( (A ∩ Bi )) = P(A ∩ Bi) = P(A|Bi )P(Bi )
i=1 i=1 i=1

∞
P
i.e., P(A) = P(A|Bi )P(Bi )
i=1

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 19 / 27

 Probability of An Event

Bayes’ Formula (cont..)

P(A ∩ Bj )
P(Bj |A) =
P(A)
P(A|Bj )P(Bj )
∴ P(Bj |A) = ∞
P
P(A|Bi )P(Bi )
i=1

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 20 / 27

 Probability of An Event

Example
A satellite is sending a binary code {0, 1}. 70% of the message is 00 s. 80% chance of
a given 0 or 1 being received correctly. Find the probability that 0 was sent if 1 is
received.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

 Probability of An Event

Example
A satellite is sending a binary code {0, 1}. 70% of the message is 00 s. 80% chance of
a given 0 or 1 being received correctly. Find the probability that 0 was sent if 1 is
received.

80%
70% ‘0’
“0” 62%

20
%

20% error 14%

“1” 38%
30% ‘1’
80%
correct 24%

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

 Probability of An Event

Example
A satellite is sending a binary code {0, 1}. 70% of the message is 00 s. 80% chance of
a given 0 or 1 being received correctly. Find the probability that 0 was sent if 1 is
received.

80%
70% ‘0’
“0” 62%

20
%

20% error 14%

“1” 38%
30% ‘1’
80%
correct 24%

Bi : i was sent (i = 0, 1)
Ai : i is received (i = 0, 1)
P(B0 ) = 0.7 ⇒ P(B1 ) = 0.3
P(A0 |B0 ) = 0.8 ⇒ P(A1 |B0 ) = 0.2
P(A1 |B1 ) = 0.8 ⇒ P(A0 |B1 ) = 0.2

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

 Probability of An Event

Example
A satellite is sending a binary code {0, 1}. 70% of the message is 00 s. 80% chance of
a given 0 or 1 being received correctly. Find the probability that 0 was sent if 1 is
received.

80%
70% ‘0’
“0” 62%

20
%

20% error 14%

“1” 38%
30% ‘1’
80%
correct 24%

Using Bayes’ formula,

Bi : i was sent (i = 0, 1)
Ai : i is received (i = 0, 1)
P(B0 ) = 0.7 ⇒ P(B1 ) = 0.3 P(A1 |B0 )P(B0 )
P(B0 |A1 ) =
P(A0 |B0 ) = 0.8 ⇒ P(A1 |B0 ) = 0.2 P(A1 |B0 )P(B0 ) + P(A1 |B1 )P(B1 )
P(A1 |B1 ) = 0.8 ⇒ P(A0 |B1 ) = 0.2 0.2 × 0.7
= = 0.37
0.2 × 0.7 + 0.8 × 0.3

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 21 / 27

 Independent Events Independent Events

Independent Events

If P(A|B) = P(A) or P(B|A) = P(B)

⇒ P(A ∩ B) = P(A)P(B)

then A and B are independent; otherwise, they are dependent.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 22 / 27

 Independent Events Independent Events

Example1: role two dice

Physically unrelated events

A = { die #2 roll > 3} die #1 B
B = { die #1 roll > 3}
6

5 A
P(A) = 18/36 = 1/2 P(B) = 18/36 = 1/2
4

3
P(A ∩ B) = 9/36 = 1/4
2

die #2
∴ P(A ∩ B) = P(A)P(B) ⇒ B and A are 1 2 3 4 5 6

independent.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 23 / 27

 Independent Events Independent Events

Example2: role two dice

Physically related events

B = { die #1 roll > 3} die #1 B C
C = { dice sum > 7}
6

5
P(B) = 1/2 P(C) = 15/36 = 5/12
4

3
P(B ∩ C) = 12/36 = 1/3
2

die #2
∴ P(B ∩ C) 6= P(B)P(C) ⇒ B and C are 1 2 3 4 5 6
dependent.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 24 / 27

 Independent Events Independent Events

Remarks: Independent versus Disjoint

In fact, disjoint events of nonzero probability must be dependent.

disjoint: P(A ∪ B) = P(A) + P(B) where P(A ∩ B) = 0.

Independent: P(A ∩ B) = P(A)P(B).
Independence is often assumed if events have no obvious
physical connection.

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 25 / 27

 Independent Events Independent Events

Independence: Collection of Events

Consider events A1 , A2 , . . .
 
Q T
If they ere independent, then P Ai = P(Ai ), hence
  i i
T
P Ai is non-increasing as i increases.
i
Events Ai , for i = 1, 2, . . . are independent if
!
\ Y
P Ai = P(Ai ), for any I = {i1 , i2 , . . . , ir }and any r > 1
i∈I i∈I

Events Ai , for i = 1, 2, . . . , are pairwise independent if

 
n
P(Ai ∩ Aj ) = P(Ai )P(Aj ) for any i 6= j. →   combinations
2

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 26 / 27

 Independent Events Independent Events

Independence: Collection of Events (cont.)

Note: independent ⇒ pairwise independent.

but pairwise independent 6⇒ independent.
Example
Draw ball from urn containing four balls numbered 1,2,3,4
A1 = {1, 2} P(A1 ) = 1/2 P(A1 ∩ A2 ) = 1/4 P(A1 ∩ A2 ∩ A3 ) = 1/4
A2 = {1, 3} P(A2 ) = 1/2 P(A1 ∩ A3 ) = 1/4 ⇒
A3 = {1, 4} P(A3 ) = 1/2 P(A2 ∩ A3 ) = 1/4
pairwise independent but not independent

Nabil Drawil (University of Tripoli) EC 636 - Fall-2019 27 / 27