ISE 3414
Fall 2025
Homework 5
1. Consider a Markov chain with states [1, 2, 3, 4] and the TPM below. Please list the classes and
state whether they are recurrent, transient, or absorbing. Calculate the period for each class
and state whether it is ergodic or not. Finally, state whether the Markov chain is irreducible
ergodic, ergodic, or not ergodic.
0.3 0.7 0 0
0.4 0.6 0 0
𝑃𝑃 = � �
0 0 1 0
0.3 0.2 0.3 0.2
2. Consider a Markov chain with states [1, 2, 3, 4] and the TPM below. Please list the classes and
state whether they are recurrent, transient, or absorbing. Calculate the period for each class
and state whether it is ergodic or not. Finally, state whether the Markov chain is irreducible
ergodic, ergodic, or not ergodic.
0 0.2 0 0.8
0 0 0 1
𝑃𝑃 = � �
1 0 0 0
0 0 1 0
3. Consider a Markov chain with states [1, 2, 3, 4, 5] and the TPM below. Please list the classes and
state whether they are recurrent, transient, or absorbing. Calculate the period for each class
and state whether it is ergodic or not. Finally, state whether the Markov chain is irreducible
ergodic, ergodic, or not ergodic.
0.3 0.3 0.4 0 0
⎡0 0 0 1 0⎤
⎢ ⎥
𝑃𝑃 = ⎢0.5 0 0 0 0.5⎥
⎢ 0 0.4 0 0 0.6⎥
⎣0 0 0 0 1⎦
4. For this question, consider Lecture 15, Example 8, reproduced here:
At all times, an urn contains two balls (where each ball is either green or red). At each stage, a
fair die is tossed. If a 3 or a 6 appears on the die, then a ball is chosen at random from the urn
and replaced by a green ball. If any other number appears on the die, then a ball is chosen from
the urn and is replaced by a red ball. Let the state of the system at any stage (𝑋𝑋𝑡𝑡 ) be the
number of red balls left in the urn after replacement 𝑡𝑡. The TPM is equal to
0 1 2
0 1/3 2/3 0
𝑃𝑃 = 1 �1/6 1/2 1/3�
2 0 1/3 2/3
a. At a given time in the distant future, what is the probability that there are 2 red balls in the
urn?
b. You’re playing a game with a friend. At each turn, they will give you $50 for each red ball in
the urn, and you will give them $50 for each green ball. On an average turn in the distant
future, how much money do you expect to earn from or owe to your friend?
5. Consider Homework 4, #2, reproduced here:
At all times, a bowl contains three pieces of fruit, which are either apples or oranges. Each hour,
someone walks into the room and chooses a piece of fruit at random. 60% of the time they will
replace it with an apple, and 40% of the time they will replace it with an orange. Let the state of
the system at any hour (𝑋𝑋𝑡𝑡 ) represent the number of apples in the bowl. The TPM is equal to
0 1 2 3 0 1 2 3
0 2/5 3/5 0 0 0 0.4 0.6 0 0
1 2/15 7/15 2/5 0 1 0.1333 0.4667 0.4 0
𝑃𝑃 = � �= � �
2 0 4/15 8/15 1/5 2 0 0.2667 0.5333 0.2
3 0 0 2/5 3/5 3 0 0 0.4 0.6
At a given time in the distant future, what is the probability that there are 2 apples in the bowl?
