UCSD ECE153 Handout #34

Prof. Young-Han Kim Tuesday, May 27, 2014

Solutions to Homework Set #6

(Prepared by TA Fatemeh Arbabjolfaei)

1. Linear estimator. Consider a channel with the observation Y = XZ, where the signal X and
the noise Z are uncorrelated Gaussian random variables. Let E[X] = 1, E[Z] = 2, σX 2 = 5,

and σZ2 = 8.

(a) Find the best MSE linear estimate of X given Y .

(b) Suppose your friend from Caltech tells you that he was able to derive an estimator with
a lower MSE. Your friend from UCLA disagrees, saying that this is not possible because
the signal and the noise are Gaussian, and hence the best linear MSE estimator will also
be the best MSE estimator. Could your UCLA friend be wrong?

Solution:

(a) We know that the best linear estimate is given by the formula

Cov(X, Y )
X̂ = (Y − E(Y )) + E(X).
σY2

Note that X and Z Gaussian and uncorrelated implies they are independent. Therefore,

E(Y ) = E(XZ) = E(X)E(Z) = 2,

E(XY ) = E(X 2 Z) = E(X 2 )E(Z) = (σX
2
+ E2 (X))E(Z) = 12,
E(Y 2 ) = E(X 2 Z 2 ) = E(X 2 )E(Z 2 ) = (σX
2
+ E2 (X)), (σZ2 + E2 (Z)) = 72,
σY2 = E(Y 2 ) − E2 (Y ) = 68,
Cov(X, Y ) E(XY ) − E(X)E(Y ) 5
= = .
σY2 σY2 34

Using all of the above, we get

5 12
X̂ = Y + .
34 17
(b) The fact that the best linear estimate equals the best MMSE estimate when input and
noise are independent Gaussians is only known to be true for additive channels. For
multiplicative channels this need not be the case in general. In the following, we prove
Y is not Gaussian by contradiction.
Suppose Y is Gaussian, then Y ∼ N (2, 68). We have

1 (y−2)2
fY (y) = √ e− 2×68 .
2π × 68

1
 On the other hand, as a function of two random variables, Y has pdf
Z ∞ y
fY (y) = fX (x)fZ dx.
−∞ x

But these two expressions are not consistent, because

Z ∞   Z ∞
0
fY (0) = fX (x)fZ dx = fZ (0) fX (x) dx = fZ (0)
−∞ x −∞
1 (0−2)2
=√ e− 2×8
2π × 8
1 (0−2)2
6= √ e− 2×68 = fY (0) ,
2π × 68
which is a contradiction. Hence, X and Y are not joint Gaussian, and we might be able
to derive an estimator with a lower MSE.

2. Additive-noise channel with path gain. Consider the additive noise channel shown in the figure
below, where X and Z are zero mean and uncorrelated, and a and b are constants.

X a b Y = b(aX + Z)

Find the MMSE linear estimate of X given Y and its MSE in terms only of σX , σZ , a, and b.

Solution: By the theorem of MMSE linear estimate, we have

Cov(X, Y )
X̂ = (Y − E(Y )) + E(X).
σY2

Since X and Z are zero mean and uncorrelated, we have

E(X) = 0,
E(Y ) = b(aE(X) + E(Z)) = 0,
2
Cov(X, Y ) = E(XY ) − E(X)E(Y ) = E(Xb(aX + Z)) = a b σX ,
σY2 = E(Y 2 ) − (E(Y ))2 = E(b2 (aX + Z)2 ) = b2 a2 σX
2
+ b2 σZ2 .

Hence, the best linear MSE estimate of X given Y is given by

2
a σX
X̂ = 2 + b σ 2 Y.
b a2 σX Z

2
3. Image processing. A pixel signal X ∼ U[−k, k] is digitized to obtain
1
X̃ = i + , if i < X ≤ i + 1, i = −k, −k + 1, . . . , k − 2, k − 1.
2
To improve the the visual appearance, the digitized value X̃ is dithered by adding an inde-
pendent noise Z with mean E(Z) = 0 and variance Var(Z) = N to obtain Y = X̃ + Z.

(a) Find the correlation of X and Y .

(b) Find the best linear MSE estimate of X given Y . Your answer should be in terms only
of k, N, and Y .

Solution:

(a) From the definition of X̃, we know P{X̃ = i + 21 } = P{i < X ≤ i + 1} = 1

2k . By the law
of total expectation, we have

Cov(X, Y ) = E(XY ) − E(X)E(Y ) = E(X(X̃ + Z)) = E(X X̃)

k−1
X
= E[X X̃ | i < X ≤ i + 1]P (i < X ≤ i + 1)
i=−k
k−1 Z i+1 k−1 k
X 1 1 1 X 1 X
= x(i + ) dx = (2i + 1)2 = (2i − 1)2
2 2k 8k 4k
i=−k i i=−k i=1
4k 2 − 1
= .
12
Pk 2
Since, i=1 i = k(k + 1)(2k + 1)/6.
(b) We have

E(X) = 0,
E(Y ) = E(X̃) + E(Z) = 0,
k−1 k−1
X 1 2 1 1 X 4k 2 − 1
σY2 = VarX̃ + VarZ = (i + ) +N = (2i + 1)2 + N = + N.
2 2k 4k 12
i=−k i=0

Then, the best linear MMSE estimate of X given Y is given by

4k2 −1
Cov(X, Y ) 12
X̂ = (Y − E(Y )) + E(X) = Y
σY2 4k2 −1
12 + N
4k 2 − 1
= Y.
4k 2 − 1 + 12N
4. Covariance matrices. Which of the following matrices can be a covariance matrix? Justify
your answer either by constructing a random vector X, as a function of the i.i.d zero mean unit
variance random variables Z1 , Z2 , and Z3 , with the given covariance matrix, or by establishing
a contradiction.

3
    
    1 1 1 1 1 2
1 2 2 1
(a) (b) (c)  1 2 2  (d)  1 2 3 
0 2 1 2
1 2 3 2 3 3

Solution:

(a) This cannot be a covariance matrix because it is not symmetric.

(b) This is a covariance matrix for X1 = Z1 + Z2 and X2 = Z1 + Z3 .
(c) This is a covariance matrix for X1 = Z1 , X2 = Z1 + Z2 , and X3 = Z1 + Z2 + Z3 .
(d) This cannot be a covariance matrix. Suppose it is, then σ23 2 = 9 > σ σ
22 33 = 6, which
contradicts the Cauchy–Schwartz inequality. You can also verify this by showing that
the matrix is not positive semidefinite. For example, the determinant is −2. Also one of
the eigenvalues is negative (λ1 = −0.8056).
Alternatively, we can directly show that this matrix does not satisfy the definition of
positive semidefiniteness by

  
  1 1 2 2
2 0 −1 1 2 3
   0  = −1 < 0.
2 3 3 −1

5. Gaussian random vector. Given a Gaussian random vector X ∼ N (µ, Σ), where µ = (1 5 2)T
and  
1 1 0
Σ = 1 4 0 .
0 0 9

(a) Find the pdfs of

i. X1 ,
ii. X2 + X3 ,
iii. 2X1 + X2 + X3 ,
iv. X3 given (X1 , X2 ), and
v. (X2 , X3 ) given X1 .
(b) What is P{2X1 + X2 − X3 < 0}? Express your answer using the Q function.
(c) Find the joint pdf on Y = AX, where
 
2 1 1
A= .
1 −1 1

Solution:

(a) i. The marginal pdfs of a jointly Gaussian pdf are Gaussian. Therefore X1 ∼ N (1, 1).
ii. Since X2 and X3 are independent (σ23 = 0), the variance of the sum is the sum
of the variances. Also the sum of two jointly Gaussian random variables is also
Gaussian. Therefore X2 + X3 ∼ N (7, 13).

4
 iii. Since 2X1 + X2 + X3 is a linear transformation of a Gaussian random vector,
 
  X1
2X1 + X2 + X3 = 2 1 1 X2  ,
X3
it is a Gaussian random vector with mean and variance
    
 1 1 1 0 2
µ = 2 1 1 5 = 9 and σ 2 = 2 1 1 1 4 0 1 = 21 .
  

2 0 0 9 1
Thus 2X1 + X2 + X3 ∼ N (9, 21).
iv. Since σ13 = 0, X3 and X1 are uncorrelated and hence independent since they are
jointly Gaussian; similarly, since σ23 = 0, X3 and X2 are independent. Therefore
the conditional pdf of X3 given (X1 , X2 ) is the same as the pdf of X3 , which is
N (2, 9).
v. We use the general formula for the conditional Gaussian pdf:
X2 | {X1 = x1 } ∼ N Σ21 Σ−1 −1


11 (x − µ1 ) + µ2 , Σ22 − Σ21 Σ11 Σ12

In the case of (X2 , X3 ) | X1 ,

   
  1 4 0
Σ11 = 1 , Σ21 = , Σ22 = .
0 0 9
Therefore the mean and variance of (X2 , X3 ) given X1 = x1 are
     
1  −1   5 x1 + 4
µ(X2 ,X3 )|X1 = 1 x1 − 1 + = ,
0 2 2
         
4 0 1   4 0 1 0 3 0
Σ(X2 ,X3 )|X1 = − 1 0 = − = .
0 9 0 0 9 0 0 0 9
Thus X2 and X3 are conditionally independent given X1 . The conditional densities
are X2 | {X1 = x1 } ∼ N (x1 + 4, 3) and X3 | {X1 = x} ∼ N (2, 9).
(b) Let Y = 2X1 +X2 −X3 . Similarly as part (a)iii., 2X1 +X2 −X3 is a linear transformation
of a Gaussian random vector,
 
  X1
2X1 + X2 − X3 = 2 1 −1 X2  ,
X3
it is a Gaussian random vector with mean and variance
    
1 1 1 0 2
2
   
µ = 2 1 −1 5 = 5 and σ = 2 1 −1 1 4 0
     1  = 21 .
2 0 0 9 −1
Thus 2X1 + X2 − X3 ∼ N (5, 21), i.e., Y ∼ N (5, 21). Thus

(Y − 5) (0 − 5)
  
5
P{Y < 0} = P √ < √ =Q √ .
21 21 21

5
 (c) In general, AX ∼ N (AµX , AΣX AT ). For this problem,
 
  1  
2 1 1   9
µY = AµX = 5 = ,
1 −1 1 −2
2
  
  1 1 0 2 1  
T 2 1 1  21 6
ΣY = AΣX A = 1 4 0 1 −1 =
   .
1 −1 1 6 12
0 0 9 1 1
   
9 21 6
Thus Y ∼ N , .
−2 6 12
6. Gaussian Markov chain. Let X, Y, and Z be jointly Gaussian random variables with zero
mean and unit variance, i.e., E(X) = E(Y ) = E(Z) = 0 and E(X 2 ) = E(Y 2 ) = E(Z 2 ) = 1. Let
ρX,Y denote the correlation coefficient between X and Y , and let ρY,Z denote the correlation
coefficient between Y and Z. Suppose that X and Z are conditionally independent given Y .

(a) Find ρX,Z in terms of ρX,Y and ρY,Z .

(b) Find the MMSE estimate of Z given (X, Y ) and the corresponding MSE.

Solution:

(a) From the definition of ρX,Z , we have

Cov(X, Z)
ρX,Z = ,
σX σZ
where,

Cov(X, Z) = E(XZ) − E(X)E(Z) = E(XZ) − 0 = E(XZ),

σX = E(X 2 ) − E(X)2 = 1 − 0 = 1,
σY = E(Y 2 ) − E(Y )2 = 1 − 0 = 1.

Thus, ρX,Z = E(XZ).

Moreover, since X and Z are conditionally independent given Y ,

E(XZ) = E(E(XZ|Y )) = E[E(X|Y )E(Z|Y )].

Now E(X|Y ) can be easily calculated from the bivariate Gaussian conditional density
ρX,Y σX
E(X|Y ) = E(X) + (Y − E(Y )) = ρX,Y Y.
σY
Similarly, we have

E(Z|Y ) = ρY,Z Y.

6
 Therefore, combining the above,

ρX,Z = E(XZ)
= E[E(X|Y )E(Z|Y )]
= E(ρX,Y ρY,Z Y 2 )
= ρX,Y ρY,Z E(Y 2 )
= ρX,Y ρY,Z .

(b) X, Y and Z are jointly Gaussian random variables. Thus, the minimum MSE estimate
of Z given (X, Y ) is linear.
 
1 ρX,Y
Σ(X,Y )T = ,
ρX,Y 1
   
E(XZ) ρX,Z
Σ(X,Y )T Z = = ,
E(Y Z) ρY,Z
 
ΣZ(X,Y )T = ρX,Z ρY,Z .

Therefore,
 
X
Ẑ = ΣZ(X,Y )T Σ−1
(X,Y )T Y
 −1  
  1 ρX,Y X
= ρX,Z ρY,Z
ρX,Y 1 Y
  
  1 1 −ρX,Y X
= ρX,Z ρY,Z
1 − ρ2X,Y −ρX,Y 1 Y
 
1  X
0 −ρ2X,Y ρY,Z

= 2
+ ρY,Z ,
1 − ρX,Y Y

where the last equality follows from the result of (a). Thus,
 
  X
Ẑ = 0 ρY,Z = ρY,Z Y.
Y

The corresponding MSE is

MSE = Σ2Z − ΣZ(X,Y )T Σ−1 Σ T

(X,Y )T (X,Y ) Z
 −1  
  1 ρX,Y ρX,Z
= 1 − ρX,Z ρY,Z
ρX,Y 1 ρY,Z
  
  1 1 −ρX,Y ρX,Z
= 1 − ρX,Z ρY,Z
1 − ρ2X,Y −ρX,Y 1 ρY,Z
 
  ρX,Z
= 1 − 0 ρY,Z
ρY,Z
= 1 − ρ2Y,Z .

7
7. Prediction of an autoregressive process. Let X be a random vector with zero mean and
covariance matrix
α α2 · · · αn−1
 
1
 α 1 α 
 
 α2 α 1
ΣX = 


 .. .. 
 . . 
α n−1 ··· 1
for |α| < 1. X1 , X2 , . . . , Xn−1 are observed, find the best linear MSE estimate (predictor) of
Xn . Compute its MSE.

Solution: We have
· · · αn−2 αn−1
 
1
 .. .. .. .. 
ΣX =  . . . . .

αn−2 · · · 1 α 
αn−1 · · · α 1
 T
By defining Y = X1 · · · Xn−1 , we have
· · · αn−2
 
1
ΣY =  ... .. ..  ,

. . 
αn−2 · · · 1
T
ΣYX = αn−1 · · ·

α ,
ΣXY = αn−1 · · ·
 
α ,
σx 2 = 1.
Therefore,
X̂n = ΣXY Σ−1
Y Y
−1
αn−2

1 ···
 . .. ..  Y
= αn−1 · · · α  ..

. . 
αn−2 · · · 1
= hT Y (where hT = ΣXY Σ−1
Y )
= 0 · · · 0 α Y (since hT ΣY = ΣXY )
 

= αXn−1 ;

and

MSE = σx 2 − ΣXY Σ−1

Y ΣYX
= 1 − hT ΣYX
αn−1
 

0 α  ... 
 
= 1 − 0 ···


α
= 1 − α2 .

8
8. Noise cancellation. A classical problem in statistical signal processing involves estimating a
weak signal (e.g., the heart beat of a fetus) in the presence of a strong interference (the heart
beat of its mother) by making two observations; one with the weak signal present and one
without (by placing one microphone on the mother’s belly and another close to her heart).
The observations can then be combined to estimate the weak signal by “cancelling out” the
interference. The following is a simple version of this application.
Let the weak signal X be a random variable with mean µ and variance P , and the observations
be Y1 = X + Z1 (Z1 being the strong interference), and Y2 = Z1 + Z2 (Z2 is a measurement
noise), where Z1 and Z2 are zero mean with variances N1 and N2 , respectively. Assume that
X, Z1 and Z2 are uncorrelated. Find the best linear MSE estimate of X given Y1 and Y2 and
its MSE. Interprete the results.

Solution: This is a vector linear MSE problem. Since Z1 and Z2 are zero mean, µX = µY1 = µ
and µY2 = 0. We first normalize the random variables by subtracting off their means to get
X ′ = X − µ, and  
Y −µ
Y′ = 1 .
Y2
Now using the orthogonality principle we can find the best linear MSE estimate X̂ ′ of X ′ . To
do so we first find
   
P + N1 N1 P
ΣY = and ΣYX = .
N1 N1 + N2 0

Thus,

X̂ ′ = ΣTYX Σ−1
Y Y
′
 
1 N1 + N2 −N1
Y′
 
= P 0
P (N1 + N2 ) + N1 N2 −N1 P + N1
P
(N1 + N2 ) −N1 Y′ .
 
=
P (N1 + N2 ) + N1 N2
The best linear MSE estimate is X̂ = X̂ ′ + µ. Thus,
P
X̂ = ((N1 + N2 )(Y1 − µ) − N1 Y2 ) + µ
P (N1 + N2 ) + N1 N2
1
= (P ((N1 + N2 )Y1 − N1 Y2 )) + N1 N2 µ) .
P (N1 + N2 ) + N1 N2
The MSE can be calculated by
2
MSE = σX − ΣTYX Σ−1
Y ΣYX
 
P   P
=P− (N1 + N2 ) −N1
P (N1 + N2 ) + N1 N2 0
P 2 (N1 + N2 )
=P−
P (N1 + N2 ) + N1 N2
P N1 N2
= .
P (N1 + N2 ) + N1 N2

9
The equation for the MSE makes perfect sense. First, note that if N1 and N2 are held constant
but P goes to infinity, the MSE tends to NN11+N N2
2
. Next, note that if both N1 and N2 go to
2
infinity, the MSE goes to σX , i.e., the estimate becomes worthless. Finally, note that if either
N1 or N2 goes to 0, the MSE also goes to 0. This is because the estimator will then use the
measurement with zero noise variance and perfectly determine the signal X.

10
 Solutions to Additional Exercises

1. Worst noise distribution. Consider an additive noise channel Y = X+Z, where the signal X ∼
N (0, P ) and the noise Z has zero mean and variance N . Assume X and Z are independent.
Find a distribution of Z that maximizes the minimum MSE of estimating X given Y , i.e., the
distribution of the worst noise Z that has the given mean and variance. You need to justify
your answer.

Solution: The worst noise has Gaussian distribution, i.e. Z ∼ N (0, N ).

To prove this statement, we show that the MSE corresponds to any other distribution of
Z is less than or equal to the MSE of Gaussian noise, i.e. MSENonG ≤ MSEG .

We know for any noise, MMSE estimation is no worse than linear MMSE estimation, so
MSENonG ≤ LMSE. Linear MMSE estimate of X given Y is given by
Cov(X, Y ) P
X̂ = 2 (Y − E(Y )) + E(X) = Y,
σY P +N
2 Cov2 (X, Y ) P2 NP
LMSE = σX − 2 = P − = .
σY P +N P +N

Note that LMSE only depends on the second moment of X and Z. So MSE corresponds to
any distribution of Z is always upper bounded by the same LMSE, i.e. MSENonG ≤ PN+N
P
.

When Z is Gaussian and independent of X, (X, Y ) are joint Gaussian. Then MSEG is
equal to LMSE, i.e. MSEG = PN+N
P
. Hence,

NP
MSENonG ≤ = MSEG ,
P +N
which shows the Gaussian noise is the worst.

2. Jointly Gaussian random variables. Let X and Y be jointly Gaussian random variables with
pdf
1 1 2 2
fX,Y (x, y) = p e− 2 (4x /3+16y /3+8xy/3−8x−16y+16) .
π 3/4

(a) Find E(X), E(Y ), Var(X), Var(Y ), and Cov(X, Y ).

(b) Find the minimum MSE estimate of X given Y and its MSE.

Solution:

(a) We can write the joint pdf for X and Y jointly Gaussian as
 h i
exp − a (x − µX )2 + b (y − µY )2 + c (x − µX ) (y − µY )
fX,Y (x, y) = q ,
2πσX σY 1 − ρ2X,Y

11
 where
1 1 −2ρX,Y
a= 2 , b= , c= .
2(1 − ρ2X,Y )σX 2(1 − ρ2X,Y )σY2 2(1 − ρ2X,Y )σX σY

By inspection of the given fX,Y (x, y) we find that

2 8 4
a= , b= , c= ,
3 3 3
and we get three equations in three unknowns
c 1
ρX,Y = − √ = − ,
2 ab 2
2 1
σX = = 1,
2(1 − ρ2X,Y )a
1 1
σY2 = 2 = .
2(1 − ρX,Y )b 4

To find µX and µY , we solve the equations

2aµX + cµY = 4,
2bµY + cµX = 8,

and find that

µX = 2, µY = 1.

Finally
1
Cov(X, Y ) = ρX,Y σX σY = − .
4

(b) X and Y are jointly Gaussian random variables. Thus, the minimum MSE estimate of
X given Y is linear

Cov(X, Y )
E(X|Y ) = (Y − µY ) + µX = −(Y − 1) + 2 = 3 − Y.
σY2
3
MMSE = E(Var(X|Y )) = (1 − ρ2XY )σX
2
= .
4
3. Markov chain. Suppose X1 and X3 are independent given X2 . Show that

f (x1 , x2 , x3 ) = f (x1 )f (x2 |x1 )f (x3 |x2 ) = f (x3 )f (x2 |x3 )f (x1 |x2 ).

In other words, if X1 → X2 → X3 forms a Markov Chain, then so does X3 → X2 → X1 .

Solution: By definition of conditional independence,

f (x1 , x3 |x2 ) = f (x1 |x2 )f (x3 |x2 ) .

12
 Therefore, using the definition of conditional density,

f (x1 , x2 , x3 ) f (x1 , x3 |x2 )f (x2 ) f (x1 |x2 )f (x3 |x2 )

f (x3 |x1 , x2 ) = = = = f (x3 |x2 ) .
f (x1 , x2 ) f (x1 |x2 )f (x2 ) f (x1 |x2 )

We are given that X1 and X3 are independent given X2 . Then

f (x1 , x2 , x3 ) = f (x1 )f (x2 |x1 )f (x3 |x1 , x2 ) = f (x1 )f (x2 |x1 )f (x3 |x2 ) ,

In this case X1 → X2 → X3 is said to form a Markov chain. Similarly,

f (x1 , x2 , x3 ) = f (x3 )f (x2 |x3 )f (x1 |x2 , x3 ) = f (x3 )f (x2 |x3 )f (x1 |x2 ),

This shows that if X1 → X2 → X3 is a Markov chain, then X3 → X2 → X1 is also a Markov

chain.

4. Proof of Property 4. In Lecture Notes #6 it was stated that conditionals of a Gaussian

random vector are Gaussian. In this problem you will prove that fact.
 
Y
is a zero-mean GRV then X|{Y = y} ∼ N ΣXY Σ−1 2 −Σ −1


If Y y, σX XY ΣY ΣYX .
X

Justify each of the following steps of the proof.

(a) Let X̂ be the best MSE linear estimate of X given Y. Then X̂ and X − X̂ are individually
zero-mean Gaussians. Find their variances.
(b) X̂ and X − X̂ are independent.
(c) Now write X = X̂ + (X − X̂). If Y = y then X = ΣXY Σ−1
Y y + (X − X̂).
(d) Now complete the proof.

Remark: This proof can be extended to vector X.

Solution:

(a) Let X̂ be the best MSE linear estimate of X given Y. In the MSE vector case section of
Lecture Notes #6 it was shown that X̂ and X − X̂ are individually zero-mean Gaussian
random variables with variances ΣXY Σ−1 2 −1
Y ΣYX and σX − ΣXY ΣY ΣYX , respectively.
(b) The random variables X̂ and X − X̂ are jointly Gaussian since they are obtained by
a linear transformation of the GRV [ Y X ]T . By orthogonality, X̂ and X − X̂ are
uncorrelated, so they are also independent. By the same reasoning, X − X̂ and Y are
independent.
(c) Now write X = X̂ + (X − X̂). Then given Y = y

X = ΣXY Σ−1
Y y + (X − X̂) ,

since X − X̂ is independent of Y.
(d) Thus X | {Y = y} is Gaussian with mean ΣXY Σ−1 2 −1
Y y and variance σX − ΣXY ΣY ΣYX .

13
5. Additive nonwhite Gaussian noise channel. Let Yi = X + Zi for i = 1, 2, . . . , n be n ob-
servations of a signal X ∼ N(0, P ). The additive noise random variables Z1 , Z2 , . . . , Zn are
zero mean jointly Gaussian random variables that are independent of X and have correlation
E(Zi Zj ) = N · 2−|i−j| for 1 ≤ i, j ≤ n.

(a) Find the best MSE estimate of X given Y1 , Y2 , . . . , Yn .

(b) Find the MSE of the estimate in part (a).

Hint: the coefficients for the best estimate are of the form hT = [ a b b · · · b b a ].

Solution:

(a) The best estimate of X is of the form

n
X
X̂ = hi Yi .
i=1

We apply the orthogonality condition E(XYj ) = E(X̂Yj ) for 1 ≤ j ≤ n:

n
X
P = hi E(Yi Yj )
i=1
Xn
= hi E((X + Zi )(X + Zj ))
i=1
n
X
= hi (P + N · 2−|i−j| ) .
i=1

There are n equations with n unknowns:

P + N/2n−2 P + N/2n−1
    
P P +N P + N/2 ··· h1
P   P + N/2 P +N ··· P + N/2n−3 P + N/2n−2 
  h2 
 
 ..   . .. .. ..   .. 
  
.= .. ..  . .
. . . .
P  P + N/2n−2 P + N/2n−3
    
··· P +N P + N/2  hn−1 
P P + N/2n−1 P + N/2n−2 ··· P + N/2 P +N hn

By the hint, there are only 2 degrees of freedom given, a and b. Solving this equation
using the first 2 rows of the matrix, we obtain
   
h1 2
 h2  1
 .. 
  P  .. 
 
 . =  .
  3N + (n + 2)P  . 
hn−1  1
hn 2

14
(b) The minimum mean square error is

MSE = E(X − X̂)X

X n
=P −P hi Yi
i=1
 
(n + 2)P
=P 1−
3N + (n + 2)P
3P N
= .
3N + (n + 2)P

15