Macroeconomics II
Resursive deterministic models
Prof. McCandless
June 14, 2022
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 1 / 37
Optimal control
Mathematics developed to control the movement of some dynamic
system
Initially to control physical systems, rockets, machines, quantum
mechanics
then applied to policy development for social systems
Soviet version (1950s): variational methods: solve a dynamic problem
US version (1950s): Bellmans equations: solving recursive systems
developed as method to control long range rockets
today: deterministic recursive systems
next class: variational methods and stochastic recursive systems
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 2 / 37
Recursive deterministic models
What is a recursive problem
Nature of the problem is the same independent of the period
1 Same maximization problem
2 Same budget constraints
3 Initial values can be di¤erent since they can change over time
Example
1 A household maximizes its discounted utility stream subject to a
budget constraint
2 If each period the utility function is the same
3 The form of the budget constraints are the same
The values in the budget constraint can change
Wealth can be di¤erent in di¤erent periods
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 3 / 37
States and controls
Three types of variables
1 State variables (those that are given at the beginning of a period)
stochastic variables chosen by "nature"
predetermined variables, whose values were chosen in pervious periods
2 Control variables (those that can be chosen inside a period)
3 Other (jump) variables (other variables of interest)
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 4 / 37
States variables are those predetermined at the beginning of a period
1 Capital in a Solow growth model (came from previous period)
2 Shock to technology (determined by nature)
3 Money stock carried over from previous period
Control variables are those chosen to maximize some objective
function
1 Investment in the period
2 Labor supplied in the period
3 Consumption in the period
4 Capital to be carried over to the next period
5 Money holding to be carried over to the next period
Other variables: those determined by the states and the choices for
the values of the controls
1 Output is determined by capital (normally a state) and by labor supply
(normally a control)
2 If output is determined and investment is a control, consumption is
determimed from budget constraints
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 5 / 37
Policy Function
The solution we look for is called a
A policy function gives
The optimizing values for the time t Controls
As a function of the values of the time t States
A policy function tells what to do based on what is happening or has
happened
was developed in the 1950’s
separately in the US and in the Soviet Union
countries needed controls for their rockets
In a recursive problem the policy function will be the same each
period
riding a bicycle is one such optimal control problem
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 6 / 37
In…nite horizon problem with states and controls
Robinson Crusoe want to maximize the in…nite horizon discounted
utility
∞
max ∑ βi u (ct +i ),
i =0
subject to the budget restrictions,
yt = f (kt ) = ct + it .
kt +1 = (1 δ)kt + it ,
kt is the predetermined state variable (no stochastic variables)
possible choices of controls
1 kt +1 (choosing kt +1 determines the required it and, from the second
budget constraint, ct
2 ct (choosing ct determines it and this determines kt +1
3 it (choosing it determines ct from one budget constraint and kt +1
from the other
As the problem is written, ct is the control
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 7 / 37
Writing the problem with capital as the control
Use the budget constraint to write
ct = f (kt ) + (1 δ)kt kt +1
Substitute this into the utility function to remove consumption and
investment to get
∞
max ∑ βi u [f (kt +i ) + (1 δ)kt +i kt +i +1 )]
i =0
In this format, kt +i +1 is a control in period t + i and will become the
state in period t + i + 1
kt +i is the state in period t + i
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 8 / 37
The value function (explained for a solow type model)
De…nition
For given values of the state variables at time t, the value function gives
the value of the discounted objective function when that objective function
has been maximized.
The value function is a function of the current state variables
The sum of the discounted objective function is being optimized
Example:
∞
V (kt ) = max
∞ ∑ βi u (f (kt +i )
f k s g s =t +1 i = 0
kt +1 +i + (1 δ)kt +i )
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 9 / 37
The value function
Let kt be the state variable.
V (kt ) is the value of
∞
∑ βi u (f (kt +i ) kt +1 +i + (1 δ)kt +i )
i =0
When the sequence of fks gs∞=t +1 has been chosen to maximize it
V (kt ) is a function of the current state variables and its value
changes when the value of the state variables change (in this case, k0 )
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 10 / 37
Recursive problems
Robinson Crusoe’s time t problem
∞
V (kt ) = max
∞ ∑ βi u (f (kt +i )
f k s g i =0 i = 0
kt +1 +i + (1 δ)kt +i )
is recursive
In time t + 1, Robinson Crusoe solves
∞
V (kt +1 ) = max
∞ ∑ βi u (f (kt +1+i )
f k s g i =0 i = 0
kt +2 +i + (1 δ)kt +1 +i )
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 11 / 37
Decomposing the time t problem
The time t problem
∞
V (kt ) = max
∞ ∑ βt u (f (kt +i )
f k s g i =0 i = 0
kt +1 +i + (1 δ)kt +i )
can be broken into two components and written as
V (kt ) = max [u (f (kt ) kt +1 + (1 δ)kt )
k t +1
#
∞
+ β max
∞ ∑ β u (f (kt +1+i )
f k s g i =0 i = 0
i
kt +2 +i + (1 δ)kt +1 +i )
or, substituting for the second line what it equals, as
V (kt ) = max [u (f (kt ) kt +1 + (1 δ)kt ) + βV (kt +1 )]
k t +1
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 12 / 37
The Bellman equation
The recursive equation
V (kt ) = max [u (f (kt ) kt +1 + (1 δ)kt ) + βV (kt +1 )]
k t +1
is called a Bellman equation
It is recursive because value of the function V (kt ) dependes on the
value of the same function V ( ) but evaluated at kt +1
It is a one period problem
One only chooses the value of kt +1
Notice that the entire future utility is captured in V (kt +1 )
Choice of kt +1 will change the value of V (kt +1 )
Lots of systems work this way
riding a bicycle, driving a car, ‡ying a glider
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 13 / 37
First order conditions
Take the derivative of V (kt ) with respect to kt +1
Get
0= u 0 (f (kt ) kt +1 + (1 δ)kt ) + βV 0 (kt +1 )
Problem is that we do not know the function V (kt +1 ) nor its
derivative V 0 (kt +1 )
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 14 / 37
Benveniste - Scheinkman envelope theorem conditions
Benveniste - Scheinkman give conditions under which one can …nd
V 0( )
Take derivative of V (kt ) in
V (kt ) = max [u (f (kt ) kt +1 + (1 δ)kt ) + βV (kt +1 )]
k t +1
with respect to kt
Get
V 0 (kt ) = u 0 (f (kt ) kt +1 + (1 δ)kt ) f 0 (kt ) + (1 δ)
which we can evaluate at kt +1
The result is called an envelope theorem
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 15 / 37
and envelope
Combine …rst order conditions
Get the Euler equation
u 0 ( ct )
= β f 0 (kt +1 ) + (1 δ) .
u 0 ( ct + 1 )
In a stationary state, where ct = ct +1 , this is
1
(1 δ ) = f 0 (k ).
β
This works out nicely, but let’s look at a general version to see why.
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 16 / 37
General version of problem
Let xt be the state variables and yt the controls
We want solve
∞
V (xt ) = max
∞ ∑ βs
f y s g s =t s = t
t
F (xs , ys )
subject to the set of budget constraints
xs +1 = G (xs , ys ).
The functions, F ( , ) and G ( , ), are the same for all periods
Both time t state variables and control variables can be in the
objective function and the budget constraints at time t.
This can be written as a Bellmans equation,
V (xt ) = max [F (xs , ys ) + βV (xt +1 )] ,
yt
subject to the budget constraints
xs +1 = G (xs , ys ),
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 17 / 37
General version of problem
Using the budget constraint, the Bellmans equation can be written as
V (xt ) = max [F (xt , yt ) + βV (G (xt , yt ))]
yt
We solve for a policy function of the form
yt = H (xt )
which gives the time t controls are functions of the time t state
variables
Notice that the problem is a functional equation and that the
solution is the function yt = H (xt )
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 18 / 37
General version of problem: the …rst order conditions
Taking the derivative of the Bellmans equation gives
0 = Fy (xt , yt ) + βV 0 (G (xt , yt ))Gy (xt , yt )
As before we can …nd the Benveniste-Scheinkman envelope theorem
V 0 (xt ) = Fx (xt , yt ) + βV 0 (G (xt , yt ))Gx (xt , yt )
If Gx (xt , yt ) = 0
The envelope condition is simply V 0 (xt ) = Fx (xt , yt )
The solution can be written as
0 = Fy (xt , yt ) + βFx (G (xt , yt ), yt +1 )Gy (xt , yt )
If the function, Fx (G (xt , yt ), yt +1 ), is independent of yt +1 ,
This equation can be solved directly for, yt = H (xt )
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 19 / 37
Conditions for the envelope theorem (from
Benveniste-Scheinkman)
Conditions are (for our form of the model)
xt 2 X where X is convex and with non-empty interior
yt 2 Y where Y is convex and with non-empty interior
F (xt , yt ) is continuous and di¤erentiable
G (xt , yt ) is continuous and di¤erentiable and invertible in yt
This gives enough structure so the envelope theorem holds
Newer, more general results in Milgrom and Segel
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 20 / 37
Approximation of the value function
What happens if Gx (xt , yt ) 6= 0 ?
One can approximate the value function numerically
Great contribution of Bellman
Choose some initial function V0 (xt )
Most any function will do
a good one is V0 (xt ) = c
where c is a constant (0, for example)
Find (approximately) the function V1 (xt )
V1 (xt ) = max [F (xt , yt ) + βV0 (G (xt , yt ))]
yt
over a su¢ ciently dense set of values from the domain of xt
su¢ cient for the degree of accuracy that one needs
One now has the function V1 (xt )
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 21 / 37
Approximation of the value function (continued)
Using this function V1 (xt ), …nd
V2 (xt ) = max [F (xt , yt ) + βV1 (G (xt , yt ))]
yt
over a su¢ ciently dense set of values from the domain of xt
one will need to interpolate the function V1 (xt )
when the needed G (xt , yt ) is not part of the relatively dense set of xt
linear interpolation is normally good enough
Using V2 (xt ) repreat the process
Get a sequence fVi (xt )gi∞=0
Bellman showed that fVi (xt )gi∞=0 ! V (xt )
Once you have V (xt ) …nding yt = H (xt ) is easy
Actually, one …nds a sequence fHi (xt )gi∞=0 ! H (xt )
while …nding fVi (xt )gi∞=0 ! V (xt )
Why does this work? Answer = β
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 22 / 37
Problems of dimensionality
How well do we choose to approximate the function
How many points in the domain of xt
If xt 2 R1 we can choose lots of points, M points
As dimensionality of xt grows (say to RN )
number of points needed is M N which can be very large
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 23 / 37
Comparing example economy to general problem 1: using
B-S
The objective function is
F (xt , yt ) = u (f (kt ) kt +1 + (1 δ)kt )
The budget constraint is
kt +1 = xt +1 = G (xt , yt ) = yt = kt +1
or
kt +1 = kt +1
The …rst order condition is
0 = Fy (xt , yt ) + βV 0 (G (xt , yt ))Gy (xt , yt )
= u 0 (f (kt ) kt +1 + (1 δ)kt ) + βV 0 (G (xt , yt )) 1
Because ∂kt +1 /∂kt = 0, the B-S condition is
V 0 (xt ) = Fx (xt , yt ) = u 0 (f (kt ) kt +1 + (1 δ)kt ) f 0 (kt ) + (1 δ)
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 24 / 37
Comparing example economy to general problem 1: using
B-S
Use this V 0 ( ) in the …rst order conditions to get
0 = u 0 (f (kt ) kt +1 + (1 δ)kt )
0
+ β u (f (kt +1 ) kt +2 + (1 δ)kt +1 ) f 0 (kt +1 ) + (1 δ) .
we can …nd the stationary state where kt = kt +1 = kt +2 = k as
1
f 0 (k ) = (1 δ)
β
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 25 / 37
Comparing example economy to general problem 2
We can solve the problem a di¤erent way, with ct as control
Let the objective function be
F (xt , yt ) = u (ct )
The budget constraint is
kt +1 = xt +1 = G (xt , yt ) = f (kt ) + (1 δ)kt ct
The Bellmans equation is
V (kt ) = max [u (ct ) + βV (f (kt ) + (1 δ)kt ct )]
ct
Notice that the budget constraint is already in V (kt +1 )
The derivative of the budget constraint is
∂G (xt , yt )
= f 0 (kt ) + (1 δ) 6= 0
∂xt
Can’t use B-S method
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 26 / 37
Approximation of the Value function
To approximate the value function need explicit functions for u (ct )
and f (kt )
Let f (kt ) = ktθ and u (ct ) = ln(ct )
Let δ = .1, θ = .36, and β = .98 (consistent with anual data for US)
The Bellmans equation is
h i
V (kt ) = max ln(ktθ kt +1 + (1 δ)kt ) + βV (kt +1 )
k t +1
Note: stationary state k = 5.537 (how do you …nd this?)
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 27 / 37
Approximation of the Value function
Choose V0 ( ) = 0 (a constant initial guess for value function)
Find V1 ( ) using
h i
V1 (kt ) = max ln(ktθ kt +1 + (1 δ)kt ) + βV0 (kt +1 )
k t +1
= max ln(kt.36 kt +1 + .9kt ) + .98 0
k t +1
for a relatively dense set of kt
Find V2 ( ) using
V2 (kt ) = max ln(kt.36 kt +1 + .9kt ) + .98 V1 (kt +1 )
k t +1
for a relatively dense set of kt . Use linear interpolation of V1 (kt +1 )
between known points
Repeat N times. Get approximate V (kt ) function
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 28 / 37
Computer program
Main program
global vlast beta delta theta k0 kt
hold off
hold all
%set initial conditions
vlast=zeros(1,100);
k0=0.06:0.06:6;
beta=.98;
delta=.1;
theta=.36;
numits=240;
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 29 / 37
%begin the recursive calculations
for k=1:numits
for j=1:100
kt=j*.06;
%find the maximum of the value function
ktp1=fminbnd(@valfun,0.01,6.2);
v(j)=-valfun(ktp1);
kt1(j)=ktp1;
end
if k/48==round(k/48)
%plot the steps in finding the value function
plot(k0,v)
drawnow
end
vlast=v;
end
hold off
% plot the final policy function
plot(k0,kt1)
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 30 / 37
Computer program
Subroutine (valfun.m) to calculate value function
function val=valfun(k)
global vlast beta delta theta k0 kt
%smooth out the previous value function
g=interp1(k0,vlast,k,’linear’);
%Calculate consumption with given parameters
kk=kt^theta-k+(1-delta)*kt;
if kk <= 0
%to keep values from going negative
val=-888-800*abs(kk);
else
%calculate the value of the value function at k
val=log(kk)+beta*g;
end
%change value to negative since "fminbnd" finds minimum
val=-val;
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 31 / 37
after 1 iteration
t
function 1
14
12
10
6
V(kt)
-2
-4
0 1 2 3 4 5 6
kt
1 .pdf
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 32 / 37
after ten iterations
t
function 10
14
12
10
6
V(kt)
-2
-4
0 1 2 3 4 5 6
kt
2 .pdf
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 33 / 37
after 50 iterations
t
function 50
14
12
10
6
V(kt)
-2
-4
0 1 2 3 4 5 6
kt
3 .pdf
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 34 / 37
after 100 iterations
t
function 100
14
12
10
6
V(kt)
-2
-4
0 1 2 3 4 5 6
kt
4 .pdf
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 35 / 37
after 200 iterations
t
function 200
14
12
10
6
V(kt)
-2
-4
0 1 2 3 4 5 6
kt
5 .pdf
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 36 / 37
The policy function after 200 interatioins
function 200
6
0
0 1 2 3 4 5 6
6 .pdf
Prof. McCandless () Macroeconomics II Resursive deterministic models June 14, 2022 37 / 37
