Solving Stochastic Money-in-the-Utility-Function Models

Finance and Economics Discussion Series Divisions of Research & Statistics and Monetary Affairs Federal Reserve Board, Washington, D.C. Solving Stochastic Money-in-the-Utility-Function Models Travis D. Nesmith 2005-52 NOTE: Staff working papers in the Finance and Economics Discussion Series (FEDS) are preliminary materials circulated to stimulate discussion and critical comment. The analysis and conclusions set forth are those of the authors and do not indicate concurrence by other members of the research staff or the Board of Governors. References in publications to the Finance and Economics Discussion Series (other than acknowledgement) should be cleared with the author(s) to protect the tentative character of these papers.

Solving Stochastic Money-in-the-Utility-Function Models Travis D. Nesmith Board of Governors of the Federal Reserve System 20th and C Sts. NW, Mail Stop 188 Washington, DC 20551 Abstract This paper analyzes the necessary and su¢ cient conditions for solving money-inthe-utility-function models when contemporaneous asset returns are uncertain. A unique solution to such models is shown to exist under certain measurability conditions. Stochastic Euler equations, whose existence is normally assumed in these models, are then formally derived. The regularity conditions are weak, and economically innocuous. The results apply to the broad range of discrete-time monetary and (cid:133)nancial models that are special cases of the model used in this paper. The method is also applicable to other dynamic models that incorporate contemporaneous uncertainty. Key words: money, asset pricing, dynamic programming, stochastic modelling, uncertainty JEL classi(cid:133)cation: C61, C62, D81, D84, E40, G12 Acknowledgements I wish to thank Richard Anderson, William Barnett, Erwin Diewert, Barry Jones, Je⁄rey Marquardt, David Mills, and Heinz Sch(cid:228)ttler for helpful discussions and comments. The views presented are solely my own and do not necessarily represent those of the Federal Reserve Board or its sta⁄. Email address: travis.d.nesmith@frb.gov (Travis D. Nesmith). Preprint submitted to FEDS 26 October 2005

1 Introduction Many monetary and (cid:133)nancial models(cid:151)ranging from cash-in-advance to capital asset pricing models(cid:151)are special cases of stochastic money-in-the-utilityfunction (MIUF) models.1 Stochastic decision problems that include monetary or (cid:133)nancial assets have been used to address a variety of topics including asset pricing [2, 15, 29, 30, 41, 51], price dynamics [23, 32, 40, 42, 43], intertemporal substitution [26], money demand [21, 33], currency substitution andexchangerates[6,34],optimalmonetarypolicy[16,17,18,19,20,25],and monetaryaggregation[3,4,5,46].Ineachcase,theusefulnessofthemodeldepends on the derivation of the stochastic Euler equations that characterize the model(cid:146)s solution. Deriving stochastic Euler equations is not straightforward in stochastic settings. The di¢ culty of the derivation depends crucially on the speci(cid:133)cation of uncertainty. However, uncertainty is rarely explicitly modeled inthisliterature.Instead,stochasticEulerequationsarejustassumedtoexist. Consequently, the validity and applicability of these models(cid:146)results is di¢ cult to ascertain. Thespeci(cid:133)cmodelusedinthispaperisbasedonBarnett,Liu,andJensen(cid:146)s[3] discrete-time model.2 The stochastic Euler equations derived from this model depend on a trade-o⁄ between an asset(cid:146)s rate of return, risk, and liquidity, instead of depending on just the trade-o⁄ between return and liquidity as is the norm in most of the monetary literature. The model also generalizes muchofthevoluminousasset-pricingliteraturein(cid:133)nance,whereonlythetwodimensional trade-o⁄ between risk and return is considered.3 In particular, the consumption capital asset pricing model (CAPM), which ignores liquidity, isaspecialcaseofthemodel[3].4 StochasticMIUFmodels,therefore,havethe capacity to integrate the monetary and (cid:133)nancial approaches to asset pricing. As the model is recursive, dynamic programming (DP) would seem to be the natural candidate for a solution method.5 DP is especially appealing for stochasticproblemswherestatesofthedynamicsystemareuncertain,because theDPsolutiondeterminesoptimalcontrolfunctionsde(cid:133)nedforalladmissible states. But, for a stochastic model, care must be taken to ensure that DP is feasible. Bertsekas and Shreve [10, Chapter 1] and Stokey and Lucas [49, Chapter 9] discuss the requirements for implementing stochastic DP in more 1 Deterministic MIUF models, cash-in-advance models, and other transaction cost models of money are functionally equivalent [28]. 2 Earlierversionsofthismodel,bothdeterministicandstochastic,areinthepapers collected in [4]. 3 An exception is [2], who use liquidity to explain [45](cid:146)s equity premium puzzle. 4 See [37] for a textbook treatment of CAPM and a roadmap for the asset-pricing literature. 5 The seminal paper is [7]. Reference texts for DP include [9, 39, 47, 49]. 2

detail. Probably, the simplest method to ensure that a solution to a stochastic problem can be found using DP is to specify the timing of the uncertainty. If all current state variables are known with perfect certainty, then further modeling of uncertainty is unnecessary. Although the timing restriction makes DP feasible, contemporaneous certainty is not always a reasonable assumption. Contemporaneous certainty is a particularly restrictive assumption for models containing assets; it implies that all asset returns are risk-free. Other simple assumptions that support stochastic DP similarly impose strong restrictionsontheapplicabilityof theresults. Forexample, theobjectivefunction or the stochastic processes can be restricted so that certainty equivalence holds. This requires either the strong assumption that the objective function is quadratic, or the strong assumption that all stochastic processes are i.i.d. or (conditionally) Gaussian. 6 DP is also feasible if the underlying probability space is (cid:133)nite or countable. Assuming countability is stronger than the norm in economics and (cid:133)nance, however, and does not meet Aliprantis and Border(cid:146)s [1] dictum that (cid:147)[t]he study of (cid:133)nancial markets requires models that are both stochastic and dynamic, so there is a double imperative for in(cid:133)nite dimensional models.(cid:148) This paper takes a di⁄erent tack. Following Bertsekas and Shreve [10], a rich in(cid:133)nite-dimensional model of uncertainty is developed that is similar to the measure-theoretic approach in [49]. Within this framework, the DP algorithm can be implemented for the stochastic problem and results can be obtained that are nearly as strong as those available for deterministic problems. In particular, the existence of a unique optimum that satis(cid:133)es the principal of optimality can be shown. I further prove that the solution inherits di⁄erentiability.Theseresultsimplythattheoptimumcanbecharacterizedbystochastic Euler equations. The main bene(cid:133)t of this approach is that applicability of the results is not restricted by strong assumptions on uncertainty. The regularity conditions(cid:151) certain measurability requirements(cid:151)are the least restrictive available, and are economically innocuous. As a result, the validity of many previous results, which depended on the existence of stochastic Euler equations, is broadly established. Although the method is speci(cid:133)cally developed for stochastic MIUF models due to the central importance of risk in (cid:133)nancial models, it should be clear that the method applies to other dynamic economic models that incorporate contemporaneous uncertainty, such as models with search costs or information restrictions. The organization of this paper is as follows. Section 2 presents the dynamic 6 This approach is commonly used in the (cid:133)nance literature [e.g. 16, 21] 3

decision problem. Section 3 lays out the measure-theoretic apparatus used to model expectations. Section 4 discusses measurable selection, which is necessary for the DP results. Section 5 reviews the stochastic DP approach to solving the decision problem. Conditions guaranteeing the existence of an optimalplanthatsatis(cid:133)estheprincipalofoptimalityaredeveloped.Theoptimal plan is shown to be stationary, non-random, and (semi-) Markov. Section 6 proves that di⁄erentiability of the solution follows from di⁄erentiability of the utility function. This result combined with the results in the previous section formally supports the derivation of the stochastic Euler equations. The last section provides a short conclusion. 2 Household Decision Problem The model is based on the in(cid:133)nite-horizon stochastic household decision problem in [3]. The model is very general in that preferences are de(cid:133)ned over an arbitrary ((cid:133)nite) number of assets and goods, but the form of the utility functionisnotspeci(cid:133)ed.Resultsprovenforthismodel,willholdformorerestrictive stochastic MIUF models. Similar results also apply to (cid:133)nite horizon versions, except that the optimal policy would be time-varying. The (representative) consumer attempts to maximize utility over an in(cid:133)nite horizon in discrete time. De(cid:133)ne C to be the household(cid:146)s survival set: a subset of the n-dimensional nonnegative Euclidean orthant. Let Y = R k+1 C where (cid:2) Rrepresentstherealnumbers.Thecompleteconsumptionspaceistheproduct space Y Y ::: (cid:2) (cid:2) Let the consumption possibility set for any period s t;t+1;:::; , S(s), 2 f 1g be de(cid:133)ned as follows: n k p c (1+(cid:26) )p a p a S(s) = 8 >>< (a s ;A s ;c s ) 2 Y (cid:12) (cid:12) (cid:12) (cid:12) j + P =1 (1 j + s j R s s (cid:20) (cid:0) 1 ) i P = p 1 (cid:3)s h (cid:0) 1 A s (cid:0) 1 i; (cid:0) s (cid:0) p 1 (cid:3)s A (cid:3)s s (cid:0) + 1 i I ;s s (cid:0) 1 (cid:0) (cid:3)s is i 9 >>= >>: (cid:12) (cid:12) >>; (1) where, for each period s, a = (a ;:::;a ) is a k-dimensional vector of s 1s ks planned real asset balances where each element a has a nominal holdingis period yield of (cid:26) , A is planned holdings of the benchmark asset which has is s an expected nominal holding period yield of R , c = (c ;:::;c ) is a ns s 1s ns dimensional vector of planned real consumption of non-durable goods and services where each element c has a price of p , I is the nominal value of is is s income from all other sources, which is nonnegative, and p is a true cost-of- (cid:3)s living index de(cid:133)ned as a function over some non-empty subset of the p .7 is 7 The assumption that a true cost-of-living index exists is trivial, because, the 4

Prices, including the aggregate price, p , and rates of return are stochastic (cid:3)s processes. Note that the construction of Y allows for short-selling. For the stochastic processes, the information set must be speci(cid:133)ed. It is assumed that current prices and the benchmark rate of return are known at the beginning of each period and current interest on all other assets is realized at the end of each period. More speci(cid:133)cally, for all i and s, p , p , R (and R ) is (cid:3)s s s 1 (cid:0) and (cid:26) are known at the beginning of period s, while (cid:26) is not know until i;s 1 is the end(cid:0)of period s. Since their returns are unknown in the current period, the assets, a , are risky. Despite the uncertainty, the constraint contains only s known variables in the current period, so the consumer can satisfy (1) with certainty. The consumer is assumed to maximize a utility function over the complete consumption space, including all assets except for the benchmark asset. Consequently,thesetupresemblesastandardmoney-in-the-utility-functionmodel. In the stochastic decision problem, however, the assets can be either monetary or (cid:133)nancial assets. Utility is assumed to be intertemporally additive; a standard assumption in expected utility models. In addition, although it is not necessary, preferences are assumed to be independent of time and depend on the distance from t only through a constant, subjective, rate of time preference, so that s t 1 1 (cid:0) U (a ;c ;a ;c ;:::) = u(a ;c ) (2) t t t+1 t+1 s s 1+(cid:24) ! s=t X where 0 < (cid:24) < is the subjective rate of time preference. The period utility 1 function u( ) inherits regularity conditions from the total utility function U ( ): The consumer, therefore, solves, at time t, s t 1 1 (cid:0) sup u(a ;c )+E u(a ;c ) (3) t t t s s 8 2 1+(cid:24) ! 39 < s= X t+1 = 4 5 subject to (a ;c ;A: ) S(s) for all s t;t+1;:::; and t;he transversals s s 2 2 f 1g ity condition s t 1 (cid:0) lim E A = 0 (4) t s s " 1+(cid:24) ! # !1 where the operator E [ ] denote expectations formed on the basis of the infort mation available at time t. The transversality condition, which is standard in limiting case is a singleton so that p = p , where p denotes the price of a (cid:3)s j (cid:3) s j (cid:3) s numØraire good or service. Note that the previous literature denoted the vector of goods and services by x . t 5

in(cid:133)nite horizon decisions, rules out unbounded borrowing at the benchmark rate of return. The necessity of transversality conditions in stochastic problems is shown in [35, 36]. This condition implies that the constraint set is bounded. It might appear that the problem given by (3) is deterministic in the current period. If this was actually the case, the model can already be solved by DP. But, allowing asset returns to be risky introduces contemporaneous uncertainty. As an alternative, contemporaneous uncertainty for goods could be speci(cid:133)ed, for example by assuming search costs. However, assuming that assets are exposed to risk is a more natural method of introducing such uncertainty. Explicitly de(cid:133)ning how expectations are formed, is the subject of the next section. 3 Expectations The decision problem de(cid:133)ned in equation (3) is not fully speci(cid:133)ed, as the expectations operator E [ ] is not formally de(cid:133)ned. Without some further struct ture, neither stochastic DP nor other solution techniques can be applied to the problem. Since the underlying probabilityspace is assumedto be uncountable, the expectation operator becomes an integral against some probability measure. A di¢ culty with the measure theory approach is that integration against a measure is not well-de(cid:133)ned for all functions.8 Therefore, the measurability of functions will be central to the ability to derive a stochastic DP solution. Furthermore, the measurable space cannot be arbitrary. The following standard de(cid:133)nition will, therefore, be used repeatedly: De(cid:133)nition 3.1 (Measurable function) Let X and Y be topological spaces and let F be any (cid:27)-algebra on X and let B be the Borel (cid:27)-algebra on Y. A X Y function f: X Y is F-measurable if f 1(B) F B B : (cid:0) X Y ! 2 8 2 Following [49], the disturbance spaces are assumed to be Borel spaces. This implies that the decision and constraint spaces for the in(cid:133)nite-horizon problem are also Borel spaces, because Euclidean spaces, Borel subsets of Borel spaces, and countable Cartesian products of Borel spaces are all Borel spaces. 8 The outer integral, which is well-de(cid:133)ned for any function, could be used. Not only is there no unique de(cid:133)nition for the outer integral, but it is not a linear operator. Consequently, expectations would not be additive. Recursive methods, such as DP, require additivity so that the overall problem can be broken into smaller subproblems. 6

The formulation of the constraint set in [3] assumed compactness. Compactness along with the additional assumptions, not made by [3], that are needed for semicontinuous problems [10, pg. 208-210], or upper hemi-continuity [49, pg. 56], would allow the problem to be solved using only Borel measurability. As in [10], instead of assuming compactness and the necessary ancillary assumptions, a solution is sought within a richer set of functions: universallymeasurable functions. Relaxing the measurability restrictions, which expands the class of possible solutions, is more appealing than imposing strong compactness restrictions. Section 5.3 discusses these trade-o⁄s further. Transition functions are commonly used to incorporate stochastic shocks into afunctionalequation[see49,pg.212].Thecurrenttreatmentissimilar,except insteadofbeginningwithtransitionfunctions,stochastickernelsareemployed. De(cid:133)nition 3.2 (Stochastic Kernel) Let X and Y be Borel spaces with B Y denoting the Borel (cid:27)-algebra. Let P(Y) denote the space of probability measures on (Y;B ). A stochastic kernel, q(dy x), on Y given X is a collection Y j of probability measures in P(Y) parameterized by x X. If F is a (cid:27)-algebra on 2 X, and (cid:13) 1(B ) F where (cid:13): X P(Y) is de(cid:133)ned by (cid:13)(x) = q(dy x), (cid:0) P(Y) (cid:26) ! j then q(dy x) is F-measurable. If (cid:13) is continuous, q(dy x) is said to be j j continuous. A stochastic kernel is a special case of a regular conditional probability for a Markov process, de(cid:133)ned in [48, De(cid:133)nition 6, pg. 226]. In abstract terms, if x is taken to represent the state of the system at time t and the system is Markovian, then, from [48, Theorem 3, pg. 226] and the properties of Markov processes, the conditional expectation operator applied to an element of Y can, formally be viewed as an integral against the stochastic kernel, i.e. E [f (x;y)] = f(x;y)q(dy x). As this de(cid:133)nition requires measurability of t j the function, a measurability assumption is necessary to de(cid:133)ne expectations, R evenbeforeasolutionissought.Ifafunction,de(cid:133)nedontheCartesianproduct ofX andY,isBorel-measurableandthestochastickernelisBorel-measurable, then f(x;y)q(dy x) is Borel-measurable [10, Proposition 7.29, pg. 144], i.e. j the conditional expectation is Borel-measurable. Integration de(cid:133)ned this way R operates linearly and obeys classical convergence theorems. Also, the integral is equal to the appropriate iterated integral on product spaces. These statements will be clari(cid:133)ed in the next section. Asthestatespacefortheproblemhasnotbeende(cid:133)ned,itmayappearthatthe stochastic kernel is limited in that it only represents conditional expectations for Markovian systems. In fact, non-Markovian processes can always be reformulated as Markovian by expanding the state space. In the sequel, the state space for the problem will be formulated so that the process is Markovian. 7

4 Measurable Selection Borelmeasurability,byitself,isnotadequatetoprovetheexistenceofthesolution. Bertsekas and Shreve [10] address this problem through a richer concept of measurability. Their apparatus includes upper semianalytic functions and measurability with regard to the universal (cid:27)-algebra. De(cid:133)nitions and relevant results are presented below; proofs can be found in [1, 10].9 The key relation is that the universal (cid:27)-algebra includes the analytic (cid:27)-algebra, which in turn includes the Borel (cid:27)-algebra. This implies that all Borel-measurable functions are analyticallymeasurable, andthat all analyticallymeasurablefunctions are universally measurable. Thus the move to universal measurability is, relative to the Borel model, relaxing a constraint, instead of imposing a restriction. By moving to universal-measurability, the information set is enriched and the measurability assumptions are technically relaxed. Such a relaxation does no damage to the economics behind the model. The results are standard and the proofs are omitted. Borel measurability is not adequate for DP, because the orthogonal projection of a Borel set is not necessarily Borel measurable Speci(cid:133)cally, if f: X Y (cid:2) ! R(cid:3) ,whereR(cid:3) istheextendedrealnumbers,isgivenandf (cid:3) : X R(cid:3) isde(cid:133)ned ! by f (cid:3) (x) = sup y Y f(x;y) then for each c R, de(cid:133)ne the set 2 2 x X f (x) > c = proj ( (x;y) X Y f(x;y) > c ) (5) (cid:3) X f 2 j g f 2 (cid:2) j g where proj ( ) is the projection mapping from X Y onto X. If f( ) is X (cid:2) Borel-measurable then (x;y) X Y f(x;y) > c (6) f 2 (cid:2) j g is Borel-measurable, but x X f (x) > c may not be Borel-measurable (cid:3) f 2 j g ([11], [22, pg. 328-329], and [49, pg. 253]). The DP algorithm repeatedly implements such projections, so the conditional expectation of functions like f ( ) will need to be evaluated, requiring that (cid:3) the function is measurable. This leads to the de(cid:133)nition of analytic sets, the analytic (cid:27)-algebra, and analytic measurability: De(cid:133)nition 4.1 (Analytic sets) A subset A of a Borel space X is analytic if there exists a Borel space Y and Borel subset B of X Y such that A = (cid:2) proj (B). The (cid:27)-algebra generated by the analytic sets of X is referred to as X the analytic (cid:27)-algebra, denoted by , and functions that are measurable with X A respect to it are called analytically measurable. 9 Further references include [22, 24, 38, 48, 49]. 8

Davidson [22, pg. 329] refers to analytic sets as (cid:147)nearly(cid:148)measurable because, for any measurable space and any measure, (cid:22), on that space,the analytic sets are measurable under the completion of the measure. The completion of a space with respect to a measure involves setting (cid:22)(E) = (cid:22)(A) for any set E such that A E B whenever (cid:22)(A) = (cid:22)(B). E⁄ectively, this assigns (cid:26) (cid:26) measure zero to all subsets of measure zero sets [22, pg. 39]. Analytic sets address the problem of measurable selection within a dynamic program, because, if X and Y are Borel spaces, and, if A X is analytic (cid:26) and f: X Y is Borel-measurable, then f(A) is analytic. This implies that ! if B X Y is analytic then proj (B) is also analytic. Analytic sets are the X (cid:26) (cid:2) smallest groups of sets such that the projection of a Borel set is a member of the group [1]. Analytic sets are used to de(cid:133)ne upper semianalytic functions as follows: De(cid:133)nition 4.2 Let X be a Borel space and let f: X R(cid:3) be a function. ! Then f( ) is upper semianalytic if x X f(x) > c is analytic c R. f 2 j g 8 2 The following result is key for the application of the DP algorithm: Lemma 4.3 Let X and Y be Borel spaces, and let f: X Y R(cid:3) be upper (cid:2) ! semianalytic, then f (cid:3) : X R(cid:3) de(cid:133)ned by f (cid:3) (x) = sup f(x;y) is upper ! y Y semianalytic. 2 Two important properties of upper semianalytic functions are that the sum of such functions remains upper semianalytic, and if f: X R(cid:3) is upper ! semianalytic and g : Y X is Borel measurable, then the composition f g ! (cid:14) is upper semianalytic. Most importantly, the integral of a bounded upper semianalytic function against a stochastic integral is upper semianalytic. This is stated as a lemma: Lemma 4.4 Let X and Y be Borel spaces and let f: X Y R(cid:3) be a upper (cid:2) ! semianalytic function either bounded above or bounded below. Let q(dy x) be j a Borel- measurable stochastic kernel on Y given X. Then g: X R(cid:3) de(cid:133)ned ! by g(x) = f(x;y)q(dy x) is upper semianalytic. j Y R Semianalytic functions have one relevant limitation. If two functions are analytically measurable, their composition is not necessarily analytically measurable. This di¢ culty can be overcome moving to the richer universally measurable (cid:27)-algebra:10 De(cid:133)nition 4.5 (Universal (cid:27)-algebra) Let X be a Borel space, P(X) be the set of probability measures on X, and let B ((cid:22)) denote the completion of B X X 10The slightly tighter, but less intutive, (cid:27)-algebra of limit measureable sets would be su¢ cient. Again, moving to the larger class does not impose any restrictions. 9

with respect to the probability measure (cid:22) P(X). The universal (cid:27)-algebra U X 2 is de(cid:133)ned by U = B ((cid:22)): (7) X X (cid:22) P(X) 2 If A U , A is called universally\measurable, and functions that are measur- X 2 able with respect to U are called universally measurable. X The universally-measurable (cid:27)-algebra is the completion of the Borel (cid:27)-algebra with respect to every Borel measure. Consequently, it does not depend on any speci(cid:133)c Borel measure. Note that every Borel subset of a Borel space X is also an analytic subset of X, which implies that the (cid:27)-algebra generated by the analytic sets is larger than the Borel (cid:27)-algebra. The fact that analytic sets are measurable under the completion of any measure implies that they are universally-measurable, so B U : x X x (cid:18) A (cid:18) Universal measurabilityis the last type of measurabilitythat will be neededto implement stochastic DP as universally measurable stochastic kernels will be used in the DP recursion. Of course, if a stochastic kernel is Borel-measurable, it is universally measurable. Integration against a universally measurable stochastic kernel operates linearly, obeys classical convergence theorems, and iterates on product spaces, as shown by the following theorem: Theorem 4.6 Let X ;X ;::: be a sequence of Borel spaces, Y = X 1 2 n 1 (cid:2)(cid:1)(cid:1)(cid:1)(cid:2) X , and Y = X X . Let (cid:22) P(X ) be given and, for n = 1;2;:::, n 1 2 1 (cid:2) (cid:2) (cid:1)(cid:1)(cid:1) 2 let q (dx y ) be a universally measurable stochastic kernel on X given n n+1 n n+1 j Y . Then for n = 2;3;:::, there exist unique probability measures r P(Y ) n n n 2 such that X B ;:::;X B 8 1 2 X1 n 2 Xn r (X X X ) = q (X x ;:::;x ) n 1 2 n n 1 n 1 n 1 \ \(cid:1)(cid:1)(cid:1)\ (cid:1)(cid:1)(cid:1) (cid:0) j (cid:0) Z Z Z X X X 1 2 n q (dx x ;:::;x )::: q (dx x )(cid:22)(dx ) (8) n 2 n 1 1 n 2 1 2 1 1 (cid:2) (cid:0) (cid:0) j (cid:0) (cid:2) j If f: Y n R(cid:3) is universally measurable, and the integral is well-de(cid:133)ned,11 ! then fdr = f(x ;:::;x )q (X x ;:::;x ) n 1 n n 1 n 1 n 1 (cid:1)(cid:1)(cid:1) (cid:0) j (cid:0) Z Z Z Z Yn X1X2 Xn q (dx x ;:::;x ) q (dx x )(cid:22)(dx ): (9) n 2 n 1 1 n 2 1 2 1 1 (cid:2) (cid:0) (cid:0) j (cid:0) (cid:2)(cid:1)(cid:1)(cid:1)(cid:2) j There further exists a unique probability measure r P(Y) such that for each 2 n the marginal of r on Y is r . n n The formal de(cid:133)nition of the conditional expectations operator is, therefore, 11The integral is well-de(cid:133)ned if either the positive or negative parts of the function are (cid:133)nite. Such a function will be called integrable. 10

the integral of the function versus r or r. This de(cid:133)nition allows universally n measurable selection: Theorem 4.7 (Measurable Selection) Let X and Y be Borel spaces, D 2 X Y be an analytic set such that D x = y (x;y) D , and f: D R(cid:3) be (cid:2) f j 2 g ! an upper semianalytic function. De(cid:133)ne f (cid:3) : proj X (D) R(cid:3) by ! f (x) = sup f(x;y): (10) (cid:3) y Dx 2 Then the set I = x proj (D) for some y D , f(x;y ) = f (x) is uni- X x x x (cid:3) f 2 j 2 g versally measurable, and for every (cid:15) > 0, there exists a universally measurable function (cid:30): proj (D) Y such that Gr((cid:30)) D and for all x proj (D) X X ! (cid:26) 2 either f[x;(cid:30)(x)] = f (x) if x I (11) (cid:3) 2 or f (x) (cid:15) if x = I and f (x) < (cid:3) (cid:3) f[x;(cid:30)(x)] (cid:0) 2 1 (12) (cid:21) 81=(cid:15) if x = I and f (x) = (cid:3) < 2 1 : The selector obtained in Theorem 4.7, f[x;(cid:30)(x)] is universally measurable. If the function (cid:30)( ) is restricted to be analytically measurable, then I is empty and (12) holds. In this case, the selector is not necessarily universally measurable. For Borel-measurable functions (cid:30)( ), the analytic result does not hold uniformly in x. The strong result given by (11) is only available for universally measurable functions. Similarly, strong results are available for Borel-measurable functions if signi(cid:133)cantly stronger regularity assumptions are maintained.12 The weaker regularity conditions are appealing, as they allow a solution without imposing restrictions on the economics of the problem. 5 Stochastic Dynamic Programming There are several issues with simply applying DP to the stochastic MIUF problem. First, the functional form of the utility function is not speci(cid:133)ed, as doing so would restrict the applicability of the results. Second, as previously discussed, there are a number of technical di¢ culties in applying DP methods in a general stochastic setting. Section 3(cid:146)s measure theory was developed to overcome these di¢ culties.13 12In particular, D must be assumed to be compact, and f must be upper semicontinuous. This is the approach taken in [49]. 13For further references to DP in measure spaces, see [11, 12, 13, 27, 31, 50]. 11

Three tasks are repeatedly performed in the DP recursion. First, a conditional expectation is evaluated. Second, the supremum of an extended real-valued function in two (vector-valued) variables, the state and the control, is found over the set of admissible control values. Finally, a selector which maps each state to a control that (nearly) achieves the supremum in the second step is chosen. Each of these steps involves mathematical challenges in the stochastic context.Anespeciallyimportantconcernismakingsurethatthemeasurability assumptions are not destroyed by any of the three steps. The (cid:133)rst and second steps require that the expectation operator can be iterated and interchanged with the supremum operator. As shown in Section 4, these requirements are met by the integral de(cid:133)nition of the expectations operator, for either the Borel- or universally-measurable speci(cid:133)cations. Step two encountersaproblemwithmeasurability,becauseoftheissuewithprojections of Borel sets also discussed in the previous section. Analytic-measurability is su¢ cient to address this particular problem, but such measurability is not necessarily preserved by the composition of two functions. By using semianalytic functions and assuming universal-measurability, not only is this problem solved, but measurable selection is also possible under mild regularity conditions. Consequently, all three steps of the DP algorithm can be implemented forthestochasticproblem.Consequently,theexistenceofanoptimalornearly optimal program can be shown and the principal of optimality holds for the (near) optimal value function. ToshowthattheseresultsareapplicabletostochasticMIUFmodels,theproblem laid out in (3) is mapped into Bertsekas and Shreve(cid:146)s general stochastic DP model. One adjustment is necessary as Bertsekas and Shreve de(cid:133)ne lower semianalyticfunctionsratherthanuppersemianalyticfunctions,becausetheir exposition addresses the (cid:133)nding the in(cid:133)mum of a function. This di⁄erence requires careful adjustment of their regularity conditions. 5.1 General Framework Following Bertsekas and Shreve [10, pg. 188-189], the general in(cid:133)nite horizon model is de(cid:133)ned as follows: De(cid:133)nition 5.1 (Stochastic Optimal Control Model) A in(cid:133)nite horizon stochastic optimal control model is an eight-tuple (X;Y;S;Z;q;f;(cid:12);g) where: X State space: a non-empty Borel space; Y Control space: a non-empty Borel space; S Control constraint: a function from X to the set of non-empty subsets of Y. The set (cid:0) = (x;y) x X;y S(x) is assumed to be analytic in X Y; f j 2 2 g (cid:2) Z Disturbance space: a non-empty Borel space; 12

q(dz x;y) Disturbancekernel:aBorel-measurablestochastickernelonZ given j X Y; (cid:2) f System function: a Borel-measurable function from X Y Z to X; (cid:2) (cid:2) (cid:12) Discount factor: a positive real number; and g One-stage value function: a upper semianalytic function from (cid:0) to R(cid:3) . The (cid:133)ltered probability space used in the stochastic optimal control model consists four elements: 1) the (Cartesian) product of the disturbance space with the in(cid:133)nite product of the state and control spaces, Z ((cid:5) (X Y) ); (cid:2) 1i=t (cid:2) i 2)a(cid:27)-algebra(generallyuniversallymeasurable)onthatproductspace;3)the probability measure de(cid:133)ned in Theorem 4.6; and, 4) the (cid:133)ltration de(cid:133)ned by the restriction to the product of the state and control spaces that have already occurred, (cid:5)s 1(X Y ) X where it is understood that each subscripted i=(cid:0)t i i s (cid:2) (cid:2) space is a(cid:16)copy of the resp(cid:17)ective space. Establishing the existence of a solution to a stochastic optimal control model means establishing the existence of an optimal policy for the problem. Specifically, the following de(cid:133)nitions from [10] are used: De(cid:133)nition 5.2 (Policy) A policy is a sequence (cid:30) = ((cid:30) ;(cid:30) ;:::) such that, t t+1 for each s t;t+1;::: , 2 f g (cid:30) (dy x ;y ;:::;y ;x ) (13) s s t t s 1 s j (cid:0) is a universally measurable stochastic kernel on Y, given X Y Y X (cid:2) (cid:2)(cid:1)(cid:1)(cid:1)(cid:2) (cid:2) satisfying (cid:30) (S(x ) x ;y ;:::;y ;x ) = 1; (14) s s t t s 1 s j (cid:0) for every (x ;y ;:::;y ;x ). If for every s, (cid:30) is parameterized by only x , t t s 1 s s s (cid:0) then (cid:30) is a Markov policy. Alternatively, if for every s, (cid:30) is parameterized s s by only (x ;x ), then (cid:30) is a semi-Markov policy. The set of all Markov polit s s cies, (cid:8), is contained in the set of all semi-Markov policies, (cid:8). If for each 0 s and (x ;y ;:::;y ;x ), (cid:30) (dy x ;y ;:::;y ;x ) assigns mass one to t t s 1 s s s t t s 1 s (cid:0) j (cid:0) some element of Y, (cid:30) is non-randomized. If (cid:30) is a Markov policy of the form (cid:30) = ((cid:30) ;(cid:30) ;(cid:30) ;:::), it is called stationary. t t t De(cid:133)nition 5.3 (Value Function) Suppose(cid:30)isapolicyforthein(cid:133)nitehorizon model. The (in(cid:133)nite horizon) value function corresponding to (cid:30) at x X 2 is V (x) = 1 (cid:12)kg(x ;y ) dr((cid:30);(cid:22) ) (cid:30) k k x " # Z k X =0 (15) = 1 (cid:12)k g(x ;y ) dr ((cid:30);(cid:22) ) k k k+t x k=0(cid:20) Z (cid:21) X where, for each (cid:30) (cid:8) and (cid:22) P(X), r((cid:30);(cid:22) ) is the unique probability 0 x 2 2 measure de(cid:133)ned in equation (4.6) and, for every k, the r ((cid:30);(cid:22) ) is the apk+t x 13

propriate marginal measure.14 The (in(cid:133)nite horizon) optimal value function at x X is V (x) = sup V (x). (cid:3) (cid:30) (cid:8) (cid:30) 2 2 0 Note that the optimal value function is de(cid:133)ned over semi-Markov policies; this is without loss of generality. Furthermore, Bertsekas and Shreve [10, pg. 216] show that the optimal value can be reached by only considering Markov policies. The advantage of including semi-Markov policies is that the optimum may require a randomized Markov policy, but only need a non-randomized semi-Markov policy. Finally, the Jankov-von Neumann theorem guarantees the existence of at least one non-randomized Markov policy so (cid:8) and (cid:8) are 0 non-empty. The following de(cid:133)nes optimality for policies: De(cid:133)nition 5.4 (Optimal policies) If (cid:15) > 0, the policy (cid:30) is (cid:15)-optimal if V (x) (cid:15) if V (x) < (cid:3) (cid:3) V (x) (cid:0) 1 (16) (cid:30) (cid:21) 81=(cid:15) if V (x) = (cid:3) < 1 : for every x X. The policy (cid:30) is optimal if V (x) = V (x). (cid:30) (cid:3) 2 In the next subsection, equation (3) is restated in this optimal control framework. The last subsection addresses what conditions are need to guarantee the existence of a (nearly) optimal policy. 5.2 Restating the Household Problem Embedding the household decision problem into this framework requires specifying the state and control spaces. Since the spaces in this problem are all (cid:133)nite Euclidean spaces, the state and control spaces will be Borel no matter how de(cid:133)ned. For a utility problem, it is natural to generally de(cid:133)ne (cid:145)prices(cid:146) as states and (cid:145)quantities(cid:146)as controls, but there is no unique speci(cid:133)cation required for the DP algorithm. Also, if the utility function demonstrated habit persistence, as in Barnett and Wu [5], lagged consumption variables would naturally be state variables in the current stage. De(cid:133)ne the period s states by x = ((cid:11) ; ), where (cid:11) = (a ;A ) and s s s s s 1 s 1 s (cid:0) (cid:0) denotes the vector of prices, interest rates and other income that were realized 14The interchange of the integral and the summation is justi(cid:133)ed by either the monotone or bounded convergence theorems. 14

at the beginning of period s, normalized by p , (cid:3)s p p = 1+(cid:26) (cid:3)s 1;:::; 1+(cid:26) (cid:3)s 1; s 1;s 1 p (cid:0) k;s 1 p (cid:0) (cid:0) (cid:3)s (cid:0) (cid:3)s (cid:16) (cid:17) (cid:16) (cid:17) p p p I (1+R ) (cid:3)s 1; 1;s ;:::; n;s ; s : (17) s 1 (cid:0) (cid:0) p (cid:3)s p (cid:3)s p (cid:3)s p (cid:3)s! The period s controls are de(cid:133)ned to be y = ((cid:18) ;c ) where (cid:18) = (a ;A ). The s s s s s s state space X is (2k +n+2)(cid:150)dimensional Euclidean space, and the control space Y is a subset of (k +n+1)(cid:150)dimensional Euclidean space. This notation is useful in what follows. Again note that, for s < , elements of a and A s s 1 may be negative, so that short-selling is allowed. The budget constraint can be used to eliminate one of the controls, because the constraint will hold exactly at every time-period for any optimal solution. Therefore, a redundant control has been speci(cid:133)ed and the set of admissible controls actually lie in a (k +n)-dimensional linear subspace of Y. Besides satisfying the budget constraint, the control variables need to be inside the survival set. By leaving in the redundant control, it is easier to explicitly specify this constraint. The controls set can be written as a function of only period s states and controls as k+1 k+1+n S(x) = y Y (y x )+ y 0 (18) 8 2 (cid:12) i (cid:0) i i j j (cid:0) k+n+2 (cid:20) 9 < (cid:12) X i=1 j= X k+2 = (cid:12) (cid:12) where the period:subscrip(cid:12)t s has been suppressed. The (cid:133)rst key assu;mption is Criterion 5.5 Assume that (cid:0) = (x;y) x X;y S(x) is analytic in f j 2 2 g X Y. (cid:2) The system function has a relatively simple form. It is de(cid:133)ned by x = (cid:11) ; = f (x ;y ;z ) = ((cid:18) ; +z ) (19) s+1 s+1 s+1 s s s s s s (cid:16) (cid:17) In words, the (cid:133)rst partition of the states evolves according to the simple rule (cid:11) = (cid:18) , and the second evolves as a state-dependent stochastic process, s+1 s according to = +z .15 If z was a pure white noise process, would s+1 s s s s be a random walk. The discount factor is de(cid:133)ned by (cid:12) = 1=(1 + (cid:24)) and satis(cid:133)es 0 < (cid:12) < 1. The one-stage value function g(x ;y ) is simply the period utility function s s (g(x ;y ) = g(y ) = u(a ;c )), so it is a function of only the controls.16 The s s s s s 15This would have to slightly modi(cid:133)ed to account for the model in [5] that includes habit persistence. 16Recall that there is a redundant control. 15

framework would allow g( ) to be time-varying or to depend on the states, however, this would complicate the derivation of stochastic Euler equations in Section 6. The remaining assumption, which completes the mapping of the stochastic utility problem into the DP model, is: Criterion 5.6 Assume that g(y) is an upper semianalytic function from (cid:0) to R +. 5.3 Existence of a Solution and the Principal of Optimality The existence of a solution to the household decision problem or equivalently the existence of a (nearly) optimal policy can now be proved. First, as it is easier, the optimal value function is shown to satisfy a stochastic version of Bellman(cid:146)s equation and the Principal of Optimality. The result is most cleanly stated using the following de(cid:133)nition: De(cid:133)nition 5.7 (State transition kernel) ThestatetransitionkernelonX given X Y is de(cid:133)ned by (cid:2) t(B x;y) = q( z f(x;y;z) B x;y) = q(f 1(B) x;y): (20) (cid:0) (x;y) j f j 2 g j j Thus, t(B x;y) is the probability that the state at time (s+1) is in B given j that the state at time s is x and the sth control is y. Note that t(dx x;y) 0 j inherits the measurability properties of the stochastic kernel. Then the following mapping helps to state results concisely: De(cid:133)nition 5.8 Let V : X R(cid:3) be universally measurable. De(cid:133)ne the opera- ! tor T by T(V) = sup g(y)+(cid:12) V (x) t(dx x;y) : (21) 0 0 8 j 9 y 2 S(x) < X Z = : ; Several lemma(cid:146)s characterize the optimal policies. The following lemma shows that the optimal value function for the problem satis(cid:133)es a functional recursion that is a stochastic version of Bellman(cid:146)s equation. Lemma 5.9 The optimal value function V (x) satis(cid:133)es V = T (V ) for (cid:3) (cid:3) (cid:3) every x X. 2 16

PROOF. Note that g(y) is upper semianalytic and non-negative. This implies that g(y) is lower semianalytic and non-positive. Also (cid:0) V (x) = 1 (cid:12)k( g(x ;y )) dr((cid:30);(cid:22) ) = V (x) (22) (cid:30) k+t k+t x (cid:30) " (cid:0) # (cid:0) Z k=0 X e and V (x) = inf V (x) = V (x). Then from Proposition 9.8 in [10, (cid:3) (cid:30) (cid:8) (cid:3) 2 (cid:0) pg. 225], e e V (x) = inf g(y)+(cid:12) V (x)t(dx x;y) (23) (cid:3) (cid:3) 0 0 y 2 Y (cid:26) (cid:0) Z X j (cid:27) since g(y) sateis(cid:133)es their assumption labeeled (N) on page 214. Taking the (cid:0) negative of each side implies the result as the negation can be taken inside the integral. 2 This necessity result implies that the optimal policy would be a (cid:133)xed point of themappingwhichisimplicitlyde(cid:133)nedinthelemma.Thefollowingsu¢ ciency result implies that a stochastic version of Bellman(cid:146)s principal of optimality holds for stationary policies. Lemma 5.10 (Principal of Optimality) Let(cid:30) = ((cid:30);(cid:30);:::) beastationary policy. Then the policy is optimal iff V = T V for every x X. (cid:30) (cid:30) 2 (cid:16) (cid:17) PROOF. Following the same argumentation as in the previous lemma, given the properties of g(y) the result holds for V (x) by Proposition 9.13 in [10, (cid:30) pg. 228], which implies the result. 2 e Before examining existence of optimal policies, note that the measurability assumptions already imply the existence of an (cid:15)-optimal policy. From Proposition 9.20 in [10, pg. 239], the non-negativity of the utility function is enough to assert the existence of an (cid:15)-optimal policy using similar arguments as in the previous lemmas. Lemma 5.11 For each (cid:15) > 0, there exists an (cid:15)-optimal non-randomized semi- Markov policy for the in(cid:133)nite horizon problem. If for each x X there exists a 2 policy for the in(cid:133)nite horizon problem, which is optimal at x, then there exists a semi-Markov (randomized) optimal policy. The fact that existence of any optimal policy is su¢ cient for the existence of a semi-Markov randomized optimal policy is important. The primary concern is with the (cid:133)rst period return or utility function. For the initial period, the semi- Markov (cid:15)-optimal policy is Markov as clearly (cid:30) (dy x ;x ) = (cid:30) (dy x ). If t s t t t s t j j (cid:15)-optimality is judged to be su¢ cient, then simply use g (cid:30)(cid:3) where (cid:30)(cid:3) is the t t (cid:133)rstelementoftheoptimalpolicy.Theprincipalofoptima(cid:16)lity(cid:17)wouldonlyhold 17

approximately, however. Similarly, if an optimal policy does actually exist, the randomness is not an issue as the optimal policy is non-random in the (cid:133)rst element. In that case, the principal of optimality may not hold as equation (3) is only guaranteed to hold for stationary policies. Consequently, minimal additional assumptions are useful. Before making these additional assumptions, de(cid:133)ne the DP algorithm as follows: De(cid:133)nition 5.12 (Dynamic Programming Algorithm) The algorithm is de(cid:133)ned recursively by V (x) = 0 x X (24) 0 8 2 V (x) = T (V (x)) x X; k = 0;1;::: (25) k+1 k 8 2 Proposition 9.14 of [10] implies that the algorithm converges for the problem as stated in the following lemma. Lemma 5.13 V = V (cid:3) 1 Unfortunately, the convergence is not necessarily uniform in x. Additionally, it is not possible to synthesize the optimal policy from the algorithm, as is the case for deterministic problems, because V , while universally measurable, is k not necessarily semianalytic for all k. The regularity assumptions are strengthened by imposing a mild boundedness assumption. Criterion 5.14 (Boundedness) Assume that i and s t;t + 1;::: , 8 8 2 f g > 0. Further assume that the single stage utility function contains no i;s points of global satiation. Assuming Criterion 5.14 leads to stronger results. First, under this boundedness condition, the DP algorithm converges uniformly for any initial upper semianalytic function not just zero. Furthermore, necessary and su¢ cient conditions for the existence of an optimal policy are available. Lemma 5.15 Assume Criterion 5.14 holds. Then for each (cid:15) > 0, there exists an(cid:15)-optimal non-randomizedstationaryMarkovpolicy.Ifforeachx X there 2 existsapolicyforthein(cid:133)nitehorizonproblem,whichisoptimal atx,thenthere exists an unique optimal non-randomized stationary policy. Furthermore, there is an optimal policy if and only if for each x X the supremum in 2 sup g(y)+(cid:12) V (x) t(dx x;y) (26) (cid:3) 0 0 8 j 9 y 2 S(x) < Z X = : ; 18

is achieved. PROOF. Criterion 5.14 and the Transversality condition given in (4) imply that g(y) is bounded over (cid:0) so that b such that (x;y) (cid:0), g(y) < b. 9 8 2 Since g(y) 0, obviously g(y) > b. Also recall that (cid:12) = 1=(1+(cid:24)) so that (cid:21) (cid:0) 0 < (cid:12) < 1. This implies that the problem satis(cid:133)es the assumption labeled (D) in [10, pg. 214]. The lemma follows from Proposition 9.19, Proposition 9.12 and Corollary 9.12.1 in [10, pg. 228]. 2 Combining this lemma with lemma 5.10 implies that there exists an optimal policy if and only if there exists a stationary policy such that V = T V (cid:30) (cid:30) for every x X. It is clear that the (cid:15)-optimal non-randomized statio(cid:16)nary(cid:17) 2 Markov policy is the universally measurable selector from Theorem 4.7. If universallymeasurableselectionisassumedtobepossible(i.e.thesetI de(cid:133)ned in Theorem 4.7 is the entire set D), then the supremum will be achieved. This assumption is weaker than requiring that Borel-measurable selection is possible, as in Stokey and Lucas [49]. The assumption is also weaker than the regularity conditions needed to solve semicontinuous models in [10], which haveBorel-measurableoptimalplans.Thefollowinglemma,whichfollowsfrom Proposition 9.17 of [10] supplies a su¢ cient condition for the supremum to be achieved. Lemma 5.16 Under Criterion 5.14, if there exists a nonnegative integer k such that for each x X, (cid:21) R, and k > k, the set 2 2 S (x;(cid:21)) = y S(x) g(y)+(cid:12) V (x) t(dx x;y) (cid:21) (27) k k 0 0 > 2 j j (cid:26) Z (cid:27) is compact in Y then there exists a non-randomized optimal stationary policy for the in(cid:133)nite horizon problem. This is a weaker condition than assuming that the constraint sets are compact or that (cid:0) is upper hemi-continuous. If the supremum in (26) is achieved for the initial state x X, the boundedness assumption implies that a unique t 2 stationary non-random Markov optimal plan exists. 6 Stochastic Euler Equations In this section, Euler equations for the stochastic decision are derived. The usefulness of stochastic Euler equations is discussed in [49, pg. 280-283]. Although necessary and su¢ cient conditions for the optimum to exist have been established, the stronger characterization given by Euler equations is often 19

needed and is always useful. Since the principal of optimality has been shown to hold, Bellman(cid:146)s equation can be used to derive stochastic Euler equations. Of course, the optimal value function needs to satisfy additional regularity conditions. In particular, it needs to be di⁄erentiable. In addition, it must be possible to interchange the order of integration. The interchange is possible, for example, if each partial derivative of V is absolutely integrable ([38] and [14,49,Theorem9.10,pg.266-257]).Inthepresentcase,itissu¢ cienttoshow that the value function is di⁄erentiable on an open subset of x , because of t Criterion 5.14. Then the value function meets the conditions in Mattner [44], particularly the locally bounded assumption, and the interchange is valid.17 The following two results are immediate implications of the envelope theorem. The (cid:133)rst proves that the optimal solution inherits di⁄erentiability. The second formally derives the stochastic Euler equations proposed in Barnett et al. [3] and, more generally, demonstrates that stochastic Euler equations can be validly derived for the class of models. Theorem 6.1 If U ( ) is concave and di⁄erentiable, then the value function is di⁄erentiable. PROOF. Let (cid:30) denote the optimal stationary non-random Markov policy. Note that at time s, (cid:30) is a function of x . To simplify notation, let V (x) = s V (x). Bellman(cid:146)s equation implies (cid:30) V (x) = g (cid:30)(x) +(cid:12) V f (cid:30)(x);z p(dz x;y) (28) j (cid:16) (cid:17) Z Z h (cid:16) (cid:17)i holdsforanyx .Notex = f(y ;z ),sothevaluefunctionwithintheintegral t t+1 t t is being evaluated one period into the future. Let x0 denote the actual initial state. For x N (x0), where N (x0) is a neighborhood of x0, de(cid:133)ne 2 J (x) = g x;(cid:30) x0 +(cid:12) V f (cid:30) x0 ;z p(dz x;y): (29) j (cid:16) (cid:16) (cid:17)(cid:17) Z Z h (cid:16) (cid:16) (cid:17) (cid:17)i In words, J (x) is the value function with the policy constrained to be the optimal policy for x0. Clearly, J (x0) = V (x0) and, x N (x0), J (x) (cid:30) 8 2 (cid:20) V (x) because(cid:30)(x0) is not the optimal policyforx = x0. If the original utility (cid:30) 6 functionU ( ) isconcaveanddi⁄erentiablethensoisu( ) andthereforeg( ). This assumption implies that J (x) is also concave and di⁄erentiable. The envelope theorem from [8] combined with the fact that the policy (cid:30) is optimal uniformly in x then implies that V (x) is di⁄erentiable for all x int(X), (cid:30) 2 As prices and rates of return are assumed to be larger than zero, the only 17The theorem actually applies to holomorphic functions, but the proof can be readily adapted for (cid:133)rst-order (real) di⁄erentiable function. 20

initial conditions for which V (x) is not di⁄erentiable are in(cid:133)nite (positive or (cid:30) negative) initial asset endowments, which can be excluded. 2 Theorem 6.2 (Stochastic Euler Equations) If U ( ) is concave and differentiable. Then the stochastic Euler equations for (3) are, @u(a (cid:3)t ;c (cid:3)t ) = @u(a (cid:3)t ;c (cid:3)t ) 1 E 1+(cid:26) p (cid:3)t @u a (cid:3)t+1 ;c (cid:3)t+1 (30) @a @c (cid:0)1+(cid:24) t 2 i;t p (cid:16) @c (cid:17)3 i 0 (cid:3)t+1 0 (cid:16) (cid:17) 4 5 and @u(a (cid:3)t ;c (cid:3)t ) = 1 E (1+R ) p (cid:3)t @u a (cid:3)t+1 ;c (cid:3)t+1 (31) @c 1+(cid:24) t 2 t p (cid:16) @c (cid:17)3 0 (cid:3)t+1 0 4 5 where a and c are the controls speci(cid:133)ed by the non-randomized optimal sta- (cid:3)t (cid:3)t tionary policy and c is an arbitrary numØraire. 0 PROOF. Using the notation from the previous proof, the envelope theorem implies that @g(x;(cid:30)(x)) @g((cid:30)(x)) V x0 = J x0 = = (32) x x @x (cid:12) @x (cid:12) (cid:16) (cid:17) (cid:16) (cid:17) (cid:12)x=x0 (cid:12)x=x0 (cid:12) (cid:12) (cid:12) (cid:12) where V (x) is a vector-valued function who(cid:12)se ith element is g(cid:12)iven by x @g(x;(cid:30)(x))=@x : (33) i The di⁄erentiability of the value function combined with the ability to interchange di⁄erentiation with integration for the stochastic integral, imply that the necessary conditions for (cid:30)(x0) to be optimal are @V (x) y = 0 (34) @y (cid:12) (cid:12)y=(cid:30)(x0) (cid:12) (cid:12) where@V=@y isavector-valuedfuncti (cid:12) onwhoseith elementisgivenby@V=@y . i It follows that the stochastic Euler equation is @g(y) @f (y;z) +(cid:12) V0 [f (y;z)] p(dz x;y) = 0 (35) @y y @y j (cid:12) Z (cid:12) Z (cid:12)y=(cid:30)(x) (cid:12) (cid:12) (cid:12) where @g=@y is a k +n+1 vector-valued function whose ith element @g=@y , i @f=@y is a k+n+1 by 2k+n+2 matrix with i,j element @f =@y . Equation j i 21

(32) can be used to replace the unknown value function, so that (35) becomes @g(y) @g(y)@f (y;z) +(cid:12) p(dz x;y) = 0: (36) @y @x @y j (cid:12) Z (cid:12) Z (cid:12)y=(cid:30)(x) (cid:12) (cid:12) The simple form of the system equation implies that(cid:12) @f=@y = I 0 ; (37) (k+n+1) (k+n+1) (k+n+1) (k+1) (cid:2) (cid:2) h i so that (36) becomes, @g(y) @g(y) +(cid:12) p(dz x;y) = 0: (38) @y @x j (cid:12) Z (cid:12) Z (cid:12)y=(cid:30)(x) (cid:12) (cid:12) Using the fact that there is a redundant control (cid:12)at the optimum, an element can be eliminated from (cid:30)(x). In particular, choose an arbitrary element of c. Denote this numØraire element by c, and the remaining n 1 elements of c 0 (cid:0) by c . Assume without loss of generality that p = p so that = 1 where 0 (cid:3) 0 (cid:0) is the element of that coincides with c. To further simplify notation, 0 0 let y denote y evaluated at the optimum at time s. Then using the obvious s(cid:3) notation, g(y ) = g (cid:18) ;c ;c and (38) implies that, for i 1;:::;k , s(cid:3) (cid:3)s (cid:3) 0(cid:3) 2 f g (cid:0) (cid:16) (cid:17) @g (cid:18) ;c ;c @g (cid:18) ;c ;c (cid:3)t (cid:3)t 0t(cid:3) (cid:3)t (cid:3)t 0t(cid:3) (cid:0) (cid:0) (cid:16) @(cid:18) (cid:17) (cid:0) (cid:16) @c (cid:17) i 0 @g((cid:18) ;c ;c ) +(cid:12) (cid:3)t+1 (cid:3)t+1 0t(cid:3)+1 p(dz x;y ) = 0: i;t+1 @ (cid:0) c j (cid:3) Z Z (cid:16) (cid:17) 0 Also, taking the derivative with regards to y (the benchmark asset) implies k+1 (cid:0) @g (cid:16) (cid:18) (cid:3)t @ ;c c (cid:3) (cid:0) t ;c 0t(cid:3) (cid:17) +(cid:12) k+1;t+1 @g((cid:18) (cid:3)t+1 ; @ c (cid:3) (cid:0) c t+1 ;c 0t(cid:3)+1 ) p(dz j x;y (cid:3) ) = 0: 0 Z Z (cid:16) (cid:17) 0 (39) Substituting the original notation proves the result. 2 The stochastic Euler equations de(cid:133)ne an asset pricing rule that is a strict generalization of the consumption CAPM asset pricing rule.18 Substituting from (31) into (30) and using the linearity of the conditional expectations operator implied by the linearity of the integral, produces 1 p @u(a ;c )=@a = E R (cid:26) (cid:3)t u a ;c (40) (cid:3)t (cid:3)t i 1+(cid:24) t " t (cid:0) i;t p (cid:3)t+1 c 0 (cid:3)t+1 (cid:3)t+1 # (cid:16) (cid:17) (cid:16) (cid:17) 18Thereare,ofcourse,notherequationsfordi⁄erentiationwithrespecttoelements of c. These equations are simpler in that they are non-stochastic. 22

where u a ;c = @u a ;c =@c. The (cid:133)rst order condition for a c 0 (cid:3)t+1 (cid:3)t+1 (cid:3)t+1 (cid:3)t+1 0 simple uti(cid:16)lity maxim(cid:17)ization(cid:16)problem fo(cid:17)r consumption goods is @u(c)=@ci = pi: @u(c)=@cj pj Similarly,theright-handsideof(40)de(cid:133)nestherelevantinformationforassets(cid:146) relative prices. 7 Conclusion LjungqvistandSargent[39,pg.xxi]statethatthereisan(cid:147)art(cid:148)tochoosingthe right state variables so that a problem can be solved through recursive techniques. They further argue that increasing the range of problems amenable to recursive techniques has been one of the key advances in macroeconomic theory. This paper has applied a di⁄erent art: carefully de(cid:133)ning the characteristics of the state and control spaces. But the motivation is similar. The choice ofspaceandthesubsequentmeasurabilityassumptionsallowstochasticMIUF models to be solved through the DP recursion. The results mirror those that are available for deterministic dynamic problems: an unique solution exists that can be di⁄erentiated to derive (stochastic) Euler equations. The method used in this paper requires regularity conditions that are less restrictive than other approaches. Even more importantly, the regularity conditions do not restrict the economics of the problem. Consequently, the results are broadly applicable to monetary and (cid:133)nancial models, particularly the many models where the existence of a solution was just assumed. The approach to modeling uncertainty can be applied to other stochastic economic models, but introducing risk, as is done in this paper, is probably the clearest motivation for introducing contemporaneous uncertainty. The stochastic MIUF problem integrates monetary and (cid:133)nance models, containing important examples from each as special cases. Further work on integrating aspects of (cid:133)nance into models of money could address both the fact that technological and theoretical advances have been steadily increasing the liquidity of risky assets and the fact there is little consensus on how to model risk. The expected utility framework, whose underpinnings are formally established in this paper, is the most commonly used approach, but it is not universally accepted. It is not yet clear whether stochastic MIUF models can incorporate alternative models of risk. Furthermore, the current model addresses the decision of an individual consumer. Embedding the decision problem in a market context would strengthen the connection between the two literatures. 23

References [1] Aliprantis, C. D., Border, K. C., 1999. In(cid:133)nite Dimensional Analysis: A Hitchhiker(cid:146)s Guide. Springer, Heidelberg and New York. [2] Bansal, R., Coleman II, W. J., Dec. 1996. A monetary explanation of the equity premium, term premium, and risk-free rate puzzles. J. Polit. Economy 104 (6), 1135(cid:150)71. [3] Barnett, W. A., Liu, Y., Jensen, M., 1997. CAPM risk adjustment for exact aggregation over (cid:133)nancial assets. Macroecon. Dynam. 1 (2), 485(cid:150) 512. [4] Barnett, W. A., Serletis, A., 2000. The Theory of Monetary Aggregation. Elsevier Science, North-Holland, Amsterdam; New York and Oxford. [5] Barnett, W. A., Wu, S., 2005. On user costs of risky monetary assets. AnnFinance 1 (1), 35(cid:150)50. [6] Basak, S., Gallmeyer, M., Jan. 1999. Currency prices, the nominal exchangerate, andsecuritypricesinatwo-countrydynamicmonetaryequilibrium. Math. Finance 9 (1), 1(cid:150)30. [7] Bellman,R.E.,1957.Dynamicprogramming.PrincetonUniversityPress, Princeton. [8] Benveniste, L. M., Scheinkman, J. A., May 1979. On the di⁄erentiability of the value function in dynamic models of economics. Econometrica 47 (3), 727(cid:150)32. [9] Bertsekas, D. P., 2000. DynamicProgrammingandOptimal Control. Vol. 1-2. Athena Scienti(cid:133)c. [10] Bertsekas, D. P., Shreve, S. E., 1978. Stochastic Optimal Control: The Discrete-Time Case. Academic Press, Inc., New York. [11] Blackwell, D., Feb. 1965. Discounted dynamic programming. Ann. Math. Statist. 36 (1), 226(cid:150)35. [12] Blackwell, D., Jan. 1970. On stationary policies (with discussion). J. Roy. Statistical Society 133, 33(cid:150)37. [13] Blackwell, D., Freedman, D., Orkin, M., Oct. 1974. The optimal reward operator in dynamic programming. Ann. Prob. 2 (5), 926(cid:150)41. [14] Blume,L.,Easley,D.,O(cid:146)Hara,M.,Dec.1982.Characterizationofoptimal plans for stochastic dynamic programs. J. Econ. Theory 28 (2), 221(cid:150)34. [15] Bohn, H., May 1991. On cash-in-advance models of money demand and asset pricing. J. Money, Credit, Banking 23 (2), 224(cid:150)42. [16] Boyle, G. W., Peterson, J. D., May 1995. Monetary policy, aggregate uncertainty, and the stock market. J. Money, Credit, Banking 27 (2), 570(cid:150)82. [17] Calvo, G. A., Vegh, C. A., Feb. 1995. Fighting in(cid:135)ation with high interest rates: The small open economy case under (cid:135)exible prices. J. Money, Credit, Banking 27 (1), 49(cid:150)66. [18] Canzoneri, M. B., Diba, B. T., 2005. Interest rate rules and price determinancy: The role of transactions services of bonds. J. Monet. Econ. 52, 329(cid:150)43. 24

[19] Chang, R., Aug. 1998. Credible monetary policy in an in(cid:133)nite horizon model: Recursive approaches. J. Econ. Theory 81 (2), 431(cid:150)61. [20] Chari, V. V., Christiano, L. J., Eichenbaum, M., Aug. 1998. Expectation traps and discretion. J. Econ. Theory 81 (2), 462(cid:150)92. [21] Choi, W. G., Oh, S., Oct. 2003. A money demand function with output uncertainty, monetary uncertainty, and (cid:133)nancial innovations. J. Money, Credit, Banking 35 (5), 685(cid:150)709. [22] Davidson, J., 1994. Stochastic Limit Theory: An Introduction for Econometricians. Oxford University Press, New York. [23] Den Haan, W. J., 1990. The optimal in(cid:135)ation path in a Sidrauski-type model with uncertainty. J. Monet. Econ. 25 (3), 389(cid:150)409. [24] Dudley, R. M., 1989. Real Analysis and Probability. Wadsworth and Brooks/Cole, Paci(cid:133)c Grove, California. [25] Dupor, B., Sep. 2003. Optimal random monetary policy with nominal rigidity. J. Econ. Theory 112 (1), 66(cid:150)78. [26] Dutkowsky, D. H., Dunsky, R. M., May 1996. Intertemporal substitution, money, and aggregate labor supply. J. Money, Credit, Banking 28 (2), 216(cid:150)32. [27] Dynkin, E. B., Juskevic, A. A., 1975. Controlled Markov Processes and their Applications. Springler-Verlag, Moscow. [28] Feenstra,R.C.,Mar.1986.Functionalequivalencebetweenliquiditycosts and the utility of money. J. Monet. Econ. 17 (2), 271(cid:150)91. [29] Finn, M. G., Ho⁄man, D. L., Schlagenhauf, D. E., Jun. 1990. Intertemporal asset-pricing relationships in barter and monetary economies: An empirical analysis. J. Monet. Econ. 25 (3), 431(cid:150)51. [30] Hansen, L. P., Singleton, K. J., Apr. 1983. Stochastic consumption, risk aversion, and the temporal behavior of asset returns. J. Polit. Economy 91 (2), 249(cid:150)65. [31] Hinderer,K.,1970.FoundationsofNonstationaryDynamicProgramming with Discrete Time Parameter. Springer-Verlag, New York. [32] Hodrick, R. J., Kocherlakota, N. R., Lucas, D., Apr. 1991. The variability of velocity in cash-in-advance models. J. Polit. Economy 99 (2), 358(cid:150)84. [33] Holman, J. A., Nov. 1998. GMM estimation of a money-in-the-utilityfunction model: The implications of functional forms. J. Money, Credit, Banking 30 (4), 679(cid:150)98. [34] Imrohoroglu, S., Nov. 1994. GMM estimates of currency substitution between the canadian dollar and the u.s. dollar. J. Money, Credit, Banking 26 (4), 792(cid:150)807. [35] Kamihigashi, T., Mar. 2003. Necessity of transversality conditions for stochastic problems. J. Econ. Theory 109 (1), 140(cid:150)49. [36] Kamihigashi, T., Aug. 2005. Necessity of the transversality condition for stochastic models with bounded or CRRA utility. J. Econ. Dynam. Control 29 (8), 1313(cid:150)29. [37] Karatzas, I., Shreve, S. E., 1998. Methods of Mathematical Finance. Springer, New York and Heidelberg. 25

[38] Lang, S., 1993. Real and Functional Analysis. Springer, New York. [39] Ljungqvist, L., Sargent, T. J., 2004. Recursive Macroeconomic Theory. MIT Press, Cambridge, Mass. [40] Lucas, Jr., R. E., Apr. 1972. Expectations and the neutrality of money. J. Econ. Theory 4 (2), 103(cid:150)24. [41] Lucas, Jr., R. E., Apr. 1990. Liquidity and interest rates. J. Econ. Theory 50 (2), 237(cid:150)64. [42] Matsuyama, K., Jan. 1990. Sunspot equilibria (rational bubbles) in a model of money-in-the-utility-function. J. Monet. Econ. 25 (1), 137(cid:150)44. [43] Matsuyama, K., Nov. 1991. Endogenous price (cid:135)uctuations in an optimizing model of a monetary economy. Econometrica 59 (6), 1617(cid:150)31. [44] Mattner, L., 2001. Complex di⁄erentiation under the integral. Nieuw Arch. Wisk. 2 (1), 32(cid:150)35. [45] Mehra, R., Prescott, E. C., Mar. 1985. The equity premium: A puzzle. J. Monet. Econ. 15 (2), 145(cid:150)61. [46] Poterba, J. M., Rotemberg, J. J., 1987. Money in the utility function: An empirical implementation. In: Barnett, W. A., Singleton, K. J. (Eds.), New Approaches to Monetary Economics: Proceedings of the Second International Symposium in Economic Theory and Econometrics. pp. 219(cid:150) 40. [47] Sargent, T. J., 1987. Dynamic Macroeconomic Theory. Harvard University Press, Cambridge, Mass. and London. [48] Shiryaev, A. N., 1996. Probability. Springer, New York. [49] Stokey, N. L., Lucas, Jr., R. E., 1989. Recursive Methods in Economic Dynamics. Harvard University Press, Cambridge, Mass. and London. [50] Strauch, R. E., Aug. 1966. Negative dynamic programming. Ann. Math. Statist. 37 (4), 871(cid:150)90. [51] Townsend, R. M., Apr. 1987. Asset-return anomalies in a monetary economy. J. Econ. Theory 41 (2), 219(cid:150)47. 26