Post-Stimulation Analysis of Monte Carlo Experiments: Interpreting Pesaran's (1974) Study of Non-Nested Hypothesis Test Statistics

International Finance Discussion Papers Number 276

Revised: March 1986

POSTSIMULATION ANALYSIS OF MONTE CARLO EXPERIMENTS: INTERPRETING PESARAN'S (1974) STUDY OF NON+NESTED HYPOTHESIS TEST STATISTICS

by

Neil R. Ericsson

NOTE: International Finance Discussion Papers are preliminary materials circulated to stimulate discussion and critical comment. References in publications to International Finance Discussion Papers (other than an acknowledgment by a writer that he has had access to unpublished material) should be cleared with the author or authors.

ate

ABSTRACT

"Monte Carlo experimentation in econometrics helps ‘solve! deterministic problems by simulating stochastic analogues in which the analytical unknowns are reformulated as parameters to be estimated." (Hendry (1980)) With that in mind, Monte Carlo studies may be divided operationally into three phases: design, simulation, and post-simulation analysis. This paper provides a guide to the last of those three, post-simulation analysis, given the design and simulation of a Monte Carlo study, and uses Pesaran's (1974) study of statistics for testing non=nested hypotheses to illustrate the techniques described. A statistic is derived for testing for significant deviations between the asymptotic and (observed) finite sample properties. Further, that statistic provides the basis for analysing discrepancies between the finite sample and asymptotic properties using response surfaces. The results for Pesaran's study indicate the value of asymptotic theory in interpreting finite sample properties and certain limitations for doing so. Finally, a method is proposed for adjusting the finite sample sizes of different test statistics so that comparisons of their power may be made. Extensions to other finite

sample properties are indicated.

Post-simulation Analysis of Monte Carlo Experiments: Interpreting Pesaran's (1974) Study of Non-nested Hypothesis Test Statistics by

Neil R. Ericsson*

1. Introduction

Monte Carlo studies in econometrics often have been criticized for imprecision present in estimating the underlying finite sample properties investigated and for the specificity of the results from the particular parameter values and sample sizes chosen, so making any conclusions very tentative at best. Hendry (1984) presents a met hodology reducing both and which aims to obtain "numerical-analytical formulae which jointly summarize the experimental findings and known analytical results in order to help interpret empirical evidence and to compute outcomes at other points within the relevant parameter space" (p. 944). That methodology affects all aspects of Monte Carlo experimentation: design, simulation, and post-simulation analysis. Illustrating such post-simulation analysis, this paper re-examines Pesaran's (1974) Monte Carlo study of statistics for

testing non-nested hypotheses.

*This research was supported in part by a grant from the Social Science Research Council to the Programme in Methodology, Inference and Modelling

in Econometrics at the London School of Economics (1979-81) while I was at the LSE. I am grateful for the financial assistance from the

S.S.R.C. although the views expressed in this paper are solely the responsibility of the author and should not be interpreted as reflecting those of the S.S.R.C., the Board of Governors of the Federal Reserve

System, or other members of their staffs. I have benefited from valuable discussions with and comments from Julia Campos, Raymond Chapman, David Hendry, Grayham Mizon, Hashem Pesaran, Denis Sargan, and Frank Srba. I am grateful to Hashem Pesaran for detailed information on his 1974 Monte Carlo experiments (including the results used for forecast tests herein), and for his permission to quote in equations (34) and (35) unpublished material

from Pesaran (1981). All response surfaces were obtained with the computer program GIVE (Hendry and Srba (1978, 1980), Hendry, Morgan, and Srba (1982)). This is a revised and shortened version of Ericsson (1982b, Chapter 3) and is forthcoming in Ericsson (1986) (but without Appendices A, C, and D).

2

Pesaran (1974) compares the finite sample properties of the Cox (1961, 1962) and F statistics for testing the mis~specification of a static single equation model. As a basis for analysing his results, Section 2 describes the asymptotic properties of those statistics. Section 3 interprets Pesaran's Monte Carlo design in light of Hendry (1984) and notes further properties of the statisties when the data generation process is the one selected by Pesaran. Section 4 explains the valuable roles that a response surface can have as a convenient summary of Monte Carlo Simulations and as an approximation to the underlying finite sample distribution function. Further, the section details how to assess the adequacy of a response surface. Section 5 compares Pesaran's experimental results directly with what asymptotic theory would predict, and, to the extent that significant discrepancies exist, attempts to explain them, employing the methodology from Section 4, Finally, Section 6 proposes adjustments such that the estimated finite sample powers of the Cox and F tests may be compared on a more equal footing in those instances where the estimated finite sample probability of type I error for the Cox statistic Significantly exceeds its nominal value. Although the approach presented relates to finite sample rejection frequencies in particular, similar strategies exist for evaluating most properties of estimators and test Statistics, including biases, simulation standard errors, and estimated

asymptotic standard errors.

2. Statistics for testing non-nested hypotheses ae By potneses This section summarizes existing analytical results for the two statistics examined by Pesaran (1974). Consider the two non-nested

hypotheses

Ho: Y = XoBo + Uy Up ~ D(0,097°Ir) (1) and

Hi: y = X,8, + u, u, ~ D(0,0,7*I7) (2) and the comprehensive hypothesis

Ho: y = X28. + Uz, Ug ~ D(0,0,7*Ip) (3)

where the dependent variable y is Tx1, T being the econometric sample size; Xj; is a Txkj matrix of regressors and 8; the corresponding kjx1 vector of coefficients (i=0,1,2); X, includes all the non-redundant variables in (X9:X,) with 8, conformable; and uj is a Tx1 vector of disturbances with mean zerd and variance o;47-Iy- (i=0,1,2). Two approaches have been Suggested for testing H, against H, when X, and X, are predetermined: following Cox (1961, 1962), Pesaran (1974) proposes evaluating a modified likelihood ratio statistic for H, and H,; alternatively, because those hypotheses are nested in H,, the restrictions implied by going from H, to H, (or H,) can be tested using the F or Wald statistics (cf. Silvey (1975, pp. 115-116)). Under H,,

Do 4 N(O,1) | . (4) and

f. a F(k2-ko, T-k2, 0) (5) where D, is the Cox statistic for which H, is assumed true, f, is the F statiscvic for testing H, against the general hypothesis H,, and 5 denotes ‘converges in distribution to, as T+", Pesaran (1982) and Ericsson (1983) derive the asymptotic properties of D, under H, and H,; however, Pesaran (1974) estimates the finite sample rejection frequencies of D, and D, (the latter being the Cox statistic for which H, is assumed true) under H, only. Noting the similarity in those approaches, it follows

from Eriesson (1983) that, under H, as a local alternative to H,,

D, g N(y,,4,) (6) and

f, g F(k,-k,, T-k,, 3) (7) where uy, and w, are the asymptotic mean and variance of D,, f, is the F statistic for testing H, against H,, and A, is the asymptotic

non-centrality parameter of that F statistic. !

3. The data generation process and experimental design eee esse and experimental design Pesaran (1974, p. 160) generates his data in the following manner: Yt = Ao + doXt + UE Ugpt ~ NID(0O,097) (8)

r/(1-r?2) 1/2 }

x 1 t > NID(o, r/(1-r2)}/2 (1-r2)7?

t=1,...,T (9) 2t

where r is the population correlation coefficient of Xt, and Zp +2 Given

that process, he considers three specifications:

Hot Vt = a + doXxt * Uot ot ~ NID(0,09*) (10) Hit Ye = a, + Diz + Ure uit ~ NID(0,0,7) (11) Hot Ye = a, + DyxXE + Cozte + UdtE Ust ~ NID(O,o,7) . (12)

Letting R be the population multiple correlation coefficient for (8)

ee

lEricsson's (1983) derivation of an equation equivalent to (6) but for D, with H, as a local alternative involves an approximation additional to the Standard asymptotic approximation. Pesaran (1982) only requires the usual asymptotic approximation but must have at least as many regressors (total) under H, as regressors in H, but not in H,. (Note that the roles of H, and H, are reversed for D,.) In practice, Ericsson's and Pesaran's approximations appear numerically similar; and, for the model in Pesaran (1974), Pesaran's (1982) asymptotic distribution for D,? is just that of f However, the formula in (6) is used throughout. See Appendix A for a detailed discussion of the statistical properties of D,, D,, f,, and f,.

For extensive discussions on the role and interpretation of non-nested hypothesis test statistics, see MacKinnon (1983) (including the comments thereon), the Journal of Econometrics issue on "non-nested models" (1983, Vol. 21, No. 1), Mizon (1984), and Mizon and Richard (1986). Godfrey and Pesaran (1983) suggest and evaluate analytical adjustments for improving the finite sample properties of the Cox Statistic.

3°

@Pesaran's (1974) notation of xz has been simplified to r, and n is nowT.

5 (R*=b,*/(b,*+0,7)), Pesaran (1974) calculates D,, D,, f,, and f,.for 500

replications of each of 400 points defined by all possible combinations of

T = (20, 40, 60, 80) cr? = (.90, .91, ..-, .99) 32 = (.80, 81, «22, 89) —_ (13)

with a,=100 and b,=2. Following Hendry's (1984, p. 940) notation and terminology, the Monte Carlo design variables are

9 = (a5,D9.P,R2) € @ = {9 | r2<1, R<1} (14) and : . Te T= [(T°,T'] (15)

where T is pre-assigned with T°=20 and T'=80.3 Equations (8) and (9)

are the data generation process (DGP); 0xT is the parameter space; equations (10), (11), and (12) are the relationships of interest4; and

the objective of the Monte Carlo study is to determine the finite sample distributions of the statistics D,, D,, f,, and f, as defined by the relationships of interest, within the specified parameter space of the DGP. More modestly, letting t be any of D,, D,, f2, and f, and 6 be the critical value associated with a test based on tT, the objective is to find the finite sample rejection frequency m7 = prob(|t|26). That probability depends upon 6 and T and can be expressed as a conditional probability

formula:

rd

i}

1 prob(|t|26 | 6,T) = g(0,T) . (16)

Thus, we wish to know (or obtain a good approximation to) g(@,T) over OxT.9

3In (14), 8 could be defined as 6 = (a9 5Dos% 5097)» noting that (b,,R7) maps one-to-one onto (by,0,97).

4wote that: both H, and H, coincide with the DGP but H, does not: specifically, H, is a mis~specified model with x, in H, replaced by zt, a variable correlated with xtz.,

DSImplicitly, g(+,-) is a function of § as well. However, because 6 is constant for each of the two types of statistics in Pesaran (1974), its presence in g(:,°) is ignored in the analysis below.

6 The DGP defined by (8) and (9) and the relationships of interest in (10)-(12) have certain implications for the properties of the statistics being examined. First, the formulae for their asymptotic distributions

have explicit representations: \,;, u,, and w, may be expressed as

Ag = Tdo2(1-r?)/oo7 , . (17)

vb, = 7T?/2b,(1-r?2)?72(14r2)/(2oor2) (18) and

w, = {4r* + (1-r?)(1+r?)?7}/(4rS) . (19)

For r? close to unity (relevant for Pesaran's experimental design; 3ee (13)), uw.? = A, and oy * 1, so the distribution of D,? is roughly a x¥7(1,4,). Because f, is asymptotically distributed as an F(1, T-k,, ee) (and hence asymptotically as a y?(1,\,)), D, should have about the same power as f, for Pesaran's experiments. Second, for a given local alternative (i.e., for a constant non-zero bey in (17) and (18)), ‘she asymptotic powers of the Cox and F tests tend to the nominal size as r* > 1. Third, because X, is fixed, f, and f, are exactly distributed as F(k,~Kg, T~k,, 0) and F(k,-k,, T-k,, 4%) where the finite sample non-centrality parameter .* is.

Ms Dot XS" (WW, )X$Do/ 097 | (20) with Wy = Xy(Xq'Xq)7'Xy" (i=0,1,2) and Xt = (x,, x2, ..., Xp)’. These analytical results are invaluable in interpreting the Monte Carlo

Simulation results, as will become apparent in the following sections.

4, Monte Carlo methodology

Cox (1970, Chapters 3 and 6), in his discussion of the empirical logistic transform, implicitly provides the basis for developing response surfaces of estimated finite sample probabilities, including both estimated

finite sample powers and estimated finite sample probabilities of type I

T error ,© Consider a binary response variable for which the probability of "success" (or, frequently later, acceptance or rejection by a particular test) is m (O < m < 1) and on which there are N observations (N > 1), S$

being "successes", Letting

A = [S(N-S)]/(N-1) , . (21) 4 L(t) = A'¥2in | — Oo<zr< 1, (22) 1-2 | and c 7 (2N)7? L*(c) = a?/2in | ——————_—_ (2N)7? < ¢ < 1-(2N)7? (23)

1- ¢- (2N)7™ in a notation similar to that found in Hendry (1984, pp. 957-961) and Mizon and Hendry (1980, p. 34), then, using results from Cox (1970, pp. 30-34, 41-42, 78-79), it can be shown that (syn) = L¥(s) - L(w) z= N(0,1) | (24)

where s = S/N and A denotes "converges in distribution to, as N > @ "7 In the context of Pesaran's (1974) Monte Carlo study (and Monte Carlo studies of powers in general), N is the number of replications ina particular experiment, S the number of replications for which the value of the test statistic lies in the critical region, and n the finite sample (i.e., finite econometric sample T) probability of the test statistic lying in the critical region.

Typically, ms is unknown; and, as an initial step in analysing Monte

' Carlo results, it is of interest to test whether m equals the (local)

asymptotic (i.e., as T + ~) power of the test (mg, say). That is easily

6see Cochran and Cox (1957, pp. 335ff), Cox (1958, pp. 113-128), and the references in Cochran and Cox (1957, p. 369) on the use of response surfaces in statistical analyses; cf. Sowey (1973).

TA proof is given in Appendix B.

8 accomplished by replacing m by ta in (24) and comparing the value of ¢(s,1qg) with the appropriate critical values for the standardized normal distribution.8 Even when t=1g is not a good approximation, (24) still provides the basis for analysing the Monte Carlo results. Without loss of generality,

wT Ta 0 = ———- exp{G*(6,T)} (25) 1- ty 1-7

where 6 is the vector of all the parameters (except T) which define thie model generating the binary random variable of interest, Gt(-,*) is some appropriate function, a is a parameter which itself may depend upon 6 and T, and 7 is subscripted by T so as to emphasize that mn (2 E(S/N)) is-a function of the econometric sample size.9 Using (25), (24) may be rewritten as

L*(s) = aL(mg) + at/?-Gt(@,T) + ©€ e x N(O,1) . (26)

8as indicated by Cox (1970, pp. 27-29), the logistic and normal distributions only differ slightly, primarily in the tail regions. So, provided that n is not too close to 0 or 1, the normal distribution should provide a good approximation ta the distribution of (24). Furthermore, the use of the normal approximation is often justified because N is typically quite "large" in Monte Carlo studies (e.g., N = 500 in Pesaran (1974)).

It is assumed throughout that 7 and tg each lie strictly within the unit interval. ’

9clearly, (25) is equivalent to (16). However, the functional form of (25) ensures that predicted powers are within the unit interval. Further, (25) has the advantage of having split g(6,T) in (16) into two components, an asymptotic term and a term involving the deviation between the finite sample and asymptotic distributions. The equivalent partition for (16) is Tp = Ta t+ (177g). By using asymptotic theory, we are able to simplify the problem of simulating mp (of 0(1)) directly to one of analytically calculating ta and simulating only (mp-mg) (of o(1), and quite possibiy o(T7!/2)), That is in line with Hammersley and Handscomb's (1964, pp. 5, 59) precept that one should solve as much of the problem as possible analytically in order to minimize the imprecision and specificity arising from simulation. With that in mind, see Hendry (1973) and Nickell (1981) for two elegant examples in which analytical formulae greatly simplify the interpretation of previous Monte Carlo studies.

If an analytical approximation to m7 better than mg is available (e.g., an Edgeworth expansion), it could appear in (25) in place of ny, further reducing the order of the term being simulated.

9

In practice, a and the functional form of G*(-+,*) are unknown although (e.g., in Pesaran (1974)) they are implicitly defined by the computer program generating the Monte Carlo data. If a and G*(*,°) were known, the exact finite sample probability (of "success", rejection) for any particular value of (8,T) could be calculated directly from (25), obviating any need for conducting Monte Carlo experiments to estimate mp. Even with a and Gt(-,°) unknown, approximations to them may be found; and further, the accuracy of those approximations may be tested, 10

From asymptotic theory, one expects that a + 1 as T + » and that

Gt(e,T) = 1T7/?G(e,T7'/?) — | (27) where G(6,T~!?/2) is O(T°) (cf. Phillips (1977, p. 474; 1982), Sargan (1980, p. 1120)). Thus, a might be expanded in powers of T7}/? about T=;

ao = ad + a, T4? + a TT? + 2 y (28) where a, is expected to be unity; and G(:,+) might be expanded in powers of T~1/2 and of the elements of e.11 Truncating both the series for a and the series for G(e,T7~?/2), the coefficients of the powers and cross-products of 6, T~!/?, and ln{ta/(1-1g)} may be estimated by least squares, correcting for heteroscedasticity using the weight Ad/2, i.e., from estimating 10There are two distinct senses in which a and G(-,-) can be known: for the particular experiments in which tg is estimated, and for any values of (0,T) in Oxf. Clearly, the latter is far more useful. That distinction also emphasizes the value of choosing an experimental design which covers a wide range of OxT.

Aneuryn-Evans and Deaton (1980, pp. 284-285) suggest an alternative framework for analysing Monte Carlo results on statistics for testing non-nested hypotheses. However, their approach did not prove fruitful.

See Eriesson (1982a) for details.

11the parameterisation of @ is not unique, and it may be worthwhile transforming "natural" parameters of the model into parameters which span the same range as L*(s) before expanding G(-,.), For instance, a parameter bounded between -1 and 1 (p, say) might better appear in 6 as p/(1-p?); a parameter bounded from below by zero (07, say) might better appear in 6 as ln(o?).

10

L*(s) = aob(mg) + a,T7?/7L (mg) +... + AM ?T772H(8,TT7?) + @ © (29)

where H(6,T~?/2) is the least squares approximation to G(6,T7?/4?) and the error term e is the combination of e« (the error arising from simulation) and A!/27~1/2(G(+,+)-H(+,°)} (the error from approximating G(-,-) by H(*,°)).

A response surface like (29) summarizes a possibly vast array of Monte Carlo simulations in a relatively simple formula which may account for much of the variation in s across experiments and may be useful for predicting mp at points within OxT but not included in the simulations. Further, the response surface may adequately approximate the underlying finite sample distribution function. Two types of information are available for inferring how. "good" a response surface like (29) is:

(A) asymptotic theory (i.e., a ,=1), and

(B) € A NID(O,1)

(cf. Hendry (1984, p. 962)). Although (B) is not directly testable, many testable implications follow from the null hypothesis that H(-°,°)=G(-,°).

(B,) oe? = 1. If H(+,*)#G(-,*), then og? > 1 because € is uncorrelated with a2/ 2972/26 (2, -)=H(+,*)}. The hypothesis og? = 1 may be tested by noting that, under the null, the residual sum of squares from (29) is distributed as a x* random variate with its degrees of freedom equal to the number of experiments less the number of regressors, provided N is large. Power under the alternative is directly related to the magnitude of AT~*{G(-,*)-H(-,°)}? over the experiments.

(B,) The error e does not include any terms involving 6, qo 1/2, and In{tg/(1-mg)}. By using OLS, e can not include any of the terms in H(-,°). However, if H(*,*)#G(*,°), e contains terms of a higher order than those included in H(*,*) (cf. Maasoumi and Phillips (1982, p. 198) and Hendry (1982, p. 210)). By initially specifying a rather general formulation for

H(-,°) and a and simplifying therefrom, the F statistic comparing the final

11 Specification against the general one helps test for the presence of such factors in the e's of the final specification.

(B,) The error e is normally distributed.

(B,) The e's are serially independent for any ordering of experiments specified prior to simulation. That follows from the independence of ¢ across experiments. If H(:,+)#G(+,*) and experiments are ordered to be (e.g.) increasing in values of 6 and T, terms in e involving 6, T7!/2, and In{ta/(1-1g)} may induce serial correlation and/or heteroscedasticity in the e's,

(Bs) H(+,*) is constant over regions of the parameter space which were not included in the estimation of (29).

Table I lists the bulk of the test statistics reported below; the convention used is that &(q) and nj(q,p) denote statistics which have central y?(q) and F(q,p) distributions respectively under a common null and against the ith alternative. Thus, &,(q) and na(q,K-m-q) both test for qth-order residual autocorrelation. There are K experiments and m regressors in the response surface under the null hypothesis.

The extent to which (A) and (B,)-(B;) are not satisfied reflects the degree of approximation of the response surface to the underlying conditional probability formula (response function) although the power of tests of (A) and (B,)-(B,) depends crucially on the number of replications per experiment, the experimental design (i.e., the points in ©xT examined), and the choice of DGP and ©xT. Finally, even if any of (A) and (B,)-(B;) are rejected, the response surface still has certain desirable properties as an approximation to the unknown function G(-,:) (White (1980b, pp. 1557157)) and it still may account for (and so summarize) much of the

inter~experiment variation.

Null

(A) (B,)

(B2)

(B;)

(B;)

(By)

(B,)

(B, )

(Bs)

lla

Table I. Criteria for evaluating response surfaces

Alternative Statistic Sources

| n2(1,K-m-1) Hendry (1984, p. 952)

>1 &,(K-m) Theil (1971, pp. 137-8) q invalid parameter n, (q,K~-m-q) Johnston (1963, p. 126) restrictions skewness (SK) and E.(2) Jarque and Bera (1980) excess kurtosis (EK) heteroscedasticity Ne (q,K-m-q) White (1980a, p. 825), quadratic in regressors Nicholls and Pagan (1983)

(q quadratic terms)

first-order ARCH E,(1) Engle (1982) first-order residual dw Durbin and Watson autocorrelation ‘ (1950, 1951), Farebrother (1980) qth-order residual E,(q)3 Box and Pierce (1970); autocorrelation Ne (q,K-m-q) Godfrey (1978), Harvey (1981, p. 173) H*(+*,*) not constant Ng (m,K-2m) Kendall over subsamples . (1946, pp. 242ff), Chow (1960, pp. 595ff) predictive failure E,(q); Hendry (1979, p. 222); over a subset of q n, (q,K-m-q) Chow (1960, pp. 594-5) obser vations),¢

Notes: a. The value of q may differ across statistics, as may the number of

regressors m and the number of experiments K across response surfaces and Monte Carlo studies.

b. We have labelled the Chow statistic n,(q,K-m-q) to highlight the

pre-eminence of the issue of constancy. The covariance test Statistic ng(m,K-2m) is often (and confusingly) referred to as the "Chow statistic" although Chow (1960, p. 592) was well aware of its presence in the literature.

c. Constancy may be tested using Chow's statistic, the covariance

statistic, or the usual y? statistic based upon the forecast errors. Often, an even more stringent test may be constructed by substituting unity for the estimated value of og? in the re..evant Statistic, thereby testing the "absolute" accuracy of the response surface. Such statistics are designated as those above, but with a prime added, e.g., &,(q) becomes &!(q).

12 5. Evaluation of Pesaran's estimated finite sample rejection frequencies

Equation (24) above is the foundation for evaluating Monte Carlo estimates of finite sample rejection frequencies. This section utilizes (24) both in assessing how close the simulation results are to what asymptotic theory would predict and in formulating response surfaces to explain significant deviations between Monte Carlo results and asymptotic theory.

The "closeness" of an estimated finite sample rejection frequency of a test statistic to its asymptotic value can be tested using $(s,1g) from (24) above where s is the fraction of replications for which the value of the statistic lies in the critical region and tg is the probability of the statistic lying in the critical region asymptotically. The values of the estimated finite sample powers and estimated probability of type I error for the Cox and F statistics, their asymptotic values, and the values of o(8,mtq) are given in Table II. Figure 1 strikingly displays how much of the inter-experimental variation in the finite sample rejection frequencies is explained by asymptotic theory. Now consider those results for each statistic in turn.

Pesaran (1974, p. 161) notes that the estimated probability of type I error for f, never differs significantly from .05, and the corresponding values of $(s,.05) in Table II confirm that. Further, the sum of squares of o(s,.05) over Pesarants (1974) nine experiments is &,(9) = 6.45, offering additional support to the hypothesis fn = .05.12

The estimated probability of type I error for D, is significantly different from (and larger than) .05 in several experiments, although the magnitude of their difference decreases as T increases, in line with

12Note that £,(9) is distributed as a y7(9) for large N (provided that m = .05) and that prob[y?(9)216.92] = .05.

12a

Table II. The asymptotic powers (mg) and estimated finite sample powers (s) of D, adf,, the estimated type I errors (s) of D, and f,, and, in each case, the value of a statistic 9(s,+) for testing the closeness of the asymptotic and

finite sample results. -

768 . O48 (£010)

458 . 056 (.010)

052 (.010)

O48 (.010)

052 (.010)

“O42 (.0109)

~~ ~~

~~

SS BR 88

-_~ ww

Notes: a. The nominal size of D, and f, is .05.

b. The total number of replications per experiment (N) is 500 for each experiment, with the multiple correlation coefficient of the DCP (R?) being .8.

Cc. Under the null hypothesis that m=ng (or that m.05 for D, ad f,), $(s,°) x N(O,1). Asterisks denote results for which the null hypothesis is rejected at the 5% level.

d. Simulation standard errors of estimated probabilities are in parentheses.

12b

Figure 1

Asympiotic and estimated finite sample rejection frequencies for Do; Dy, and f3

—_ fo)

Oo

Estimated finite sample power (s)

size (s) of Dg r=)

Estimated

0 2 4 6 8 1.0 Asymptotic power (ra)

Legend

Asymptotic power or size

— =—— Rejection frequency for Do or Dy (T = 20) — ease Rejection frequency for Do or Dy (T = 40) = «= = Rejection frequency for f3 (T= 20) quaenona= Rejection frequency for. f3 (T = 40)

13

asymptotic theory (cf. Figure 1). Further, for T=80, the three estimated probabilities are individually and jointly insignificantly differert from .05.

The estimated finite sample power of D, differs significantly from its asymptotic power in three experiments although, as with the probability of type I error, the difference between estimated and asymptotic values decreases as T increases. Also, as with the probability of type I error, the estimated values are insignificantly different from their asymptotic values for T=80. 13

That the estimated finite sample power for f, differs from its asymptotic power as markedly as it does in three experiments is a surprise at first blush, and requires some explanation. As noted in Sections 2 and 3, f, is exactly distributed as F(1, T-k,, a*) but only asymptotically as F(1, T-k,, A3;), where NB and \, are the finite sample and asymptotic

non-centrality parameters in (20) and (17).14 a% is not equal to |,

13No test can be made for {T=80, r2=.90} because S$

= See Cox (1970, pp. 33, 42, 78) on the treatment of experiments with

N. S=0 or S=N.

VAror Pesaran's model,

fs = {y' (W.-W, )y/(k~k, )}/{y' (Ip-W. )y/ (T-k, )} . Noting that (I7-W,)X, = 0 and (W,-W,)X,8, = (W.-W, )X$8% where g§ is the Sub-vector of parameters in 8, corresponding to Xt, then, under H,,

f, = {u*' (W.-W, )u*/(k,-k, )}/{u*' (Ip-W, )u*/ (T-k,)}

where u* = xtgt + u,. If X, is fixed in repeated samples, or at least if X, and u, are independent in each sample (and so X, may be conditioned upon; see Schmidt (1976, pp. 93-94ff)), then, conditional upon X,,

u* ~ N(Xtgt, oo2*Ip) . Because (W,-W,) and (Ip-W,) are each idempotent and of ranks (k,-k,) and (T-k,), respectively,

u*'(W-W, u*/oo2 ~ x2(ka7k,, AZ) and u*' (Ip-W, )u*/a9? ~ x2(T-k,, 0) where Ae = BO XS (Wa-W, )XSBS/ 007

noting that (Ip-W,)X$ = 0. Further, the two x* variates above are independent because (I7-W,)(W,-W,) = 0 (see Rao (1973, p. 187)), so

f, ~ F(k,-k,, T-k,, 5) (exactly) .

14

in general, so f, is not (generally) distributed as an F(1, TrKz, Ag). Further, ’* varies across replications because the x's and z's are generated for each replication. Because the distribution of f, is a non-linear function of a, which is itself a non-linear function of X,*, the expected fraction of rejections by f, need not equal the probability of rejectior associated with the given critical value for F(1, T-k,, ha)! However, (7) still holds, so, for large N, $(S,1q) should be approximately a standardized normal variate, provided that T is large enough. If X,* were known for each replication (or if it had been held constant over replications within experiments and were known for each experiment), the exact probability of rejection for f, could be calculated for each replication (and so for each experiment). As it is, X,* is not known, so only the asymptotic formula in (7) may be used. As noted for D,, the differenze between the estimated finite sample power of f, and its asymptotic power shrinks as T increases. 16

Because Pesaran's (1974, p. 162) published results contain just nine experiments (effectively eight for D, and f,), only highly restrictive response surfaces may be fitted to them. Expanding G(e,T7?/2) and a in to i/2 only and truncating at the first derivative (all a priori and

arbitrarily) leads to the rather simple response surface

15~the discrepancies between the theoretical and estimated finite sample powers cf f, might have arisen from the approximation used to calculate the probabilities associated with the non-central F distribution. However, Appendix C shows that the errors involved are not large enough to explain those discrepancies.

16, similar analysis applies to the distribution of f, under H,, with f, ~ F(kK,7kg, T-ky, 43) and f, 4 F(k,~ko, T7kz, Az) where \* and

A, are defined in an obvious manner. Under H, (i.e., with g7=0), however , vs = \, = 0; so the exact and asymptotic distributions of f. are identical, and not a function of X, at all. The results for f, in Table It are in line with that discussion.

15 L¥(s) = (oo + a,T7?/?)L(mg) + AMAT 2 (ky + KTM?) + € AV2(q 52 + a, RTT? + KT? + «,TT1) +e (30)

where & = In{ta/(1~-mg)}. Equation (30) may be too general a model for the

estimated probabilities of interest; so, in the usual way, coefficients of the regressors in (30) may be tested to be zero or to satisfy other constraints. The final response surfaces for the estimated finite sample

powers of D, and f, are

L¥(s) - L(g) = AM ?T7/2(1.16 - .502) (31) (.23) (.17) C.24] [.16]

K = 8 R?2 = .75 Gg = 1.02 &,(8) = 10.2 &4(8) = 10.7 n, (8,6) = 1.03 no(1,5) = .02 &3(6) = 6.3 n,(2,4) = .0O4 E5(2) = .7 SK =-.5 EK = -.4

ne(3,11) = .24 €,(1) = 1.8 dw = 2.49 no(2,12) = .85 and

L*(s) - L(mg) = A?/?T71/2(-1.07 - 1.208) (32) (243) (.30) {.32] (.34]

K= 8 R*2 = .66 Ge = 1.96 £&,(8) = 6.8 &1(8) = 26.0 n,(8,6) = .84 n2(1,5) = 1.29 §€3(6) = 23.1 (2,4) = .6 &5(2) = .8 SK =-.0 EK = 1.2

nNe(3,11) = .70 €,(1) = 1.8 dw = 2.23 ng(2,12) = .01

respectively. .R? is the unadjusted squared multiple correlation coefficient!7, 6e is the square root of the estimated residual

variance, and (-) and [:] respectively denote conventionally calculated and

17R2 may lie outside the unit interval because (30) has no constant term (see Schmidt (1976, pp. 3-5)). In particular, R* is less than zero in the

general response surface for D, because L*(s) has a large negative mean. However, Ge, not R?, is the appropriate measure of the goodness-of-fit

for the response surfaces, so small or negative values of R? per se are not worrisome.

In this paper, "R?2" refers both to the unadjusted squared multiple correlation coefficient for response surfaces and to the squared population multiple correlation coefficient for the DGP (8). The uses are distinct and no confusion should arise from using a common notation.

16 White (1980a) coefficient standard errors. Tests of (B,) use eight sets of results for which T and r? take the same values as the fitted results, but for which the R? of the DGP is 85.18 Noting that the entire discussion of response surfaces above applies equally well for any estimated probability (including the estimated probability of type I error), response surfaces for the estimated probability of type I error for D, are estimated

as well, the final specification of which is

‘

L*(s) - L(.05) = at?T710.5 (33) (1.5) [1.1] K = 8 R? = .65 Ge = .89 §€,(8) = 6.5 €&4(8) = 5.2 n,(8,7) = .80 no(1,5) = .00 §&3(7) = 5.5 n,(2,5) = 1.70 €5(2) = 1.2 SK = .4 EK = ~1.1

ne(l,14) = 2.07 §€7(1) = 1.0 dw = 1.80 no(1,14) = .12

The estimated probability of type I error for f, need not be analysed further because it does not deviate significantly from its asymptotic value.

The summary statistics indicate that the response surfaces for D, and D, show no signs of mis-specification nor are the restrictions that a, = 1 and k, = 0 (for D,) and a, = 1 and Ko = O (for D,) rejected. Given the data available, those response surfaces support the conjectures that a = 1, de? = 1, and that very simple expansions of a and G(e,T7?/2) are sufficient to explain the observed Monte Carlo experiments. Further, both

restricted response surfaces predict the eight out-of-sample observations

18see Appendix D for all the results. The results tor {T=80, r?=.90} are not included in any of the response surfaces although they could have been in those for Dj. Experiments are ordered as in Table II. .

In general, experiments for prediction could be chosen by random selection or stratified selection. Cf. Cox (1958), Wilks (1962).

The statistics —,(°), SK, EK, and n,(°,°) use all sixteen experiments because of the degrees of freedom involved; the others (where appropriate) use only the eight published in Pesaran (1974) (i.e., for which R® = .80). However, their values alter only slightly if all sixteen are included.

17 with acceptable accuracy, no matter whether the relative or absolute measure of forecast accuracy is used.19

The response surface for f, presents more of a puzzle. The

restrictions in (32) are not rejected, and there is no indication of serial correlation in either the unrestricted or restricted response surfaces (and hence no indication that higher order terms from the expansions of a and G(0,T~!/2) ought to be present in those response surfaces). However, o9” is significantly greater than unity in both response surfaces (see &,(6)), possibly indicative of the problems with f, noted above. For comparison, Pesaran (1981) obtains the following response surfaces for D, and f, when using a DGP and relationships of interest similar to (8)-(12) but with the

number of non-overlapping regressors in (10) and (11) varying from one to four ,29

L*(s) = .967L(ma) + AM%2(.034 - 7.61277? + 1.124,T7?) (34) (.019) (.025) (.80) (.18)

K = 108 R? = .982 Ge = 1.30

n2(1,104) = 3.02 €,(104) = 176. dw = 1.07

L*(s) = -977L (mq) + ATM ?(-,031 + 1.622T7? - 2.184477?) (35) (.023) (.026) (.90) (.15)

K = 108 R= .978 G9 = 1.52

n2(1,104) = 1.00 §&,(104) = 240. dw = 1.44

These response surfaces concisely summarize results for over one hundred experiments and satisfy many of the criteria (A) and (B,)-(B,). However,

as with (31) and (32), the response surface for f, fits worse (in terms of

19tn the response surfaces for D,, it was not necessary to include any "terms of 0(67!)", i.e., terms resulting from the additional approximation in (6) (see Ericsson (1983)). That suggests that the normal distribution approximates the asymptotic distribution of D, quite well.

20pesaran (1982), the published version of Pesaran'ts (1981) working paper, unfortunately does not include these response surfaces.

18 6e) than the one for D,, so that issue may be worthwhile investigating further.

In general, response surfaces appear quite valuable in summarizing Pesaraa'ts (1974) experimental results. Simple, well-determined response surfaces are obtained using only eight experiments for their estimation. Further, those response surfaces satisfactorily predict the outcomes of eight additional experiments for which one of the experimental design parameters (R?) is considerably different from its value in all the experiments used for estimation, indicating the robust nature of those response surfaces. Even so, the response surfaces are limited by the relatively small number of experiments and the restrictiveness of the parameter space. Such specificity could be reduced markedly by performing more experiments over a more broadly defined parameter space (e.8-, one including lagged dependent variables, several non-overlapping regressors, k )#k,, and autocorrelated exogenous regressors), thereby allowing more

sophisticated and (hopefully) reliable inferences to be drawn.

6. Adjustments for the estimated probability of type I error

In comparing the properties of the Cox and F statistics, it is only reasonable to consider both the probability of type I error and the probability of type II error. The estimated probability of type I error for D, significantly exceeds .05 in several instances (predominantly for smaller sample sizes and smaller values of r?), whereas the estimated probability of type I error for f, never does. To account for both types of error, Pesaran (1974, p. 163) uses a simple linear loss function in which type I errors receive the same (or even twice as much) weight as type II errors, concluding that "the [Cox] test is preferable to the F-test when

the sample size is small (n $ 40) and the correlation between the competing

19 set [sic] of explanatory variables is large". Pesaran's conclusions may not depend so much upon the asymptotic nature of the Cox test (vs. the exact nature of the F test) as upon the comparison of estimated type I and type II errors for which the density of the statistic (for the type I error) at its critical values is a small fraction of the density of the statistic (for the type II error) at one of the critical values.

To see that, consider the extremely simplified example in which the test statistic t is exactly distributed as N(0,1) under H, and N(y,1) under H, and the critical value (to be chosen) for a symmetric two-tailed test is 5, or 8, (59 > 6, > 0) with 6, slightly greater than 6,. The mean y is non-zero and is assumed negative without loss of generality. Further, suppose that, for 6,, power is in an intermediate range, SO yp = “do. Simplifying the example even more (but not appreciably weakening the argument), suppose that p = -5, = -1.96 (see Figure 2). Atu, the densities under H, and H, are approximately .058 and .399 respectively. Noting that the probability of t being greater than +1.96 is negligible under H, and that the test is symmetric, it follows that a test using 6, would be preferred to one using 6,, even if the probability of type I error were weighted three times as heavily as the probability of type II error. In the context of the Cox and F tests, because Pesaran was comparing statistics for which the probability of type I error was small (= .05) and the power (and hence the probability of type II error) was usually between .10 and .90, his criterion might well have favoured the test statistic with the larger probability of type I error (i.e., the Cox statistic), even had the two test statistics had identical distributions but critical values

corresponding to different probabilities of type I error.

19a

Figure 2

The density of t under Hg and Hy | Density at x 00

40

.30

.20

10

— 4.0

20

Ideally, one would choose critical values for D, and f, giving identical probabilities of type I error and use those critical values when estimating the finite sample powers of D, and f,, but that is not feasible because the finite sample distribution of D, is not known. As an alternative, one could solve for the critical value of f, which would give it the same probability of type I error (ap, say) as the estimated probability of type I error for D, (sp, say).21 (Table III lists “hose critical values for T = 20 and T = 40.) Then, using those critical values, the power of f, could be estimated for each Monte Carlo experiment. However, because that approach would depend upon knowing the values of the dependent variable and regressors for all replications of all experiments (information which is not currently available), a somewhat inferior method is adopted, namely, calculating the power of f, using its asymptotic non-centrality parameter ()\,), evaluated at the adjusted critical value. Those (adjusted) powers for f, and the estimated powers of D, are presented in the last two columns of Table III and indicate much smaller differences between the powers of the tests: based on D, and f, than are apparent in Pesaran's (1974, p. 162) Table 1. Even so, f, does appear slightly more

powerful than D, at higher powers, and vice versa at lower powers.

1. Concluding remarks

This paper describes and implements an approach for obtaining numerical-analytical formulae (response surfaces) which integrate existing analytical knowledge with experimental results. Response surfaces can help summarize and interpret Monte Carlo simulations, and May reasonably

approximate the unknown finite sample conditional probability formula for

2Icf. Sargan (1976, pp. 444ff) who suggests how to improve the efficiency of the estimated probability by using a control variate.

20a

Teble III. The powers of the F test (f,) with non-centrality 1, when evaluated at critical values corresponding to ap = .05 and ar = Sp, og being the size of the F test and Sp the estimated size of the Cox test.

Note: a. Simulation standard errors of estimated probabilities are in parentheses.

21 the relationship(s) of interest and DGP considered. To evaluate the closeness of approximation, this paper presents a framework for inference about response surfaces. Illustrating this methodology, Pesaran's (1974) Monte Carlo simulations of the finite sample rejection frequencies of the Cox and F tests are re-examined. Further, adjustments are made in order that the estimated finite sample powers of various tests may be compared with each other on a more equal footing in those instances where the estimated finite sample probability of type I error for one of the tests differs significantly from the nominal value. Although this paper is concerned with tests in particular, the methodology is much more general: similar techniques have been successful in analysing a wide variety of estimators, both consistent and inconsistent, linear and non-linear (ef. Engle, Hendry, and Trumble (1985), Campos (1986), Hendry (1984), and references therein). Although Monte Carlo experimentation does not replace analysis, the two can complement each other effectively to provicle

convenient formulae for interpreting empirical findings.

22 Appendix A: The distributions of D,, f,, D,, and f, under H,: in general and for Pesaran's (1974) model in particular

This appendix derives the distributions of the test statistics of interest, using the formulae in Ericsson (1983). Further, for Pesaran's (1974) data generation process and relationships of interest, formulae explicitly in terms of his design parameters are found, The notation is the same as that in Ericsson (1983) (and that above) except that the econometric sample size is denoted by T (not n), the regression coefficients are £j (not aj), and N is the matrix Z(Z'Z)7'Z' (not the number of replications).

From Ericsson (1983),

Do g N(0,1) . (A1) and

f. a F(kg7ky, T-k2, 0) (A2) under H,. Further,

Do FI N (uo 2 Wo) (A3)

under H,, where

Vo = 7 (Sott,)/(20,vt2) , Wo = {(Co+t,)?/(4,)}evar(q*)/o,2 , and var(q™)/o,? = N(CotEirS2)/ (Totti)? ~ Al CarCa)/L (Cotl1 Ea} + (C3-Sx)/E2" , (A4) with sn een Tee (i 20) , do = plim, 6'Xt'NXtS/T , 6; = plim, 6'Xt'(Q,Q,)i71Q,xté/T (i 21), fs) = BtvT (6 a non-zero constant) , Qi = NPy = NX; (X4'NXq)7?X4'N (i = 0, 1, 2) , and N = Z(Z'Z)7*Z' , (A5)

Z being the Txm matrix of observations on m instruments, which is equal to

23

the entire set of regressors X, in Pesaran (1974). Further, under 4,,

‘ y' (Ip-P,)' (Ip-P2 )y/ (T-k, ) ut (Q,7Q, )u/ (ka-ky ) (n6) ul (Ip-P,)' (Ip-P2 )u/ (T-k, ) where us = ~Xtpi + u, and u, ~ N(O, o,7*I7) . Hence w' (Q,-Qo )U/ {51 2(Ky~Ky)} gy X2(Ky7Kyy Ay) / (Ka~ky) (AT) and u' (Ip-P,)' (Ip-P,)u/{o,2(T-k,)} 4 x2(T-k,,0)/(T-k,) (A8)

where A, = C)/0,7. These random variates are independent asymptotically, so their ratio f, is asymptotically distributed as a singly non-central F,

f. a F(k,-kKy, T7K,, Az) . (A9) Assuming X, fixed in repeated samples, or at least assuming X, and u, independent and X, to be conditioned upon (see Schmidt (1976, pp. 93-94ff)), then |

uo ~ = N(XTBT, 0,7*Ip) ; (A10) and, with Z -%, the results in (A7), (A8), and (A9) are exact (i.e., with qa Substituted by ~), with

Me = BT'Xt'(Q,-Q,)XtBt/o,? (A11) replacing i, in (A7) and (AQ).

In Monte Carlo studies, it is often more convenient (and less expensive computationally) to evaluate (for instance) both D, and D, under Ho, than to evaluate D, under H, and H,. Effectively, D, under H, behaves

like D, under H, because each statistic is evaluated under a non-nested

alternative. Using the formulae in (A3)-(A5), it readily follows that

24

D, 3 N(u,., ) (A12)

under H,, where

da = 7 (0 (0)*S (1) (200V%t(2)) Wi = ((0(9)*E(1))?/(4E(2)) evar (al*))/o42 , (A13) and var (q(*))/o9? = HCC (0)+B(1)75(2))/ (0(0) #51) )? ~ ACC (2)7 (3) )/ 1 (C0) tB(1) 5 (2)} + ((3)7S(4))/E(2)7 (A14) with

S(i) = (i) 7 O(i+1) G20) ,

d(o) = plimy 6*'xt'nNxd6*/T , b(i) = plimy 6*'X$'(Q,Q,)i71Q,x¢6*/T (i 21) , and se = gtyt (s* a non-zero constant) . (A15)

That is, the role of the parameters and variables in the model H, is exchanged with that of the parameters and variables in the model H,. (Superfluous parentheses around subscripts and superscripts denote that those roles have been exchanged in the definition of the subscripted or Superscripted variable.) Likewise, the distribution of f, under H, is

Py a F(k-k,, T-k,, Az) (A16) where i, = S(0)/ 07s If Z = X, and X, is fixed, f, is exactly distributed as an F(k,-k,, T-k,, A%) where AS = BS'X3'(Q,-Q, )X3BG/ 00?

= 8*'X$"(Q,-Q, )X$6*/ (0977) (A17)

(and note that A, = plim, A*).

‘in Pesaran's (1974) model, the formulae above may be simplified because (i) Z = X, and (ii) X, = (Xi: X: Xt). Letting

Wy = Xq(Xq'Xy)7'Xy! and My = Ip-Wy (i=0,1,2), Z = X, implies that

25 Qi = Py = Wy and Q,xXt = NX} = W,Xt = Xt (i=0,1). The expressions above for (4) and re Simplify to

(0) = plim, 6*'xt'xte*/T ,

o(i) = plim, 6**xX$'(W,W Jim Wyxte*/T = (i 2 1), (a18) and AS = ONTXEN(Ip-W, )XE6*/ (0927) = 8*'xt'M, xt6*/ (0,27) . (A19)

Adopting the following notation:

M4 = plim, X$'xt/T , 2 vo M+ = plim, X, Xo/T ,. m,, = plim, X,'X,/T , Mo = plim, X,'X,/T , Moo = plim, X,'X,/T , (A20)

Mo, = Mo’, and my, = m4", then $(4) may be written as

(0) = 6*'m,48*

*"ms1m, 1.71 (m, oMy9~*mo,m,,~*)im 1m, ,8* (i 21). (A21)

(i) = 6 Given the population moments of the regressors, c(i) (i=0,...,4) may be calculated; and hence so can be Hi» W,, and A;, knowing o,?. From those, the (approximate) asymptotic powers of the tests based on f,; and D, are derived. (See Appendix C for details.)

The formulae for $(4) and 4, may be simplified further by noting two particular features of the structure of X,: that X, is a Tx3 matrix with X$, X, and Xt all being Tx1 vectors; and that X$, X, and xt

satisfy the properties given in (8) and (9) where xt = (X1sXoe5050-,Xp)",

é X = (1,1,...,1), and Xf = (Z1,Z2,++-,27)'. The relevant moment matrix is

N yy 1 |

T71E(X,'X,)

It follows that

Mee = [1] 0 1 0 Mra = Lo ya

Mi+ =

To

Y

{r2/(1-r2)} 3/2

for positive Y.

{(8$)?(1-R?)/R?

xt'xt x'xt

xt'xt Xt Xt XE Ze

Moo

pi/2

26

xt'x x'X xt'x

Xt

2

' xt'xt ' x'xt t XT XT

XE 2 Zt 2

(A22) y?+1

(A23)

The four-fold product of matrices for o(i) (appearing in

the parentheses of the formulae for $(;) in (A21)) simplifies to

“4 0

0 vue) | ’ (a2H) so o(1) = (6*)7(r?)1, i=0,...,5. Substituting into (A13) and (a14),

wy = > 6*(1=r?) 172 (14r?2)/(209r?) (A25)

Ww, = {4r* + (1-r*)(14+r?)}/(Cur®) and (A26)

hs = (8*)?(1-r?)/o,2 (A27)

Those formulae for u,, w,, and A, were used to calculate the asymptotic

powers of f, and D, under H, from Pesaran's model.

27 It is interesting to note that vi? = (6*)2(1-r?)(14r?)?/(4og2r*) | (A28)

so that for r? close to unity,

u

Wy? (8*)2(1-r2)/o92 = dg (A29)

and Ww = 1. (A30)

Hence D,? and f, have nearly identical asymptotic distributions; and, given a symmetric two-tailed test for D,, D, and f, should have roughly the same

power. Even when r? is not so close to unity, the approximations in (A29)

and (A30) are still reasonable, as can be seen in Table A.I below for

Pesaran's published results.

Table A.I. A comparison of the parameters in the approximate asymptotic distribution of D, (w, and y,) with the one in the asymptotic distribution of f, (A,;). R? for the DGP is .80.

Parameter

28

Appendix B: Proof of the asymptotic normality of the empirical logistic function.

It is to be shown that

s - (2N)7?} T o(s,7) = A'M?{in | —————-—---__ ] - in 1- s - (2N)7?} 1-7

} q N(O.1). (BI)

S is the number of "successes" out of a total of N observations (N > 1) on a binary response variable with probability of "success" equal to 1

(0 < <1), Ss (3S/N) is the fraction of "successes", A = S(N-S)/(N-1), and A cenotes "converges in distribution to, as N +o", Results

in Cox (1970, pp. 30-34, 41-42, 78-79), Mann and Wald (1943), and Cramer (1946, pp. 254, 299-300) are used extensively.

Noting that E(S) = Nm and var(S) = Na(1-1), define U such that

S = Nar + UW. (B2) Hence E(U) = O and var(U) = n(1-m). The term in braces in (B1) is m + (U/YN) - (2N)7? 1 - a — (U/YN) - (2N)7? {+} = ln - i.n T 1- = U/{n(1-m)¥N} + =O (NT?) (B3)

(ef. Cox (1970, p. 33, (3.11)) with (his) a=-.5). The variable A in (B1) is

A = Ns(1-s) + Op(N®) = Na(i-m) + Op(N?/?) , (By) so Ai/2 = N2/2{a(1-1) + Op(N71/?)}172 = {Nn(i-n)}272 + O5(N®) (B5)

Substituting (B3) and (B5) into (B1),

o(S,m) = U/{m(imm) F172 + Op(NT?) (B6) Sinze S is the sum of N independent and identically distributed random variates, each with mean m and variance m(1-7), and U = (S - Na)/YN, then U/{1(1-7)}*/? converges in distribution to N(0,1) as N > © by the Lindeberg-Lévy variant of the central limit theorem (Cramér (1946, p. 215)).

Hence (B6) converges in distribution to N(0,1) as N > @, ||

29 Appendix C: The calculation of probabilities associated with non-central x? and F distributions, and with the normal distribution

The discrepancies between the theoretical and estimated finite sample powers of f, might have arisen from the approximation used to calculate the probabilities associated with the non-central F distribution. This appendix, which closely follows the substance and notation of Keridall and Stuart (1973, pp. 237-239, 241, 262-264), shows that the errors involved do not appear large enough to explain those discrepancies. For completeness and clarity in presentation, a related approximation (of the non-central ,? distribution by a central one) is given first. See Johnson (195°) on other approximations to the non-central xy? and F distributions.

The probability of a non-central yy? random variate exceeding

xq(v,0), the 100(1-a) per cent point of the central y? distribution, is

P= S dy?(v,A) . (C1) X5,6¥ 90)

where x?(v,A) is the non-central y? distribution with degrees of freedom v and non-centrality parameter’ 4, as given in Kendall and Stuart (1973,

p. 238, equation (24.18)). Equating the first two moments of the non-central x2 random variate to those of py?(-,0) (a central y? random

variate multiplied by a factor of proportionality p, to be determined),

then Ps J dy2(v*,0) (C2) X56v0)/p where x] (v + ))? vo Ba ¢3) and > = (v + 2h) . (ch) v

+ d

30

Having solved for v* and the lower limit point of the integral in (C2), the integral itself may be calculated numerically, e.g., with the NAg (1977) routine GO1BCF, linearly interpolating for P if v* is not integral.

Table C.I compares the approximation given in (C2) with exact values of the integral in (C1) for the values of v and '\ appearing in Patnaik (1949, p. 207, Table 1). Patnaik's own approximation to the exact probability involves not only linearly interpolating P for non~integral v*, but also interpolating P for the value of the lower limit of the integral in (C2). (C2) (using GO1BCF) appears to be a better approximation to the exact probability than Patnaik's approximation, but only marginally so, With both approximating the exact probability quite well.

The probability of a singly non-central F random variate exceeding

Fy(vi,v2,0), the 100(1i-a) per cent point of the (central) F distribution, is

P= f GF(v,9Va,k) (c5) Fav. 5V2 0)

where F(v,,v,,A) is the singly non-central F distribution with the degrees of freedom in the numerator and denominator of the F-ratio being v, and v,, respectively; and A is the non-centrality parameter of the y? random variate in the numerator , the non-centrality parameter of the ,? random variate in the denominator being zero (see Kendall and Stuart (1973,

p. 262, equation (24.105))). Equating the first two moments of the nor-~central y* random variate in the numerator to those of a central ,?

random variate (as above), then

eo

Pos S dF(v*,v,,0) (C6)

{vi/ (vy tA) FFg (vy v2 20) where 2 ye = (v, + A) (C7)

Vv, + 2h

31 From v,, v2, A, and a, the approximation to P in (C6) may be calculated numerically, e.g., with the NAg (1977) routine GO1BBF, linearly

interpolating for P in (C6) if v*

is not integral. Table C.II compares the approximation given in (C6) with exact values of the integral in (C5) for values of v,, v,, and \ appearing in Patnaik (1949, p. 222, Table 7).. patnaik's own approximation to the exacz -

probability involves both linearly interpolating P for non-integral v* and interpolating P for the value of the lower limit of the integral in (C6). The latter interpolation only slightly affects the values obtained, and both approximations perform well over a wide range of powers.

The probability of a normal variate with mean y and variance o? being

less than a certain critical value z is z* , (2n)7?72 f exp(-u2/2) du (C8) =O where z* = (z-u)/o . This integral is calculated directly with the NAg

(1977) routine SI5ABF. The integral corresponding to the upper tail is

calculated in a similar manner with NAg (1977) routine S15ACF.

32

Table (@.I. A comparison of two approximations to the non-central y?(v,\)

distribution with points on the distribution.

Probability

Patnaik's approximation

Xalvs0) Exact

Probability

Approximation from (C2)

33

Table C.II. A comparison of two approximations to the singly non-central F(v,,v2,A) distribution with points on the distribution.

Probability

dA | Fglvi.v2,0) Patnaik's Approximation Exact approximation from (C6) probability

34

Appendix D: Pesaran's experimental results far the multiple correlatio coefficient of the DGP (R?) being .80 and .8.

Table D.I. Estimated probabilities for the Cox and F tests when R? = -8.

Probability of | Probability of | Probability of | Probability of | Type I error: Type II error: accepting H, rejecting H, rejecting both | accepting both | probability of | probability of and and H, and H, Hy and H, rejecting H, accepting H,

rejecting H, accepting H,

-786 .634 | .050 .026 | .032 .022 | .132 .318 | .082 6. |{«C«w (.018) (.022) | (.010) (.007) | (.008) (.007) | (.015) (.021) | (.012) (.010) | (.017) (.021)

- 600 -4O4 058 . 028 .032 314 540 086 056 : 564 (.022) (.022)}(.010) (. (.007) (.008) | (.021) (.022) | (.013) (.010) | (. (.022)

174 09% | . . . .026 | .754 2 | .072)—(«k . 878 (.013) | (. . . . (.019) (.016) | (. . . (.015)

.118 = .210 (.014) (.018)

664 756 (.019) (.019)

012 | . . 002 «= 6.020 |. . 020 (.005) | (. . (.002) (.006) | (. . (.006)

-030 .018 | .020 .020 | yoy 578 | . .03 524 (.008) (.006) | (.006) (.006)] (.022) (.022) | (. . (.022)

Note: a. Simulation standard errors of estimated probabilities are in parentheses.

35

Table D.II. Estimated probabilities far the Cox and F tests when R* = .& .

Probability of | Probability of | Probability of | Probability of | Type I error: | Type II error: accepting H, rejecting Hy rejecting both | accepting both | probability of | probability of and and rejecting H, accepting H, rejecting H, accepting H,

| | r?{D test F test |] D test F test |D test F test;D test F test]D test F test;D test

-768 | .O44 .026 | . . 060 .184 | .086 O48 | . (.016)@ (.019) | (.009) (.007) | (.009) (.007) | (.011) (.017) | (.013) (.010) | (.014)

-688 -540 -O54 . . 224 . -088 -056 (.021) (.022)|(.010) (. . . (.019) (. (.013) (.010)

254 116 . . . . * 674 . O72

. . . 062 (.011) (. . . (.011)

008 | .0O70 . . . O74 (.004) } (.011) (. . . (.012)

528 86.76 | Cs .022 »©6©.02H | . 056 (.022) (.022) | (.008) (.008)} (.007) (.007) | (.022) (.022) | (.010)

Note: a. Simulation standard errors of estimated probabilities are in parentheses.

References

Aneuryn-Evans, G. and A. Deaton (1980) "Testing linear versus logarithmic regression models", Review of Economic Studies, 47, 1, 275-291.

Box, G.E.P. and D.A. Pierce (1970) "Distribution of residual autocorrelations in autoregressive-integrated moving average time series models", Journal of the American Statistical Association, 65, 332, 1509-1526. as

Campos, J. (1986) "Finite sample properties of the instrumental variables estimator for dynamic simultaneous equation sub-systems with ARMA disturbances", Journal of Econometrics, forthcoming.

Chow, G.C. (1960) "Tests of equality between sets of coefficients in two linear regressions", Econometrica, 28, 3, 591-605.

Cochran, W.G. and G.M. Cox (1957) Experimental Designs, New York, John Wiley and Sons, second edition.

Cox, D.R. (1958) Planning of Experiments, New York, John Wiley and Sons.

Cox, D.R. (1961) "Tests of separate families of hypotheses" in J. Neyman (ed.) Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, University of California Press, volume I, 105-123.

Cox, D.R. (1962) "Further results on tests of separate families of hypotheses", Journal of the Royal Statistical Society, Series B, 24, 2, 406-424,

Cox, D.R. (1970) The Analysis of Binary Data, London, Methuen.

Cramer, H. (1946) Mathematical Methods of Statistics, Princeton, Princeton University Press.

Durbin, J. and G.S. Watson (1950) "Testing for serial correlation in least squares regression. I", Biometrika, 37, 3 and 4, 409-428.

Durbin, J. and G.S. Watson (1951) "Testing for serial correlation in least squares regression. II", Biometrika, 38, 1 and 2, 159-178.

Engle, R.F. (1982) "Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation", Econometrica, 50, 4, 987-1007.

Engle, R.F., D.F. Hendry, and D. Trumble (1985) "Small-sample properties of ARCH estimators and tests", Canadian Journal of Economics, 18, 1,

66793.

Ericsson, N.R. (1982a) "Testing linear versus logarithmic regression models: a comment", Review of Economic Studies, 49, 3, 477-481.

37

Ericsson, N.R. (1982b) Testing Non-nested Hypotheses in Systems of Linear Dynamic Economic Relationships, Ph.D. thesis, University of Loncon.

Ericsson, N.R. (1983) "Asymptotic properties of instrumental variables statistics for testing non-nested hypotheses", Review of Economic Studies, 50, 2, 287-304.

Ericsson, N.R. (1986) "Post-simulation analysis of Monte Carlo experiments: Interpreting Pesaran's (1974) study of non-nested hypothesis test statistics", Review of Economic Studies, forthcoming.

Farebrother, R.W. (1980) "The Durbin-Watson test for serial correlation when there is no intercept in the regression", Econometrica, 48, 6, 1553-1563; and (1981) "Erratum", Econometrica, 49, 1, 277.

Godfrey, L.G. (1978) "Testing against general autoregressive and moving average error models when the regressors include lagged dependert variables", Econometrica, 46, 6, 1293-1301.

Godfrey, L.G. and M.H. Pesaran (1983) "Tests of non-~nested regression models: Small sample adjustments and Monte Carlo evidence", Journal of Econometrics, 21, 1, 1337154.

Hammersley, J.M. and D.C. Handscomb (1964) Monte Carlo Methods, London, Chapman and Hall.

Harvey, A.C. (1981) The Econometric Analysis of Time Series, Oxford, Philip Allan.

Hendry, D.F. (1973) "On asymptotic theory and finite sample experiments", Economica, 40, 158, 2107217.

Hendry, D.F. (1979) "Predictive failure and econometric modelling in macroeconomics: the transactions demand for money" in P. Ormerod (ed.) Economic Modelling, London, Heinemann Education Books, 217-242.

Hendry, D.F. (1980) "Monte Carlo experimentation in econometrics", Discussion Paper No. 8035, CORE, Universite Catholique de Louvain, Louvain-la-Neuve, Belgium, November.

Hendry, D.F. (1982) "A reply to Professors Maasoumi and Phillips", Journal of Econometrics, 19, 2/3, 203-213.

Hendry, D.F. (1984) "Monte Carlo experimentation in econometrics", Chapter 16 in Z. Griliches and M.D. Intriligator (eds.) Handbook of Econometrics, Amsterdam, North-Holland Publishing Company, volume II, 937-976.

Hendry, D.F., M. Morgan, and F. Srba (1982) "GIVE guide", mimeo, London School of Economics.

Hendry, D.F. and F. Srba (1978) "Technical manual for GIVE: General instrumental variables estimation of linear equations with lagged dependent variables and first order autoregressive errors", mimeo, London School of Economics, October.

38

Hendry, D.F. and F. Srba (1980) "AUTOREG: A computer program library for dynamic econometric models with autoregressive errors", Journal of sconometrics, 12, 1, 85-102.

Jarqu2, C.M. and A.K. Bera (1980) "Efficient tests for normality, nomoscedasticity and serial independence of regression residuals", Economics Letters, 6, 3, 255-259.

Johnson, N.L. (1959) "On an extension of the connexion between Poisson and 2

x° distributions", Biometrika, 46, 3 and 4, 352-363.

Johnsson, J. (1963) Econometric Methods, New York, McGraw-Hill Book Company.

Kendall, M.G. (1946) The Advanced Theory of Statistics, Volume II, London, Charles Griffin.

Kendail, M.G. and A. Stuart (1973) The Advanced Theory of Statistics, Volume 2, Inference and Relationship, London, Griffin, third edition,

Maasoumi, E. and P.C.B. Phillips (1982) "On the behavior of inconsistent :nstrumental variable estimators", Journal of Econometrics, 19, 2/3, 183-201.

MacKinnon, J.G. (1983) "Model specification tests against non-nested alternatives", Econometric Reviews, 2, 1, 85-110.

Mann, H.B. and A. Wald (1943) "On stochastic limit and order relationships", Annals of Mathematical Statistics, 14, 3, 217-226.

Mizon, G.E. (1984) "The encompassing approach in econometrics", Chapter 6 in D.F. Hendry and K.F. Wallis (eds.) Econometrics and Quantitative Heonomics, Oxford, Basil Blackwell, 135-172.

Mizon, G.E. and D.F. Hendry (1980) "An empirical application and Monte Garlo analysis of tests of dynamic specification", Review of Economic Studies, 47, 1, 21-45.

Mizon, G.E. and J.-F. Richard (1986) "The encompassing principle and its application to testing non-nested hypotheses", Econometrica, forthcoming.

Nicholls, D.F. and A.R. Pagan (1983) "Heteroscedasticity in models with lagged dependent variables", Econometrica, 51, 4, 1233-1242.

Nickell, S. (1981) "Biases in dynamic models with fixed effects", Econometrica, 49, 6, 1417-1426.

Numerical Algorithms Group (NAg) (1977) NAg FORTRAN Library Manual, Mark 6 (FLM6), Oxford, Numerical Algorithms Group Ltd.

Patnaik, P.B. (1949) "The non-central y?- and F-distributions and their applications", Biometrika, 36, 1 and 2, 202-232.

Pesaran, M.H. (1974) "On the general problem of model selection", Review of Economic Studies, 41, 2, 153-171. .

39

Pesaran, M.H. (1981) "Comparison of local power of alternative tests of non-nested regression models", mimeo, Trinity College, Cambridge, June.

Pesaran, M.H. (1982) "Comparison of local power of alternative tests of non-nested regression models", Econometrica, 50, 5, 1287-1305.

Phillips, P.C.B. (1977) "Approximations to some finite sample distributions associated with a first-order stochastic difference equation", Econometrica, 45, 2, 463-485; and (1982) "Erratum", Econometrica, 50, 1, 274.

Rao, C.R. (1973) Linear Statistical Inference and Its Applications, New York, John Wiley and Sons, second edition.

Sargan, J.D. (1976) "Econometric estimators and the Edgeworth approximation", Econometrica, 44, 3, 421-448; and (1977) "Erratum", Econometrica, 45, 1, 272.

Sargan, J.D. (1980) "Some approximations to the distribution of econometric criteria which are asymptotically distributed as chi-squared", Econometrica, 48, 5, 110771138.

Schmidt, P. (1976) Econometrics, New York, Marcel Dekker.

Silvey, S.D. (1975) Statistical Inference, London, Chapman and Hall.

Sowey, E.R. (1973) "A classified bibliography of Monte Carlo studies in econometrics", Journal of Econometrics, 1, 4, 3777395.

Theil, H. (1971) Principles of Econometrics, New York, John Wiley and Sons.

Wilks, S.S. (1962) Mathematical Statistics, New York, John Wiley and Sons.

White, H. (1980a) "A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity", Econometrica, 48,

4, 817-838.

White, H. (1980b) "Using least squares to approximate unknown regression functions", International Economic Review, 21, 1, 149-170.

IFDP

NUMBER

276

275

274

273

272

271

270

269

268

267

266

4O

International Finance Discussion Papers

TITLES

1986

Post-simulation Analysis of Monte Carlo Experiments: Interpreting Pesaran's (1974) Study of Non-nested Hypothesis Test Statistics

A Method for Solving Systems of First Order Linear Homogeneous Differential Equations When the Elements of the Forcing Vector

are Modelled as Step Functions

International Comparisons of Fiscal Policy:

The OECD and the IMF Measures of Fiscal

Impulse

An Analysis of the Welfare Implications of

Alternative Exchange Rate Regimes: An

Intertemporal Model with an Application 1985

Expected Fiscal Policy and the Recession of 1982

Elections and Macroeconomic Policy Cycles

Assertion Without Empirical Basis: An Econometric Appraisal of Monetary Trends

in ... the United Kingdom by Milton Friedman

and Anna J. Schwartz Canadian Financial Markets: The Government's Proposal for Reform

Was It Real? The Exchange Rate Interest Differential Relation, 1973-1984

The U.K. Sector of the Federal Reserve's Multicountry Model: The Effects of Monetary and Fiscal Policies

Optimal Currency Basket in a World of Generalized Floating: An Application to the Nordic Countries

AUTHOR(s)

Neil R. Ericsson

Robert A. Johnson

Garry Schinasi

Andrew Feltenstein David Lebow Anne Sibert

William H. Branson Arminio Fraga Robert A. Johnson

Kenneth Rogoff Anne Sibert

David F. Hendry Neil R. Ericsson

Garry J. Schinasi

Richard Meese Kenneth Rogoff

Hali J. Edison

Hali J. Edison Erling Vardal

Please address requests for copies to International Finance Discussion Papers, Division of International Finance, Stop 24, Board of Governors of

the Federal Reserve System, Washington, D.C.

20551.

IFDP NUMBER 265

264

263

262

261

260

259

258

257

256

255

254

253

252

251

250

44

International Finance Discussion Papers ae ee ee eciss ion rapers

TITLES

Money Demand in Open Economies: Substitution Model for Venezuela

A Currency

Comparing Costs of Note Issuance Facilities and Eurocredits

Some Implications of the President's Tax Proposals for U.S. Banks with Claims on Developing Countries

Monetary Stabilization Policy in an Open Economy

Anticipatory Capital Flows and the Behaviour of the Dollar

Simulating Exchange Rate Shocks in the MPS and MCM Models: An Evaluation

Trade Policy for the Multiple Product Declining Industry

Long Memory Models of Interest Rates, the Term Structure, and Variance Bounds Tests

Currency Substitution and the New Divisia Monetary Aggregates: The U.S. Case

The International Transmission of Oil Price Effects and OPEC's Pricing Policy

U.S. Banks' Lending to Developing Countries: A Longer-Term View

Conditional Econometric Modelling: An Application to New House Prices in the United Kingdom

Low Pushing: Doctrine and Theory

1984 (partial listing) Postwar Financial Policies in Taiwan, China

Foreign Exchange Constraints and Growth Possibilities in LDCs

The Determination of Front-end Fees on Syndicated Eurocurrency Credits

AUTHOR('s)

Jaime Marquez Rodney H. Mills

Allen B. Frankel

Marcus H. Miller Arnold Kling Arnold Kling Catherine Mann Gary S. Shea Jaime Mar quez Jaime Marquez Henry S. Terrell Rod Mills

Neil R. Ericsson

David F. Hendry

William Darity, Jr.

Robert F. Emery

Jaime Marquez

Rodney H. Mills Henry S. Terrell