Can Macro Variables Used in Stress Testing Forecast the Performance of Banks?
Abstract
When stress tests for the banking sector use a macroeconomic scenario, an unstated premise is that macro variables should be useful factors in forecasting the performance of banks. We assess whether variables such as the ones included in stress tests for U.S. bank holding companies help improve out of sample forecasts of chargeoffs on loans, revenues, and capital measures, relative to forecasting models that exclude a role for macro factors. Using only public data on bank performance, we find the macro variables helpful, but not for all measures. Moreover, even our best-performing models imply bands of uncertainty around the forecasts so large as to make it challenging to distinguish the implications of alternative macro scenarios.
Finance and Economics Discussion Series Divisions of Research & Statistics and Monetary Affairs Federal Reserve Board, Washington, D.C. Can Macro Variables Used in Stress Testing Forecast the Performance of Banks? Luca Guerrieri and Michelle Welch 2012-49 NOTE: Staff working papers in the Finance and Economics Discussion Series (FEDS) are preliminary materials circulated to stimulate discussion and critical comment. The analysis and conclusions set forth are those of the authors and do not indicate concurrence by other members of the research staff or the Board of Governors. References in publications to the Finance and Economics Discussion Series (other than acknowledgement) should be cleared with the author(s) to protect the tentative character of these papers.
Can Macro Variables Used in Stress Testing Forecast the Performance of Banks?∗ Luca Guerrieri and Michelle Welch July 23, 2012 Abstract Whenstresstestsforthebankingsectoruseamacroeconomicscenario, anunstated premise is that macro variables should be useful factors in forecasting the performance of banks. We assess whether variables such as the ones included in stress tests for U.S. bank holding companies help improve out of sample forecasts of chargeoffs on loans, revenues, and capital measures, relative to forecasting models that exclude a role for macro factors. Using only public data on bank performance, we find the macro variables helpful, but not for all measures. Moreover, even our best-performing models imply bands of uncertainty around the forecasts so large as to make it challenging to distinguish the implications of alternative macro scenarios. JEL Classification Codes: C53, G17, G38. KEYWORDS:forecastcombinations,macrovariables,measuresofbankingconditions, stress tests. ∗ We thank Valentin Bolotnyy, Francisco Covas, Rochelle Edge, Lutz Kilian, and Andreas Lehnert for useful comments and discussions. Remaining errors are our own. The views expressedinthispaperaresolelytheresponsibilityoftheauthorsandshouldnotbeinterpreted asreflectingtheviewsoftheBoardofGovernorsoftheFederalReserveSystemorofanyother person associated with the Federal Reserve System. 1
1 Introduction Stress tests are one of the tools used by banks to understand their risk exposures. Bank regulators use stress tests to verify that banks will still be able to maintain adequate levels of capital under stressful but plausible circumstances. One type of stress test, referred to as “scenario analysis,” involves the application of historical or hypothetical scenarios to assess the impact of various events on the performance of banks. Scenario analysis was an integral part of the Comprehensive Capital Analysis and Review (CCAR), a supervisory assessment of bank holding companies conducted by the U.S. Federal Reserve in 2011 and 2012, as well as of the Supervisory Capital Assessment Program (SCAP) in 2009. In these three instances, the 19 largest U.S. bank holding companies were asked to submit capital plans extending 9 quarters out and reflecting macroeconomic baseline and stress scenarios formulated by the Federal Reserve.1 As mandated by the Dodd-Frank Wall Street Reform and Consumer Protection Act, scenario analysis will continue to be an integral part of stress tests for the largest bank holding companies in the United States.2 An unstated premise of stress tests built around a macroeconomic scenario is that the macro variables should be useful factors in forecasting the performance of banks. We assess whether variables such as the ones included in the CCAR scenarios help improve out of sample forecasts of chargeoffs on loans, revenues, and capital measures, relative to forecasting models that exclude a role for macro factors. Furthermore, we construct confidence bands around the conditional forecasts of our measures of performance for the banking sector.3 1All bank holding companies with total assets in excess of $50 billion were required to participate in CCAR. 2Othercountrieshavereliedonstresstestsinsupervisingthebankingsector. Perhapsmostprominently, theEuropeanBankingAuthorityconductedstresstestsbasedonscenarioanalysisthroughouttheEuropean Union in 2009, 2010, and 2011. 3Weuseatop-downapproachthatreliesonpublicdataonly. Regulatorsandeachbankholdingcompany 2
The forecast method that we use is the equal-weighted average of simple models as first proposed by Bates and Granger (1969). This approach has been found to yield results near the frontier of best performance in a varied range of applications, see, for instance Stock and Watson (2004), who also provide extensive references to the literature on forecast combinations. Faust and Wright (2008) opine that “its empirical success is part of the folklore of forecasting,” and like many others include the equal-weighted combination of simple models as a standard benchmark. Specifically, we focus on Root Mean Squared Errors (RMSEs) for a battery of forecast combination models conditional on macro variables, chosen because they areincludedinCCARscenarios, andarandomwalkmodel. Comparisonsagainst a random walk model are interesting on at least three accounts: 1) the random walk model beats purely auto-regressive models in RMSE; 2) it has no role for macro variables; 3) it is a standard benchmark model for forecast assessment. The scenarios used in the recent stress tests of bank holding companies in the UnitedStateswerenottailoredtothebusinessmodelofanyonespecificcompany. We focus on forecasting aggregate performance for the companies involved in stress testing in the United States. It is not our purpose to tailor our regressions to encompass bank-specific factors. A priori, it is not clear whether or not bank specific factors would reveal more robust relations between bank performance measures and macro factors. We leave that pursuit for further work. Interestinstresstesting, priortothefinancialcrisis, wasmostlycircumscribed to practitioners. Early contributions that document the use of time series models for stress testing are those of Blaschke, Jones, Majnoni, and Peria (2001), Kalirai and Schleicher (2002), and of Bunn, Drehmann, and Cunnigham (2005). In a recent paper, Covas, Rump, and Zakrajsek (2012) model PPNR’s six sub components as well as net chargeoffs to generate a path of the tier-1 common have access to a greater wealth of data than what is publicly disclosed. 3
risk based capital ratio. Quagliariello (2009), gathers contributions on the topic of stress testing from regulators around the world. A feature common among the early work on stress-testing is the evaluation of modelsbasedonin-sampleperformance. LikeCrookandBanasik(2012), wetake an explicit out-of-sample approach. While there is an established literature that uses financial factors to facilitate the forecast of macro aggregates, few papers construct forecasts of financial variables based on macro factors.4 Our paper is among the few that follow this latter course. Intuitively, even slow moving variables, like the unemployment rate, may incorporate useful information for some of the banking measures that exhibit little high-frequency variation, such as the tier-1 capital ratio. Therestofthepaperisstructuredasfollows: Section2spellsoutthedetailsof the forecast models used and describes the data; Section 3 lays out our empirical findings; section 4 concludes. 2 Description of Method The measures of performance for the banking sector for our analysis are aggregates for the top 25 bank holding companies by total assets assessed quarterly. Using call Report data, the total assets for these companies amounted to $9.3 trillion in 2011Q3, or 74% percent of the assets of all commercial banks. We considered measures of performance, from three classes of financial variables: credit measures, revenue measures, and capital measures. All measures are derived from the Consolidated Reports of Condition and Income (Call Report) of the Federal Deposit Insurance Corporation. We selected net chargeoffs on loans and leases (chargeoffs for short) as a credit measure, pre provision net 4For the former, see for instance Estrella and Hardouvelis (1991) and Estrella and Mishkin (1998). For the latter, see Bellotti and Crook (2009). 4
revenue (PPNR) and net interest margin (NIM) as revenue measures, and the tier-1 regulatory capital ratio as a capital measure. To forecast each measure based on macro variables, we use a simple forecast combinationapproach. ForeachmacrovariableVi andforeachbankingmeasure, C, we estimate a simple regression. This regression takes the form: C = α+βC +γ Vi +γ Vi +γ Vi +γ Vi +u (1) t t−1 1 t−1 2 t−2 3 t−3 4 t−4 t We forecast out C from regression i according to the scheme CF,i = α+βCF,i +γ Vi +γ Vi +γ Vi +γ Vi +u (2) t+j t+j−1 1 t+j−1 2 t+j−2 3 t+j−3 4 t+j−4 t We construct the average forecast of each measure by taking the unweighted average of the forecasts of that measure across models: i(cid:88)=N CF,i CF = t+j (3) t+j N i=1 The macro variables we consider are: real GDP growth, unemployment rate (alternatively the change and level), the growth rate of the national house price index, the term spread, the growth rate of the S&P500 index, the implied volatility of the S&P500 index options (VIX), and the real interest rate. All of these variables were included in the baseline and stress scenarios produced by the Federal Reserve Board as part of the Comprehensive Capital Analysis and Review in both 2011 and 2012. WefocusoncomparingtheRootMeanSquaredError(RMSE)forourforecast combination models and for the forecast implied by a random walk (i.e., a no change forecast). According to the random walk model CF = C . (4) t+j t In the calculation of the RMSE, we take a pseudo-out-of-sample approach and keeptheestimationwindowconstantat40quarters. Startingwiththeestimation 5
window from 1986Q1 to 1995Q4 (allowing for a lag length of four quarters) we construct pseudo-out of sample forecasts going out nine quarters – the length of the planning horizon in the U.S. stress test exercise. We compute RMSEs at each forecast horizon, progressively moving the estimation window and using an assessment window which spans 1996q1 to 2011Q4. Because we want to mimic thestresstestingprocess,wetaketheevolutionofthemacrovariablesasobserved in the data. Similarly, within a stress test, the evolution of macro variables would be implied by a given scenario. When comparing the RMSE for the random walk and the RMSE for the forecast combination models, we test for equal predictive ability using the procedure recommended by Clark and West (2007). 2.1 Data The banking data is from the quarterly Consolidated Reports of Condition and Income(CallReport)thateverynational, statemember, andinsurednonmember bank is required to file on the last day of each quarter by the Federal Financial Institutions Examination Council (FFIEC). The Federal Deposit Insurance Corporation is tasked as the overseer, collecting and reviewing all submissions. Call Report data used in this analysis are cleaned and adjusted for bank mergers and acquisitions, using structure data from the National Information Clearinghouse (NIC) on mergers and acquisitions.5 Foreign entities are excluded and domestic subsidiaries are aggregated up to the parent, bank-holding-company (BHC), level. We aggregate our measures of banking conditions for the top 25 BHCs, as ranked by total assets, which is 5Bank balance sheet variables are adjusted for mergers between commercial banks by comparing balance sheet values at the end of the quarter with those at the beginning of the quarter, accounting for amounts acquired or lost during the period because of mergers. For information on the merger-adjustment procedure for income, see the appendix in English and Nelson (1998). 6
assessed quarterly. The banking data in our regressions start in 1985q1 and end in 2011q3. Quarterly flows for total net chargeoffs are expressed as a percentage of total loans and leases; quarterly flows for PPNR are expressed as a percentage of total assets. The tier-1 regulatory capital is expressed as a quarterly ratio of risk weighted assets; and, the net interest margin is expressed as the percentage of net interest income over interest earning assets. Aggregation to the level of bank holding company starting with data for commercial banks may introduce measurement error. As an alternative to the Call Report data we conduct our analysis using data from the FR Y-9C Consolidated Financial Statements for Bank Holding Companies (FR Y-9C). BHCs with total consolidatedassetsof$500millionormorearerequiredtofiletheFRY-9Creport on the last day of the quarter. The Federal Reserve acts as the overseer of this data, collecting, processing and publishing it. The FR Y-9C report is designed to parallel the Call Report in terms of the definition of data items. As a result, we are able to perform our analysis with consistent definitions across measures of banking conditions. The only deterrent to using FR Y-9C data instead of Call Report data is that the FR Y-9C data of our selected banking measures begins six years after the Call Report data counterpart. We present the results from our analysis using FR Y-9C data in Section 3.1. We refer to the macro factors used in the models as two groups of aggregate factors dubbed “macro” and “financial.” The macro group includes the unemployment rate, real GDP growth, and the growth in the house price index.6 The financial group includes a term spread measure, the growth of the S&P500 index, the S&P500 Volatility Index (VIX), and a short-term real interest rate.7 6The unemployment rate is from the Bureau of Labor Statistics; it is a quarterly average of monthly data. Real GDP is obtained from the National Income and Products Accounts; it is the log difference of the chained-weighted index. The house price index is from CoreLogic and is log-differenced. 7The term spread measure is the difference between the yield on a 10 year U.S. government bond and a 7
Most of the forecast combinations we consider include a forecast conditional on the unemployment rate. Figure 1 provides motivation for our interest in unemployment. The top panel shows percentile curves for the change in the unemployment rate over a horizon between 1 and 4 quarters. Remarkably, the bottom panel highlights that periods when the quarterly change in the unemployment rate is above the 75th percentile can pick up NBER recessions well ahead of the official NBER announcements. 3 Results Figures 2 to 5 allow a comparison of the RMSEs for different combinations of simple models with each figure focusing on a different measure of banking conditions. We consider eight different models. Model 1, the broadest, is a combination of forecasts conditional on two groups of variables, the macro group and the financial group. Model 5 only includes the macro group, while models 2 to 4 are intermediate models that eliminate one of the financial variables at a time. Model 6 and 7 pare down the macro variables. Finally, model 8 considers the performance of a forecast combination that includes only the financial group. Each figure shows two sets of results – we alternatively include the level or the change of the unemployment rate. In each figure we use a color scheme to facilitate the comparison of results. The lowest RMSEs at each horizon are shown against a deep green background, and the highest RMSEs are shown against a red background. Shades from green to orange are used for intermediate results. We determine when macro variables improve upon the random walk forecast using the test of Clark and West (2007). Under the null hypothesis, the random walk model is the data-generating process. Then parameters that are zero in 3-month U.S. treasury bill. The S&P500 index is obtained from Standard&Poors and log differenced. The VIX is from the Chicago Board Options Exchange. Finally the short-term real interest rate is the interest rate on a three-month U.S. treasury bill. 8
population are correctly set to zero in sample, implying a gain in efficiency. Conversely, the alternative model introduces noise into the forecasting process that inflates its RMSE in sample. Accordingly, Clark and West (2007) recommend a downward adjustment of the sample RMSE for the alternative hypothesis. Thus, it is possible to reject the null of equal predictive ability even when, in sample, the RMSE of the alternative hypothesis is higher than the RMSE of the random walk model. We use a one sided test. In the tables, the RMSEs for which we reject the null of equal predictive ability at the 5% significance level are highlighted in bold face. Figure2focusesonresultsfortotalnetchargeoffs. Model5,withthechangein unemployment,realGDP,andHPIhasthelowestRMSEatallhorizonsandbeats a random walk also at all horizons. Models 6 and 7, both include unemployment, butdropHPIandGDPinturn. Inbothcases,thecombinationforecaststillbeats a random walk forecast, based on RMSE, if a bit more modestly. In particular, HPI seems to help at reducing the RMSE at the shorter forecast horizons. These three models are consistent with the hypothesis that sudden changes in unemployment can reduce the ability of borrowers to repay their loans, resulting in substantial increases in chargeoffs. By contrast, the combination of models that includes financial variables only (model 8) has the worst performance in terms of RMSE – well above a random walk. Moreover, the figure shows that inclusionofthefinancialvariablessubstantiallyworsenstheforecastperformance. Figures 3 and 4 show results for our revenue measures, respectively PPNR and NIM. In this case, the broadest model, Model 1, performs best in terms of RMSE. Model 8, which had the highest RMSE for total net chargeoffs, displays a relatively good performance for PPNR and NIM. Overall, however, even the best-peforming models show more modest gains relative to a random walk than in the case of total net chargeoffs. In the case of PPNR, even the best performing 9
model does not beat a random walk at all horizons. Figure 5 focuses on tier-1 capital. Model 6, with the level of unemployment and GDP displays the lowest RMSE and beats the random walk forecast at all horizons. With the tier-1 capital ratio, the forecast combination that has the worst performance includes financial variables only. Overall, we were not able to beat a random walk across all horizons for all of the measures of banking conditions that we considered. The relative gains in RMSE were most pronounced for chargeoffs and modest for NIM and tier- 1 capital. Figure 6 shows the banking measures considered against the NBER recession dates. Total net chargeoffs show clear procyclicality. NIM and the tier- 1 capital ratio, while much less volatile, also show some increases in recessions. By contrast, pre-provision net income does not follow one pattern across the three recessions spanned by the data available. For instance, in the most recent recessions, pre-provision net income shows multiple peaks and troughs. Even when the forecast combination outperformed the random walk forecast, the best performing models we could formulate were still saddled with a substantial degree of forecast uncertainty. As an example, Figure 7 shows forecasts for each of the measures of banking conditions considered. The estimation sample ends in 2009Q2, leaving 9 quarters for the assessment window till the end of our sample in 2011Q3. Even when we do beat a random walk, the forecast uncertainty bands in Figure 7 imply a striking degree of uncertainty for each point forecast at different horizons even when compared to the abnormal variation observed in each series coinciding with the recent financial crisis. While we cannot claim to have formulated the most efficient forecast model possible for each of the measures of banking conditions considered, we interpret our results as a cautionary factor in the analysis of capital plans produced by bank holding companies as part of a stress test exercise. 10
3.1 Sensitivity Analysis We perform sensitivity analysis regarding several dimensions of the benchmark forecast exercise. To verify that aggregation of data for commercial banks in the Call Report dataset to the level of Bank Holding Company did not skew the benchmark results, we use alternative data from form FR-Y9C filings that does not require aggregation. Furthermore, we forecast alternative aggregate measures for the universe of U.S. commercial banks, instead of focusing on the largest U.S. bank holding companies. We consider a shorter sample that ends before the recent financial crisis. Finally, we consider sensitivity to alternative choices for the size of the estimation window – in turn 60, or 80 quarters, instead of 40 quarters in the benchmark results. In all cases, to conserve space, we focus on chargeoffs on loans and leases and do not report sensitivity results for the additional measures of banking conditions considered above. Figure 8 shows results for the Form Y9C dataset.8 Figure 9 shows results for an aggregate measure of chargeoffs for all U.S. commercial banks using Call Report data. In both cases, model number 5 with unemployment, GDP, and HPI all in differences remains the best performing forecast combination and the model that includes all the variables in our financial group the worst. Moreover, the performance of the best model still beats that of the random walk model in terms of lower RMSE. We conclude that neither the aggregation procedure to bank holding company in the Call Report dataset, nor consideration of the top 25 bank holding companies only skews our benchmark results. Figure 10 shows RMSEs based on a sample of data and assessment window that stop before the recent financial crisis. The last 40-quarter estimation sample considered ends in 2005Q3, leaving 9 quarters for the last assessment window spanning from 2005Q4 to 2007Q4. Even the best forecast combination model – still model 5 – fails to beat a random walk in terms of RMSE at all of the 8Our Y9C sample starts in 1997Q1 and ends in 2011Q3. 11
horizons considered. The deterioration in performance is more marked for model number 7, that includes only the unemployment rate and HPI. We conclude that the inclusion of HPI brings about an important improvement in performance of the forecast combination especially when considering a sample that includes the recent financial crisis. Finally figures 11 and 12 show results for a rolling estimation sample of 60 and 80 quarters, respectively, instead of 40 quarters in the benchmark. We conclude that the main results in the benchmark experiment continue to hold with these alternative estimation windows. 3.2 Sensitivity of Forecasts conditional on CCAR macro scenarios In this section we return to the original motivation for our forecast comparisons, the application to macro stress testing. We generate forecasts of our four measures of banking conditions conditional on the macro scenarios included in the most recent stress test for bank holding companies conducted by the Federal Reserve, CCAR 2012. Two scenarios were included, a baseline scenario, and a severe stress scenario. The stress scenario is meant to represent “highly adverse conditions”, while the baseline is meant to capture “expected economic conditions.”9 Considering both scenarios allows us to assess the relative sensitivity of the banking conditions forecasts to the baseline and stress CCAR scenarios. To construct the forecasts conditional on CCAR scenarios, the estimation sample ends in 2011q3. Each CCAR scenario includes all the macro variables needed by the alternative models considered in the previous sections. The scenarios extend 13 quarters out. The forecasts we present stop 9 quarters out, as the bank holding companies in the stress test are only required to produce capital 9FormoreinformationonthedesignoftheFederalReserveCCAR2012scenariosseeBoardofGovernors of the Federal Reserve System (2011). 12
plans for the next 9 quarters. Figure 13 shows dynamic forecasts conditional on either the stress scenario or the baseline scenario for each of the measures of banking conditions considered. For each measure we selected the best performing model out of the 8 models assessed above.10 The figure also shows a 2-RMSE uncertainty band centered around each forecast. OneofthestrikingfeaturesthatemergefromFigure13isthattheuncertainty bands for the baseline scenario encompass the point forecasts conditional on the stress scenarios. Only in the case of total net chargeoffs and tier-1 capital ratio do the point forecast veer outside of the uncertainty bands towards the end of the capital planning horizon. It is also interesting to consider the sensitivity of the point forecast to the different scenarios. The difference between the point forecasts for PPNR and NIM is modest. By contrast, for total net chargeoffs and tier-1 capital, going 9 quarters out, the difference between the point forecasts for the baseline and stress scenarios is sizable. In both cases, it is about half of the increase observed during the recent financial crisis. However, notice that Tier-1 capital is predicted to increase in a severe recession. This is in accordance with the pattern observed in the data. As shown in Figure 6, the tier-1 capital ratio increased during each of the three recessions for which we have data from the Call Report dataset. 4 Conclusion Thispapercontributestotheempiricalunderpinningsofstresstestexercises. For some but not all measures of aggregate banking conditions, forecasts conditional on macro variables outperform random walk forecasts in terms of root mean 10The models we used for chargeoffs, PPNR, NIM, and tier-1 capital are, respectively: model 5 with unemployment in differences, model 1 with unemployment in differences, model 1 with unemployment in differences, and model 6 with unemployment in levels. 13
squared errors. The largest gains are for total chargeoffs on loans and leases. We found relatively more modest gains for net interest income and the tier-1 capital ratio. However, even our best performing model did not beat a random walk at all horizons for Pre Provision Net Income. Regardless of the gains, we find large RMSEs for the forecasts of all the measures of banking conditions. The RMSEs are large even when compared to the large and abnormal variation for each of the series during the recent financial crisis. When we apply our preferred forecast models to macro scenarios used in most recent stress test conducted by the Federal Reserve, CCAR 2012, we find little sensitivity of the banking measures to scenarios that are meant to capture large macroeconomic differences. In all cases, for most of 9-quarter forecast horizon, the point estimates for the stress scenario are inside the 2-RMSE uncertainty bands around the forecast conditional on the baseline scenario. We cannot claim to have formulated the most efficient forecast model possible foreachofthemeasuresofbankingperformanceconsidered. Indeed,wehaveused only publicly available data, while regulators and each bank holding company have access to a greater wealth of information. Nonetheless, we interpret our results as a cautionary factor in the analysis of capital plans produced by bank holding companies as part of a stress test exercise. At the very least, our results highlightthatregulatorsmayfinditdifficulttoexplaintheirjudgmentofdifferent bankholdingcompaniestooutsideobserversbyrelyingexclusivelyonpublicdata. 14
References Bates, J. M. and C. W. J. Granger (1969). The combination of forecasts. Operations Research Quarterly 20, 451–68. Bellotti, T. and J. Crook (2009). Credit scoring with macroeconomic variables using survival analysis. Journal of the Operations Research Society 60(12), 1699–1707. Blaschke, W., M. T. Jones, G. Majnoni, and S. Peria (2001). Stress testing of financial systems: an overview of issues, methodologies, and fsap experiences. IMF Working Paper, 1. Board of Governors of the Federal Reserve System (2011). Comprehensive capital analysis and review 2012: Methodology and results for stress scenario projections. Bunn, P., M. Drehmann, and A. Cunnigham (2005). Stress testing as a tool for assessing systemic risk. Bank of England Financial Stability Review. Clark, T. and K. West (2007). Approximately normal tests for equal predictive accuracy in nested models. Journal of Econometrics 138, 291–311. Covas, F., B. Rump, and E. Zakrajsek (2012). Stress-testing u.s. bank holding companies: A dynamic panel quantile regression approach. Mimeo, Federal Reserve Board. Crook, J. and J. Banasik (2012). Forecasting and explaining aggregate consumer credit delinquency behaviour. International Journal of Forecasting 28, 145–160. Estrella, A. and G. A. Hardouvelis (1991). The term structure as predictor of real economic activity. Journal of Finance 46(2), 555–576. Estrella, A. and F. Mishkin (1998). Predicting u.s. recessions: Financial variables as leading indicators. Review of Economics and Statistics 80(1), 45– 61. 15
Faust, J. and J. Wright (2008). Comparing greenbook and reduced form forecasts using a large realtime dataset. Journal of Business and Economic Statistics 146, 293–303. Kalirai,H.andM.Schleicher(2002).Macroeconomicstesstesting: Preliminary evidence from austria. OENB Financial Stability Report number 3. Meese, R. A. and K. Rogoff (1983). Empirical exchange rate models of the seventies: Dotheyfitourofsample.Journal of International Economics 14, 3–24. Quagliariello, M. (Ed.) (2009). Stress-Testing the Banking System. Cambridge University Press. Stock,J.H.andM.W.Watson(2004).Combinationforecastsofoutputgrowth in a seven-country data set. Journal of Forecasting 23(6), 405–430. 16
Figure 1: The Level and Changes in the U.S. Unemployment Rate and NBER Recessions 17
Figure 2: Total Net Chargeoffs: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.046 0.046 0.045 0.044 0.044 0.042 0.045 0.044 0.051 2 step 0.073 0.071 0.068 0.067 0.065 0.060 0.064 0.062 0.084 3 step 0.099 0.093 0.088 0.084 0.081 0.074 0.079 0.075 0.116 4 step 0.125 0.119 0.112 0.106 0.100 0.090 0.095 0.091 0.153 5 step 0.153 0.146 0.136 0.129 0.120 0.106 0.110 0.107 0.190 6 step 0.177 0.172 0.160 0.152 0.142 0.121 0.123 0.121 0.228 7 step 0.199 0.198 0.182 0.175 0.165 0.135 0.136 0.136 0.264 8 step 0.217 0.223 0.203 0.198 0.187 0.149 0.148 0.150 0.300 9 step 0.230 0.216 0.201 0.195 0.187 0.160 0.159 0.162 0.272 Average forecast 0.153 0.149 0.139 0.134 0.128 0.111 0.112 0.113 0.190 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.046 0.047 0.046 0.046 0.046 0.045 0.049 0.048 0.051 2 step 0.073 0.073 0.070 0.069 0.068 0.064 0.069 0.068 0.084 3 step 0.099 0.095 0.090 0.087 0.085 0.078 0.085 0.083 0.116 4 step 0.125 0.122 0.114 0.110 0.105 0.095 0.102 0.099 0.153 5 step 0.153 0.148 0.138 0.132 0.124 0.111 0.117 0.115 0.190 6 step 0.177 0.174 0.162 0.155 0.145 0.126 0.131 0.130 0.228 7 step 0.199 0.200 0.184 0.177 0.167 0.140 0.145 0.145 0.264 8 step 0.217 0.224 0.205 0.200 0.189 0.154 0.157 0.159 0.300 9 step 0.230 0.218 0.204 0.198 0.191 0.166 0.169 0.171 0.272 Average forecast 0.153 0.150 0.141 0.136 0.130 0.116 0.119 0.120 0.190 Estimation Sample: Rolling window of 40 qtrs starting in 1986q1, ending in 2009q3. Assessment window: 1996q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 18
Figure 3: Pre-Provision Net Revenue: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.073 0.067 0.067 0.068 0.068 0.068 0.070 0.066 0.068 2 step 0.074 0.071 0.071 0.073 0.074 0.073 0.077 0.071 0.072 3 step 0.086 0.079 0.079 0.081 0.082 0.081 0.086 0.078 0.081 4 step 0.087 0.083 0.083 0.085 0.086 0.084 0.094 0.080 0.084 5 step 0.089 0.087 0.087 0.090 0.090 0.089 0.099 0.086 0.087 6 step 0.096 0.090 0.090 0.092 0.093 0.092 0.103 0.090 0.090 7 step 0.092 0.092 0.093 0.096 0.096 0.096 0.108 0.094 0.091 8 step 0.103 0.095 0.097 0.099 0.099 0.100 0.111 0.097 0.093 9 step 0.092 0.096 0.098 0.100 0.100 0.101 0.113 0.098 0.095 Average forecast 0.070 0.064 0.065 0.067 0.067 0.067 0.076 0.064 0.063 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.073 0.067 0.067 0.068 0.069 0.069 0.071 0.068 0.068 2 step 0.074 0.072 0.072 0.074 0.075 0.074 0.079 0.073 0.072 3 step 0.086 0.080 0.081 0.083 0.084 0.083 0.090 0.082 0.081 4 step 0.087 0.084 0.084 0.087 0.088 0.087 0.098 0.084 0.084 5 step 0.089 0.089 0.089 0.091 0.092 0.092 0.104 0.091 0.087 6 step 0.096 0.091 0.092 0.094 0.095 0.096 0.108 0.095 0.090 7 step 0.092 0.094 0.095 0.098 0.099 0.100 0.113 0.100 0.091 8 step 0.103 0.096 0.099 0.101 0.103 0.104 0.118 0.104 0.093 9 step 0.092 0.098 0.101 0.103 0.104 0.106 0.120 0.106 0.095 Average forecast 0.070 0.066 0.067 0.069 0.070 0.071 0.082 0.071 0.063 Estimation Sample: Rolling window of 40 qtrs starting in 1986q1, ending in 2009q3. Assessment window: 1996q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 19
Figure 4: Net Interest Margin: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.023 0.023 0.023 0.024 0.024 0.024 0.025 0.025 0.023 2 step 0.030 0.029 0.030 0.030 0.030 0.031 0.034 0.031 0.030 3 step 0.037 0.035 0.036 0.036 0.036 0.038 0.042 0.038 0.037 4 step 0.046 0.042 0.043 0.043 0.043 0.045 0.050 0.046 0.044 5 step 0.049 0.043 0.043 0.043 0.044 0.045 0.050 0.049 0.045 6 step 0.054 0.047 0.047 0.047 0.049 0.050 0.055 0.055 0.049 7 step 0.057 0.050 0.051 0.052 0.054 0.055 0.060 0.061 0.052 8 step 0.061 0.054 0.054 0.056 0.059 0.060 0.065 0.069 0.056 9 step 0.065 0.061 0.061 0.063 0.067 0.068 0.072 0.077 0.062 Average forecast 0.045 0.039 0.039 0.040 0.041 0.042 0.047 0.045 0.041 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.023 0.023 0.023 0.024 0.024 0.024 0.024 0.025 0.023 2 step 0.030 0.030 0.030 0.031 0.031 0.031 0.034 0.032 0.030 3 step 0.037 0.036 0.036 0.037 0.037 0.038 0.042 0.040 0.037 4 step 0.046 0.043 0.043 0.045 0.045 0.046 0.051 0.049 0.044 5 step 0.049 0.043 0.043 0.045 0.046 0.047 0.051 0.052 0.045 6 step 0.054 0.047 0.047 0.049 0.050 0.050 0.055 0.057 0.049 7 step 0.057 0.050 0.050 0.053 0.055 0.055 0.059 0.064 0.052 8 step 0.061 0.054 0.053 0.057 0.060 0.060 0.063 0.071 0.056 9 step 0.065 0.060 0.059 0.063 0.066 0.066 0.067 0.078 0.062 Average forecast 0.045 0.039 0.039 0.041 0.042 0.042 0.046 0.047 0.041 Estimation Sample: Rolling window of 40 qtrs starting in 1986q1, ending in 2009q3. Assessment window: 1996q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 20
Figure 5: Tier-1 Capital Ratio: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.211 0.174 0.163 0.164 0.162 0.152 0.153 0.154 0.202 2 step 0.388 0.320 0.296 0.292 0.286 0.268 0.264 0.269 0.375 3 step 0.565 0.464 0.435 0.430 0.421 0.404 0.411 0.404 0.528 4 step 0.738 0.614 0.587 0.585 0.573 0.565 0.575 0.569 0.674 5 step 0.893 0.755 0.731 0.732 0.717 0.719 0.729 0.728 0.807 6 step 1.045 0.897 0.878 0.884 0.869 0.880 0.888 0.898 0.938 7 step 1.189 1.042 1.023 1.032 1.016 1.032 1.038 1.064 1.083 8 step 1.322 1.179 1.159 1.169 1.155 1.171 1.168 1.221 1.222 9 step 1.440 1.317 1.298 1.311 1.296 1.304 1.230 1.380 1.339 Average forecast 0.971 0.851 0.833 0.834 0.822 0.829 0.806 0.852 0.889 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.211 0.174 0.163 0.164 0.162 0.152 0.150 0.153 0.202 2 step 0.388 0.320 0.297 0.292 0.288 0.268 0.259 0.269 0.375 3 step 0.565 0.463 0.434 0.428 0.420 0.399 0.395 0.395 0.528 4 step 0.738 0.609 0.583 0.577 0.566 0.551 0.543 0.546 0.674 5 step 0.893 0.745 0.721 0.718 0.701 0.692 0.675 0.687 0.807 6 step 1.045 0.882 0.861 0.861 0.842 0.839 0.810 0.836 0.938 7 step 1.189 1.021 1.000 1.001 0.981 0.980 0.940 0.985 1.083 8 step 1.322 1.154 1.130 1.132 1.111 1.108 1.050 1.129 1.222 9 step 1.440 1.287 1.266 1.269 1.246 1.235 1.098 1.279 1.339 Average forecast 0.971 0.840 0.820 0.817 0.803 0.799 0.749 0.808 0.889 Estimation Sample: Rolling window of 40 qtrs starting in 1991q1, ending in 2009q3. Assessment window: 2001q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 21
Figure 6: Measures of Banking Conditions and NBER Recessions 22
Figure 7: Conditional Forecasts of Banking Conditions and Confidence Intervals∗ Total Net Charge Offs 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 2006 2007 2008 2009 2010 2011 tnecreP Pre Provision Net Revenue 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 2006 2007 2008 2009 2010 2011 tnecreP Net Interest Income 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 2006 2007 2008 2009 2010 2011 tnecreP Tier 1 Capital Ratio 12 10 8 6 4 2 0 2006 2007 2008 2009 2010 2011 tnecreP ∗ The solid lines show observed data. The dashed lines show conditional forecasts. For chargeoffs we used model 5 with unemployment in differences. For PPNR and NIM we used model 1 with unemployment in differences; and for Tier 1 Capital ratio we used model 6 with unemployment in levels. The dotted lines show a 1-RMSE confidence band around the point estimate. The dashed-dotted lines show a 2-RMSE confidence band around the point estimate. 23
Figure 8: Total Net Chargeoffs – Using FR-Y9C Form Data: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.100 0.095 0.096 0.094 0.089 0.093 0.105 0.091 0.100 2 step 0.166 0.161 0.156 0.148 0.139 0.136 0.155 0.132 0.187 3 step 0.238 0.235 0.221 0.211 0.197 0.185 0.201 0.182 0.284 4 step 0.297 0.298 0.277 0.266 0.247 0.224 0.235 0.221 0.372 5 step 0.340 0.349 0.325 0.314 0.290 0.259 0.267 0.254 0.445 6 step 0.382 0.402 0.376 0.362 0.333 0.292 0.295 0.286 0.517 7 step 0.401 0.433 0.401 0.390 0.359 0.306 0.308 0.303 0.567 8 step 0.402 0.452 0.414 0.406 0.376 0.314 0.320 0.312 0.601 9 step 0.359 0.410 0.379 0.376 0.354 0.311 0.328 0.303 0.510 Average forecast 0.285 0.313 0.292 0.284 0.267 0.239 0.250 0.235 0.394 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.100 0.095 0.096 0.094 0.089 0.094 0.105 0.092 0.100 2 step 0.166 0.163 0.157 0.149 0.140 0.138 0.157 0.135 0.187 3 step 0.238 0.237 0.223 0.214 0.200 0.188 0.205 0.188 0.284 4 step 0.297 0.300 0.279 0.269 0.251 0.228 0.241 0.228 0.372 5 step 0.340 0.351 0.327 0.316 0.293 0.263 0.273 0.260 0.445 6 step 0.382 0.402 0.377 0.364 0.335 0.295 0.302 0.292 0.517 7 step 0.401 0.433 0.402 0.390 0.359 0.310 0.316 0.310 0.567 8 step 0.402 0.452 0.414 0.407 0.377 0.319 0.331 0.320 0.601 9 step 0.359 0.411 0.381 0.379 0.358 0.318 0.342 0.316 0.510 Average forecast 0.285 0.314 0.293 0.286 0.269 0.243 0.256 0.241 0.394 Estimation Sample: Rolling window of 40 qtrs starting in 1998q1, ending in 2009q3. Assessment window: 2008q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 24
Figure 9: Total Net Chargeoffs – Aggregate of All U.S. Commercial Banks: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.045 0.043 0.043 0.042 0.041 0.042 0.044 0.043 0.046 2 step 0.067 0.061 0.059 0.057 0.055 0.052 0.055 0.053 0.072 3 step 0.092 0.082 0.078 0.075 0.072 0.068 0.073 0.069 0.102 4 step 0.114 0.102 0.095 0.091 0.086 0.079 0.085 0.080 0.131 5 step 0.140 0.126 0.117 0.112 0.106 0.098 0.102 0.097 0.163 6 step 0.162 0.149 0.137 0.131 0.124 0.109 0.112 0.108 0.196 7 step 0.183 0.173 0.157 0.153 0.145 0.122 0.124 0.121 0.229 8 step 0.199 0.193 0.173 0.171 0.164 0.133 0.133 0.132 0.260 9 step 0.211 0.191 0.177 0.174 0.168 0.145 0.144 0.143 0.238 Average forecast 0.141 0.130 0.120 0.117 0.113 0.101 0.102 0.100 0.165 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.045 0.043 0.044 0.043 0.042 0.043 0.045 0.045 0.046 2 step 0.067 0.062 0.060 0.058 0.057 0.054 0.058 0.057 0.072 3 step 0.092 0.084 0.080 0.077 0.074 0.071 0.077 0.074 0.102 4 step 0.114 0.104 0.097 0.093 0.089 0.083 0.091 0.086 0.131 5 step 0.140 0.128 0.120 0.115 0.110 0.102 0.109 0.105 0.163 6 step 0.162 0.151 0.139 0.134 0.128 0.114 0.120 0.116 0.196 7 step 0.183 0.175 0.159 0.156 0.149 0.128 0.133 0.131 0.229 8 step 0.199 0.195 0.176 0.174 0.168 0.139 0.144 0.142 0.260 9 step 0.211 0.194 0.180 0.178 0.173 0.151 0.155 0.154 0.238 Average forecast 0.141 0.132 0.122 0.120 0.115 0.105 0.108 0.106 0.165 Estimation Sample: Rolling window of 40 qtrs starting in 1986q1, ending in 2009q3. Assessment window: 1996q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 25
Figure 10: Total Net Chargeoffs – Shortened Sample Ending in 2007Q4: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.037 0.041 0.040 0.040 0.040 0.038 0.039 0.041 0.046 2 step 0.051 0.058 0.058 0.057 0.057 0.051 0.049 0.055 0.068 3 step 0.064 0.069 0.067 0.066 0.065 0.058 0.056 0.062 0.082 4 step 0.083 0.084 0.083 0.079 0.077 0.071 0.070 0.074 0.101 5 step 0.108 0.100 0.097 0.090 0.088 0.081 0.080 0.086 0.120 6 step 0.137 0.120 0.115 0.107 0.105 0.098 0.094 0.100 0.144 7 step 0.165 0.140 0.134 0.125 0.121 0.112 0.105 0.115 0.167 8 step 0.192 0.159 0.153 0.143 0.139 0.128 0.117 0.132 0.188 9 step 0.220 0.179 0.174 0.164 0.158 0.145 0.135 0.148 0.210 Average forecast 0.132 0.114 0.111 0.105 0.102 0.095 0.089 0.098 0.132 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.037 0.042 0.042 0.042 0.043 0.041 0.044 0.046 0.046 2 step 0.051 0.060 0.060 0.060 0.061 0.055 0.056 0.062 0.068 3 step 0.064 0.071 0.070 0.069 0.069 0.062 0.062 0.069 0.082 4 step 0.083 0.087 0.085 0.082 0.082 0.075 0.076 0.082 0.101 5 step 0.108 0.103 0.100 0.093 0.092 0.086 0.085 0.093 0.120 6 step 0.137 0.123 0.118 0.110 0.109 0.102 0.099 0.108 0.144 7 step 0.165 0.142 0.137 0.128 0.125 0.116 0.111 0.123 0.167 8 step 0.192 0.162 0.156 0.146 0.143 0.132 0.123 0.139 0.188 9 step 0.220 0.182 0.177 0.167 0.162 0.149 0.141 0.156 0.210 Average forecast 0.132 0.116 0.113 0.107 0.105 0.099 0.094 0.105 0.132 Estimation Sample: Rolling window of 40 qtrs starting in 1986q1, ending in 2007q4. Assessment window: 1996q1 to 2010q1 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 26
Figure 11: Total Net Chargeoffs – 60 Quarter Estimation Window: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.053 0.048 0.048 0.046 0.046 0.048 0.051 0.049 0.051 2 step 0.088 0.078 0.075 0.073 0.071 0.072 0.077 0.072 0.088 3 step 0.121 0.108 0.102 0.098 0.094 0.092 0.097 0.094 0.129 4 step 0.153 0.144 0.136 0.132 0.125 0.117 0.117 0.120 0.177 5 step 0.185 0.183 0.173 0.167 0.159 0.144 0.139 0.147 0.226 6 step 0.214 0.222 0.210 0.203 0.192 0.171 0.156 0.175 0.277 7 step 0.241 0.259 0.244 0.235 0.224 0.195 0.172 0.200 0.326 8 step 0.263 0.293 0.274 0.264 0.253 0.215 0.186 0.221 0.373 9 step 0.280 0.286 0.270 0.264 0.254 0.228 0.199 0.234 0.346 Average forecast 0.187 0.191 0.181 0.176 0.169 0.154 0.144 0.158 0.232 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.053 0.048 0.048 0.046 0.046 0.048 0.051 0.049 0.051 2 step 0.088 0.078 0.075 0.073 0.071 0.072 0.078 0.073 0.088 3 step 0.121 0.108 0.103 0.100 0.096 0.094 0.099 0.096 0.129 4 step 0.153 0.145 0.137 0.133 0.127 0.119 0.120 0.122 0.177 5 step 0.185 0.184 0.173 0.168 0.160 0.146 0.142 0.149 0.226 6 step 0.214 0.222 0.210 0.203 0.193 0.172 0.159 0.176 0.277 7 step 0.241 0.259 0.243 0.235 0.224 0.195 0.175 0.200 0.326 8 step 0.263 0.292 0.273 0.263 0.252 0.214 0.188 0.220 0.373 9 step 0.280 0.285 0.269 0.263 0.253 0.227 0.200 0.232 0.346 Average forecast 0.187 0.191 0.182 0.176 0.170 0.155 0.146 0.159 0.232 Estimation Sample: Rolling window of 60 qtrs starting in 1986q1, ending in 2009q3. Assessment window: 2001q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 27
Figure 12: Total Net Chargeoffs – 80 Quarter Estimation Window: Root Mean Squared Errors for Forecast Combinations and for Random Walk Model A: Forecast combinations using the change in unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.066 0.073 0.068 0.069 0.066 0.059 0.052 0.060 0.088 2 step 0.116 0.135 0.123 0.125 0.118 0.106 0.093 0.105 0.162 3 step 0.168 0.188 0.175 0.178 0.168 0.152 0.135 0.152 0.222 4 step 0.218 0.228 0.216 0.220 0.208 0.188 0.167 0.190 0.266 5 step 0.265 0.260 0.249 0.253 0.241 0.219 0.196 0.222 0.300 6 step 0.309 0.285 0.273 0.276 0.263 0.238 0.214 0.241 0.327 7 step 0.350 0.304 0.290 0.292 0.278 0.252 0.227 0.256 0.349 8 step 0.385 0.317 0.303 0.302 0.289 0.262 0.237 0.268 0.364 9 step 0.415 0.336 0.321 0.319 0.305 0.276 0.250 0.284 0.386 Average forecast 0.270 0.242 0.230 0.230 0.218 0.197 0.176 0.199 0.279 B: Forecast combinations using the level of unemployment Root Mean Squared Error Random Walk (1) (2) (3) (4) (5) (6) (7) (8) 1 step 0.066 0.073 0.067 0.068 0.065 0.059 0.051 0.058 0.088 2 step 0.116 0.133 0.121 0.123 0.115 0.102 0.088 0.099 0.162 3 step 0.168 0.185 0.172 0.174 0.163 0.145 0.125 0.143 0.222 4 step 0.218 0.225 0.212 0.215 0.202 0.179 0.153 0.177 0.266 5 step 0.265 0.256 0.244 0.247 0.233 0.208 0.179 0.205 0.300 6 step 0.309 0.280 0.267 0.269 0.254 0.226 0.195 0.223 0.327 7 step 0.350 0.299 0.284 0.284 0.269 0.238 0.207 0.236 0.349 8 step 0.385 0.311 0.296 0.294 0.278 0.246 0.215 0.245 0.364 9 step 0.415 0.330 0.313 0.309 0.292 0.259 0.224 0.258 0.386 Average forecast 0.270 0.239 0.227 0.225 0.212 0.189 0.164 0.188 0.279 Estimation Sample: Rolling window of 80 qtrs starting in 1986q1, ending in 2009q3. Assessment window: 2006q1 to 2011q3 Bold indicates that model has superior predictability relative to the Random Walk model at that horizon, according to test suggested by Clark and West (2007) VIX data (models 1 and 2) begin in 1990q1 Key to models 1. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX, real interest rate 2. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog), VIX 3. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread, SP500(dlog) 4. (4 lags): Unemployment, real GDP (dlog), HPI(dlog), term-spread 5. (4 lags): Unemployment, real GDP (dlog), HPI(dlog) 6. (4 lags): Unemployment, real GDP (dlog) 7. (4 lags): Unemployment, HPI(dlog) 8. (4 lags): Term-spread, SP500(dlog), VIX, RRINT 28
Figure 13: Forecasts and Confidence Intervals of Banking Conditions Conditional on CCAR 2012 Macro Scenarios∗ Total Net Charge Offs 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 2006 2008 2010 2012 tnecreP Pre Provision Net Revenue 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 2006 2008 2010 2012 tnecreP Net Interest Income 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 2006 2008 2010 2012 tnecreP Tier 1 Capital Ratio 18 16 14 12 10 8 6 4 2 0 2006 2008 2010 2012 tnecreP ∗ The solid lines show observed data. The dashed lines show forecasts conditional on the CCAR stress scenario; the dotted lines show forecasts conditional on the CCAR baseline scenario. For chargeoffs we used model 5 with unemployment in differences. For PPNR and NIM we used model 1 with unemployment in differences; and for Tier 1 Capital ratio we used model 6 with unemployment in levels. The dash-dot lines show a 2-RMSE confidence band around the point estimate of the forecast conditional on the stress scenario. The dot-point lines show a 2-RMSE confidence band around the point estimate of the forecast conditional on the baseline scenario. 29
Cite this document
Luca Guerrieri and Michelle Welch (2012). Can Macro Variables Used in Stress Testing Forecast the Performance of Banks? (FEDS 2012-49). Board of Governors of the Federal Reserve System, Finance and Economics Discussion Series. https://whenthefedspeaks.com/doc/feds_2012-49
@techreport{wtfs_feds_2012_49,
author = {Luca Guerrieri and Michelle Welch},
title = {Can Macro Variables Used in Stress Testing Forecast the Performance of Banks?},
type = {Finance and Economics Discussion Series},
number = {2012-49},
institution = {Board of Governors of the Federal Reserve System},
year = {2012},
url = {https://whenthefedspeaks.com/doc/feds_2012-49},
abstract = {When stress tests for the banking sector use a macroeconomic scenario, an unstated premise is that macro variables should be useful factors in forecasting the performance of banks. We assess whether variables such as the ones included in stress tests for U.S. bank holding companies help improve out of sample forecasts of chargeoffs on loans, revenues, and capital measures, relative to forecasting models that exclude a role for macro factors. Using only public data on bank performance, we find the macro variables helpful, but not for all measures. Moreover, even our best-performing models imply bands of uncertainty around the forecasts so large as to make it challenging to distinguish the implications of alternative macro scenarios.},
}