Benchmarking Operational Risk Models

Finance and Economics Discussion Series Divisions of Research & Statistics and Monetary Affairs Federal Reserve Board, Washington, D.C. Benchmarking Operational Risk Models Filippo Curti, Ibrahim Ergen, Minh Le, Marco Migueis, and Robert Stewart 2016-070 Please cite this paper as: Curti, Filippo, Ibrahim Ergen, Minh Le, Marco Migueis, and Robert Stewart (2016). “Benchmarking Operational Risk Models,” Finance and Economics Discussion Series 2016-070. Washington: Board of Governors of the Federal Reserve System, http://dx.doi.org/10.17016/FEDS.2016.070. NOTE: Staff working papers in the Finance and Economics Discussion Series (FEDS) are preliminary materials circulated to stimulate discussion and critical comment. The analysis and conclusions set forth are those of the authors and do not indicate concurrence by other members of the research staff or the Board of Governors. References in publications to the Finance and Economics Discussion Series (other than acknowledgement) should be cleared with the author(s) to protect the tentative character of these papers.

Benchmarking Operational Risk Models Filippo Curti, Ibrahim Ergen, Minh Le, Marco Migueis, and Robert Stewart ∗ March 2, 2016 Abstract The 2004 Basel II accord requires internationally active banks to hold regulatory capital for operational risk, and the Federal Reserve’s Comprehensive Capital Analysis and Review (CCAR) requires banks to project operational risk losses under stressed scenarios. As a result, banks subject to these rules have measured and managed operational risk more rigorously. But some types of operational risk particularly legal risk - are challenging to model because such exposures tend to be fat-tailed. Tail operational risk losses have significantly impacted banks’ balance sheets and income statements, even post crisis. So, operational risk practitioners, bank analysts, and regulators must develop reasonable methods to assess the efficacy of operational risk models and associated equity financing. We believe benchmarks should be used extensively to justify model outputs, improve model stability, and maintain capital reasonableness. Since any individual benchmark can be misleading, we outline a set of principles for using benchmarks effectively and describe how these principles can be applied to operational risk models. Also, we provide some examples of the benchmarks that have been used by US regulators in assessing Advanced Measurement Approach (AMA) capital reasonableness and that can be used in CCAR to assess the reasonableness of operational risk loss projections. We believe no single model’s output and no single benchmark offers a comprehensive view, but that practitioners, analysts, and regulators must use models combined with rigorous benchmarks to determine operational risk capital and assess its adequacy. JEL Classification: G21, G28 Keywords: Banking Regulation, Risk Management, Operational Risk, Benchmarking ∗Filippo Curti and Ibrahim Ergen are from the Federal Reserve Bank of Richmond and can be reached at Filippo.Curti@rich.frb.org and Ibrahim.Ergen@rich.frb.org, respectively. Minh Le is from the Federal ReserveBankofClevelandandcanbereachedatMinh.B.Le@clev.frb.org. MarcoMigueisisfromtheBoard ofGovernorsoftheFederalReserveSystemandcanbereachedatMarco.A.Migueis@frb.gov. RobertStewart isfromtheFederalReserveBankofChicagoandcanbereachedatRobert.T.Stewart@chi.frb.org. Wethank AzamatAbdymomunov, BakhodirErgashev, MikeGibson, DavidJones, PaulRallo, andparticipantsatthe Governance, Compliance, and Operational Risk (GCOR) Conference of the Risk Management Association (2015)forhelpfulcomments. Theviewsexpressedinthepaperarethoseoftheauthorsanddonotnecessarily represent those of the Board of Governors of the Federal Reserve System, the Federal Reserve Bank of Chicago, the Federal Reserve Bank of Cleveland, or the Federal Reserve Bank of Richmond.

1 Introduction US banks with total assets greater than $250 billion or foreign exposure greater than $10 billion are required to use the advanced measurement approach (AMA) to compute operational risk loss exposure at the 99.9% confidence level according to the final Basel rule.1 To estimate operational risk loss exposure, most US banks subject to AMA use the loss distribution approach (LDA). With the LDA, loss frequency and severity are modeled separately and then convoluted through Monte Carlo simulation or analytical techniques, such as Panjer recursion.2 Modeling high quantiles of a distribution is challenging even when data is plentiful and the distribution is well behaved; the challenges increase when modeling large losses that occur infrequently as is often the case with operational risk losses - particularly legal losses - as documented by Danielsson (2002) and Danielsson (2008). Many researchers have documented the uncertainty around operational risk capital estimates measured at the 99.9th quantile. Mignola and Ugoccioni (2006) find that LDA estimates of the 99.9th quantile suffer from considerable uncertainty due to data scarcity and the fat-tailed nature of operational losses. Similarly, Cope et al. (2009) find that AMA estimates suffer from significant uncertainty and criticize the Basel standard for operational risk due to challenges in extrapolating beyond the empirical data to the 99.9th quantile. The challenges of estimating the 99.9th quantile were also addressed by Neˇslehov´a et al. (2006), who highlighted the potential pitfalls with the application of extreme value theory (EVT) to operational risk when parameter estimates result in infinite mean models. Finally, Opdyke and Cavallo (2012) detail some of the challenges of modeling the 99.9th quantile when faced with truncated data and using maximum likelihood estimation. In addition to Basel regulatory capital requirements, the Federal Reserve requires bank 1The rule can be found at 12 CFR part 3 for the Office of the Comptroller of the Currency (OCC) and 12 CFR part 217 for the Board of Governors of the Federal Reserve System. 2Panjer (1981). 1

holding companies (BHCs) with assets exceeding $50 billion to conduct annual stress tests: the Comprehensive Capital Analysis and Review (CCAR). As part of CCAR stress testing, BHCs are required to project operational risk losses over a nine-quarter time horizon assuming various economic scenarios. But BHCs have struggled to find meaningful relationships between operational risk losses and the macroeconomy. The uncertainty associated with estimating links between the macroeconomy and operational risk is inherently high due to large and infrequent loss events dominating operational risk exposure. This uncertainty is compounded by the short length of the datasets used for estimation and by the difficulty in defining appropriate dates for operational loss events. Some operational loss event types, such as internal fraud, occur over extended periods of time; while other operational loss event types, such as legal cases, may result in payouts years after the occurrence date. Therefore, the operational risk models used in stress testing remain immature and will likely be significantly enhanced as practitioners and regulators refine the process. The above challenges are not the only issue operational risk models face: models are also affected by banks’ incentives to take on tail risk as discussed by Rajan (2006), Acharya et al. (2010) and Haldane (2010). Judgment is inevitable in statistical modeling, and identical datasets can lead different modelers to different conclusions; even in cases when modelers face identical incentives, reasonable and honest people often disagree on what model to use. But when modelers face skewed incentives, modeling frameworks are inevitably skewed. Due to government guarantees that create subsidies, banks have incentives to minimize equity financing including operational risk capital as documented by Admati (2014) and the International Monetary Fund (2014). Therefore, bank modelers have incentives to adopt models that underestimate operational risk exposure. To overcome these modeling issues and skewed incentives, we believe extensive benchmarking should be employed by banks, regulators, and analysts. The Federal Reserve 2

and the Office of the Comptroller of the Currency (OCC) define benchmarking as ”the comparison of a given model’s inputs and outputs to estimates from alternative internal or external data or models.”3 Effective benchmarking of different models requires unique techniques. Therefore, we propose the following five principles for developing operational risk benchmarks: 1) benchmarks should be conceptually meaningful, 2) benchmarks should be simple, 3) multiple benchmarks should be used, 4) discrepancies between model output and benchmarks should be explained, and 5) benchmarks cannot replace models. 1. Benchmarks should be conceptually meaningful. Gabaix and Laibson (2008) argue that models should be ”conceptually insightful”; similarly, we believe that operational risk benchmarks should be conceptually meaningful. In other words, the link between the benchmark and the model output should be intuitive. Furthermore, benchmarks should facilitate comparisons of model results and capital figures with peer banks, between subdivisions of banks, and through time. 2. Benchmarks should be simple. Benchmarks should be simple to compute, explain, and understand. A complicated benchmark is not useful because benchmarks are often used to explain model results to managers, regulators, and non-practitioners. In addition, a benchmark requiring complicated estimation will likely be more prone to estimation error. 3. Multiple benchmarks should be used. No single benchmark can track a complex model. Modelers should use a series of benchmarks that address different aspects of the modeling framework. 4. Discrepancies between model output and benchmarks should be explained. 3See OCC Bulletin 2011-12, ”Supervisory Guidance on Model Risk Management,” and Board SR letter 11-7, ”Supervisory Guidance on Model Risk Management.” 3

When benchmark results do not match model output reasonably well, the validity of results should be doubted; moreover, the disparity should be explainable or changes should be made to the model. 5. Benchmarks cannot replace models. Benchmarks do not aim to replace models but rather to allow for simple comparisons that inform model developers and users regarding the robustness of the model and, thus, enhance confidence in the model output. These five principles may sometimes conflict, and practitioners will face tradeoffs when applying them. For example, a more sophisticated benchmark may be necessary to address certain model assumptions; therefore, practitioners will face tradeoffs between keeping the benchmark simple and providing a conceptually meaningful benchmark. Furthermore, the five principles are likely not exhaustive and should be adjusted as operational risk modeling develops. Still, we believe these principles lay out a reasonable approach for appropriate use of benchmarks in operational risk modeling frameworks. The remaining sections illustrate the use of benchmarks with operational risk modeling at US banks. Section II describes the data used in the analysis. Section III explains the AMA and CCAR benchmarks considered by the Federal Reserve. Section III.a) focuses on benchmarks driven by financial statement information, Section III.b) discusses benchmarks derived from banks’ internal loss data, and Section III.c) discusses benchmarks that use both financial statement information and internal loss data. Finally, Section IV concludes. 2 Data As part of CCAR, BHCs with assets over $50 billion are required to provide operational loss data to the Federal Reserve, and this data is used for our analysis. At the time of 4

analysis, data was submitted by thirty-one BHCs. The BHCs must report information on all their operational loss events above an appropriate collection threshold including: dollar amount, occurrence date, discovery date, accounting date, Basel II event type, and Basel II business line.4 In addition, the analysis uses financial statement variables collected from the FR Y-9C reports, including total assets, risk-weighted assets, total deposits, gross income, and number of employees. The AMA capital figures used in our analysis are taken from the FFIEC 101 Schedule S, and fifteen BHCs were reporting AMA capital numbers at the time of the analysis. Pooling data from various institutions leads to unique data challenges. BHCs report data with different start dates and reporting lags. Some BHCs have more than ten years of data, while others have less than five. In addition, BHCs have different loss collection thresholds. Finally, missing data is inevitable. To address these challenges, our analysis uses a universal internal threshold of $20,000, and banks with missing data for any particular benchmark were excluded from the results for that benchmark. Furthermore, the analysis uses data from 2005Q1 to 2014Q3 to balance reporting quality with a sufficient time period. Finally, gross loss amounts, adjusted for inflation, are used.5 3 Benchmarks 3.1 Benchmarks Based on Financial Statement Information For at least a century, capital ratios have been used to assess bank safety; therefore, practitioners are well versed in the simple logic behind capital ratios.6 The numerator acts as a measure of soundness, the denominator as a proxy for risk, and larger ratios signal a 4For a detailed description of the operational risk data provided by BHCs to the Federal Reserve, see the FR Y-14Q reporting form and instructions at www.federalreserve.gov/apps/reportforms. 5TheinflationratesweuseareobtainedfromFREDat: http://research.stlouisfed.org/fred2/series/PCEPILFE# and represent the increase in personal consumption expenditures excluding food and energy. 6Mitchell (1909). 5

safer bank. Historically, equity financing usually makes up the numerator, while metrics such as total assets and total deposits have been used as denominators. In the following benchmarks, we use the AMA operational risk capital and the BHC stressed operational loss projections from CCAR as the numerator or measure of soundness.7 In this section, ratios of AMA operational risk capital to five financial statement variables are presented for the fifteen BHCs subject to operational risk AMA requirements and for which Schedule S data was reported in 2014Q3. Similarly, ratios of the BHC stressed projections from CCAR to five financial statement variables are presented for the thirty-one BHCs subject to CCAR requirements. The reporting banks included in this sample were at varying stages of operational risk model development. Specifically, only eight of the thirty-one banks had AMA models approved by regulators. The following five capital ratio benchmarks are in-line with the principles outlined in the introduction, as they are simple and provide multiple perspectives on exposure. Also, these capital ratios can be used for comparisons between banks and across time. 3.1.1 AMA Operational Risk Capital / Total Assets & BHC Stress Projection / Total Assets Abdymomunov (2014) and Abdymomunov and Curti (2015) showed that total operational risk losses are significantly correlated with the asset size of banks; also, total assets are a simple and transparent benchmark. Therefore, we believe total assets are a useful operational risk benchmark. However, some banks have significant off-balance sheet exposures due to derivatives and securitization that are not represented in the total assets figure, while other banks do not. This discrepancy should lead practitioners to exercise caution when using this benchmark for horizontal comparisons. Table 1 displays various percentiles and the average of the AMA capital estimates 7StressedlossprojectionsareasensiblemeasureofsoundnessbecauseBHCsarerequiredtoholdacapital buffer equivalent to these loss projections on top of their minimum regulatory capital. 6

divided by total assets for the fifteen banks subject to AMA requirements. Table 1: AMA Operational Risk Capital / Total Assets Percentile Ratio 10th 0.75% 25th 0.93% 50th 1.08% 75th 1.24% 90th 1.55% Average 1.16% Sample Size 15 Table 2 displays various percentiles and the average of the BHC stressed projections divided by the total assets for the thirty-one banks subject to CCAR. Table 2: BHC Stress Projection / Total Assets Percentile Ratio 10th 0.24% 25th 0.46% 50th 0.63% 75th 0.84% 90th 1.20% Average 0.69% Sample Size 31 Ratios between AMA capital and total assets range from .7% to 1.6%, with an industry median of 1.2%; while ratios between BHC stressed operational losses and total assets range from .2% to 1.2%, with an industry median of 0.7%. Banks with less developed operational risk modeling tend toward the lower ratios. 3.1.2 AMA Operational Risk Capital / Risk Weighted Assets & BHC Stress Projection / Risk Weighted Assets Total assets are limited as a measure of exposure because they do not account for differences in asset riskiness. To address this limitation, the Basel accords created 7

”risk-weighted assets” (RWA) - a measure of exposure in which assets of different riskiness are weighted differently. Table 3 and Table 4 below display this benchmark: Table 3: AMA Operational Risk Capital / Risk Weighted Assets Percentile Ratio 10th 1.21% 25th 1.37% 50th 1.98% 75th 2.17% 90th 2.88% Average 1.91% Sample Size 15 Table 4: BHC Stress Projection / Risk Weighted Assets Percentile Ratio 10th 0.31% 25th 0.59% 50th 0.86% 75th 1.23% 90th 2.39% Average 1.13% Sample Size 31 Ratios between AMA capital and total RWAs range from 1.2% to 2.9%, with an industry median of 2.0%; while ratios between BHC stressed operational losses and total RWAs range from .3% to 2.4%, with an industry median of .9%. To account for operational risk exposure, the Basel framework requires banks to multiply AMA operational risk capital estimates by 12.5 to obtain operational risk RWAs, which are then summed to credit and market RWAs to calculate total RWA. Total RWA are then used for regulatory capital ratios. A table presenting operational risk RWAs as a percentage of total RWAs is included in Appendix A. Operational risk RWAs as a percentage of total RWAs range from 15% to 36% with a median of 25% for US AMA BHCs in 2014Q3. These percentages demonstrate that operational risk is a significant risk for US BHCs. 8

3.1.3 AMA Operational Risk Capital / Basic Indicator Approach Capital & BHC Stress Projection / Basic Indicator Approach Capital Gross income (GI) is the proxy used in the Basel standardized approaches for operational risk capital calculations. We calculate gross income for a year by summing net interest income (item 3) and total noninterest income (item 5.m) from the schedule HI of the FR Y-9C report. Then, to calculate the Basic Indicator Approach (BIA) capital for 2014Q3, we average gross income for the three previous years (2011, 2012, and 2013), excluding negative values, and multiply this average by 15%. Tables 5 and 6 display this benchmark. Table 5: AMA Operational Risk Capital / Basic Indicator Approach Capital Percentile Ratio 10th 1.2 25th 1.4 50th 1.7 75th 2.0 90th 2.2 Average 1.9 Sample Size 15 Table 6: BHC Stress Projection / Basic Indicator Approach Capital Percentile Ratio 10th 0.5 25th 0.6 50th 0.9 75th 1.2 90th 1.9 Average 1.1 Sample Size 31 Ratios between AMA capital and BIA capital range from 1.2 to 2.2, with an industry median of 1.7; while ratios between BHC stressed operational losses and BIA capital range from .5 to 1.9, with an industry median of 0.9. In 2014Q3, fourteen of the fifteen US AMA BHCs reported operational risk capital greater than the capital they would be required to 9

hold under the BIA, which indicates that the BIA likely underestimates the operational risk exposure of large US BHCs. Although gross income has been used across the industry to proxy operational risk, its reliability as a proxy has not been established.8 3.1.4 AMA Operational Risk Capital / Total Deposits & BHC Stress Projection / Total Deposits This benchmark is simple and difficult to game. However, it should not be used to compare banks with different funding models. For example, despite facing operational risk comparable or greater than traditional banks, investment banks rely much less on deposit funding than traditional banks. Tables 7 and 8 display this benchmark. Table 7: AMA Operational Risk Capital / Total Deposits Percentile Ratio 10th 1.01% 25th 1.36% 50th 1.69% 75th 3.55% 90th 7.50% Average 3.07% Sample Size 15 Table 8: BHC Stress Projection / Total Deposits Percentile Ratio 10th 0.32% 25th 0.65% 50th 0.96% 75th 1.95% 90th 3.58% Average 1.51% Sample Size 31 8Recently, the Basel Committee proposed revisions to the standardized approaches for operational risk partly aimed at improving the risk sensitivity - which replace gross income as a proxy for operational risk. Basel Committee on Banking Supervision (2014). 10

RatiosbetweenAMAcapitalandtotaldepositsrangefrom1.0%to7.5%,withanindustry median of 1.7%; while ratios between BHC stressed operational losses and total deposits range from .3% to 3.6%, with an industry median of 1.0%. 3.1.5 AMA Operational Risk Capital / Number of Employees & BHC Stress Projection / Number of Employees The number of employees is a useful operational risk benchmark because employees are likely a driver of operational risk. The more employees a firm has, the more opportunities for an employee to cause losses intentionally or unintentionally. Tables 9 and 10 display this benchmark: Table 9: AMA Operational Risk Capital / Number of Employees Percentile Ratio 10th 60,997 25th 70,004 50th 79,768 75th 126,803 90th 201,584 Average 114,326 Sample Size 15 Table 10: BHC Stress Projection / Number of Employees Percentile Ratio 10th 19,131 25th 31,776 50th 44,926 75th 75,140 90th 114,656 Average 68,299 Sample Size 31 Ratios between AMA capital and number of employees range from 61.0k to 201.6k, with an industry median of 79.8k; while ratios between BHC stressed operational losses 11

and total deposits range from 19.1k to 114.7k, with an industry median of 44.9k. The benchmark ratios presented above compare banks’ capital estimates to financial statement information and, thus, can be used to rank banks. However, these benchmarks offer limited insight into whether the industry ratios are reasonable. For example, a ratio above the industry median may not be conservative because the reasonableness of the industry median ratio as a benchmark depends on the soundness of models across the industry. To assess a safe level of capital, further analysis is required using banks’ loss histories. 3.2 Benchmarks Based on Banks Internal Losses To understand and compare banks’ operational risk loss exposure, loss experience should be considered. Past loss experience is likely a good proxy for operational risk loss exposure going forward, as past losses relate to the quality of risk controls and the riskiness of the business environment. However, caution should be applied when making comparisons based on internal loss benchmarks because banks are at different levels of data collection. Some banks have collected up to twelve years of data and have a mature collection process; while others have collected only a few years of data, with questionable quality in the collection process. Still, the following five internal loss-based benchmarks can be used to better understand banks’ capital adequacy. 3.2.1 AMA Operational Risk Capital / Maximum Loss & BHC Stress Projection / Maximum Loss To begin, a simple comparison of the banks’ capital projections divided by their largest recorded loss is computed. Tables 11 and 12 display this benchmark. 12

Table 11: AMA Operational Risk Capital / Maximum Loss Percentile Ratio 10th 2.6 25th 3.0 50th 4.2 75th 9.2 90th 12.0 Average 6.0 Sample Size 15 Table 12: BHC Stress Projection / Maximum Loss Percentile Ratio 10th 0.8 25th 2.2 50th 3.2 75th 6.2 90th 9.5 Average 4.4 Sample Size 31 RatiosbetweenAMAcapitalandthelargesthistoricallossrangefrom2.6to12.0, withan industry median of 4.2; while ratios between BHC stressed operational losses and the largest historical loss range from .8 to 9.5, with an industry median of 3.2. Estimates resulting from prudent AMA and CCAR models should exceed the firm’s largest loss, but some banks are still in the early stages of developing credible models. 3.2.2 AMA Operational Risk Capital / Maximum 4 Quarters Loss & BHC Stress Projection / Maximum 9 Quarters Loss AMA capital is designed to provide a buffer for operational risk losses over a four quarter horizon, while BHC stressed operational loss projections are designed to provide an estimate of losses under stressed scenarios over a nine quarter horizon. Therefore, comparing the operational risk capital model outputs to actual worse case scenarios experienced by banks 13

over those time periods provides a valuable insight into the efficacy of the models. Tables 13 and 14 display these benchmarks: Table 13: AMA Operational Risk Capital / Maximum 4 Quarters Loss Percentile Ratio 10th 1.5 25th 1.9 50th 3.5 75th 4.7 90th 5.6 Average 3.6 Sample Size 15 Table 14: BHC Stress Projection / Maximum 9 Quarters Loss Percentile Ratio 10th 0.6 25th 1.1 50th 2.0 75th 3.7 90th 5.5 Average 2.8 Sample Size 31 Ratios between AMA capital and largest consecutive four quarter losses range from 1.5 to 5.6, with an industry median of 3.5; while ratios between BHC stressed operational losses and largest consecutive nine quarters losses range from .6 to 5.5, with an industry median of 2.0. Given that most banks have less than fifteen years of data, safety and soundness should likely lead to ratios well above one for both AMA and CCAR models. 3.2.3 AMA Operational Risk Capital / Average 4 Quarters Loss & BHC Stress Projection / Average 9 Quarters Loss Comparing AMA capital and stressed loss projections to average losses is also useful. Tables 15 and 16 display these benchmarks. 14

Table 15: AMA Operational Risk Capital / Average 4 Quarters Loss Percentile Ratio 10th 5.6 25th 7.4 50th 11.6 75th 13.8 90th 16.5 Average 11.4 Sample Size 15 Table 16: BHC Stress Projection / Average 9 Quarters Loss Percentile Ratio 10th 1.6 25th 2.2 50th 3.5 75th 6.4 90th 9.6 Average 4.6 Sample Size 30 Ratios between AMA capital and average four quarter losses range from 5.6 to 16.5, with an industry median of 11.6; while ratios between BHC stressed operational losses and average nine quarters losses range from 1.6 to 9.6, with an industry median of 3.5. Given that AMA capital is supposed to cover tail losses and BHC stressed projections are supposed to cover stressed losses, both of these ratios should be significantly above one. 3.2.4 AMA Operational Risk Capital / Most Recent 4 Quarters Loss & BHC Stress Projection / Most Recent 9 Quarters Loss Recent losses may be more representative of a firm’s current risk exposure. Therefore, a benchmark that highlights recent losses may be useful to identify concerning trends, but this benchmark is more volatile than other benchmarks (see the coefficient of variation statistics in Appendix B). Tables 17 and 18 display these benchmarks. 15

Table 17: AMA Operational Risk Capital / Most Recent 4 Quarters Loss Percentile Ratio 10th 16.0 25th 23.3 50th 39.8 75th 47.5 90th 63.8 Average 38.3 Sample Size 15 Table 18: BHC Stress Projection / Most Recent 9 Quarters Loss Percentile Ratio 10th 2.1 25th 3.0 50th 5.2 75th 11.1 90th 13.2 Average 7.1 Sample Size 30 Ratios between AMA capital and most recent four quarter losses range from 16.0 to 63.8, with an industry median of 39.8; while ratios between BHC stressed operational losses and average nine quarters losses range from 2.1 to 13.2, with an industry median of 7.1. 3.2.5 AMA Operational Risk Capital / 99.9 Empirical Bootstrap (4 quarters) & BHC Stress Projection / 99.9 Empirical Bootstrap (9 quarters) This benchmark uses a LDA framework, but with some unique assumptions to ensure comparability. The process to compute this benchmark begins with separating each bank’s losses into the seven Basel event types which are used as units of measure. Then, like a traditional LDA, loss frequency and severity are modeled separately and assumed independent. To model frequency, let N(t) be a Poisson random variable with intensity λ, which represents the number of losses in the time interval [0,t] (assumed to be four 16

quarters for the AMA benchmark and nine quarters for the CCAR benchmark): (λt)n P(N(t) = n) = expλt , (1) n! To model severity, assume X is a positive random variable describing the magnitude i of losses. While traditional LDA models assume that loss severity follows a parametric distribution, this benchmark uses the empirical distribution of observed loss severities to define the severity distribution. To combine the frequency and severity distributions, we calculate event type aggregate losses, S(t): N(t) (cid:88) S(t) = X (2) i i=1 Monte Carlo simulation is used to obtain the distribution of aggregate losses for each eventtype. Ineachsimulationpath, first, alosscountisdrawnfromthePoissondistribution; then, the corresponding number of loss severities are drawn from the empirical distribution and these severities are added up. To obtain the 99.9th quantile of the total annual loss distribution, the 99.9th quantile of the aggregate annual loss distributions of the seven event types are summed (i.e., we assume full dependency between event types). Tables 19 and 20 display these benchmarks: Table 19: AMA Operational Risk Capital / 99.9 Empirical Bootstrap (4 quarters) Percentile Ratio 10th 0.8 25th 1.0 50th 1.5 75th 2.7 90th 3.0 Average 1.7 Sample Size 15 17

Table 20: BHC Stress Projection / 99.9 Empirical Bootstrap (9 quarters) Percentile Ratio 10th 0.2 25th 0.6 50th 1.0 75th 1.5 90th 2.6 Average 1.3 Sample Size 31 Ratios between AMA capital and the historical simulation benchmark range from .8 to 3.0, with an industry median of 1.5; while ratios between BHC stressed operational losses and the historical simulation benchmark range from .2 to 2.6, with an industry median of 1.0. By using the empirical distribution of severities as the severity distribution, this benchmark assumes that a bank will never experience a loss event bigger than the largest loss event the bank has experienced in the past. Therefore, this benchmark should likely be a floor for AMA models. The historical simulation benchmark is more complicated than the other benchmarks explored making results susceptible to estimation error, but the benchmark is conceptually meaningful as it provides for easy comparison between banks, while including many of the assumptions of the more sophisticated AMA models used by banks. Furthermore, the historical simulation benchmark can be useful to assess whether changes in model estimates are sensible. For example, the historical simulation benchmark can provide a robustness check for data updates. If the historical simulation estimate increases by 100% for a particular unit of measure, an increase of 80% on the model estimate is likely reasonable; while if the historical simulation estimate barely changes after a data update, then a 80% change in model estimates is likely the result of an unstable model. 18

3.3 Benchmarks Based on Financial Statement Information and Internal Loss Data As discussed in Section I, the 99.9th quantile standard for AMA raises modeling challenges. The following benchmark, which combines financial statement information and internal loss data, offers insight into reasonable AMA capital numbers given past industry losses. Specifically, we use the distribution of ratios of cumulative four consecutive quarters of operational losses to total assets in the quarter prior to the loss window to derive a benchmark for AMA capital. To estimate this benchmark we use data for all AMA banks from 2000Q1 to 2014Q3, which includes more than 600 ratios. The distribution of these ratios is used to identify the ratio number between AMA capital and total assets that would reflect the 99.9 confidence level, if exposure was homogenous across the industry and unchanged from what was observed in the past. Figure 1 displays the distribution of ratios: Figure 1: Distribution: One Year OpRisk Loss / Total Assets The99.9quantileforthisdistributionslightlyexceeds1.3%. Usingsimilarlogic, inFigure 2, the distribution of ratios is presented using gross income instead of total assets: 19

Figure 2: Distribution: One Year OpRisk Loss / Gross Income The 99.9 quantile for this distribution of ratios slightly exceeds 34%. Similar combined benchmarkscanbecomputedforCCAR,usingnine-quarterlossesinsteadofthefourquarter losses used for AMA. The distribution of ratios for CCAR is displayed in Figures 3 and 4. Figure 3: Distribution: Nine Year OpRisk Loss / Total Assets 20

Figure 4: Distribution: Nine Quarter OpRisk Loss / Total Assets These distributions show the range of operational losses experienced in the industry in thelastfifteenyears. However, restraintshouldbeusedwhenevaluatingthehighquantilesof these distributions to assess the credibility of AMA and CCAR estimates. Despite using the full history of industry losses reported to the Federal Reserve, the distribution of observed ratios still only includes around 600 ratios and, thus, the estimate of the 99.9th quantile suffers from significant uncertainty, as it results from interpolation of the largest two data points. Moreover, operational loss exposure varies significantly across banks and is not always proportional to size. 4 Conclusion This paper provides guidelines for using benchmarks in conjunction with operational risk models. Operational risk is difficult to model at high quantiles and difficult to link to macroeconomic factors. Furthermore, banks’ incentives to minimize capital lead banks’ operational risk modelers to make less conservative assumptions in their models. Therefore, 21

benchmarks should be an integral part of any operational risk modeling framework. Benchmarks should be used by modelers, regulators, and analysts to gauge the efficacy of the models and assess the reasonableness of results. Still, misuse of operational risk benchmarks may be more detrimental than failing to use benchmarks at all. Therefore, we offer five principles to ensure robustness for operational risk benchmarks. Benchmarks should be conceptually meaningful. Benchmarks should be simple. Multiple benchmarks should be used. Benchmarks should explain discrepancies. And benchmarks cannot replace models. Operational risk benchmarks must be practical, easily understood, and lead to meaningful changes. But benchmarks cannot replace the detail found in models; rather, operational risk benchmarks allow for comparability across banks and over time. Furthermore, practitioners will have to balance the different principles within the modeling framework when tracking and recalibrating their operational risk models. These simple principles offer direction around what role operational risk benchmarks should play in an operational risk modeling framework. The benchmarks presented in this paper illustrate the range of outcomes in the industry. Thus, this paper allows practitioners to compare their models estimates with the estimates from other large US BHCs. These benchmarks are examples of the benchmarks currently being used by US banks and regulators, but the examples should not be considered complete. Rather, benchmarks should constantly be re-evaluated and updated as models change, regulations change, and economic factors change. With these inevitable changes, new operational risk benchmarks should be developed and poorly performing benchmarks should be retired. The process must be dynamic. But a sound operational risk modeling framework combined with ample use of operational risk benchmarks offers the best way forward for measuring and managing operational risk. 22

References Abdymomunov, A. (2014). Banking sector operational losses and macroeconomic environment. Working Paper,Federal Reserve Bank of Richmond. Abdymomunov, A. and Curti, F. (2015). Improving the robustness of operational risk modeling through external data scaling. Working Paper,Federal Reserve Bank of Richmond. Acharya, V., Cooley, T., Richardson, M., and I., W. (2010). Manufacturing tail risk: A perspective on the financial crisis of 2007-09. Foundations and Trends in Finance, 4:247–325. Admati, A.(2014). Statementforsenatecommitteeonbanking, housingandurbanaffairssubcommitteeon financial institutions and consumer protection. Hearing on ”Examining the GAO report on expectations of government support for bank holding companies”, July 31, 2014. Basel Committee on Banking Supervision (2014). Operational risk - revisions to the simpler approaches. Bank of International Settlements. Cope, E., Mignola, G., Antonini, G., and Ugoccioni, R. (2009). Challenges and pitfalls in measuring operational risk from loss data. The Journal of Operational Risk, 4(4):3–27. Danielsson, J. (2002). The emperor has no clothes: Limits to risk modeling. Journal of Banking & Finance, 26(7):1273–1296. Danielsson, J. (2008). Blame the models. Journal of Financial Stability, 4(4):321–328. Gabaix, X. and Laibson, D. (2008). The Seven Properties of Good Models, The Foundations of Positive and Normative Economics: A Handbook. Oxford University Press. Haldane, A. (2010). The $100 billion question. Comments by Andrew G Haldane, Executive Director, Financial Stability, Bank of England, at the Institute of Regulation & Risk, Hong Kong, 30 March 2010. International Monetary Fund (2014). Global financial stability report. Washington: International Monetary Fund, April 2014. Mignola, G. and Ugoccioni, R. (2006). Sources of uncertainty in modeling operational risk losses. The Journal of Operational Risk, 1(2):33–50. Mitchell, W. C. (1909). The decline in the ratio of banking capital to liabilities. Quarterly Journal of Economics, 23(4). Neˇslehov´a, J., Embrechts, P., and Chavez-Demoulin, V. (2006). Infinite mean models and the lda for operational risk. The Journal of Operational Risk, 1(1):3–25. Opdyke, J. and Cavallo, A. (2012). Estimating operational risk capital: The challenges of truncation, the hazards of maximum likelihood estimation, and the promise of robust statistics. The Journal of Operational Risk, 7(3):3–90. Panjer, H. (1981). Recursive evaluation of a family of compound distributions. ASTIN Bulletin, 12:22–26. Rajan, R. (2006). Has financial development made the world riskier? European Financial Mangement, 12(4):499–533. 23

Appendix A AMA Operational Risk RWA / Total RWA Percentile Ratio 10th 15.1% 25th 17.1% 50th 24.7% 75th 27.2% 90th 36.0% Average 23.9% Sample Size 15 24

Appendix B Descriptive Statistics Benchmarks based on financial statement information: AMA - Financials Ratio N Mean SD CV 10% 25% 50% 75% 90% OpRisk Capital / Total Assets 15 1.2% 0.4% 36.8% 0.7% 0.9% 1.1% 1.2% 1.7% OpRisk Capital / Gross Revenue 15 28.2% 16.0% 56.9% 17.0% 21.3% 24.3% 30.7% 33.5% OpRisk Capital / Total RWA 15 1.9% 0.7% 34.2% 1.2% 1.3% 2.0% 2.2% 3.0% OpRisk Capital / Total Deposits 15 3.1% 2.7% 87.8% 0.9% 1.3% 1.7% 4.5% 8.2% OpRisk Capital / BIA 15 1.9 1.0 50.5% 1.1 1.4 1.7 2.0 2.2 OpRisk Capital / Number of Employees 15 114,325 76,280 66.7% 60,564 67,098 79,767 137,119 232,117 CCAR - Financials Ratio N Mean SD CV 10% 25% 50% 75% 90% BHC Stress / Total Assets 31 0.7% 0.4% 52.4% 0.2% 0.4% 0.6% 0.9% 1.2% BHC Stress / Gross Revenue 31 15.9% 11.2% 70.4% 6.9% 9.4% 13.2% 17.3% 27.9% BHC Stress / Total RWA 31 1.1% 0.9% 79.6% 0.3% 0.5% 0.9% 1.2% 2.4% BHC Stress / Total Deposits 30 1.5% 1.5% 97.6% 0.2% 0.5% 0.8% 1.3% 2.3% BHC Stress / BIA 31 1.1 0.7 70.4% 0.5 0.6 0.9 1.2 1.9 BHC Stress / Number of Employees 31 68,298 70,782 103.6% 19,131 31,710 44,925 82,004 114,655 25

Benchmarks based on banks’ internal losses: AMA - Internal Losses Ratio N Mean SD CV 10% 25% 50% 75% 90% OpRisk Capital / Max Loss 15 6.0 3.8 64.6% 2.6 3.0 4.2 9.2 12.0 OpRisk Capital / Max 4q 15 3.6 1.9 54.0% 1.5 1.9 3.5 4.7 5.6 OpRisk Capital / Average 4q 15 11.4 5.0 44.0% 5.6 7.4 11.6 13.8 16.5 OpRisk Capital / 99.9 Emp. Bootstrap 15 1.7 0.9 54.3% 0.8 1.0 1.5 2.7 3.0 OpRisk Capital / Most Recent 4q 15 38.2 20.7 54.1% 16.0 23.3 39.8 47.5 63.8 OpRisk Capital / Historic 99.9 TA Ratio 15 90.5% 33.3% 36.8% 19.5% 68.9% 81.9% 94.3% 108.7% OpRisk Capital / Historic 99.9 GI Ratio 15 84.4% 43.4% 51.5% 9.9% 54.1% 69.6% 89.0% 98.0% CCAR - Internal Losses Ratio N Mean SD CV 10% 25% 50% 75% 90% BHC Stress / Max Loss 31 12.6 16.4 130.0% 1.2 2.2 5.6 15.2 39.5 BHC Stress / Max 9q 31 2.8 2.3 82.1% 0.6 1.1 2.0 3.7 5.5 BHC Stress / Average 9q 30 4.6 3.2 69.3% 1.7 2.2 3.8 6.9 10.3 BHC Stress / 99.9 Emp. Bootstrap 31 1.3 1.0 78.1% 0.2 0.6 1.0 1.6 2.6 BHC Stress / Most Recent 9q 30 7.1 4.9 68.4% 2.2 2.9 5.7 12.1 12.9 BHC Stress / Historic 99.9 TA Ratio 31 37.1% 19.6% 53.0% 12.7% 24.8% 33.6% 47.9% 65.6% BHC Stress / Historic 99.9 GI Ratio 31 30.6% 21.5% 70.4% 13.2% 18.1% 25.5% 33.4% 53.9% 26