Second Brain

SR 11-7 - Model Risk Management Guidance - Overview

12 min read

SR 11-7 - Model Risk Management Guidance - Overview

Summary

SR 11-7 (April 4, 2011) is the joint Federal Reserve / OCC supervisory guidance that became the de facto standard for governing quantitative models. It defines a model broadly (input, processing and reporting components; judgment-based inputs included if the output is quantitative), defines model risk as arising from fundamental error or incorrect/inappropriate use, and names effective challenge — critical analysis by parties with the incentives, competence and influence to force change — as its guiding principle. It organises model risk management into three pillars: (1) disciplined development, implementation and use (§IV); (2) validation with three core elements — conceptual soundness, ongoing monitoring and outcomes analysis — plus vendor-model validation (§V); and (3) governance, policies and controls — board and senior management, policies, roles, internal audit, external resources, a firm-wide model inventory and documentation (§VI). Rigor must scale with materiality throughout. The framework transfers almost unchanged to marketing analytics, where the target quantity (incremental effect) is never observed — see SR 11-7 - Applying Model Risk Management to Marketing Analytics.

Overview

Document. “SR Letter 11-7 Attachment”, issued by the Board of Governors of the Federal Reserve System and the Office of the Comptroller of the Currency, dated April 4, 2011, titled Supervisory Guidance on Model Risk Management. The OCC issued the same text as its Bulletin 2011-12 (the bulletin number is not printed in the attachment; it is the standard companion citation). 21 numbered pages plus a contents page; five footnotes.

Research question (in effect). How should an institution that relies on quantitative models manage the risk that decisions based on those models are wrong — not just by validating models, but across the whole lifecycle and organisation?

Contribution. Earlier OCC/Fed guidance “pay[s] particular attention to model validation”; SR 11-7 “expands on existing guidance—most importantly by broadening the scope to include all aspects of model risk management” (§II, p. 2). Its lasting contributions are the broad model definition, the two-source taxonomy of model risk, the effective challenge principle and the three-element validation framework.

Key principles in one line each.

  1. Anything that turns data and assumptions into quantitative estimates is a model, including its reporting layer (^def-model).
  2. Model risk comes from wrong models and from right models used wrongly (^def-model-risk).
  3. Challenge only counts when the challenger has incentives, competence and influence (^def-effective-challenge).
  4. Users are poor challengers of favourable outcomes (^thm-user-challenge-asymmetry).
  5. Validation must be independent and can block model use (^thm-validation-gate).
  6. In-sample fit and holdout performance are “not substitutes for back-testing” (^def-backtesting).
  7. “Banks are expected to validate their own use of vendor products” (SR 11-7 - Validation of Vendor and Third-Party Models).
  8. Without documentation and a firm-wide inventory, aggregate model risk cannot be assessed (SR 11-7 - Model Inventory and Documentation Standards).

Main Content

§I Introduction (p. 1)

Banks “rely heavily on quantitative analysis and models in most aspects of financial decision making” — underwriting credits; valuing exposures, instruments and positions; measuring risk; “managing and safeguarding client assets”; determining capital and reserve adequacy; “and many other activities” (§I, p. 1). “In recent years, banks have applied models to more complex products and with more ambitious scope, such as enterprise-wide risk measurement, while the markets in which they are used have also broadened and changed” (p. 1), partly spurred by U.S. regulatory capital rules based on the Basel Committee framework, but also independently, as banks have increased “the use of data-driven, quantitative decision-making tools.”

The economic framing that motivates the whole document:

Definition: The Two Costs of Models (SR 11-7, §I, p. 1)

Models “can improve business decisions, but models also come with costs”:

  • a direct cost — “of devoting resources to develop and implement models properly”;
  • indirect costs — “the possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused.”

“Those consequences should be addressed by active management of model risk.”

Footnote 1 (p. 1) — scope of “banks”: national banks and other OCC-supervised institutions, plus bank holding companies, state member banks and other institutions supervised by the Federal Reserve Board.

The introduction then gives the section road-map reproduced in the Source Map below.

§II Purpose and Scope (p. 2)

  • Purpose: “comprehensive guidance for banks on effective model risk management.” Validation “plays a critical role,” but “sound development, implementation, and use of models are also vital elements,” and model risk management “encompasses governance and control mechanisms such as board and senior management oversight, policies and procedures, controls and compliance, and an appropriate incentive and organizational structure.”
  • Relationship to prior guidance (fn. 2): the OCC’s model-validation guidance OCC 2000-16 (May 30, 2000), other bulletins and Comptroller’s Handbook booklets; Federal Reserve SR Letter 09-01 on the Market Risk Rule (validation standards, validation documentation, back-testing); the Fed’s Trading and Capital-Markets Activities Manual; and the advanced-approaches risk-based capital rules (12 CFR 3 App. C; 12 CFR 208 App. F; 12 CFR 225 App. G), which contain explicit validation requirements.
  • Rationale and expectation for existing practice: the guidance is issued “Based on supervisory and industry experience over the past several years” and “expands on existing guidance—most importantly by broadening the scope to include all aspects of model risk management.” It acknowledges that “Many banks may already have in place a large portion of these practices, but all banks should ensure that internal policies and procedures are consistent with the risk management principles and supervisory expectations contained in this guidance” (§II, p. 2). In other words, the bar is conformity of internal policies and procedures to the stated principles, not adoption of a brand-new programme.
  • Proportionality: “practical application of this guidance should be customized to be commensurate with a bank’s risk exposures, its business activities, and the complexity and extent of its model use.” Example given: a community bank “using relatively few models of only moderate complexity” may need “significantly less involved” steps than a larger bank with extensive or complex model use.

Principle: Proportionality / Materiality (SR 11-7, §II p. 2; §III p. 5; §VII p. 21)

The extent of model risk management scales with risk. Where model use is “less pervasive and has less impact on their financial condition,” a less complex approach may suffice; “where models and model output have a material impact on business decisions … and where model failure would have a particularly harmful impact on a bank’s financial condition, a bank’s model risk management framework should be more extensive and rigorous” (p. 5). The same principle recurs for validation rigor (p. 9), governance sophistication (p. 16) and the conclusion (p. 21).

§VII Conclusion (p. 21)

The conclusion restates that “Many of the activities described in this document are common industry practice,” but every bank should confirm its practices conform to the principles for development, implementation, use and validation, maintain “strong governance and controls,” and apply the guidance “commensurate with a bank’s risk exposures, its business activities, and the extent and complexity of its model use.” No new content is introduced.

Structure of the framework

flowchart TD
  A["Model risk (§III)<br/>error + misuse; individual + aggregate"] --> B["Effective challenge<br/>incentives · competence · influence"]
  B --> C["Pillar 1: Development, implementation, use (§IV)"]
  B --> D["Pillar 2: Validation (§V)"]
  B --> E["Pillar 3: Governance, policies, controls (§VI)"]
  D --> D1["Conceptual soundness"]
  D --> D2["Ongoing monitoring<br/>process verification, overrides, benchmarking"]
  D --> D3["Outcomes analysis<br/>back-testing, parallel analysis, early warning"]
  D --> D4["Vendor / third-party products"]
  E --> E1["Board & senior mgmt · Policies · Roles"]
  E --> E2["Internal audit · External resources"]
  E --> E3["Model inventory · Documentation"]

Source Map

Every section, subsection and footnote of the source, with the note that captures it.

Source locationContentNote
Contents pageSection list with page numbersthis note
§I Introduction, p. 1Bank reliance on models; direct vs indirect coststhis note ()
Footnote 1, p. 1Definition of “banks” (OCC/Fed-supervised institutions)this note
§I road-map, p. 2Section guidethis note
§II Purpose and Scope, p. 2Purpose, rationale (supervisory/industry experience), broadened scope, expectation that existing policies conform, proportionalitythis note ()
Footnote 2, p. 2Prior guidance (OCC 2000-16, SR 09-01, Trading Manual, advanced-approaches rules)this note
§III Overview, pp. 3-5Model definition (3 components), model quality, model risk (2 sources), aggregate risk, effective challenge, limits/conservatism, materiality, 3 pillarsSR 11-7 - Model Definition, Model Risk and Effective Challenge
Footnote 3, p. 3Qualitative approaches outside the model definition still need rigorous controlSR 11-7 - Model Definition, Model Risk and Effective Challenge
§IV intro, p. 5Development as judgment-laden, multidisciplinary activitySR 11-7 - Model Development, Implementation and Use
Footnote 4, p. 5Smaller banks relying on vendor modelsSR 11-7 - Model Development, Implementation and Use and SR 11-7 - Validation of Vendor and Third-Party Models
§IV Model Development and Implementation, pp. 5-7Statement of purpose, theory, data quality & proxies, testing, multiple tests, judgmental adjustments, systems integrationSR 11-7 - Model Development, Implementation and Use
§IV Model Use, pp. 7-8Users as source of challenge; weakness and asymmetry of user challenge; reports; uncertainty and ranges; conservatism; transition to validationSR 11-7 - Model Development, Implementation and Use
Footnote 5, p. 8GAAP compliance of uncertainty adjustmentsSR 11-7 - Model Development, Implementation and Use
§V intro, pp. 9-10Validation definition; all components; vendor parity; rigor; independence; incentives; competence; authority & influence; pre-use gate; ongoing validation; annual review; thresholdsSR 11-7 - Validation Independence, Validation Gate and Periodic Review
§V Key Elements, p. 11Three core elements listSR 11-7 - Validation Independence, Validation Gate and Periodic Review
§V.1 Evaluation of Conceptual Soundness, pp. 11-12Developmental evidence, alternatives, assumptions, data relevance, sensitivity analysis, stress testing, action plan, qualitative judgmentSR 11-7 - Conceptual Soundness, Sensitivity Analysis and Stress Testing
§V.2 Ongoing Monitoring, pp. 12-13Purpose and frequency, process verification, code control, user-developed applications, reports, repeated tests, override analysis, benchmarkingSR 11-7 - Ongoing Monitoring, Process Verification, Overrides and Benchmarking
§V.3 Outcomes Analysis, pp. 13-15Definition, range of tests, parallel outcomes analysis, back-testing, VaR example, interpretation, long horizons & early warning, thresholds, limited back-testingSR 11-7 - Outcomes Analysis and Back-Testing
§V Validation of Vendor and Other Third-Party Products, pp. 15-16Vendor selection, vendor evidence, own-use validation, customization, data representativeness, in-house knowledge, contingencySR 11-7 - Validation of Vendor and Third-Party Models
§VI intro, p. 16Why governance matters; governance scales with model usageSR 11-7 - Governance Roles, Policies, Internal Audit and External Resources
§VI Board of Directors and Senior Management, p. 17Framework, delegation, senior-management duties, reporting, risk toleranceSR 11-7 - Governance Roles, Policies, Internal Audit and External Resources
§VI Policies and Procedures, pp. 17-18Approval & annual review, required coverage, accuracy targets, validation standards, documentation, roles, external resourcesSR 11-7 - Governance Roles, Policies, Internal Audit and External Resources
§VI Roles and Responsibilities, pp. 18-19Ownership / controls / compliance; model owner; control staff; exceptions; compliance trackingSR 11-7 - Governance Roles, Policies, Internal Audit and External Resources
§VI Internal Audit, p. 19Role, independence, audit checklist, evaluating validatorsSR 11-7 - Governance Roles, Policies, Internal Audit and External Resources
§VI External Resources, p. 20Benefits vs costs, scope of work, designated internal party, contingencySR 11-7 - Governance Roles, Policies, Internal Audit and External Resources
§VI Model Inventory, p. 20Coverage, firm-wide inventory, variations, inventory fieldsSR 11-7 - Model Inventory and Documentation Standards
§VI Documentation, p. 21Standard, incentives, who documents, vendor docs, validation reportsSR 11-7 - Model Inventory and Documentation Standards
§VII Conclusion, p. 21Restatement; proportionalitythis note
(applied, not in source)Mapping to marketing analytics, acceptance protocol, red flagsSR 11-7 - Applying Model Risk Management to Marketing Analytics

The source contains no tables or figures; the only worked example is the VaR back-test (§V, pp. 14-15), captured in SR 11-7 - Outcomes Analysis and Back-Testing.

Examples

Example: Reading SR 11-7 as a lifecycle (generic illustration)

A forecasting model for weekly sales moves through the guidance as follows: a statement of purpose and documented theory/data choices (§IV) → pre-use validation by an independent reviewer covering conceptual soundness, sensitivity and stress tests (§V.1) → approval, restriction or rejection at the validation gate (§V intro) → entry in the model inventory with owner, uses, restrictions and next validation date (§VI) → ongoing monitoring with process verification, override tracking and benchmarking (§V.2) → outcomes analysis: back-tests at the forecast horizon against predetermined thresholds (§V.3) → at least annual review, and re-validation after material change (§V intro) → internal audit checks that this cycle actually ran (§VI).

Connections

See Also

Graph View

Backlinks