WIXLIB

A RANDOM WALK THROUGHMODEL RISK MANAGEMENTPresentation by Federal Reserve and OCCFRB Richmond/Charlotte MRM ForumSeptember 2016

Background/Context2 Agencies now have a half-decade of experience in evaluatingMRM frameworks that are based on the 2011 model riskmanagement guidance (MRMG)OCC and FRS have established formal supervisory processes forevaluating MRM frameworksAlso are integrating MRM assessments into broader supervisoryassessments (such as of business lines or risk stripes or forCCAR/DFAST)Tying together assessments of individual models or modelingareas with assessments of “programmatic” elements of MRMOCC and FRS continue to work closely together on MRMassessments, as well as on industry outreach

Background/Context (cont)3 In this presentation we are trying to highlight positive practicesand examples seen over the years (“glass half full” or the“inverse of MRAs”)“Random” means we are not planning to touch all aspects ofMRM – picking & choosing a selection in no particular orderIn each area we visit, we will touch on some points, not all (i.e.,not an exhaustive discussion)Most of these are “based on a true story” with some tweaksNot necessarily describing minimum expectations, but differentways to meet minimum expectations Practices noted here are not intended to be new or enhancedstandards Instead, a description of what has worked well for some firms

Stops on Our Walk4 GovernanceModel designParallel validationOverlaysVendor modelsPolicies and proceduresInternal audit

Governance5 Significant attention and internal discussion to evaluate current MRMpractices (gap analyses) and develop remediation plansFirm-wide committee and governance of MRM with better integrationacross LOBs – more systematic policies, procedures, and practicesAttention to monitoring the entire cycle of development,implementation, use, and validationEntire firm views MRM as a process, not a project or an eventModel evaluation resources/expertise broadened - not only technicalreview process, but also includes risk managers, finance, audit, IT, etc.Greater attention to MRM by board members and carefulconsideration of information board members need to oversee MRM Reports appropriately scaled for board vs senior managementHealthy skepticism about model output and appreciation of limitations Firm pays attention to model materiality, scales MRM accordingly

Model Design6 Firm-wide standards for model design, as well as more detailedprocedures to guide development of specific model typesClear process for deciding to build the model, with properapprovals that follow established policies & proceduresIntended use of the model is clear and approval is needed forexpanding/altering useClear documentation about the original model design andprocessing components – allows third party to follow the path ofdevelopmentEvidence that model developers have conducted research aboutindustry/academic theory & practice relating to the modelConsideration of data choices and accounting for exposurecharacteristicsAssumptions clearly identified and tested

Parallel Validation7 Firm chooses to conduct validation activities as model developmentproceeds, rather than waiting for developers to finish the modelFirm recognizes that it should have additional safeguards topreserve effective challenge if it pursues such “parallel validation” Validation staff not involved in making any decisions about the model Policy states clearly that validation’s role is to determine if there are modellimitations; developer has to decide how to address those limitations Once a discrete stage of the development process is completed anddocumented, then corresponding validation activities begin But clarity that this is not the “final say” from validation on this topic Once all model stages finished, next step in validation commences Taking into account previous results but also taking a fresh look at all of thedevelopmental evidence and how the model works in full Evaluating all the parts of the model as an integral whole is fundamental toassessing it thoroughly Model should then be reviewed on an ongoing basis

MRM for Model Overlays8 Firms includes overlays/overrides in overall MRM programHas clear and detailed policies and procedures for overlaysFirm makes distinction between one-time vs ongoing overlays (and tries tominimize the latter)Firm has well documented support for overlay rationale (first line)Strong effective challenge for model overlays (scaled by materiality)Internal standards for tracking model and overlay performance, separatelyand togetherFirm also ensures that it applies effective challenge to overlays set bymanagement committeesCombination of technical expertise & business expertise in overlay reviewConfirmation that overlays called “conservative” are indeed soFirms revisits underlying model to explore recalibration/redevelopmentMore significant overlays require higher-level management approval

MRM for Vendor Models9 MRM for vendor models fits into broader approach for vendormanagement And linked to other supervisory guidance on vendor managementSame or similar standards for development, implementation & use applySample test data sent to vendor before purchase (with extreme values)Internal benchmark model built for more important vendor modelsAppropriate skepticism for vendor models deemed “industry standard”Obtain sufficient developmental evidence that allows for an evaluationof conceptual soundness during validationVendor model validation looks at the specific manner in which they willbe applied and implemented at that firm (e.g., checking all the settings)Clearly understand model limitations and assumptions of vendor modelsto determine whether the model is appropriate for the intended useUpdates to vendor methodology, data, etc are tracked and reviewed

Policies and Procedures10 Firm has policies that apply to models everywhere in the organization, andthen procedures that apply to specific modeling areasP&P are appropriately cross-referenced – for example, MRM policy hasdescription of audit’s role, but also links to firm’s broader audit standardsP&P tie to supervisory expectations (and not just for MRMG but otherareas such as vendor management, audit, and IT/change controls)Evidence that P&P serve as key guiding documents to help ensure practicesare consistent and sufficiently rigorous, and to minimize chances formisinterpretationClear to all in the firm which P&P are the “law of the land” and there arenegative consequences for not adhering to P&PP&P regularly updated – and not just ahead of examsFirm has mechanisms to assess conformance with P&P - such as QA function

Internal Audit11 Audit makes a full assessment of the whole MRM framework, not justdiscrete elements (e.g., not just validation)Planning and scopes are clearly and communicated ex anteAudit focuses on evaluating processes and controls, not individual models But conducts sampling and transaction testing to confirm Audit has sufficient standing in the organization to feel comfortablepointing out shortcomings and to have them addressedWork is consistent with other existing supervisory expectations for auditWork is rigorous, appropriately critical, and has clear conclusions Audit is able to identify thematic issues and root causesAudit specifies clearly what was covered and what was notRegular reports provided to senior mgmt and the board (or its delegate)Supervisors observe that issues they find in first and second line have beenalready identified by audit

Closing Thoughts12 As outlined above, we are seeing a lot of good practices and alot of resources devoted to MRMBut we note a few areas still needing attentionGood policies without compliance do not help Uneven application of better practices, even within a bank How to keep it going will be a challenge – ensure sustainabilityand regular updates Good to demonstrate within the firm that MRM is worth thesubstantial costs (not just to make supervisors happy) Supervisors will continue attention on MRM as firms progress We will maintain our outreach to the industry and help promotebetter practices

Break 9:45 – 10:00

MRM in CCARPanel DiscussionModerator: Vishant SharmaFederal Reserve Bank Of AtlantaSeptember, 2016Page 1

Agenda Panelist Introduction Panel Choice Discussion Q&A SessionPage 22

MRM in CCAR: Panelists Deniz Senturk, SVP/ Head of Model Risk Management, StateStreet Elizabeth Mays, Chief Model Risk Officer, PNC Angela Reindollar, Model Risk Director, Comerica BankPage 33

Managing Model Risk in CCAR/DFASTProcesses to Ensure Forward LookingRisk ManagementDeniz Senturk, State StreetPage 4

Managing Model Risk in CCAR/DFAST Processesto Ensure Forward Looking Risk Management:Critical Success FactorsCorrectly Scoping the Work, enabled by effective processesShared Accountability, enabled by effective reportingFull Transparency coupled with the Risk AppetiteOnly Eliminating Risk # 36 and Effectively Managing other RisksPage 5

Actionable ReportingModel Risk Appetite should be based on sources of riskacross the model life cycleRegular and actionable metrics/reporting will enable us tomonitor and take action to reduce model risk Input: Errors related to inputs Processing: Inadequate modelingmethodology or assumptions Output: Poor model performance,stability, or robustnessModel validation Deficiencies related to validationImplementation and planned useModel landscapeIllustrativeX% Percent of total modelsX%All modelsValidatedModel development % of models with “high” Errors related to moving model toproduction and planned usemodel riskIllustrative Number ofmodelsPost validation application and use Inappropriate actual use of model Inappropriate compensating controlsInherent model sensitivityScore: Uncontrollable risks which arise due toinherent uncertainty of modeled environmentHigh Inherent RiskScoreModerateResidualRisk ScoreLowAgile and simplified but comprehensive information sharing is the keyPage 6 X%% models currently not validatedDocumentationModel life cycleModel LandscapeModel developmentX%Not validatedX% CommentsComment on areas with highestconcentration of high/ risk modelsComment on interconnectedness amongmodels

CCAR Models: Key Sensitivity Factors% of Models Driven by Key FactorsModel Family139%34%32%31%Risk Rating2CCAR - AElevatedCCAR -BElevatedCCAR - CModerateCCAR - DModerateCCAR: EModerateCCAR: FModerate28%24%18%14%13%11%6%2%Interconnectedness of Regulatory modelsNodes highlighted for CCAR – Model APage 7Top 3 Factors(1) Liquidity(2) Interest Rates(3) Accounting/Balance sheet Factors;Credit; Equity; Foreign Exchange; InternalFactors; Macro Factors(1) Collateral/Pool Factors(2) Macro Factors(3) Deal/Bond Factors(1) Foreign Exchange(2) Equity(3) Liquidity(1) Credit; Equity; Interest Rates; InternalFactors(1) Accounting/Balance sheet Factors; ForeignExchange(1) Internal Factors(2) Credit(3) Equity; Interest Rates

Challenges in Managing the ModelDevelopment and Validation Timelinefor Stress-Testing ModelsElizabeth Mays, PNCPage 8

The ChallengeEvery YearMRM must complete severaldozen model reviews by a datethat is set in stone.A significant number of thesemodels are modified or fullyredeveloped.Developers want to work ontheir models as long aspossible, causing furtherdelays in validation.Validation work piles up in thelast few weeks before thesubmission.ConstraintsFinitenumber ofvalidationresourcesModels to beredevelopednot completelyclear until latesummer whenregulatoryfeedback isreceived

Parallel Review Process Allows Validation to beCompleted Shortly after Development s3ModelBuild4FinalModelModel DocumentationValidation Report5Validation Milestones12345Model DesignData ReviewPreliminaryAnalysisPreliminaryModel ReviewFinal ModelApprovalOngoing feedback from validators to developers,while maintaining independence“Observation and Request Log” documentsinteractions“Any Showstopper” Issues Identified earlyValidation finished shortly afterdevelopment