Model Risk Management: Navigating SS1/23

Model risk is a risk in its own right. With the release of the Prudential Regulation Authority’s (PRA) Supervisory Statement (SS1/23) in May 2023, it is clear that UK regulators are serious about enforcing effective and sustainable model risk management (MRM) standards. Critically, UK banks* with approval to calculate regulatory capital requirements through internal models will need to assess whether their model risk management frameworks meet the expectations of the PRA. This is only the beginning of the process as the standard is expected to cover the entire banking sector in due course.

Most large banks will have established frameworks that cover model governance, lifecycle management, validation and model risk mitigants that are in line with the PRA’s principles. However, the statement’s intentionally broad definitions of what constitutes a model as well as how to apply model risk tiering could become a chronic impediment if not adequately and effectively addressed early on in the remediation journey.

As the 17 May 2024 deadline approaches, banks will need to identify and remediate deficiencies across their model risk management systems, processes and governance in a manner that is both rigorous and consistent.

*UK incorporated banks, building societies and PRA-designated firms.

A new bar is set

The Supervisory Standard (SS) aims to support firms in strengthening their policies, procedures and practices across the model lifecycle and model risk management through five core principles:

1. Model Identification and Model Risk Classification

2. Governance

3. Model Development, Implementation and Use

4. Independent Model Validation

5. Model Risk Mitigation

A significant challenge for financial institutions will be interpreting and applying the principles outlined in the standard – especially those that are intentionally broad or non-prescriptive (i.e., model definition and model risk classification). As a result, firms may find themselves stretching the existing capacity of their MRM capabilities as they apply a much broader definition of what constitutes a model.

However, the standard’s proportionality and risk-based considerations will be significant levers that senior management can use to direct their effort and oversight where it is especially required.

Principles and challenges

While the SS will stand alongside existing supervisory expectations, firms will need to conduct an initial selfassessment of their implemented MRM frameworks against the principles and prepare a remediation plan to address identified deficiencies and shortcomings.

Monocle's approach

Model identification and model risk classification (Workstream A)

Large banks can have hundreds or even thousands of models across the enterprise. The foundation of model risk management is therefore a comprehensive, reliable and risk-based view of all these computations with specific consideration for:

Model Definition. The PRA has provided a broad model definition that is wide enough to include computations
that may not ordinarily or historically have been classified as a model.

In order to meet the requirements of the PRA while ensuring a manageable and realistic model scope, Monocle recommends firms take a practical and structured approach to defining their model definition.

Therefore firms should consider a variety of factors that are critical for an effective model tiering framework such as model complexity, model use, business impact and a host of additional factors that have been incorporated into Monocle’s tiering approach.

As a starting point in model identification, firms should determine the models that are critical for strategic and operational decision-making. Further engagement should be considered with specific business stakeholders as well as model users/owners/developers to identify models and computations that were not previously included as part of model risk management oversight.

The PRA defines a model as “A quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into output. The definition of a model includes input data that are quantitative and/or qualitative in nature or expert judgement-based, and output that are quantitative or qualitative.“

“Notwithstanding the above definition, where material deterministic quantitative methods such as decision-based rules or algorithms that are not classified as a model have a material bearing on business decisions and are complex in nature, firms should consider whether to apply the relevant aspects of the MRM framework to these methods”

Tiering Criteria

Model risk management framework refinement (Workstream B)

Financial institutions will need to refine their model risk management frameworks to align with the PRA’s expectations across the five principles. Most firms will already have the necessary documented policies and procedures in place and will therefore refine and enhance their frameworks rather than completely overhaul them.

Model risk management gap analysis (Workstream C)

Once all applicable models have been identified and categorised by risk, a gap assessment should be performed to ensure proportional oversight and controls are in place – particularly for tier 1 and tier 2 models.

Model risk management framework remediation (Workstream D)

Addressing all identified shortcomings and deficiencies will require a structured remediation plan with achievable deadlines, considering the 12-month compliance period. A sprawling project such as this will require active
contribution from, and collaboration between, the three lines of defence – model owners/users/developers, model validators and internal audit – as well as review and approval from senior management.

Additionally, remediation offers the opportunity for enhancements and strategic improvements to be implemented, particularly as financial institutions make increasing use of advanced analytics that have inherent characteristics not accommodated by traditional model risk management frameworks.

Emerging model risk management trends

• Machine learning

Machine learning (ML), a prevalent form of artificial intelligence, uses quantitative models to fit a set of data to make predictions, recommendations or decisions without explicitly being programmed. This falls neatly into the Supervisory Statement’s expanded model definition without being mentioned directly.

Banks will need to consider two significant features of their ML models – their complexity as well as their business impact.

Complexity: ML applications are capable of being retrained on new data as it becomes available in order to adjust and potentially improve the model over time. This dynamism is a key difference compared to traditional statistical models but introduces the risk of model drift, which should be addressed through enhanced and continuous monitoring (using key model performance indicators) and validation.

Additionally, the statement refers to the risk factors of model “explainability” and transparency, as well as the risk of data bias – two significant issues particularly for convoluted neural networks. Financial institutions will need to pay special attention to this when risk tiering their ML models.

Business Impact: The Bank of England’s and Financial Conduct Authority’s latest machine learning survey found the business area of Treasury and Credit – arguably the most important functions in a bank – to hold the highest percentage of ML applications that were considered critical to the business area. With the SS’s focus on business decision impacts, banks will need to be aware of how machine learning outputs are supporting materially important business decisions, and categorise these models accordingly.

• Climate risk

Climate risk management is becoming a standard part of the risk management and reporting landscape –particularly climate risk scenario analysis and stress testing. However, climate forecasts contain substantial uncertainty due to a number of factors including non-linear outcomes, significant data gaps, reliance on various assumptions and proxies, long time frames and the novelty of forecasting physical and transition risks for the banking industry. Much like the intricacies of ML, banks will need to consider the idiosyncrasies of their climate risk models including their rapid advancement as well as their business impacts.

Rapid Advancement: Risk modelling capabilities around climate are improving as granular, sector-specific data becomes more readily available, benchmark climate scenarios become accepted by the industry and climate modelling becomes increasingly embedded in banks’ risk management frameworks. However, the pace of change requires banks to adopt a more agile approach to model validation that must find alternatives to back testing and other traditional model validation techniques ill-suited to climate risk. Banks can opt for sensitivity testing, benchmarking and reviewing the conceptual soundness of methodologies, key drivers and assumptions on a regular basis.

Business Impact: For most banks, their climate risk modelling will be focused on the impact on credit risk and its various stages including credit underwriting, impairment, collateral management, and regulatory capital. Climate models with significant impacts like these will need to be identified, categorised and managed accordingly. As is the case with AI and ML, the impact and rapid pace of change in the climate modelling space will need to be adequately assessed and addressed by banks.

How Monocle can assist

With Monocle’s four workstream project approach we aim to assist our clients to interpret and apply the Supervisory Statement while identifying and remediating any shortcomings and deficiencies.

Following an extensive assessment of the statement, Monocle has developed a robust methodology that will aid banks in their interpretation of each principle and guide them in the implementation thereof.

Monocle’s vast experience and comprehensive understanding of the statement ensures that, through a detailed model level gap analysis and the subsequent refinement of existing MRM frameworks, our clients will have a wellstructured workflow embedded in their business. Ultimately, this will optimise model risk operations and improve efficiency across all aspects of their model risk management capabilities.