What is Model Risk Management

The number of large institutions relying on models is rising and the rate will continue across all industries into the future. The benefits of the models created with advanced analytics techniques are too great for companies and organizations to ignore. They must build their own models to compete.

The types of models that are emerging in this new landscape continue to grow. Financial institutions have been using quantitative models and other types of models for some time to meet regulatory requirements, such as provisioning and stress testing. Now models are designed for other business needs like pricing, planning, and asset-liquid management.

Big data advanced analytics and the need to manage data analytics are driving sophisticated models designed for customer relationship management, anti-money laundering, and fraud detection.

As the dependency on models continues to grow so does the need for model risk management. Financial institutions have already invested substantial resources in developing sophisticated model risk management frameworks. However, each model is unique in its complexity and one institution’s Model Risk Management (MRM) framework will not necessarily be the solution for another.

In this article, we’ll look at model risk management and break down all of its components so you can get a clear picture of what it is, what it’s important, and best practices for managing your own model risks.

What is a Model in Model Risk Management?

The “model” part in the term “model risk management” is the designed and engineered model that an organization is using and which is being assessed for risk. Model risk management activities are active throughout a model lifecycle from development to deployment and beyond. Model management is critical to mitigate model risks and identify incorrect or misused models before they cause irreparable damage.

What is Model Risk Management?

Model risk management is a term that describes the management of a model’s risk. To understand model risk management you must understand model risk – the thing it’s designed to manage.

With the recent big data revolution, predictive models are being integrated into more and more business processes. This comes with many benefits, but also exposes institutions to new kinds of risks.

When businesses are making decisions based on bad models, the consequence can be severe. Prior to the financial crisis of 2088, model risk management within financial institutions was driven by best practices rather than regulatory standards. The crisis resulted in regulators around the world reigning in model risk issues across the industry to better assess capital adequacy among other major exposures that come into play during a financial downturn.

In 2011, the Federal Reserve Board (FRB) and the Office of Comptroller of the Currency (OCC) issued a joint regulation specifically targeting Model Risk Management ( SR 11-7 and OCC Bulletin 2011-12). This regulation laid the foundation for assessing model risk for financial institutions around the world.

Although it was the model failure in financial institutions that spurred this push, today data scientists are building models for various industries not just financial models but still must adhere to the same model risk management standards.

Model Risk Management Framework

The FDIC has set a standard for MRM that can be broken down into three main components.

Model Development

The first responsibility to manage model risk falls on those who develop, implement, and use the models. Model development relies on the experience and judgment of the data scientist building the models. Development and model implementation processes should align with model governance and control policies.

Model Validation

Prior to deployment models should be reviewed by an independent group for model validation. Model validation is the set of processes and activities intended to independently verify that models are performing as designed. The model validation process is intended to provide an effective challenge to each model and reveal any model error in design.

The Key elements of validation include:

1. Evaluation of conceptual soundness. The model is assessed for quality design and construction. A review is conducted examining documentation and evidence supporting the methods used and variables selected for the model.
2. Ongoing Model Monitoring. This step is done to confirm the model is performing as intended. It’s critical to see whether changes in products, exposures, market conditions, and other variables require an adjustment to the model design.
3. Outcomes Analysis. This step of validation is important because it compares model outputs to actual outcomes. Model forecasts are compared to predictions to check for accuracy.

Model Governance

Model governance provides explicit support and structure to risk management functions through policies defining relevant risk management activities, procedures that implement those policies, allocation of resources, and mechanisms for testing that policies and procedures are being carried out as specified. This should include documentation of model development and validation with enough detail to allow parties unfamiliar with a model to understand how it operates, including its limitations and key assumptions.

What are the Types of Model Risk?

Model risk is defined as the potential loss an institution may incur as a consequence of decisions that are based on the output of internal models as a result of errors in the development, implementation, or use of models.

Model risk arises primarily because of potential errors in the models and inappropriate usage or implementation of a model. These errors and inaccuracies can cause significant monetary losses, poor organizational decision-making, and cause reputational damage.

To best understand model risk it’s best to clearly define what is a model. A model is a quantitative system or mathematical representation that processes input data to derive quantitative estimates or different variables.

A model contains a set of variable assumptions and data for inputs, processes, outputs, and scenarios. It applies mathematical, statistical, financial, and economic data and techniques in a model. A model contains three major components:

1. Inputs: Data and assumptions of the model
2. Process: Processes that transform inputs into quantitative estimates
3. Reporting: Expression of estimates into valuable information for management

The following are common sources of model risk:

Bad Data: The data one uses in a model can be inaccurate, incomplete, or distorted. Clean and accurate data is critical for developing an effective model. Bad data can compromise the entire model and introduce a significant amount of risk.

Improper Implementation: The incorrect or incomplete implementation of a model can lead to inaccurate or erroneous results. This leads to the significant risk of poor organizational decision-making and possible reputational damage.

Methodology Errors: Statistical methodologies have their own errors, such as sampling errors and standard errors. These errors occur in regression modeling.
Unrealistic Assumptions: Unrealistic and incorrect assumptions can alter the intended parameters of a model and introduce risk. When fitting a model’s parameters, one error can alter the model’s calibration.

Misinterpretation: Misinterpreting the results of a model introduces significant risks. With misinterpreted results a misinformed course of action often follows.

Wrong Model: Getting ahead of the competition in business is exciting but can cloud judgment. Wrongful model usage is the failure to understand how a model applies to a challenge can doom an opportunity and result in a significant loss in resources.

What are the Basic Principles of Risk Management?

Although our focus is specifically on model risk management, the basic principles of risk management still apply.

• Organizational Context: Every organization is affected by various factors in its environment. For example, an organization may be immune to change in import duty whereas a different organization operating in the same sector could be at risk. There are also differences in communication channels, institutional risk culture, and risk management procedures.
• Stakeholder Involvement: The risk management process involves stakeholders at nearly every step of decision-making. Often they remain aware of even the smallest decisions made.
• Organizational Objectives: When considering risk it’s important to keep the organizations’ objectives in mind. The risk management process should address uncertainty.
• Reporting: Communication is foundational to risk management. The accuracy of the information is equally important. Decisions should be made on the best available information.
• Roles and Responsibilities: Risk management must be transparent and inclusive. It should take into account the human factors and ensure that everyone knows their roles at each stage.
• Support Structure: Team members should be dynamic, diligent, and responsive to change. Every member should understand their intervention at each stage of the project management lifecycle.
• Early Warning Indicators: Keep track of early signs of risk translation into a problem.
• Review Cycle: Keep evaluating inputs at each step of the process. Identify, assess, respond and review.

Model Risk Management Framework

A model risk management framework combines strong governance principles with thorough documentation in the design, development, validation, and deployment of a model.

The key elements of a model risk management framework should include:

• Model Definition: The problem statement and intended purpose of the model output, including all development decisions, techniques, and datasets used.
• Risk Governance: Establishing a clear process to evaluate risks across each model including policies, procedures, and controls to be implemented.
• Lifecycle Management: Identifying dependencies and factors of importance for continued usage of the model through its lifecycle.
• Assessment: Independently assessing and verifying to ensure all assumptions and decisions made during development are appropriate.

A proper framework helps organizations prepare for model risk management. Fulfilling the practical aspects of implementation and ongoing monitoring of risk requires the use of a software solution called Machine Learning Operations (MLOps).

What are the Tools used in Model Risk Management?

The growth in machine learning applications has resulted in new tools to help with model risk management. In particular, machine learning operations (MLOps) are used for the full lifecycle of building models, testing data and algorithms, deploying models at scale, maintaining models for ongoing relevance, and providing a system to record and document how models work and why they should be trusted.

MLOps is defined as a set of practices at the interaction of machine learning, DevOps, and Data Engineering. MLOps is mostly focused on the deployment and management of models into production which directly ties into model risk management.

Its primary goals are to operationalize and automate model development and deployment processes at scale and improve the quality of production models. This is critical for the efficient and effective achievement of business and regulatory requirements.

How MLOps Helps with Model Risk Management

An MLOps platform automates systematic processes for ensuring that a model is built and validated properly. MLOps provides four key capabilities across the data science lifecycle:

Management

Scaling data science and models requires robust project management capabilities. This helps avoid key-person risk, which can impede progress when a data scientist suddenly leaves for a new job. When they leave, they often leave with critical information nobody else understands. MLOps enables collaboration and governance of complex models.

Development

The core work for the data scientist is building models. For large organizations, scaling the work of model building comes with big obstacles. For efficiency, data scientists need instant access to a broad array of tools, languages, libraries, integrated development environments, and compute resources. Not having an integrated platform results in a substantial loss of productivity. Enterprises are not able to keep track of all related artifact versions with respective software elements at scale without an MLOps platform.

Deployment

MLOps allows for fast and flexible deployment of models at scale because it eliminates the need for data scientists to rely on IT or software developers to deploy each model. An MLOps platform ensures that all resources required for deployment are immediately available to stakeholders.

Monitoring

MLOps integrates a strong model maintenance system. It includes model retraining and rebuilding with a full history and context of original melding work and previous versions are kept intact and are easily understood by any stakeholder.

Conclusion

The use of Model Risk Management is critical for avoiding the unintended consequences from model development, inputs, or outputs.

A proper model risk management framework implemented with the right MLOps platform is the safest and most resource-effective way for organizations and enterprises to operate. To keep pace with AI and ML developments, model risk management must perform end-to-end management for models.

Today, the majority of MRM functions do not have comprehensive standards tailored for AI and ML. These are needed to address specific challenges, including bias detection, ethical questions, and explainability. Further vulnerabilities are caused by a lack of appropriate AI and ML tools and infrastructure.

Datatron was built on solving this pain point, and as a result, speeds up deployments, detects problems early, and ultimately leads to increased efficiency and the ability to manage multiple models at scale.