What is Model Governance and Why it’s Important

Businesses including financial institutions, pharmaceuticals, manufacturing, and more rely on data scientists to create models to improve specific functions of the business, like fraud prevention. Once these models leave the lab environment, there’s very little visibility on what happens to them. Questions like, “How are the models being deployed?” and “Are the models making the right decisions?” and “Are there biases?” must be answered.

During development, machine learning models are bound by specific assumptions, rules, and expectations. Once the models are deployed, the results in the real world can differ significantly from the results in controlled development environments.

Having good documentation showing the lifecycle is useful but insufficient when models become unreliable. AI model governance brings accountability and traceability to machine learning models – they ask and give answers to these crucial questions:

• What input variables are coming into the model?
• What are the output variables?
• What were the previous versions like?
• Who has access to the model?
• Has there been any unauthorized access?
• How is the model behaving with certain metrics?

Part of what makes it challenging to answer these questions is the current state of enterprise data science. It’s evolved into a siloed situation where scientists use different toolsets to develop their models. Some develop models on SAS, and others use R, Python, or any of the many machine learning software libraries available.

Because they’re operating in their own silos, it creates a challenge for a central IT or governing organization to ensure proper governance and audit models across the business. In this article, we’re going to take a closer look at model governance, model risk, model risk management, and the best ways to stay in compliance with your AI/ML models.

What is Model Governance in machine learning?

Regulatory compliance Case

Model governance in machine learning comprises multiple areas. The first one revolves around knowing whether your model behaves the way you want them to behave compared to when it was trained. Most models are trained with a limited data set. For example, you might train the data based on a small city in California. When you run that same model for the entire country, it will likely have a different output and decision-making.

So you need some sort of model governance to make sure you’re able to capture whether your model has bias, drift, or other performance concerns. This can be particularly important for financial institutions in finance when you’re using a model for credit or loan approval where you need to justify why you are rejecting certain demographics. When and if somebody challenges a bank or a regulated industry, they need to provide evidence and an audit report to show they are in regulatory compliance.

Reputation Management Case

The other important aspect to model governance is reputation management. We’ve seen cases with social media companies where their algorithm ends up labeling population groups with what looks to the public as racist. Although the people behind the company didn’t mean for this to happen, they must still manage the public relations disaster, reputation hit, and loss of substantial profits.

Business Impact

Even where there might not be impact to compliance or reputation, models can have negative consequences to the bottom lines of the company. For example, a model that was trained to gain operational efficiencies or improve profits might be shown to work well in the lab. But when deployed in production with real-world data, the same models might behave differently and result in financial losses. The perfect example of this is a 2021 news article on how an online real estate company lost billions of dollars because they failed to realize their models were not behaving as intended.

What is Model Risk Governance?

Model risk governance is focused on risk compliance elements for regulated industries like health care and financial services. These industries receive a lot more scrutiny from the government on how to measure risk.

In the financial services sector, loan approval gets a lot of attention as well as investment choices from financial advisors. Healthcare models are now making decisions on whether a person has a particular illness or not, so a wrong decision based on lack of data can be a huge issue.

Risk governance allows for a third party to come in and audit your model for risk and compliance issues. Validating data in the lab is easy, but working with a partner that can validate models with actual production data is a much better way to manage risk.

How Do You Create a Governance Model?

First and foremost, even before any models are being built, the business needs to clearly articulate and document the goals for these models. From there, think about the various issues that you want to keep track of. Then build a governance framework around that.

There are multiple ways you can create a governance model – you can inject some sort of SDK into what the data scientists are doing as part of the code development, for example. However, this is solely focused on validating what the data scientists are doing in the lab environment. This method sometimes misses what that means in a production environment.

Most important is creating a governance model with the capability to validate a model in production without touching the machine learning models directly. Ideally, a third party could come in and monitor decisions on the output of a model without touching the code directly.

Model Governance in AI and ML

Artificial intelligence (AI) is an umbrella term and machine learning (ML) is a subset of AI, AI doesn’t have to be machine learning. You can create a simple mathematical regression model that is also an AI model – it doesn’t have to be ML. Machine learning goes a little deeper where you enable it to do machine-based learning, whereas with AI, you use a more direct code-based approach that is still intelligent but doesn’t go quite as deep as ML.

This is an essential point as with the popularity of machine learning; many tools are focused primarily on machine learning governance and explainability.

However, many banks have been using models for years, and they are in the form of various mathematical models that allow them to address various use cases. Some of these use cases include loan approvals, anti-money laundering, customer acquisition, and more. Even though many of these models are not based on machine learning, they are a form of AI models.

Model governance needs to work for both AI and ML since many enterprises that adopted AI many years ago will need governance tools to integrate with their existing infrastructure, databases, software, and more

Making Your Model Comply with Relevant Regulations

Regulatory compliance is something that every business in AI is trying to achieve. For example, in Europe, today, companies are trying to have some sort of report that shows their models are compliant with local regulations. Currently, there is no tool that does that; you have to build some kind of template.

Third-party companies like Datatron provide solutions for this by taking in data and additional logs from companies and feeding it into a kind of regulation template for the country or area to help ensure models are compliant with relevant regulations.

How to Monitor Models Post Deployment

In the past, post-deployment monitoring required more work for data scientists beyond developing their models. Today, companies like Datatron can provide post-deployment monitoring without the need to tap into the model itself. Now, output data as well as machine, system, infrastructure logs plus metrics, and other information is used to monitor AI models running in live production.

Why You Should Care about Model Governance

As already noted, the three biggest reasons to care about governance are compliance, reputation management, and business impact. Going deeper, as revolutionary as artificial intelligence is, it’s still in its early developmental stages compared to the traditional software products versus what it will be in the future. AI is not perfect and needs to be checked to see if it makes the right decisions. There’s simply too much at stake.

Many times AI models are simply not making the right decisions. There are countless examples of this happening on a daily basis. Even if a model is trained correctly, over time it will experience drift – it will change. Because of the inevitable drift, you need a way to monitor and capture that change so the business and their data scientists can see what changed and make adjustments to the model.

For example, before COVID, a model could accurately predict how people go to restaurants to have dinner based on foot traffic. But post-COVID that model no longer makes sense.

The Future of Model Governance

Although most of model governance today focuses on model risk management around compliance, we’re seeing a trend towards social responsibility. There are underlying impacts that haven’t been fully surfaced outside these areas. Some prominent social media companies have come under fire, as well as one of the most popular real estate platforms for issues with their AI algorithms.

These issues were not in violation of any government regulations. Still, the financial fallout from poorly behaving models could generate bad press and significantly impact the company’s bottom line.

Just like how there is still a growing list of monitoring tools and methodologies for software, there will be an ever-increasing list of similar software for AI. AI/ML is substantially more complex, and we are only in its infancy.

If these companies had been using an advanced production risk management governance tool like Datatron, they could have avoided costly AI errors and avoided public embarrassment and financial loss.

What to Look for in an AI Governance Solution

The best AI model governance solutions focus on simplifying the entire process of monitoring and compliance assurance, while not causing significant disruptions to existing workflows and allowing key stakeholders in the organizations to have visibility into what the models are doing.

Most solutions today try to focus on how to help the data scientist figure out whether models are running correctly in the lab environment with a limited training data set. Furthermore, it requires specific integration that adds to the data scientists’ workflow.

It is also critical to use a governance tool that is agnostic to the model development platform, thereby empowering data scientists to develop best-in-class models using the right ML development platform. The ML world is too nascent; there is no such thing as a end-to-end platform that can handle the entire lifecycle of development to deployment. Enterprises need to be cautious not to buy into the marketing nonsense and only find themselves locked into an overpriced tool that brings no benefits. Even worse, it further creates dissolution in the enterprises that AI projects are a waste of money.

However, what is more relevant and helpful, is measuring in a production environment without interfering with or tapping into what the data scientists are doing. The best way to think of this question is in layers of defense.

The first line of defense starts with the data scientist, whose job is to ensure model validation using some kind of explainability tool, which is necessary from a developmental perspective but not good enough for actual production validity.

The second line of defense helps with the production aspect by monitoring elements while the model is working. The third and most sophisticated line of defense has the capability to provide a detailed audit report that shows what the models are doing with only the output data.

Model Governance Checklist

When choosing a model risk solution for risk management, make sure you choose a platform that goes beyond just compliance control. All enterprise-level businesses need to be confident about their model inventory, model development, and model management practices.

Make sure your AI monitoring and governance solution offers the following:

• A dashboard that quickly gives you an overview of your AI model’s health in real-time
• An overall health score that provides intuitive and easy-to-understand measures of all models operating in your system
• Bias, drift, performance, and anomaly detection
• Alerts for when a model varies from its pre-defined performance thresholds
• Custom metrics to align with your organization’s KPI metrics
• Activity log & audit trail
• An agnostic tool that can work with the various popular ML development platforms
• Integrate with existing infrastructure, including databases, software, and more