cancel
Showing results for 
Search instead for 
Did you mean: 

Introduction to Model Monitoring

Introduction to Model Monitoring

This article showcases the Model Monitoring capabilities that come with DataRobot, which include service health, data drift, and accuracy monitoring.

Deployments

Deploying a model as a REST API through DataRobot’s deploy method, means you have access to the wide range of monitoring capabilities available in the Deployments page.

Figure 1. Landing page of DataRobotFigure 1. Landing page of DataRobot

 

Figure 2. Deployments pageFigure 2. Deployments page

 

You’ll find a summary of all deployments that you have created (using DataRobot) or that have been shared with you. At the top of the page, you’ll see a summary for these deployments: how many there are, how many predictions have happened recently, and (most importantly) a synopsis of service health, data drift, and accuracy.

Below this information. You’ll find a list of all of the available deployments. To do a deep dive into one of them, you can just click on it.

Overview

When you select a model from the Deployments inventory page, DataRobot opens to that model’s Overview page. The overview provides a model-specific summary that describes the deployment, including the information you provided when creating the deployment and any model replacement activity.

You will see information similar to Figure 3.

Figure 3. Overview tabFigure 3. Overview tab

 

Service Health

The Service Health tab tracks metrics about a deployment’s ability to respond to prediction requests quickly and reliably. You can use this information to understand bottlenecks and assess capacity, which is critical to proper provisioning.

Figure 4. Service Health tabFigure 4. Service Health tab

 

 

Data Drift

When you deploy a model through DataRobot, training data is automatically uploaded with the model. The Data Drift dashboard leverages that data to help you to analyze a model’s performance after it has been deployed. The dashboard provides three interactive visualizations to identify the health of a deployed model over a specified time interval.


Figure 5. Data Drift tabFigure 5. Data Drift tab

Accuracy

The Accuracy tab enables you to analyze the performance of model deployments over time, using standard statistical measures and visualizations. Use this tool to determine whether a model’s quality is decaying and if you should consider replacing it. The Accuracy tab renders insights based on the problem type and its associated metrics—metrics that vary depending on regression or binary classification projects.

 

Figure 6. Accuracy tabFigure 6. Accuracy tab

 

Monitoring and Notification Settings

DataRobot provides a wide range of options to customize your deployments (as shown in Figures 7 and 8). For example, you can set a specific threshold for data drift based on your tolerance. When data drift is larger than that defined limit, DataRobot will flag the deployment’s data drift as either yellow or red, depending on the defined threshold.

Figure 7. Monitoring settingsFigure 7. Monitoring settings

 

Lastly, you can set the notifications you receive as well as the frequency for the notifications. Do you want to be warned only when something important happens, or do you want to receive an email notification every 2 hours even when your deployment is working as intended?

Figure 8. Notification settingsFigure 8. Notification settings

Sharing a Deployment

Every deployment in DataRobot can be shared with your colleagues. Figure 9 shows the procedure for sharing a deployment: simply insert the email of the person you want to share your project with.

Figure 9. Sharing a deploymentFigure 9. Sharing a deployment

 

There are also different roles you can assign based on the governance protocols of your company. An individual can be a consumer, a user, or an owner of a deployment. (More information on the exact permissions for each role can be found in the in-app documentation.)

More Information

If you’re a licensed DataRobot customer, search the in-app documentation for Manage deployments or Optimization metrics. Also search Authentication, roles, and permissions, then locate more information for "Deployment roles and permissions."

Contributors