Humility Rules

  

The Humility tab is useful for configuring rules that enable models to recognize, in real-time, when they make uncertain predictions or receive data they haven't seen before. In these cases, you can specify actions to take for individual predictions, allowing you to have desired behaviors with rules that depend on different triggers. You use humility rules to associate trigger conditions with corresponding actions. Using humility rules helps to mitigate risk for model predictions in production.

Figure 1. Humility tabFigure 1. Humility tab

You create humility rules for a deployment from the Humility > Rules tab. If you have not yet enabled humility for a model, toggle on Enable humility. Then, create a new rule by selecting Add Rule. Alternatively, you can select Use Recommended Rules, which will generate two automatically configured humility rules: one is a trigger for uncertain predictions, and the other if for handling outliers for the most important numeric feature.

Clicking Add Rule brings up options for configuring the new rule. Click the pencil icon to give the new rule a name.  Directly below are two dropdown lists: one to select a trigger that detects a rule violation and one to select the action to apply to the violating prediction (i.e., the trigger).

Figure 2. Add Rule buttonFigure 2. Add Rule button Figure 3. New rule entryFigure 3. New rule entry

Creating a new humility rule

There are three triggers available, and each trigger has additional values to configure how it detects violations. 

Trigger types

Figure 4. Uncertain PredictionsFigure 4. Uncertain Predictions

Uncertain Prediction detects whether a prediction's value that falls between defined thresholds. You set the lower-bound and upper-bound thresholds for prediction values, by either entering the values manually or clicking Calculate to use computed values derived from the holdout set. For regression models, the trigger detects any values outside of the configured thresholds; for binary classification models, the trigger detects any prediction’s probability value that is inside of the thresholds.

Figure 5. Outlying inputs (outliers)Figure 5. Outlying inputs (outliers)

Outlying Input detects if the input value of a numeric feature is outside of the configured thresholds. To configure the outlying input trigger, select a numeric feature and set the lower-bound and upper-bound thresholds for its input values. Enter the values manually or click Calculate to use computed thresholds derived from the training data of the model (computed thresholds is available only for models built within DataRobot). 

Figure 6. Low Observation Region (input value outside thresholds)Figure 6. Low Observation Region (input value outside thresholds)

Low Observation Region detects if the input value of a categorical feature is not included in the list of specified values. For this trigger, select a categorical feature and enter one or more values. Any input value that appears in prediction requests but does not match the indicated values triggers the corresponding action.

Actions

After you select any trigger, choose a corresponding action to apply for a rule violation. There are three actions available; each can be used with any of the three triggers.

Figure 7. No operation actionFigure 7. No operation action

No operation results in no changes being made to the detected prediction value.

Figure 8. Override prediction actionFigure 8. Override prediction action

  

Override prediction modifies predicted values for rows violating the trigger with the value configured by the action. Set a value that will overwrite the returned value for predictions violating the trigger. For binary classification and multiclass models, the indicated value can be set to either of the model’s class labels (e.g., “True” or “False”). For regression models, manually enter a value or use the maximum, minimum, or mean provided by DataRobot (available only for models built within DataRobot).

Figure 9. Throw Error actionFigure 9. Throw Error action

Throw error returns a 480 HTTP error with the predictions for any row in violation of the trigger.  This also contributes to the data error rate on the Service Health tab. You can use the default error message provided, or specify your own custom error message. This 480 HTTP error and message will appear as part of the predictions.

When you’ve finished configuring the rule, click Add to save it.

When you’re set with all humility rule changes for the deployment, click Submit; if you navigate away from the tab before submitting the rule changes, they will not be saved.


Figure 10. Add/Save humility rulesFigure 10. Add/Save humility rules

Once saved and submitted, DataRobot will monitor the deployment using the new rules along with any previously created ones. After a rule is created, the prediction response body returns the humility object. 

Figure 11. Example API responseFigure 11. Example API response

  

Editing humility rules

To edit a rule, select the pencil icon for that rule next to the name of the rule. You can change the trigger, action, and any associated values for a rule. When you are finished, click Save changes.

Figure 12. Edit rulesFigure 12. Edit rules

 

To delete a rule, select the trash can icon for that rule.

To re-order the rules listed, simply drag-and-drop them to the desired order.

Figure 13. Reorder rulesFigure 13. Reorder rules

 

After you have made any of the changes described above, click Submit to finalize your edits (in addition to having selected Save changes after editing a rule). Submit is the only way to permanently save rule changes.

Viewing humility data over time 

After configuring rules and making predictions with humility monitoring enabled, you can view the humility data collected for a deployment from the Humility > Summary tab.

Figure 14. View collected humility dataFigure 14. View collected humility data

  

The X-axis measures the range of time during which predictions have been made for the deployment. The Y-axis measures the number of times humility rules have triggered for the given period of time.

The controls for model version and the data time range selectors work the same as those available on the Data Drift tab.

More information

Check out these Community resources:

And these DataRobot.com resources:

If you’re a licensed DataRobot customer, search the in-app Platform Documentation for Humility tab.

Questions or comments?

We'd like to hear from you, about this feature or others! If you have any questions or comments, please click Comment and add them here. Looking forward to hearing from the community.

Labels (2)
Version history
Revision #:
5 of 5
Last update:
3 weeks ago
Updated by:
 
Contributors