Evaluating the Leaderboard and Beyond

cancel
Showing results for 
Search instead for 
Did you mean: 

Evaluating the Leaderboard and Beyond

(Article updated November 2020.)

This article discusses how to interpret the Leaderboard and explains the useful actions you can take from here. To access it, click the Models tab; the Leaderboard opens by default.

Figure 1. LeaderboardFigure 1. Leaderboard

Models are ranked on the Leaderboard based on an optimization metric. Although this metric is chosen automatically before you start your project, you can customize it in Advanced Options.  

Figure 2. Optimization metricFigure 2. Optimization metric

You can change how the metric is displayed on the Leaderboard by clicking on the Metric dropdown. 

Figure 3. Change MetricFigure 3. Change Metric

The icons next to each model tells you which tool was used to build it. The example below shows an XGBoost model, a Python model, and some DataRobot models. You may also see icons for R, Keras, and other open source tools. 

Figure 4. IconsFigure 4. Icons

The Feature List & Sample Size column shows which feature list and sample size was used for building the model.

Figure 5. Feature List and Sample SizeFigure 5. Feature List and Sample Size

You can create a new feature list on the Data tab, and then retrain any model on the Leaderboard using that feature list. Simply click on the icon next to the feature list name, and choose the list that you want to use. 

Figure 6. Feature RetrainFigure 6. Feature Retrain

Figure 7 shows how to retrain a model on a different feature list.

Figure 7. Retraining a model with a different feature listFigure 7. Retraining a model with a different feature list


In addition to different feature lists, the column also shows different sample sizes of the data, such as 100%, 80%, 64%, 32%, and 16%. To retrain a model using a new sample size, click on the icon next to the sample size (see Figure 8). 

Figure 8. Retraining a model with a different sample sizeFigure 8. Retraining a model with a different sample size

From the Learning Curves tab you can also examine the learning trajectory of models. The displayed graph shows how the performance changes when DataRobot uses different sample sizes. This is useful because it shows how well the models learn with the addition of more data. If the curves are steep, then it suggests that adding more data might improve performance; if they are flat, then it suggests that you have reached the threshold of diminishing returns with respect to adding more data to the model. 

Figure 9. Learning Curves tabFigure 9. Learning Curves tab

Under the Model Comparison tab, you can compare the performance of two models directly.  You can look at Dual Lift, Lift, ROC Curves, and Profit Curves for each model. 

Figure 10 (multiple different screens)

Figure 10 (multiple different screens)Figure 10 (multiple different screens)

 

Sometimes it’s important to know how fast your model is. You can also compare the speed and accuracy of each model on the Speed vs Accuracy tab, where the accuracy is on the Y-axis and the speed is on the X-axis. This is useful if you need to score your data at a very low latency. For example, blenders can be very accurate but have an increased processing time. In that scenario you may want to choose a model that has a much faster speed for a minor reduction in accuracy. 

Figure 11. Speed vs  AccuracyFigure 11. Speed vs  Accuracy

There are usually blenders on the Leaderboard. These are automatically generated in Autopilot, but can be turned off in Advanced Options before starting the project. You can easily build your own blenders with a few clicks on the Leaderboard. You can create a number of different blenders using Averages, Medians, GLM, and more (see Figure 12). 

Figure 12. Creating an Average blendFigure 12. Creating an Average blend

Finally, from the Leaderboard you can easily share your project with a colleague. Simply click on the Share icon at the top right of the browser, enter the email of the person you want to share it with, select a governance role, and click Share

Figure 13. Sharing a projectFigure 13. Sharing a project

More information

Community articles:

If you’re a licensed DataRobot customer, search the in-app Platform Documentation for Leaderboard overview.

Version history
Revision #:
12 of 12
Last update:
a week ago
Updated by:
 
Contributors