I'm struggling to understand how feature effects differs from feature fit. I really just want to know which model I should move to production.
Which one tells me what i want to know about my model? And then can I just ignore the other?
Solved! Go to Solution.
The short answer is that the default settings for Feature Fit and Feature Effects answer two different questions.
Of course, as with all DataRobot visualizations and metrics, they work in conjunction to show you different aspects of your model. There are several ways to describe why and how they are different, but those descriptions may not all matter to your use case. So, in order to give you the most directed answer, I need to know more about your particular question.
Specifically, when you ask "Which one tells me what i want to know about my model?", what is it that you want to know so that you can make a decision?
Longer form, some ideas to consider:
Feature Fit and Feature Effects complement each other. Neither is best considered in a vacuum.
Feature Fit (Predictions and Actuals) and Feature Effects (Partial Dependence) used to be displayed in the same graph, called Model X-Ray. Several (many...) versions ago, these were split into two graphs because they do two different things.
In the broad sense, here is the difference between the two:
Feature Fit is on the Evaluation tab. Its default settings are showing you how well the model does when predicting a particular subset of data (whether bins or categories) within a feature.
For example: In the Lending Club dataset, B is the most common Grade, according to the distribution plot below the graph. How well a particular model does on predicting Bs ( or Cs, As, Ds, ...) might help in your evaluation of which model to choose among close competitors. Or which to winnow out. Evaluating different Grade values would also include all of the things that naturally go along with being in that category (i.e., other values that coincide with being graded as B or C or A).
Feature Effects is on the Understand tab. Its default settings can help the user understand what the model thinks the effect of that feature is (hence the name) on changing the outcome of predictions. To create this visualization, all other values in a sample are kept the same as they are in the original data, and different observed values of the feature in question are hypothetically tested in every row. That keeps all other things constant and sees how the model responds to isolating a change in the value of that one feature.
For example: with the same Lending Club example and a target looking at the riskiness of each loan, Partial Dependence shows what the model would think the collective risk was if we took a sample of applications and set them all to Grade A. What if we then set them all to B? What is the difference between the two? Let's say 3%. That would indicate that - all else being equal - the model thinks the difference between A and B grades is worth about 3%. That's not about understanding reality (actual A loans vs actual Bs, with changes in other features), it's about understanding the model. Since the feature is being isolated, by using Partial Dependence, I am learning how the model does respond, not how I think it "should" respond.
Since they are born from a common origin, one could swap from one to the other in the UI by simply changing the selected checkboxes to display different lines, but then you're using - and, IMO, thinking of - the functionality as one did with Model X-Ray. There are two purposes: Evaluating the model and Understanding the model. The defaults for each are set to assist each of those purposes.
Ahh, ok, so one is about how my model is performing, but the other helps me peer inside my model for how its working.
my feature fit results look good so I can use that to help justtify we should use this model
This is great - thank you!