I started to use datarobot platform as free user.
By using the autopilot option I was able to build a regression model (light boosted tree) with a R2 of about 60% on test set. Then I deployed the model and tested it on a blind dataset. The R2 this time decreased from 60% to about 35%.
I tried to improve the model accuracy but w/o success.
Someone can help me?
Yes, my colleague has a good point. If your sample size is insufficient then you can also experience that difference. You could use the Learning Curves to help determine whether you sample size is too small.
Without knowing the number of features, or the size of the testing set I can just speculate the decrease of R2 during scoring (when using the model on test set). Here are 2 potential problems that you may have:
1. Sample size too small: not enough representative data in the training set.
Outcome: overfitting, models memorizes but don't learn the patterns.
Potential Solution: Increase the train set size
2. Maybe the model used is not the most appropriate.
Potential Solution: click on Get More Accuracy (as advised)
Hi firstname.lastname@example.org, thanks for the question. Were you able to check the feature impact and feature effects of the different datasets for comparison? Am thinking about drift causing your given results.