cancel
Showing results for 
Search instead for 
Did you mean: 

rules of creating backtest

rules of creating backtest

Hello,

 

I tired to predict "Anomaly Detection" and created 5 backtest areas.

I thought that backtest's train/validation range would be same as default.

But when I created 5 backtests, the ranges of tran/validation were different.

What is the rule making backtest range?

anomaly.PNG

Labels (1)
1 Solution

Accepted Solutions

Hi @cookie_yamyam ,

Thanks for your question. 

Though ideally, we would want it to be the same across all backtests there are not enough data points in some parts of your training data.(You can see it in the observations histogram). The difference in backtest length I assume is due to that. 

DataRobot uses the following criteria to set backtests - For OTV, backtests require at least 20 rows in each validation and holdout fold and at least 100 rows in each training fold. For time series, backtests require at least 4 rows in validation and holdout and at least 20 rows in the training fold. 

Do note that this is the best heuristic suggestion based on tests across 1000's of TS datasets but still the user has the flexibility to edit and change each of the backtest.

I am attaching the Backtests doc for more information on this. 

 

Let me know if this solves your query.

View solution in original post

2 Replies

Hi @cookie_yamyam ,

Thanks for your question. 

Though ideally, we would want it to be the same across all backtests there are not enough data points in some parts of your training data.(You can see it in the observations histogram). The difference in backtest length I assume is due to that. 

DataRobot uses the following criteria to set backtests - For OTV, backtests require at least 20 rows in each validation and holdout fold and at least 100 rows in each training fold. For time series, backtests require at least 4 rows in validation and holdout and at least 20 rows in the training fold. 

Do note that this is the best heuristic suggestion based on tests across 1000's of TS datasets but still the user has the flexibility to edit and change each of the backtest.

I am attaching the Backtests doc for more information on this. 

 

Let me know if this solves your query.

jenD
DataRobot Employee
DataRobot Employee

@cookie_yamyam If this answer worked for you, can you mark this as accepted? Thanks!