How do holdout score and forecast distance relate?
I am trying to build time automated time series model. I need models that run inference 1 to 3 time units into the future. I can train models and get a RMSE holdout score to rank the models.
However, is that score only based on 1 time unit in the future inferences or 3 time unit inferences in the future? How exactly is holdout score calculated when you want forecast distance to be more than 1? Essentially is the evaluation running multiple 3 step forward inferences and scoring them and factoring that into the RMSE if I set forecast distance to (1,3)?
Hi, the holdout score is based on all time steps into the future that you've specified in the forecast distance. In your example, the model is running inference on 1 time step into the future, 2 time steps into the future, and 3 time steps into the future. The holdout score you see is calculated using these 3 data points for each date within the specified holdout timespan.
The only exception would be the dates near the end of the holdout set when we can't see that far into the future. For example, let's say the last date in our holdout set is May 5, 2023. For May 4, we'd only be able to run inference 1 time step into the future.
You can also get a sense of how RMSE changes with forecast distance by looking at the Forecasting Accuracy plot under the Evaluate tab for your model (see screenshot).