How closely will the model forecast for the weeks or months ahead? The essential distinction from fitting accuracy is that you are now predicting into an unknown future, as opposed to reproducing the historical data whose measurements were known to you. This forecasting accuracy component is often called out-of-sample accuracy, since the future is necessarily outside the sample of historical data we have about the past.
There are three approaches that can be used to measure forecasting accuracy. These are:
Wait and see in real time
Use holdout samples
Create retrospective evaluationsWait and See is the time-honored practice: make your forecasts now and then compare them against the actual Sales that materialize. One recommendation is to be careful to keep the forecast horizon fixed at lead time or planning fence for your company.