How to Pick a Forecasting Model

Consider two race cars, each representing a forecasting model, that are set loose on a track with minimal room to accelerate before they begin to encounter obstacles in their lanes. Depending on how much room they have to get to speed and what obstacles they encounter, one car might seem faster than another at any given point. But how to determine which car is fastest overall?

A researcher can’t rely on their typical method, which is to use most of the historical data available to build the parameters for the model and then to use a smaller, more recent set of data to verify it. Where researchers choose to draw the line between the set used for making estimates and the set used for evaluation is known as “the splitting point.” Researchers try to pick a splitting point that leaves enough data in set one to build the model and enough in set two to evaluate it.

The process, says Smetanina, “is relatively ad hoc. There’s a lot of judgment and no methodology in the literature. But the choice is crucial.”

The proposed method avoids the splitting-point problem and instead examines each model’s performance against the other’s as a function of time—in effect, says Smetanina, comparing the performance of the two race cars over “all terrain.”

The result is a more robust model-selection process. Rather than rely on separate sets of historic data chopped into two epochs, Richter and Smetanina have developed a technique for estimating how well models perform under general conditions.

Ultimately, the researchers created two methods to compare forecasting performance—one that aggregates past performance to measure historical performance, and another that forecasts which model is most likely to outperform in the future. In an environment where things are changing, it can help to have these two perspectives, Smetanina explains. A model that has performed well in the past won’t necessarily continue doing so, and conversely, a model that hasn’t worked well before could do well in the future.