Forecasting

Forecasting realities and understanding randomness
On June 21st of this year, SAS hosted the first webcast in a two-part series called Worst Practices in Forecasting. Speaker Mike Gilliland, Marketing Manager for Forecasting for SAS, said it was time for companies to stop chasing the dream of a perfect forecast and start focusing on the realities of what forecasting can accomplish.

If a company has a product with a long history of stable demand, says Gilliland, given no new variables, high forecast accuracy should be possible. However, most products and services dont fit that profile. In that case, companies shouldnt invest vast resources trying to achieve results that simply arent obtainable. As Gilliland points out, when there is a high degree of randomness and volatility in demand, accuracy is limited.

So what do companies do that qualifies as worst practices? Well, lots of things. But the root problem seems to be that companies have unrealistic expectations of forecasting. A sober, reasoned approach might save companies money, time and frustration.

Making the model fit
One mistake companies make is to try to fit a forecasting model to past results and then use that model to predict future trends. Gilliland calls this overfitting the model to history. The objective, he reminds us, is to forecast the future; not manhandle the model until it explains the past.

Its easy to predict the past. A model can always be designed that will explain the past: Gilliland gives the example of stock market commentators, glibly explaining why the market did whatever odd thing it did that day. The information is interesting, perhaps, but not entirely useful when it comes to deciding where to put your money tomorrow. Says Gilliland, this technique of modeling depends on two assumptions:

The forecasting model captures some systematic pattern in past behavior that underlies market demand.
That same systematic pattern will continue, unchanged, into the future.

The problem is that these assumptions, and the models based on them, dont take into account the randomness of the marketplace. What if a companys product showed up on Oprah? The Oprah Winfrey effect causes demand for products to shoot up exponentially; no statistical model based on history could predict or account for such a radical shift in demand.

A better practice, according to Gilliland, is to fit the model to structure, not randomness. First, seek to understand the underlying pattern or rule that guides demand, the pattern you are trying to forecast, then fit the model to it.

What factors impact forecast accuracy for your product? If you know the structure that underlies demand for your product, if there is little randomness in or deviation from that structure, and if that structure doesnt change over time, than forecast accuracy is more possible. But, Gilliland reminds us, if you have a model that doesnt fit that structure, if there is considerable volatility and randomness in demand for your product, and if you can assume that there will be changes in buying behavior over time, then a high degree of accuracy becomes less possible. Says Gilliland, Have a realistic expectation of what accuracy is reasonable given your demand patterns.

The bad business of benchmarks
So where did all those unreasonable expectations come from? Well, one source is industry benchmarks. Says Gilliland, those best-in-class forecast performances need to be viewed with extreme caution. Gilliland notes that those benchmarks are often self-reported survey data, not subject to any sort of audit. The benchmarks may lack consistent measurements, and they dont take into consideration the relative difficulty or ease of forecasting certain products. Its possible that that best-in-class forecast was so good simply because it forecast a long-term, stable, highly forecastible product.

The problem arises when companies set these benchmarks as goals. Sometimes such high expectations are simply unreasonable. If your company experiences high volatility in demand, if you roll out lots of new products with short life cycles, if you do a lot of promotional activities that cause spikes in demand, then forecasting for you may simply be less accurate, given your history.

Better practices
Instead of focusing on forecast accuracy numbers, says Gilliland, the better practice is to concentrate efforts on continually improving your forecast process. Improve performance by reducing errors in the forecast, reducing biases and agendas that may negatively impact forecast results, increasing FVA (forecast value added), and dedicating fewer resources to achieving results that are as good as or better than those garnered from more costly models.

When management sets an unreasonable goal say, less than 20% mean absolute percent error (MAPE) forecasting staff may become demoralized or even cheat to accomplish those goals. Equally, a goal that is set too low may be achieved too easily, perhaps matching the accuracy that could be accomplished by a much less expensive nave forecast.

A far better practice, says Gilliland, is to ignore others in the field who may have better forecasting numbers but who may also have far greater forecastibility built in to their products. Focus instead on setting reasonable accuracy objectives, objectives based on your products and their demand histories. A nave forecast can help you set a baseline for accuracy, the lowest level you should be able to achieve. Then set out to beat that baseline. If you continue to refine your process, decreasing errors and noise, increasing accuracy and efficiency, then you have truly succeeded at forecasting.

Author: Shannon Perry, journalist, theforecaster