In medical terms, the suffix –oma is used to denote a morbid condition of a part, usually some kind of growth. According to Gary Johnson, managing director of Inpharmation, a forecast can suffer from various –oma conditions as well, when a forecast becomes too large and complex.
At the Pharma Forecasting Excellence USA conference sponsored by eyeforpharma, Johnson addressed the question, “Can a forecast be too clever for its own good?” Too often, in a fervor to craft a truly thorough, comprehensive model, forecasters instead create a model that is simply too complex, one that is subject to error and one that contains factors which may have little or no genuine impact on accuracy. So, what are the potential consequences of making forecasts that are too complicated?
Two types of forecast quality
Says Johnson, there are really two quality issues at work here. Quality type one is simply a measure of error – how many mistakes are there? Forecast models that involve a lot of hugely complicated code and formulae are wide open to error. Between one and five per cent of cells in a spreadsheet will contain an error – that’s the nature of human limitation – and the more complicated the spreadsheet, the higher the likelihood and number of errors. Additionally, the more complicated the code or formula of a cell in a spreadsheet, the harder that spreadsheet becomes to audit. Prewritten codes which can be debugged and then used over and over provide a solution. Proven codes and formulae which can be dropped into a spreadsheet easily and without fear of mistakes produce forecasts of high type one quality.
Quality type two has to do with building a model that works. The problem, according to Johnson, is that many models which look like models are in fact nothing more than highly sophisticated calculators. A close look at the numbers reveals that nearly all the most important information is based on guesswork. Numbers like projected market share, for example, are “pulled out of the air,” says Johnson, but their specious origins are disguised by the complexity of the model. As Johnson points out, “A complex spreadsheet which is populated largely by guesses is not a quality forecast.”
Zooming in on a spreadsheet until the “elemental level” is reached reveals the somewhat apocryphal nature of the numbers. “I assert that it’ll take X years for product Y to reach peak share.” The problem with this, Johnson says, is that people are notoriously poor guessers on issues like peak share.
A far better approach is a simpler one. What predictors of behavior are already known? For example, product uptake curves: will it be fast or slow? We know that uptake differs from country to country, and that these differences tend to be consistent. Product uptake is slower in the UK and faster in Italy, for example. If the new therapy fills a hole in the market, then uptake is generally much swifter. And if the new therapy fits with existing practices (switching from a tablet to a tablet), the uptake is quicker than if the therapy requires radical change in practices (patients will now be sensitive to sunlight and will have to come to your office after hours). Simple, evidence-based models which exploit known behaviors are far better than ones based on guesswork.
Even the experts don’t guess as well as we might imagine they will. As Paul Meehl posited in his book Clinical vs. Statistical Prediction, simple statistical models did as well or better than the experts in accurately predicting future behaviors. Says Johnson, if you’re forecasting something that happens over and over again, look for the simple patterns that are already out there, and use them as the basis for the forecast.
Keep it simple
In virtually every study done comparing the accuracy of complex models versus simple ones, the simple models are always at least as good and often better. Yet, if you ask forecasting experts if complexity generates a better, more accurate model, the overwhelming majority would say yes.
However, in studies of econometric models, evidence points to little difference between the two in terms of accuracy. And a study of extrapolation methods reveals that simpler models (like exponential smoothing) contained far fewer errors than more complex models (like Box-Jenkins). And this same phenomenon has shown up everywhere – in conjoint analysis, extrapolation, econometrics, even in expert judgment. Simplicity yields clarity; complexity fogs it.
Best of all is a combination of several methods. Viewing a problem from many angles provides the most complete view. The average of the numbers from several different forecasts is always more accurate than the numbers from any single model. Don’t have a favorite, Johnson says; combining techniques answers more questions and eliminates more errors.
Model-oma: creating a model that’s too complex
Several bad practices feed into the phenomenon Johnson terms “model-oma.” Word-oma is the tendency of people to sit around and brainstorm ideas, then throw them all into a spreadsheet before they’ve tested the validity of those ideas. This “rush to the spreadsheet” results in complex models full of untested ideas. Words, according to Johnson, must always be tested. It doesn’t really matter what doctors say they’re doing or what branch managers say is important, what matters are those actions that have genuine impact.
Event-oma causes forecasts full of non-events. Johnson gave as an example an occurrence in the UK. Doctors weren’t generally paid much and weren’t really performing all that well. The government came to the conclusion that the reason some very basic health care practices weren’t being done was because those practices were terribly difficult. To counter this, a system of rewards was put in place, and targets were set. One target had to do with statin use. Statin prescriptions did increase, and doctors claimed that the reward system had a great deal to do with their prescribing decision. However, a look at the trajectory reveals that statin use was already rising, and extrapolating the curve showed that the target and reward had little or no actual impact. And, of course, everyone accounted for the non-event of targets and rewards in their forecasts.
Segment-oma – the drive to include all possible segments in a forecast – should be subject to serious quality control, says Johnson. Before a segment is included, forecasters need to be certain that the segment is of appreciable size to merit inclusion and that there are measurable differences in behavior between the segments. Trying to include all the potential segments overcomplicates the forecast and risks including those with little or no discernible impact.
One example of segment-oma is the tendency of forecasters to assign too much importance to FDA labels. If a label says that this new drug can be used on people who’ve already tried that drug, for example, there’s an assumption that the people described necessarily become a target segment for the new therapy. However, studies of physician compliance demonstrate a certain lack of adherence to these labels – doctors are, generally, perfectly comfortable disregarding the label when making prescribing decisions. The label information is therefore useless in designating segments. Inclusion in a forecast should be reserved for genuine segments, Johnson says, based on proven facts about behaviors, not assumptions.
Technique-oma refers to the tendency among forecasters to use “heroic,” sophisticated techniques to produce forecast models. For example, conjoint analysis can be either “sublimely complicated,” Johnson says, or ridiculously simple. The technique of conjoint analysis sets out to build a computer model of the different elements of a drug that are important to doctors when they’re making prescription decisions. Each therapy has a set of attributes (speed of onset, for example), and these attributes contribute to its “likeability.” A drug that has fast onset and is easily administered is more inherently likeable than one that’s slow and burdensome to administer. Determining a drug’s likeability profile can be a complicated, arduous process (conjoint analysis), or it can be relatively simple. The simple, self explicated technique involves taking a drug, rating how good all its different attributes are and weighing the factors appropriately. This system works at least as well and often better than a more complex analysis. And the fancier version has the added disadvantage of fooling people into thinking that the conjoint analysis on its own is sufficient to build a forecast. It’s not.
There’s a great deal more to a drug’s success than its likeability quotient. So, asks Johnson, how important is it to add the impact of drivers other than the profile? Order of entry to the market is extremely important and should not be overlooked; likewise the efforts and resources allocated to a drug’s promotion have marked effects on the therapy’s success. Adding just these simple drivers lends a level of accuracy comparable to all the conjoint analysis.
The all-in-one model (conjoint plus drivers) provides a highly accurate prediction of therapy class shares. The more drivers and models that are added to the mix, the more accurate the results.
Common-sense models for the win
Studies bear out the notion that common sense and simplicity in forecasting are best, even in such a complex marketplace. As Johnson says, both quality types one and two are improved by not trying to be “too clever.” In fact, models that are too overthought can actually introduce complications that don’t actually exist in the market. Says Johnson, “simple forecasting techniques can estimate all the key numbers in a market model.” If you zoom in on your forecast, what does it reveal? Clarity or more confusion?
Author: Shannon Perry, journalist, theforecaster
