Get better stilts: Uncertainty and The Number

Here’s the latest from my friend, regional analysis wizard and former real estate developer Dr. Peter Mallow, in our series of explorations about how the way we do economic analysis often sets us up for trouble.

In this one, Pete is taking on one of my favorite we-all-know-it-but-we-don’t-want-to-admit-it-and-then-it-bites-us topics: the fact that even our best predictions are built on inherent uncertainties.  We can’t avoid that, but we can’t pretend it doesn’t exist, either.  So we ought to know what we’re looking at, and deal with it.

 

You can read Pete’s previous work here and here, and your can review an annotated version of a presentation we do here, and if you’re really a glutton you can listen to one of our presentations here.

Take it away, Pete!

___

If you read the past couple of posts on this topic, you learned that The Number can often be misleading or plain wrong.  For those that missed those posts, “The Number” refers to the new dollars, jobs, and taxes that an economic impact analysis claims will materialize from a new public project or a company coming to town. The expert or software has done some kind of magic with the data and returned a single Number that supposedly best describes how great the public project or company will be for the community.

People like The Number because it’s simple and it’s easy to understand.  However, it is almost always, by definition, wrong.  Even the best intentioned, well-meaning analysis is most certainly wrong when it reports one Number – that’s Statistics and Probability 101. We often excuse our Number’s lack of accuracy when we realize how far off The Number was through some variant of “garbage in, garbage out.”  We easily claim that the data or assumptions driving the analysis were flawed, and we can blame some combination of uncontrollable factors.

But this is an over-simplification of the problem. Uncertainty is, fundamentally the real problem, and most of the time uncertainty is the root cause of the Number turning out to be wrong.  Uncertainty exists everywhere in the analysis, whether the data is finely tuned or back of napkin.  Yet we give the uncertainty inherent in our analyses very little attention.

To better understand uncertainty, think of walking on stilts.  The smaller the base of the stilt the harder it is to balance and walk. The width of the base of the stilt represents how certain you are that The Number is actually correct.  In this case uncertainty takes the form of four different stilts – more on that in a moment. But think for a minute about the width of those stilts – how strong or weak, stable or wobbly, each one of them could be.

people on stilts
I think these guys need wider stilts. From Flickr Creative Commons

The more certain you think you are of your analysis results, the more narrow the range of results you will consider as possible outcomes.  If you’re so sure of your analysis that you can say, “this number is, most definitely, absolutely, the thing that is going to happen!”  then the stilt holding you up is very narrow- in fact, it’s only one number wide.  If you know that there’s a range of possible outcomes – if the results could vary – then admitting that range of possibilities means that your stilt is wider and more stable –its strength does not depend on just one number.

We know instinctively that wider stilts mean a stronger and safer walking experience.

If we build our plans on the basis of one Number, and we don’t account for other possibilities, then it is as though we are walking on very skinny stilts.  All it takes is a little variation, something relatively minor to go wrong, and the plans we made on the basis of those assumptions will go all to pieces.

 

There are four of these stilts, or types of uncertainty: The economists have defined them with the following words:

  • Stochastic,
  • Parameter,
  • Heterogeneity, and
  • Structural.

Don’t worry, we will peel back the jargon.

Here’s the main thing to remember: uncertainty is always present – you cannot escape it.  But by understanding the types of uncertainty, and carefully checking your “stilts” to make sure they are as wide and solid as possible, you can have greater confidence in The Number.

Here are the four basic types of uncertainty that we need to check our stilts for:

 

Stochastic Uncertainty (aka randomness)

Stochastic uncertainty is the randomness of life.  It basically means that you don’t know exactly what will happen until after the thing happens.  Here’s an example: if you toss a coin into the air, you know it will either be heads or tails. But which one?  You won’t know the answer until you toss the coin.

In terms of economic development there will always be some randomness about the project or new company that you cannot control nor predict – at least, not until after it has happened.

 

Parameter Uncertainty

A parameter is a set of measurable characteristics that define an object. How you define these characteristics is not set in stone, and if what happens differs from the parameters, then the results will be different as well.

For example, a high tech industry can be defined the types of jobs it contains (i.e. computer scientists, executives, sales people, administrative, engineers, etc.).  The specific mix of these jobs will be different for every company.  However, when you are looking at the economic analysis of a high tech industry, you will be working within a parameter – you will be using an assumption about the types of jobs that a new company within that industry will employ.

If a new company says it will bring in 1,000 new jobs, it’s possible that their employment could include any possible combination of job types that equal 1,000. But based on the type of industry, the economic impact analysis will probably assume a certain set of parameters – a typical or average or idealized mix of job types that it assumes the new business will create.  But this specific business might not fit those parameters.  If the new company ends up with a larger than typical number of remote sales jobs and administrative people, for example, then the parameter assumptions that fed into the economic impact analysis.  When that occurs, The Number will not reflect what actually happened.

Another way to think about parameter uncertainty is our coin example from before.  We know a fair coin has a 50 percent chance of landing heads.  However, if we toss it 100 hundred times and find that 54 were heads.  Is this wrong?  No, it is parameter uncertainty.  Just because we know the odds are 50-50 doesn’t mean that 54-46 isn’t entirely possible.

 

Heterogeneity Uncertainty

Heterogeneity is a complex way of saying no two jobs are the same. Take, for example, a cashier job at Costco and one at Wal-Mart. Both positions require the same tasks and responsibilities, but people working in those jobs may be making very different wages for doing fundamentally the same work. In the economic impact analysis, we usually assume an average or typical or idealized income, but what actually happens can vary widely from that assumption.

 

Structural Uncertainty

Structural uncertainty is inherent any type of methodological approach, like the process used to develop any kind of economic study.  Economic impact analyses can be done in a number of different ways, ranging from complex input/output methods, to simple arithmetic estimates, and any number of methods in between.  They can also be done for different time periods. The choice of the method, the time period and how the parameters interact cause structural uncertainty. Remember every model is an abstraction of reality. Yet all too often only one model and its parameters are provided as the best abstraction of reality.

Uncertainty is always present. If you don’t analyze how uncertainty impacts the assumptions that are holding up your studies, that uncertainty will eventually make matchsticks out of the wooden legs that you’re standing on. But there is good news: you can make informed decisions to reinforce your analysis based on your analysis of uncertainty.

Most importantly, exploring and admitting uncertainty will probably lead you to report The Number as range of possibilities – a set of numbers, rather than a single one.  Think of that range as the width of your stilts — the wider the range, the stronger the base, the less the uncertainty, and the more credible your estimates of jobs, dollars, and/or tax receipts will ultimately be.

Leave a Reply

Your email address will not be published. Required fields are marked *