subscribe : Posts | Comments

Human Biases Verses Statistical Models

0 comments

The genius of the human mind lies in the creation and systematic use of mathematical formulas, which are the underpinnings of science, rather than in intuitive leaps of faith. It is statistics which bestows science its predictability and objectivity. The ability to predict is the hallmark of science and separates it from fly by the seat of your pants conjecture. Most of us have seen Money Ball. It’s the true story based on a book how in 1999 how Paul DePodesta an economics major from Harvard joined the Oakland Athletics organization as an assistant to general manager Billy Beane. DePodesta was a key figure in Michael Lewis’ book Money Ball which thrust the analyt-ical principles of sabermetrics into the mainstream of baseball recruiting and turned the Oakland A’s into a winning team. DePodesta, a young Harvard economist, proposed the idea of selecting players based upon their statistical performance using sabermetrics, a sophisticated statistical analysis of baseball statistics which calculated new statistics such as runs above average (an es-timate of the runs a player through base running, hitting and defense was worth above of below an average player).

When applied to human behavior, predictability is a far more complicated affair than when ex-plaining the relatively simple behavior of chemicals, but statistics still provides controls over the error rate and builds a far more accurate model than any other approach.

The beauty of statistics is that it is a set of methods that can be applied to any and all subject areas. “A statistical consultant knows that he or she doesn’t need a medical degree to work on a research project with doctors”(Anderson, 2008). Of course the Statistician cannot initially throw everything into our model building but that’s where expert’s knowledge specific to the sector being analyzed (Transportation, Biotech, Educations, etc.) may be useful to zone in on which variables look most promising as predictors, but it is the mechanical mathematical process which validates them as such.

Discriminant analysis, for example, can be used to decide whether an experimental drug will succeed and the statistician needs no expertise pharmaceuticals but does need an expert in phar-macology to define the data to be entered into the statistical model. All that’s needed is know-ledge of the statistical procedure, how to apply it and the knowledge of a person who is an expert in the specific subject field to be measured to provide lots and lots of data!

There is a wide body of research that demonstrates decisions made by using simple objective standardized statistical procedures, even if they omit relevant variables, are far superior to those made by experts who try to mentally juggle variables, even if they have more variables at their disposable. Especially if they have a large number of variables at their disposable. Indeed the attempt to deal mentally with a large number of factors falls victim to biases and simply are sub-ject to failures according to everything we know about cognitive processes and their limitations. And far from being able to assess “the whole is greater than the sum of its parts”relationships, in situations where “everything influences everything”even experts are left with overloads, necessi-tating poor quality guessing while a simple statistical model can easily assemble interaction terms.

In every test simple statistical procedures beats expert“intuition”. Human judgment is predis-posed by the very way the brain evolved to be wrong more often than not.

As put by Ruscio (2002) after surveying 50 years of research on judgment “The accuracy of judgments made in a methodological way from just a few relevant pieces of information is …. superior to that of judgments made by experts who combine a wide array of information in an intuitive manner…unaided human judgment cannot compete with a more mechanical process that involves a comparatively simple combination of a small handful of relevant variables…unaided human judgment is simply incapable of dealing effectively with large amounts of complex information.”

Frost (2013) has stated “Studies in the cognitive and behavioral sciences have consistently shown that our brains are not up to the task. However, statistical analyses can solve these problems. Studies that date back to at least the 1950s have repeatedly shown that even simple statistical models can produce better predictions than expert judgment”.

As examples he offers “Orly Ashenfelter, a Princeton economist, fit regression models that pre-dict the price of wine vintages. These models include predictors such as temperatures and rain-fall, among others. Wine experts dismissed the regression models until the model’s predictions beat the experts to the punch on identifying several ‘vintages of the century” and “Statistician Vinny Bruzzese has developed a methodology to statistically analyze the story structure of the script in order to predict profitability. Based on the analysis, Bruzzese suggests alterations to the script that are designed to increase the predicted profits. He is reportedly expanding his business and some industry executives are saying that everyone will be doing this soon.”

Humans are filled with biases, cognitively overload with even a minute level of mental processing, and for motivational and ego purposes put on an air that they are weighing “all” the variables. Computers running statistical packages have none of these short-comings.

References:

Anderson, K. (2008) https://www.hitpages.com/doc/6125894942851072/2#pageTop

Frost,J.(2013) http://blog.minitab.com/blog/adventures-in-statistics/expanding-the-role-of-statistics-to-areas-traditionally-dominated-by-expert-judgment

Ruscio, J. (2002) The emptiness of holism, Skeptical Inquirer, 26, 2, p 46-50

Leave a Reply

Your email address will not be published. Required fields are marked *


*