When Qualitative Data is MORE Accurate: Death by Data

Aaron pinged me with follow-up questions, telling me I'd been sniffing glue if I thought that quantitative data was less reliable than qualitative.  Gifted and respected colleague, I don't think you can ever say that one method is inherently superior to the other -- much like you can't say that Linux is more/less secure than Windows.  Both can be very highly secured, and both can be as secure as a wet paper sack.

I think that there can be cases where qualitative data is actually more reliable than quantitative data.  Quantitative research can create the illusion of reliable fact, because it has numbers, which makes is seem concrete and absolute.  However, due to bias and errors, quantitative data can be wildly inaccurate.  

As one example, the Space Shuttle Challenger loss (which occurred 27 years ago today, 27-January) was due to catastrophic failure of O-ring seals, and was precipitated by erosion of the O-rings on prior flights. The erosion of the seals by hot gases should never have occurred, and was an indicator of a failed test, and a failure mode of the O-ring.  However, because the engineers presumed that eroding completely through the O-ring was required in order for a failure to occur, and the erosion had only eaten through 1/3 of the O-ring after the timed test, they presumed that the duration and erosion were correlated.  The engineers made an unsupported leap of faith on very few data points that appeared to fit a linear progression (since humans are nothing if not pattern-matching machines, so they found a pattern where none existed, like a Rorschach ink blot).  From this erroneous extrapolation, they created an "erosion model" for the margin of safety.  After an additional test caused the data point for erosion to fall near the curve, they believed the bogus model was predictive, and treated failure modes as an actual Margin of Error.  This was quoted in a section of the Rogers Commission Report on the Space Shuttle Challenger by R.P. Feynman (http://www.ralentz.com/old/space/feynman-report.html) as having been a contributing factor to the disaster.  It was qualitative, fit a curve, and seemed to have significance (because they backed into the curve from the numbers), so seemed reliable.  However, it was really a chart of failure modes, not of safety, and was uncorrelated.

In this instance, the quantitative research was bogus, but believed because measurements and the fit to forecasted curve gave the illusion of science.