Quantitative v. Qualitative

"Science is what we understand well enough to explain to a computer.  Art is everything else we do."

-Donald Knuth

Quantitative research is conducted using empirical information that can be sensed and measured.  The intent for quantitative analysis is to provide for a means of analysis via mathematics, statistical, or computational analysis.  Quantitative research is often, but not exclusively, based on establishing a hypothesis, and conducting experiments provides for a testable hypothesis based on the ability to measure.  By its nature, quantitative research provides for (presumably) precise measurements within a sample of the whole problem space, and uses inference to make assumptions about the state of the whole.  Quantitative analysis gives up on being able to measure the complete state of the whole, so that precise measurements can be made.

Qualitative research is comparatively new as a means of conducting research yielding truth.  Through the use of qualitative research methods, researchers are able to create new understandings of problems that can be quite useful.  It seems that qualitative research tries to get a sense of the whole picture, painting a landscape of the entire problem space, then seeking to fill in the missing gaps to develop a working hypothesis.  Qualitative analysis abandons the notion of precise measurements, preferring instead to gain a sense of the complete state of the whole problem space.

To provide a gross generalization, I think of two people shooting arrows at a target, Jane and Bob, who both score 50 points in an archery competition.  Jane has the finest archery equipment, but poor eyesight.  Bob has the eyes of an eagle, but poor equipment.  When the two shoot, Jane gets consistently tight patterns, grouping 10 arrows in a few square inches, but her shot misses the bullseye entirely, and she scores 10 scores of 5.  Bob's poor equipment means his arrows aren't going precisely where he is aiming, but his aim is true.  Bob has arrows across the target, including one in the target bullseye, but his arrows are very far apart.  Jane is a model of quantitative research, where great precision can be delivered, but it might not be on target.  Bob is the model of qualitative research, where significant analysis of the whole yields truth, but with imprecision.  In my mind, quantitative analysis is focused on precision, while the focus of qualitative analysis is accuracy, even if the answers are not precise.  Both can deliver substantive results and provide for the truth, but qualitative seems far more suitable in areas where there are insufficient measurements available.

I must admit that I am far more comfortable with the idea of quantitative data... but experience has taught me that it is no more or less authoritative than qualitative, since both are at the mercy of measurement error, sampling error and reporting bias, which create bad data.  I think the curse of qualitative research is that the numbers make the answers seem more precise, but understanding how the research was conducted is often more important than the numbers.  I stopped believing Gallup polls when I learned 20 years ago that their most significant polling audience were college students 18-24 (as easiest to get to participate in a poll), and that they had discarded several areas of the country as too conservative, including Ohio. By targeting specific demographics, they had tainted their inputs, and ruined their ability to forecast the opinion of the whole due to their sampling error.  Gallup may have fixed their sampling process since then, but it ruined their credibility with me.

At a more basic level, understanding the sequence of questions in a survey, and the specific phrases used to introduce and explain the survey is important to understanding what bias may have been introduced to the survey experience... just one more area where quantitative research can gather information that has had bias injected at the sampling source.

In Information Security, we often struggle with a dearth of quantitative data, and there has been great lament in the community over that.  We've been left for 30+ years with expert opinion and standards of good practice, which have to necessarily make gross assumptions about what "good" is, since all industries place a different value on differing qualities of desirable outcome.  However, that doesn't mean that there aren't good answers, just that it takes more research and expert knowledge to discern the truth.

Consider that the area of cybercrime goes largely unreported, and that even acts of embezzlement are substantially under reported (typically 10%), even though withholding knowledge of the crime harms society (Tragedy of the Commons). Due to lack of quantitative data (and the Holy Grail, actuarial tables), we're not able to state authoritatively what can stop cybercrime.  I have faced suppression of reporting cybercrime occurring in my organization many times in my career, and it's always a bitter pill. When the Russian mob or Chinese hackers take down databases and grab consumer financial information, society (and those consumers) should know.  Yet, it often is unreported, or whitewashed.

The most repugnant, and one that caused me nightmares, was when I had caught a pedophile with GIGS of child porn images, It was the most abhorrent thing I've seen professionally, a true face of evil. He was fired, his pr0n files erased, and it went unreported, and that was hard to live with, because I fear what may have happened to a child as a result of this predator-in-training walking free.  That would now be a crime, to not report, but was not at the time. I lost a lot of sleep over that.  Anyway... Just one example of why there are gaps in cybercrime stats.

Fortunately, my profession is starting to get to predictive methods, and use actual quantitative analysis using sampling, Monte-Carlo simulations, game theory simulations, and Bayesian stats, and the tide is starting to turn.  However, most of our public measurements about cybercrime are based on qualitative methods (e.g. survey) that purport to provide hard numbers (quantitative measures), so we have a ways to go before we are able to speak with authority and not have our methods (rightly) challenged by criminologists.

That is what is comforting about the Donald Knuth quote I started this posting with -- because both art and science have a place in our field, and I think one of the markers of a risk professional is when you are able to be comfortable with both approaches.  Neither quantitative nor qualitative are inherently inferior, as both provide pathways to arrive at the truth, but the art comes in knowing when to pick which approach.