Lying with Numbers

I've gotten feedback on "why quantitative measures are better", and how statistics provide a firm footing for higher-order discussions in our field.  Bullocks.

"There are three kinds of lies: lies, damned lies, and statistics." 
-Attributed by Mark Twain to Benjamin Disraeli, but actual source unknown

Gentle reader, please don't get sucked into the mythology of numbers.  I once was working with a veneer cutting laser system that was supposed to yield a 5-10% improvement in optimization of raw materials (which, if you've priced exotic hardwood veneer, you'll know that 1/4" of the really high-end stuff is about $.25, so gaining 1/4" on 5000 cuts a day really adds up).  After three days, the customer was yelling that our equipment was flawed and not calibrated correctly, because their waste scrap had gone UP not down.  

It turns out that the measurements were precise, but humans were gaming the system.  Through observation, we found substantial staff resistance to change had thwarted our efforts - they would take a precise measurement using the laser measuring systems, and then round up 1/4" to 3/8", just as they always had when using a measuring tape.  However, the human system meant that a 1/4" of rounding was actually barely sufficient sometimes, and was on a bell curve where the initial measurement was fractionally short.  

Why it had gotten worse was because, with precise measurements by lasers, the bell curve was flattened into a tight plateau.  The entire rounding padding added by the cutting staff was scrap, since the laser cutter was never "too short".  Precision of laser measurement had been confused with accuracy, and a sampling bias had created a substantially deviant wrong answer with high precision.  Once we got the staff to only round up 1/100", the customer got all the savings they had forecast, and the scrap pile was nothing but sawdust. (In the Hollywood version, I probably would have gotten a mahogany and teak inlaid armoire in appreciation.)

Yes, numbers can DEFINITELY lie, because the context gets twisted.

You have to look VERY carefully at what the number is reporting.  So, if you'll indulge me, let me tell the truth, while providing misleading statistics that would enable me to lie.

Example: You're picking participants for a research project, and you want to include a person with little experience in their field, and a person with a lot of experience.

Given the following, who do you choose?

1. Amanda's full-time experience spans 3 decades.
2. Bob has worked as a security guard for the Archdiocese for 32 years.
3. Chris logged 300,000 miles this year as a trucker.
4. Doug was Police Chief in Columbus for 15 years.
5. Emily served as a judge for 27 years.

In order, who had the most experience, from least to most?

------

you know how this works.

I have to add space.

If I don't, your eyeballs will immediately jump to the answer.

------

So, examine the statistics provided carefully, then see if you agree...

1. Amanda's full-time experience spans 3 decades.
2. Bob has worked as a security guard for the Archdiocese for 32 years.
3. Chris logged 300,000 miles this year as a trucker.
4. Doug was Police Chief in Columbus for 15 years.
5. Emily served as a judge for 27 years.

In order, who had the most experience, from least to most?

Answer:

Chris, Emily, Bob, Amanda, Doug.  (3, 5, 2, 1, 4)

How?

1. Chris is a trucker, hired this morning, and helped organize the records for the company today, documenting 300,000 miles of travels for the company.  It took her an hour to log those trips.
2. Emily has been judge of the church bake competition for 27 years.  This takes her an hour a year at the annual church picnic.  She has 27 hours experience, 27x more than Chris.
3. Bob works as security guard for the annual Archdiocese Casino Day fund-raiser, watching the cash box.  He has 32 days experience spread across 32 years, more than 25x more experience than Emily.
4. Amanda has 12 years experience, from 1999-2011, even though, on first reading, you might think she has 30+ years.  Her experience spans 3 decades, the '90s, '00s, and '10s.
5. Doug has 15 years experience as Police Chief, working 42 hours a week, for 15 years full-time experience.

I point this out, because I've seen these kinds of statistical lies in my career.  :-)

When Qualitative Data is MORE Accurate: Death by Data

Aaron pinged me with follow-up questions, telling me I'd been sniffing glue if I thought that quantitative data was less reliable than qualitative.  Gifted and respected colleague, I don't think you can ever say that one method is inherently superior to the other -- much like you can't say that Linux is more/less secure than Windows.  Both can be very highly secured, and both can be as secure as a wet paper sack.

I think that there can be cases where qualitative data is actually more reliable than quantitative data.  Quantitative research can create the illusion of reliable fact, because it has numbers, which makes is seem concrete and absolute.  However, due to bias and errors, quantitative data can be wildly inaccurate.  

As one example, the Space Shuttle Challenger loss (which occurred 27 years ago today, 27-January) was due to catastrophic failure of O-ring seals, and was precipitated by erosion of the O-rings on prior flights. The erosion of the seals by hot gases should never have occurred, and was an indicator of a failed test, and a failure mode of the O-ring.  However, because the engineers presumed that eroding completely through the O-ring was required in order for a failure to occur, and the erosion had only eaten through 1/3 of the O-ring after the timed test, they presumed that the duration and erosion were correlated.  The engineers made an unsupported leap of faith on very few data points that appeared to fit a linear progression (since humans are nothing if not pattern-matching machines, so they found a pattern where none existed, like a Rorschach ink blot).  From this erroneous extrapolation, they created an "erosion model" for the margin of safety.  After an additional test caused the data point for erosion to fall near the curve, they believed the bogus model was predictive, and treated failure modes as an actual Margin of Error.  This was quoted in a section of the Rogers Commission Report on the Space Shuttle Challenger by R.P. Feynman (http://www.ralentz.com/old/space/feynman-report.html) as having been a contributing factor to the disaster.  It was qualitative, fit a curve, and seemed to have significance (because they backed into the curve from the numbers), so seemed reliable.  However, it was really a chart of failure modes, not of safety, and was uncorrelated.

In this instance, the quantitative research was bogus, but believed because measurements and the fit to forecasted curve gave the illusion of science.

Quantitative v. Qualitative

"Science is what we understand well enough to explain to a computer.  Art is everything else we do."

-Donald Knuth

Quantitative research is conducted using empirical information that can be sensed and measured.  The intent for quantitative analysis is to provide for a means of analysis via mathematics, statistical, or computational analysis.  Quantitative research is often, but not exclusively, based on establishing a hypothesis, and conducting experiments provides for a testable hypothesis based on the ability to measure.  By its nature, quantitative research provides for (presumably) precise measurements within a sample of the whole problem space, and uses inference to make assumptions about the state of the whole.  Quantitative analysis gives up on being able to measure the complete state of the whole, so that precise measurements can be made.

Qualitative research is comparatively new as a means of conducting research yielding truth.  Through the use of qualitative research methods, researchers are able to create new understandings of problems that can be quite useful.  It seems that qualitative research tries to get a sense of the whole picture, painting a landscape of the entire problem space, then seeking to fill in the missing gaps to develop a working hypothesis.  Qualitative analysis abandons the notion of precise measurements, preferring instead to gain a sense of the complete state of the whole problem space.

To provide a gross generalization, I think of two people shooting arrows at a target, Jane and Bob, who both score 50 points in an archery competition.  Jane has the finest archery equipment, but poor eyesight.  Bob has the eyes of an eagle, but poor equipment.  When the two shoot, Jane gets consistently tight patterns, grouping 10 arrows in a few square inches, but her shot misses the bullseye entirely, and she scores 10 scores of 5.  Bob's poor equipment means his arrows aren't going precisely where he is aiming, but his aim is true.  Bob has arrows across the target, including one in the target bullseye, but his arrows are very far apart.  Jane is a model of quantitative research, where great precision can be delivered, but it might not be on target.  Bob is the model of qualitative research, where significant analysis of the whole yields truth, but with imprecision.  In my mind, quantitative analysis is focused on precision, while the focus of qualitative analysis is accuracy, even if the answers are not precise.  Both can deliver substantive results and provide for the truth, but qualitative seems far more suitable in areas where there are insufficient measurements available.

I must admit that I am far more comfortable with the idea of quantitative data... but experience has taught me that it is no more or less authoritative than qualitative, since both are at the mercy of measurement error, sampling error and reporting bias, which create bad data.  I think the curse of qualitative research is that the numbers make the answers seem more precise, but understanding how the research was conducted is often more important than the numbers.  I stopped believing Gallup polls when I learned 20 years ago that their most significant polling audience were college students 18-24 (as easiest to get to participate in a poll), and that they had discarded several areas of the country as too conservative, including Ohio. By targeting specific demographics, they had tainted their inputs, and ruined their ability to forecast the opinion of the whole due to their sampling error.  Gallup may have fixed their sampling process since then, but it ruined their credibility with me.

At a more basic level, understanding the sequence of questions in a survey, and the specific phrases used to introduce and explain the survey is important to understanding what bias may have been introduced to the survey experience... just one more area where quantitative research can gather information that has had bias injected at the sampling source.

In Information Security, we often struggle with a dearth of quantitative data, and there has been great lament in the community over that.  We've been left for 30+ years with expert opinion and standards of good practice, which have to necessarily make gross assumptions about what "good" is, since all industries place a different value on differing qualities of desirable outcome.  However, that doesn't mean that there aren't good answers, just that it takes more research and expert knowledge to discern the truth.

Consider that the area of cybercrime goes largely unreported, and that even acts of embezzlement are substantially under reported (typically 10%), even though withholding knowledge of the crime harms society (Tragedy of the Commons). Due to lack of quantitative data (and the Holy Grail, actuarial tables), we're not able to state authoritatively what can stop cybercrime.  I have faced suppression of reporting cybercrime occurring in my organization many times in my career, and it's always a bitter pill. When the Russian mob or Chinese hackers take down databases and grab consumer financial information, society (and those consumers) should know.  Yet, it often is unreported, or whitewashed.

The most repugnant, and one that caused me nightmares, was when I had caught a pedophile with GIGS of child porn images, It was the most abhorrent thing I've seen professionally, a true face of evil. He was fired, his pr0n files erased, and it went unreported, and that was hard to live with, because I fear what may have happened to a child as a result of this predator-in-training walking free.  That would now be a crime, to not report, but was not at the time. I lost a lot of sleep over that.  Anyway... Just one example of why there are gaps in cybercrime stats.

Fortunately, my profession is starting to get to predictive methods, and use actual quantitative analysis using sampling, Monte-Carlo simulations, game theory simulations, and Bayesian stats, and the tide is starting to turn.  However, most of our public measurements about cybercrime are based on qualitative methods (e.g. survey) that purport to provide hard numbers (quantitative measures), so we have a ways to go before we are able to speak with authority and not have our methods (rightly) challenged by criminologists.

That is what is comforting about the Donald Knuth quote I started this posting with -- because both art and science have a place in our field, and I think one of the markers of a risk professional is when you are able to be comfortable with both approaches.  Neither quantitative nor qualitative are inherently inferior, as both provide pathways to arrive at the truth, but the art comes in knowing when to pick which approach.