He uses statistics as a drunken man uses lamp-posts -- for support rather than illumination -- Andrew Lang (1844-1912)
One is hard pressed to go a day in the IT world without reading an article, white paper or seeing a PowerPoint without a metric, poll, statistic or some sort of number attempting to make a point.
But have you ever considered, in the spirit of Juvenal's "quis custodiet ipsos custodies" -- who guards the guards -- where that data really came from?
Data from Gartner, Forrester, J.D. Power are often accepted as the final word, used to set corporate security strategies and direction. For example, when Forrester claims their results are based on a survey of 500 CISOs, how did they ensure the respondents are legitimate CISOs? Just how did they get that data? These are just a few of the types of validation questions that should be asked before taking action on the data.
The reason to deal with their results with some hesitation is that Forrester, like many other firms, uses paid surveys. While it is uncertain what percentage of their respondents are paid survey based, the fact that they are used, may taint their results.
A paid survey is a type of statistical survey where the participant is rewarded through an incentive program, generally entry into a sweepstakes program or a small cash reward, for completing the survey. Traditional surveys are usually unpaid, as with Gallup or Quinnipiac University polls.
Paid survey companies such as Opinion Miles Club and e-Rewards are two of many firms that have large databases of eager participants. What they offer are rewards for those who complete surveys; be in the form or airline miles, gifts cards, and the like. Research, marking and analyst firms use them as they are a huge pool of those willing to answer surveys.
It seems like a legitimate barter; you give them the information they want, they give you a reward. While there are problems with that model; this is not the place to detail them. But two of the most significant issues are that when there are incentives, it may compromise the data. More importantly, the incentive is substantially low for the data required.
What this means is data for significant technology questions are being answered at a minimum wage rate, when the qualifier to answer the question requires being a senior technologist.
Let me give you an example from a recent survey from e-Rewards.
Every survey has a qualification section. Before one can participate in the survey, the polling firm needs to ensure that the respondent is qualified. In this one, the first qualifier was the respondent's job title. The selection criteria had 12 selections, ranging from senior-most business leader (owner, president, C-level executive), VP in IT, to student, retired, or not currently employed.
Sign up for CIO Asia eNewsletters.