You are reading your third 40+ page report of the morning. Your eyes roll over the neatly formatted lines of Times New Roman 12pt font, the single spacing lending an almost hypnotic cadence to your reading. As your mind opens itself to accepting thoughts of shopping lists future and weekends past, your head begins to drift into its usual after lunch power nap position … KAZZING! without warning, you are awakened by an intense adrenaline surge and there it is.
“The name’s Percent. Eighty Nine Percent,” croons the Sultry Statistic.
Like all heroes, the Sultry Statistic is powerful, yet often misunderstood. In the passenger experience space, statistics are often derived from data collected through surveys or questionnaires. This is both a popular and cost effective methodology to attain a “statistically significant” result. It can however, inadvertently introduce ambiguity, or shades of grey, into the reported results.
To illustrate the point, let’s examine our hero a little more closely. Let’s say that Eighty Nine Percent claims that by 2015 “89% of passengers will want mobile flight updates”. This of course, at first glance, conveys a very strong message. However, if we look a little deeper we discover that this statistic may imply a range of unintended meanings. Consider the following scenarios under which the underlying data may have been collected:
- Passengers are surveyed: “Would you like mobile flight updates?” [Yes | No]
- Passengers are questioned: “How would you like to receive flight updates?” [empty textbox]
- Passengers are interviewed: “How was your passenger experience today?”
Given scenario one above, it is likely that most passengers would respond “Yes”. The reality is that phrasing the question in this manner makes the data collection process almost meaningless. More importantly, it sends an implicit (and possibly incorrect) message that “mobile flight updates” are really important to the passenger experience.
Looking at scenario two, we can see that removing the suggestion of a specific response from the question is likely to generate a wider range of responses. This in turn would most likely result in a smaller percentage figure for the “mobile flight updates” category. Naturally, this data collection method would lead to the inference that “mobile flight updates” are less important to the passenger experience (when compared with scenario one).
The third scenario removes any leading suggestion of what factors may be important to the passenger’s experience. This could result in a very wide range of responses, which may or may not specifically relate to “mobile flight updates”. Once again, the data collected in this manner could result in a different inference regarding which factors are important to the passenger experience.
The above example reminds us that survey and questionnaire data collection methodologies can only report on the questions that are explicitly asked. By their very nature, surveys and questionnaires assume a “closed world” perspective, i.e. what is not asked must be false. From this observation, we can generalise that these data collection tools are best suited to confirmatory, rather than exploratory research.
In the context of research aimed at discovering factors which influence passenger experience, the research should, ideally begin with an exploratory phase. During this initial phase the aim would be to limit the “open world” of possible factors to a finite, “closed world” set (scenario three). On the basis of the finite set of factors, a validating questionnaire (scenario two) could be issued to provide assurance that the closed world set contained all the key responses. It is at this point that the administration of a survey (scenario one) would lend statistic significance to the reported results while reducing the likelihood of ambiguity.
Related: Interesting paper regarding importance of wording and interview skills. A topical example of misundertsanding resulting from mis-use of statistics.