The Cynic’s Guide to Travel Industry Surveys

The travel industry remains awash in bad data, ill-conceived and poorly executed research projects. Some of the biggest and best known travel and research companies are the worst culprits, with the ever-present survey as the typical platform of choice.

Here is a quick outline that will help you identify bogus research when you see it. These points may sound extreme, but they represent only part of the standards serious researchers might impose. They are really quite simple to understand and appreciate.

The implication here, in case you missed it, is that most travel industry research is not serious. As you consider what the latest survey quotes are teaching you, the appropriate conclusion may well be “nothing whatever.”

 1)    Context

Serious researchers do not cite random figures without explaining where they came from, why they are relevant, and where you can read them in their original setting.

When you hear comments such as “travel costs were 3% lower when using this method,” the lack of context most likely means that they’d prefer you not look at the source material or they can’t really cite effective references.

2)    Sample

Effective surveys involve carefully identifying what population is to participate, how these individuals will be reached, and then taking the time to contact and engage them. Researchers should explain how they obtained their sample, how large it was, and how the sample size relates to the total population that might have been surveyed.

Surveys that allow participants to self-select might tell us something–but it’s almost impossible to discern what that might be. Another word for this is “useless.”

The sample must also be designed to answer the question as accurately as possible, not produce the answer the survey designer might like to see.

3)    Questions

Unless you see the question that was asked, the response isn’t especially instructive. Questions must be clear and unbiased (most are neither).

The phrasing of the question must also allow people who don’t have an opinion, or who think the question is silly (a distinct possibility much of the time) to avoid answering or select a neutral answer (one which is interpreted as neither positive or negative).

Good surveys shouldn’t force an answer where the individual has none–to do so imposes a bias on the result. Ideally, you should be able to review all the questions that comprised a survey.

4)    Tabulation and Method

Skipped questions are not “responses” to be interpreted however one desires. Researchers should describe the methods they employed to compile the data and interpret the conclusions.

The researcher should also be able to describe how accurate the survey is believed to be, based upon the size and composition of the sample and its relationship to what the survey is trying to describe.

Thus, a survey of a company’s travelers can be said to represent their views of all participate, or if the sample is sufficiently large so as to reflect the unbiased views of the majority. A survey of ten travelers from a population of 1,000 is probably not that useful.

An interpretation of survey results which claims that “travel costs were 3% lower when using this method” is only useful if we know the question asked, who answered it in a way that supports the 3% claim, and why the responders are representative of anything in particular.

Researchers who don’t describe their methods probably don’t have one that they would like you to examine.

5)    Biases

We should know who paid for the survey, who created the questions (an independent researcher or the sponsor) and what they survey was designed to determine. The answers may not have influenced the result, but potential biases should be disclosed.

For further information please see my comments on travel industry research available here.