Once again a new year brings another round of what passes for industry research. Although notoriously over-surveyed, the travel industry remains awash in bad data, ill-conceived and poorly executed research projects, and self-serving studies that are relevant more to the next round of funding or the next newsletter sale than to developing a real understanding of markets and trends.
Eventually the industry may get better at labeling useless research for what it is (the trend is not positive, however), but for now the very few good studies routinely drown amidst the hyperbolae of research that can’t connect with real insight–or those that connect all too well because the result was fairly evident before the process began.
Nowhere is the problem more acute than in the online travel and social media worlds. High-priced research typically reinforces conventional wisdom and assumptions while key customer and behavior questions remain unresolved.
I’ve wondered aloud in past articles why major trade groups show such slight interest in these issues. If the online and social media worlds have such monumental consequences, what precisely could be more important to their members?
Here are a few suggestions for modeling forthcoming research projects. These are similar to suggestions I’ve made in public for 15 years, and hopefully they will help you appreciate the limitations of today’s travel research and be positioned to improve it in the future.
Successful studies need wide participation and sponsorship. Those funded and controlled by a single company or clique are not necessarily bad, but this adds complexities and concerns that are avoidable through planning and execution that strives to include more viewpoints. Addressing the needs of a broad constituency increases both value and integrity.
2) Sampling Is Key
Few research projects undertaken in the travel industry describe how the study sample was selected, what the resulting accuracy and margin for error are, or the size of the sample. This is because these are among the most challenging aspects of valid research–requiring time, expertise, and money to address properly.
Most researchers simply ignore them; the resulting studies are little better than worthless.
If you peek under the covers only slightly at a surprising number of major industry studies, you’ll discover that the sample essentially self-selects. The researchers won’t explain how their conclusions in this environment are valid because they aren’t–and they can’t.
There are many parts of a study that has sufficient statistical validity to become the basis for real-world conclusions and predictions, but one is usually that a valid sample must be defined and identified in advance of the research and then the study must continue until it reaches the sample as defined. More work than most researchers want.
3) Questions, Questions
Any question-based research should disclose the questions used and how these are presented. Forming valid questions is a significant undertaking–which is frequently botched.
During 2010 I was treated to a trade conference where the expert presented study results to show the importance of the field where he was the market leader and that was the subject of his presentation. The self-selected sample were asked a variety of simplistic questions with many obvious answers:
“Is cost-control important in your business?”
Have you ever met anyone who would answer “no?”
When the presenter reached some study questions that were clearly silly, he remarked,
“Well, my staff assembled these questions and I should have reviewed them better.”
In other words, the presentation is a waste of everyone’s time–which also says something about the extent to which some conference organizers vet their presentations.
4) Seek Wide Input
Limiting control over a study to its sponsors or other “insiders” cannot but color the result as self-serving. Enlightened researchers learned along ago that the “best and brightest” often don’t work for them and they seek such talent out wherever they can find it.
5) Dump Hyperbolae; Focus On Quality
The industry doesn’t need another round or praise describing how great the opportunity of the day may be. What’s needed is thorough research and careful answers that relate to business concerns and allow the reader to reliably take action.
This simple definition disqualifies most of the fluff-laden e-commerce and social media studies of the past few years. There are people who know how to do real research–it’s a mystery when their input is so clearly lacking in the major reports of today.
As there are competent researchers there are also competent interpreters who can make connections between abstract numbers and real business situations. Their work ought to by key to any research project. A study lacking informed, usable conclusions should be first into the worthless bucket.
There’s a saying that teaches thus:
“Premises that are absurd when projected into the future were absurd to begin with.”
Researchers and readers alike need to apply logical tests in order to understand the validity of a study’s conclusions.
For instance, most predictions about the fantastic growth of mobile, social media-based, or other new media travel purchases assume a level of personal computer use and literacy throughout society that is simply absurd within the time frames considered.
Clearly studies are failures when they cannot withstand the test of reasonableness.
8) Forget “Guru” Mentality
Research is ongoing and the “final word” on most topics will likely never be written. Successful research projects are willingly subject to critical review and are revised in light of new viewpoint and data.
A premise holding that the oracle has now spoken and nothing further may be added only highlights the underling weakness of the research in question.