The Cynic’s Guide to Travel Industry Surveys

The travel industry remains awash in bad data, ill-conceived and poorly executed research projects. Some of the biggest and best known travel and research companies are the worst culprits, with the ever-present survey as the typical platform of choice.

Here is a quick outline that will help you identify bogus research when you see it. These points may sound extreme, but they represent only part of the standards serious researchers might impose. They are really quite simple to understand and appreciate.

The implication here, in case you missed it, is that most travel industry research is not serious. As you consider what the latest survey quotes are teaching you, the appropriate conclusion may well be “nothing whatever.”

 1)    Context

Serious researchers do not cite random figures without explaining where they came from, why they are relevant, and where you can read them in their original setting.

When you hear comments such as “travel costs were 3% lower when using this method,” the lack of context most likely means that they’d prefer you not look at the source material or they can’t really cite effective references.

2)    Sample

Effective surveys involve carefully identifying what population is to participate, how these individuals will be reached, and then taking the time to contact and engage them. Researchers should explain how they obtained their sample, how large it was, and how the sample size relates to the total population that might have been surveyed.

Surveys that allow participants to self-select might tell us something–but it’s almost impossible to discern what that might be. Another word for this is “useless.”

The sample must also be designed to answer the question as accurately as possible, not produce the answer the survey designer might like to see.

3)    Questions

Unless you see the question that was asked, the response isn’t especially instructive. Questions must be clear and unbiased (most are neither).

The phrasing of the question must also allow people who don’t have an opinion, or who think the question is silly (a distinct possibility much of the time) to avoid answering or select a neutral answer (one which is interpreted as neither positive or negative).

Good surveys shouldn’t force an answer where the individual has none–to do so imposes a bias on the result. Ideally, you should be able to review all the questions that comprised a survey.

4)    Tabulation and Method

Skipped questions are not “responses” to be interpreted however one desires. Researchers should describe the methods they employed to compile the data and interpret the conclusions.

The researcher should also be able to describe how accurate the survey is believed to be, based upon the size and composition of the sample and its relationship to what the survey is trying to describe.

Thus, a survey of a company’s travelers can be said to represent their views of all participate, or if the sample is sufficiently large so as to reflect the unbiased views of the majority. A survey of ten travelers from a population of 1,000 is probably not that useful.

An interpretation of survey results which claims that “travel costs were 3% lower when using this method” is only useful if we know the question asked, who answered it in a way that supports the 3% claim, and why the responders are representative of anything in particular.

Researchers who don’t describe their methods probably don’t have one that they would like you to examine.

5)    Biases

We should know who paid for the survey, who created the questions (an independent researcher or the sponsor) and what they survey was designed to determine. The answers may not have influenced the result, but potential biases should be disclosed.

For further information please see my comments on travel industry research available here.

 

Open Travel and Industry Mythology

This analysis appeared August 12, 2013 in Cornerstone Information System’s “Insight & Opinion” section.


You may have heard of something called “open travel” or “open booking,” that is about to change corporate travel procurement. It says that travelers will book whatever they want as long as they don’t exceed budgets and fulfill other vague management requirements, such as paying with a corporate credit card.

Travel Management Companies will have to find new roles (no one is quite sure what those might be) and corporate travel managers will see their responsibilities changed, or substantially diminished.

Proponents of this new travel management strategy, which include some substantial travel buyers and data managers, assure us that travelers are buying where the want anyway, so clearly it’s best to make the best of the inevitable.

What’s interesting is that this isn’t new.

Prior to the mid 1980s, when consolidating travel procurement with a few designated travel management companies became popular, it was the default purchasing system–set your budget and let the rest take care of itself.

I’m confident that proponents of “open travel” believe that technology has advanced so much over the last 30 years that the essential management problems that resulted in consolidated travel management are no longer issues.

If we simply wait long enough, smart phones will solve everything–they’ll even change human nature.

Can Less Be More?

There are few analytical reports describing open travel’s effectiveness–those that do exist are contradictory and most are poorly executed(i). As a business strategy, open travel advances several logical fallacies that we should try to avoid:

Hasty Generalization

It doesn’t necessarily follow that because some, even most, travel management programs perform poorly and that they are at odds with new technology, that all such programs must do the same.

It also doesn’t follow that travelers in general will make informed and rational purchasing decisions absent a centrally managed travel program because some travelers appear to do so, for some of their trips, at least some of the time.

Open travel’s proponents need to conclusively demonstrate that the business rules they suggest the industry adopt are not based upon the behavior of a small sample operating under exceptional conditions.

Faulty Dilemma

Because there are clear shortcomings with travel management practice, we are not necessarily left with a “strength through weakness” strategy that allows travelers to book whatever they want as the alternative. There are other choices.

The Big Picture

Centralized travel management exists because it is effective. Vendors extend favorable pricing and other services to purchasers because they believe the benefits exceed the cost.

Part of what vendors presume is that travel managers will influence selection and behavior. Travel programs that consistently deliver such results are those that succeed.

Why would vendors offer similar benefits where buyers stop trying to do these things Control is an essential component of preferred pricing, and open travel is signal for higher prices, not lower.

The mythology of the travel industry asserts that technology allows individuals to find as good or better discounts in the marketplace as are available through managed travel programs.

Again, that’s nothing new, the assertion has been made for as long as there have been centralized travel programs. In practice there are always exceptional situations, but consistently poor discounts are signs of a poor travel programs, not testimonials for open travel.

Technology makes traveler shopping somewhat easier, but it doesn’t make informed buyers or change human behavior. Again with exceptions, travelers do not usually share the management goals of their companies–part of a travel manager’s role is to provide structure for those goals.

A traveler’s agenda is more personal and can always be validated by countless rationalizations and “this time is different” conclusions. There is no lack of creativity in this area.

To expect individual and company goals to align so as to correctly and consistently influence individual traveler behavior is to assume that people will stop behaving like people because they have better smart phone applications and are free to use them.

Open travel is less efficient than centralized travel management, not more. It assumes pricing practices that don’t today exist, people acting in ways they don’t normally do, and, even if these point are granted, that it is the best use of a traveler’s time to research prices and keep sufficiently informed so as to make good decisions.

It’s a theory that assumes much and delivers little.
(i) Unless a study describes a sound methodology, an adequate sample size, and the precise questions it tried to answer, which is almost never the case, it falls into this category. Please see my paper on this topic.

Improving Travel Industry Research

Once again a new year brings another round of what passes for industry research. Although notoriously over-surveyed, the travel industry remains awash in bad data, ill-conceived and poorly executed research projects, and self-serving studies that are relevant more to the next round of funding or the next newsletter sale than to developing a real understanding of markets and trends.

Eventually the industry may get better at labeling useless research for what it is (the trend is not positive, however), but for now the very few good studies routinely drown amidst the hyperbolae of research that can’t connect with real insight–or those that connect all too well because the result was fairly evident before the process began.

Nowhere is the problem more acute than in the online travel and social media worlds. High-priced research typically reinforces conventional wisdom and assumptions while key customer and behavior questions remain unresolved.

I’ve wondered aloud in past articles why major trade groups show such slight interest in these issues. If the online and social media worlds have such monumental consequences, what precisely could be more important to their members?

Here are a few suggestions for modeling forthcoming research projects. These are similar to suggestions I’ve made in public for 15 years, and hopefully they will help you appreciate the limitations of today’s travel research and be positioned to improve it in the future.

    1)    Broaden the Base

Successful studies need wide participation and sponsorship. Those funded and controlled by a single company or clique are not necessarily bad, but this adds complexities and concerns that are avoidable through planning and execution that strives to include more viewpoints. Addressing the needs of a broad constituency increases both value and integrity.

    2)    Sampling Is Key

Few research projects undertaken in the travel industry describe how the study sample was selected, what the resulting accuracy and margin for error are, or the size of the sample.  This is because these are among the most challenging aspects of valid research–requiring time, expertise, and money to address properly.

Most researchers simply ignore them; the resulting studies are little better than worthless.

If you peek under the covers only slightly at a surprising number of major industry studies, you’ll discover that the sample essentially self-selects. The researchers won’t explain how their conclusions in this environment are valid because they aren’t–and they can’t.

There are many parts of a study that has sufficient statistical validity to become the basis for real-world conclusions and predictions, but one is usually that a valid sample must be defined and identified in advance of the research and then the study must continue until it reaches the sample as defined.  More work than most researchers want.

    3)    Questions, Questions

Any question-based research should disclose the questions used and how these are presented.  Forming valid questions is a significant undertaking–which is frequently botched.

During 2010 I was treated to a trade conference  where the expert presented study results to show the importance of the field where he was the market leader and that was the subject of his presentation.  The self-selected sample were asked a variety of simplistic questions with many obvious answers:

“Is cost-control important in your business?”

Have you ever met anyone who would answer “no?”

When the presenter reached some study questions that were clearly silly, he remarked,

“Well, my staff assembled these questions and I should have reviewed them better.”

In other words, the presentation is a waste of everyone’s time–which also says something about the extent to which some conference organizers vet their presentations.

    4)    Seek Wide Input

Limiting control over a study to its sponsors or other “insiders” cannot but color the result as self-serving. Enlightened researchers learned along ago that the “best and brightest” often don’t work for them and they seek such talent out wherever they can find it.

    5)    Dump Hyperbolae; Focus On Quality

The industry doesn’t need another round or praise describing how great the opportunity of the day may be. What’s needed is thorough research and careful answers that relate to business concerns and allow the reader to reliably take action.

This simple definition disqualifies most of the fluff-laden e-commerce and social media studies of the past few years. There are people who know how to do real research–it’s a mystery when their input is so clearly lacking in the major reports of today.

    6)    Analyze

As there are competent researchers there are also competent interpreters who can make connections between abstract numbers and real business situations. Their work ought to by key to any research project. A study lacking informed, usable conclusions should be first into the worthless bucket.

    7)    Validate

There’s a saying that teaches thus:

“Premises that are absurd when projected into the future were absurd to begin with.”

Researchers and readers alike need to apply logical tests in order to understand the validity of a study’s conclusions.

For instance, most predictions about the fantastic growth of mobile, social media-based, or other new media travel purchases assume a level of personal computer use and literacy throughout society that is simply absurd within the time frames considered.

Clearly studies are failures when they cannot withstand the test of reasonableness.

    8)    Forget “Guru” Mentality

Research is ongoing and the “final word” on most topics will likely never be written. Successful research projects are willingly subject to critical review and are revised in light of new viewpoint and data.

A premise holding that the oracle has now spoken and nothing further may be added only highlights the underling weakness of the research in question.