What Would A Proof of Open Booking Look Like?

One can spend considerable time and energy developing specific and detailed reviews of flawed theories and business proposals. Although appreciative comments and note are often the result, it’s important to recognize that this isn’t how business strategies should be developed.

The burden of proof rests with the proponents of new strategies and theories to use adequate and appropriate reasoning, and specific proofs to show that these are valid–not with everyone else to show why they are not.

Open Booking, as presently announced, is an ill-conceived business strategy that rests upon faulty logic, inadequate data, poor research, and a suspension of belief in how the real world operates. Its proponents have, or should have, certain obligations to correct these errors and show how Open Booking then remains a valid business premise.

Absent a much improved business case, the travel community is justified in rejecting Open Booking’s imaginary benefits without further argument. The disinclination of others to spend time refuting unsupported theories gives them no credence whatever.

Here’s a concise outline of how Open Booking must be proven. It’s not unrealistic and doesn’t assume more that a correct application of available evidence.

I’ve also attached a “conclusion” as to how likely we are to ever see any of these points addressed.

1)    Assume the Burden of Proof

Advocates of any theory, business or otherwise, carry the burden of proof which requires them to adequate demonstrate why their ideas are valid.

Statements about Open Booking such as “travelers are booking directly with suppliers and often times spending less money than if they go through a managed program,” or “there have been studies that have validated this but unfortunately, the status quo has swept them under the rug” are irrelevant commentary and establish nothing.
When you make such claims, it’s your job to substantiate them. Produce your evidence or abandon your claims.

Conclusion: The Burden of Proof is troublesome and inconvenient. It’s much more fun to make random, unsupported claims and suggest that people who disbelieve you should know better. The logical basis for Open Booking is so shaky that were unlikely to see a rush to defend it with more that opinion and speculation.

2)    Clearly and Comprehensively State the Proposition

Any business theory requires a detailed, comprehensive statement of what it is and how its proponents expect it to work. For Open Booking, that means more than the vague statements about how everyone is doing it so the result is therefore inevitable. The business proposition needs to be positively defended in order to be valid. Simply because flaws can be identified in the travel management process does nothing to advance the cause of Open Booking–there are other equally choices available.

The case for Open Booking also needs to consider business operations in the real world in detail and discuss how Open Booking affects each of them. A few short PowerPoint presentations do little to advance this discussion.

Conclusion: Building a correct business case is a lot of work, and Open Booking is a moving target that seems to evolve along a new line as soon as someone points out its shortcomings. It’s unlikely that anyone will expend the effort to improve this picture.

3)    Define Specific, Unambiguous Proofs That Your Assertions Are Correct

Once you’ve explained the business case for Open Booking, show us the clear proof-points that demonstrate the theory is valid and worth the effort. Not travelers are booking directly with suppliers and often times spending less money than if they go through a managed program” but how much, how often, under what conditions, and to what degree does this have to be so to offset costs and business risks?

Conclusion: If Open Booking could be substantiated in this way, someone would have tried to do so by now. The fact that proofs and evidence are abandoned in favor of opinion and anecdote is itself a demonstration of Open Booking’s failure.

4)    Use Objective, Comprehensive, Accurate, and Scientifically Correct Data

Forget self-selected surveys, tiny samples, biased questions, and the general lack of controls that infests almost all travel industry research. Produce data that can be defended, use it to establish your proofs, and then your Open Booking business proposition might have some validity.

Conclusion: Almost all travel research is useless and contrived to establish the preconceptions its authors want to perpetuate. This is unlikely to change anytime soon. As best (and this is conceding a great deal) the data in support of Open Booking are ambiguous.

Open Booking’s proofs and research should be straightforward and, if correct, should silence critics when accompanies by a comprehensive business proposition. It’s time this evidence is forthcoming.

5)    Comprehensively Describe How You Did Your Research

What precisely was your sampling methodology? How are your conclusions sustained by the raw data? What is an alternate interpretation of the data and how do you answer that interpretation? What would researchers have to do to replicate your research? Who sponsored your research and what are their and your predispositions?

Conclusion: Real research is transparent, fully explained and disclosed, and replicable. Spurious research sustains one-time conclusions or hides behind a proprietary cloak. This type of transparency and disclosure is very rare in the travel industry and non-existent as concerns Open Booking.

6)    State What You Cannot Yet Prove and How This Affects Your Conclusions

Scientific research acknowledges its shortcomings and identifies what cannot yet be proven as well as what can. It also admits areas where future evidence might disprove the theory. The quality of your interpretation of the evidence in support of your claims is as important as what that evidence specifically shows.

Open Booking lacks a real statement of its comprehensive business case, real proof-points that are offered to establish its validity, scientific evidence sufficient to establish the vague claims made in behalf of it, and a rational analysis of its very real deficiencies.

Conclusion: Open Booking’s proponents are no more likely to improve their process or develop their evidence in this area than they are in any other. Remember, if you are an advocate of Open Booking, you have the responsibility to develop and present your adequate evidence before anyone is obliged to give your ideas credence.

It’s not up to me or anyone else to disprove Open Booking–the burden rests with you. The six areas discussed here should be a minimum expectation.

The Cynic’s Guide to Travel Industry Surveys

The travel industry remains awash in bad data, ill-conceived and poorly executed research projects. Some of the biggest and best known travel and research companies are the worst culprits, with the ever-present survey as the typical platform of choice.

Here is a quick outline that will help you identify bogus research when you see it. These points may sound extreme, but they represent only part of the standards serious researchers might impose. They are really quite simple to understand and appreciate.

The implication here, in case you missed it, is that most travel industry research is not serious. As you consider what the latest survey quotes are teaching you, the appropriate conclusion may well be “nothing whatever.”

 1)    Context

Serious researchers do not cite random figures without explaining where they came from, why they are relevant, and where you can read them in their original setting.

When you hear comments such as “travel costs were 3% lower when using this method,” the lack of context most likely means that they’d prefer you not look at the source material or they can’t really cite effective references.

2)    Sample

Effective surveys involve carefully identifying what population is to participate, how these individuals will be reached, and then taking the time to contact and engage them. Researchers should explain how they obtained their sample, how large it was, and how the sample size relates to the total population that might have been surveyed.

Surveys that allow participants to self-select might tell us something–but it’s almost impossible to discern what that might be. Another word for this is “useless.”

The sample must also be designed to answer the question as accurately as possible, not produce the answer the survey designer might like to see.

3)    Questions

Unless you see the question that was asked, the response isn’t especially instructive. Questions must be clear and unbiased (most are neither).

The phrasing of the question must also allow people who don’t have an opinion, or who think the question is silly (a distinct possibility much of the time) to avoid answering or select a neutral answer (one which is interpreted as neither positive or negative).

Good surveys shouldn’t force an answer where the individual has none–to do so imposes a bias on the result. Ideally, you should be able to review all the questions that comprised a survey.

4)    Tabulation and Method

Skipped questions are not “responses” to be interpreted however one desires. Researchers should describe the methods they employed to compile the data and interpret the conclusions.

The researcher should also be able to describe how accurate the survey is believed to be, based upon the size and composition of the sample and its relationship to what the survey is trying to describe.

Thus, a survey of a company’s travelers can be said to represent their views of all participate, or if the sample is sufficiently large so as to reflect the unbiased views of the majority. A survey of ten travelers from a population of 1,000 is probably not that useful.

An interpretation of survey results which claims that “travel costs were 3% lower when using this method” is only useful if we know the question asked, who answered it in a way that supports the 3% claim, and why the responders are representative of anything in particular.

Researchers who don’t describe their methods probably don’t have one that they would like you to examine.

5)    Biases

We should know who paid for the survey, who created the questions (an independent researcher or the sponsor) and what they survey was designed to determine. The answers may not have influenced the result, but potential biases should be disclosed.

For further information please see my comments on travel industry research available here.

 

Improving Travel Industry Research

Once again a new year brings another round of what passes for industry research. Although notoriously over-surveyed, the travel industry remains awash in bad data, ill-conceived and poorly executed research projects, and self-serving studies that are relevant more to the next round of funding or the next newsletter sale than to developing a real understanding of markets and trends.

Eventually the industry may get better at labeling useless research for what it is (the trend is not positive, however), but for now the very few good studies routinely drown amidst the hyperbolae of research that can’t connect with real insight–or those that connect all too well because the result was fairly evident before the process began.

Nowhere is the problem more acute than in the online travel and social media worlds. High-priced research typically reinforces conventional wisdom and assumptions while key customer and behavior questions remain unresolved.

I’ve wondered aloud in past articles why major trade groups show such slight interest in these issues. If the online and social media worlds have such monumental consequences, what precisely could be more important to their members?

Here are a few suggestions for modeling forthcoming research projects. These are similar to suggestions I’ve made in public for 15 years, and hopefully they will help you appreciate the limitations of today’s travel research and be positioned to improve it in the future.

    1)    Broaden the Base

Successful studies need wide participation and sponsorship. Those funded and controlled by a single company or clique are not necessarily bad, but this adds complexities and concerns that are avoidable through planning and execution that strives to include more viewpoints. Addressing the needs of a broad constituency increases both value and integrity.

    2)    Sampling Is Key

Few research projects undertaken in the travel industry describe how the study sample was selected, what the resulting accuracy and margin for error are, or the size of the sample.  This is because these are among the most challenging aspects of valid research–requiring time, expertise, and money to address properly.

Most researchers simply ignore them; the resulting studies are little better than worthless.

If you peek under the covers only slightly at a surprising number of major industry studies, you’ll discover that the sample essentially self-selects. The researchers won’t explain how their conclusions in this environment are valid because they aren’t–and they can’t.

There are many parts of a study that has sufficient statistical validity to become the basis for real-world conclusions and predictions, but one is usually that a valid sample must be defined and identified in advance of the research and then the study must continue until it reaches the sample as defined.  More work than most researchers want.

    3)    Questions, Questions

Any question-based research should disclose the questions used and how these are presented.  Forming valid questions is a significant undertaking–which is frequently botched.

During 2010 I was treated to a trade conference  where the expert presented study results to show the importance of the field where he was the market leader and that was the subject of his presentation.  The self-selected sample were asked a variety of simplistic questions with many obvious answers:

“Is cost-control important in your business?”

Have you ever met anyone who would answer “no?”

When the presenter reached some study questions that were clearly silly, he remarked,

“Well, my staff assembled these questions and I should have reviewed them better.”

In other words, the presentation is a waste of everyone’s time–which also says something about the extent to which some conference organizers vet their presentations.

    4)    Seek Wide Input

Limiting control over a study to its sponsors or other “insiders” cannot but color the result as self-serving. Enlightened researchers learned along ago that the “best and brightest” often don’t work for them and they seek such talent out wherever they can find it.

    5)    Dump Hyperbolae; Focus On Quality

The industry doesn’t need another round or praise describing how great the opportunity of the day may be. What’s needed is thorough research and careful answers that relate to business concerns and allow the reader to reliably take action.

This simple definition disqualifies most of the fluff-laden e-commerce and social media studies of the past few years. There are people who know how to do real research–it’s a mystery when their input is so clearly lacking in the major reports of today.

    6)    Analyze

As there are competent researchers there are also competent interpreters who can make connections between abstract numbers and real business situations. Their work ought to by key to any research project. A study lacking informed, usable conclusions should be first into the worthless bucket.

    7)    Validate

There’s a saying that teaches thus:

“Premises that are absurd when projected into the future were absurd to begin with.”

Researchers and readers alike need to apply logical tests in order to understand the validity of a study’s conclusions.

For instance, most predictions about the fantastic growth of mobile, social media-based, or other new media travel purchases assume a level of personal computer use and literacy throughout society that is simply absurd within the time frames considered.

Clearly studies are failures when they cannot withstand the test of reasonableness.

    8)    Forget “Guru” Mentality

Research is ongoing and the “final word” on most topics will likely never be written. Successful research projects are willingly subject to critical review and are revised in light of new viewpoint and data.

A premise holding that the oracle has now spoken and nothing further may be added only highlights the underling weakness of the research in question.