A recent post at Dan Ariely’s Predictably Irrational Blog about Microsoft’s Mojave Experiment reminds us of a cynical moment during our research for Billion-Dollar Lessons when we concluded that, too often, marketing is when companies lie to their customers, and market research is when companies lie to themselves.

Microsoft’s experiment purports to show that “Vista is great, it is just that people are prejudiced against it.” Microsoft makes its point by bringing together 140 avowed Vista haters who had actually never used Vista. They just concluded from word-of-mouth and from reading reviews that Microsoft’s latest operating system was slow, buggy, etc. When shown a demo of “Mojave,” which they were told was a new Microsoft operating system but which was really Vista, the survey participants were enthusiastic. Thus, Microsoft’s conclusion that the problem with Vista is one of customer perception rather than the actual capabilities of the product itself.

Dan, who knows a lot about experiment design, points out that Microsoft’s experiment was fatally flawed. He acknowledges that getting people to experience Vista without preconceptions is a good idea. The problem is that the experiment provided little of the actual Vista experience, like installation and actual use without the benefit of a canned demo or an expert guide. This, Dan concludes, “transforms the experiment into a PR move and not a true experiment on the capability of Vista.”

Microsoft’s approach is not unlike the market research done by Iridium, the satellite venture that cost Motorola and its partners $5 billion to build. Iridium went bankrupt less than a year after the service was launched, and its assets were sold for $25 million. In a key piece of market research, Arthur D. Little began a survey with this statement: “There will soon be a new personal telephone service which at a reasonable cost will provide you with the capability to be reached or to place calls anywhere in the world using satellite technology, which is not limited in coverage like a cellular phone. To access the service you would have a small handset that fits in your pocket…” When asked whether they’d like such a device, respondents were quite positive, and Motorola enthusiastically reported the survey results at an investor conference. The problem with the survey question, as phrased, was that it was more marketing than market research. The responses validated the appeal of the concept but did little to validate the actual offering. For that to happen, the respondents would have had to understand that “reasonable cost” meant $3,000 per handset plus $3 per minute plus monthly fees, that “anywhere” meant anywhere with a line-of-sight view of rapidly orbiting satellites, and that “fits in your pocket” meant you needed a pocket that would hold a brick.

Motorola and other Iridium investors went on to lose billions because they believed that their market research was valid. We hope Microsoft will not confuse its marketing campaign with true market research.