July 29, 2008

“The Mojave Experiment:” Bad Science, Bad Marketing

I guess I should first admit I hate the show Punk’d. I mean, here’s a guy who is famous for lying about his age so he seems hipper, telling us that his show’s purpose it to deflate the big egos on other stars, and show them what truly matters in life. So he sets up situations where anyone would get upset, and then laughs when he upsets people. I call *cough*bullshit*cough*. (Also *cough*jerkface*cough*.)

So I have to admit I’m not predisposed to like The Mojave Experiment, where Microsoft took a bunch of “regular folks” XP users who were afraid of Vista, and told them Microsoft was going to show them a secret new operating system — which was actually Vista.

UNSURPRISINGLY, these people mostly said they liked Vista.

Now, if you read this blog, you know I pretty much hate Microsoft, because of their incredibly shady business practices (moreso in the early 1990s) and their shoddy products, most especially their operating systems, whose crappy user experience and programmer interfaces hold back the advance of technology. However, I’m not going to rail on Vista here. Seriously, I’m not.

What I am going to rail on is this “experiment.” (I use that word advisedly.)

--

I hate bad science. Hate it. Hate. So let’s look at not one, not two, but FOUR, yes FOUR (ah-ah-ah!) key flaws in this experiment, any single one of which would render its results meaningless:

The Placebo Effect: Every time I do a software release, no matter how minor, even if I just change one word, in French, to another French word, someone will send me mail or post on a forum, “Thanks, this release seems a lot faster!” Do I make fun of them? Or videotape them and put it on a blog? No. Because it’s just human nature. If we are told something is new-and-improved, we prime ourselves to believe it (c.f. Blink by Malcolm Gladwell, which I’ll refer to again in a bit) and make it so in our minds.

This is why we have, for example, blind taste tests: because humans are proven to not be able make dispassionate judgments about subjects they already know about. So, if you say to someone, “Hey, I’m giving you a top-secret peek at a new operating system from Microsoft, you’re incredibly lucky and special, and I really value your opinion!” of COURSE they are going to like it. They almost can’t not like it.

The Pepsi Challenge Effect: “The Pepsi Challenge” was a blind taste test that Pepsi overwhelmingly won (again, from Blink). Yet, most people still drink Coke. Why? Gladwell’s thesis is that a single sip of a soft drink is very different from drinking a whole can, which is the smallest unit most people imbibe. Pepsi usually wins the challenge because it's a sweeter drink, and initially people respond to this extra sweetness. But after drinking a can, Pepsi becomes cloying.

So, here I am, sat down in front of Mojave-err-Vista, and all I've ever used is XP. Well, look, nobody is doubting the graphics are prettier in Vista. It looks nice compare to XP (it should — they hired the guy who designed Aqua for Mac OS X).

I play with Mojave, and, yes, some system tasks are easier. Again, nobody doubts there are things that work much better. When I plug my iSight camera into Vista it shows up as a device and offers to let me take pictures in the Vista Explorer thingy. That’s kind of cool! Hey, I kind of like Mojave-nee-Vista!

Except, those glossy features aren’t why people downgrade from Vista to XP. Those are not the reason people hate on Vista!

Now, again, look — I don’t use Vista or XP for anything but games. I liked using Vista better, until the new UFO (X-Com) game that I had played great on XP, and wouldn’t launch at all on Vista. Then I bailed. That’s my story. There are apparently hundreds of others.

You, personally, may never have encountered a piece of hardware or an app that didn’t work on Vista, and you might be perfectly happy with it. I’m not going to try to argue you out of that happiness. My point is that the problems that Vista has become famous for are not the kinds of problems you encounter in a few minutes of playing with it in a controlled environment.

Vista is known for people initially liking it, then after a while discovering it’s not working for them, and “downgrading” to XP. This study has told us exactly what we already knew: that, initially, people like Vista. (Initially, people like having sex without condoms, too... it’s simply not a very good criterion all by itself.)

The Perfectly Controlled Environment Effect: Microsoft set up the hardware. Microsoft brought the accessories. Microsoft picked the software. Microsoft sat people down with Vista experts driving the mouse, and walked people through Vista. What an INCREDIBLE SHOCKER that in this INCREDIBLY TIGHTLY CONTROLLED ENVIRONMENT Vista performed OK!

Microsoft had set up an environment with a philosophy similar to Apple’s: “Look, we work well with this hardware and software, and too bad if you want something different.” Unfortunately, that’s NOT why people choose Windows. They hack together their own machines, and they want their software to still run.

Did any of these customers bring in their favorite games and try to play them? Did they bring in their graphics tablets and discover they fail?

Did any of the test machines ever say, “Oh, I’m sorry, Windows Genuine Advantage has determined that you may be running an invalid copy of Windows, so please jump through these hoops or we’ll disable some of your hardware”? I’m going to guess no. But I’ve seen this message a lot. And I own three valid licenses to Windows.

The Personal Tutor Effect: If you sit anyone down with an expert in a particular program, and the expert walks them through the features and answers their every question, chances are good that person is going to report that she had a good experience with the program. Very good, indeed.

Personal training is so important to customer experience that Apple thinks of it as a key asset of its Apple Stores. But Microsoft doesn’t have Apple Stores in real life. Or any analog. It’s you and a box with a holographic sticker on it. Good luck!

--

Microsoft has managed to prove that if you have a friendly expert on a controlled machine (with Vista pre-installed) showing a carefully selected subset of Vista features to an ignorant XP user for a few minutes, the XP user will often say he finds Vista acceptable. Wow.

This so-called experiment of Microsoft’s is an insult to science, and to our intelligence. And I am dying to see the out-takes from their shoot. I mean, how many people do you suppose like being told, “Hey, this giant, unpopular monopolistic software company just made an ass out of you! Ha ha! Our leading scienticians just PROVED that you LOVE VISTA and WANT TO MARRY IT. You are TOTALLY GAY for Vista! Haaaaaaa HAAAAAAA!”

Vista may or may not be an upgrade in user experience for most Windows customers. I personally prefer the feel of Vista over XP when the former works as well as the latter, but Vista has failed me on several occasions, and I also don’t enjoy running games MORE slowly than XP.

I've got to imagine that the Microsoft customers who took all the damn time to upgrade their machines to Vista, determined it was unworkable, and then had to take all the time to go BACK to XP, probably did so for a reason, possibly even a valid reason, and not because they had been swayed by bad word-of-mouth. I further imagine that these customers are completely livid at having Microsoft not say, “Oh, sorry, we’ll get right on those bugs,” but, instead, “You’re just stupidly following the crowd, and if you’d just free your mind up, you’ll discover you actually love Vista... hater.”

Is “Our Customers Are Stupid and Have No Idea What They Really Want” really Microsoft’s new mantra?

Again, wow.

Labels: , ,