Focus groups, polling and the “American Idol Delusion”

“I’m watching you!”

When we pre-tested our first game, “Flashback,” the high-schoolers we polled universally responded that they would NOT play an American history game.  It was the academic course many indicated they hated most, and that playing “a game” revolving around history sounded as attractive as drinking a brussel sprout milkshake.  Yet within days of launching Flashback, the site went wild and within 10 days had 200,000 page views.  In post-game interviews, 97% of the kids who said they would never play our game said they would definitely play it again.

What does this tell us?  About the same thing that election pollsters already know but rarely tell anyone otherwise they’d be out of a job:  polling is unreliable.  In a recent New York Magazine piece, Gavin Polone wrote about a similar mistrust he has of focus groups, the small test audiences (generally 48 men and women) that rate TV pilots.  Mr. Polone observes, “The idea that such a small sample can represent the whole market for a show does seem ridiculous and bound to deliver anomalous results.”  Some of those results are notorious.  Focus groups gave extremely low ratings to “Seinfeld” and “Friends.”

So what is wrong with polls and focus groups, and even more puzzling, why do we continue to use them?

Without doubt, where polling goes off the rails is that they rely on personal opinion — and opinion that is derived in a questionable context.  We call it, the American Idol Delusion.  You know what I’m talking about:  in the early episodes of any season of American Idol, a judge asks a contestant, “So do you think you’re the next American Idol?”  And the contestant brashly exclaims, “Absolutely, I’m a phenomenal singer.”  And then when that singer opens his or her mouth, the worst screeching and out-of-tune caterwauling gushes out.  Randy pales and moans, “No no no, dawg — that’s just horrible!  Can’t you hear yourself?!”

No, they can’t.  Hardly any of us can hear or see ourselves objectively — or anyone else, for that matter.  Prejudice and self-delusion color our opinions.  So why do we continue to use polls and focus groups?  Because they generate quantitative results.  “54% are fearful about the economy…  7% are undecided.”  It all sounds so scientific.  But what are they measuring?  Opinions.  Employing “tools” (i.e. polls and focus groups) that do no more than measure opinions in order to make decisions can result in epic disasters.  (48 men and women, somewhere, LOVED “Homeboys in Outer Space” — or don’t you remember that series?)

So what is worth measuring?  Actions.  Yup.  You know, those things that happen in the real world and speak louder than words.  You may think you’re the next Whitney Houston, but if when you open your mouth you are Roseanne Barr yowling the National Anthem — well, it really doesn’t matter what you think.

That’s why when we build games that also need to serve as assessment tools, we measure the data generated by actions.  Not opinions.

I also mentioned context as important to evaluating the action.  With the Nuvana platform, most of the actions generated by players happens in a social environment.  That is, you do something, and you publish it to the community.  Something very powerful and magical happens within that context:  authenticity.  People who fill out polling questionnaires in private have very different, and generally less honest answers, than when they answer in a community environment.  Let’s face it:  it’s much harder to b.s. peers in public.  Everyone is watching and listening.  If it’s a good community, they will call you out — but hopefully in a supportive way.  That’s what Nuvana strives for:  authentic actions and self-correction by the involvement of the community.

We see this happen all the time with Nuvana games.  The ethics of a social network based on authentic actions always trend toward the moral high ground.  It’s not totally surprising.  There was an interesting phenomenon discovered by Melissa Bateson, a behavioral biologist at the University of Newcastle-upon-Tyne who was in charge of her psych department’s coffee fund.  She found that by putting up a picture of a pair of eyes, donations to the coffee jar went up threefold.

Hmm.  Maybe we should change our name to NooVana.  With the double O’s in the shape of eyes.

Seriously, we have no interest in playing the role of Big Brother for the gamified universe — but we are seeking to drive authentic behaviors for ethical communities.  The actions that result can be transformed into verifiable data with which one can build trustworthy assessments.

What does that mean in real world terms?  Well, back to Mr. Polone who finishes his New York magazine piece by writing:  “Really, this [focus group] testing ritual seems pretty outdated in the Internet age. Why stick with such a small sample size when we have the technology to quickly get the opinions of a much bigger audience? A network could put all of its pilots up on its website and let the entire populace of interested viewers vote for their favorites and leave comments on how they would like to see the show improved before it went to series.”

Sounds like the perfect premise for a Nuvana game, doesn’t it?

— JvK

Tagged , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: