The Twitter prediction business

In July 2009 actor Sacha Baron Cohen’s comedy film Bruno made an impressive one-day debut of $US14.4 million at US and Canadian box offices. But the following day it dropped precipitously, falling 39 per cent to $US8.8 million. The audience had reacted quickly and badly to what they saw of the film, media reports surmised, and had voted with their tweets.

Even before then, many big companies were working hard to take account of the “Twitter effect.” Last year a pair of researchers from Hewlett Packard’s Lab in Palo Alto used a computer algorithm to crunch the positive or negative sentiments expressed in 2.9 million Twitter messages about 24 movies. The result, they claimed, perfectly predicted the box-office performance of each film, with an accuracy of over 97 per cent in the opening weekend – they’re now in the process of patenting it. Quantifying sentiment in this way, according to its boosters, isn’t only useful for its insights into our mood – it can also help us understand the direction in which things are headed. In October of last year, a team of researchers at Indiana University classified 9.7 million Twitter posts under six mood categories (happiness, kindness, alertness, sureness, vitality and calmness), and reckoned that by doing so they could predict changes in the Dow Jones Industrial Average.

The upshot of all this is a new kind of gold rush among companies like Google, Facebook, Twitter and Cisco to exploit our data and make use of it. To help them do so, they’ve hiring data analysts by the dozen. These new data-crunchers have manufactured a whole pseudo-scientific vocabulary for what they do – they talk about tracking the “social index”, and refer to themselves rather grandly, as “sentiment analysts” or practitioners of “info-demiology.” Whether they’re trying to work us out on the basis of what we’re typing into search engines, who we know or what we like, their rise tells us something significant. As the initial utopian impulse which grew up around all things web-related gives way to a new realism about what social networking can achieve, the balance of power on the web is slowly shifting to the number-crunchers and data analysts who have the resources to exploit it. For social networking systems like Facebook and Twitter making more use of the data we’re throwing their way is even more pressing. We’re playing on their turf, after all, and sooner or later they’re going to need to pay the rent.

But it’s not only companies who’ve something to gain from all this. In the last few years social scientists and pollsters have begun looking afresh at the kind of “self-reporting” which happens on blogs and social networks for what it can tell us about ourselves and which way we’re headed. For social researchers, what’s so wonderful about all this data is its freshness and its immediacy. Not only that, it short-circuits the whole awkward process of asking us loaded questions. The business of working out who we are and what we want is still young – the data we’re leaving behind online means that it’s about to grow up.

This article will appear in Wired UK’s June edition. James Harkin is Director of Flockwatching. His book Niche: Why the market no longer favours the mainstream is published by Little, Brown.

Leave a Reply

Your email address will not be published.