The observations of Murdoch Junior were revealing; if he was looking for a little pity from parliamentarians, he might have done better to repeat it this time around. It might prove a fitting epitaph for his career, as well as for the power and mystique of the mainstream media. Despite all the bluster about the influence of the Murdoch press, the truth is that they were only ever as powerful as they were allowed to be – and, long before the phone-hacking scandal, that power was on the wane. Rupert Murdoch bought the Times and the Sunday Times in the early 1980s, just as traditional, class-based politics was beginning to crumble. The upshot was that both Labour and the Conservative parties were looking for new ways to reach out to voters, and for a long time courting Murdoch and the rest of the mainstream media seemed the easiest way to do it. “I wish they’d leave me alone,” Murdoch said last time around, when interrogated by the committee members about his meetings with prime ministers. It was his best line of the day.

Murdoch’s News Corporation is a strange hybrid of family shop and publicly owned company. All the same, its changing fortunes exemplify the problems which have beset the broadsheet media during its twilight years. Newspapers used to enjoy a collective monopoly over the kind of news and information they served up – it was newspaper journalists who did most of the original work polishing up facts and breaking new stories, setting the agenda for TV and radio to follow. Around the same time that Murdoch was buying his British broadsheets, many other family–fun newspapers sold up to large publicly owned conglomerates. To recoup the vast sums of money they had spent and squeeze out a little extra for their shareholders, these groups began to fire journalists and buy in more of their news from industry agencies. The cost cutting was most ferocious in the United States, particularly among the large regional papers and general interest magazines which had the firmest grip on their audiences. At the same time, newspapers kept expanding their remit, gobbling up whole new areas of interest and muscling into the public conversation about everything from reality TV to the Royal Family to hold the attention of their audience. Throughout the 1990s, most newspapers put on a huge amount of weight, but much of the extra bulk consisted of little more than sections that were advertising vehicles.

Straining themselves to maintain their audience, general interest newspapers have in recent years diluted their authority and their reputation for quality and authority. In 2009, a study published by Nick Davies, the Guardian journalist whose dogged determination kept news of the hacking scandal alive, found that of the five most prestigious general interest papers in Britain – the Times, the Guardian, the Independent, the Daily Telegraph and the Daily Mail – 60 per cent of their output consisted wholly or mainly of news service stories or press releases. A further 20 per cent of the stories were slightly adapted from the same. Put simply, newspapers were plumping up their product with filler; with generic news instead of original reporting. The result was that the news they fed us began to look pretty much the same. One of the ironies of the News of the World’s closure is that it was one of the newspapers which was both profitable and produced original investigative journalism, even if we didn’t like what it was looking at.

General interest newspapers thrived when they were able to assemble a captive audience and exert a hold over it. But in the age of Google, Facebook and YouTube, with so much free information to choose from, the spell is wearing off. The internet has allowed us to become what the philosopher Daniel Dennett and the cognitive scientist Steven Pinker call “informavores” – creatures who exhibit an almost unlimited appetite for the information we find there. We graze around vast virtual universes at our leisure, ruthless information predators slicing through the undergrowth as we seek to locate exactly what we’re looking for. With an almost unlimited menu at our fingertips, many of us have become highly skilled at using search engines to narrow our field of attention and get exactly where we want to go as well as to use social media to share good stuff around.

In most developed countries, newspaper circulation and the advertising which went along with it are in long term retreat. Twenty out of 30 OECD countries are witnessing a decline in the circulation of their newspapers, according to a study published in 2010 by the OECD itself. Sales of American papers had plummeted 30 per cent in the previous three years, it discovered, and those of British papers had dropped 25 per cent in the same period.

Rupert Murdoch’s British papers now find themselves not big enough to create their own ecosystem and not focused enough to thrive inside an existing one. James Murdoch is well aware of all this – which is why he’s always been more interested in television and new media than in the newspapers that were bequeathed to him. So, in a way, is David Cameron. Conventional wisdom has it that his two most senior advisors, Andy Coulson and Steve Hilton, were the yin and yang of the Downing Street machine. Coulson was the modern-day Mondeo Man, employed to mouth the concerns of News International and its red-top, red-blooded readership. Hilton bellowed feel-good mantras about the big society next door. Not any more. As News International’s crisis deepened, it threw Cameron’s friend Coulson to the wolves.
Meanwhile, Hilton was married to Google communications executive Rachel Whetstone, and, for a time, accompanied her to live near Google’s California headquarters before returning to London in July 2009. He drunk so deep from the well of Bay Area internet mythology that he’s now gone back there for a “sabbatical.” His internal strategy bulletins, regularly leaked by disgruntled government ministers who don’t understand what he’s talking about, are full of buzzwords like “transparency” and “the post-bureaucratic age”.

In a speech a few years ago on the proposed “Silicon Roundabout” in Shoreditch, east London, David Cameron declared that “the founders of Google have said they could never had started their company in Britain” because of its restrictive copyright system. There followed the announcement of a review of Britain’s intellectual property law to identify barriers to innovation and help out new internet companies, to the delight of Google and the consternation of much of the traditional media.

Governments need to support innovation and investment, and Google has been infinitely more inventive with new technology than the stale suits at News Corp. But, by helping us to zoom straight to our prey and hang out with our friends online, new media giants end up with a minute-by-minute measure of what we like and what we’re after – and they can track our trail. This kind of power is not lost on politicians.

During his run for the presidency Barack Obama’s campaign team spent $5m advertising with Google, much of it to buy up sponsored links associated with popular search keywords. Anyone who typed “Barack Muslim” into Google was directed to a page telling them that Obama wasn’t a Muslim. Anyone who typed in “diabetes” was directed to a website that informed them that they might not be covered for the illness under John McCain’s healthcare plans. In the run-up to the last general election, the Tories did likewise. When Google users searched for Gordon Brown, for example, they were confronted with a sponsored link to a tory party webpage lambasting ‘Charlie Whelan’s new militant tendency.’

Even as we leave our computers behind to move around our towns and cities, the mobile phones in our pockets are using positioning technology to search for their location. With our permission, applications like Google’s Latitude and Foursquare allow us to see the whereabouts of our friends on a map as we make our way through towns and cities, and scribble electronic graffiti on anything we encounter along the way.

In retrospect, the messages routinely left behind by friends on each other’s mobiles look like a fuzzy, late 20th century prototype of the mass of information we’re now leaving behind on the internet and social media. The difference is that we’re handing it over ourselves, further aggrandizing new media at the expense of the old. It’s reassuring to know that the thoroughly modern strategists in Downing Street understand this, and are keenly aware which way the wind is blowing. Unlike poor old Rupert, these new media boys and girls don’t even have to go in through the back door.

James Harkin is Director of social research agency Flockwatching. His book Niche: The Missing Middle, and why business needs to specialize to survive (Little, Brown) will be published as a paperback in August.

Chowdhury’s decision to post the data was a brave one, but it didn’t reckon on the feeding frenzy which often accompanies the low-key release of sensitive information onto the net. When his huge file was chanced upon and investigated by a passing internet user, it was discovered to contain no less than twenty three million search keywords for 650, 000 AOL users over three months earlier that year. Within hours, it had been pilfered by nimble internet users and pasted up all over the more anarchic corners of the net. But in the brouhaha over privacy and data protection which followed, during which Abdur Chowdhury was loudly fired and his research unit closed down, it was too easy to forget that all of them had voluntarily queued up to type all this material into an gigantic database, one which was capable as a result of compiling a comprehensive ledger of each of their internet-mediated thoughts just as soon they wrote them. Within days, whole websites had sprung up dedicated to understanding what that data meant. One of the most popular is called AOL Stalker. The goal of AOL Stalker, as its name suggests, is to make AOL’s vast data bank of information entirely searchable by those who wanted to browse through the searches of AOL’s users. Its founder is a reserved and slightly suspicious 26 year-old Swedish hacker called Hjalmar and, when I tracked the man him down to Sweden and spoke to him on Skype, he forwarded me a popular sequence of searches for one AOL customer, whose data had been downloaded more than fifty thousand times. The anonymous internet user, identified in AOL’s data only as user 672368, seems to be a young woman, and the three months of chronological searches that we have for her tells us a great deal about her life and her mental state. At the beginning of March 2006, she appears to be in the early stages of pregnancy:

2006-03-01 18:54:10 Body fat calliper

2006-03-05 08:53:23 Curb morning sickness

2006-03-09 18:49:37 Get fit while pregnant

Two days later, her mood seems to darken.

2006-03-11 03:52:01. He doesn’t want the baby

2006-03-11 03:52:58. You’re pregnant he doesn’t want the baby

Soon things are back to normal, and her enthusiasm for having the baby returns.

2006-03-14 19:11:28. Baby names and meanings

2006-03-28 09:28:25. Maternity clothes

2006-03-29 10:01:39. Pregnancy workout videos

2006-03-29 10:12:38. Buns of steel video

It’s not long, however, before user 672368 seems to be having second thoughts.

2006-04-17 11:00:02 Abortion clinics charlotte nc

2006-04-17 11:40:22 Greater Carolinas Womens Center

2006-04-17 21:14:19 Can Christians be forgiven for abortion

2006-04-17 22:22:07 Roe vs. Wade

2006-04-18 06:50:34 Effects of abortion on fibroids

2006-04-18 15:14:03 Abortion clinic charlotte

2006-04-18 16:14:07 Symptoms of miscarriage

A few days later, she is thinking about engagement rings.

2006-04-20 16:58:37 Engagement rings

On the same day, however, abortion is still weighing on user 672368′s mind.

2006-04-20 17:53:49 High-risk abortions

Two days later, the decision seems to have been made on her behalf.

2006-05-22 18:17:53 Recover after miscarriage

Only several days later, her thoughts turn once again to marriage.

2006-05-06 21:22:18 www.weddingchannel.org

2006-05-26 19:32:52 Demetrios bridesmaid dresses

2006-05-27 07:25:45 Marry your live-in.

What Hjalmar had nudged me in the direction of seems to be the inchoate story of someone’s life – the affecting and real-life story of one young woman’s pregnancy, her subsequent wrestle with the fact that her baby might be not be wanted by her partner, and her eventually miscarriage. As the sequence of searches ends, user 672368 appears to be recovering from that miscarriage, and looking forward to marriage with her partner. All of this, of course, can only be inferred. No one really knows what is going on inside our heads just from the keywords we type, because the internet is not a complete approximation of our thought-processes or what we’re up to. All the same, there’s something awesomely fascinating about what all this electronic chatter can tell us about ourselves On its own it doesn’t make much sense. Join up the dots, however, and it can leave us with a frighteningly immediate route into our collective psyche. The hodgepodge of thoughts, desires and impulses which tumble out when we sit in front of our internet search boxes tends to short-circuit our rational selves, and makes for a uniquely powerful way of tapping into our collective mood which goes right under the radar of more traditional measures.

It’s not only subterranean hackers who are trying to make sense of it. In the last few years social scientists and market researchers have begun looking afresh at the kind of “self-reporting” which happens on blogs and social networks for what it can tell us about ourselves and which way we’re head. The upshot is a new kind of gold rush among companies like Google and Cisco who are clever or rich enough to exploit our data and make some use of it. Many of the same authorities and institutions that we thought we had left behind when we migrated online eco-systems like Facebook and Google have quietly become peeping toms there, and spend a great deal of time monitoring the traffic and thinking about how they can crunch it to their own advantage. To help them do so, they’ve hiring data analysts by the dozen. These new data-cruncher have even invented a whole vocabulary – they talk about tracking the “social index” or “social graph”, and refer to themselves rather grandly, as “sentiment analysts” or practitioners of the science of “info-demiology.” All the same, their rise tells us something significant. What it means is that, as the initial utopian impulse which grew up around all things web-related gives way to a new realism about what social networking can achieve, the balance of power on the web is slowly shifting to the number-crunchers and data analysts who have the resources to exploit it. For social networking systems like Facebook and Twitter the imperative to make more use of the data we’re throwing their way is even more pressing. We’re playing on their turf, after all, and sooner or later they’re going to need to pay the rent.

*

About a decade ago AOL entered into a long-term agreement to have its searches by Google, which means those searches made public by Abdur Chowdhury were immediately forwarded to Google’s search box as soon as they were asked. Since three out of five internet searches anywhere in the world are answered by the company, it’s no surprise that Google has always been keenly aware of the uses to which its data might be put. In 2008 I sat at a table in the company’s plush London offices and was shown how to use one of the new tools to emerge from its laboratory, Google Trends which uses our search words to track our enthusiasm for different subjects over time. The executive who’d invited me in demonstrated his brand new gizmo by using it to compare the numbers of people who searched for Barack Obama versus those who searched for Hilary Clinton during their race to be the Democratic Party’s nomination for the 2008 American Presidential Election. Looking at the peak and troughs of support for each and comparing public interest in the two over time, it was very easy to chart very accurately the rise in public fascination with Obama and the waning of interest in Hilary Clinton. Just as Chowdhury had predicted, what’s so wonderful about search data is its freshness and its immediacy. Given the symbiotic relationship between our fragmented thought-processes and the words we end up typing into search engines, what comes out looks like an eerily instantaneous chronicle of the public mood. To Google’s managers, all this has the potential to be seriously useful. In 2008, the same year the company invited me into its London offices, it launched Google Flu Trends, an ambitious attempt to use its searches to predict epidemics of flu in advance of medical authorities. Other big companies are experimenting with ways to use this kind of data to track public health problems or even short-term economic trends.

Mining for search data gold, however, isn’t as easy as it sounds. For one thing, it’s difficult to know what we’re measuring here. Are our searches for Michael Jackson, for example, an indication of his popularity or his lingering infamy? There are lots of different reasons why we might be searching, and no easy way to find out. If this kind of data is any use, it’s only at the aggregate level, where analysts to trawl through it to decipher patterns in the ether. Even then, it’s far from foolproof. Take Google Flu Trends. Just as Abdur Chowdhury predicted, the great thing about Google Flu Trends is that its search data can be collected and analysed instantaneously, whereas traditional flu surveillance systems can take days or even weeks to process. But that doesn’t mean it’s more accurate. According to a major study published by the University of Washington last year, in fact, it’s substantially less accurate than traditional systems the medical authorities. The problem for Google Flu Trends is that lots of different viruses can give us flu-like illnesses, but that doesn’t mean we’ve got flu. The stuff we’re tying into the net, if it can tell us anything, is usually better at telling us what we’re thinking is happening than what’s really happening. And, as we know from the lightning pace at which information finds its way around the net, this kind of “info-demiology” lends itself very well to “viral” bouts of social hysteria.

Search data, of course, isn’t the only tool that data analysts have at their disposal. When millions of us migrated to spend our lives on online social networks like Facebook, a more promising avenue for researchers opened up – to work out who we are on the basis of who we know. 2007 was the year in which Mark Zuckerberg began to present Facebook as an all-knowing “social graph” which would open our eyes to the ties which bind people to each other without anyone ever knowing. In that same year, a few MIT students decided to take him at his word. Wondering what we were unconsciously telling others about ourselves by ‘friending’ people there, the pair used data downloaded from Facebook’s MIT network and a piece of software to try to predict the sexuality of some of their fellow students who didn’t report their sexual preferences on the basis of who they knew. Though there was no scientific way of proving their results, their knowledge of the students they predicted to be gay proved that they were right in ever case they checked. Simply by looking at who people know on Facebook, in other words, they were able to predict whether that person was gay.

This idea of working out who people are on the basis of the company they keep is called social network analysis, and not at all new. To sociologists its known as the “homophily principle” – the tendency of similar kinds of people to hang together. It was a kind of social network analysis, for example, which led the 4th American Infantry Division to Saddam Hussein’s hole in the ground in December 2003. Other recent studies have tried to use it to predict everything from who might be a terrorist to the likelihood that we’ll end up happy or fat. Online social networks, however, have added high-tech fuel to the theory. It’s easy to see how it might work. If your online friends are all Muslim or Jewish it’s likely that you will be too. It might even come in handy – the police, for example, might investigate a murder by trawling through a list of the victims online acquaintances. All the same, it seems a circular and slightly primitive way of understanding who we are. We don’t necessarily act like our friends; often, in fact, we choose friends precisely because they’re so different from ourselves. And even if we did work out that someone is gay or Muslim, what would that really tell us – do people with the same religion or sexual preference really think alike?

*

In July 2009 actor Sacha Baron Cohen’s comedy film Bruno made an impressive one-day debut of $US14.4 million at US and Canadian box offices. The following day, however, it dropped precipitously, falling 39 per cent to $US8.8 million. Americans and Canadians had reacted quickly and badly to what they saw of the film, media reports surmised, and had voted with their tweets. Since then, many big companies have been working hard to take account of the “Twitter effect.” The result is a growing cadre of “sentiment analysts” who are paid to trawl through our electronic chatter and find out what we make of products of all kinds. One of them is a sprightly woman called Margaret Francis. When I visited her in her office in San Francisco last year, she gave me a crash course in how to do it. Francis works for a company called Scout Labs (it’s since changed it’s name to Lithium), and she and her colleagues have trained a computer programme to recognise thousands of words which come up in publicly accessible online conversation. She doesn’t need to know whether the people she’s eavesdropping on are gay or Muslim; she doesn’t even need to know their names. The only thing she cares about is what they’re saying about her clients; what comes out the other end, she showed me, is a bar chart tracking the murmurings of online opinion about different companies.

Sentiment analysis is a growth profession. While we dart around online, teams of ethnographic bird-watchers are looking over our shoulder to find out what we’re tweeting about. Last year a pair of researchers from Hewlett Packard’s Lab in Palo Alto used a computer algorithm to crunch the positive or negative sentiments expressed in 2.9 million Twitter messages about 24 movies. The result, they claimed, perfectly predicted the box-office performance of each film, with an accuracy of over 97 per cent in the opening weekend – they’re now in the process of patenting it. Quantifying sentiment in this way, according to its boosters, isn’t only useful for its insights into our mood – it can also help us understand the direction in which things are headed. In October of last year, a team of researchers at Indiana University classified 9.7 million Twitter posts under six mood categories (happiness, kindness, alertness, sureness, vitality and calmness), and reckoned that by doing so they could predict changes in the Dow Jones Industrial Average. It’s not as easy as it sounds. Recognising the nuances of online conversation can be a tricky business, and Margaret Francis has one of her analysts read through the data to look for discrepancies and make sure the computer’s getting it right. “We had motherfucker on the list as a negative word”, she told me, “and then we were like ‘why does the machine think this is negative and a person thinks this it’s positive?’ And then you look at the content, and it says ‘badass motherfucker’ – and motherfucker is only a bad word, it turns out, if it isn’t preceded by ‘badass motherfucker’ or ‘ ‘that righteous motherfucker’. There are all these permutations of motherfucker that are good.”

In her previous career Margaret Francis spent much of her time dissecting customers into different demographic groups, but when I asked her about it she rolled her eyes as if it were ancient history. “You’re taking me back years now”, she said. If the stuff that she does works, it’s only because the audience itself is changing its form. In the time we spend online we’re developing passionate attachments to the most recherché of interests and flocking together with those who feel the same way. Those we join up with online aren’t usually our friends, but those with whom we share a passion or an affinity. Identify with something, however, and it’s easier for other people to identify you with it too. For half a century everyone from marketers to political parties have been using demographic information about us – how old we are and our gender, our ethnicity, our sexual preferences and where we live – to help them predict what we might want and how we might act. Most of it was guesswork: reading between the lines of large datasets, asking questions of small samples of the population in the hope of blowing them up into more general significance. It was always a little flimsy, an unreliable guide to how we were. This is different. Instead of identifying people via their accidental demographic characteristics, it identifies them by the things they’re interested in, which usually happen to be the stuff they really like. It’s not perfect, of course. Tracking real-time buzz about a product or a film in real-time is usually much too late. If the film’s a turkey, it’s a bit too late for the studio to get its money back. In any case, this is a brutal and unforgiving way to understand what works and what doesn’t – the best stuff usually takes time to breathe and develop. But for anything susceptible to being built up or understood over time, the opportunities are almost limitless. For everyone from TV shows to pop bands to political sub-cultures, these new tools up whole new routes to connecting with a real audience (rather than a statistically imaginary one) and growing audiences more organically than ever before.

But it’s not only businesses and institutions who have something to learn from it. Just as Abdur Chowdhury guessed and those subterranean data hounds seemed to realise, all that information about us online is likely to be a huge free gift to social researchers as well as the downright prurient. The masses of data which now exist about us on places like Facebook, Twitter, blogs and the net seem sure to revolutionise the slightly arcane world of polling and social research. The industry, after all, is still young – its origins can be traced to the middle decade of the 20th Century. One of its early pioneers was a bunch of well-meaning British anthropologists who, in 1937, took to the streets to report the behaviour of ordinary people in cognito because they didn’t trust more traditional sociological ways of understanding how ordinary people behaved and believed. In what became known as the “mass-observation” experiments – it all started with a letter on the pages of the New Statesman – hundreds of observers were sent out to mingle among the natives and quietly report back on the everyday habits and behaviour of the British public. This new kind of self-reported mass observation has the potential to be every bit as illuminating – if only we know what to look for, and which questions to ask.

©James Harkin. James Harkin is Director of the strategic research agency Flockwatching. His book Niche: Why the market no longer favours the mainstream was published in March by Little, Brown

So let’s start with the state of the product rather than the medium: news. Long before the arrival of the internet and online news, I argue in my book Niche, many of the big public conglomerates which control the mainstream press were shaving away at their product to deliver more value for their shareholders. At the same time, faced with competition from television and the internet for our attention, many of them kept expanding their remit, spread themselves ever more thinly over more and more subject areas – everything from reality TV to the Royal Family – in an effort to retain the attention of their audience. The result is that throughout the 1990′s many newspapers put on a huge amount of weight, but much of the extra bulk consisted of little more than advertising vehicles hitched clumsily to a flagging editorial engine in a desperate effort to deliver eyeballs to advertisers. After years of these painful cuts, newspapers and general-interest magazines have been forced to take the plunge into a vast ocean of online information and social media. Faced with new threats from online media and new pressure on their profits, many of those same newspaper groups have responded by announcing yet more rounds of cost-cutting, laying off more journalists and spreading themselves even more thinly over more subject areas. And now, with the industry is such a parlous state, you can see the temptation, stoked by a bunch of flaky internet gurus; given the strain on news resources, why not just strive to be just part of the conversation on social media? Why not get readers to gather the news themselves and then spread it around?

Well I think that strategy isn’t going to work, and it’s a good thing too. In fact, I think, what’s called for is much more radical root-and-branch approach, and it involves paying closer attention to the project. Thrown into vast online eco-systems like Google and Facebook, the new news operators that I see flourishing and growing into profitability are those which stake out a coherent and distinctive niche within those eco-systems and use that distinctiveness to rope in an audience which really believes in the product and identifies with it. The success stories I’m inspired by are those like Politico – set up by émigrés from the Washington Post, Politico is written solely for political news junkies and has broken some of the most interesting political stories in Washington in recent years. In only two years Politico has grown a presence in Washington bigger than any other news organisation, while the general-interest magazines Time and Newsweek continue to cut back and lay off reporters. Politico has succeeded, just like Mark’s employer The Economist, by defining itself very narrowly. Rather like The Economist, too, it has a very distinctive house style, and its tone is very deliberately insiderish, designed to cultivate the sensibility of being in an in-crowd. Or take the online celebrity gossip magazine TMZ, run out of Los Angeles and the website which was the first to break the story of Michael Jackson’s death. Or The Register, a quirky, contrarian British tech newsletter which has cultivated a global following and which makes heaps of ad money. Just like Politico, both these sites have staked out a niche which they’re obviously passionate about, and attracted an enthusiastic audience from far and wide as a result. The tone of their reporting is deliberately insiderish and even cryptic, and many of their readers think of themselves as fans as much as readers.

Stake out a coherent niche producing a certain kind of news within the vast, almost limitless expanse of the internet and an audience can come flocking to you from anywhere in the world. Not only that, but it’s distinctive enough for them to want to identify with it, they’re much more likely to want to spread it around the place on social media. In some cases it means that they’re also likely to want to pay for it, but even if that’s not an option then targeted advertising and the changing nature of advertising demographics mean that advertisers are going to pay a premium for sure and certain knowledge of what an audience is really really interested in. And if you can’t make money out of sales or advertising, then it’s often possible to make money around a product that people really identify with. I’m amazed at the number of friends I have who only rarely buy The Guardian or read it for free online, but who seem quite happy to pay thirty pounds a month to meet other Guardian readers through its online dating service Soulmates. Stake out a distinctive enough niche, too, and the audience can become genuinely useful, because it actually knows things. The reason why TMZ was able to beat the world’s media to the news of Michael Jackson’s death, for example, is because it has an unrivalled team of sources working in hospitals who really love and trust TMZ, and who want to be involved in it’s operation. But this is a far cry from the mantras of the internet gurus about crowd-sourcing or ‘creating a conversation’ with the audience, which is really just old-fashioned customer engagement, and often deeply patronising. It’s about staking out a niche specialised enough to attract an audience which actually knows stuff – everything from astrophysics to Syrian politics to many of the subjects traditionally covered by investigative journalism – and then trying to make use of the information they have to offer.

My rule of thumb when talking about the future of the media is to ignore anyone who talks about networked anything – be it networked politics, networked journalism or anything else. Being part of the conversation isn’t enough, and talking up febrile buzzwords like mash-ups and crowd-sourcing are often rather desperate attempts by middle-aged executives to seem relevant and forward-thinking. If we really want to be bold and forward-thinking, my provocation would be to stand back and think about the product rather than the medium – to think about what news used to mean, what it means nowadays, and what it can mean in the future. Rather than all the hype and hokum about social media, I think, carving out specialised news areas is the best way to keep professional journalism alive, afloat and profitable in the years ahead.

James Harkin is the author of Niche: Why the market no longer favours the mainstream. This is a version of a talk presented by the author to the European News Journalists Network in Brussels on 10 October 2011.

Now there were good reasons for thinking that something was afoot, in 2009. First of all in Moldova and then in Iran, protesters and dissidents took to the streets and seemed to using social media to organise and outwit the police. To many technophiles hovering around State Department policy-making circles, the excitement was palpable. And not only there.In a series of blog posts fired off within hours of the demonstrations breaking out Iran, the Atlantic’s Andrew Sullivan proclaimed Twitter “the critical tool for organizing the resistance in Iran”; he even threw in a little electronic agitprop, declaring that “the revolution will be twittered.” Another prominent cheer-leader was Clay Shirky, a prominent internet evangelist and Professor of Telecommunications in NYU. Shirky is the smartest and most articulate of the internet gurus, and in his book Here Comes Everybody he makes an important claim, that from now on freedom of association and the freedom to publish things online were going to be central to protecting our political liberties. “To speak online is to publish, and to publish online is to connect with others. With the arrival of globally accessible publishing, freedom of speech is now the freedom of the press, and the freedom of the press is freedom of association.” Taking the example of a group of Belorussian activists who’d outflanked the secret police by organising their demo on a blog, Shirky predicted that the internet would prove especially useful in regimes which keep a tight rein on the means of communication, because dissidents could use it to give the authorities the slip. “The government can’t intercept the group members in advance, because there is no group in advance.”

Now this is the first error which I think is made by the cause of internet freedom – to equate the freedom to write things on Facebook with real and substantial freedoms like freedom of speech and freedom of association. Because what happened in Iran in 2009? Well for one thing, if you read the reports of Iranian expatriates and books like Death to the Dictator, the internet wasn’t quite as important as the internet gurus claim. Only a vanishingly small number of people in Tehran actually use Twitter, for example – 60, according to one survey. Many more use Facebook and mobiles, but they did so cautiously. It turns out that the activists involved in those demonstrations had a much more healthy scepticism about the power of the internet to liberate them, because they knew full that the Iranian government was taking steps to shut down and monitor social media. If the Iranian protesters were a little paranoid about mobile phones and the net, they had every reason to be. While some demonstrators were busy rounding up virtual support from the outside world, the police were scanning social networking sites to round up them. The Iranian authorities and their allies were quick to get into the swing of social media, and were soon flooding mobiles and the internet with false information and videos of dubious authenticity in order to intimidate, divide or demoralise the opposition. “Dear citizen”, read one cheery text sent to known protesters, “according to received information, you have been influenced by the destabilizing propaganda which the media affiliated with foreign countries have been disseminating.” The freedom to publish afforded by the internet, in other words, was important, but it was equally important to the authorities who used new media and social media for very sophisticated systems of surveillance. Rely too heavily on the internet as a means of fomenting an uprising, in other words, and it’s very likely you’ll find the Secret Police knocking at your door. And we’ve seen this again and again in the Arab Spring, from Mubarak cutting off the internet and mobile phone networks to the Syrian Government, possibly using expertise lent them by the Iranians, to shut down or monitor mobile and social networks in Homs, Deraa and Damascus. And what happens then is a cat-and-mouse game between the activists, the monitors (and, often, the American State Department and the CIA) to see whose programming and hacking skills are better – a battle which might not always be won by the right side, and which if pursued too vigorously can distract from building a political movement.

Following on from that opposition between internet freedom and internet surveillance, I want to talk about a second problem with the notion that the internet is a panacea for activists under authoritarian regime and a second opposition. I call it the difference between strong ties and weak ties. What I mean by strong ties aren’t family bonds or even always community bonds but the kinds of trust and mutual respect necessary to build up a political movement under difficult conditions. Evangelists for new media, everyone from Mark Zuckerberg to sociologists who talk about network theory, favour what they call weak ties – the kinds of ties which bring us all together on social media, and which Facebook so powerful. And when internet gurus talk about collective action – which they equate with online collaboration – it’s the weak ties which they tell us do most of the work. Weak ties are certainly what produce flash mobs, like that famous example when all those hipsters were invited by email to turn up in Macy’s in New York and stared at a single item of furniture. It’s all great fun, but the problem with flash mobs built out of electronic ties and made up of people who have no prior relationship to each is that they turn out to be flashes in the pan – no good at all for building really engaged, strong political movements. Weak ties, particularly made out of a global electronic diaspora which wants to empathise with the movement from the safety of their spare rooms in New York – as happened in Iran 2009 with Andrew Sullivan and Clay Shirky – can be very useful in getting messages out of the country, as very useful as a kind of emotional catharsis for those involved. But they can also distract an indigenous movement from building a real and indigenous movement at home, confuse activists who don’t always know who is saying these things, and lend ammunition to their enemies at home who want to portray their movement as a foreign plot. A tellingly example of the utter weakness of weak ties happened when the Sudanese government set up a Facebook page calling for a protest against the Sudanese government, naming a specific time and place – then simply arrested those who showed up. So I’d argue that the arguments of internet gurus about the power of weak electronic ties are useless when it comes to politics, and when exported abroad, have the potential to be disastrous. Given the inchoate nature of many of these popular movements, just about the last thing they need from the American State Department is the idea that the authorities are going to be no match for a new kind of networked, shiftless, leaderless, just-in-time disorganised organisation. They had enough of that already. And think about this – as we move towards democratic elections in Egypt, all this can have real consequences. The Muslim Brotherhood and the puritanical Salafist movements are built around the kind of strong ties which have been years and decades in the making. If we encourage Egyptian democrats to rely too heavily on a kind of Facebook democracy, we risk distracting them from the work of building sustainable political movements. And if that were to happen, Egyptian democracy itself might turn out to have been a flash in the pan.

The last opposition I want to talk about is truth versus hearsay, and here, for the sake of balance, I’d like to have a go at the enemies of the American State Department. Because within a year of the launch of this great crusade for ‘internet freedom’, launched by Hillary Clinton in a carefully choreographed series of speeches in 2009, it had fallen flat on its face. Why? Because an upstart movement of hackers called Wikileaks published a huge trove of nearly a quarter of a million of it secret diplomatic cables, at which point the State Department’s commitment to internet freedom proved to be very hollow indeed. They went ballistic. But left uncontested in all the venom hurled in Assange’s direction is his idea of ‘radical transparency’, which tends to equate freedom, democracy and even the public sphere with the ready availability of online information. And the reason why it was left uncontested is because the State Department largely agreed, at least until it was mugged by it. It’s interesting to think about what Julian Assange and the American State Department have in common. Just like many policy wonks in Washington Julian Assange hasn’t been shy to credit for the Arab uprisings, and his cables were certainly read widely in those countries, but what the citizens of Tunisia, Egypt and Syria knew about their rulers far exceeded the tittle-tattle available on Wikileaks; as we say in Belfast, the dogs in the Cairo street knew what was going on in Mubarak’s prisons. If we rely too much on that trove of tantalising embassy cables on the internet, all we really learn is what American State Department thinks about what’s going in the rest of the world, not what’s really going on in those countries. The banners of ‘internet freedom’ and ‘internet transparency’ signed up to by both the American State Department and the Wikileakers, I’d argue, are simplistic and often arrogant ways to look at how the world works. Pay them too much attention and we risk losing the real truth about events going on on the ground.

What the American State Department, the internet gurus and the data evangelists all miss is that mere access to online information doesn’t on its own open the door to power, freedom, democracy, or even necessarily get us closer to the truth. The obsessive Western focus on social media and online data robs politics and culture of its content, and risks blinding to the real dynamics of change in those countries. As we continue to watch the citizens of those countries open their own door to freedom, we need to make sure we’re not leading them down a blind alley.

Human Rights lecture, delivered by James Harkin at Georgia Tech in Atlanta on 15th September 2011 to mark the first anniversary of the Arab uprisings. James Harkin is the Director of Flockwatching

Not any more. As dog-tracks have dwindled along with the working-class communities which gave rise to them, a new class of patron has arrived to claim the greyhounds that no-one wants. Walk around any big city and you’ll see them: students, hipsters and arty young professionals, many of them novices to dog-owning, daintily dragging their pets around the shops while fiddling for the starter poop bags they picked up at the dog home. It’s a happy turn-around, because the truth is that most retired greyhounds are abandoned as soon as their racing days are done. Worse, many of them are callously slaughtered. Luckily, they’ve discovered a whole new gig – as the preferred pooch for bohemian young urbanites in need of something skinny and good-looking to drag around the place.

It’s not hard to see why retired greyhounds make such excellent accessories. For a start they’re thrillingly low-maintenance. The mature greyhound is in need of only two brisk walks a day – if it didn’t need to use the loo, it probably wouldn’t bother to go out at all. It could catch a squirrel any time it wants to, but it’s outgrown the restless angst which drives the younger canine to chase itself around the place – it’s done all its running, has won all its medals, it doesn’t feel it has anything left to prove. A greyhound is a dog for people who like cats – it doesn’t require any attention, and when not sleeping it spends most of its time rehearsing a series of far-out yoga positions, many of which involve lying with its legs in the air. Rather than barking at you until it’s blue in the face until you give it something to eat, a greyhound will look at you mournfully, as if it’s sorry it even had to ask. It prides itself on being both clean and quiet, which makes it ideal for the bijou little rabbit-hutches (what an irony) which are the only thing young professionals can now afford. Not that the greyhound’s complaining, you understand. It knows the deal – it’s an accoutrement every bit as much as a poodle, but for a younger and hipper crowd. Anything’s better than the dogs home, and if that means wearing an outrageous tartan body warmer just so his human associate gets noticed on the way to the shops, it seems like a small price to pay.

What the owner gets out of this is a little more difficult to fathom. Greyhound chic, at least in part, must be related to nostalgia for the greyhounds rough and ready racing past. From bingo to pub quizzes, many pursuits traditionally favoured by the lower orders have recently been reworked by urban trendies. Then there’s the ethical dimension. Being seen with a greyhound marks you out as someone with the desire to do your bit. Reclaimed from the doggie equivalent of the knackers yard, recycled into a life of domestic luxury – if pushed a greyhound’s owner will even show you its racing medals, and ruminate wistfully on what might have been if he hadn’t showed up. Then there’s the sheer snob-factor of going for a walk with the canine equivalent of a super-model on your arm – and watching it tip-toe elegantly past the pit-bulls and mastiffs preferred by the hoi polloi. Just don’t imagine it’s going to fare very well in a fight.

What’s the evidence? Walking down the Walworth Road in South East London on Monday evening, I tried to send a text five times from the O2 network and each time the system failed. At least one other person had similar problems, but other reports on social media complained that network coverage went down in Hackney and in Peckham. This is unusual, but might happen for a number of reasons. It’s entirely possible that the mobile network was simply overloaded, of course, with panicked local residents trying to ensure the safety of loved ones.

Just to be sure, I asked the Metropolitan Police. The question I posed late last week was “Has the Met at any time in the previous four days attempted to shut down or otherwise restrict mobile networks, Blackberry private networks or the internet in the affected areas? If so, how so?” I asked this question of their press office three times. Eventually a spokesperson for the Metropolitan Police came back to with the bald answer. “We are not prepared to discuss.” Which, given that the operation is no longer ongoing, is very far from a blunt denial.

The question of whether the authorities restrict or clamp down on coverage is intriguing for a number of reasons. For a start, it tells us something interesting about the way that the various departments of department work together with industry to manage crises. But it’s also of much broader political significance. In recent years authoritarian regimes everywhere from Iran to Belarus have successfully restricted access to their mobile networks in an attempt to damp down protests, and been roundly condemned for it by Western. The American State Department, for example, has expended a great deal of time and effort furnishing activists with the software to outwit this kind of surveillance. The outbreaks of lawlessness we’ve seen this week were very far from political protests. On the other hand, we do need to know how and on what occasions the authorities can move to shut down mobile networks – and what the consequences might be. And there’s a precedent for all this. Last October, the inquest into the 7/7 bombings heard that a senior officer from City of London Police had invoked powers to restrict use of the O2 network around Aldgate Tube station to members of the emergency services with special handsets. Lady Justice Hallett, the coroner, even noted that she had problems using her mobile on the day. July 7, 2005 bombings in London. That decision, the inquest heard, was limited to the 02 network and to the City of London Aldgate area. It was taken by a City of London Police superintendent at 11.40am under something called the Access Overload Control (ACCOLC) system. The mobile operator 02 duly restricted access for two hours after the bombings, and kept the restriction in place for the next five hours – which might well have hampered the emergency response. There were problems with other networks too on July 7, but that was largely because they were overloaded with people trying to get in touch with their family and friends.

But that was different to the ACCLOC decision, which proved deeply controversial. A report by the London Assembly in 2006 went as far as to criticise that decision. Some police officials might even have misled the London Assembly about the problems which occurred on the day. At the time of the 2006 report, the City Police, according to The Economist’s writer about technology and public policy Kenneth Cukier, initially failed to admit that it went outside the chain of command to request that a wireless operator provide telecom capacity to certain emergency-responders by degrading service for everybody else, even though the Metropolitan Police had earlier decided this was unnecessary.

O2 told me that they took no steps to restrict network coverage during the riots in the last week. But there were lots of other network operators for the police to call upon. New media can be used by the perpetrators of violence and disorder, but for the most part they weren’t using social media but semi-private networks like Blackberry Messenger. New and social media can certainly be used to spread panic dis-information, possibly a larger threat in instances of domestic disorder like we’ve seen in the last week. But at times like this a panicky over-reaction by the authorities usually makes things worse. Access to online and mobile networks is now as ubiquitous as oxygen, and cutting off the supply can itself sow the seeds of panic and paralysis – and prevent ordinary people from breathing deeply and then getting on with their lives.

It doesn’t last long. The first shop to take a bashing is Payday Loans, the local loan shark. When that’s emptied a few hundred young people, most of them wearing hoodies or scarfs, races up road towards Peckham Rye and I follow. The excitement, it turns out, is because another group has forced its way into a fashion retailer, Blue Inc., and is making off with all of its stock. A few hundred of us stand back across the road waiting for something to happen while, a few yards away a few yards up the road in the mouth of Peckham Rye, another phalanx of baton-wielding visors stands guard over the rest of the shops. A sweet-looking boy of no more than about eight emerges from Blue Inc. beaming and holding a clutch of funky T-shirts, still in their hangers. As he makes it back to our side of the street, he gets a huge round of applause.

A few older boys wearing hoodies run forward from our group to hurl rocks and abuse at the line of policemen, who occasionally respond by charging in our direction, at which point everyone races back to the relative safety of Peckham Library. This goes on for several hours. At one point a convoy of strange little box-like vehicles arrives and races past the library in the direction of Peckham High Street, and everyone runs in panic. “Those are the feds, man”, a black girl of about ten tells her friend, as if she’s talking about the bogeyman. “The ones who really hurt you.” I’m chatting to an Irish gypsy woman and her family about what’s happening elsewhere when all of a sudden the police rush towards us and I lose her in the melee. In between the rock-throwing and baton charges, a chubby black man rushes back through the no-man’s land to pick up one last suitcase full of gear from Blue Inc. and raises another cheer. “There’s nothing in your size, mate”, the man beside me shouts, and everyone laughs.

The atmosphere is akin to school sports day, or a visit to a rowdy open-air cinema. A few of them try to set a fire outside Blue Inc. but they can’t get it going. When they move on to the pawnbrokers and spend twenty minutes trying to prise the grille open, they’re beginning to bore the girl who handed out the Haribos. “Why don’t they do the hair shop, have you seen the products they keep in the back of that hair shop?” Even if the younger ones find it funny, many of the older people I speak to despair of all this looting. At least half those here are gawkers like me; some young black men I talk to are shaking their heads at what happens, and one white guy wearing sportswear and tattoos complains that “I’m all in favour of people having a tear at the police, but they shouldn’t be doing their own shops, do you know what I mean?” When the hoodies finish with the loan sharks and pawnbrokers and start cleaning out a local fashion boutique, an angry young black woman berates one of them. “You’re taking the piss, man. That woman hand-stitches everything, she’s built that shop up from nothing. It’s like stealing from your mum.” A girl holding a looted wedding dress smiles sheepishly, stuck for anything to say. A EuroLines bus with French name-plates slow-coaches up Peckham high street and the tourists get more of London than they bargained for.

As I leave Peckham high street and walk back towards the Old Kent Road, a teenager carrying a boxed Nintendo Wii nearly knocks me over, and then apologises and goes on his way. He’s on his way back from the Argos just across the road from where I live. By the time I get there everything is being carried out – it’s like the generation game without the conveyor belt – and floods of young people are still arriving to claim their share. Eventually a police car arrives and an lucky pair of teenage girls drops a flat-screen TV and run off screaming. Across the road, the young professionals who moved in across the road from me are standing milling around looking on, one of them still in her socks. They’re enjoying the show as much as everyone else.

That’s the irony of all this. For years now, wonks and idea-entrepreneurs have been banging on about ‘social capital’ ‘community engagement’, ‘social networks’ and ‘transformative communities’. Only rarely do their schemes take hold; they’re prefer to talk amongst each other, and they’ve succeeded only in enhancing their own station. For all the senseless destruction, on Monday night I talked to more locals, and got more of an insight into my local community, than in the previous five years put together. In the dog days of August, it feels like an alternative summer festival, a sort of Big Chill for under-stimulated, under-employed urban youth. You don’t even have to get your feet muddy.

What the wonks and idea-hawkers forgot about was something as simple as politics – going out there and getting your message across rather than sitting in conference rooms bleating about health inequality. It doesn’t make much sense for politicians to bleat and tub-thump about ‘responsibility’ and invoke Victorian morality – they’ve been doing that for decades, it’s gotten us nowhere and it’s not really their job. What they need to too, especially those on the left, is to offer these young people a way of understanding their situation – without which politicians are part of the problem. New Labour felt, quite understandably, that it didn’t much need the loosely-employed poor anymore. In so far as it did address them, wonks talked about their ‘poverty of aspiration’. But that was in the boom years. Now, and for many of these young people, there’s not much much left to aspire to.

There’s much more than brute criminality here. This is pure inchoate rage and ennui. Given the lack of politics or direction, of course, there’s the danger that it might turn much uglier. In Peckham blacks and white were rubbing along quite nicely – just like in America, race is much less salient than it was thirty years ago. But, in the absence of any politics, the reaction to the riots might be different. In Eltham, there’s been a rise in vigilantism and some evidence that the EDL are muscling in on the disturbances. Next time the fires may be more difficult to put out.

James Harkin is the author of Niche, and Director of Flockwatching. An abbreviated version of this appeared on 10 August 2011 on the news pages of The Guardian (with Paul Lewis). Read the news story here

•••• (4 stars)

•••• (4 stars)