How Amazonfail was born

Amazon is the world's leading e-tailer

Bill Thompson spent his Easter weekend watching an internet giant stumble.

While millions of people tuned in to Doctor Who and Red Dwarf over the Easter weekend my holiday entertainment was provided by typing 'amazonfail' into the Twitter search engine and watching the stream of outraged posts about the company that used to be the world's favourite bookstore flow across my laptop screen.

The PR nightmare started at some point on Sunday when an angry post on the LiveJournal blog site by author and publisher Mark Probst broke through into online consciousness. He had noticed that his book The Filly, though still listed on Amazon's US website, had lost its sales ranking data and was no longer appearing in relevant searches.

An e-mail from an Amazon representative informed Probst that his book, a romance featuring gay characters, had been classed as 'adult' and removed from the ratings system so that the search pages would be more 'family-friendly.'

Two other high profile gay romance books, Transgressions by Erastes and False Colors by Alex Beecroft, had also lost their sales ranking. This meant it was only possible to find them by looking specifically for the title and author, so people searching for similar books, or similar titles were unlikely to find or buy them.

It emerged that thousands of other books had been similarly delisted, including such radical texts as The Well of Loneliness and John Barrowman's autobiography, while a little research by interested bloggers found Playboy: The Complete Centerfolds, the Parent's Guide to Homosexuality and Hitler's Mein Kampf were all still searchable and proudly displayed.

BILL'S LINKS Amazon have clearly broken the bond of trust with a large number of their readers, and it will take a long time to recover. Bill Thompson Mark Probst posting on Amazonfail Amazonfail search on Twitter search Dear Author analysis Girl With A One Track Mind

Two years ago this would have resulted in a collection of angry, interlinked blog postings. A year ago there would have been a Facebook group to join. But this time it was the Twitter microblogging service that led the way, with thousands of tweets linked by the tag 'amazonfail'.

The timing was perfect. It was a slow news weekend on what is an extended holiday in many parts of the world. Amazon's ability to respond quickly was limited, while the echo chamber of Twitter, LiveJournal and Facebook meant that the noise of outrage quickly reached a crescendo.

Many of those affected were heavy net users because authors have realised that engaging with the online audience is a good way to boost visibility and sales. In fact I first became aware of the issue after furious tweets from a friend, the sex blogger Abby Lee, after her book Girl With A One Track Mind dropped out of Amazon search results.

Reputation plummeted

Within hours Amazon's reputation had plummeted among those - whether gay or straight - who think that being inclusive about homosexuality does not make something 'adult' or merit banishment to the ghetto of unranked titles.

As I write this the company has claimed that the episode was an unfortunate and does not reflect a new policy, and I'm tempted to believe that it was never their intention to delist or downgrade books that are about gay, lesbian, transsexual or bisexual issues - or those written by authors who are not aggressively heterosexual in their appetites.

Some careful analysis by "Jane" on the Dear Author site indicates that the problem lies in the metadata, the additional details about each book added by Amazon and publishers. Books classed as "gay", "lesbian", "transgender", "erotic" or "sex" have been filtered while the Playboy book, whose content is classed as "nude", not "sex", remains.

In fact some sort of filtering seems to have been going on since at least early February when former gay stripper Craig Seymour saw the sales ranking on his memoir disappear. He complained at the time and the ranking reappeared.

What this seems to show is that Amazon are trying to make search results more 'family-friendly'. A children's author I know has been in discussion with Amazon over this for well over year after clearly adult titles kept cropping up when people searched for books she had written.

Broken bond

There's nothing wrong with this, and Amazon aren't violating the First Amendment or even, I suspect, breaking the terms of their agreement with publishers by doing it. However they have clearly broken the bond of trust with a large number of their readers, and it will take a long time to recover.

Filtering in itself is not a bad things, and every time we do a search or turn to a web service we expect some sort of selection to take place, otherwise we would be faced with too much information to cope with. But filtering has to be done in an open, transparent and configurable way, like Google's 'safe search' which offers a simple all or nothing model that can easily be turned off.

There's a lot of speculation about what actually happened, with some putting forward a view that this was a carefully planned move by US fundamentalists who were gaming Amazon to cause just this sort of public upset and one or two people claiming they were behind it all.

But I'm pretty sure that the error was in the algorithm, in the set of rules that Amazon's systems apply when selecting books to display, coupled perhaps with some unfortunate choices about how certain books should be tagged in the metadata Amazon holds.

This is not, however, a source of comfort. When a book is misfiled in my local Borders it may result in a few lost sales, but for the whole of Amazon to 'misplace' my book may mean nobody in the world buys it. If the filtering had affected people writing about how to keep sheep as domestic pets instead of gay fiction we might not have noticed the error.

The consequences of living by the algorithm do not just affect Amazon, they affect all of us as we increasingly rely on recommendation systems to suggest books to buy, friends to add on social networks, emails to take notice of and places to visit. We have put our faith in Google PageRank and 'Amazon recommends', and found them wanting, yet we do not have an alternative.

The only real solution is the one that fixes so many other problems. We need transparency for the algorithms just as we need transparency about MPs expenses, police behaviour, programme interfaces and how the deep packet inspection that will deliver targeted adverts to our web pages actually works.

Of course we then have to trust the companies, agencies and government departments involved to implement the systems in accordance with the published specifications, and it's unlikely that Amazon or any other e-commerce service will willingly publish its source code for inspection. Perhaps the UN should commission internationally recognised 'algorithm inspectors' just as it has 'weapons inspectors'.

Bill Thompson is an independent journalist and regular commentator on the BBC World Service programme Digital Planet