Throughout the last century, a certain London address was inundated with strange requests. People across the globe wrote to 221B Baker Street, seeking to procure the services of its famous fictional detective – that is, to persuade Sherlock Holmes to apply his deductive prowess to the most pressing conundrums of the day, from the Watergate scandal of the Nixon era to the mysterious and unsolved murder of Swedish prime minister Olof Palme in the mid-1980s.

The feeling that Holmes was more than a fictional character – that he in some way existed in reality, albeit wreathed in mystery and shadow and the London fog – was audaciously cultivated from the get-go. In the original tales, Conan Doyle offered the character of John Watson as an extant individual – a doctor (and writer) who happened to live and work with the master detective, and whose chronicles referred to real events. The short stories were serialised in The Strand magazine. To the audience, it seemed the events unfolded in ‘real time’, so to speak: put to paper in diary form by an actual person swept up in their flux.

The device was an ingenious one, offering the same sense of reality that (a century later) the mock documentary afforded to directors in movies like The Blair Witch Project. Such was the identification with the Holmes character that Conan Doyle found himself a figure of hate – and at times a target of abuse – when, eventually, he extinguished his sleuth in the foaming waters of the Reichenbach Falls.

Later, as a concession to readers’ demands – as well as to a certain lightness of his pocket – Conan Doyle reanimated his famous detective. It was only the first of many resurrections.

What is remarkable about Holmes, then, is the way the character really does seem to possess an independent existence, stepping out from the pages of the original stories and into the slipstream of historical time. We find Holmes reappearing in successive epochs, refined and rearticulated, applying his awesome mental powers to the crises of the day. In one incarnation we are treated to a Holmes who bands up with Sigmund Freud to thwart crime; in another we discover Holmes fighting the Nazi menace – and so on. The current, incredibly popular BBC series starring Benedict Cumberbatch and Martin Freeman has transplanted some of the original Conan Doyle tales onto the panorama of twenty-first-century London.

Holmes’ ability to transcend historical epochs is complemented by a concomitant skill in defying the limitations of genre. The Holmes of Guy Ritchie’s Hollywood spectaculars is almost of the superhero mould (a muscular incarnation who battles multiple bad guys with his honed physical skills), while the 1957 Hammer film adaptation of The Hound of the Baskervilles descends into Gothic horror. Even the Fox series House, which follows a grumpy but brilliant doctor, is a play on the Holmes archetype (the clues are abundant: ‘House’ is a pun, since many houses are also ‘homes’; the medic’s central confidant is one Dr James Wilson, who, of course, shares his initials and profession with John Watson). House is particularly interesting in as much as the Holmes character is wholly divorced from the pursuit of crime.

We have seen comedy versions in which Watson provides the true genius, with his partner merely a convenient front (Without a Clue); cartoon versions with Holmes depicted as a mouse and a dog (The Great Mouse Detective and Sherlock Hound, respectively); depictions of Holmes paired up with other fantasy figures (Batman: The Brave and the Bold); films that portray a teenage Sherlock Holmes (Young Sherlock Holmes); and novels that show Holmes during his twilight years (The Bee Keeper’s Apprentice)

Why has Sherlock Holmes proved so durable? The conventional response references his remarkable ‘deductive’ abilities. But if we put aside the question of whether Holmes’ reasoning is really an example of the deductive method (it is not), we see that what is really being revealed in and through Holmesian ‘deduction’ is that the great detective’s mind is a clinical instrument that uses unadulterated and ruthless reasoning to cut through the paraphernalia of the everyday, revealing the truth behind the epiphenomena of events.

In a certain way, Holmes’ Rolls Royce brain is very much a product of the enlightenment: it represents the endeavour to deploy pure reason, untrammelled by emotion and prejudice. And yet, as the philosopher John Gray points out, the twentieth century was, in many ways, a crisis point for reason, with the idea of collective rationality choking on the industrial smog of the death camps and the Gulags.

In our everyday life, Gray argues, belief in rational systems has become ever more untenable, with those systems – ‘from the security software we install on our home computers to the mathematical formulae used by hedge funds to trade vast sums of money’ – increasingly subject to attack.

Consequently, he says, it is impossible to explain the durability of the Holmes character with regards to its application of reason. Gray suggests that Holmesian ‘reason’ – because of its almost omnipotent scope and intensity – is actually more akin to a supernatural power, a form of enchantment, ‘a clairvoyant eye for detail … [that] demonstrates the enduring power of magic.’ Gray naturally references Holmes’ creator, who also felt the limitations of pure reason and ‘found consolation in spiritualism’.

Gray is correct, though the loss of faith in reason began earlier than his off-the-cuff remarks suggest: it can be traced back to the eighteenth century, when Jacobi detected the first particles of ‘nihilism’ in Spinozan rationalism, and Kant critiqued pure reason through his devastating exposé of its antinomies. But Gray is also wrong in as much as the crisis does not invalidate the archetype that seeks to personify reason. If anything, it provides an impetus for that endeavour.

Gray’s analysis misses literature’s Freudian inflection, its element of wish-fulfilment. It is precisely when rationality is found wanting that the imagination conjures forth a figure whose crooked pipe and deerstalker cap embody an idealised and perfected human reason operating and ‘deducing’ in a fictionalised vacuum, a realm unhindered by historical and social realities. Holmes is required by fantasy precisely when pure reason seems to falter.

Along with a crisis of reason, the Holmes character might also be perceived as a response to the parallel crisis of science – or at least our collective experience of science as a social force.

In the epoch of feudal medievalism, the general population had less education than today. But in that period, scientific innovation was more directly applicable to everyday life. Labour-saving techniques (three-field crop rotation, more effective fertiliser, deep ploughing, application of water power to threshing and milling, enhancements in tools, etc.) assumed and required some degree of specialist knowledge by the individual peasant, land-labourer or journeyman who implemented them.

The experience of the contemporary worker is quite different. Consider a cashier at Tesco – and I say this as someone who has spent a considerable period of time working as one. He or she is in possession of a remarkable technology: a light sensor capable of translating an image or pattern into electronic information that then registers instantaneously as numbers on a digital gauge. Yet despite the wonder of that, despite a technology embodying the knowledge developed during the history of science, the cashier that deploys the machine experiences the sensor only in terms of the slide of one item after another across the flat surface of an impervious black square – a mind-numbing repetition that is sustained hour by hour, month by month, year by year. As Karl Marx so eloquently noted, the mechanisation of labour implies the separation of the worker from the scientific-technological knowledge in the machinery he or she sets into motion:

The machine which possesses skill and strength in place of the worker, is itself the virtuoso … Science which compels the inanimate limbs of the machinery, by their construction to act purposefully, as automaton, does not exist in the worker’s consciousness, but rather acts upon him through the machine as an alien power, as the power of the machine itself.

Conan Doyle was writing in the aftermath of the first industrial revolution and he repeatedly describes Holmes’ mind in terms of a finely honed machine. At one point, for instance, the eponymous hero describes his mind as ‘like a racing engine’. Elsewhere, Dr Watson observes of his cerebral companion: ‘He was, I take it, the most perfect reasoning and observing machine that the world has seen.’

The sense of Holmes’ brain as a machine is supplemented by Conan Doyle’s outline of its workings: he explains how Holmes has a finite quantitative capacity to store particular facts and must, therefore, discard others. The detective’s brain is, he writes, much like an attic ‘stocked with all the furniture that he is likely to use.’

The point seems remarkably prescient, hinting at the modus operandi of the modern computer. The contemporary BBC reboot, Sherlock, accentuates the theme, with Cumberbatch providing us with a Holmes whose magnetic eyes flicker with the luminous glow of a computer terminal. When Holmes is embroiled in the processes of ‘deduction’, we are treated to the image of his mind’s inner landscape; various images and icons flash up in whirlwind succession, with the master detective able to pinpoint and affix pertinent information much as one might do with a digital touch screen.

In other words, though the standard analysis of Holmes is liable to (correctly) infer his inadequacy (his alienation from others as a result of his inability to address the demands of social etiquette), the character also represents a profound overcoming of alienation. In Holmes, the science that manifests as a social power in and through industry and its fragmented division of labour – a science that confronts the worker as an implacable, mysterious and often alien presence – is reappropriated so that it becomes, once again, a living function of an individual (which is why Holmes’ mind is likened to a machine). The full and awesome consequences of that redeployment are made manifest as Holmes masters the complexity and chaos that face him by the pristine instrument of his own un-alienated intellect.

Consequently, though Holmes is often thought of as a rather inhuman character, there is at least one way in which he can be regarded as a more fully embodied person. In a time when the productive process manifests itself as a force set against us, Holmesian reason represents the scientific aspect of production exhibited as a property of a given individual.

It is perhaps no coincidence that when, in the finely attuned BBC remake, Watson delivers his eulogy at the grave of his (supposedly) deceased friend, he describes Holmes as ‘the most human … human being that I’ve ever known.’ It’s a comment once again in the spirit of the Conan Doyle original – ‘The Final Problem’ – where Watson describes Holmes as ‘the best and the wisest man whom I have ever known’.

Paradoxically, it is that overcoming of alienation that sets Sherlock Holmes apart. His brilliance – the symbiosis of science as a social power and the individuality it is normally set against – renders Holmes an ideal type, a human being given over to the ideality of reason and science at the expense of visceral emotion. As Conan Doyle writes in ‘The Sign of Four’: ‘The emotional qualities are antagonistic to clear reasoning.’

Holmes is therefore a lonely figure, exiled from the emotional life of the species more broadly. In the overcoming of a specific form of human alienation, Holmes becomes fundamentally isolated. To paraphrase Nietzsche: human, yes, but all too human.

It is a contradiction at the heart of Holmes’ being, one that elevates him from being merely uniquely brilliant to possessing an aspect of tragedy.

Holmes’ relationship with Watson is crucial. Watson provides the link with broader humanity, the possibility of an emotional connection with other human beings. Yet the more Holmes finds redemption through his friendship with Watson, the less he is truly himself, since it is his freedom from emotion that allows him to transcend the social alienation to which the rest of us are in thrall.

For Holmes, therefore, love must necessarily be a form of self-immolation.

This contradiction produces an exquisite tension in the psyche of the reader or viewer. On the one hand, we are desperate for the friendship between Holmes and Watson to be – shall we say? – consummated, with Holmes finally giving Watson some genuine affection or even love. It is in the short story ‘The Adventure of the Three Garridebs’ that Conan Doyle comes closest. Watson suffers a bullet wound and Holmes shows his concern, but this rare instant of emotion is hinted rather than stated. Watson notices how Holmes’ ‘firm lips were shaking’ – and this, we learn, is the only time in their adventures that Watson has caught ‘a glimpse of a great heart as well as of a great brain.’

Such obliqueness is necessary. When Holmes comes close to showing emotion, we can’t help but feel instinctively uncomfortable – disorientated, discombobulated – because the display involves the ontological unravelling of the Holmes archetype.

The BBC series plays on the contradiction. Whenever Holmes is at his most soulful, the sentiment is snatched away a moment later, revealed as a subterfuge that Holmes has kept under his hat. At the end of the second series, for instance, an emotional Holmes stands on the edge of a building, prepared to plunge to his doom for the sake of Watson and Mrs Hudson. Yet the next episode reveals that the event was staged by Holmes to thwart his nemesis, with the emotion nothing more than an actor’s trick. Indeed, Holmes callously abandons Watson, allowing his friend to believe him dead for the next three years.

In a riff on the same theme, the third series of Sherlock sees Holmes becoming part of a romantic couple. This is disorientating for the viewer … and even Watson seems positively unnerved at the loved-up detective canoodling with his girlfriend and whispering sweet nothings in her ear. Again, the relationship seems to violate the essence of the character itself – until, in the next sequence, we discover it to be a calculated charade designed to manipulate his so-called partner into allowing access to a building relevant to an ongoing investigation.

The deft skill of the writers allows Holmes to occupy a type of limbo, teetering on the edge of emotional redemption but never quite grasping it, for to do so would be to annihilate his own unique and lonely brand of humanity.

The Seven-Per-Cent Solution, a pastiche novel written in the 1970s and later made into a film, showed Holmes pairing up with Sigmund Freud to prevent the outbreak of a Europe-wide war. The novel not only depicts Freud, it also offers a starkly Freudian premise: Holmesian pure reason is a pathological reaction to a traumatic childhood event, since under hypnosis the fictional Freud incites Holmes to reveal how he witnessed his mother being murdered by his father. But although the Freudian take on the Holmes character is ingenious, it nevertheless fails, as anyone who has seen the film will attest. The image of Holmes quietly weeping is perhaps moving, but the viewer feels a deep dislocation. The character looks like Holmes and even talks like him … and yet doesn’t feel like him at all. The disequilibrium is similar to our response to the full-blown romantic Holmes depicted – albeit fleetingly – in the current series.

We, the readers or viewers, can and must feel the possibility that Holmes actually has a genuine emotional warmth and even love for those close to him. But that possibility can only ever be hinted, for the moment it is actualised the Holmes character dies a death more permanent and enduring than the one inflicted on him at the Reichenbach Falls. For Holmes is a contradiction that must never be resolved – and this, ultimately, is the source of his ever-lasting appeal.