This is no London Dungeon shop of horrors – which is not to say there are no scares. Kerry Stewart’s 1993 installation The Boy From the Chemist Is Here to See You certainly gave me the creeps. It consists of a door with a frosted glass panel through which you see the refracted face of a child, actually a charity box figure, its frozen painted features adding to the unease.

This unholy marriage of conceptual art and supernatural dread is good evidence for The Horror Show’s claim that gothic subculture is the true dissident virus of the modern British imagination. The story told here starts to the hypnotic chant of Bela Lugosi’s Dead by goth pioneers Bauhaus, which makes the argument with every echoing beat that punk was always gothic, and goth its natural evolution.

The show succeeds in saying something fresh about the well-worn story of Britain’s 1970s youth rebellion: instead of artist Jamie Reid’s now-cliched Sex Pistols imagery it includes his painting of a giant owl-like green monster materialising on top of a suburban house. Monster on a Nice Roof dreams of impossible creatures coming to destroy normality. Another such monster from beyond materialises when you encounter one of 6’ 3” performance artist Leigh Bowery’s costumes, his green leather gimp mask, fake breasts and cloak towering above you.

The devil is in the detail, as they say, and it’s the enthusiasm of the curators – artists Iain Forsyth and Jane Pollard, together with Somerset House’s Claire Catterall – for arcana of pop culture that makes their thesis as rich and knotty as a forest. Who knew Mark E Smith of the Fall was a fan of the late Victorian horror writer Arthur Machen, author of The Great God Pan? They do, and they prove it with letters from Smith to the Arthur Machen Society. Another echo of classic horror literature is the sequence of artworks created by Richard Wells for Mark Gatiss’s recent BBC dramatisation of The Mezzotint by MR James: as you study the antiquated prints of a country house, at first sight identical, a ghostly figure starts to make its way across the lawn towards you.

There is a romanticism at the heart of this exhibition, a quest for a dream of eldritch Britain that’s always slightly out of reach. The film director Nic Roeg has a case to himself: it includes his copy of Daphne du Maurier’s stories that inspired his masterpiece Don’t Look Now. Elsewhere, there are black and white shots of the making of Robin Hardy’s folk horror The Wicker Man, and Christopher Lee’s personal script.

You can’t blame an exhibition called The Horror Show for wallowing in nostalgia. Horror is not healthy nor is it meant to be. Nostalgia for Joy Division is a case in point. Ian Curtis, singer with the terrifyingly beautiful Manchester band, was already dead by suicide when we 1980s teenagers passed around their records like holy relics. Yet Curtis haunts this show – and even, it suggests, modern British art. Kevin Cummins’s photograph of a snowbound Manchester motorway, ethereally slipping from grim reality into ghostly nothingness, comes from his shoot of the group in January 1979. Graham Dolphin’s 2012 sculpture, Door (Joy Division Version) tears you apart a decade on: it seems to be a door preserved from some long-demolished squat, painted a sepulchral pale grey, covered in graffiti – “RIP IAN”, “IAN C”.

This ghostly door is shown opposite photographs of Rachel Whiteread’s House, the most enduring yet also most elusive monument of modern British art: this concrete cast of a demolished house was itself demolished, a masterpiece that survives only as a ghost. We’re provocatively invited to compare Whiteread and Joy Division as artists.

There is another Britain, this exhibition convinces you, that exists only as a web of imagination, a phantom realm that defies the reality of the everyday like a ghost channel taking over your TV. An obsession of the curators is the infamous 1992 BBC Ghostwatch broadcast, that appeared to be a live broadcast interrupted by supernatural powers. This is shown in eerie fragments. You can see why viewers were freaked out – and why Inside No 9 recreated it.

I am starting to think all exhibitions of contemporary art should be curated by artists. Pollard and Forsyth don’t get snagged by the laboured rationalities that often crush shows. You need to think like an artist in order to be able to connect so many gothic strands, strike a pose that’s funny and serious at the same time, and leave us unsure whether we should laugh or scream or cry.

A good comedy moment is the mask of Bollo the gorilla from The Mighty Boosh. It connects nicely with Angus Fairhurst’s Pietà, a picture of himself being cradled naked and apparently dead in the arms of a stuffed gorilla. Fairhurst, who died in 2008, is another of the exhibition’s ghosts, walking in the shadows here, whispering of art’s power to resist.