The poet William Hayley was born in the Pallants in Chichester, probably on the site of the new extension to the Pallant House art gallery. In 1790, he was offered the position of Poet Laureate, which he declined. This indicated the esteem in which he was held in his time, although since then his stock has fallen. ‘Everything about that man is good, except his poetry,’ remarked the poet Robert Southey to Samuel Taylor Coleridge. Not quite everything, perhaps: he had an illegitimate son, Thomas Alphonso Hayley, by a serving maid. But he was a friend to artists and other writers, offering the support that a poet who was also in possession of a country estate on the South Downs, at Eartham, not to mention a ‘marina villa’ at Felpham, could afford to do.

Hayley is the starting point for the captivating small exhibition ‘Poets in the Landscape’, which examines the inspiration that poetry gave to art in the Romantic period. As nobody now reads Hayley, it is difficult to know how far his own verse influenced his contemporaries’ vision of the landscape. But his circle was remarkable. George Romney spent every summer for 20 years at Eartham, where Hayley tried to rally his depressive spirits by playing quoits. The neo-Classical sculptor John Flaxman became a friend after Hayley procured the commission for him to carve a memorial to the poet William Collins in Chichester Cathedral. We meet Thomas Alphonso Hayley as a boy, dressed as Puck, in a portrait by Romney. He later became Flaxman’s pupil, and returned the compliment to Romney by making a portrait medallion of him. Hayley also befriended Mrs Charlotte Smith, a Chichester woman who married at 15, had nine children in as many years, only to find that her husband then landed the family in a debtor’s prison. She supported herself and her progeny through writing, which was praised for its ‘true feeling for rural nature’ by Wordsworth.

Hayley rescued William Blake from the grind of an engraver’s life in south London by installing him at Felpham. There followed the only three years that Blake spent in the country, happily walking and communing with the spirit world that was so vivid to him. Hayley furnished him with commissions, such as one to illustrate his own Ballads… Founded on Anecdotes Relating to Animals: the exhibition shows Blake’s interpretation of a dog rescuing a man from a crocodile. Blake found Hayley’s well-intentioned patronage tedious. It got between him and his higher mission. Exasperation boiled over when he found a drunken soldier in his garden; having marched him down to the inn, Blake then had to defend himself against a charge of sedition that was instigated as an act of revenge. From this, he was eventually acquitted, thanks partly to the support of the villagers, who plainly regarded him as a respectable crackpot.

After Blake, visionary painting, reinforced by literature, retained its association with the area. Samuel Palmer, Edward Calvert and George Richmond established a brotherhood of Ancients at Shoreham in Kent, making tiny engravings that, in Palmer’s case, were often accompanied by quotations from Milton. In the 20th century, their work was an inspiration to Graham Sutherland and Paul Drury, who not only visited Shoreham, but took to wearing long cloaks in imitation of the Ancients. After the Second World War, literature again impregnated the landscapes of neo-Romantics such as John Piper and Michael Ayrton.

This is the perfect show for a gallery such as Pallant House: intimate, revealing, focused on place. As the train rumbled past Arundel on a gold kissed April evening, were they rabbits that I saw moving in the fields? Or angels?

‘Poets in the Landscape; the Romantic Spirit in British Art’ is at Pallant House Gallery, Chichester (01243 774557; www.pallant.org.uk) until June 10.