Portside

Stephen Sondheim achieved such acclaim – for deepening the content and extending the lyrical ingenuity of musical theatre – that, from the age of 50, each major birthday was celebrated with tribute concerts in London, New York or both.

Watching the composer-lyricist of Sweeney Todd and Follies at such events – taking a bow, with his wry smile – it was impossible not to reflect on our luck in coinciding with the life of someone who would clearly stand in the history of the genre alongside such geniuses as Cole Porter, Irving Berlin, George and Ira Gershwin, Leonard Bernstein, Kurt Weill, Richard Rogers and Oscar Hammerstein.

Sondheim had an unusually direct connection to this creative lineage. As an 11-year-old, living in Pennsylvania as the only child of a divorced and distant mother, he became friends at prep school with James Hammerstein, son of the celebrated librettist.

Oscar became a surrogate father to Sondheim, who often joked that, if Jamie’s dad had been a plumber, he would have spent his own life fixing pipes. But this suggestion of helpless influence overlooks the fact that the young Sondheim, a piano prodigy, had already written a musical – By George, a satirical history of his school – which encouraged the Broadway legend to mentor him.

Hammerstein gave his protege tutorials on rhyme, characterisation and storytelling, and set a demanding piece of homework: to write four musicals – one adapted from a good play, one from a not so good play, one from a non-dramatic source, and one from an original idea.

These can be seen as the templates for Sondheim’s mature output. Three shows – West Side Story (1957), A Funny Thing Happened on the Way to the Forum (1962) and The Frogs (1974) – are based on good plays, by Shakespeare, Plautus and Aristophanes respectively. Saturday Night (1954), Do I Hear a Waltz? (1965), Company (1970), Sweeney Todd (1979), and Merrily We Roll Along (1981) were adapted from lesser theatrical texts.

Non-dramatic sources that he musicalised were the memoirs of the stripper Gypsy Rose Lee, which became Gypsy (1959); the fairytales by Grimm and others that encouraged Into the Woods (1986); and movies (by Ingmar Bergman and Ettore Scola) that inspired A Little Night Music (1973) and Passion (1994).

Original ideas triggered Anyone Can Whistle (1964), Follies (1971), Pacific Overtures (1976), Sunday in the Park With George (1984), and Assassins (1990). The subject matters of those free-floating shows were: small-town politics; old variety performers suffering marital and mental breakdowns; the Japanese-American trade wars of the mid 19th-century; the painting of a Pointilliste masterpiece; and the subset of Americans who have killed or tried to kill a US president.

For some critics, the improbability of such material as musical theatre narrative showed Sondheim was artistically killing the father-figure of Hammerstein, who was widely seen as a cosy and conservative figure. Sondheim, though, identified Hammerstein’s Show Boat, Oklahoma!, Carousel and South Pacific as hugely radical for the time, moving American musical theatre on from soppy operetta to deal with real people in serious situations.

The 17-year-old Sondheim also worked as what he called “a $25-a-week gofer” on Rodgers and Hammerstein’s 1947 show Allegro, an experimental piece in which a man looks back at his life. It was in conscious homage to Hammerstein that he later used a similar psychological song-cycle form for Company, while Pacific Overtures contains thematic nods to South Pacific, which also dealt with American-Asian military and racial tension. Sondheim always remained true to Hammerstein’s vision that a song within a musical was “a one-act play which either intensifies a moment or moves the story forward”.

Where he most departed from the traditional Broadway model was in desiring to write both music and lyrics, rather than the more common collaborative division between composer at the piano and librettist with a notepad.

Sondheim tended to bracket as lesser works West Side Story and Gypsy, which had music by Leonard Bernstein (whom he loved) and Jule Styne (whom he didn’t). He had hoped to write the songs for Gypsy on his own, but its star, Ethel Merman, felt that the young man was too modernist to provide the lush orchestral score she wanted. Sondheim considered A Funny Thing Happened on the Way to the Forum, his first show in charge of both halves of the song, as the true start of his career and, apart from brief sentimentally-inspired reunions with Bernstein and Rodgers, never again wrote words for another composer.

“I’ve never found anybody I could work with as quickly as myself, or with less argument,” Sondheim told the New York Times in 1966, a joke that was deadly serious.

Yet Sondheim described himself as “naturally a collaborative animal”, and freely admitted that only Sweeney Todd and Passion were entirely his own ideas – or, the word he preferred, “notions”. The rest came from the writers of the “book” (musical theatre’s term for the narrative and dialogue). The most frequent of these were George Furth, John Weidman, Hugh Wheeler and James Lapine. They provided storyline, structure and thematic ideas; Sondheim’s independence came in the insistence that the stuff that was sung would be only his.

This encouraged the view, which he never quite eluded, that Sondheim’s lyrics were greater than his music. To make the comparison at all ignored Sondheim’s rationale for doing both jobs – that the two elements are complementary and cross-influential.

His lyrics are extravagantly clever, packed with rhymes that give no quarter to Cole Porter. In Follies, he offers in one scene alternative numbers for the performer playing Phyllis, depending on the level of dance skills the actress has. The first song, The Ballad of Lucy and Jessie, reveals the divided personality of the character through the story of two women who crave each other’s traits. The rhyming and rhythm are thrillingly slick: “Lucy wants to be dressy / Jessie wants to be juicy / Lucy wants to be Jessie / And Jessie Lucy / You see.” However, the alternate version, Ah, But Underneath, is even more of a tongue-twister, with a quatrain that builds to a climactic triple rhyme: “In the depths of her interior / Were fears she was inferior / And something even eerier / But no-one dared to query her superior exterior.”

In an anti-love song from Company, a troubled couple reflect on their relationship in a verse that, unexpectedly, has the joke rhymes falling internally rather than climactically. It is, they reflect: “The concerts you enjoy together / Neighbours you annoy together / Children you destroy together / That keep marriage intact.”

The way in which, in that lyric, the three examples of togetherness rise through the cosy via the funny to that dark parental punchline typifies the comedic sensibility central to Sondheim’s writing. His jokes can be very dark, most notoriously in A Little Priest, from Sweeney Todd, where the murderous barber and pie shop owner Mrs Lovett discuss the various professional flavours that his victims might lend her pastries.

Someone in a Tree, from Pacific Overtures, most spectacularly achieves the theory of a song as a compressed play. This Act One finale quartet, which Sondheim identified as his favourite among his compositions, moves fluidly between three timescales. Two old men and the childhood version of one of them remember witnessing, from a high branch, the signing of an American-Japanese treaty in 1853.

In the haiku-like lines that Sondheim uses throughout the libretto, one veteran remembers overhearing shreds of negotiation: “‘But we want…’ / ‘No, you can’t / And we won’t…’ / ‘But we need it, / And we want…’ / ‘Will you grant–?’ / ‘If you don’t…’ / ‘We concede it…’” Those 25 words, in which one voice channels two others, supremely demonstrate Sondheim’s often-stated desire to bring theatrical lyrics as close to natural speech as possible.

But Sondheim’s words would not have been so memorable without his compositional ability and agility. The thunderous opening chorus of Sweeney Todd, its instruments including a piercing factory whistle, is one of the most exciting and terrifying theatrical soundscapes ever achieved.

And, refuting those naysayers who claimed that Sondheim couldn’t write hummable numbers, Send in the Clowns has been recorded more than 500 times by different artists.

As a composer, Sondheim was the musical equivalent of Phyllis in Follies, who – reports the lyric of Ah, But Underneath – could be “sly, wry, cool, or coy, or candid”. He was equally adept at harmonies, ballads, choral numbers, love songs, laments and comic ditties, and had an ability to pastiche any musical form, whether operetta, opera, patter songs or military marches.

Such protean fluency as a composer was useful because so many numbers are cut – or urgently inserted – during the frequently brutal preview period before a musical opens. Sondheim was so prolific in the cause of making a show work that a CD, Unsung Sondheim (1993), and a sheet music book, The Almost Unknown Sondheim (2016), featured more than 50 songs which, though thrown away on the road to a Broadway opening, sounded anything but throwaway.

Sondheim often asked more of audiences than they were prepared to give. He performed the score of Anyone Can Whistle to 33 different groups of potential backers before the budget was raised for a premiere that ran for only nine performances; the premiere of Merrily We Roll Along closed after 16 shows. Almost none of his musicals went into profit on their first production.

Crucially, though, he was not ignoring public taste so much as slowly shaping it. Works that had once sat uneasily on Broadway or in London’s West End won over ticket-buyers and critics with subsequent productions. This revisionism was helped by a series of reputation-changing revivals in the less commercially brutal environment of British subsidised or off-West End theatre.

They included a 2017 Follies at the National Theatre, and stagings of Sunday in the Park with George, A Little Night Music and Assassins at London’s Menier Chocolate Factory in 2005, 2008 and 2014; and of Sweeney Todd and Gypsy, both starring Imelda Staunton, at the Chichester festival theatre in 2011 and 2014. In the first decades of the 21st century, there were important New York revivals of Sweeney Todd, Gypsy, and Passion.

Although honoured that his Broadway shows also graduated to opera houses, he refused offers to create original work for such elite spaces. He was discouraged by the handful of performances given to new operas, and the medium’s resistance to rewriting a work in the light of audience reaction.

Revivals of his work helped to shatter the idea that Sondheim’s songs had greater intellectual than emotional intelligence. Losing My Mind, from Follies, when sung by Staunton at the National, was revealed to be a clinically precise account of a nervous breakdown that happened to be written in the form of a torch song. In the same production, Philip Quast, as arrogant diplomat Ben, so convincingly broke down in his final number, Live, Laugh, Love that some in the audience believed the actor was unwell or had forgotten his lines.

Sondheim published, in two volumes, what is probably the best autobiography by a major creative talent, Finishing the Hat: Collected Lyrics (1954-1981) with Attendant Comments, Principles, Heresies, Grudges, Whines and Anecdotes and Look, I Made a Hat: Collected Lyrics (1981–2011) with Attendant Comments, Amplifications, Dogmas, Harangues, Digressions, Anecdotes and Miscellany. The memoir included Sondheim’s lyrical mission statement, declaring that four principles underlay “everything I’ve ever written”. These were: “Content Dictates Form, Less is More, God is in the Details – all in the service of Clarity.”

The books fascinatingly revealed his perfectionism. For a single line in Sweeney Todd, he pored over numerous maps of London seeking a two-syllable street name, in which a Victorian judge might plausibly have lived, that was stressed on its first rather than second syllable. He admitted to “blushing” over rare imperfect rhymes in early shows, and latterly would only accept “perfect” (or true) rhymes, in which the sounds of the final syllables match (home / roam, convey / dismay), rather than near or false rhymes.

Made distrustful of love first by his parents’ failed marriage and then by being gay in a long historical period of homophobia, Sondheim was often considered shy and solitary. Many singers of Send in the Clowns have concluded that its explanation of isolation was autobiographical, while Barcelona, a duet for reluctant lovers in Company, and the refrain “artists are bizarre, fixed, cold”, in Sunday in the Park with George, can reasonably be seen as confessional. However, in maturity, Sondheim sustained two successive relationships with younger men: first with the dramatist Peter Jones and then the digital technologist Jeff Romley. He seemed to have achieved in his private life an ease, personability and generosity that he had always suggested in interviews.

In his final decade, Sondheim worked with playwright David Ives on a musical, intended for New York’s Public theatre, based on two films by the Spanish director Luis Buñuel. A section was previewed, to acclaim, in 2017. Sondheim’s theatrical immortality, though, lay in his back catalogue and his influence on others. After Lin Manuel Miranda translated some Sondheim lyrics into Spanish for a 2009 Broadway revival of West Side Story, the older composer became the sort of admiring mentor to Miranda that Hammerstein had been to Sondheim.

Sweeney Todd, A Little Night Music, Pacific Overtures and Passion seem certain to be fixtures in the theatrical and operatic repertoires, while Company, Follies and A Funny Thing Happened can expect regular Broadway and West End revivals, as can two classics that Sondheim regarded as only half his: West Side Story and Gypsy. Merrily We Roll Along is likely to be heard less often, but contains a lyric of huge significance. One character, Franklin Shepard, is a Broadway composer whose adult failures and disillusion are seen to contradict his youthful belief that: “But for those who dare / the world is there / to change.”

In 1981, after the 16th and final Broadway performance seemed to confirm Sondheim’s reputation as a minority taste, Stephen Sondheim might reasonably have felt that he, like Shepard, had failed in his artistic aims. Now, though, it’s clear that Sondheim not only dared to change musical theatre, but succeeded, forever. His work simply had to wait to rhyme with his time.

[Mark Lawson is a Guardian writer and broadcaster]

We have a small favour to ask. Tens of millions have placed their trust in the Guardian’s high-impact journalism since we started publishing 200 years ago, turning to us in moments of crisis, uncertainty, solidarity and hope. More than 1.5 million readers, from 180 countries, have recently taken the step to support us financially – keeping us open to all, and fiercely independent.

With no shareholders or billionaire owner, we can set our own agenda and provide trustworthy journalism that’s free from commercial and political influence, offering a counterweight to the spread of misinformation. When it’s never mattered more, we can investigate and challenge without fear or favour. We couldn’t do this without readers like you.

Ahead of what is certain to be a pivotal year for America, we’re raising $1.25m to fund our reporting in 2022. If you can, please consider making a year-end gift so that we can continue to deliver journalism that sheds a light on injustice and drives meaningful change. It is reader support that makes our high-impact journalism possible and gives us the emotional support and motor energy to keep doing journalism that matters.

Unlike many others, Guardian journalism is available for everyone to read, regardless of what they can afford to pay. We do this because we believe in information equality.

We aim to offer readers a comprehensive, international perspective on critical events shaping our world – from the crackdown on voting rights to corrosive racial inequality and the erosion of democratic norms. We are committed to upholding our reputation for urgent, powerful reporting on the climate emergency, and made the decision to reject advertising from fossil fuel companies, divest from the oil and gas industries, and set a course to achieve net zero emissions by 2030.

Every contribution, however big or small, powers our journalism and sustains our future. Support the Guardian from as little as $1 – it only takes a minute. Thank you.