Sir Keir Starmer achieved what many sage minds in Westminster believed was impossible.

Labour smashed itself to bits in 2019. But Sir Keir put the party back together and swept back to power with an epic majority five years later.

Yet, turn on the radio or glance at the headlines and Labour appears to be flirting with the unthinkable – giving him his P45.

In the last few days I’ve spoken to 30 people across government and the party – ministers, MPs, advisers – to try to work out, as the prime minister makes his way to Labour’s annual party conference in Liverpool, how much trouble is he really in?

“How many more data points do you need to see that he is just not very good at being prime minister?”

Unforgiving as that sounds, the Whitehall figure who posed that question is far from the only person who has seen Starmer’s government up close to conclude that fundamentally, the prime minister is just not suited to many aspects of the job.

For months polling has suggested the public feels the same.

Sir Keir is an intelligent, incredibly diligent, serious-minded politician who has achieved extraordinary things. But time and again the same problems are raised, summed up brutally by another senior figure: “His judgement on people has proven to be flawed, see endless staffing resets.”

“His judgement on policy is flawed, hence U-turns, and he can’t communicate, and has been unpopular in the public’s eyes for a long time.”

Others point to a slow pace of decision making in Downing Street, suggesting the prime minister thinks “too like a chairman, not a chief executive,” says another insider.

And “he only gets to the right answer having exhausted all the other options,” rather than moving fast using a gut political instinct.

Another senior party figure says: “He doesn’t think like a leader.”

Everyone in the Labour party is fed up with months of shocking polls.

But others are fed up of colleagues complaining about the boss, whether the small number who do so in public, or the much higher number who do so in private.

“It’s like being England manager,” one cabinet minister jokes. “Everyone thinks they can do a better job.”

You can hear the frustration in others’ voices, highlighting the “outstanding job” Starmer has done on the world stage – wrangling President Donald Trump, or leading support for Ukraine.

“He hasn’t put a foot wrong,” when it comes to foreign policy, says a cabinet minister.

Another praises the “sensible changes” the prime minister has just made to Downing Street, saying “people now are now in jobs that will suit their abilities better” after the reshuffle, and promising the public will soon see the benefit.

There can be no hiding from a very messy few weeks, with the forced exits of Angela Rayner and Lord Mandelson, and the circus around the Mayor of Greater Manchester Andy Burnham’s not very hidden ambition.

But there are other believers in what they see as Starmer’s steady and relentless leadership, as one party source jokes: “There were a couple of weeks where we couldn’t get of bed without accidentally tearing our own arms off.”

They predict “that could all go away if we have a decent couple of months”.

Perhaps. What even some of the prime minister’s staunchest allies believe is that, he must explain more clearly, and vigorously, to the party – and to all of us – what the government is actually doing and why.

Sir Keir himself admitted as much in the summer, saying: “We haven’t always told our story as well as we should.”

So in the next few days, and live on the programme tomorrow, expect the prime minister to try hard to make a clearer pitch to the country. After a horrible few weeks, aides say the prime minister is in a “better place, clearer about the argument he wants to make”.

Announcements like the Hillsborough Law, the returns deal with France, and yesterday’s confirmation that ministers will go ahead with digital ID cards will convey, No 10 hopes, a sense of momentum.

And you can expect him to build on the argument he’s been making about patriotism in politics, trying to reclaim some of the territory Reform UK have been trying to grab.

There are two problems though.

Firstly, even Starmer’s biggest fans in government know fluid communication is not his strong point. He doesn’t see the camera, or the TV sofa as his friend.

A strong defender of Starmer tells me: “I wish he was better at telling a story, people don’t understand how deeply he feels it and how passionate he is, but he personally isn’t good at it.”

Secondly, and according to one senior MP this is the bigger issue: “We’re just not clear about what we are doing or who we are for.”

Another staffer tells me: “There is no fire, no passion, no vision – we are timid and flat-footed.”

Even if the prime minister suddenly developed the silky charm and persuasiveness of George Clooney, or the bombastic nature of Piers Morgan, the question becomes, what is the story he wants to tell?

Multiple exits from the No 10 communications team tell you that working this out is not an easy task.

One source says: “Until we know the story we are trying to tell we will get buffeted around.”

Is reforming schools, hospitals or welfare at the top of the list? Building houses or becoming a green “superpower”? Or being at the forefront of technology, or protecting our borders, or growing the economy? Or is it now this priority of “patriotic renewal”?

If you have that many priorities, how can the public understand what you care about most? To govern is to choose after all, if you’re not doing much choosing, are you doing much governing?

One source suggested frankly, even for those who work closely alongside him, it’s often hard to work out what the prime minister thinks, telling me: “He often appears unknowable.”

“He compartmentalises his life – personal, professional, and political – in a way that works for him, but leaves colleagues unsure of his true views.”

In a sense it seems extraordinary that the leader of a political party can leave colleagues unsure of their views, after 14 months in office, and four years as leader in opposition.

But his early years in charge were dominated by internal fights for control of the party, and over antisemitism, not over ideas.

One source recalls: “We had lots of fights to win in opposition, but they were never, ever about what the parties’ priorities should be.”

The prime minister has a huge chance in the next few days to spell out to his party, and to the public, exactly where he stands, and exactly what he believes in specific terms.

After a horrible summer, and a very tricky year, it is desperately needed if the party is to get back on track.

“There is a way through this, but he has to assert himself,” says one cabinet minister, desperate for the prime minister to remind his party and the country why they elected him in the first place.

Ministers and Downing Street are frustrated by chatter about Starmer’s leadership, especially the behaviour of Andy Burnahm in recent days.

“Everyone needs to just shut up and get on with their own jobs,” one government source says.

Another says: “the Burnham stuff is just ridiculous.”

“He doesn’t have a seat in Parliament, wouldn’t necessarily find or win one,” and one minister snipes, “any chance to have a dig – Burnham will never miss an opportunity”.

They predict confidently, the party “doesn’t want regicide or disruption, they just want us to do well”.

But leadership chatter is a symbol that all is not well. And that sense of assurance is far from universal.

One senior figure outside government warns me, “they don’t realise they are in serious, serious trouble – this is literally a head-in-hands-moment”.

There are real discussions in different pockets of the Labour movement about whether Starmer should still lead the party.

And the sweaty bars and fringe meetings of the party conference provide the perfect backdrop for those out to make trouble.

Yet it’s clear from multiple conversations that it’s not only mischief makers who are pondering what to do.

“Frustration is wide and deep,” says a senior MP.

One insider who has been part of conversations about the chances of changing the leader says, “it’s not flighty people, not just newbies without a clue, they’re not idiots”.

Even on the government payroll one minister worries, “the country is desperate for a disrupter, and it’s just not who he is,” questioning whether Starmer can be the right man for this political moment.

Here’s the fundamental question, asked by many, expressed by one senior figure: “Do we think Keir can improve? People don’t think he can, although they wish he would. But there is a sinking feeling.”

Sign up for the Off Air with Laura K newsletter to get Laura Kuenssberg’s expert insight and insider stories every week, emailed directly to you.

Some MPs are even pointing to the example of the Democrats’ failure to tell former President Joe Biden to go, which left the party no time to build up another candidate.

The source continues we “dread and fear being the people who didn’t do what was necessary to stop the country electing Farage”.

Others would baulk at that comparison. The prime minister has only had just over a year in charge. The problems Labour inherited on taking office were profound.

Centre left political parties in many countries are grappling with how to deal with demographics, immigration, an era where public money is very tight.

There are more than three years until the next general election. There is time to start wrestling with all of that.

The electorate has shown politicians they can move in droves, and fast. So many ministers plead for patience.

One wishes his colleagues would not just “get a grip”, but remember they were all elected on a promise to stop political chaos.

“Anyone who thinks the challenging world is rectified by a change in leader, I don’t understand the rationale. When we look at how unstable things are, and talk about leadership, how would the bond markets respond, when our primary responsibility is stabilising the economy?”

Another source dismisses those predicting doom in the party, “nothing is irrevocable, and nothing is inevitable”.

Perhaps the rebooted No 10 will start getting better results.

Perhaps, and No 10 fervently hopes, Sir Keir will wow the crowds at conference, and display why he earned the job to start with, and won the election in the first place.

The prime minister’s position is also, frankly, protected by the fact there is no consensus about who could do a better job than him.

Even if Labour were to conclude universally, which it is far from doing, that Starmer’s leadership is somehow beyond repair, that reality is a smack in the face for those who’d seek a change.

As one Whitehall figure suggests: “problem one, is he good at being PM? No. But next problem, do you have an upgrade? Hmm…”

Labour’s answer to that question right now, notwithstanding Andy Burnham’s antics, is, “well, we’re not sure”.

A lack of clarity over who would be next, and fresh memories of the political shambles of the Conservatives’ change of leader, both put the brakes on anyone seeking to make a change any time soon.

As ever in politics two things that might appear to be contradictory are in fact, both true.

Is it mad on paper for Labour to be even vaguely contemplating changing their leader so soon, without an obvious strong successor, when they ran on a promise of no drama? For many, yes, it’s crackers.

But is the discussion about whether the PM can carry on like this just idle chit-chat? Not at all.

One senior figure sums up the situation with no pleasure – discussion about changing the Labour leader “is both lunatic, and real”.

The leadership’s hope is that a strong conference without too many noises off, and an effective speech from the platform by the prime minister himself might calm everything down.

A political get together, a party conference for the politicians, members, and associated hangers-on, would never transform a prime minister’s fortunes with the public overnight. But it can make a big difference to how a party feels about itself.

For Starmer to have a chance of restoring his position with the public he must first, once again, persuade his party that he is the man for the job.

But without restoring his party’s faith in him, convincing Labour of his merits, his own position could well be at risk.

One insider who knows the prime minister well suggested: “He is not a natural explainer… he states positions but doesn’t repeatedly argue for them.

“This has led to an aversion to debate, treating persuasion as weakness.”

Yet politics is, above all else, the art of persuasion. That is perhaps the dilemma for Labour. That is the problem for him.

BBC InDepth is the home on the website and app for the best analysis, with fresh perspectives that challenge assumptions and deep reporting on the biggest issues of the day. And we showcase thought-provoking content from across BBC Sounds and iPlayer too. You can send us your feedback on the InDepth section by clicking on the button below.