Newly-appointed shadow housing secretary Sir James Cleverly has accused Sir Keir Starmer of fuelling public frustration over the housing of asylum seekers.

Sir James told BBC Radio 4 the prime minister had “amplified” the situation by saying on Monday there was “lots of housing available” for the rising numbers of homeless people and asylum seekers.

“It’s that kind of disconnect that I think is driving real frustration and that’s what I want to address,” the Conservative MP told the Today programme.

Labour, which has pledged to deliver 1.5 million homes, said the country was still “living with the consequences” of the Tories’ “disastrous decision to abolish mandatory housing targets”.

A Labour spokesperson added: “While Labour is working in partnership with regions to turn the tide on the acute and entrenched housing crisis, the Conservatives haven’t changed and they haven’t once apologised for the mess they left behind.”

Sir James kicked off his new job with a visit to a housing project in Hillingdon, north London, with Tory leader Kemi Badenoch, who handed him the role on Tuesday in a reshuffle of her top team.

The former foreign secretary and home secretary has been on the backbenches since losing out to Badenoch in last year’s Tory leadership contest.

His new role makes him the opposition counterpart to Angela Rayner in her housing, communities and local government brief, but not in her deputy prime minister post.

Sir James told the Today programme: “We’ve got a government that made big promises when it came to housebuilding and is spectacularly failing to deliver on those promises and that is generating a lot of frustration, particularly with young people waiting to get on the housing ladder.”

He said that was “amplified by the prime minister sitting at the liaison committee claiming there are plenty of spare houses for asylum seekers, while people are struggling to get on the housing ladder”.

He was referring to comments made by the prime minister on Monday, when he was being grilled by the liaison committee, which is made up of select committee chairs.

Labour MP Dame Meg Hillier, who chairs the Treasury committee, asked the PM about where the government was going to house homeless people, given that the price of temporary accommodation had been driven up by the need to house asylum seekers.

He replied: “Oh, there is lots of housing and many local authorities that can be used, and we’re identifying where it can be used.”

Pressed for specific examples, Sir Keir said he would write to the committee.

The government has said it wants to work in partnership with councils and in June Home Office minster Angela Eagle said it was looking to buy tower blocks and former student accommodation to house migrants as an alternative to hotels.

In his Today interview, Sir James was asked about demonstrations outside hotels believed to be housing asylum seekers.

There has been a series of protests outside the Bell Hotel in Epping, Essex, since an asylum seeker was charged with sexual assault.

“I understand the desire to protest,” he said.

“There is never any excuse for rioting and we are seeing that agitators, both on the left and the right, are descending upon these communities to try and stir things up and there’s never an excuse for that, but I do understand why local people are frustrated.”

Sir James was also asked for his view on leaving the European Convention on Human Rights (ECHR), after Badenoch launched a review to examine the issue.

He would not say whether, like Badenoch has said, he was also “increasingly of the view” that the UK should leave the international human rights treaty.

He told the BBC this would “not necessarily be a silver bullet” but that if the review leads to it becoming party policy he would abide by that.

Badenoch echoed his point, telling reporters: “I wouldn’t bring someone into the shadow cabinet if they didn’t agree with me.”