The leader of the Unite union says Labour is not defending working people and they are turning away from the party “in droves”.

Sharon Graham said Labour should be “seriously concerned” after the union voted to potentially rethink its relationship with the party, which could result in it formally cutting ties and funding.

It comes after Unite said it had suspended the membership of Deputy Prime Minister Angela Rayner over her handling of bin strikes in Birmingham. A Labour source said Rayner quit Unite in April and defended her action on workers’ rights.

A party spokesperson said the Labour government was responsible for the “biggest upgrade in workers’ rights in a generation”.

Delegates at Unite’s policy conference voted to rethink their relationship with Labour should any of its members be made redundant in the course of the long-running bin strike.

The vote also saw the union decide to suspend Rayner over her role in the strikes.

The deputy prime minister has urged workers to accept a deal tabled by Birmingham’s Labour-run city council to end the dispute, saying the authority had “moved significantly to meet the demands of the workers”.

Speaking on BBC Radio 4’s Today programme on Saturday, Unite’s general secretary said Unite members “don’t believe that Labour defends workers in the way we thought they would”.

Rayner was attempting a “Houdini act” by focusing on “whether she wasn’t or was a member at this juncture,” she said, and Labour should instead be asking where it was “going wrong”.

Ms Graham said its members reflected “what everyday people are saying” about the government.

“I have real difficulty in the way that Labour are making decisions,” she said, “in terms of what they tried to do on winter fuel, what they tried to do to people with disabilities, what they’re doing to workers”.

The government faced major political pressure over its planned cuts to winter fuel payments and welfare, including from the left of the Labour party – which were subsequently reversed and significantly watered-down.

Unite is one of a number of unions which are affiliated with Labour – giving it seats on the party’s ruling national executive committee and delegates to its annual conference.

It is also Labour’s biggest union funder through the affiliation fees that members pay to the party – currently totalling £1.2m a year.

Ms Graham said disaffiliation was a possibility and that she was under pressure to call an emergency rules conference – where a decision about disaffiliation could be made.

Members needed to see that affiliation was “worth something,” she said.

“At this present moment in time, it is hard to justify it, if I’m being honest. Would that money be better spent on frontline services for my members?”

She said access to political power was useful but not “if you’re walking into a room and that political power keeps saying, ‘computer says no’.”

The BBC understands Rayner stopped paying for her Unite membership in April. On Friday, a Labour source called her suspension a “silly stunt”.

Unite members walked out in January over plans to downgrade some roles as part of the city council’s attempts to sort out its equal pay liabilities.

Unite has also urged the council to guarantee long-term pay for Grade 4 bin lorry drivers, claiming in April that bin lorry drivers’ pay could fall from £40,000 to £32,000 under new council plans.

An all-out indefinite strike was announced in March, and a deal to end industrial action has not yet been reached.

Conciliation service Acas has been mediating in the negotiations since May, but talks broke down on Wednesday. The council’s Labour leader John Cotton said the authority had “reached the absolute limit of what we can offer”.

Unite said on Friday it had also suspended Cotton’s union membership. He told the BBC that he was a “proud, lifelong trade unionist” determined to “end the scandal of gender pay inequality”.

On Friday, Birmingham Edgbaston MP Preet Kaur Gill said the union did not need to “get involved” as a “fair deal” was on the table.

She told the BBC Radio 4’s The World Tonight that Ms Graham “should have felt confident in her local officers that were negotiating” and they were “getting close to a deal before she got involved”.

A Labour Party spokesperson said: “The Labour government has introduced the biggest upgrade in workers’ rights in a generation to address low pay, insecure work, and poor working conditions, which will benefit 15 million workers across the country.”

They added: “Only Labour is delivering the change working people voted for and so deserve.”