After criticism of the decision, the chairman Sean Kennedy along with John Grafton, Duncan Macpherson and Willie MacKay have resigned from the committee.
Independent councillor MacKay has also resigned as a councillor, while Grafton has been suspended by the Scottish Liberal Democrat group on Highland Council.
Kennedy, an Independent councillor, has resigned from the Highland Independents group on the local authority.
The sixth councillor, Scottish Conservative Ruraidh Stewart, has been approached for comment.
A Scottish Conservative spokesperson said: “We do not support the decision taken by the licensing committee.
“Whatever the extenuating circumstances behind it, this vote has caused understandable anger and it would be appropriate for it to be reviewed.”
Brown’s taxi driver’s licence – which allows him to drive – had been suspended in January 2024 after details of his offences emerged.
His separate operator’s licence allows his vehicle to be used as a taxi business.
The committee’s decision is to be reviewed by a meeting of the full council.
