Sir James Cleverly said he would not have compared a government minister to a Gestapo officer, after Kemi Badenoch’s comments about Education Secretary Bridget Phillipson.

The Conservative leader made the Nazi comparison while criticising the government’s decision to end a tax exemption for private schools, an insult which drew heavy criticism from Labour.

Appearing on the BBC’s Sunday with Laura Kuenssberg programme, Sir James, the shadow housing secretary, criticised Labour’s education record but when pressed on whether he would have made the remark himself, said: “No, I probably wouldn’t have done.”

Badenoch has not apologised for the comment, which Phillipson said showed she was “not fit to be prime minister”.

Badenoch was also highly critical of Phillipson at Prime Minister’s Questions earlier this week, describing her as a “spiteful class warrior”.

The Conservatives have accused Labour of targeting families with children at private schools with VAT to raise money for more teachers in state schools, while instead overseeing a fall in teacher numbers.

Badenoch told the Spectator earlier this month that the education secretary had “acted like a Gestapo officer” in doing so.

Asked about the insult, Sir James said: “The point is Bridget Phillipson has pursued an agenda which has nothing to do with raising money, nothing to do with recruiting teachers, it is about punishing people who spend their own money to send their children to private school.”

Pressed on whether he thought the language was appropriate, Sir James said: “So that wouldn’t have been the phraseology that I use, but you can see Kemi’s explanation there.”

Phillipson has herself been criticised by the Conservatives for labelling comments made by Tory MP Nick Timothy as “racist” and calling for him to be sacked. Timothy, who described a Muslim religious gathering in central London in March as an “act of domination”, denied the comment amounted to racism.

Sir James continued: “When Kemi highlights the fact that these policies are vindictive and they are about class war rather than recruiting teachers, she gets vilified by the Labour Party.

“And when people in the Labour Party, literally minutes earlier, describe a Conservative MP as being racist, no-one seems to bat an eyelid.”

Pressed on whether politicians had a duty to maintain a tone of respect in their exchanges, Sir James said: “We have a robust democracy and I think one of the dangers in a desire to be collegiate and calm is the voters think that we are not holding each other to account.

“So I do think we need to be robust and I do think when we see people, particularly Secretaries of State, particularly the government, doing things, not for the good of the country but to scratch a party political itch, I think it’s absolutely right that we call it out.”

Labour Party chairwoman Anna Turley said Badenoch’s shadow ministers “won’t defend the indefensible because they know full well Kemi Badenoch’s comments were completely appalling and deeply inappropriate”.

Sign up for our Politics Essential newsletter to read top political analysis, gain insight from across the UK and stay up to speed with the big moments. It’ll be delivered straight to your inbox every weekday.