Black Country MP Ian Austin has joined the criticism of Cannock Chase MP Aidan Burley by claiming the Conservative Party had attempted to 'brush the scandal under the carpet'.
Tory MP Mr Burley attended a stag party in France where guests dressed as Nazis and made toasts to the Third Reich.
French prosecutors have opened an investigation into the party. Under French law, it is a crime to wear uniforms, insignia or emblems linked to the Nazi regime, unless they are being used for a film, play or historical exhibition.
Mr Burley has apologised for his behaviour and was removed from his post as a parliamentary private secretary (PPS) to Transport Secretary Justine Greening.
But Dudley North MP Mr Austin said the Prime Minister should go further:
"David Cameron and the Tories tried to brush this under the carpet, but the scandal surrounding Aidan Burley's disgraceful conduct is not going away," said Mr Austin.
"Surely the Prime Minister must take the whip off Mr Burley while this investigation is taking place."
In a message to constituents, Mr Burley voiced his "deepest regret", adding: "There are no excuses for my foolish behaviour which, over the past two weeks, has caused so much distress to so many people.
"I made a bad error of judgment and you deserved better from your local MP.
"Being involved in a stag party where an SS uniform was worn was wrong and offensive. It was the wrong decision on my part; crass and insensitive.
"I am deeply sorry, and want to take this opportunity to offer the people of Cannock Chase an unreserved, wholehearted and full apology for the terrible offence this incident has undoubtedly caused."
Mr Burley said his family had "been through hell" as a result of the publicity surrounding the party.
And he insisted: "I have no sympathies whatsoever with Nazism, racism, or fascism... I personally did not participate in any alcohol-fuelled attempted toasts by other guests to the Third Reich.
"Nor did I participate in any chants, offensive or otherwise. As the video showed, I left the restaurant immediately when that inexcusable behaviour by other guests started."
Mr Austin later told the BBC: "It's quite right that Aidan Burley apologised to the Jewish community last week, because they'll take particular offence at what happened, but for me it goes much wider than that.
"For me, what Britain did in the Second World War - defeating the Nazis, standing up to fascism alone when the rest of Europe was overrun - for me, that's our finest hour.
"It shows why we can claim a particular attachment to values like democracy and freedom and fairness. It's what makes us the country that we are.
"So, far from simply disgracing himself, I think he has actually undermined what it means to be British. That's why I think that we should all be taking this much more seriously."