The veil is a "mark of separation" which makes non-Muslims feel uncomfortable, Tony Blair said yesterday.
But it was merely part of the broader issue of the failure of some Muslims to integrate, he claimed.
"There is a minority where there is a problem," said the Prime Minister.
The comments, made at Mr Blair's monthly Downing Street press conference, came despite warnings from Birmingham MP Khalid Mahmood (Lab, Perry Barr) that politicians were fanning extremism by constantly criticising Muslims.
Mr Blair supported the Yorkshire school which has suspended a Muslim teaching assistant for refusing to remove her veil, telling journalists: "I back their handling of the case".
And he defended Cabinet Minister Jack Straw, the Leader of the Commons, who prompted the debate over veils by revealing he asked constituents to remove them when they attended his advice surgeries.
Echoing a phrase used by Mr Straw, the Prime Minister said: "It is a mark of separation and that is why it makes other people from outside the community uncomfortable."
He added: "There is a broader question and that is the general question of how we make sure the Muslim community integrates in British society."
He said there was an issue, which was apparent across Europe, about how Islam "comes to terms and is comfortable with" the modern world.
He said: "People want to know that the Muslim community in particular, but actually all minority communities, have got the balance right between integration and multi-culturalism."
The debate over veils, sparked by Mr Straw, heated up after Kirklees Council suspended Aishah Azmi, a teaching assistant at Headfield Church of England Junior School in Dewsbury, West Yorkshire.
Earlier this week, Mr Mahmood warned that politicians and the media were stoking divisions by focusing excessive attention on Muslims.
He said: "This is a debate which is polarising communities and the only people who benefit are extremists."
Both the BNP and Muslim radicals were benefiting, he added.
Mr Mahmood called for Muslim scholars to look at the question of veils.
Similar warnings about the effect of a constant spotlight on Muslims have come from John Denham, chairman of the Commons Home Affairs Committee, and Dr Mohammad Abdul Bari, secretary-general of the Muslim Council of Britain.
But Ms Azmi's MP, Shahid Malik, said ministers had been right to give their views and said it had resulted in helpful debate
In his press conference, Mr Blair also described the issue of faith schools as "difficult" for the Government.
Education Secretary Alan Johnson is to change the law to allow local councils to force new schools to accept up to a quarter of its pupils from other or no religions.