Deputy Prime Minister John Prescott has announced he is to give up his grace-and-favour mansion, Dorneywood.
He said he had made the personal decision because continued controversy over the residence was "getting in the way" of his role in Government.
But Tories said the public was not interested in where Mr Prescott lived but in the failings of the health service and benefits system, dismissing the move as "slightly trivial".
In a statement Mr Prescott said: "Like other Cabinet Ministers before me, I have used Dorneywood as a place to stay, to relax and to work.
"But I am well aware that my use of it is now a subject of public controversy and criticism and a matter of concern amongst some MPs and the Labour Party.
"I have accepted that my continued use of Dorneywood is getting in the way of doing my job in government.
"I have told the Prime Minister that it is my personal decision that I no longer want to be the official resident. He has accepted this decision."
Mr Prescott will hope the move will deflect criticism that began with the revelations of his affair with diary secretary Tracey Temple.
He was stripped of his giant department in Tony Blair's swingeing reshuffle on May 5 but retained his cabinet ministerial salary, and perks including Dorneywood and his flat in Admiralty House - the scene of his affair with diary secretary Ms Temple.
Mr Blair announced Mr Prescott was to chair nine Cabinet committees and deputises for Mr Blair when he is away.
The clamour for Mr Prescott to quit the estate increased at the weekend when he was photographed playing croquet at Dorneywood with staff.
The Conservatives have accused Mr Prescott of retaining perks while "seemingly doing very little".
At Prime Minister's questions last week Mr Blair was asked if he had asked Mr Prescott to give up Dorney-wood. Mr Blair replied then: "I have no intention whatever of discussing the reshuffle or matters associated with it."
Last night Labour MP Stephen Pound told BBC News it was a "dignified statement" but added: "I think he could have done it a week or so ago."
Mr Pound likened the photos of the croquet match to George Galloway's appearance in a red catsuit that left his reputation "utterly destroyed".
Labour MP Ian Gibson said he was "pleased" that Mr Prescott had made the decision to quit Dorneywood.
"I think this is a gesture towards recognising that this kind of lavish behaviour is unnecessary."
Conservative party chair-man Francis Maude claimed Mr Prescott is "deeply damaged goods now".
"This is a Government that's past its sell-by date. John Prescott is past his sell-by date some time ago."
He insisted though that where Mr Prescott lived was "slightly trivial" and not what matters to the public who were more concerned about the health service, the Home Office and tax credits.