The Government stood accused of an “outrageous stealth tax” after it emerged that more than four million pensioners will be worse off following changes to simplify the tax system unveiled in the Budget.

At the moment, people over the age of 65 are not taxed on the first £10,500 of their income, but this will be frozen for existing pensioners and from next April new pensioners will have a tax-free allowance of £9,205, bringing them into line with the general population.

Around half of pensioners will be affected by the changes, which will raise around £1 billion annually for the Government.

The average loss for pensioners has been put at £80 a year, rising to £197 for those retiring from next April.

Saga director-general Ros Altmann said: “This is an outrageous assault on decent middle-class pensioners.

“This Budget contains an enormous stealth tax for older people. Over the next five years, pensioners with an income of between £10,500 and £24,000 will be paying an extra £3 billion in tax while richer pensioners are left unaffected.

“There is nothing in this Budget for savers, there is nothing to improve the annuity market, nothing to appease the damage of quantitative easing and nothing to support Isa changes and shelter older people’s money in cash.

“This Budget is terrible news for pensioners.”

The move will save the Treasury £360 million next year, £670 million in 2014, more than £1 billion in 2015 and £1.25 billion in 2016.

A statement from the Treasury said: “We are taking a decision that the best way to help pensioners is through the triple lock, guaranteeing increases in the state pension.

“There is a goal of a simpler tax system and a simpler personal allowance system.”

Chancellor George Osborne said the current age-related allowances were a “particularly complicated” feature of the tax system, which many pensioners do not understand, requiring around 150,000 pensioners to fill in self-assessment forms.

He said: “So over time we will simplify the tax system for pensioners by doing away with the complexity of the additional age-related allowances for anyone reaching the age of 65 on or after April 6 2013 and I will freeze the cash value of the allowance for existing pensioners until it aligns with the personal allowance.

“This will protect the existing level of allowance pensioners have, while introducing a single personal allowance for all.”

But TUC general secretary Brendan Barber said: “The Chancellor’s decision to raise more than a billion extra pounds in tax from pensioners by freezing age allowances will come back to haunt him. It’s already being dubbed ‘the granny tax’.

“Pretending that pensioners will be grateful because it will simplify their tax is a vain hope. Instead they will see that they are being asked to pay more while the super-rich have kept all the pensions tax relief - a heavy burden for ordinary taxpayers.

“And with ever more increases in the state pension age threatened in a new White Paper - a burden that falls more heavily on the poor, the regions and those with manual jobs - older workers and pensioners are paying a heavy price for a pro-rich Budget.”

Joanne Segars, chief executive of the National Association of Pension Funds (NAPF), said pensioners were the “biggest losers” in the Budget.

She said: “This will come as a blow to millions of pensioners who have paid into the tax system throughout their working lives.”

But she welcomed the Government’s plans to introduce a simple, single-tier state pension of around £140, based on contributions.

The Chancellor confirmed the introduction of a single-tier pension for future pensioners, which he said would cost no more than the current system in any year and represented “a further major simplification of our tax and benefit system”.

The plans come ahead of a huge overhaul of the pensions system with the automatic enrolment of up to 10 million workers in schemes from this autumn.