A Labour government would scrap the coalition's "bedroom tax", Ed Miliband is to announce ahead of the party's annual conference.

In an effort to regain the initiative after a string of revelations about the bitter in-fighting of the Blair/Brown years, Mr Miliband will condemn the controversial measure as "a symbol of an out-of-touch, uncaring Tory Government".

Labour insists the £470 million expected to be saved by the welfare reform could instead be covered by closing "shady" tax loopholes. On the eve of his party's conference in Brighton, Mr Miliband will say the "most important issue facing families in Britain" is "the cost of living crisis".

He will say: "Under David Cameron life is getting harder and harder, with prices rising faster than wages in 38 of the 39 months that he has been in Downing Street, and working people are an average of almost £1,500 a year worse off under his Government.

"But we have a Tory-led Government which listens only to a privileged few. Tax cuts for millionaires and tax breaks for hedge funds." He will say the "bedroom tax" was the product of a Government that "stands up for the privileged few - but never for you".

Mr Miliband will say: "We'll scrap the bedroom tax by abolishing the shady schemes of tax loopholes for the privileged few which the Tories keep inventing. Tax cuts for hedge funds, the billion pound black hole created with a scheme for workers to sell their rights for shares, and by tackling scams which cheat the taxpayer in construction."

The announcement came as Labour feared their annual gathering could be overshadowed by the publication of Gordon Brown's spin doctor's memoirs. Former special adviser Damian McBride confessed to leaking stories to the press about former home secretaries John Reid and Charles Clarke at a time when he was determined to ensure Mr Brown succeeded Tony Blair as prime minister.

The Tories claimed Labour's sums did not add up and the party would need to increase borrowing to cover the cost of the "bedroom tax" policy. The Government was already cracking down on the use of intermediary firms to dodge tax, and scrapping the shares for rights scheme would yield nothing in 2015/16 and just £5 million in 2016/17, the Conservatives claimed. They argued that pension funds, rather than hedge funds, would be hit by reinstating the stamp duty reserve tax charge.

Treasury Minister Sajid Javid said: "Labour's first policy commitment, after three years of waiting, is more spending on housing benefit, funded by a tax on pensions and more borrowing. That sums up Labour's record in office and shows it's still the same old Labour. Despite promising 'discipline' on borrowing, Ed Miliband has shown he is too weak to deliver."

But Grainia Long, chief executive of the Chartered Institute of Housing, backed the Labour announcement, saying: "The bedroom tax is not fair - it has become clear since its introduction that this policy is causing hardship and suffering for thousands of people across Great Britain, as many in the housing industry warned it would."