Franklin House in Bournville - reputedly where the Curly Wurly was invented - will be transformed into an apartment block with up to 96 homes, a garden terrace and a coffee house in a £16 million redevelopment.
It becomes the second scheme funded solely by Chinese cash and will be delivered through a joint venture between city developer Court Collaboration and Hong Kong-based Headland Developments.
Plans for flats at the six-storey office block were withdrawn in summer 2011 before being re-submitted and then rejected by the council in spring 2012, amid claims it could have led to the closure of the Cadbury factory.
The chocolate giant still opposes the scheme but new permitted development rights open the door to a residential project.
It becomes the first Birmingham project backed by Headland but managing director John Gunning said he had more plans to invest in the city.
He told the Post: "The momentum of HS2 in the last four months has grown from what looked like a paperwork plan to something really significant.
"It just seems like the right time because after years of lagging behind the city is on a positive trajectory."
Work is set to start on Franklin House, in Bournville Lane, in a matter of months and is expected to be completed by Christmas 2015, with one- and two-bedroom apartments.
Court Collaboration founder Anthony McCourt said that, while plans for the 1960s office block would be a high-end scheme, it would be affordable for first-time buyers.
It is the second Chinese-backed property development headed up by the 31-year-old, after 130 Colmore Row, which was let to popular restaurant chain Lasan last month.
Mr McCourt, himself a Bournville resident, said it was a rare opportunity for a large scheme to come on the market in the model village and he planned a landmark scheme.
Among the plans are a residents-only outlet, such as a coffee shop, and a car park for 50 vehicles.
He said: "You don't get many chances to build something like this in Bournville. Opportunities like this don't come round every day.
"This is not just about putting apartments in, the plan is to maximise the opportunities for residential including leisure space.
"We are going to think very hard about how people want to live in the city's suburbs and deliver what modern people want – maybe a quality coffee shop for residents.
"There will be one and two-bedroom apartments and access to a roof terrace garden across from the factory. It is going to be idyllic."
Mr McCourt said the scheme was funded by Chinese investors who were "putting in several million pounds of their own, hard-earned, cash".
Earlier plans for Franklin House came before Birmingham City Council in 2012 but were rejected over fears of noise from the nearby chocolate factory.
At the time, applicant First Property General Partner threatened to take the planning committee to appeal, in a bid to force through the proposals.
It also drew objections from Cadbury's then owner Kraft, although the developer was at pains to point out that the factory is surrounded by residential properties.
A spokesperson for Cadbury owner Mondelez International said it opposed the development.
He said: "We have not changed our position since 2012 and we still believe it is inappropriate to have residential property so close to a factory which is running 24 hours a day, seven days a week.
"Although it is not fully operational throughout the night, it is working around the clock and people need to be aware."
Mr Gunning, originally from Selly Oak, said UK investments were a new focus for the firm and he believed there were strong prospects in Birmingham.
He added: "Birmingham has a large Chinese population and that certainly had a bearing on our decision. One of the investors has connections here and they will have picked up the phone and given them a call, asked for advice and got the thumbs up."