The head of a Birmingham radio station was accused of "a form of mismanagement" by the Charity Commission after tens of thousands of pounds collected for victims of South Asian disasters was kept in a bank account for more than five years.
But Dr Arun Bajaj, the founder of the Radio XL Asian music channel and chairman of the Institute of Asian Business, told The Birmingham Post he had kept hold of the money to make sure it was distributed fairly.
And he added the commission's order to hand over the money immediately mean it could be misused.
The commission launched two inquiries after the Wolverhampton-based Divine Onkar Mission charity complained in 2001 it had not been given nearly £30,000 collected on its behalf for victims of cyclones in the Indian state of Orissa.
During its investigations, the commission found a total of £123,000 raised for various charities had been kept in an account run by Dr Bajaj instead of being handed over to charities. This money had been raised for reconstruction after various disasters, including the 2004 Indian Ocean tsunami and the earthquake in Kashmir the year after.
The commission said there was no question of Dr Bajaj misusing the money, but claimed "the failure to release funds constituted a form of mismanagement", and said it felt justified issuing a demand for the money to be handed over immediately.
But Dr Bajaj said he had promised the money to the charities only after specific reconstruction work in the areas affected had been carried out. He said: "A lot of the work was being done in India and Pakistan, and how are you going to control it out there?
"There were long-term projects like building an orphanage and setting up a school being carried out. But it's very difficult to control, especially in India, with all the corruption.
"There's never been any question of me misusing this money, even when I went out there I paid for my own flights and expenses. This has been all my own extra work, I don't show off about it, but it's disheartening to be treated like a criminal when trying to do good."
He said he felt honour-bound to ensure the promises made to Radio XL listeners were kept.