The drive to make school food healthier has led children to abandon canteen meals and buy their lunch elsewhere.
The Government-backed School Food Trust found fewer children were eating school dinners in the wake of controversy over the cheap junk food which was being served.
Caterers blamed the decline on the media coverage of school meals in the past two years and the fact that lunches now had more "healthy" options.
TV chef Jamie Oliver mounted a national campaign to improve school dinners after his Channel 4 series revealed how unhealthy many canteen meals were.
Ministers responded with a £220 million package of measures to improve school meals, which included setting up the School Food Trust to promote reform.
Dame Suzi Leather, chair of the Trust, said: "It is no surprise to us to see a small drop in take up, considering the huge amount of negative coverage school meals have received in the last 18 months.
"However, what is really encouraging is that some schools who are now offering healthier meals have seen take-up increase."
The Trust's report, based on a survey of local authorities in England, said two-thirds of primary school caterers and half of secondary caterers reported a drop in the take-up of school meals.
There was a six per cent decline in take-up in primary schools and 5 per cent decline in secondary schools between the 2004-05 and 2005-06 financial years.
Some areas saw a drop of 23 per cent, while others saw a 9 per cent increase in the take-up of school dinners.
In primary schools, the fact that cooks were serving "more healthy options" resulted in "more pupils bringing in packed lunches instead, the report said.
"In secondary schools, the same reason was felt to have resulted in more pupils buying meals elsewhere."
The School Food Trust was set up last year as ministers responded to a public outcry over the quality of food served.
It has been set a target for increasing take-up of school meals.
The Government has announced that junk food will be banned from canteens and vending machines while lunches will have to contain more fresh fruit and vegetables and less fat, sugar and salt.
The Trust's report raised concerns that one in five primary schools had no facilities for preparing food.