The star, 38, who has an estimated fortune of £150 million, said that he finds it "hard to talk about modern-day poverty".
He cited a family he met while making one of his previous TV shows who ate unhealthy, fast food but had splashed out on a huge TV. Oliver said that poor communities in other countries had a better grasp of good food.
He told the Radio Times: "Some of the most inspirational food in the world comes from areas where people are financially challenged. The flavour comes from a cheap cut of meat, or something that's slow-cooked, or an amazing texture's been made out of leftover stale bread."
The campaigning chef added: "I'm not judgmental, but I've spent a lot of time in poor communities, and I find it quite hard to talk about modern-day poverty. You might remember that scene in Ministry Of Food, with the mum and the kid eating chips and cheese out of Styrofoam containers, and behind them is a massive TV. It just didn't weigh up."
Oliver, whose new Channel 4 show, Jamie's Money Saving Meals, is designed to help people save on their food bill, added: "The fascinating thing for me is that seven times out of 10, the poorest families in this country choose the most expensive way to hydrate and feed their families. The ready meals, the convenience foods."
The Naked Chef said: "I meet people who say, 'You don't understand what it's like.' I just want to hug them and teleport them to the Sicilian street cleaner who has 25 mussels, 10 cherry tomatoes, and a packet of spaghetti for 60 pence, and knocks out the most amazing pasta. You go to Italy or Spain and they eat well on not much money. We've missed out on that in Britain, somehow."
The Child Poverty Action Group (CPAG) insisted that low income can be a barrier to healthy eating. Imran Hussain, the group's head of policy, said: "Jamie Oliver has made a huge contribution to improving school meals and we're grateful for the support his foundation has given us in our work on free school meals.
"He is right to say that healthy food doesn't always have to be expensive - one of CPAG's ambassadors, the food blogger Jack Monroe, is an excellent example of this - but for many families it's low income which gets in the way of healthy eating."
A Department for Work and Pensions spokesman said: "The Government has taken action to help families with the cost of living, including increasing the tax-free personal allowance to £10,000, which will save a typical taxpayer over £700; freezing council tax for five years; and freezing fuel duty. Our welfare reforms with the introduction of Universal Credit will make three million households better off - the majority of these from the bottom two-fifths of the income scale."