I agree about some tropical fruits (mangoes and durians especially), but I've had bananas straight off the tree in Southeast Asia and they really didn't taste any different to me.

Bananas are an exception because they are actually best picked green and allowed to ripen off the tree. I don't recall what goes wrong if you don't. This makes them the rare fruit that is perfectly happy with industrial production and shipping.

Another such exception is avocados: http://en.wikipedia.org/wiki/Avocado#Harvest_and_postharvest (second paragraph under Harvest and Postharvest)