For fresh, cheap fruit and vegetables that last longer than those you get at the supermarket, head to Tennessee Orchard. Tennessee Orchard is primarily an apple orchard which is open to the public. Their range of apples changes with the seasons. They also sell a wide variety of fresh vegetables which are firm and tasty. They also stock local honey, jams and chutneys, along with apple juice. Prices are much cheaper than the supermarkets and the produce lasts MUCH longer than the regular markets. It's a family-run shop, very basic but very friendly. Their hours are Friday - Sunday, 8am-4pm.
I just buy what's fresh and readily available. However, if you buy produce near the end of season, it's usually a lot cheaper, especially the apples.