I've been at the market many times in the summer season (think: sun, summer afternoons and evenings, live music, organic produce and samples) and found it wonderful! The market is surrounded by a lovely garden and located right by the Toronto Botanical Gardens, which boasts lush gardens and a small cafe (organic!!) to satiate any mid-walk cravings. The building by the Farmers' Market is an air-conditioned botanical library (one of the biggest in Canada, I believe).
The great thing about this market is its size. The farmers are very nice and will answer any questions you may have. Right by the stands are patio tables and chairs where you can enjoy your meal. Sometimes in the summer, they'll have people dressed in vegetable costumes and samples to go around.
Now that winter has hit Toronto, the market has moved operations indoors. Think of a cozy cafe - that's what it's like inside the building.
Great place, great people, and great food.