The Botanical Garden is a place I like to return to often and spend countless hours in relative silence.
Check out the First Nations Garden, the gardens along Pie IX (section 6), the Chinese garden, Japanese garden and the Rose garden. Wander through their greenhouses and be amazed at the beauty of the collection.
The setting is lovely and the plants, surprisingly, don't rush you around.
There's no better place to release your inner misanthrope and spend some precious time in peace and the company of silent beautiful friends with fancy Latin names.
When to visit: during the Lantern festival in October. Spend the afternoon wandering the gardens, then at nightfall stroll to the Chinese garden and see Chinese lantern art at its best.
When not to visit: when it's still too soon after winter. I don't think anything gets going before mid-April at least.