The Toronto Botanical Gardens is made up of many smaller themed gardens and spaces like the Teaching Garden, Arrival Courtyard, and Beryl Ivey Knot Garden so there's usually a different favourite section for everyone.
You may want to avoid attending during summer weekends as this is prime wedding picture season, parking is usually very limited. I suggest taking the TTC Lawrence bus #54 as it stops right at the corner of Leslie and Lawrence.
I like dropping by during the fall as the temperatures are cooler and the autumn foliage is stunning, especially the Japanese maples!