If you've only got a few days in Toronto, it's definitely worth stopping by the bar at Canoe restaurant. It's on the 54th floor of the TD Bank building downtown and for a pricey drink you get a view that is well worth the money, overlooking the lake, stadiums, CN tower and Union station. I think the best part about Canoe is that the service is really stellar. They really make you feel welcome in an unobtrusive way, even if you're not super dressed up or look like you have money. I've been here twice now, years apart, and what I remember most are the views and just how lovely the service was.
The bar isn't huge, so this wouldn't work for large groups. It's better for a quiet drink with a few friends, as you each sit in a stool overlooking a grand drop down to the city below.